Target detection method and device, electronic device, unmanned vehicle and storage medium

Abstract

This disclosure relates to a target detection method and apparatus, electronic device, unmanned vehicle, and storage medium. The target detection method includes: acquiring image signals and lidar signals; determining a classification prediction map, pose prediction map, and depth information prediction map of a target object based on the image signals; correcting the depth information prediction map of the target object based on the lidar signals; and determining the detection output result of the three-dimensional target object based on the classification prediction map, pose prediction map, and corrected depth information prediction map. This disclosure, by introducing lidar signals as auxiliary input to the model, can compensate for the deficiencies of a single image signal, resulting in a more accurate 3D target detection output.

Description

Technical Field

[0001] This disclosure relates to the field of target detection, particularly to the field of autonomous driving, and specifically to a target detection method and apparatus, electronic equipment, autonomous vehicle, and storage medium. Background Technology

[0002] Currently, autonomous vehicles are used to automatically transport people or goods from one location to another. These vehicles collect environmental information through onboard sensors and complete the automated transport. Autonomous delivery vehicles controlled by autonomous driving technology greatly improve the convenience of production and daily life, and save labor costs.

[0003] In the fields of autonomous driving or self-driving cars, 3D object detection is typically required using image signals. Typical 3D object detection methods use images captured by a monocular camera as real-time input, and then use a neural network model to predict the 3D pose information of target objects in the scene, achieving intelligent scene perception. Most related 3D object detection methods rely on a single image input signal, using a model to extract features to obtain a scene classification prediction map, a pose prediction map, and a target object center depth prediction map. These are then decoded to obtain the final detection output. Summary of the Invention

[0004] The inventors discovered through research that the biggest challenge of related 3D target detection methods is the inability to obtain an accurate prediction map of the target object's center depth due to the lack of scene depth information in the image, resulting in very low model accuracy.

[0005] In view of at least one of the above technical problems, this disclosure provides a target detection method and apparatus, electronic device, unmanned vehicle and storage medium. By introducing LiDAR signal as an auxiliary input to the model, the defects of a single image signal can be made up for, and a more accurate 3D target detection output can be obtained.

[0006] According to one aspect of this disclosure, a target detection method is provided, comprising:

[0007] Acquire image signals and lidar signals;

[0008] Based on the image signal, determine the target object's classification prediction map, pose prediction map, and depth information prediction map;

[0009] The depth information prediction map of the target object is corrected based on the lidar signal;

[0010] The detection output of the 3D target object is determined based on the target's classification prediction map, pose prediction map, and corrected depth information prediction map.

[0011] In some embodiments of this disclosure, the depth information prediction map includes a surface depth prediction map of the scene where the target object is located.

[0012] In some embodiments of this disclosure, the correction of the depth information prediction map of the target object based on the lidar signal includes:

[0013] The depth correction map is determined based on the surface depth prediction map of the scene where the target object is located and the lidar signal;

[0014] The depth information prediction map of the target object is corrected based on the depth correction map.

[0015] In some embodiments of this disclosure, determining the depth correction map based on the surface depth prediction map of the scene where the target object is located and the lidar signal includes:

[0016] The predicted radar signal is determined based on the surface depth prediction map of the scene where the target object is located.

[0017] Low-pass filtering is applied to the predicted radar signal and the lidar signal to obtain the smoothed predicted radar signal and the smoothed lidar signal.

[0018] Differential low-pass filtering is performed on the predicted radar smoothed signal and the lidar smoothed signal to determine the depth correction map.

[0019] In some embodiments of this disclosure, determining the predicted radar signal based on the surface depth prediction map of the scene where the target object is located includes:

[0020] The predicted radar signal is determined based on the surface depth prediction map of the scene where the target object is located and the mask signal of the radar signal.

[0021] In some embodiments of this disclosure, the step of performing differential low-pass filtering on the predicted radar smoothing signal and the lidar smoothing signal to determine the depth correction map includes:

[0022] The difference between the predicted radar smoothed signal and the lidar smoothed signal is calculated to determine the difference map;

[0023] The difference map is low-pass filtered to determine the depth correction map.

[0024] In some embodiments of this disclosure, the depth information prediction map further includes a center depth prediction map of the target object.

[0025] In some embodiments of this disclosure, determining the classification prediction map, pose prediction map, and depth information prediction map of the target object based on the image signal includes:

[0026] Based on the image signal, a classification prediction map of the target object, a pose prediction map of the target object, and a surface depth prediction map of the scene in which the target object is located are determined.

[0027] Based on the target object's pose prediction map and the surface depth prediction map of the scene in which the target object is located, determine the target object's center depth prediction map.

[0028] In some embodiments of this disclosure, correcting the depth information prediction map of the target object based on the depth correction map includes:

[0029] The center depth prediction map of the target object is corrected based on the depth correction map to obtain the corrected center depth prediction map.

[0030] In some embodiments of this disclosure, the step of determining the detection output result of the three-dimensional target object based on the target's classification prediction map, pose prediction map, and corrected depth information prediction map includes:

[0031] The target interface is decoded from the target classification prediction map, pose prediction map and corrected target object center depth prediction map to obtain the target object pose information.

[0032] The target object's pose information and the predicted surface depth map of the scene in which the target object is located are output as the detection results.

[0033] In some embodiments of this disclosure, acquiring the lidar signal includes:

[0034] The sparse depth map, which is obtained by projecting the raw LiDAR scanning data onto the image coordinate system based on the camera's extrinsic and intrinsic parameters, is used as the LiDAR signal.

[0035] According to another aspect of this disclosure, a target detection apparatus is provided, comprising:

[0036] The signal acquisition module is configured to acquire image signals and lidar signals;

[0037] The prediction map determination module is configured to determine the classification prediction map, pose prediction map, and depth information prediction map of the target object based on the image signal.

[0038] The depth correction module is configured to correct the depth information prediction map of the target object based on the lidar signal;

[0039] The detection result determination module is configured to determine the detection output result of the three-dimensional target object based on the target's classification prediction map, pose prediction map, and corrected depth information prediction map.

[0040] In some embodiments of this disclosure, the depth information prediction map includes a surface depth prediction map of the scene where the target object is located.

[0041] In some embodiments of this disclosure, the depth correction module includes a correction map determination unit and a depth correction unit, wherein:

[0042] The correction map determination unit is configured to determine a depth correction map based on the surface depth prediction map of the scene where the target object is located and the lidar signal;

[0043] The depth correction unit is configured to correct the depth information prediction map of the target object based on the depth correction map.

[0044] In some embodiments of this disclosure, the correction map determination unit is configured to determine a predicted radar signal based on a surface depth prediction map of the scene where the target object is located; perform low-pass filtering on the predicted radar signal and the lidar signal to obtain a predicted radar smoothed signal and a lidar smoothed signal; and perform differential low-pass filtering on the predicted radar smoothed signal and the lidar smoothed signal to determine a depth correction map.

[0045] In some embodiments of this disclosure, the correction map determination unit is configured to determine the predicted radar signal based on the surface depth prediction map of the scene where the target object is located and the mask signal of the radar signal, when the predicted radar signal is determined based on the surface depth prediction map of the scene where the target object is located.

[0046] In some embodiments of this disclosure, the calibration map determination unit is configured to, when determining a depth calibration map by performing differential low-pass filtering on the predicted radar smoothing signal and the lidar smoothing signal, calculate the difference between the predicted radar smoothing signal and the lidar smoothing signal to determine a differential map; and perform low-pass filtering on the differential map to determine the depth calibration map.

[0047] In some embodiments of this disclosure, the depth information prediction map further includes a center depth prediction map of the target object.

[0048] In some embodiments of this disclosure, the prediction map determination module is configured to determine a classification prediction map of the target object, a pose prediction map of the target object, and a surface depth prediction map of the scene where the target object is located based on the image signal; and to determine a center depth prediction map of the target object based on the pose prediction map of the target object and the surface depth prediction map of the scene where the target object is located.

[0049] In some embodiments of this disclosure, the depth correction unit is configured to correct the center depth prediction map of the target object based on the depth correction map to obtain a corrected center depth prediction map.

[0050] In some embodiments of this disclosure, the detection result determination module is configured to decode the target interface of the target's classification prediction map, pose prediction map and corrected target object center depth prediction map to obtain target object pose information; and output the target object pose information and the surface depth prediction map of the scene where the target object is located as the detection result.

[0051] In some embodiments of this disclosure, the signal acquisition module is configured to project the raw LiDAR scanning data onto the image coordinate system as a sparse depth map, based on camera extrinsic and intrinsic parameters, as the LiDAR signal.

[0052] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0053] Memory, used to store instructions;

[0054] A processor is configured to execute the instructions, causing the electronic device to perform operations implementing the target detection method as described in any of the above embodiments.

[0055] According to another aspect of this disclosure, an unmanned vehicle is provided, including a target detection device as described in any of the above embodiments, or including an electronic device as described in any of the above embodiments.

[0056] According to another aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the target detection method as described in any of the above embodiments.

[0057] This disclosure, by introducing LiDAR signals as auxiliary input to the model, can compensate for the deficiencies of a single image signal and obtain a more accurate 3D target detection output. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 These are schematic diagrams of some embodiments of the target detection method disclosed herein.

[0060] Figure 2a The diagram illustrates some other embodiments of the target detection method disclosed herein.

[0061] Figure 2b This is a schematic diagram of image signals in some embodiments of this disclosure.

[0062] Figure 2c This is a schematic diagram of lidar signals in some embodiments of this disclosure.

[0063] Figure 2d This is a schematic diagram of the surface depth prediction map of the scene where the target object is located in some embodiments of this disclosure.

[0064] Figure 2e This is a schematic diagram illustrating the prediction of radar signals in some embodiments of this disclosure.

[0065] Figure 3 These are schematic diagrams of some embodiments of the target detection device disclosed herein.

[0066] Figure 4 This is a schematic diagram of some embodiments of the depth correction module of this disclosure.

[0067] Figure 5 This is a schematic diagram of the structure of some embodiments of the electronic device disclosed herein.

[0068] Figure 6 This is a schematic diagram of the system architecture of some embodiments of the unmanned vehicle disclosed herein.

[0069] Figure 7 This is a schematic diagram of the hardware structure of some embodiments of the unmanned vehicle disclosed herein. Detailed Implementation

[0070] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0071] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0072] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0073] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0074] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0075] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0076] The inventors discovered through research that the technical problem with the related technology lies in:

[0077] 1. Inaccurate depth estimation: Because conventional cameras cannot capture the depth information of a scene, it is impossible to obtain an accurate depth prediction map from a single image signal. Since depth prediction is the most critical factor affecting the performance of 3D object detection models, current technologies often fail to achieve good detection results.

[0078] 2. Difficulty in Guaranteeing Synchronization Between Radar and Camera: To obtain better detection results, in addition to the image signal, extra sensor signals need to be introduced as auxiliary inputs for 3D object detection. The most effective method is to introduce the LiDAR scanning signal and project it onto the image coordinate system through the camera's extrinsic and intrinsic parameter matrices to obtain a sparse depth ground truth supervision signal. However, in autonomous driving scenarios, extrinsic and intrinsic parameters are difficult to calibrate accurately, and the triggering times of the radar and camera are difficult to synchronize. Therefore, the obtained depth ground truth signal is misaligned with the original image signal. This makes it difficult to use the LiDAR signal to assist in image prediction correction.

[0079] 3. Limited Model Output: Existing 3D object detection methods often only output sparse object poses and cannot perceive more scene geometric information (such as dense depth maps of the scene).

[0080] In view of at least one of the above technical problems, this disclosure provides a target detection method and apparatus, electronic device, unmanned vehicle and storage medium. The above embodiments of this disclosure will be described below through specific examples.

[0081] Figure 1 These are schematic diagrams of some embodiments of the target detection method disclosed herein. Figure 2a The diagram illustrates other embodiments of the target detection method of this disclosure. Preferably, this embodiment can be performed by the target detection device or the unmanned vehicle of this disclosure. Figure 1 and Figure 2a As shown, the method may include at least one of steps 100-400, wherein:

[0082] Step 100: Acquire image signals and lidar signals.

[0083] Figure 2b This is a schematic diagram of image signals in some embodiments of this disclosure. Figure 2c This is a schematic diagram of lidar signals in some embodiments of this disclosure.

[0084] In some embodiments of this disclosure, step 100, the step of acquiring the lidar signal, may include: projecting the original lidar scanning data onto the image coordinate system based on camera extrinsic and intrinsic parameters to obtain a sparse depth map, which is then used as the lidar signal.

[0085] In some embodiments of this disclosure, image signals are used to obtain multiple predicted maps of a scene, and radar signals are used to correct the "target center depth predicted map" therein.

[0086] In some embodiments of this disclosure, the input to the target detection model includes image signals and lidar signals. The lidar signal is a sparse depth map generated by projecting the raw scan data onto the image coordinate system based on camera extrinsic and intrinsic parameters.

[0087] In some embodiments of this disclosure, such as Figure 2a As shown, the lidar signal is the input true value lidar signal.

[0088] In some embodiments of this disclosure, the target detection device can be installed in driverless vehicles or autonomous vehicles such as unmanned cars.

[0089] In some embodiments of this disclosure, image signals are acquired by cameras on unmanned or autonomous vehicles, and raw scan data of lidar signals are acquired by lidar on unmanned or autonomous vehicles.

[0090] In some embodiments of this disclosure, the target detection method of this disclosure can be applied to the fields of unmanned driving or autonomous driving.

[0091] Step 200: Determine the classification prediction map, pose prediction map, and depth information prediction map of the target object based on the image signal.

[0092] In some embodiments of this disclosure, the depth information prediction map may include a surface depth prediction map of the scene where the target object is located and a center depth prediction map of the target object.

[0093] In some embodiments of this disclosure, such as Figure 2a As shown, step 200, namely, the step of determining the classification prediction map, pose prediction map, and depth information prediction map of the target object based on the image signal, may include at least one of steps 210 and 220, wherein:

[0094] Step 210: Determine the classification prediction map of the target object, the pose prediction map of the target object, and the surface depth prediction map of the scene where the target object is located based on the image signal. Figure 2d This is a schematic diagram of the surface depth prediction map of the scene where the target object is located in some embodiments of this disclosure.

[0095] In some embodiments of this disclosure, the image signal is used to obtain multiple prediction maps of a scene, such as a classification prediction map of the target object, a pose prediction map of the target object, and a surface depth prediction map of the scene in which the target object is located.

[0096] In some embodiments of this disclosure, step 210 may include: the model first extracts features from the input image to obtain three prediction map outputs (classification, pose, and depth).

[0097] In some embodiments of this disclosure, classification, pose, and depth are the outputs of a 3D object detection network. Classification is the category to which an object point belongs, such as car / pedestrian / bicycle. Pose is the position and orientation of the object; position refers to the offset in the 2D plane, and orientation includes predicted length, width, and height dimensions and predicted rotation angle, such as a 20-degree left turn. Depth is the distance between the object and the shooting point.

[0098] In some embodiments of this disclosure, step 210 may include: the model extracts features through the convolution operation of a convolutional neural network to obtain a prediction map output.

[0099] In some embodiments of this disclosure, the neural network model can be a stack of convolutional neural networks (CNNs).

[0100] Step 220: Determine the center depth prediction map of the target object based on the pose prediction map of the target object and the surface depth prediction map of the scene where the target object is located.

[0101] In some embodiments of this disclosure, step 220 may include: the "pose prediction map" and the "depth prediction map" are then further calculated by the model to obtain the "target center depth prediction map".

[0102] In some embodiments of this disclosure, Figure 2a The depth prediction map in this embodiment is a surface depth prediction map of the scene where the target object is located.

[0103] In some embodiments of this disclosure, Figure 2a In this embodiment, the depth refers to the surface depth of the scene where the target object is located, that is, the distance between the object surface and the entire scene surface and the shooting point.

[0104] In some embodiments of this disclosure, the target center depth is the distance between the geometric center of the object and the shooting point. Depth is the distance between the object surface and the shooting point.

[0105] In some embodiments of this disclosure, the center depth is obtained from the depth based on the rotation angle and length, width and height dimensions of the object, wherein the rotation angle and length, width and height dimensions of the object are in the pose prediction map.

[0106] Step 300: Correct the depth information prediction map of the target object based on the lidar signal.

[0107] In some embodiments of this disclosure, "classification," "pose," and "target center depth" are essential information for decoding 3D targets, while "depth" serves as an intermediate output auxiliary signal for correcting the "target center depth." Since the accuracy of the "target center depth" obtained from the image is very low, radar signals need to be introduced for correction.

[0108] In some embodiments of this disclosure, such as Figure 2a As shown, step 300, namely, the step of correcting the depth information prediction map of the target object based on the lidar signal, may include at least one of steps 310 and 320, wherein:

[0109] Step 310: Determine the depth correction map based on the surface depth prediction map of the scene where the target object is located and the lidar signal.

[0110] In some embodiments of this disclosure, such as Figure 2a As shown, step 310, namely, the step of determining the depth correction map based on the surface depth prediction map of the scene where the target object is located and the lidar signal, may include at least one of steps 311-313, wherein:

[0111] Step 311: Determine the predicted radar signal based on the surface depth prediction map of the scene where the target object is located.

[0112] Figure 2e This is a schematic diagram illustrating the prediction of radar signals in some embodiments of this disclosure.

[0113] In some embodiments of this disclosure, the error between the predicted radar signal and the "input true radar signal" can represent the error between the "target center depth prediction map" and the true target center depth.

[0114] In some embodiments of this disclosure, step 311 may include: determining the predicted radar signal based on the surface depth prediction map of the scene where the target object is located and the mask signal of the radar signal.

[0115] In some embodiments of this disclosure, step 311 may include multiplying the "depth prediction map" by a 0-1 mask of the radar signal to obtain a sparse "predicted radar signal".

[0116] In some embodiments of this disclosure, step 311 may include: multiplying the "depth prediction map" element-wise with a 0-1 mask of the radar signal. Multiplying by a point in the mask that is 0 results in a 0, indicating an invalid point. Multiplying by a point in the mask that is 1 retains the original value, indicating a valid point. A signal in which valid points occupy less than a predetermined proportion of all points is considered a "sparse" signal.

[0117] Step 312: Perform low-pass filtering on the predicted radar signal and the lidar signal respectively to obtain the predicted radar smoothed signal and the lidar smoothed signal.

[0118] In some embodiments of this disclosure, the low-pass filter can be a mean filter.

[0119] In other embodiments of this disclosure, the low-pass filter can be a low-pass filter employing Gaussian filtering, bilateral filtering, median filtering, maximum-minimum filtering, or other similar methods.

[0120] Due to practical issues, the pixels of the "predicted radar signal" and the "input true radar signal" are not aligned, which can interfere with the calibration.

[0121] This application applies low-pass filtering to the predicted radar signal and lidar signal to obtain a smoothed radar signal. Low-pass filtering can increase the robustness of the radar signal to pixel misalignment.

[0122] Step 313: Perform differential low-pass filtering on the predicted radar smoothed signal and the lidar smoothed signal to determine the depth correction map.

[0123] In some embodiments of this disclosure, step 313 may include: calculating the difference between the predicted radar smoothing signal and the lidar smoothing signal to determine a difference map; and performing low-pass filtering on the difference map to determine a depth correction map.

[0124] In some embodiments of this disclosure, the difference map can represent the prediction error of the target center depth. Because radar signals are sparse, the difference map needs to be further low-pass filtered to obtain a smooth, dense error map, i.e., a "depth correction map".

[0125] The purpose of low-pass filtering in step 312 of this disclosure is to alleviate the problem of pixel misalignment. The purpose of low-pass filtering in step 312 of this disclosure is to change sparse to dense.

[0126] Step 320: Correct the center depth prediction map of the target object based on the depth correction map to obtain the corrected center depth prediction map of the target object.

[0127] In some embodiments of this disclosure, step 320 may include: summing the depth correction map with the center depth prediction map of the target object to obtain the corrected center depth prediction map of the target object.

[0128] Step 400: Determine the detection output result of the three-dimensional target object based on the target's classification prediction map, pose prediction map, and corrected depth information prediction map.

[0129] In some embodiments of this disclosure, step 400 may include: decoding the target interface of the target's classification prediction map, pose prediction map and corrected target object center depth prediction map to obtain target object pose information; and outputting the target object pose information and the surface depth prediction map of the scene where the target object is located as detection results.

[0130] In some embodiments of this disclosure, step 400 may include: the "corrected center depth map", the "classification prediction map" and the "pose prediction map" are used together as input to the decoder to obtain the final 3D object detection output result.

[0131] In some embodiments of this disclosure, the final output of the target detection, that is, the decoded content, includes the 3D world coordinates (x, y, z) of the geometric center of the target object, as well as the length, width, and height of the object.

[0132] In the above embodiments of this disclosure, the intermediately generated "depth prediction map" is also used as the model output to achieve multi-level perception of scene geometric information.

[0133] The input to the model in the above embodiments of this disclosure includes two parts: image signals and lidar signals. The image signals are used to obtain multiple predicted maps of the scene, and the lidar signals are used to correct the "target center depth prediction map" among them. The corrected "target center depth map" is then compared with other predicted maps. Figure 1 The target is used as input to execute the target decoding algorithm, resulting in the final 3D target detection output. In addition, the "depth prediction map" output from the intermediate output of the model in the above embodiments of this disclosure is also output as a deep perception of scene geometric information.

[0134] The above embodiments of this disclosure provide a 3D target detection method based on lidar correction in autonomous driving.

[0135] The embodiments disclosed above can improve the prediction effect of target center depth without significantly increasing model complexity, and obtain more accurate detection results.

[0136] The embodiments disclosed above introduce lidar signals as auxiliary inputs to the model, thereby compensating for the deficiencies of a single image signal and obtaining more accurate 3D target detection output.

[0137] The embodiments disclosed above propose a solution to the misalignment between lidar signals and image signals, thereby improving the robustness of the enhancement model and achieving efficient utilization of lidar signals.

[0138] In addition to the target object pose information, the embodiments of this disclosure output a dense depth prediction map of the scene, thereby improving the scene perception capability.

[0139] Figure 3 These are schematic diagrams illustrating some embodiments of the target detection device disclosed herein. For example... Figure 3 As shown, the target detection device of this disclosure may include a signal acquisition module 500, a prediction map determination module 600, a depth correction module 700, and a detection result determination module 800, wherein:

[0140] The signal acquisition module 500 is configured to acquire image signals and lidar signals.

[0141] In some embodiments of this disclosure, the signal acquisition module 500 may be configured to project the raw LiDAR scanning data onto the image coordinate system based on camera extrinsic and intrinsic parameters into a sparse depth map, which serves as the LiDAR signal.

[0142] The prediction map determination module 600 is configured to determine the classification prediction map, pose prediction map, and depth information prediction map of the target object based on the image signal.

[0143] In some embodiments of this disclosure, the depth information prediction map may include a surface depth prediction map of the scene where the target object is located and a center depth prediction map of the target object.

[0144] In some embodiments of this disclosure, the prediction map determination module 600 is configured to determine a classification prediction map of the target object, a pose prediction map of the target object, and a surface depth prediction map of the scene where the target object is located based on the image signal; and to determine a center depth prediction map of the target object based on the pose prediction map of the target object and the surface depth prediction map of the scene where the target object is located.

[0145] The depth correction module 700 is configured to correct the depth information prediction map of the target object based on the lidar signal.

[0146] The detection result determination module 800 is configured to determine the detection output result of the three-dimensional target object based on the target's classification prediction map, pose prediction map, and corrected depth information prediction map.

[0147] Figure 4 These are schematic diagrams illustrating some embodiments of the depth correction module of this disclosure. For example... Figure 4 As shown, the depth correction module of this disclosure (e.g.) Figure 3 The depth correction module 700 in the embodiment may include a correction map determination unit 710 and a depth correction unit 720, wherein:

[0148] The correction map determination unit 710 is configured to determine a depth correction map based on the surface depth prediction map of the scene where the target object is located and the lidar signal.

[0149] In some embodiments of this disclosure, the correction map determination unit 710 may be configured to determine a predicted radar signal based on a surface depth prediction map of the scene where the target object is located; perform low-pass filtering on the predicted radar signal and the lidar signal to obtain a predicted radar smoothed signal and a lidar smoothed signal; and perform differential low-pass filtering on the predicted radar smoothed signal and the lidar smoothed signal to determine a depth correction map.

[0150] In some embodiments of this disclosure, the low-pass filter can be a mean filter.

[0151] In other embodiments of this disclosure, the low-pass filter can be a low-pass filter employing Gaussian filtering, bilateral filtering, median filtering, maximum-minimum filtering, or other similar methods.

[0152] In some embodiments of this disclosure, the correction map determination unit 710 may be configured to determine the predicted radar signal based on the surface depth prediction map of the scene where the target object is located and the mask signal of the radar signal, when the predicted radar signal is determined based on the surface depth prediction map of the scene where the target object is located.

[0153] In some embodiments of this disclosure, the correction map determination unit 710 may be configured to, when performing differential low-pass filtering on the predicted radar smoothing signal and the lidar smoothing signal to determine the depth correction map, calculate the difference between the predicted radar smoothing signal and the lidar smoothing signal to determine the differential map; and perform low-pass filtering on the differential map to determine the depth correction map.

[0154] The depth correction unit 720 is configured to correct the center depth prediction map of the target object based on the depth correction map, so as to obtain the corrected center depth prediction map of the target object.

[0155] In some embodiments of this disclosure, such as Figure 3 As shown, the detection result determination module 800 can be configured to decode the target interface by analyzing the target's classification prediction map, pose prediction map, and corrected target object center depth prediction map to obtain the target object pose information; and output the target object pose information and the surface depth prediction map of the scene where the target object is located as the detection result.

[0156] In some embodiments of this disclosure, the target detection device is used to perform any of the embodiments described above (e.g., Figures 1-2a The operation of the target detection method described in any embodiment.

[0157] The embodiments of this disclosure introduce additional LiDAR signals as auxiliary input, thereby enabling the correction of the depth information prediction map and achieving higher 3D target detection accuracy. The entire correction process of the embodiments of this disclosure is convenient to implement, does not affect the overall model processing time, and ensures real-time performance.

[0158] To address the misalignment between the image and the radar signal, the embodiments of this disclosure employ a low-pass filtering smoothing method to mitigate the issue and improve the robustness of the model.

[0159] In addition to the conventional target object detection results, the embodiments of this disclosure can also output dense depth prediction values ​​for the entire scene, thereby achieving deep perception of scene geometric information.

[0160] Figure 5 This is a schematic diagram illustrating the structure of some embodiments of the electronic device disclosed herein. For example... Figure 5 As shown, the electronic device includes a memory 51 and a processor 52.

[0161] The memory 51 is used to store instructions, and the processor 52 is coupled to the memory 51. The processor 52 is configured to execute instructions stored in the memory to implement the methods involved in the above embodiments.

[0162] like Figure 5 As shown, the electronic device also includes a communication interface 53 for exchanging information with other devices. Additionally, the electronic device includes a bus 54, through which the processor 52, communication interface 53, and memory 51 communicate with each other.

[0163] The memory 51 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive. The memory 51 may also be a memory array. The memory 51 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.

[0164] Furthermore, processor 52 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0165] According to another aspect of this disclosure, an unmanned vehicle is provided, comprising any of the embodiments described above (e.g., Figure 3 or Figure 4 The target detection device described in the embodiments, or including any of the embodiments described above in this disclosure (e.g., Figure 5 The electronic device described in the embodiment).

[0166] Figure 6 This is a schematic diagram of the system architecture of some embodiments of the unmanned vehicle disclosed herein. Figure 7 This is a schematic diagram of the hardware structure of some embodiments of the unmanned vehicle disclosed herein. (See attached diagram.) Figure 6 and Figure 7 As shown, the unmanned vehicle mainly consists of four parts: chassis module 1, autonomous driving module 2, cargo box module 3, and remote monitoring and propulsion module 4.

[0167] like Figure 6 As shown, the autonomous driving module 2 may include a core processing unit (Orin or Xavier module) 21, a traffic light recognition camera, front, rear, left and right surround view cameras, a lidar 24, a positioning module (such as Beidou, GPS, etc.) 25, an inertial navigation unit 26, a switch 27, and a blind spot radar.

[0168] In some embodiments of this disclosure, such as Figure 6 As shown, lidar 24 can be a multi-line lidar.

[0169] In some embodiments of this disclosure, such as Figure 6 As shown, the traffic light recognition camera may include a first traffic light recognition camera 221 and a second traffic light recognition camera 222.

[0170] In some embodiments of this disclosure, such as Figure 6 As shown, the surround view cameras can include a front camera 231, a rear camera 232, a right camera 233, and a left camera 234.

[0171] In some embodiments of this disclosure, such as Figure 6 As shown, the blind spot filling radar may include a front blind spot filling radar 281, a rear blind spot filling radar 282, a right blind spot filling radar 283, and a left blind spot filling radar 284.

[0172] In some embodiments of this disclosure, the core processing unit 21 may be implemented as an autonomous driving sensor and core processing unit suite.

[0173] In some embodiments of this disclosure, the core processing unit 21 can be implemented as in any of the embodiments described above (e.g., Figure 3 or Figure 4 The target detection device described in the embodiments, or including any of the embodiments described above in this disclosure (e.g., Figure 5 The electronic device described in the embodiment).

[0174] In some embodiments of this disclosure, the camera and the autonomous driving module can communicate with each other. In order to improve the transmission speed and reduce wiring, GMSL link communication can be used.

[0175] In some embodiments of this disclosure, such as Figure 6 and Figure 7As shown, chassis module 1 mainly includes chassis 10, battery 11, power management device 12, chassis controller 13, motor driver 14, drive motor 15, main battery 16, and wireless communication module 17. Battery 11 provides power to the entire autonomous vehicle system. Power management device 12 converts the battery output into different voltage levels usable by various functional modules and controls power-on and power-off. Chassis controller 13 receives motion commands from the autonomous driving module and controls the autonomous vehicle's steering, forward movement, reverse movement, braking, etc.

[0176] In some embodiments of this disclosure, such as Figure 6 As shown, the remote monitoring streaming module 4 consists of a front monitoring camera 41, a rear monitoring camera 42, a left monitoring camera 43, a right monitoring camera 44, and a streaming module 45. The streaming module 45 transmits the video data collected by the monitoring cameras to the backend server for viewing by backend operators. The wireless communication module 17 communicates with the backend server through the antenna 33, enabling backend operators to remotely control the unmanned vehicle.

[0177] In some embodiments of this disclosure, such as Figure 6 As shown, the cargo box module 3 includes a cargo box 31, a display and interaction module 32, and an antenna 33. The cargo box 31 serves as the cargo-carrying device for the unmanned vehicle. The display and interaction module 32 is installed on the cargo box 31, allowing interaction between the unmanned vehicle and the user. Users can perform operations such as picking up items, storing goods, and purchasing goods through the display and interaction module. The type of cargo box can be changed according to actual needs. For example, in a logistics scenario, the cargo box 31 can include multiple sub-boxes of different sizes, which can be used to load goods for delivery. In a retail scenario, the cargo box 31 can be set as a transparent box so that users can clearly see the products for sale.

[0178] In some embodiments of this disclosure, the cargo box 31 may be a delivery box.

[0179] The embodiments disclosed above can improve the prediction effect of target center depth in autonomous driving scenarios without significantly increasing model complexity, and obtain more accurate detection results.

[0180] The embodiments disclosed above introduce lidar signals as auxiliary inputs to the model, thereby compensating for the deficiencies of a single image signal and obtaining more accurate 3D target detection output.

[0181] The embodiments disclosed above propose a solution to the misalignment between lidar signals and image signals, thereby improving the robustness of the enhancement model and achieving efficient utilization of lidar signals.

[0182] In addition to the target object pose information, the embodiments of this disclosure output a dense depth prediction map of the scene, thereby improving the scene perception capability.

[0183] According to another aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement any of the embodiments described above (e.g., Figure 1 or Figure 2a The target detection method described in the example).

[0184] In some embodiments of this disclosure, the computer-readable storage medium may be a non-transitory computer-readable storage medium.

[0185] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0187] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0189] The signal acquisition module 500, prediction map determination module 600, depth correction module 700, detection result determination module 800, correction map determination unit 710, depth correction unit 720, target detection device and electronic equipment described above can be implemented as a general-purpose processor, programmable logic controller (PLC), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component or any suitable combination thereof for performing the functions described in this application.

[0190] The present disclosure has been described in detail so far. To avoid obscuring the concept of the present disclosure, some details known in the art have not been described. Based on the above description, those skilled in the art can fully understand how to implement the technical solutions disclosed herein.

[0191] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing the relevant hardware to implement them. The program can be stored in a non-transitory computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0192] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A target detection method, comprising: Acquire image signals and lidar signals; The target object is determined by the image signal, including a classification prediction map, a pose prediction map, and a depth information prediction map. The depth information prediction map includes a surface depth prediction map of the scene where the target object is located. The depth information prediction map of the target object is corrected based on the lidar signal; The detection output of the 3D target object is determined based on the target's classification prediction map, pose prediction map, and corrected depth information prediction map. The step of correcting the depth information prediction map of the target object based on the lidar signal includes: The predicted radar signal is determined based on the surface depth prediction map of the scene where the target object is located. Low-pass filtering is applied to the predicted radar signal and the lidar signal to obtain the smoothed predicted radar signal and the smoothed lidar signal. Differential low-pass filtering is performed on the predicted radar smoothed signal and the lidar smoothed signal to determine the depth correction map; The depth information prediction map of the target object is corrected based on the depth correction map.

2. The target detection method according to claim 1, wherein, The step of determining the predicted radar signal based on the surface depth prediction map of the scene where the target object is located includes: The predicted radar signal is determined based on the surface depth prediction map of the scene where the target object is located and the mask signal of the radar signal.

3. The target detection method according to claim 1, wherein, The step of performing differential low-pass filtering on the predicted radar smoothed signal and the lidar smoothed signal to determine the depth correction map includes: The difference between the predicted radar smoothed signal and the lidar smoothed signal is calculated to determine the difference map; The difference map is low-pass filtered to determine the depth correction map.

4. The target detection method according to any one of claims 1-3, wherein, The depth information prediction map also includes a center depth prediction map of the target object; The process of determining the classification prediction map, pose prediction map, and depth information prediction map of the target object based on the image signal includes: Based on the image signal, a classification prediction map of the target object, a pose prediction map of the target object, and a surface depth prediction map of the scene in which the target object is located are determined. Based on the target object's pose prediction map and the surface depth prediction map of the scene in which the target object is located, determine the target object's center depth prediction map.

5. The target detection method according to claim 4, wherein, The correction of the depth information prediction map of the target object based on the depth correction map includes: The center depth prediction map of the target object is corrected based on the depth correction map to obtain the corrected center depth prediction map.

6. The target detection method according to claim 5, wherein, The detection output results of determining the 3D target object based on the target's classification prediction map, pose prediction map, and corrected depth information prediction map include: The target interface is decoded from the target classification prediction map, pose prediction map and corrected target object center depth prediction map to obtain the target object pose information. The target object's pose information and the predicted surface depth map of the scene in which the target object is located are output as the detection results.

7. The target detection method according to any one of claims 1-3, wherein, The acquisition of lidar signals includes: The sparse depth map, which is obtained by projecting the raw LiDAR scanning data onto the image coordinate system based on the camera's extrinsic and intrinsic parameters, is used as the LiDAR signal.

8. A target detection device, comprising: The signal acquisition module is configured to acquire image signals and lidar signals; The prediction map determination module is configured to determine the classification prediction map, pose prediction map and depth information prediction map of the target object based on the image signal, wherein the depth information prediction map includes the surface depth prediction map of the scene where the target object is located. The depth correction module is configured to correct the depth information prediction map of the target object based on the lidar signal; The detection result determination module is configured to determine the detection output result of the three-dimensional target object based on the target's classification prediction map, pose prediction map, and corrected depth information prediction map. The depth correction module includes a correction map determination unit and a depth correction unit, wherein: The calibration map determination unit is configured to determine the predicted radar signal based on the surface depth prediction map of the scene where the target object is located; perform low-pass filtering on the predicted radar signal and the lidar signal to obtain the predicted radar smooth signal and the lidar smooth signal; and perform differential low-pass filtering on the predicted radar smooth signal and the lidar smooth signal to determine the depth calibration map. The depth correction unit is configured to correct the depth information prediction map of the target object based on the depth correction map.

9. The target detection device according to claim 8, wherein: The correction map determination unit is configured to determine the predicted radar signal based on the surface depth prediction map of the scene where the target object is located and the mask signal of the radar signal, when the predicted radar signal is determined based on the surface depth prediction map of the scene where the target object is located.

10. The target detection device according to claim 8, wherein: The calibration map determination unit is configured to, in the case of determining a depth calibration map by performing differential low-pass filtering on the predicted radar smoothed signal and the lidar smoothed signal, calculate the difference between the predicted radar smoothed signal and the lidar smoothed signal to determine a differential map; and perform low-pass filtering on the differential map to determine the depth calibration map.

11. The target detection device according to any one of claims 8-10, wherein, The depth information prediction map also includes a center depth prediction map of the target object; The prediction map determination module is configured to determine the classification prediction map of the target object, the pose prediction map of the target object, and the surface depth prediction map of the scene where the target object is located based on the image signal; and to determine the center depth prediction map of the target object based on the pose prediction map of the target object and the surface depth prediction map of the scene where the target object is located.

12. The target detection device according to claim 11, wherein: The depth correction unit is configured to correct the center depth prediction map of the target object based on the depth correction map to obtain the corrected center depth prediction map.

13. The target detection device according to claim 12, wherein: The detection result determination module is configured to decode the target interface from the target's classification prediction map, pose prediction map, and corrected target object center depth prediction map to obtain the target object pose information. The target object's pose information and the predicted surface depth map of the scene in which the target object is located are output as the detection results.

14. The target detection device according to any one of claims 8-10, wherein: The signal acquisition module is configured to project the raw LiDAR scanning data onto the image coordinate system based on camera extrinsic and intrinsic parameters to form a sparse depth map, which serves as the LiDAR signal.

15. An electronic device comprising: Memory, used to store instructions; A processor is configured to execute the instructions, causing the electronic device to perform operations that implement the target detection method as described in any one of claims 1-7.

16. An unmanned vehicle, comprising a target detection device as claimed in any one of claims 8-14, or comprising an electronic device as claimed in claim 15.

17. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the target detection method as described in any one of claims 1-7.

Images