Target detection method and device, electronic equipment and storage medium

Abstract

The application provides a target detection method and device, electronic equipment and a storage medium. The method comprises: acquiring a view image and point cloud data, the view image and the point cloud data being acquired for the same space; based on the point cloud data, performing depth prediction on the view image and / or performing visibility segmentation on a bird's-eye view under the space to obtain a spatial occlusion relationship; and based on the spatial occlusion relationship, performing feature aggregation on the view image and the point cloud data, and performing target detection based on the aggregation result. The method and device provided in the embodiments of the application perform depth prediction on the view image and / or perform visibility segmentation on the bird's-eye view under the space based on the point cloud data to obtain the spatial occlusion relationship, and use the spatial occlusion relationship to guide the feature aggregation of the view image and the point cloud data, thereby avoiding the problem of reverse projection in the aggregation process and avoiding the introduction of noise in the aggregation process. Thus, target detection is performed based on the aggregation result, and accurate and reliable target detection is achieved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a target detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, multi-view Figure 3 3D object detection is a crucial component of various applications, including autonomous driving. Besides lower deployment costs, pure vision-based perception systems offer significant advantages in detecting small targets and recognizing road features such as traffic signs and lane markings. Currently, 3D vision perception technology based on Bird's Eye View (BEV) provides a unified space for multiple perspectives, enabling target detection by learning spatial features from view images and point cloud data within the BEV.

[0003] However, this method inevitably causes the spatial features of the bird's-eye view to aggregate the spatial features of the back-projected 2D image, resulting in unsatisfactory target detection performance. Summary of the Invention

[0004] This invention provides a target detection method, apparatus, electronic device, and storage medium to address the shortcomings of unsatisfactory target detection results in the prior art.

[0005] This invention provides a target detection method, comprising:

[0006] Acquire view images and point cloud data, wherein the view images and point cloud data are collected for the same space;

[0007] Based on the point cloud data, depth prediction is performed on the view image and / or visibility segmentation is performed on the bird's-eye view in the space to obtain spatial occlusion relationships;

[0008] Based on the spatial occlusion relationship, feature aggregation is performed on the view image and the point cloud data, and target detection is performed based on the aggregation result.

[0009] According to a target detection method provided by the present invention, the step of performing depth prediction on the view image and / or visibility segmentation on the bird's-eye view in the space based on the point cloud data to obtain spatial occlusion relationships includes:

[0010] Based on the depth information of the point cloud data, depth prediction is performed on each pixel in the view image to obtain the predicted pixel depth of each pixel in the view image;

[0011] Based on the point cloud data, the bird's-eye view is segmented by visibility to obtain the voxel visibility results of each voxel in the bird's-eye view;

[0012] The spatial occlusion relationship is determined based on the predicted pixel depth and / or the voxel visibility results.

[0013] According to a target detection method provided by the present invention, the step of performing depth estimation on each pixel in the view image based on the depth information of the point cloud data includes:

[0014] Based on the positional correspondence between the point cloud data and the view image, and the depth information of the point cloud data, the point cloud supervision depth of each pixel in the view image is determined.

[0015] Based on the point cloud supervised depth and the image features of the view image, depth prediction is performed on each pixel in the view image.

[0016] According to a target detection method provided by the present invention, the step of performing visibility segmentation on the bird's-eye view based on the point cloud data to obtain the voxel visibility results of each voxel in the bird's-eye view includes:

[0017] The point cloud data is converted to the space where the bird's-eye view is located to obtain projected point cloud data;

[0018] The system detects whether each voxel in the bird's-eye view has a corresponding radar point in the projected point cloud data, and determines the voxel visibility result of each voxel based on the detection result.

[0019] According to a target detection method provided by the present invention, the step of detecting whether each voxel in the bird's-eye view has a corresponding radar point in the projected point cloud data, and determining the voxel visibility result of each voxel based on the detection result, includes:

[0020] Based on the historical spatial features of the bird's-eye view at a historical moment, and the coordinate transformation information from the historical moment to the current moment, the spatial features of the bird's-eye view at the current moment are enhanced.

[0021] Based on the enhanced spatial features, it is detected whether each voxel in the bird's-eye view has a corresponding radar point in the projected point cloud data, and the voxel visibility result of each voxel is determined based on the detection result.

[0022] According to a target detection method provided by the present invention, the method involves performing depth prediction on the view image and / or visibility segmentation on the bird's-eye view in the space based on the point cloud data to obtain spatial occlusion relationships; performing feature aggregation on the view image and the point cloud data based on the spatial occlusion relationships; and performing target detection based on the aggregation results, including:

[0023] Based on the target detection model, the point cloud data is used to perform depth prediction on the view image and / or visibility segmentation on the bird's-eye view in the space, spatial occlusion relationship is obtained and applied, feature aggregation is performed on the view image and the point cloud data, and the aggregation result is used for target detection.

[0024] The target detection model is trained based on sample point cloud data, sample view images, target detection labels, depth labels, and visibility labels.

[0025] According to the object detection method provided by the present invention, the training steps of the object detection model include:

[0026] Determine an initial model, which includes at least two candidate detection heads;

[0027] Based on the initial model, at least two target detection results corresponding to the sample point cloud data and the sample view image, a depth prediction result of the sample view image, and a visibility prediction result of the sample bird's-eye view are determined; the at least two target detection results are output by the at least two candidate detection heads respectively;

[0028] Based on the depth prediction results and the depth labels, the depth prediction loss is determined;

[0029] Based on the visibility prediction results and the visibility labels, the visibility segmentation loss is determined;

[0030] Based on the target detection label and the at least two target detection results, the detection loss is determined;

[0031] Based on the depth prediction loss, the visibility prediction loss, and the detection loss, the initial model is iterated to obtain the target detection model.

[0032] The present invention also provides a target detection device, comprising:

[0033] The acquisition unit acquires view images and point cloud data, wherein the view images and point cloud data are acquired in the same space;

[0034] Determine spatial occlusion units, and based on the point cloud data, perform depth prediction on the view image and / or perform visibility segmentation on the bird's-eye view under the space to obtain spatial occlusion relationships;

[0035] The detection unit performs feature aggregation on the view image and the point cloud data based on the spatial occlusion relationship, and performs target detection based on the aggregation result.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the target detection methods described above.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target detection method as described above.

[0038] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the target detection method as described above.

[0039] The target detection method, apparatus, electronic device, and storage medium provided by this invention use point cloud data to perform depth prediction on view images and / or visibility segmentation on bird's-eye views in space to obtain spatial occlusion relationships. These spatial occlusion relationships are then used to guide feature aggregation of view images and point cloud data, thereby avoiding the problem of back projection during aggregation and preventing the introduction of noise during the aggregation process. Based on the aggregation results, target detection is performed, achieving accurate and reliable target detection. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is one of the flowcharts of the target detection method provided by the present invention;

[0042] Figure 2 This is a schematic diagram of the occlusion of various targets in a bird's-eye view provided by the present invention;

[0043] Figure 3 This is a flowchart illustrating the process of determining spatial occlusion relationships provided by the present invention;

[0044] Figure 4 This is the second flowchart of the target detection method provided by the present invention;

[0045] Figure 5 This is a schematic diagram of the target detection device provided by the present invention;

[0046] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0048] Currently, spatial features in a view image are searched by predefining a set of 3D reference points in the bird's-eye view. However, this method inevitably leads to the aggregation of spatial features from the bird's-eye view with those from the back-projected 2D image. In other words, the occlusion relationships of targets in space are unclear during view transformation. This unclear occlusion leads to two problems: first, targets occluded by foreground obstacles incorrectly aggregate features from the foreground image; second, voxels in the bird's-eye view are projected onto the same point on the image plane along the straight line between the optical center and the target. Therefore, the unclear occlusion relationships introduce non-negligible noise into target detection, resulting in unsatisfactory target detection performance.

[0049] To address the aforementioned problems, this invention proposes a target detection method that clearly defines occlusion relationships, thereby ensuring the accuracy of target detection. Figure 1 This is one of the flowcharts of the target detection method provided by the present invention, such as... Figure 1 As shown, the method includes:

[0050] Step 110: Acquire view image and point cloud data, wherein the view image and point cloud data are acquired for the same space;

[0051] Specifically, the view image is a 2D image captured by a camera, while the point cloud data is a 3D point cloud collected by radar. For example, during target detection in autonomous driving, the view image can be obtained by acquiring images of the space surrounding the vehicle from onboard cameras. This view image can be a single image or multiple images captured by onboard cameras. Additionally, point cloud data of the space surrounding the vehicle can be obtained using a LiDAR sensor installed on the vehicle. It should be noted that the view image acquired by the onboard camera and the point cloud data acquired by the LiDAR sensor are data collected from the same space, containing spatial features of different dimensions of that space. Here, the view image can reflect the spatial features of various targets in the space in 2D, such as the distance between pedestrians and vehicles. As an image, the view image can clearly reflect the semantic information of the targets contained in the space, such as the traffic status indicated by traffic lights. It is understandable that the accuracy and comprehensiveness of obtaining spatial information about targets in space from view images are limited. Therefore, point cloud data acquired by LiDAR sensors can richly, accurately, and densely reflect the spatial information of targets in space in the 3D dimension. Thus, view images and point cloud data corresponding to the same space can be registered and fused with spatial features acquired from different dimensions to achieve target detection within the same space.

[0052] Step 120: Based on the point cloud data, perform depth prediction on the view image and / or perform visibility segmentation on the bird's-eye view in the space to obtain the spatial occlusion relationship;

[0053] Specifically, the accurate and dense depth information contained in point cloud data can be used to predict the depth of a view image. By obtaining the predicted depth information of each pixel in the view image, the spatial occlusion relationships of various objects in the 2D dimension can be predicted. For example, at the same pixel in the view image, the larger the depth value of the object, the farther the object is from the camera, and the greater the probability that the object is occluded at the bottom layer of the image; the smaller the depth value, the closer the object is to the camera, and the less likely the object is to be occluded at the bottom layer of the image. It is understandable that extracting the depth information from the point cloud data into the view image, obtaining the depth information of each pixel in the view image, and thus determining the spatial occlusion relationships between various objects in the view image, makes the aggregation of spatial features in the view image and the spatial features of the corresponding bird's-eye view from the point cloud data more accurate.

[0054] It should be noted that depth estimation based solely on view images is easily affected by camera calibration parameters, resulting in poor generalization ability. Therefore, point cloud data can also be used as visibility segmentation labels to perform visibility segmentation on the bird's-eye view in space. This involves segmenting the bird's-eye view into voxels and determining the visibility of each voxel. Here, visibility refers to whether each voxel in the bird's-eye view is occupied, reflecting whether the target corresponding to that voxel is occluded. If the voxel is unoccupied in the bird's-eye view, its visibility is considered visible, and the probability that the target corresponding to that voxel is occluded in the lower layers of the image is lower. Conversely, if the voxel is occupied in the bird's-eye view, its visibility is considered invisible, and the probability that the target corresponding to that voxel is occluded in the lower layers of the image is higher.

[0055] Understandably, spatial occlusion relationships of targets in space can be obtained by performing depth prediction on only the view image using point cloud data, or by performing visibility segmentation on only the bird's-eye view using point cloud data. Furthermore, combining depth prediction on 2D images with visibility segmentation on 3D bird's-eye view space can yield more accurate and stable results.

[0056] The spatial occlusion relationship here can be reflected in the mutual occlusion relationships between targets in space in 2D and / or 3D dimensions. Therefore, based on this spatial occlusion relationship, the image features of the view image are aggregated with the spatial features of the corresponding bird's-eye view in the point cloud data. This avoids aggregating the image features of view images with occlusion relationships onto the spatial features of the same voxel in the bird's-eye view, thereby improving the accuracy of target detection based on the aggregation result. For example, Figure 2 This is a schematic diagram of the occlusion of various targets in a bird's-eye view provided by the present invention, such as... Figure 2 As shown, 210 represents a voxel along the ray direction; 220 represents a foreground obstacle; and 230 represents an occluded detection target. Specifically, after obtaining the spatial occlusion relationship, when aggregating the view image features with the spatial features of the bird's-eye view corresponding to the point cloud data, it is possible to avoid aggregating the features of the view image of the foreground obstacle 220 onto the occluded detection target 230. Simultaneously, it also prevents voxels 210 along the line between the optical center and the target from being projected onto the same point in the image plane in the bird's-eye view.

[0057] Step 130: Based on the spatial occlusion relationship, perform feature aggregation on the view image and the point cloud data, and perform target detection based on the aggregation result.

[0058] Specifically, after obtaining the spatial occlusion relationships, the spatial occlusion relationships of each target in the view image and each voxel in the bird's-eye view can be used to aggregate the image features of the view image and the spatial features of the bird's-eye view in the space corresponding to the point cloud data. This aggregated result improves the accuracy of target detection based on the aggregated result. Here, the aggregated result refers to the aggregated image features of the view image and the spatial features of the bird's-eye view that reflect the occlusion relationships of each target in space.

[0059] The method provided in this invention, based on point cloud data, performs depth prediction on view images and / or visibility segmentation on bird's-eye views in space to obtain spatial occlusion relationships. These spatial occlusion relationships are then used to guide feature aggregation of view images and point cloud data, thereby avoiding the problem of back projection during aggregation and preventing the introduction of noise during aggregation. This achieves high-quality 3D feature reconstruction, and target detection is performed based on the aggregation results, thus achieving accurate and reliable target detection.

[0060] Based on any of the above embodiments Figure 3 This is a schematic diagram of the process for determining spatial occlusion relationships provided by the present invention, as shown below. Figure 3 As shown, step 120 includes:

[0061] Step 310: Based on the depth information of the point cloud data, perform depth estimation on each pixel in the view image to obtain the estimated pixel depth of each pixel in the view image;

[0062] Here, the depth information of point cloud data reflects the distance between various targets in space and the vehicle in three-dimensional space. Specifically, the depth information of point cloud data is denser, more accurate, and more concentrated than the depth information in the view image. The depth information of the point cloud data at each point in space can be used to predict the depth of each corresponding pixel in the view image, thus obtaining the predicted pixel depth of each pixel in the view image. For example, the depth information of the point cloud data corresponding to a pixel in the view image can be directly used as the predicted pixel depth. This predicted pixel depth includes a depth value that incorporates depth information from three-dimensional space, as well as the predicted probability of each depth value.

[0063] Understandably, compared to obtaining the expected pixel depth of each pixel in the view image solely from the depth information of the view image, extracting the depth information of three-dimensional space into the view image makes the depth prediction information in the obtained view image more accurate and avoids the influence of the camera on the depth prediction information of the view image.

[0064] Step 320: Based on the point cloud data, perform visibility segmentation on the bird's-eye view to obtain the voxel visibility results of each voxel in the bird's-eye view;

[0065] Specifically, the LiDAR points contained in the point cloud data can be used as supervisory labels for visibility segmentation of voxels in a bird's-eye view, thus obtaining the voxel visibility results in the bird's-eye view. For example, if a voxel in the bird's-eye view contains a LiDAR point, its book visibility result is visible, and the voxel is not occupied; if the voxel in the bird's-eye view does not contain a LiDAR point, its book visibility result is invisible, and the voxel is occupied. It can be understood that by judging the visibility of each voxel in the bird's-eye view based on point cloud data, the image features of the view image can be appropriately assigned to their actual occupied voxels in the bird's-eye view.

[0066] It should be noted that the embodiments of the present invention do not limit the order in which steps 310 and 320 are executed. Steps 310 and 320 can be executed one at a time, sequentially, or simultaneously.

[0067] Step 330: Determine the spatial occlusion relationship based on the predicted pixel depth and / or the voxel visibility results.

[0068] Specifically, the spatial occlusion relationship at a given location can be determined by combining the estimated pixel depth of that pixel in the view image with the voxel visibility result of the voxel corresponding to that pixel. It is understandable that determining the spatial occlusion relationship based on the estimated pixel depth and voxel visibility result makes the determined spatial occlusion relationship more accurate and stable. It should be noted that the spatial occlusion relationship can also be determined using only the estimated pixel depth or voxel visibility result.

[0069] The method provided in this invention, based on point cloud data, obtains the expected pixel depth of the view image and / or the visibility results of each voxel in the bird's-eye view. By utilizing the distance information provided by the depth information of the point cloud data, the spatial occlusion relationship of each target in space is determined from the 2D and / or 3D dimensions, thereby improving the accuracy of the spatial occlusion relationship.

[0070] Based on any of the above embodiments, step 310, which involves estimating the depth of each pixel in the view image based on the depth information of the point cloud data, includes:

[0071] Based on the positional correspondence between the point cloud data and the view image, and the depth information of the point cloud data, the point cloud supervision depth of each pixel in the view image is determined.

[0072] Based on the point cloud supervised depth and the image features of the view image, the expected pixel depth of each pixel in the view image is determined.

[0073] Here, the positional correspondence between point cloud data and the view image reflects the correspondence between points in the point cloud data and points in the view image. This correspondence can be determined using the 3D coordinates of the point cloud data, the rotation and translation matrices from the point cloud coordinate system to the camera coordinate system, and the camera's intrinsic matrix. After obtaining this correspondence, the depth information of the point cloud data can be used as the point cloud supervised depth for the corresponding pixels in the view image. This supervised depth can then be used to supervise the image depth information of the view image, enabling depth prediction for each pixel in the view image.

[0074] Furthermore, the image features of the view image here can characterize the spatial and semantic information contained in the view image itself. It should be noted that after obtaining the point cloud supervised depth, the point cloud supervised depth can be extracted into the image features of the view image to achieve depth prediction for each pixel in the view image. Specifically, the depth perception range of the view image can be divided into multiple sub-intervals. For example, the depth perception range of [0, 64] meters can be divided into N sub-intervals. Through the point cloud supervised depth and the image features of the view image, depth prediction is performed on each pixel in the view image, making the depth information of the view image more accurate and obtaining image features of the view image containing accurate image depth information.

[0075] Specifically, firstly, the projection coordinates of the point cloud data in the camera coordinate system can be determined using the 3D coordinates of the point cloud data and the rotation and translation matrices from the point cloud data coordinate system to the camera coordinate system. For example, this can be calculated using the following formula:

[0076]

[0077] In the formula, Represents the projection coordinates (x, y) of the point cloud data in the i-th camera. c ,y c ) and point cloud supervision depth d c ;R i R represents the rotation matrix from the point cloud data coordinate system to the camera coordinate system. i ∈R 3×3 P represents point cloud data; t i t represents the translation matrix from the point cloud data coordinate system to the camera coordinate system. i ∈R 3×3 .

[0078] Next, the coordinates of the pixels corresponding to the point cloud data in the pixel coordinate system of the view image can be obtained through the camera's intrinsic matrix. For example, it can be calculated using the following formula:

[0079] [u,v,1]=K i ·[x c / d c y c / d c 1] T

[0080] In the formula, (u,v) represents the coordinates of the pixel corresponding to the point cloud data in the pixel coordinate system of the view image; K i Let K represent the intrinsic matrix of the i-th camera. i ∈R 3×3 ;

[0081] Furthermore, after obtaining the positional correspondence between pixels in the point cloud data within the view image, and the point cloud supervised depth of each pixel, the view image can be upsampled and image feature encoded to match the view image size. Then, based on the image features and the point cloud supervised depth, depth prediction is performed on each pixel in the view image, for example, using the following formula:

[0082] depth_estimation(upsample(F i ))

[0083] In the formula, depth_estimation represents depth estimation; upsample represents upsampling; F i Represents the image features of the view image.

[0084] Understandably, considering the significant uncertainty in depth, after obtaining the point cloud supervised depth of each pixel in the view image, the predicted probability corresponding to the point cloud supervised depth can be calculated using the Gaussian distribution of the point cloud supervised depth and the cross-entropy loss function. That is, the predicted pixel depth can be obtained. The Gaussian distribution can be calculated using the following formula:

[0085] X~N(d c ,σ 2 )

[0086]

[0087] In the formula, X represents a random variable; N represents the sample size; σ represents the standard deviation of the Gaussian distribution; d∈d1,d2,d3…,d N , represents the center depth of each sub-interval of the depth perception range of the view image; e represents the natural constant.

[0088] The method provided in this invention extracts the depth information of point cloud data into the image features of a view image based on the depth information of point cloud data, and performs depth prediction. This enables the determination of spatial occlusion relationships between targets in 2D space, improves the accuracy and stability of the predicted pixel depth of the view image, and thus improves the accuracy and stability of spatial occlusion relationships.

[0089] Based on any of the above embodiments, step 320 includes:

[0090] The point cloud data is converted to the space where the bird's-eye view is located to obtain projected point cloud data;

[0091] The system detects whether each voxel in the bird's-eye view has a corresponding radar point in the projected point cloud data, and determines the voxel visibility result of each voxel based on the detection result.

[0092] Specifically, the point cloud data can be transformed into the space of the bird's-eye view using rotation and translation matrices from the 3D coordinate system of the point cloud data to the coordinate system of the bird's-eye view, thus obtaining projected point cloud data. This projected point cloud data is the data corresponding to the 3D point cloud data in the space of the bird's-eye view, and can be calculated using the following formula:

[0093]

[0094] In the formula, P e (x e ,y e ,z e () represents projected point cloud data; The rotation matrix representing the point cloud data from its 3D coordinate system to the coordinate system in the space of the bird's-eye view. The translation matrix representing the shift from the 3D coordinate system of the point cloud data to the coordinate system of the bird's-eye view. P represents point cloud data.

[0095] Furthermore, the visibility of each voxel in the bird's-eye view can be determined by detecting whether a corresponding radar point exists in the projected point cloud data. If a radar point exists, the detection result is "visible"; if no radar point exists, the detection result is "invisible". After obtaining the detection results, the voxel visibility result can be obtained by calculating binary cross-entropy.

[0096] The method provided in this embodiment of the invention, based on point cloud data, performs voxel visibility segmentation on each voxel in a bird's-eye view, determines the voxel visibility result of each voxel, realizes the determination of the spatial occlusion relationship of each target in 3D space, and improves the accuracy of the spatial occlusion relationship.

[0097] Based on any of the above embodiments, the step of detecting whether each voxel in the bird's-eye view has a corresponding radar point in the projected point cloud data, and determining the voxel visibility result of each voxel based on the detection result, includes:

[0098] Based on the historical spatial features of the bird's-eye view at a historical moment, and the coordinate transformation information from the historical moment to the current moment, the spatial features of the bird's-eye view at the current moment are enhanced.

[0099] Based on the enhanced spatial features, it is detected whether each voxel in the bird's-eye view has a corresponding radar point in the projected point cloud data, and the voxel visibility result of each voxel is determined based on the detection result.

[0100] Here, historical spatial features can reflect the spatial information of the bird's-eye view at historical moments, and can be used to compensate for the spatial information of the bird's-eye view at the current moment, so as to obtain a more comprehensive spatial information of the bird's-eye view at the current moment.

[0101] Specifically, firstly, historical spatial features can be obtained by extracting spatial features from the bird's-eye view at a historical time. Then, by using coordinate transformation information from the historical time to the current time, the bird's-eye view at the current time can be aligned with the historical time's bird's-eye view. Furthermore, the historical spatial features can be used to automatically compensate for the spatial features of the current time, thereby enhancing the spatial features of the bird's-eye view at the current time. This can be calculated using the following formula:

[0102]

[0103]

[0104] In the formula, The spatial features representing the alignment of historical spatial features at a historical moment with the spatial features at the current moment; Q represents the coordinate transformation matrix from a historical moment to the current moment; T-1 Historical spatial characteristics of historical moment T-1; Indicates the enhanced spatial features; Q p p represents the spatial characteristics of point cloud data; Q represents point cloud data. T This indicates the spatial characteristics of the bird's-eye view at the current moment.

[0105] It is understandable that by making full use of the surround view and time cues to capture time information, the spatial features of the bird's-eye view at the current moment can be enhanced, making the spatial features based on the enhanced features more comprehensive and accurate. In turn, the voxel visibility results of visibility detection based on the enhanced spatial features are more accurate and comprehensive.

[0106] Furthermore, after obtaining the spatial features enhanced by the feature enhancement, a more comprehensive bird's-eye view with more three-dimensional information can be obtained. Then, by checking whether each voxel in the more comprehensive bird's-eye view with more three-dimensional information has a corresponding radar point in the projected point cloud data, and determining the voxel visibility result of each voxel based on the detection result, a more comprehensive and accurate voxel visibility result is obtained.

[0107] Based on any of the above embodiments, steps 120 and 130 include:

[0108] Based on the target detection model, the point cloud data is used to perform depth prediction on the view image and / or visibility segmentation on the bird's-eye view in the space, spatial occlusion relationship is obtained and applied, feature aggregation is performed on the view image and the point cloud data, and the aggregation result is used for target detection.

[0109] The target detection model is trained based on sample point cloud data, sample view images, target detection labels, depth labels, and visibility labels.

[0110] Specifically, in the practical application stage, point cloud data and view images collected in the same space can be input into the target detection model. The target detection model uses the point cloud data to perform depth prediction on the view image and / or visibility segmentation on the bird's-eye view in the space, obtains and applies spatial occlusion relationships, performs feature aggregation on the view image and point cloud data, and applies the aggregation results to perform target detection.

[0111] Prior to this, an object detection model can be obtained through model training. Specifically, sample point cloud data can be used to predict depth in sample view images and / or perform visibility segmentation on the bird's-eye view in space. Spatial occlusion relationships can be obtained and applied. Feature aggregation can be performed on the image features of the sample view images and the spatial features of the corresponding bird's-eye view in the sample point cloud data, and the aggregation results can be used for object detection to obtain the trained detection results. The object detection model can be iteratively trained with the aim of reducing the loss between depth labels and predicted pixel depth, and / or the loss between visibility labels and visibility results, as well as the loss between trained detection results and object detection labels, to obtain the final object detection model that can be used to obtain accurate object detection results.

[0112] Furthermore, feature aggregation of view images and point cloud data within the spatial context of the bird's-eye view can be achieved by using predicted pixel depth and voxel visibility results as adaptive learnable weights to aggregate image features from the view images within the spatial features of the bird's-eye view. For example, this can be calculated using the following formula:

[0113]

[0114] In the formula, v hitThe viewpoint of the camera is represented by ; i represents the camera at the i-th viewpoint; Indicates voxel visibility results; Indicates the expected pixel depth; DeformAttn represents deformable attention; Q p The bird's-eye view represents the spatial characteristics of the bird's-eye view; ρ(p,i,j) represents the positional correspondence between the bird's-eye view and the view image; F i Represents the image features of the view image.

[0115] It should be noted that ρ(p,i,j) here, that is, the positional correspondence between the bird's-eye view and the view image, can be obtained through the learnable grid-type parameters Q∈R corresponding to the bird's-eye view. H×W×C For the query point p = (x, y), first calculate its coordinates (x, y) in the bird's-eye view. w ,y w The result can be calculated using the following formula:

[0116]

[0117]

[0118] In the formula, W represents the width of the bird's-eye view; H represents the height of the corresponding bird's-eye view.

[0119] Next, define a set of pillar-shaped anchor heights. Then the coordinates of the query point P in real space can be obtained as follows: It is understandable that this can be achieved through the camera's intrinsic matrix K. i ∈R 3×3 External matrix T i ∈R 3×4 It can calculate the pixel coordinates (x, y) of the query point p projected from the real space onto the view image. ij ,y ij The value of z can be calculated using the following formula: ij ·[x ij y ij 1] T =K i ·T i ·[x w y w z j 1] T Furthermore, the positional projection relationship between the bird's-eye view and the view image corresponding to the point cloud data can be denoted as ρ(p,i,j)=(x ij ,y ij ).

[0120] It should also be noted that, in order to more accurately aggregate the view images into the spatial features of the bird's-eye view and avoid the influence of factors such as inaccurate calibration and errors caused by vehicle vibration on the internal and external matrices of the camera, a set of offsets Δp for the mapped pixel coordinates ρ(p,i,j) can be predicted by the decoder in DETR (End-to-End Object Detection with Transformers). mnl Then DeformAttn represents deformable attention, which can be expressed as: In the formula, A mnl This indicates the attention weight for prediction. Indicates A mnl Standardize; w′ m and W m The learnable weights are represented by m and n, respectively, which represent the number of attention heads and sampling offsets. l This represents the image features of the view image. Specifically, the attention weight A... mnl Normalization can usually be achieved using the softmax function. By constraining the sum of attention weights of features in the softmax activation layer to 1, the weights for image feature aggregation can be effectively controlled. For example, A can be calculated using the following formula: mnl =softmax(Z) n In the formula, Z n This represents the attention weights output by the attention weight prediction layer. However, due to the nature of ray tracing, to avoid projecting spatial features from other locations along the ray onto the same point in the view image, this embodiment of the invention employs adaptively learned feature aggregation weights, such as A. mnl = sigmoid(Z n ).

[0121] Based on any of the above embodiments, the training steps of the target detection model include:

[0122] Determine an initial model, which includes at least two candidate detection heads;

[0123] Based on the initial model, at least two target detection results corresponding to the sample point cloud data and the sample view image, a depth prediction result of the sample view image, and a visibility prediction result of the sample bird's-eye view are determined; the at least two target detection results are output by the at least two candidate detection heads respectively;

[0124] Based on the depth prediction results and the depth labels, the depth prediction loss is determined;

[0125] Based on the visibility prediction results and the visibility labels, the visibility segmentation loss is determined;

[0126] Based on the target detection label and the at least two target detection results, the detection loss is determined;

[0127] Based on the depth prediction loss, the visibility prediction loss, and the detection loss, the initial model is iterated to obtain the target detection model.

[0128] Here, by selecting two candidate detection heads with different learning strategies, such as the DETR detection head and the center point detection head, the accuracy of object detection based on aggregated features can be further improved. It should be noted that using detection heads with different learning strategies can alleviate the overfitting problem in aggregated feature learning under a single task, which is beneficial for further improving the accuracy of object detection. Furthermore, the detection loss here can be constructed using the DETR loss function and the center point loss function corresponding to the two candidate detection heads, respectively. Therefore, the total loss of the initial model during training can be obtained by combining the depth prediction loss, visibility segmentation loss, and detection loss. For example, the total loss of the initial model can be calculated using the following formula:

[0129] l final =λ1l detr +λ2l centerpoint +λ3l depth +λ4l visibility

[0130] In the formula, l final l represents the total loss of the initial model. detr This indicates the DETR detection loss of the detection head; centerpoint Indicates the detection loss at the center point of the detection head; l depth Indicates the depth prediction loss; l visibility λ represents the visibility segmentation loss; λ1, λ2, λ3, and λ4 represent the loss weights corresponding to each loss.

[0131] Based on any of the above embodiments Figure 4 This is the second flowchart of the target detection method provided by the present invention, as shown below. Figure 4 As shown, the method includes:

[0132] First, acquire view images and point cloud data collected from the same space;

[0133] Next, the view image is processed through a backbone network to extract its features. This backbone network can be constructed using a multi-layer feature pyramid network (FPN), performing feature extraction and information fusion at different levels of feature maps. Upsampling is then performed to match the view image size. The image features and point cloud data are then input into the DCA (Deep Attention Convolutional Network) module. The DCA (Visual Attention Convolutional Network) module applies the point cloud data to predict the depth of the view image, obtaining the predicted pixel depth for each pixel in the view image.

[0134] Additionally, point cloud data is input into the VCA module. The VCA module applies the point cloud data to perform visibility segmentation on the spatial bird's-eye view, obtaining the voxel visibility results for each voxel. Visibility segmentation of the spatial bird's-eye view can be achieved by aligning the historical spatial features of the bird's-eye view from a historical time point with the coordinate transformation information from the historical time point to the current time point using a bird's-eye view alignment model. Based on the aligned spatial features, the spatial features of the bird's-eye view at the current time point are enhanced using a temporal self-attention mechanism to obtain the enhanced spatial features of the bird's-eye view. The enhanced spatial features and point cloud data can then be input into the VCA module. Based on the enhanced spatial features, the VCA module detects whether each voxel in the bird's-eye view contains a corresponding radar point in the projected point cloud data, and determines the voxel visibility result for each voxel based on the detection results.

[0135] Furthermore, based on the predicted pixel depth output by the DCA module and the voxel visibility results output by the VCA module, the occlusion-aware cross-referencing module guides the image features of the view image to perform feature aggregation between the image features and spatial features in the bird's-eye view, obtaining the aggregation result. This occlusion-aware cross-referencing module is composed of six bird's-eye view spatiotemporal transformation blocks, which can effectively query the visual information of the view image.

[0136] Finally, based on the aggregation results, target detection is performed using the DETR detection head and center point detection head in the multi-head learning framework of the target detection model to obtain target detection results corresponding to the view image and point cloud data.

[0137] Based on any of the above embodiments Figure 5 This is a schematic diagram of the target detection device provided by the present invention, as shown below. Figure 5 As shown, it includes:

[0138] The acquisition unit 510 acquires a view image and point cloud data, wherein the view image and point cloud data are acquired for the same space;

[0139] The spatial occlusion unit 520 determines the spatial occlusion relationship by performing depth prediction on the view image and / or visibility segmentation on the bird's-eye view under the space based on the point cloud data.

[0140] The detection unit 530 performs feature aggregation on the view image and the point cloud data based on the spatial occlusion relationship, and performs target detection based on the aggregation result.

[0141] The apparatus provided in this invention performs depth prediction on a view image and / or visibility segmentation on a bird's-eye view in space based on point cloud data to obtain spatial occlusion relationships. It then applies these spatial occlusion relationships to guide feature aggregation of the view image and point cloud data, thereby avoiding the problem of back projection during aggregation and preventing the introduction of noise during aggregation. Based on the aggregation results, it performs target detection, achieving accurate and reliable target detection.

[0142] Based on any of the above embodiments, the spatial occlusion unit is specifically used for:

[0143] Based on the depth information of the point cloud data, depth prediction is performed on each pixel in the view image to obtain the predicted pixel depth of each pixel in the view image;

[0144] Based on the point cloud data, the bird's-eye view is segmented by visibility to obtain the voxel visibility results of each voxel in the bird's-eye view;

[0145] The spatial occlusion relationship is determined based on the predicted pixel depth and / or the voxel visibility results.

[0146] Based on any of the above embodiments, the spatial occlusion determination unit further includes a depth prediction unit, which is specifically used for:

[0147] Based on the positional correspondence between the point cloud data and the view image, and the depth information of the point cloud data, the point cloud supervision depth of each pixel in the view image is determined.

[0148] Based on the point cloud supervised depth and the image features of the view image, depth prediction is performed on each pixel in the view image.

[0149] Based on any of the above embodiments, the spatial occlusion unit further includes a visibility segmentation unit, which is specifically used for:

[0150] The point cloud data is converted to the space where the bird's-eye view is located to obtain projected point cloud data;

[0151] The system detects whether each voxel in the bird's-eye view has a corresponding radar point in the projected point cloud data, and determines the voxel visibility result of each voxel based on the detection result.

[0152] Based on any of the above embodiments, the visibility segmentation unit is further used for:

[0153] Based on the historical spatial features of the bird's-eye view at a historical moment, and the coordinate transformation information from the historical moment to the current moment, the spatial features of the bird's-eye view at the current moment are enhanced.

[0154] Based on the enhanced spatial features, it is detected whether each voxel in the bird's-eye view has a corresponding radar point in the projected point cloud data, and the voxel visibility result of each voxel is determined based on the detection result.

[0155] Based on any of the above embodiments, the detection unit is specifically used for:

[0156] Based on the target detection model, the point cloud data is used to perform depth prediction on the view image and / or visibility segmentation on the bird's-eye view in the space, spatial occlusion relationship is obtained and applied, feature aggregation is performed on the view image and the point cloud data, and the aggregation result is used for target detection.

[0157] The target detection model is trained based on sample point cloud data, sample view images, target detection labels, depth labels, and visibility labels.

[0158] Based on any of the above embodiments, the detection unit further includes a training unit, which is specifically used for:

[0159] Determine an initial model, which includes at least two candidate detection heads;

[0160] Based on the initial model, at least two target detection results corresponding to the sample point cloud data and the sample view image, a depth prediction result of the sample view image, and a visibility prediction result of the sample bird's-eye view are determined; the at least two target detection results are output by the at least two candidate detection heads respectively;

[0161] Based on the depth prediction results and the depth labels, the depth prediction loss is determined;

[0162] Based on the visibility prediction results and the visibility labels, the visibility segmentation loss is determined;

[0163] Based on the target detection label and the at least two target detection results, the detection loss is determined;

[0164] Based on the depth prediction loss, the visibility prediction loss, and the detection loss, the initial model is iterated to obtain the target detection model.

[0165] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a target detection method, which includes: acquiring a view image and point cloud data, wherein the view image and point cloud data are acquired for the same space; based on the point cloud data, performing depth prediction on the view image and / or performing visibility segmentation on the bird's-eye view in the space to obtain spatial occlusion relationships; based on the spatial occlusion relationships, performing feature aggregation on the view image and the point cloud data, and performing target detection based on the aggregation results.

[0166] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the target detection method provided by the above methods. The method includes: acquiring a view image and point cloud data, wherein the view image and point cloud data are acquired for the same space; performing depth prediction on the view image and / or performing visibility segmentation on the bird's-eye view in the space based on the point cloud data to obtain spatial occlusion relationships; performing feature aggregation on the view image and the point cloud data based on the spatial occlusion relationships, and performing target detection based on the aggregation results.

[0168] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the target detection methods provided by the above methods. The method includes: acquiring a view image and point cloud data, wherein the view image and point cloud data are acquired for the same space; performing depth prediction on the view image and / or performing visibility segmentation on a bird's-eye view in the space based on the point cloud data to obtain spatial occlusion relationships; performing feature aggregation on the view image and the point cloud data based on the spatial occlusion relationships, and performing target detection based on the aggregation results.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target detection method, characterized in that, include: Acquire view images and point cloud data, wherein the view images and point cloud data are collected for the same space; Based on the point cloud data, depth prediction is performed on the view image and visibility segmentation is performed on the bird's-eye view in the space to obtain the spatial occlusion relationship; Based on the spatial occlusion relationship, feature aggregation is performed on the view image and the point cloud data, and target detection is performed based on the aggregation result; The feature aggregation based on the spatial occlusion relationship of the view image and the point cloud data includes: Using the predicted pixel depth and voxel visibility results in the spatial occlusion relationship as adaptive learnable weights, the image features of the view image are aggregated in the spatial features of the bird's-eye view corresponding to the point cloud data.

2. The target detection method according to claim 1, characterized in that, Based on the point cloud data, the process of performing depth prediction on the view image and visibility segmentation on the bird's-eye view in the space to obtain spatial occlusion relationships includes: Based on the depth information of the point cloud data, depth prediction is performed on each pixel in the view image to obtain the predicted pixel depth of each pixel in the view image; Based on the point cloud data, the bird's-eye view is segmented by visibility to obtain the voxel visibility results of each voxel in the bird's-eye view; The spatial occlusion relationship is determined based on the predicted pixel depth and the voxel visibility results.

3. The target detection method according to claim 2, characterized in that, The process of estimating the depth of each pixel in the view image based on the depth information of the point cloud data includes: Based on the positional correspondence between the point cloud data and the view image, and the depth information of the point cloud data, the point cloud supervision depth of each pixel in the view image is determined. Based on the point cloud supervised depth and the image features of the view image, depth prediction is performed on each pixel in the view image.

4. The target detection method according to claim 2, characterized in that, The process of performing visibility segmentation on the bird's-eye view based on the point cloud data to obtain the voxel visibility results of each voxel in the bird's-eye view includes: The point cloud data is converted to the space where the bird's-eye view is located to obtain projected point cloud data; The system detects whether each voxel in the bird's-eye view has a corresponding radar point in the projected point cloud data, and determines the voxel visibility result of each voxel based on the detection result.

5. The target detection method according to claim 4, characterized in that, The process of detecting whether each voxel in the bird's-eye view has a corresponding radar point in the projected point cloud data, and determining the voxel visibility result of each voxel based on the detection result, includes: Based on the historical spatial features of the bird's-eye view at a historical moment, and the coordinate transformation information from the historical moment to the current moment, the spatial features of the bird's-eye view at the current moment are enhanced. Based on the enhanced spatial features, it is detected whether each voxel in the bird's-eye view has a corresponding radar point in the projected point cloud data, and the voxel visibility result of each voxel is determined based on the detection result.

6. The target detection method according to any one of claims 1 to 5, characterized in that, Based on the point cloud data, depth prediction is performed on the view image and visibility segmentation is performed on the bird's-eye view in the space to obtain spatial occlusion relationships; Based on the spatial occlusion relationship, feature aggregation is performed on the view image and the point cloud data, and target detection is performed based on the aggregation result, including: Based on the target detection model, the point cloud data is used to perform depth prediction on the view image and visibility segmentation on the bird's-eye view in the space. Spatial occlusion relationships are obtained and applied to perform feature aggregation on the view image and the point cloud data. The aggregation results are then used for target detection. The target detection model is trained based on sample point cloud data, sample view images, target detection labels, depth labels, and visibility labels.

7. The target detection method according to claim 6, characterized in that, The training steps of the object detection model include: Determine an initial model, which includes at least two candidate detection heads; Based on the initial model, at least two target detection results corresponding to the sample point cloud data and the sample view image, a depth prediction result of the sample view image, and a visibility prediction result of the sample bird's-eye view are determined; the at least two target detection results are output by the at least two candidate detection heads respectively; Based on the depth prediction results and the depth labels, the depth prediction loss is determined; Based on the visibility prediction results and the visibility labels, the visibility segmentation loss is determined; Based on the target detection label and the at least two target detection results, the detection loss is determined; Based on the depth prediction loss, the visibility prediction loss, and the detection loss, the initial model is iterated to obtain the target detection model.

8. A target detection device, characterized in that, include: The acquisition unit acquires view images and point cloud data, wherein the view images and point cloud data are acquired in the same space; Determine the spatial occlusion unit, and based on the point cloud data, perform depth prediction on the view image and visibility segmentation on the bird's-eye view under the space to obtain the spatial occlusion relationship; The detection unit performs feature aggregation on the view image and the point cloud data based on the spatial occlusion relationship, and performs target detection based on the aggregation result; The detection unit is specifically used for: Using the predicted pixel depth and voxel visibility results in the spatial occlusion relationship as adaptive learnable weights, the image features of the view image are aggregated in the spatial features of the bird's-eye view corresponding to the point cloud data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the target detection method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the target detection method as described in any one of claims 1 to 7.

Images