A monocular image 3D target detection method based on neural volume rendering

Abstract

The application discloses a monocular image 3D target detection method based on neural volume rendering. The original input RGB image is first subjected to a 2D image backbone network to extract 2D image features; a multi-plane image interval sampling is used to obtain a position view cone feature; the 2D image features and the position view cone feature are fused to obtain a position perception view cone feature; the position perception view cone feature is processed by using a 3D convolution network to establish a neural volume rendering feature based on a signed distance field; the optimized neural volume rendering feature is obtained according to the neural volume rendering feature in combination with a reconstruction consistency constraint of volume rendering and a zero level set constraint of the signed distance field; finally, the 3D detection feature is obtained through grid sampling, and the 3D target detection result is obtained by inputting a general target detection head. The monocular image 3D target detection method based on neural volume rendering is first proposed, which can effectively perform a monocular image 3D target detection task and simultaneously predict a 3D occupation, and has higher detection precision.

Description

Technical Field

[0001] This invention relates to an image processing method in computer vision and autonomous driving, specifically a monocular 3D target detection method based on neural body rendering. Background Technology

[0002] Monocular 3D object detection is one of the most important and challenging problems in computer vision. In this task, given a single image, we need to detect objects of the desired category and output their position and size in 3D space. An efficient monocular 3D object detector can be widely used in many fields based on visual perception, such as autonomous vehicles and security robots, and can also provide data support for the training of other downstream computer vision tasks.

[0003] Significant progress has been made in monocular 3D object detection in recent years. However, existing monocular methods typically rely on depth estimation to achieve their relational constraints from 2D images to 3D space.

[0004] However, the current 3D representation derived from display depth estimation has some limitations.

[0005] First, 3D features obtained from depth estimation or pseudo-LiDAR exhibit a non-uniform distribution throughout the 3D space. Specifically, they have a higher density at close range, but the density decreases with increasing distance.

[0006] Secondly, the final 3D object detection performance heavily depends on the accuracy of depth estimation, and improving the accuracy of depth estimation remains a significant challenge.

[0007] Therefore, this representation method cannot generate dense and reasonable 3D features for monocular image 3D object detection tasks.

[0008] In the field of monocular 3D object detection, while there are approaches that utilize scene geometric cues to enhance detectors, many existing methods explicitly utilize these cues, such as estimating depth maps and backprojecting them into 3D space. Due to the increase in dimensionality from 2D to 3D, such explicit methods lead to sparsity in the 3D representation and result in the loss of significant information, especially for distant and occluded objects. Summary of the Invention

[0009] To address and alleviate the problems existing in the background technology, this invention innovatively proposes a method for 3D target detection of monocular images based on neural body rendering. The method can effectively perform 3D target detection tasks of monocular images and simultaneously generate 3D occupancy.

[0010] The application is to re-express the intermediate 3D representation in monocular image 3D target detection as a 3D representation similar to neural radiance field (NeRF), thereby generating dense set of 3D geometry and occupancy information, according to which 3D target detection is carried out.

[0011] The technical scheme adopted by the application is:

[0012] 1) For the original input RGB image, first extract 2D image features through a 2D image backbone network;

[0013] The 2D image backbone network adopts a Resnet34 neural network.

[0014] 2) Use a multi-plane image interval sampling method to obtain the three-dimensional coordinates of the viewing cone corresponding to the input RGB image, and normalize the three-dimensional coordinates as position viewing cone features;

[0015] 3) Fuse the 2D image features and the position viewing cone features by means of a query-key value pair to obtain position-aware viewing cone features;

[0016] 4) Use a 3D convolution network to process the position-aware viewing cone features to establish a neural volume rendering feature based on a signed distance field;

[0017] 5) According to the reconstruction consistency constraint of the neural volume rendering feature combined with the zero level set constraint of the signed distance field, obtain the optimized neural volume rendering feature;

[0018] 6) After the optimized neural volume rendering feature is sampled through a grid, 3D detection features are obtained, and the 3D target detection results are obtained by inputting the 3D detection features into a general target detection head.

[0019] All the above steps of the application constitute a monocular image 3D target detection network as a whole.

[0020] The step 2) is specifically: according to the pre-defined near depth z n and far depth z f , a depth range is established, i.e. the depth range from the near depth z n to the far depth z f , D depths are sampled from the depth range, the sampling process uses equal depth intervals and random perturbations, each depth establishes a depth plane, each depth plane is divided into a plurality of grid pixels, each depth plane and the 2D image feature resolution (HxW) of step 1) are consistent, i.e. have (HxW) pixel grids, each grid pixel takes its own plane coordinates on the depth plane plus the depth of the depth plane as its own three-dimensional coordinates; and the 3D representation three-dimensional matrix obtained after normalizing the three-dimensional coordinates of all grid pixels of all depth planes is taken as the position viewing cone features.

[0021] The 3D position coordinate p = [u, v, z]T of the view frustum of each grid pixel in each depth plane is composed of the plane coordinate [u, v]T of the grid pixel in the depth plane and the depth [z]T.

[0022] The step 3) query-key-value method specifically comprises a position query module, a key mapping module, a value mapping module and a fusion module comprising a softmax function, and is used for fusing the position frustum feature and the 2D image feature to construct a position-aware frustum feature.

[0023] The position query feature Q is obtained by inputting the position frustum feature into the position query module for processing, and the key mapping feature K and the value mapping feature V are obtained by inputting the 2D image feature into the key mapping module and the value mapping module respectively for processing.

[0024] Then, the position query feature Q and the key mapping feature K are multiplied, and then the multiplication means multiplication corresponding to the same position, and then the multiplication result is scaled in the depth dimension by using the softmax function and multiplied by the value mapping feature V to obtain the position-aware frustum feature.

[0025] The position query module f q is:

[0026] f q : F pos → Q

[0027] wherein F pos is the position frustum feature, Q is the position query feature, and the learnable function f q realizes the mapping from F pos to Q;

[0028] The key mapping module f k is:

[0029] f k : F image → K

[0030] wherein F image is the 2D image feature, K is the key mapping feature, and the learnable function f k realizes the mapping from F image to K;

[0031] The value mapping module f v is:

[0032] f v : F image → V

[0033] wherein F image is the 2D image feature, V is the value mapping feature, and the learnable function fv implementing a mapping from F image to V;

[0034] The fusion module is:

[0035] F P = Softmax(QK, dim=D) V

[0036] wherein F P is a position-aware view-frustum feature, Q, K, V are distributed as the aforementioned position query feature, key mapping feature and value mapping feature, dim represents the dimension of depth, and Softmax(·, dim=D) is a softmax function calculation along the aforementioned depth sampling axis (sampling D depth dimensions).

[0037] In the step 4), the neural volume rendering feature based on the signed distance field is specifically four parts including a signed distance field feature, a volume density feature, a 3D intermediate feature and an RGB color feature, which are respectively realized by a 3D convolution network with three learnable parameters and a Laplace cumulative distribution function with one learnable parameter:

[0038] After constructing the position-aware view-frustum feature, the position-aware view-frustum feature is input to a first 3D convolution network to output total 3D features, and any one-dimensional feature is taken from the total 3D features as a signed distance field (SDF) feature, while the position-aware view-frustum feature is also input to a second 3D convolution network to output total 3D features, and the remaining dimensional features except the signed distance field (SDF) feature are taken from the total 3D features as 3D intermediate features, and the signed distance field feature is converted and processed into a volume density feature required for volume rendering by a Laplace cumulative distribution function.

[0039] The first 3D convolution network is:

[0040] f1: F P → F sdf

[0041] wherein F P is a position-aware view-frustum feature, F sdf is a signed distance field feature, and a 3D convolution network f1 with learnable 3D parameters realizes a mapping from F P to F sdf ;

[0042] The second 3D convolution network is:

[0043] f2: F P → F 3D

[0044] wherein F P is a position-aware view-frustum feature, F 3DFor 3D intermediate features, a 3D convolution network f2 with learnable 3D parameters is used to realize the mapping from F P to F 3D ;

[0045] where the Laplace cumulative distribution function is:

[0046] αΨ β : F sdf → F density

[0047] where ψ β is a Laplace cumulative distribution function with zero mean and β as the scaling factor, α and β are the first and second learnable parameters respectively, F sdf is the signed distance field feature, F density is the volume density feature, and ψ β realizes the mapping from the signed distance to the volume density, which is used to form a homogeneous volume density.

[0048] The signed distance field (SDF) feature has multiple small lattices, each of which serves as an element, and each small lattice has a scalar value as the value of the element. This is to encode the position-aware view frustum feature into three-dimensional space to obtain a three-dimensional feature with more information for neural volume rendering.

[0049] In the method, the volume density feature and the 3D intermediate feature are combined to establish a loss constraint for supervised training and learning, that is, the overall detection network is trained, specifically:

[0050] Before the training supervision process, each RGB image has a sparse LiDAR point (LiDAR point) to form a point cloud, which serves as the depth data and training label of the RGB image.

[0051] The 3D intermediate feature is input into a third 3D convolution network to obtain the corresponding radiation field as the RGB color feature. After obtaining the volume density feature and the RGB color feature, the volume density feature and the RGB color feature are recovered from the 3D space through volume rendering to obtain the reconstructed RGB image and the corresponding reconstructed depth map.

[0052] where the third 3D convolution network is:

[0053] f3: F 3D → F RGB

[0054] where F 3D is the 3D intermediate feature, F RGB is the RGB color feature, and a 3D convolution network f3 with learnable 3D parameters realizes the mapping from F 3D to F RGB ;

[0055] and the original depth map is obtained by projecting the sparse LiDAR points (LiDAR points) that the original input RGB image has by itself;

[0056] Color consistency loss is established between the reconstructed RGB image and the original input RGB image, and sparse depth Figure 1 consistency loss is established between the reconstructed depth map and the original depth map obtained by projecting the sparse LiDAR points.

[0057] In this way, the SDF is supervised by the LiDAR corresponding to the original input RGB image, so that the SDF element values corresponding to all LiDAR points are 0 (LiDAR points usually only appear on the surface of an object).

[0058] In step 5), the model is trained in a self-supervised manner, and the model is optimized when the loss function converges, wherein the loss function is:

[0059] arg min λ rgb L rgb +λ depth L depth +λ sdf L sdf

[0060] wherein L rgb represents the color consistency loss between the reconstructed RGB image obtained by volume rendering and the current original input RGB image, L depth represents the sparse depth Figure 1 consistency loss between the reconstructed depth map obtained by volume rendering and the sparse depth map constructed by projecting the LiDAR points (LiDAR) to the camera imaging plane, L sdf represents the zero level set constraint; the reconstruction consistency is ensured by L rgb and L depth , and the zero level set constraint of the signed distance field is ensured by L sdf . λ rgb , λ depth , λ sdf are adjustable weights of the color consistency loss, the sparse depth Figure 1 consistency loss, and the zero level set constraint, respectively.

[0061] The specific losses are as follows:

[0062] The color consistency loss Lrgb including smooth mean absolute error L smoothL1 and structural similarity error L SSIM :

[0063] L rgb = λ smoothL1 L smoothL1 + λ SSIM L SSIM

[0064] where λ smoothL1 and λ SSIM are the corresponding adjustable weights of smooth mean absolute error L smoothL1 and structural similarity error L SSIM ;

[0065] The reconstructed depth map obtained by volume rendering and the sparse depth map constructed by projecting the laser radar points (LiDAR) to the camera imaging plane Figure 1 consistency loss L depth :

[0066]

[0067] where N depth represents the number of effective points after projecting the laser radar points to the camera imaging plane, represents the reconstructed depth map obtained by rendering, Z gt represents the sparse depth map obtained by projection, and |||1 represents the 1-norm;

[0068] Regarding the zero level set constraint, the signed distance field describes the geometric surface in the scene with a set of 0 values, and all laser radar points (LiDAR) are on the geometric surface, and the constraint L sdf is:

[0069]

[0070] where N gt represents the number of effective laser radar points in the 3D target detection range, F sdf is a signed distance field feature, (x, y, z) represents the three-dimensional coordinates corresponding to the effective laser radar points, and |||2 represents the 2-norm.

[0071] Two reconstruction losses are established in the method, one is to establish a reconstruction loss between the reconstructed RGB image and the original input RGB image, and the other is to establish a reconstruction loss between the reconstructed depth map and the original depth map.

[0072] The numerical integration formula used in step 5) of volume rendering is as follows:

[0073]

[0074]

[0075]

[0076] where r represents a ray emitted from the camera optical center, represents the RGB color value of ray r reflected on the image, represents the depth value of ray r reflected on the image, T i represents the accumulated transparency of ray r from depth plane 1 to depth plane i, D is the number of the sampled depth planes, z i represents the corresponding depth value of depth plane i, represents the RGB color according to z i from the volume density feature F density sampled, represents the RGB color according to z i from the RGB color feature F RGB sampled, represents the depth interval between depth plane j and depth plane j+1.

[0077] In the step 6), the feature voxel V f is obtained by coordinate sampling from the 3D intermediate feature according to the pre-defined voxel space coordinates, the density voxel V density is obtained by coordinate sampling from the volume density feature according to the pre-defined voxel space coordinates, and the feature voxel V density is specially weighted by the density voxel V f to obtain the 3D detection feature finally sent into the general object detection head, and the 3D detection feature is input into the 3D detection head to obtain the detection result of the depth, which is specifically represented as:

[0078] V density = G(F density )

[0079] V f = G(F 3D )

[0080] V 3D = V f · tanh(V density )

[0081] where G(·) represents the grid sampling operation, tanh(·) represents the hyperbolic tangent activation function, which is used to scale the density voxel, and V 3D is the 3D detection feature finally sent into the general object detection head.

[0082] The neural volume rendering feature input into the general object detection head includes the density voxel V densityand the feature voxels V f , respectively, are sampled from the aforementioned body density features F density and 3D intermediate features F 3D over a predefined 3D grid.

[0083] The present application uses the 3D features supervised by volume rendering after being weighted by the body density to input into the 3D detection head to predict the detection result.

[0084] The present application combines the extracted 2D image features and the corresponding normalized position cone features to construct the position-aware cone features fused with 3D position information, and then uses the position-aware cone features to create the signed distance field feature (SDF) and the RGB color feature (RGB color), and further extracts the voxel weighted prediction to obtain the detection result.

[0085] The present application uses the reconstructed RGB image and the reconstructed depth map generated from the signed distance field feature and the RGB color feature by volume rendering, and supervises the training and learning by establishing the loss between the original RGB image and the LiDAR points, and optimizes the implicit modeling of the scene through the zero-level set of the signed distance field.

[0086] Specifically, the present application uses the signed distance function (Signed Distance Functions, SDF) to model the scene, thereby facilitating the generation of dense neural volume rendering features. And these neural volume rendering features are regarded as neural radiance fields (Neural Radiance Fields, NeRF), and then the classic volume rendering technology is used to reconstruct the RGB image and the depth map, which enables the network to infer the dense 3D geometric information and the occupancy (3D-Occupancy) situation, effectively performs the monocular image 3D target detection task, and produces the 3D occupancy.

[0087] The present application has the following beneficial effects:

[0088] The present application first proposes a monocular image 3D target detection method based on neural volume rendering, which maintains high detection accuracy and can predict 3D occupancy; and uses a self-supervised method to optimize the neural volume rendering features during training, without the need for additional manpower for data labeling. BRIEF DESCRIPTION OF DRAWINGS

[0089] Figure 1 The functional module diagram of the present application example.

[0090] Figure 2 The monocular image 3D target detection method based on neural volume rendering designed in the present application example is a step flow chart. DETAILED DESCRIPTION

[0091] The application will be further described below in conjunction with the drawings and specific embodiments.

[0092] 1) For the original input RGB image, first extract 2D image features through a 2D image backbone network;

[0093] The 2D image backbone network in the specific implementation adopts a Resnet34 neural network.

[0094] 2) Use the 2D image features to generate a 3D representation three-dimensional matrix close to the NeRF representation, as the positional view frustum feature;

[0095] The step 2) is specifically: establish a depth range from the near depth z_n to the far depth z_f according to the predefined near depth z_n and far depth z_f, sample D depths from the depth range, the sampling process uses equal depth intervals and random perturbations, each depth establishes a depth plane, each depth plane is consistent with the 2D image feature resolution (H x W) of the step 1) in terms of having (H x W) pixel grids, each grid pixel takes its own planar coordinates on the depth plane plus the depth of the depth plane as its three-dimensional coordinates; and the 3D representation three-dimensional matrix obtained after normalizing the three-dimensional coordinates of all grid pixels of all depth planes is taken as the positional view frustum feature.

[0096] In each depth plane, the view frustum 3D position coordinates p = [u, v, z]T of each grid pixel are composed of the planar coordinates [u, v]T of the grid pixel on the depth plane plus the depth [z]T.

[0097] 3) Fuse the positional view frustum feature and the 2D image feature to construct a location-aware view frustum feature;

[0098] The query-key value method specifically includes a location query module, a key mapping module, a value mapping module, a fusion module, and a softmax function, and is used for fusing the positional view frustum feature and the 2D image feature to construct a location-aware view frustum feature.

[0099] The location query module is:

[0100] f q : F pos → Q

[0101] Where F pos is the positional view frustum feature, Q is the location query feature, and the learnable function f q realizes the mapping from F pos to Q.

[0102] The key mapping module is:

[0103] f k: F image → K

[0104] where F image is a 2D image feature, K is a key mapping feature, and a learnable function f k achieves the mapping from F image to K.

[0105] The value mapping module is:

[0106] f v : F image → V

[0107] where F image is a 2D image feature, V is a value mapping feature, and a learnable function f v achieves the mapping from F image to V.

[0108] The fusion module is:

[0109] F P = Softmax(QK, dim=D)V

[0110] where F P is a position-aware view cone feature, Q, K, and V are the position query feature, the key mapping feature, and the value mapping feature described above, and Softmax(·, dim=D) is a softmax function calculation along the aforementioned depth sampling axis (sampling D depth dimensions).

[0111] 4) The position-aware view cone feature is processed using a 3D convolutional network and a Laplace cumulative distribution function to obtain a corresponding neural volume rendering feature, and then combined with volume rendering processing to optimize the prediction to obtain the detection result of the depth.

[0112] The neural volume rendering feature based on the signed distance field in the step 4) specifically includes four parts of a signed distance field feature, a volume density feature, a 3D intermediate feature, and an RGB color feature, which are respectively realized by a 3D convolutional network with 3 learnable parameters and a Laplace cumulative distribution function with 1 learnable parameter.

[0113] where the first 3D convolutional network is:

[0114] f1: F P → F sdf

[0115] where F P is a position-aware view cone feature, F sdf is a signed distance field feature, and a 3D convolutional network f1 with learnable 3D parameters achieves the mapping from F P to F sdf .

[0116] where the second 3D convolutional network is:

[0117] f2: F P → F 3D

[0118] where F P is a position-aware view sphere feature, F 3D is a 3D intermediate feature, and a 3D convolutional network f2 with learnable 3D parameters implements the mapping from F P to F 3D .

[0119] where the third 3D convolutional network is:

[0120] f3: F 3D → F RGB

[0121] where F 3D is a 3D intermediate feature, and F RGB is an RGB color feature, and a 3D convolutional network f3 with learnable 3D parameters implements the mapping from F 3D to F RGB .

[0122] where the Laplace cumulative distribution function is:

[0123] αΨ β : F sdf → F density

[0124] where α, β are learnable parameters, ψ β is a Laplace cumulative distribution function with zero mean and β as the scaling dimension, F sdf is a signed distance field feature, and F density is a volume density feature, and ψ β implements the mapping from the signed distance to the volume density for forming a homogeneous volume density.

[0125] 5) Before the specific implementation of 3D target detection, the method also needs to be trained and learned, including self-supervised training of the volume rendering part and supervised training of the 3D target detection part. The supervised training of the 3D target detection part is irrelevant to the general concept and innovation point, and will not be described here. The learning process of the volume rendering part is specifically:

[0126] The model can be trained in a self-supervised manner, and the model optimization is completed when the loss function converges, where the loss function is:

[0127] arg min λ rgb L rgb +λ depth L depth +λ sdfL sdf

[0128] Each loss is specifically:

[0129] Color consistency L between the reconstructed RGB image from volume rendering and the current original input RGB image rgb Including smooth mean absolute error L smoothL1 And structural similarity error L SSIM , λ smoothL1 And λ SSIM Are their corresponding adjustable weights respectively:

[0130] L rgb = λ smoothL1 L smoothL1 + λ SSIM L SSIM

[0131] Consistency L between the reconstructed depth map from volume rendering and the sparse depth map constructed by projecting laser radar points (LiDAR) to the camera imaging plane depth :

[0132]

[0133] Regarding the zero level set constraint, the signed distance field describes the geometric surface in the scene with the set of 0 values, and all laser radar points (LiDAR) are on the geometric surface, and the constraint L sdf is:

[0134]

[0135] 6) The neural volume rendering features input into the general object detection head include density voxels V density And feature voxels V f , which are respectively sampled from the aforementioned volume density features F density And 3D intermediate features F 3D Through a predefined 3D grid:

[0136] V density = G(F density )

[0137] V f = G(F 3D )

[0138] V 3D = V f · tanh(V density )

[0139] Wherein, G(·) represents a grid sampling operation, tanh(·) represents a hyperbolic tangent activation function, which is used to scale the density voxels, V 3DTo be fed into the general object detection head for 3D detection features.

Claims

1. A monocular image 3D target detection method based on neural volume rendering, characterized in that: The method comprises the following steps: 1) For the original input RGB image, first extract 2D image features through a 2D image backbone network; 2) Use a multi-plane image interval sampling method to obtain the three-dimensional coordinates of the viewing cone corresponding to the input RGB image, and normalize the three-dimensional coordinates as the position viewing cone features; 3) Fuse the 2D image features and the position viewing cone features by means of a query-key value pair method to obtain position-aware viewing cone features; 4) Process the position-aware viewing cone features using a 3D convolution network to establish a neural volume rendering feature based on a signed distance field; 5) Obtain optimized neural volume rendering features according to the neural volume rendering features combined with the reconstruction consistency constraint of volume rendering and the zero level set constraint of the signed distance field; 6) After the optimized neural volume rendering features are sampled through a grid, 3D detection features are obtained, which are input into a general object detection head to obtain 3D object detection results.

2. The monocular image 3D target detection method based on neural volume rendering according to claim 1, characterized in that: The step 2) is specifically: according to the predefined near depth z n and far depth z f establish a depth range, sample D depths from the depth range, the sampling process uses equal depth intervals and random perturbations, each depth establishes a depth plane, each depth plane is divided into a plurality of grid pixels, each grid pixel takes the plane coordinates of itself on the depth plane plus the depth of the depth plane as its three-dimensional coordinates; then the three-dimensional matrix obtained after normalizing the three-dimensional coordinates of all grid pixels of all depth planes is taken as the positional view frustum feature.

3. The monocular image 3D target detection method based on neural volume rendering according to claim 2, characterized in that: In each depth plane, the 3D position coordinates of the viewing cone of each grid pixel p = [u, v, z]T are composed of the plane coordinates [u, v]T of the grid pixel in the depth plane and the depth [z]T.

4. The monocular image 3D target detection method based on neural volume rendering according to claim 1, characterized in that: The step 3) query-key value pair method specifically comprises a position query module, a key mapping module, a value mapping module, and a fusion module comprising a softmax function, which is used to fuse the position viewing cone features and the 2D image features to construct the position-aware viewing cone features; The position query features Q are obtained by inputting the position viewing cone features into the position query module, and the key mapping features K and the value mapping features V are obtained by inputting the 2D image features into the key mapping module and the value mapping module, respectively; Then, the position query features Q and the key mapping features K are multiplied, and then the multiplied result is scaled in the depth dimension using the softmax function and multiplied with the value mapping features V to obtain the position-aware viewing cone features; specifically, The position query module f q is: f q : F pos → Q wherein F pos is a location frustum feature, Q is a location query feature; The key mapping module f k is: f k : F image → K where F image is a 2D image feature, K is a key map feature; The value mapping module f v is: f v : F image → V where F image is a 2D image feature, and V is a value mapping feature. The fusion module is: F P = Softmax(QK, dim=D) V where F P is a position-aware pyramid feature, Q, K, V are the position query feature, key map feature and value map feature, and dim represents the dimension of the depth.

5. The monocular image 3D object detection method based on neural volume rendering according to claim 1, wherein: In the step 4), the neural volume rendering feature based on the signed distance field is specifically: The position-aware viewing cone features are input into a first 3D convolution network to output total 3D features, and one-dimensional features are taken from the total 3D features as signed distance field features; at the same time, the position-aware viewing cone features are also input into a second 3D convolution network to output total 3D features, and features of the remaining dimensions except the signed distance field features are taken from the total 3D features as 3D intermediate features; then the signed distance field features are converted into volume density features required for volume rendering by a Laplace cumulative distribution function; specifically: The first 3D convolution network is: f1 : F P → F sdf where F P is a position-aware view-frustum feature, F sdf is a signed distance field feature; The second 3D convolution network is: f2: F P → F 3D where F P is a position-aware frustum feature, F 3D is a 3D intermediate feature; The Laplace cumulative distribution function is: αΨ β : F sdf → F density where ψ β is the Laplace cumulative distribution function with zero mean and scale β, and α, β are the first and second learnable parameters, respectively, F sdf is the signed distance field feature, and F density is the volume density feature.

6. The monocular image 3D object detection method based on neural volume rendering according to claim 1 or 5, wherein: In the method, the volume density features and the 3D intermediate features are combined to establish a loss constraint for supervised training and learning, specifically: The 3D intermediate features are input into a third 3D convolutional network to obtain RGB color features; after the volume density features and the RGB color features are obtained, the reconstructed RGB image and the corresponding reconstructed depth map are recovered from the 3D space through volume rendering based on the volume density features and the RGB color features; The third 3D convolutional network is as follows: f3: F 3D → F RGB where F 3D is a 3D intermediate feature, F RGB is an RGB color feature; And the original input RGB image is used to project the original depth map from the laser radar points (LiDAR points) carried by the original input RGB image; Color consistency loss is established between the reconstructed RGB image and the original input RGB image, sparse depth map consistency loss is established between the reconstructed depth map and the original depth map projected by the laser radar points, and the following loss constraints are also established: the three-dimensional coordinates of each laser radar point carried by the original input RGB image are found to correspond to the elements in the signed distance field features, and the zero level set constraint is established in the case that the elements are found, with the values of the elements being 0.

7. The monocular image 3D target detection method based on neural volume rendering according to claim 6, characterized in that: In step 6), the feature voxels V are obtained from the 3D intermediate feature by coordinate sampling according to the predefined voxel space coordinates f The density voxels V are obtained from the volume density feature by coordinate sampling according to the predefined voxel space coordinates density The 3D detection feature is obtained after the special weighting processing of the feature voxels V density The 3D detection feature is obtained after the special weighting processing of the feature voxels V f The final 3D detection feature is obtained after the special weighting processing of the feature voxels V, which is input into the general object detection head, and the 3D detection feature is input into the 3D detection head to obtain the detection result of the depth, which is specifically represented as V density = G(F density ) V f = G(F 3D ) V 3D = V f tanh(V density ) where G(·) denotes a grid sampling operation, tanh(·) denotes a hyperbolic tangent activation function used to scale the density voxels, V 3D are the final 3D detection features fed into a general-purpose object detection head.

Images