Implicit road surface field learning method for roadside monocular three-dimensional target detection

Abstract

The application discloses a kind of implicit road surface field learning methods for roadside monocular three-dimensional target detection, specifically: obtaining roadside image, camera parameter and road calibration plane, using the way of calibration plane and residual height field combination to continuously model complex road geometry, construct bird's-eye grid and project image features to bird's-eye space, predict the residual height relative to calibration plane, obtain continuous road surface field;Then through road data, smoothing, plane regularity and consistency constraint, jointly train with three-dimensional detection network, output target three-dimensional position, size and orientation.The application can effectively improve the three-dimensional detection precision, robustness and practicality in complex roadside scene.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and vehicle-road cooperative perception technology, and in particular relates to an implicit road surface field learning method for roadside monocular three-dimensional target detection. Background Technology

[0002] Monocular 3D target detection refers to estimating the position, dimensions (length, width, height), and orientation of an object in a scene within a single image captured by a single ordinary RGB camera. Compared to LiDAR, multi-view cameras, or millimeter-wave radar solutions, monocular solutions offer lower hardware costs, more flexible deployment, and easier maintenance, making them particularly suitable for large-scale road infrastructure construction scenarios.

[0003] In vehicle-to-everything (V2X) or roadside perception scenarios, cameras are typically mounted several meters or even higher above the ground. This provides advantages such as a wider viewing angle, broader coverage, and less obstruction, enabling more comprehensive perception information for autonomous vehicles, traffic management platforms, and digital road infrastructure. However, precisely because roadside cameras are mounted at higher heights, have larger pitch angles, and exhibit significant differences in extrinsic parameters at different locations, their imaging geometry is more complex compared to vehicle-mounted cameras. This makes it difficult to directly apply many monocular 3D detection algorithms designed for vehicle-mounted perspectives in roadside scenarios.

[0004] One common approach in existing methods is to introduce a road plane prior. This means assuming the entire road surface is approximately a mathematical plane and estimating the contact relationship between the target and the road accordingly. This method is effective on flat roads, but in real urban and highway environments, roads are often not a single plane. For example, there are on-ramps and off-ramps, bridge approach roads, undulating slopes, grade-separated interchanges, multi-level roads, median strips, and height differences between pedestrian and vehicular lanes. If a single plane is still used to represent the entire road segment, systematic height errors will be introduced, especially at long distances, where these errors will be amplified into significant depth errors and positioning deviations.

[0005] Another type of existing approach attempts to describe roads using segmented planes or pixel-by-pixel plane parameters. While these methods are more flexible than single-plane representations, they still have shortcomings. On the one hand, discontinuous transitions can easily occur between segmented planes, which does not conform to the physical law that real roads are usually relatively smooth in space. On the other hand, these methods often only achieve strong supervision in local areas with labeled targets, and the geometric constraints on road areas not covered by targets are insufficient, making unstable predictions prone to occur. In addition, many existing methods simply input road information as auxiliary features into the target detection network, and there is a lack of strong constraints between road branches and target detection branches, failing to truly utilize road geometry to eliminate the inherent ambiguities in monocular depth estimation.

[0006] Monocular depth ambiguity refers to the phenomenon where a single pixel may correspond to multiple different depth positions along the line of sight in space; without additional geometric constraints, it is difficult to determine its true distance based solely on the appearance of the image. This problem is even more pronounced for roadside high-position cameras, where distant targets exhibit smaller scale variations in the image. Therefore, constructing a road geometry representation that can both represent complex road surface shapes and be strongly coupled with the 3D target detection process is a crucial technical problem that urgently needs to be solved in the field of roadside monocular 3D target detection. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies, such as overly simple road surface representation, insufficient adaptability to complex road scenes, difficulty in effectively eliminating monocular depth ambiguity, and loose coupling between road geometry and target detection. Therefore, this invention provides an implicit road surface field learning method for roadside monocular 3D target detection.

[0008] An implicit road surface field learning method for roadside monocular 3D target detection according to the present invention includes the following steps:

[0009] Step 1: Acquire road scene images captured by a roadside monocular camera, and obtain the corresponding camera intrinsic parameter matrix, extrinsic parameter matrix, and road calibration plane parameters. The camera intrinsic parameters are used to describe the camera focal length and principal point position, the camera extrinsic parameters are used to describe the attitude relationship between the camera coordinate system and the world coordinate system, and the road calibration plane parameters are used to characterize the reference geometric relationship of the road in the camera coordinate system.

[0010] Step 2: Extract features from the road scene image to obtain image feature maps of at least one scale, and construct a bird's-eye view grid based on the road's horizontal and vertical coordinates. The bird's-eye view is a coordinate representation of the scene viewed from above the road, used to uniformly map road areas at different distances onto a two-dimensional ground coordinate plane.

[0011] Step 3: Calculate the reference plane height of each bird's-eye view grid position based on the road calibration plane parameters, and combine the camera projection relationship to back-map each bird's-eye view grid onto the image feature map for sampling, thereby obtaining bird's-eye view features. Here, the back-mapping is the inverse perspective mapping, which is used to map the features in the image view to the spatial position on the road plane.

[0012] Step 4: Fuse the bird's-eye view features with the road calibration plane parameters and input them into the residual height prediction network to obtain the residual height of each bird's-eye view grid position relative to the reference plane height, and construct an implicit road surface field based on the reference plane height and the residual height.

[0013] Step 5: Jointly train the implicit road surface field with the 3D target detection network so that the target bottom ground point and the implicit road surface field satisfy the geometric consistency constraint, and output the target category, 3D position, size and orientation.

[0014] Step 6: In the inference phase, the trained network is used to process the input roadside monocular image and output the target 3D detection result.

[0015] Furthermore, the road calibration plane parameters satisfy the following plane equations:

[0016] ;

[0017] Where X, Y, and Z are the spatial coordinates in the camera coordinate system. , , , For planar parameters; for any bird's-eye view grid position The height of the reference plane is obtained based on the plane equation. Thus, the prior road reference height is obtained.

[0018] Furthermore, the feature extraction step employs a convolutional neural network backbone and a feature pyramid structure to extract multi-scale image features; the convolutional neural network is used to extract local texture, edge, and shape information, and the feature pyramid structure is used to fuse high-resolution detail information and low-resolution semantic information to meet the detection needs of small targets, distant targets, and large targets.

[0019] Furthermore, the residual height prediction network is a lightweight convolutional network, including at least three convolutional layers, with normalization layers and nonlinear activation layers between each convolutional layer, in order to complete the modeling of local road undulation patterns while ensuring inference speed.

[0020] Furthermore, the implicit road surface field is represented as:

[0021] ;

[0022] in, Indicates the coordinate position of the road from a bird's-eye view. The final height at the location, This indicates the reference height determined by the calibration plane. This represents the local height compensation amount output by the residual height prediction network.

[0023] The implicit road surface field is a road representation in the form of a continuous function. Instead of directly dividing the road into multiple planes, it outputs a height value for each bird's-eye view position, thus forming a one-to-one mapping relationship between two-dimensional position and height value. The implicit road surface field is represented in 2.5D, that is, each plane position corresponds to only one main road height.

[0024] Furthermore, the 3D object detection network includes a Transformer-based detection head. The Transformer is a network structure that uses an attention mechanism to model global relationships. It models global information of the image through an encoder and outputs a fixed number of candidate object prediction results through object query vectors in the decoder, thereby reducing redundant computation caused by traditional candidate box enumeration.

[0025] Furthermore, joint training includes at least the following loss function:

[0026] Road data loss is used to monitor residual height using the bottom center point of the labeled target.

[0027] The road smoothing loss is used to constrain the variation in residual height between adjacent bird's-eye view grids, so that the predicted road surface meets the physical characteristics of continuity and smoothness.

[0028] Planar regularization loss is used to limit the residual height from deviating excessively from the reference plane when the road is nearly flat.

[0029] The road consistency loss is used to constrain the height of the predicted target bottom center to be consistent with the height of the implicit road surface field, and to constrain the implicit road surface field to remain compatible with the reference plane in areas without significant undulations.

[0030] Furthermore, the road data loss is achieved by constraining the difference between the actual residual height and the predicted residual height at the bottom center position of the target annotation; the actual residual height is obtained by the difference between the actual bottom center height of the target and the height of the reference plane.

[0031] Furthermore, the road consistency loss is constructed through the following two types of consistency relationships:

[0032] The first type of consistency relationship is that the predicted bottom center height of the target is consistent with the height of the implicit road surface field at the corresponding plane position.

[0033] The second type of consistency relationship is that at the location of the predicted target, the implicit road surface field maintains a controllable deviation from the reference plane.

[0034] By using two types of consistency relationships, the road modeling branch and the target detection branch are transformed from a loosely coupled auxiliary relationship to a strongly geometrically coupled relationship.

[0035] Furthermore, the method of the present invention is applicable to the three-dimensional detection of vehicles, pedestrians, cyclists or other road users in ramps, uphill and downhill slopes, upper and lower levels of interchanges, road edge elevation differences, areas near the central median strip and other non-planar roadside traffic scenarios.

[0036] The present invention also provides an implicit road surface field learning device for roadside monocular three-dimensional target detection, comprising:

[0037] The data acquisition module is used to acquire images from roadside monocular cameras, camera intrinsic parameter matrices, extrinsic parameter matrices, and road calibration plane parameters.

[0038] The feature extraction module is used to extract multi-scale image features from the input image.

[0039] The bird's-eye view projection module is used to construct a bird's-eye view grid and map image features to the bird's-eye view feature space according to the inverse perspective mapping.

[0040] The surface field construction module is used to construct an implicit road surface field based on the prediction results of the reference plane height and residual height.

[0041] The detection coupling module is used to jointly constrain the implicit road surface field with the 3D target detection results.

[0042] The results output module is used to output the target category, 3D position, 3D size, and orientation information.

[0043] The present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0044] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0045] The beneficial technical effects of this invention compared to the prior art are as follows:

[0046] This invention introduces an implicit road surface field for roadside monocular 3D target detection tasks and uses a combination of calibration plane and residual height field to continuously model complex road geometry. Then, by using road data loss, road smoothing loss, plane regularization loss and road consistency loss, the road geometry and target 3D detection results are strongly coupled and trained, thereby effectively improving the 3D detection accuracy, robustness and practicality in complex roadside scenes. Attached Figure Description

[0047] Figure 1 This is a schematic diagram comparing existing road surface representation methods with the implicit road surface field representation method of the present invention.

[0048] Figure 2 This is a schematic diagram of the overall network framework of the present invention.

[0049] Figure 3 This is a schematic diagram of the geometric relationship of the implicit road surface field of the present invention.

[0050] Figure 4 This is a schematic diagram comparing the 3D detection effects of the present invention and the comparative method in a typical scenario. Detailed Implementation

[0051] The present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0052] This invention proposes a continuous implicit road surface field for modeling road height in a bird's-eye view coordinate system. Unlike representing a road as a single plane or several discrete planes, this method... Figure 1 As shown, this invention outputs a continuous height value for each bird's-eye view location, thus forming a function mapping from the two-dimensional road plane position to the height value. To balance geometric prior and learning flexibility, this invention adopts a hybrid modeling approach of "calibration plane + residual height field": the calibration plane provides a global geometric reference, and the residual height field is used to compensate for local deviations caused by complex road shapes. This invention projects image features onto a bird's-eye view grid through inverse perspective mapping. Inverse perspective mapping can be understood as: given the camera's internal parameters and mounting posture, the visual features originally located in the image plane are mapped inversely to their actual spatial positions on the road, thereby enabling the network to model road geometric relationships within a unified road coordinate system. In terms of object detection, this invention adopts a Transformer-based 3D object detection framework. Transformer is a type of neural network structure that uses an "attention mechanism" to model the association of long-distance information. Its advantage lies in its ability to simultaneously integrate contextual information from the entire image, providing good expressive power for crowded scenes, long-distance scenes, and multi-object scenes. This invention does not require significant modifications to the Transformer ontology. Instead, it couples the output target 3D bounding box with the implicit road surface field through loss constraints, so that the prediction results are geometrically "landed" on a reasonable road surface.

[0053] The overall network framework of the implicit road surface field learning method for roadside monocular 3D target detection of the present invention is as follows: Figure 2 As shown, it includes the following steps:

[0054] Step 1: Acquire road scene images captured by a roadside monocular camera, and obtain the corresponding camera intrinsic parameter matrix, extrinsic parameter matrix, and road calibration plane parameters. The camera intrinsic parameters are used to describe the camera focal length and principal point position, the camera extrinsic parameters are used to describe the attitude relationship between the camera coordinate system and the world coordinate system, and the road calibration plane parameters are used to characterize the reference geometric relationship of the road in the camera coordinate system.

[0055] The road calibration plane parameters satisfy the following plane equations:

[0056] ;

[0057] Where X, Y, and Z are the spatial coordinates in the camera coordinate system. , , , For planar parameters; for any bird's-eye view grid position The height of the reference plane is obtained based on the plane equation. Thus, the prior road reference height is obtained.

[0058] Step 2: Extract features from the road scene image to obtain image feature maps of at least one scale, and construct a bird's-eye view grid based on the road's horizontal and vertical coordinates. The bird's-eye view is a coordinate representation of the scene viewed from above the road, used to uniformly map road areas at different distances onto a two-dimensional ground coordinate plane.

[0059] The feature extraction step uses a convolutional neural network backbone and a feature pyramid structure to extract multi-scale image features. The convolutional neural network is used to extract local texture, edge and shape information, and the feature pyramid structure is used to fuse high-resolution detail information and low-resolution semantic information to meet the needs of detecting small targets, distant targets and large targets.

[0060] Step 3: Calculate the reference plane height of each bird's-eye view grid position based on the road calibration plane parameters, and combine the camera projection relationship to back-map each bird's-eye view grid onto the image feature map for sampling, thereby obtaining bird's-eye view features. Here, the back-mapping is the inverse perspective mapping, which is used to map the features in the image view to the spatial position on the road plane.

[0061] Step 4: Fuse the bird's-eye view features with the road calibration plane parameters and input them into the residual height prediction network to obtain the residual height of each bird's-eye view grid position relative to the reference plane height, and construct an implicit road surface field based on the reference plane height and the residual height.

[0062] The residual height prediction network is a lightweight convolutional network, consisting of at least three convolutional layers. Normalization layers and nonlinear activation layers are set between each convolutional layer to model local road undulation patterns while ensuring inference speed.

[0063] Implicit road surface field geometry relationship as follows Figure 3 As shown, specifically:

[0064] ;

[0065] in, Indicates the coordinate position of the road from a bird's-eye view. The final height at the location, This indicates the reference height determined by the calibration plane. This represents the local height compensation amount output by the residual height prediction network.

[0066] The implicit road surface field is a road representation in the form of a continuous function. Instead of directly dividing the road into multiple planes, it outputs a height value for each bird's-eye view position, thus forming a one-to-one mapping relationship between two-dimensional position and height value. The implicit road surface field is represented in 2.5D, that is, each plane position corresponds to only one main road height, which is suitable for most road surface target grounding detection scenarios.

[0067] Step 5: Jointly train the implicit road surface field with the 3D target detection network so that the target bottom ground point and the implicit road surface field satisfy the geometric consistency constraint, and output the target category, 3D position, size and orientation.

[0068] The 3D object detection network includes a Transformer-based detection head. The Transformer is a network structure that uses an attention mechanism to model global relationships. It models global information of the image through an encoder and outputs a fixed number of candidate object prediction results through the object query vector in the decoder, thereby reducing the redundant computation caused by traditional candidate box enumeration.

[0069] Joint training should include at least the following loss functions:

[0070] Road data loss is used to monitor residual height using the bottom center point of the labeled target.

[0071] The road smoothing loss is used to constrain the variation in residual height between adjacent bird's-eye view grids, so that the predicted road surface meets the physical characteristics of continuity and smoothness.

[0072] Planar regularization loss is used to limit the residual height from deviating excessively from the reference plane when the road is nearly flat.

[0073] The road consistency loss is used to constrain the height of the predicted target bottom center to be consistent with the height of the implicit road surface field, and to constrain the implicit road surface field to remain compatible with the reference plane in areas without significant undulations.

[0074] The road data loss is achieved by constraining the difference between the actual residual height and the predicted residual height at the bottom center position of the target annotation; the actual residual height is obtained by the difference between the actual bottom center height of the target and the height of the reference plane.

[0075] Road consistency loss is constructed through the following two types of consistency relationships:

[0076] The first type of consistency relationship is that the predicted bottom center height of the target is consistent with the height of the implicit road surface field at the corresponding plane position.

[0077] The second type of consistency relationship is that at the location of the predicted target, the implicit road surface field maintains a controllable deviation from the reference plane.

[0078] By using two types of consistency relationships, the road modeling branch and the target detection branch are transformed from a loosely coupled auxiliary relationship to a strongly geometrically coupled relationship.

[0079] Step 6: In the inference phase, the trained network is used to process the input roadside monocular image and output the target 3D detection result.

[0080] The 3D detection performance of the BEVHeight++ algorithm in typical scenes compared to the present invention is as follows: Figure 4As shown, in Scene 1, a roadside camera captures a straight stretch of road with multiple vehicles distributed at different distances. The BEVHeight++ algorithm fails to detect several vehicles in the distant area, resulting in significant missed detections. This is primarily due to the unreliability of height-based BEV projections at long distances due to reduced height cues. In contrast, this invention successfully recalls these distant targets by utilizing the learned implicit road surface field, providing a more accurate geometric prior for long-range depth estimation. In Scene 2, a vehicle is turning in the lower left corner of the image. The BEVHeight++ algorithm generates a predicted bounding box with a significantly different orientation from the true orientation, indicating its difficulty in achieving accurate orientation estimation in such challenging situations. Conversely, the heading angle predicted by this invention is highly consistent with the actual vehicle orientation. This improvement likely benefits from the consistency constraint of the road surface, which ensures that the predicted 3D bounding box is geometrically compatible with the underlying road surface, thus providing additional regularization for orientation estimation. In Scene 3, a tricycle appears on the right side of the scene. The BEVHeight++ algorithm incorrectly identified the tricycle as a car, as evidenced by the predicted bounding box being significantly larger than the actual object's size. However, our invention avoids this misclassification and makes a more reasonable prediction. This indicates that our method's road-aware-based BEV feature representation preserves richer geometric information about different object categories, leading to more discriminative classification. In summary, these qualitative results consistently demonstrate that our invention achieves better detection performance than the BEVHeight++ algorithm in terms of recall for distant vehicles, accuracy of orientation estimation, and robustness in object classification.

[0081] This invention includes at least the following key steps:

[0082] 1. Acquire input data: Collect roadside monocular images and read camera intrinsic and extrinsic parameters as well as road calibration plane parameters.

[0083] 2. Extracting image features: Multi-scale image features are generated using a convolutional backbone network and a feature pyramid network.

[0084] 3. Construct a bird's-eye view grid: Regularly discretize the road along the horizontal x-direction and vertical z-direction to form a bird's-eye view grid.

[0085] 4. Calculate the reference height of each grid based on the calibration plane, and back-project the position of each grid onto the image feature map for sampling to form a bird's-eye view feature.

[0086] 5. The bird's-eye view features and planar parameters are concatenated and input into the residual height prediction network to output a residual height map.

[0087] 6. Add the reference height and the residual height to obtain the implicit road surface field.

[0088] 7. Jointly train the implicit road surface field with the 3D target detection results.

[0089] Example 1:

[0090] Overall process.

[0091] This embodiment provides an implicit road surface field learning method for roadside monocular 3D target detection. The input is assumed to be an RGB image captured by a roadside camera. The image size can be set according to deployment requirements, for example, an 864×1536 resolution can be selected. Corresponding to the image, the system reads the camera intrinsic parameter matrix K and extrinsic parameter matrix. and road calibration plane parameters .

[0092] The intrinsic parameter matrix K describes the relationship between pixel coordinates and imaging rays; the extrinsic parameter matrix... The calibration plane parameters are used to describe the rotation and translation relationship of the camera relative to the road scene; the road calibration plane parameters are used to characterize the reference road plane in the camera coordinate system. The "calibration" here can be obtained from the traditional camera calibration process, or it can be provided by manual measurement, laser-assisted measurement, road mapping results, or automatic calibration algorithms.

[0093] After acquiring the basic input, a convolutional neural network (CNN) backbone is first used to extract image features. CNNs are effective at extracting local texture, edges, shape, and semantic structure information. To accommodate targets at different scales, a feature pyramid network can be further used to fuse features from different levels, enabling the network to retain both high-level semantics of large, near-field targets and local details of small, distant targets.

[0094] Subsequently, a regular grid is established from a bird's-eye view. The bird's-eye view, often abbreviated as BEV (Bird's Eye View), can be understood as the planar coordinate system used when looking down from a road surface. In this coordinate system, the horizontal coordinate... Indicates the left and right position of the road, and the vertical coordinate. It indicates the vehicle's direction of travel or the road's direction of extension. Compared to traditional image coordinate systems, the BEV coordinate system is more suitable for representing road geometry because different depth positions on the same road surface have a more uniform scale meaning in BEV.

[0095] For any bird's-eye view grid position The reference height of this point in the camera coordinate system can be calculated based on the road calibration plane equation. This height can be considered a reference height "if the road is completely flat or only sloped as a whole". Further, this 3D point is mapped onto the image feature map through camera projection, and image features at the corresponding location are extracted using differentiable sampling. Differentiable sampling means that the sampling process can participate in the backpropagation of network gradients, thereby jointly optimizing the projected features and subsequent network parameters during training.

[0096] After completing the above operations, a bird's-eye view feature map is obtained. This feature map is then concatenated with the planar parameter channels and input into a residual height prediction network. This network is preferably a lightweight convolutional neural network, for example, it can contain three convolutional layers, each followed by batch normalization and a non-linear activation function. The output of this network is... This refers to the height compensation amount of each bird's-eye view position relative to the reference plane.

[0097] The implicit road surface field is obtained through the following relationship:

[0098] ;

[0099] This formula shows that the final road height is composed of two parts: "global planar trend" and "local detail correction". For most relatively flat roads, It will approach zero; for complex road sections such as ramps, slopes, and grade-separated interchanges, This allows us to learn clearly non-zero values, which can be used to characterize the deviation between the real road and the plane assumption.

[0100] Example 2:

[0101] Joint coupling of implicit road surface field and 3D target detection.

[0102] In traditional detection methods, road branches and detection branches often exist in parallel, meaning that road information only provides some auxiliary features, but the detection results themselves may not strictly adhere to road geometric relationships. This invention, however, emphasizes the strong geometric coupling between the two.

[0103] Specifically, for targets such as vehicles, pedestrians, or cyclists on the road, the bottom center point of their 3D bounding box should typically be in contact with or nearly in contact with the road surface. Therefore, if the bottom center of a target is located in a bird's-eye view coordinate system... Then its vertical height Should be with This constraint effectively matches the depth estimation problem, which originally had multiple solutions along the imaging ray direction. It confines the problem to a smaller feasible geometric space, thereby significantly reducing depth uncertainty.

[0104] To achieve the above objectives, this invention sets up multiple loss functions during the training phase.

[0105] One aspect is road data loss. The location of the true bottom center point of the target can be obtained using bounding boxes. Substituting this location into the baseline plane equation, the true residual height can be calculated, and a difference constraint can be applied between this and the residual height predicted by the network. This provides direct supervision for road surface learning at the location where the target exists.

[0106] Secondly, there is the road smoothing loss. Real roads typically do not exhibit drastic, irregular height jumps between adjacent locations. Therefore, setting a penalty term for the residual height difference between adjacent bird's-eye view grids can make the learned surface more continuous and smooth, avoiding distortion caused by local noise or sparse annotations.

[0107] Thirdly, there is the planar regularization loss. When the scene road is approximately planar, the network should not generate large residual shifts without justification. Therefore, by imposing constraints on the absolute value of the residuals, the model can be made to deviate from the calibration plane only when there is sufficient evidence, thereby improving the model's stability and interpretability.

[0108] Fourthly, there is the road consistency loss. This loss simultaneously constrains the bottom center height of the predicted target to be consistent with the implicit road surface, and also constrains the implicit road surface to maintain a reasonable relationship with the reference plane. In this way, the target prediction result not only "looks like a target," but must also "fall on a reasonable road surface," thereby achieving a unification from visual semantic constraints to geometric constraints.

[0109] Example 3:

[0110] Further explanation of key terms and network structure.

[0111] 1. Regarding "implicit representation": Implicit representation does not directly describe a road using several explicit geometric segments, such as planes, polygons, or triangular meshes. Instead, it uses a function to return the road height at a specific location when a query is needed. During review, it can be understood as a "functional road model." The advantages of this approach are its continuity, compactness, and ease of integration with neural network training.

[0112] 2. Regarding "2.5D": 2.5D does not refer to a complete arbitrary 3D volume representation, but rather to a single principal height value y corresponding to a given position (x,z) in a 2D plane. For most target detection tasks involving vehicles, pedestrians, bicycles, etc., located on the road surface, a single plane position corresponding to a single principal road height is sufficient. Therefore, it can represent road undulations without requiring the construction of a complex voxel space, resulting in lower computational overhead.

[0113] 3. Regarding "Inverse Perspective Mapping (IPM): IPM is an abbreviation for Inverse Perspective Mapping, which can be understood as the process of "reversely extrapolating information from the image's perspective back to the road coordinate system." Because roadside cameras capture perspective images, distant targets appear smaller, while nearby targets appear larger. However, from a bird's-eye view, this perspective scaling is weakened, making it more suitable for uniformly processing targets at different distances and road structures.

[0114] 4. Regarding the "Transformer Detector Head":

[0115] The core of Transformer lies in its attention mechanism. This mechanism can be likened to "allowing the network to learn which locations in the image should be prioritized for the current target." In complex roadside traffic scenarios, where multiple targets are subject to occlusion, scale differences, and background interference, Transformer can better integrate global semantics and contextual information, thereby improving detection performance.

[0116] 5. Regarding "Strongly Coupled Training":

[0117] Strong coupling does not simply mean that multiple modules are trained in the same network, but rather that the output of one module directly influences the constraints of another module. For example, in this invention, the bottom position of the target 3D bounding box is explicitly constrained to the road surface. Therefore, the road surface field branch is no longer just a dispensable auxiliary branch, but an integral part of the target geometry prediction.

[0118] Example 4:

[0119] Example of training and inference process.

[0120] In a specific training embodiment, the invention can be trained using a roadside dataset containing target category, 2D bounding box, 3D bounding box, and camera calibration information. The optimizer is the AdamW optimizer, and the learning rate, weight decay, batch size, and number of training epochs can be set according to the hardware platform and dataset size. The road-related loss term and the basic detection loss term are weighted and summed to form the total loss; the weights can be obtained through validation set tuning.

[0121] In a typical embodiment, the bird's-eye view range can be set horizontally to [-51.2m, 51.2m] and vertically to [0m, 102.4m], with a spatial resolution of 0.4m, corresponding to a 256×256 bird's-eye view grid. Of course, under different road deployment conditions, the bird's-eye view range and resolution can also be adaptively adjusted according to the road width, camera height, intersection type, and desired detection distance.

[0122] During the inference phase, only a single roadside image and corresponding calibration parameters need to be input to output the target category, the target's 3D center point position, the target's length, width, height, and orientation angle at once. No additional LiDAR or multi-view camera assistance is required, thus providing good ease of deployment.

[0123] Example 5:

[0124] Applicable scenarios and expansion methods.

[0125] This invention can be widely applied to smart intersections, urban main roads, highway ramps, toll station entrances and exits, park roads, port roads, airport ground traffic areas, and other scenarios that require fixed high-position cameras for three-dimensional perception.

[0126] In extended implementations, implicit road surface fields can be used not only for vehicle detection but also for detecting various types of traffic participants such as pedestrians, cyclists, tricycles, and construction vehicles. Furthermore, if sidewalks, non-motorized vehicle lanes, or elevated roads of different levels exist in the scene, the invention can be extended to multi-surface joint modeling scenarios by adding multi-surface modeling branches or layered road representation methods.

[0127] In another extended implementation, the implicit road surface field can be further supervised by combining road segmentation results, sparse laser points, map priors, lane line information, or high-precision map information to further improve the accuracy of road geometry restoration.

[0128] In summary, this invention introduces an implicit road surface field for roadside monocular 3D target detection tasks and employs a combination of calibration plane and residual height field to continuously model complex road geometry. Furthermore, it utilizes road data loss, road smoothing loss, plane regularization loss, and road consistency loss to strongly couple the road geometry with the target 3D detection results during training, thereby effectively improving the accuracy, robustness, and practicality of 3D detection in complex roadside scenarios. All equivalent substitutions, improvements, or modifications made based on the core concept of this invention should fall within the protection scope of this invention.

Claims

1. An implicit road surface field learning method for roadside monocular 3D target detection, characterized in that, Includes the following steps: Step 1: Acquire road scene images captured by roadside monocular cameras, and obtain the camera intrinsic parameter matrix, extrinsic parameter matrix, and road calibration plane parameters corresponding to the images. The camera intrinsic parameters are used to describe the camera focal length and principal point position, the camera extrinsic parameters are used to describe the attitude relationship between the camera coordinate system and the world coordinate system, and the road calibration plane parameters are used to characterize the reference geometric relationship of the road in the camera coordinate system. Step 2: Extract features from the road scene image to obtain image feature maps of at least one scale, and construct a bird's-eye view grid based on the road's horizontal and vertical coordinates. The bird's-eye view is a coordinate representation of the scene viewed from above the road, used to uniformly map road areas at different distances onto a two-dimensional ground coordinate plane. Step 3: Calculate the reference plane height of each bird's-eye view grid position based on the road calibration plane parameters, and combine the camera projection relationship to back-map each bird's-eye view grid onto the image feature map for sampling to obtain bird's-eye view features. Here, the back-mapping is the inverse perspective mapping, which is used to map the features in the image view to the spatial position on the road plane. Step 4: Fuse the bird's-eye view features with the road calibration plane parameters and input them into the residual height prediction network to obtain the residual height of each bird's-eye view grid position relative to the reference plane height, and construct an implicit road surface field based on the reference plane height and the residual height. Step 5: Jointly train the implicit road surface field with the 3D target detection network to ensure that the target's bottom ground point and the implicit road surface field satisfy geometric consistency constraints, and output the target category, 3D position, size and orientation; Step 6: In the inference phase, the trained network is used to process the input roadside monocular image and output the target 3D detection result.

2. The implicit road surface field learning method for roadside monocular 3D target detection according to claim 1, characterized in that, The road calibration plane parameters satisfy the following plane equation: ； Where X, Y, and Z are the spatial coordinates in the camera coordinate system. , , , For planar parameters; for any bird's-eye view grid position The height of the reference plane is obtained based on the plane equation. Thus, the prior road reference height is obtained.

3. The implicit road surface field learning method for roadside monocular 3D target detection according to claim 1, characterized in that, The feature extraction step uses a convolutional neural network backbone and a feature pyramid structure to extract multi-scale image features. The convolutional neural network is used to extract local texture, edge and shape information, and the feature pyramid structure is used to fuse high-resolution detail information and low-resolution semantic information to meet the needs of detecting small targets, distant targets and large targets.

4. The implicit road surface field learning method for roadside monocular 3D target detection according to claim 1, characterized in that, The residual height prediction network is a lightweight convolutional network, including at least three convolutional layers. Normalization layers and nonlinear activation layers are set between each convolutional layer to complete the modeling of local road undulation patterns while ensuring inference speed. The implicit road surface field is represented as follows: ； in, Indicates the coordinate position of the road from a bird's-eye view. The final height at the location, This indicates the reference height determined by the calibration plane. This represents the local height compensation amount output by the residual height prediction network; The implicit road surface field is a road representation in the form of a continuous function. Instead of directly dividing the road into multiple planes, it outputs a height value for each bird's-eye view position, thus forming a one-to-one mapping relationship between two-dimensional position and height value. The implicit road surface field is represented in 2.5D, that is, each plane position corresponds to only one main road height.

5. The implicit road surface field learning method for roadside monocular 3D target detection according to claim 1, characterized in that, The 3D object detection network includes a Transformer-based detection head. Transformer is a network structure that uses an attention mechanism to model global correlations. It models global information of the image through an encoder and outputs a fixed number of candidate object prediction results through the object query vector in the decoder, thereby reducing redundant computation caused by traditional candidate box enumeration. The joint training includes at least the following loss function: Road data loss is used to monitor residual height using the bottom center point of the labeled target; Road smoothing loss is used to constrain the variation of residual height between adjacent bird's-eye view grids, so that the predicted road surface meets the physical characteristics of continuity and smoothness. Planar regularization loss is used to limit the residual height from deviating excessively from the reference plane when the road is nearly flat; The road consistency loss is used to constrain the height of the predicted target bottom center to be consistent with the height of the implicit road surface field, and to constrain the implicit road surface field to remain compatible with the reference plane in areas without significant undulations.

6. The implicit road surface field learning method for roadside monocular 3D target detection according to claim 5, characterized in that, The road data loss is achieved by constraining the difference between the actual residual height and the predicted residual height at the bottom center position of the target annotation; the actual residual height is obtained by the difference between the actual bottom center height of the target and the height of the reference plane. The road consistency loss is constructed through the following two types of consistency relationships: The first type of consistency relationship is that the predicted bottom center height of the target is consistent with the height of the implicit road surface field at the corresponding plane position; The second type of consistency relationship is that at the location of the predicted target, the implicit road surface field and the reference plane maintain a controllable deviation; By using two types of consistency relationships, the road modeling branch and the target detection branch are transformed from a loosely coupled auxiliary relationship to a strongly geometrically coupled relationship.

7. The implicit road surface field learning method for roadside monocular 3D target detection according to claim 1, characterized in that, The method is applicable to the three-dimensional detection of vehicles, pedestrians, cyclists or other road users in ramps, uphill and downhill slopes, upper and lower levels of interchanges, road edge elevation differences, areas near the central median strip and other non-planar roadside traffic scenarios.

8. An implicit road surface field learning device for roadside monocular three-dimensional target detection, characterized in that, include: The data acquisition module is used to acquire images from roadside monocular cameras, camera intrinsic parameter matrices, extrinsic parameter matrices, and road calibration plane parameters. The feature extraction module is used to extract multi-scale image features from the input image; The bird's-eye view projection module is used to construct a bird's-eye view grid and map image features to the bird's-eye view feature space according to the inverse perspective mapping; The surface field construction module is used to construct an implicit road surface field based on the prediction results of the reference plane height and residual height. The detection coupling module is used to jointly constrain the implicit road surface field and the three-dimensional target detection results; The results output module is used to output the target category, 3D position, 3D size, and orientation information.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 10.

Images