Scene depth estimation method and system based on road end multi-view images
Patent Information
- Application Number
- CN202610764994.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-29
- Publication Date
- 2026-08-18
AI Technical Summary
[0005]本发明提供了基于路端多视角图像的场景深度估计方法、系统，通过全局注意力机制克服卷积神经网络感受野受限的问题，有效融合多视角几何特征，并结合高精度深度回归策略，以解决现有技术在复杂场景下全局感知能力弱和深度重建精度不足的问题
1.本发明突破了稀疏视角特征隔离，实现物理空间的高精度对齐，由于传统视觉几何模型在应用于路端稀疏视角时极易匹配失效，本发明引入了基于鸟瞰图空间的占据一致性损失，将各视角的预测深度投影至统一的3D空间进行网格防碰撞与尺度对齐，有效弥补了极端视角差异造成的特征隔离，显著提升了多视角深度估计的几何一致性，使路端图像在路口三维重建与交通场景分析等应用中具备更高的可靠性与泛化能力。
Smart Images
Figure SMS_5 
Figure SMS_24 
Figure SMS_25
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, specifically to a method and system for scene depth estimation based on multi-view images from roadside locations. Background Technology
[0002] Scene depth estimation is a core task in computer vision and autonomous driving, aiming to recover the geometry of a 3D scene from a 2D image, providing crucial spatial information for roadside perception, vehicle-to-everything (V2X) communication, and digital twins. Inferring depth using geometric constraints of the same scene from different viewpoints offers advantages over active sensors like LiDAR, including lower cost, richer texture information, and ease of large-scale deployment. However, the complex backgrounds, varying lighting conditions, and large areas of weak texture in real-world road scenes pose significant challenges to achieving high-precision and robust depth information acquisition.
[0003] Currently, mainstream technologies all have significant drawbacks. Traditional stereo vision matching algorithms rely on the luminosity consistency between image patches, performing reasonably well in textured regions, but easily failing in areas with weak texture, reflectivity, or repetitive textures. The generated depth maps contain numerous holes and noise, and the computational complexity is high, making it difficult to meet real-time requirements. With the rise of deep learning, convolutional neural network-based methods have alleviated the texture dependency problem, but the inherent local convolution operations of CNNs limit the receptive field, making it difficult to capture long-distance dependencies and global contextual information. In multi-view geometric scenes, the parallax between different viewpoints can be large, and feature fusion lacking global information reduces matching accuracy, especially when dealing with slender objects or edge regions, easily resulting in the loss of details. Furthermore, existing depth estimation models have bottlenecks in depth value regression prediction. Many methods discretize continuous depth values into a classification problem, limiting the prediction results to preset depth intervals and failing to achieve sub-pixel-level high-precision reconstruction; while simple direct regression methods output continuous values, they often fail to converge to the global optimum, leading to blurred depth maps or distorted geometric structures.
[0004] Therefore, a scene depth estimation method based on multi-view images from the roadside is needed. Summary of the Invention
[0005] This invention provides a method and system for scene depth estimation based on multi-view images from the roadside. It overcomes the problem of limited receptive field in convolutional neural networks by using a global attention mechanism, effectively integrates multi-view geometric features, and combines a high-precision depth regression strategy to solve the problems of weak global perception capability and insufficient depth reconstruction accuracy in complex scenes in existing technologies.
[0006] The technical solution of this invention is as follows: The scene depth estimation method based on multi-view images of the roadside includes the following steps: Step S1. Acquire road segment images from four perspectives and preprocess them simultaneously. Cut the road segment images from the four perspectives into multiple image blocks for feature extraction to obtain image tokens. Add a learnable camera token and a set of register tokens to each perspective. Concatenate the image tokens, camera tokens and register tokens from the four perspectives to obtain an image feature sequence. Step S2. After performing intra-frame self-attention processing on the obtained image feature sequence, local spatial features are obtained. Then, global self-attention processing is performed on the local spatial features to obtain the first image feature. Intra-frame self-attention processing and global self-attention processing are performed on the first image feature in sequence to obtain the second image feature. This process is repeated for Q-1 iterations to obtain the Qth image feature. Then, the Q-th image features are split and the following branch decoding process is performed: The upper branch performs camera parameter regression head processing on the Q-th image features to obtain the camera intrinsic and extrinsic parameters corresponding to the Q-th image features; The next branch performs depth regression head processing on the Q-th image features to obtain the estimated depth map; Convergence is achieved using depth loss function, camera pose loss function, and occupancy consistency loss function.
[0007] Furthermore, intra-frame self-attention processing computes attention only between tokens within the same image.
[0008] Furthermore, global self-attention processing is used to jointly compute attention among all tokens across the four perspectives.
[0009] Furthermore, occupy the consistency loss function The details are as follows: set up For camera and camera The physical overlap region projected onto the BEV space, within which... Represents the two-dimensional coordinates of the i-th viewpoint in BEV space. The occupancy probability grid at the location, Represents the two-dimensional coordinates of the j-th viewpoint in BEV space. The occupancy probability grid at each location, where N represents the total number of camera indices.
[0010] Further, let's go to the next branch, as follows: Using the updated camera token, the rotation and translation matrices for each image are predicted via a multilayer perceptron.
[0011] Further, the next branch is as follows: Using the updated image token, features are reconstructed and upsampled by the decoder, and finally, an estimated depth map consistent with the input resolution is regressed through a convolutional layer.
[0012] Furthermore, images of the road segment from four different perspectives are acquired and preprocessed simultaneously. The specific process is as follows: Color images were acquired from four fixed viewpoints along the roadside, and the image sequence from the four viewpoints was set to be synchronized in time. , , where i represents the i-th camera viewpoint, and noise reduction processing is performed on the color image.
[0013] Furthermore, the Q-1 iterations are as follows: When Q=13, the local spatial features are first obtained by performing intra-frame self-attention processing on the obtained image feature sequence. Then, the local spatial features are subjected to global self-attention processing to obtain the first image feature. The first image feature is then subjected to intra-frame self-attention processing and global self-attention processing in sequence to obtain the second image feature. This process is repeated 12 times to obtain the thirteenth image feature.
[0014] A scene depth estimation system based on multi-view images from roadside locations includes the following: The multi-view image feature sequence construction module acquires road segment images from four perspectives and performs preprocessing simultaneously. It then cuts the four road segment images into multiple image blocks for feature extraction to obtain image tokens. Learnable camera tokens and a set of register tokens are added to each perspective. Finally, the image tokens, camera tokens, and register tokens from the four perspectives are concatenated to obtain the image feature sequence. The alternating attention multi-task decoding training module performs intra-frame self-attention processing on the obtained image feature sequence to obtain local spatial features. Then, it performs global self-attention processing on the local spatial features to obtain the first image feature. The first image feature is then subjected to intra-frame self-attention processing and global self-attention processing in sequence to obtain the second image feature. This process continues until the Q-1th image feature is obtained. Then, the Q-th image features are split and the following branch decoding process is performed: The upper branch performs camera parameter regression head processing on the Q-th image features to obtain the camera intrinsic and extrinsic parameters corresponding to the Q-th image features; The next branch performs depth regression head processing on the Q-th image features to obtain the estimated depth map; Convergence is achieved using depth loss function, camera pose loss function, and occupancy consistency loss function.
[0015] Furthermore, the intra-frame self-attention processing module in the multi-view image feature sequence construction module only calculates attention between tokens within the same image.
[0016] The beneficial effects of this invention are as follows: 1. This invention breaks through the feature isolation of sparse viewpoints and achieves high-precision alignment in physical space. Since traditional visual geometric models are prone to matching failure when applied to sparse viewpoints at road ends, this invention introduces an occupancy consistency loss based on bird's-eye view space. The predicted depth of each viewpoint is projected onto a unified 3D space for mesh anti-collision and scale alignment, which effectively compensates for feature isolation caused by extreme viewpoint differences and significantly improves the geometric consistency of multi-view depth estimation. This enables road end images to have higher reliability and generalization ability in applications such as intersection 3D reconstruction and traffic scene analysis.
[0017] 2. This invention unifies multi-view images into feature sequences, breaking the limitations of single-image processing. By segmenting the image into tokens, local information from different perspectives can coexist in the same sequence, laying the foundation for cross-view interaction. Furthermore, a learnable camera token is introduced for each perspective, enabling it to autonomously capture the geometric characteristics of that perspective during the attention process, achieving implicit modeling of camera parameters and reducing reliance on precise extrinsic parameter calibration. Simultaneously, a set of register tokens is added to absorb common features of the global scene and accommodate invalid regions such as background and occlusion, effectively separating noise. Finally, the image tokens, camera tokens, and register tokens from the four perspectives are concatenated to form a unified input, providing a complete and high-quality data structure for subsequent alternating attention processing.
[0018] 3. This invention achieves deep fusion and iterative refinement of local details and cross-viewpoint geometric relationships by alternately executing intra-frame and global self-attention. Intra-frame self-attention focuses on the interior of each viewpoint, preserving spatial details such as texture and edges, and avoiding information loss caused by premature globalization. Global self-attention breaks down viewpoint barriers, establishing dense associations among all tokens, and automatically searching for and matching corresponding regions from different viewpoints. Alternating between the two forms a gradual optimization process, enabling feature representation to maintain fine-grained perception. The resulting image features are rich in both local details and global geometric relationships, providing high-quality shared features for upper-branch camera parameter regression and lower-branch depth map regression, significantly enhancing geometric consistency across multiple tasks.
[0019] 4. This invention has extremely high robustness for complex long-tail scenarios. When faced with complex environments such as roadside reflections, tree occlusion, large areas of weak texture, and rain and snow, this invention designs a depth loss function based on uncertainty weighting. This enables the method to adaptively reduce gradient interference in noisy areas, avoid misleading training by erroneous supervision signals, and ensure reliability in all-weather environments.
[0020] 5. In a single forward propagation, this invention uses two parallel prediction heads to simultaneously regress a high-precision camera intrinsic and extrinsic parameter matrix and a depth map at the corresponding resolution, providing a direct and high-quality data foundation for subsequent 3D Gaussian reconstruction. Detailed Implementation
[0021] To better understand the above technical solutions, specific implementation methods will be described in detail below. It should also be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0022] This embodiment provides a scene depth estimation method based on multi-view images of the roadside, including the following steps: S1. Acquire road segment images from four perspectives and preprocess them simultaneously. Cut the road segment images from the four perspectives into multiple image blocks for feature extraction to obtain image tokens. Add a learnable camera token and a set of register tokens to each perspective. Concatenate the image tokens, camera tokens and register tokens from the four perspectives to obtain an image feature sequence.
[0023] In this embodiment of the invention, RGB images acquired from four fixed viewpoints along the roadside are first preprocessed. The input is set to a time-synchronized sequence of images from the four viewpoints. , Let i represent the viewpoint of the i-th camera. The original image is denoised to remove environmental noise interference such as rain and snow, and necessary distortion correction and size normalization are performed to ensure the geometric consistency of the input data.
[0024] Each image is divided into multiple non-overlapping image patches. A pre-trained self-supervised visual model, DINOv2 (Distillation with No Labels version 2), is used to extract tokens from each image patch. Assuming the image resolution is... The patch size is Then each image generates an image token. ,in K represents the number of tokens, and C represents the token feature dimension.
[0025] To give the model the ability to perceive camera geometry, a special geometric token is attached to the token sequence for each viewpoint. Specifically, a learnable camera token is added to each image. and a set of register tokens Finally, all tokens from the four perspectives are concatenated to form the input sequence, i.e., the image feature sequence.
[0026] S2. After performing intra-frame self-attention processing on the obtained image feature sequence, local spatial features are obtained. Then, global self-attention processing is performed on the local spatial features to obtain the first image feature. Intra-frame self-attention processing and global self-attention processing are performed on the first image feature in sequence to obtain the second image feature. This process is repeated for Q-1 iterations to obtain the Qth image feature. Then, the Q-th image features are split and the following branch decoding process is performed: The upper branch performs camera parameter regression head processing on the Q-th image features to obtain the camera intrinsic and extrinsic parameters corresponding to the Q-th image features; The next branch performs depth regression head processing on the Q-th image features to obtain the estimated depth map.
[0027] In this embodiment of the invention, an alternating attention mechanism is used in the layer encoder to balance local feature extraction and global geometric matching.
[0028] Intra-frame self-attention Attention is calculated only between tokens within the same image to obtain local spatial features.
[0029] Global self-attention jointly computes attention across all tokens in all four perspectives.
[0030] Furthermore, in this invention, the two attention operations described above are executed alternately for a total of 12 loops. Each loop includes one intra-frame and one global attention operation, for a total of 24 attention calculations. Based on the obtained image feature sequence, intra-frame self-attention processing is performed in the encoder to obtain local spatial features. Then, global self-attention processing is performed on the local spatial features to obtain the first image feature. The first image feature is then fed back into the intra-frame self-attention processing and used again as a local spatial feature. After global self-attention processing, the second image feature is obtained. The second image feature is then fed back into the intra-frame self-attention processing to obtain the third local spatial feature. After global self-attention processing, the third image feature is obtained, and so on, for a total of 12 loop operations, finally obtaining the thirteenth image feature. The two attention operations are detailed below: Specifically, assuming the first The input of the layer is This process can be represented as: in, Representation layer normalization; It is intra-frame self-attention; It is global self-attention; Representing intermediate hidden states, these are input features. go through , The result is obtained by performing the operation and adding the residual to the original input; It is the first The final output of the layer.
[0031] After processing by the layer encoder, the updated image token and camera token are output. At this point, the thirteenth image feature is fed into two parallel prediction heads for decoding: The camera parameter regression head, or the upper branch, utilizes the updated camera token. The rotation and translation matrices of each image are predicted using a multilayer perceptron (MLP).
[0032] The high-precision depth regression head, or downbranch, utilizes the updated image token. In this embodiment of the invention, a Dense Prediction Transformer is used to perform feature recombination and upsampling via a decoder, and finally, an estimated depth map consistent with the input resolution is regressed through convolutional layers. .
[0033] In this embodiment of the invention, the method is trained using uncertainty-based depth loss, camera pose loss, and occupancy consistency loss, based on training data, to output the final depth map.
[0034] By collecting data offline, the intersection was reconstructed into a 3D Gaussian model, and the image and depth of the roadside view were rendered in the model.
[0035] Specifically, to improve the robustness of the method in complex areas such as road surface reflections and tree occlusion, an uncertainty-based depth loss function was designed. The predicted depth value based on this invention... and true depth The depth loss function combines L1 error and gradient error, weighted by confidence, to optimize the model. The formula is as follows: in, Represents the depth loss function; To balance the regularization term; The confidence level is obtained through the DINO feature extraction model; To predict depth, i.e., to estimate depth; For true depth.
[0036] This loss function can automatically reduce the weights of uncertain regions in the model, such as the sky or the boundaries of dynamic objects, to avoid erroneous supervision signals interfering with training.
[0037] To ensure geometric consistency, camera pose loss is introduced. Calculate and predict pose With truth value Losses between: To align the depth of the four views, a differentiable projection layer is defined to project the depth maps and features of the four views onto the same BEV mesh. Given the first... Image pixel coordinates from each perspective and the predicted pixel depth The camera's intrinsic parameter matrix is The extrinsic parameters of the camera to the world coordinate system are The three-dimensional point of this pixel in the 3D world coordinate system. for: in, Represents the rotation matrix. The translation vector is represented by the translation vector, and together they determine the camera's position and orientation in world coordinates.
[0038] This 3D point cloud is discretized into an occupancy probability grid in a BEV space by voxelization.
[0039] Within the shared field of view of the four cameras, the depth predictions projected onto the BEV should perfectly overlap. To penalize misalignment, an occupation consistency loss is introduced. : set up For camera and camera The physically overlapping region projected onto the BEV space. Within this region, minimizing the difference in occupancy probabilities among the various cameras: Represents the two-dimensional coordinates of the i-th viewpoint in BEV space. The occupancy probability grid at the location, Represents the two-dimensional coordinates of the j-th viewpoint in BEV space. The occupancy probability grid at each location; N represents the total number of camera indices.
[0040] The final total loss function is After training, the method of this invention can directly output high-precision depth maps of the four perspectives in a single forward propagation based on the input road segment images from the four perspectives, which can be used for subsequent intersection 3D reconstruction or traffic scene analysis.
[0041] This embodiment provides a scene depth estimation system based on multi-view images from the roadside, including the following: The multi-view image feature sequence construction module acquires road segment images from four perspectives and performs preprocessing simultaneously. It then cuts the four road segment images into multiple image blocks for feature extraction to obtain image tokens. Learnable camera tokens and a set of register tokens are added to each perspective. Finally, the image tokens, camera tokens, and register tokens from the four perspectives are concatenated to obtain the image feature sequence. The alternating attention multi-task decoding training module performs intra-frame self-attention processing on the obtained image feature sequence to obtain local spatial features. Then, it performs global self-attention processing on the local spatial features to obtain the first image feature. The first image feature is then subjected to intra-frame self-attention processing and global self-attention processing in sequence to obtain the second image feature. This process continues until the Q-1th image feature is obtained. Then, the Q-th image features are split and the following branch decoding process is performed: The upper branch performs camera parameter regression head processing on the Q-th image features to obtain the camera intrinsic and extrinsic parameters corresponding to the Q-th image features; The next branch performs depth regression head processing on the Q-th image features to obtain the estimated depth map; Convergence is achieved using depth loss function, camera pose loss function, and occupancy consistency loss function.
Claims
1. A scene depth estimation method based on multi-view images of the roadside, characterized in that, Includes the following steps: Step S1. Acquire road segment images from four perspectives and preprocess them simultaneously. Cut the road segment images from the four perspectives into multiple image blocks for feature extraction to obtain image tokens. Add a learnable camera token and a set of register tokens to each perspective. Concatenate the image tokens, camera tokens and register tokens from the four perspectives to obtain an image feature sequence. Step S2. After performing intra-frame self-attention processing on the obtained image feature sequence, local spatial features are obtained. Then, global self-attention processing is performed on the local spatial features to obtain the first image feature. Intra-frame self-attention processing and global self-attention processing are performed on the first image feature in sequence to obtain the second image feature. This process is repeated for Q-1 iterations to obtain the Qth image feature. Then, the Q-th image features are split and the following branch decoding process is performed: The upper branch performs camera parameter regression head processing on the Q-th image features to obtain the camera intrinsic and extrinsic parameters corresponding to the Q-th image features; The next branch performs depth regression head processing on the Q-th image features to obtain the estimated depth map; Convergence is achieved using depth loss function, camera pose loss function, and occupancy consistency loss function.
2. The scene depth estimation method based on multi-view images at the road end according to claim 1, characterized in that, The intra-frame self-attention processing calculates attention only between tokens within the same image.
3. The scene depth estimation method based on multi-view images at the road end according to claim 1, characterized in that, The global self-attention processing jointly computes attention among all tokens in the four perspectives.
4. The scene depth estimation method based on multi-view images at the road end according to claim 1, characterized in that, The occupation consistency loss function The details are as follows: set up For camera and camera The physical overlap region projected onto the BEV space, within which... Represents the two-dimensional coordinates of the i-th viewpoint in BEV space. The occupancy probability grid at the location, Represents the two-dimensional coordinates of the j-th viewpoint in BEV space. The occupancy probability grid at each location, where N represents the total number of camera indices.
5. The scene depth estimation method based on multi-view images at the road end according to claim 1, characterized in that, The upper branch is as follows: Using the updated camera token, the rotation and translation matrices for each image are predicted via a multilayer perceptron.
6. The scene depth estimation method based on multi-view images at the road end according to claim 1, characterized in that, The lower branch is as follows: Using the updated image token, features are reconstructed and upsampled by the decoder, and finally, an estimated depth map consistent with the input resolution is regressed through a convolutional layer.
7. The scene depth estimation method based on multi-view images at the road end according to claim 1, characterized in that, The process of acquiring road segment images from four different perspectives and performing preprocessing simultaneously is as follows: Color images were acquired from four fixed viewpoints along the roadside, and the image sequence from the four viewpoints was set to be synchronized in time. , , where i represents the i-th camera viewpoint, and noise reduction processing is performed on the color image.
8. The scene depth estimation method based on multi-view images at the road end according to claim 1, characterized in that, The Q-1 iterations of the loop are as follows: When Q=13, the local spatial features are first obtained by performing intra-frame self-attention processing on the obtained image feature sequence. Then, the local spatial features are subjected to global self-attention processing to obtain the first image feature. The first image feature is then subjected to intra-frame self-attention processing and global self-attention processing in sequence to obtain the second image feature. This process is repeated 12 times to obtain the thirteenth image feature.
9. A scene depth estimation system based on multi-view images of the roadside, applied to the scene depth estimation method based on multi-view images of the roadside as described in claim 1, characterized in that, Includes the following: The multi-view image feature sequence construction module acquires road segment images from four perspectives and performs preprocessing simultaneously. It then cuts the four road segment images into multiple image blocks for feature extraction to obtain image tokens. Learnable camera tokens and a set of register tokens are added to each perspective. Finally, the image tokens, camera tokens, and register tokens from the four perspectives are concatenated to obtain the image feature sequence. The alternating attention multi-task decoding training module performs intra-frame self-attention processing on the obtained image feature sequence to obtain local spatial features. Then, it performs global self-attention processing on the local spatial features to obtain the first image feature. The first image feature is then subjected to intra-frame self-attention processing and global self-attention processing in sequence to obtain the second image feature. This process continues until the Q-1th image feature is obtained. Then, the Q-th image features are split and the following branch decoding process is performed: The upper branch performs camera parameter regression head processing on the Q-th image features to obtain the camera intrinsic and extrinsic parameters corresponding to the Q-th image features; The next branch performs depth regression head processing on the Q-th image features to obtain the estimated depth map; Convergence is achieved using depth loss function, camera pose loss function, and occupancy consistency loss function.
10. The scene depth estimation system based on multi-view images at the road end according to claim 9, characterized in that, The intra-frame self-attention processing module in the multi-view image feature sequence construction module only calculates attention between tokens within the same image.