A target ranging system based on a self-supervised monocular depth estimation method
By using a self-supervised monocular depth estimation method and a YOLO model, the accuracy and generalization problems of traditional monocular target ranging systems in complex scenarios are solved, achieving high-precision and stable target ranging results, which are suitable for indoor positioning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TONGJI UNIV
- Filing Date
- 2024-04-12
- Publication Date
- 2026-05-15
AI Technical Summary
Traditional monocular target ranging methods perform poorly in complex scenes or when feature points are lacking, and the calculation process based on geometric methods is sensitive to background interference. Supervised depth estimation models suffer from problems such as difficulty in obtaining labeled data, high cost, and poor generalization.
We employ a self-supervised monocular depth estimation method, which combines a self-supervised depth estimation model and a pre-trained YOLO model with a depth prediction network and a pose estimation network to generate depth maps for dense scenes and track targets. We also utilize a semantic segmentation model to reduce the complexity of foreground segmentation, quickly generate target masks, and calculate real-time target ranging.
It significantly reduces the need for scene annotation, enhances the system's generalization ability, achieves meter-level indoor positioning accuracy, has better ranging accuracy than UWB devices, and has high stability, which can meet the needs of practical applications.
Smart Images
Figure CN118172399B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, specifically to a target ranging system based on a self-supervised monocular depth estimation method. Background Technology
[0002] Traditional monocular target ranging methods mainly rely on geometric principles and camera calibration parameters. For example, Chinese invention patent application 202210810678.0 provides a monocular ranging system for downhole unmanned vehicles based on a geometric ranging model and Kalman filtering method. This type of method requires precise camera calibration and performs poorly in complex scenes or when feature points are lacking. In addition, traditional geometric methods obtain scene depth information based on complex calculation processes, making them very sensitive to background interference.
[0003] With the continuous development of technologies such as machine learning, the vision-based target ranging problem has gradually transformed into a scene depth estimation problem. Dense and accurate depth information can accurately describe the distance of objects in a scene from the shooting position, thereby achieving accurate target ranging. Depth information can be obtained through hardware devices such as LiDAR or through cue-based or machine learning-based depth estimation methods. Depth information obtained through hardware is mostly sparse, and the hardware has demanding operating conditions and high costs. Traditional supervised depth estimation methods based on machine learning mostly suffer from many problems such as the difficulty and high cost of obtaining labeled data, and poor model generalization. Therefore, target ranging systems based on this type of depth estimation model have demanding environmental requirements and poor robustness. Summary of the Invention
[0004] Purpose of the invention
[0005] Existing technologies reveal that traditional visual ranging systems based on geometric methods or supervised depth estimation models suffer from numerous problems in acquiring target distances, such as background interference and poor generalization. To overcome these issues, this invention innovatively constructs a target ranging system based on a monocular self-supervised depth estimation method.
[0006] Technical solution of the present invention:
[0007] A target ranging system based on a self-supervised monocular depth estimation method includes: a system input module, a depth estimation module, a target detection module, a target tracking module, a data processing module, a foreground segmentation module, and a system output module. The system's working mechanism is as follows:
[0008] The monocular images or videos captured by the camera equipment are first input into the depth estimation module and the target detection module;
[0009] The depth estimation module uses a pre-trained self-supervised depth estimation model to map monocular images into dense scene depth maps of the same size;
[0010] The object detection module uses a pre-trained YOLO model to output a list of detection boxes containing the detected object category, object location, and confidence level;
[0011] The target tracking module performs target ID matching based on the output of the target detection module, thereby achieving target tracking;
[0012] The data processing module decomposes the monocular RGB image and the scene depth map obtained by the depth estimation module into several RGB image and depth map sub-block pairs based on the target detection results, thereby reducing the complexity of the input data of the foreground segmentation module and accelerating the semantic segmentation process.
[0013] The foreground segmentation module receives the output of the target detection module and quickly generates the target mask corresponding to each RGB sub-block; the foreground segmentation module uses the generated mask to separate the target depth information in the scene depth map and calculate the real-time target ranging result;
[0014] The system output module labels the tracking results and target distance on a monocular RGB image, and then horizontally stitches the RGB image with the scene depth pseudo-color map to construct the output image.
[0015] Beneficial effects
[0016] Compared with traditional target ranging methods, this invention significantly reduces the need for scene annotation, greatly enhances the generalization ability, and can meet the requirements of meter-level indoor positioning in terms of ranging accuracy. Attached Figure Description
[0017] Figure 1 This is a structural diagram of the monocular ranging system of the present invention;
[0018] Figure 2 This is a diagram of the depth prediction network structure in the depth estimation module of this invention.
[0019] Figure 3 This is a diagram of the pose estimation network structure in the depth estimation module of this invention.
[0020] Figure 4 This is a schematic diagram of adjacent frame pixel mapping in the depth estimation module of this embodiment of the invention;
[0021] Figure 5 This is a diagram illustrating the system's operational effect in an embodiment of the present invention.
[0022] Figure 6 This is a comparison diagram of the ranging accuracy of UWB devices in the embodiments of the present invention;
[0023] Figure 7 This is a comparison chart of the accuracy of the embodiments of the present invention and the UWB solution;
[0024] Figure 8 This is a map of the stability test field in an embodiment of the present invention;
[0025] Figure 9 The figure shows the system stability test results of an embodiment of the present invention. Detailed Implementation
[0026] The technical solutions provided in this application will be further described below with reference to specific embodiments and accompanying drawings. The advantages and features of this application will become clearer from the following description.
[0027] A target ranging system based on a self-supervised monocular depth estimation method includes: a system input module, a depth estimation module, a target detection module, a target tracking module, a data processing module, a foreground segmentation module, and a system output module. The system structure and the working mechanisms between the various functional modules are as follows: Figure 1 As shown.
[0028] The system works as follows:
[0029] Monocular images or videos captured by camera equipment are first input into the depth estimation module and the object detection module. The depth estimation module uses a pre-trained self-supervised depth estimation model to map the monocular image into a dense scene depth map of the same size.
[0030] The object detection module uses a pre-trained YOLO model to output a list of detection boxes containing the detected object category, object location, and confidence level.
[0031] The target tracking module performs target ID matching based on the output of the target detection module, thereby achieving target tracking.
[0032] The data processing module decomposes the monocular RGB image and scene depth map into several RGB image and depth map sub-block pairs based on the target detection results, thereby reducing the complexity of the input data to the foreground segmentation module and accelerating the semantic segmentation process.
[0033] The foreground segmentation module receives the output from the target detection module and quickly generates a target mask corresponding to each RGB sub-block. The foreground segmentation module then uses the generated mask to separate the target depth information from the scene depth map and calculates the real-time target ranging result.
[0034] The system output module labels the tracking results and target distance on a monocular RGB image, and then horizontally stitches the RGB image with the scene depth pseudo-color map to construct the output image.
[0035] The specific details of each module are as follows:
[0036] First, the system input module decomposes the input video into a monocular RGB image sequence and inputs it frame by frame into the monocular target ranging system.
[0037] II. The depth estimation module is implemented using a self-supervised monocular depth estimation model, which consists of two parts: a depth prediction network and a pose estimation network. During the training of the self-supervised monocular depth estimation model, the outputs of the depth prediction network and the pose estimation network jointly complete the construction of the network supervision signal, guiding the training of the model. In the actual use of the self-supervised monocular depth estimation model, only the depth prediction network is needed to complete the mapping from monocular RGB images to scene depth maps.
[0038] The depth prediction network takes a monocular RGB image as input and outputs a single-channel scene depth map. The network employs an encoder-decoder architecture. Figure 2 As shown.
[0039] The network encoder is built on ResNet-18, where:
[0040] Input represents the input monocular three-channel RGB image;
[0041] The Conv layer represents an image preprocessing layer consisting of a convolutional layer with a kernel size of 7×7, a stride of 2, and padding of 3, and a BatchNormalization (BN) layer.
[0042] MaxPooling represents a max pooling layer with a pooling window of 3×3 and a stride of 2.
[0043] Conv_Block represents a downsampling layer consisting of a convolutional layer with a kernel size of 3×3, a stride of 2, and padding of 1, and a BN layer.
[0044] The encoder output is a list consisting of five feature maps at different levels.
[0045] The decoder utilizes multi-scale fusion to perceive feature information across multiple levels. The image upsampling process is implemented using 3×3 padding convolutions and bilinear interpolation. Figure 2 This is represented as Deconv. To fuse feature maps from different levels, the model uses a skip-connection method to fuse feature maps of the same size as the upsampled result from the decoder, using these as input to the next layer of the decoder. This process... Figure 2 This is represented as Concat. The decoder output is a 64-channel depth map of the same size as the input image, and the last channel is taken as the model's predicted depth, i.e., Output in the figure.
[0046] The pose estimation network takes two adjacent frames from the training dataset as input and outputs the pose vector between those two adjacent frames. The pose estimation network structure is as follows: Figure 3 As shown, the feature extraction part of the pose network is the same as the encoder of the depth prediction network. The high-dimensional feature map output by the feature extraction part is then convolved to finally output 6-dimensional data containing 3D rotation vectors and 3D translation vectors.
[0047] The self-supervised monocular depth estimation model used in this invention constructs a supervision signal using the image reconstruction process between adjacent frames. The image reconstruction process between adjacent frames refers to mapping the original pixels of the reference frame to the projected pixels of the target frame using depth information and pose information between adjacent frames, assuming that the adjacent frames are static scenes. After obtaining the coordinates of the projected pixels, the gray values of the projected pixels are obtained using bilinear interpolation. The reconstruction of the reference frame can be completed using the gray values of the projected pixels.
[0048] The pixel mapping process described above, which utilizes depth information and pose information between adjacent frames to realize the original pixels of the reference frame to the projected pixels of the target frame, is as follows: Figure 4 As shown, where p n For frame I n A pixel on, p n in I n The pixel coordinates in (x) are r ,y r ), p n-1 For p n In frame I n-1 The mapping method for the projection points on the surface consists of five steps:
[0049] (1) Based on the predicted depth value, the reference frame pixel point p n Two-dimensional coordinates (x) r ,y r Projected onto 3D coordinates based on the reference frame;
[0050] (2) Synthesize the three-dimensional rotation matrix and the three-dimensional translation matrix by utilizing the relative attitude between the reference frame and the target frame;
[0051] (3) Project the three-dimensional coordinates based on the reference frame onto the three-dimensional coordinate system based on the target frame according to the rotation matrix and the translation matrix;
[0052] (4) Project the predicted depth values onto the two-dimensional coordinates of the target frame to obtain frame I. n-1 p n The projection point p n-1 ;
[0053] (5) Using bilinear interpolation, the four pixels closest to the mapping point are used. The grayscale values of the mapping points are calculated to realize the reconstruction of the reference frame.
[0054] The loss function in model training consists of the photometric loss L between the reconstructed image and the reference image. ssim Geometric consistency loss I G And smoothing loss I S It consists of three weighted parts.
[0055] L ssim The photometric error primarily characterizes the accuracy of image reconstruction between adjacent frames, obtained by calculating the photometric error between the reconstructed image and the reference image. However, dynamic objects in the dataset and the lighting conditions during dataset acquisition can severely impact the photometric loss. This invention improves the reliability of the photometric error by generating a static mask. Specifically, a semantic segmentation model is first used to filter out vehicles and pedestrians in the dataset to generate a static background mask. An effective pixel set V is then constructed based on the pixels in the background mask that have successfully mapped to adjacent frames. The weighted loss function during training is calculated based on the effective pixel set. Since dynamic points have a greater photometric loss than static points, the model uses the L1 loss method, which is robust to outliers, to calculate the photometric loss L of set V. ssim .
[0056] L G The main characteristic is the consistency of scene depth generated across different frames. The specific calculation method involves using a reference frame (I... a ) and target frame (I b Relative attitudes (P) between ab ) and the predicted depth map of the reference frame (D a Synthesized target frame depth map Based on the synthesized depth map of the target frame With predicted depth map (D b The depth difference between the two values is used to construct the geometric consistency loss L. G The actual calculation of geometric consistency loss is shown in Equation 1, where p represents any pixel in the set of valid points V.
[0057]
[0058] For smoothness loss L S The model adopts the method in the reference paper [1], using the first derivative of the depth value along the spatial direction as a smoothing loss. This method has been shown to constrain the smoothness of image edge guidance.
[0059] The loss function L used in model training is shown in Equation 2, where α, β, and γ are weight parameters set before training:
[0060] L=αL ssim +βL S+γL G (2)
[0061] III. The object detection module is implemented using the YOLO object detection model. YOLO (You Only Look Once) is a deep learning model for real-time object detection. Its main working principle is as follows: First, the input RGB image is divided into a fixed-size grid. Each grid cell is responsible for detecting objects in the image and predicting multiple candidate boxes. Each candidate box is defined by a set of parameters, including the object's location (center coordinates x and y), size (width and height), and a confidence score indicating whether the box contains the object. The model then uses a softmax layer to predict the probability score for each category to determine the object's category. Finally, the model filters out candidate boxes with low confidence and further uses post-processing methods such as Non-Maximum Suppression (NMS) to retain the most likely target boxes. The model's output is the detected object category and its bounding box.
[0062] The object detection module takes a monocular RGB image as input and outputs a list of detected objects in the image, including their categories and bounding boxes, as determined by the YOLO model. This output forms the basis for the object tracking algorithm and also enables the segmentation of small RGB sub-blocks containing objects from a large-scale scene RGB image, thereby reducing the inference time of the foreground segmentation module and improving system speed.
[0063] IV. The target tracking module takes the target detection results of the current frame and the target detection results of the previous frame as input. This module calculates the degree of overlap between the detection boxes of the current frame and the detection boxes of the previous frame to achieve target matching in consecutive frames and thus complete target tracking.
[0064] Fifth, the data processing module decomposes the monocular RGB image and scene depth map into several RGB image and depth map sub-block pairs based on the target detection results, thereby reducing the complexity of the input data of the foreground segmentation module and accelerating the semantic segmentation process.
[0065] VI. The foreground segmentation module is implemented using a DeeplabV3 pre-trained model. The module input consists of several RGB image and depth map sub-block pairs segmented from the target detection results. The module output is the foreground mask of the target sub-block, which is generated based on the RGB image sub-blocks and can separate the target depth information in the corresponding depth map sub-blocks, thereby accurately calculating the target distance.
[0066] 7. The system output module labels the tracking results and target distance on the monocular RGB image, and horizontally stitches the RGB image with the scene depth pseudo-color map to construct the output image.
[0067] Verification of the implementation effect of the technical solution of this invention:
[0068] Figure 5 The image shows the real-time performance of the system. The left side is a monocular RGB image, with the detected target highlighted in red. The top left corner of the border indicates the target category, while the top right corner indicates the real-time calculated target distance and the target ID used for tracking. The right side is a scene density depth map output in real-time by the depth estimation module. In this pseudo-color map, each pixel's RGB value corresponds to the depth value D. i It is obtained by multiplying by a coefficient α (α is 15) and then using the JET mapping rule.
[0069] D′ i =α*D i (3)
[0070] This invention employs a centimeter-level ultra-wideband (UWB) ranging device as a high-precision comparison scheme. UWB is based on the transmission and reception of short-pulse signals, calculating distance by measuring the signal propagation time. The UWB device uses a two-way ranging method to collect distance data between the base station and the tag. The measured accuracy of the UWB device is as follows: Figure 6 As shown. Within the range of 3 to 8 meters, the UWB ranging scheme can achieve an average ranging accuracy of less than 8 cm, which can be used as a high-precision comparison scheme for this invention.
[0071] Figure 7 The results show the comparison of target ranging accuracy of the present invention with that of a high-precision UWB scheme in a real-world scenario. The experimental results show that, in actual testing, the average absolute ranging error of the present invention is less than 0.36m compared with the UWB scheme. It is a high-precision ranging scheme that can support the realization of a high-precision positioning system.
[0072] The present invention further evaluates the stability and accuracy of the invention in practical applications by performing target ranging on pedestrians walking back and forth on a fixed trajectory. Figure 8 The stability test site is described, with the trajectory starting at (0, 3.75) and turning point at (0, 12.2). Pedestrians walk back and forth on the specified path, and the system performs target distance measurement on the pedestrians at the shooting point (0, 0). Figure 9 The results of stability tests were presented. During the periodic back-and-forth movement of pedestrians, the distance measurement results of the present invention can cover or approach the actual minimum and maximum distances in each walking cycle. These test results indicate that the present invention has strong stability and higher practical application value.
[0073] The above description is merely a description of preferred embodiments of this application and is not intended to limit the scope of this application in any way. Any changes or modifications made by those skilled in the art based on the above-disclosed technical content should be considered as equivalent and valid embodiments and fall within the scope of protection of the technical solution of this application.
[0074] Reference papers:
[0075] [1]Ranjan A,Jampani V,Balles L,et al.Competitive collaboration:Jointunsupervised learning of depth,camera motion,optical flow and motionsegmentation[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2019:12240-12249.
Claims
1. A target ranging system based on a self-supervised monocular depth estimation method, characterized in that, include: The system comprises an input module, a depth estimation module, an object detection module, an object tracking module, a data processing module, a foreground segmentation module, and a system output module; the system's working mechanism is as follows: The monocular images or videos captured by the camera equipment are first input into the depth estimation module and the target detection module; The depth estimation module uses a pre-trained self-supervised depth estimation model to map monocular images into dense scene depth maps of the same size; The object detection module uses a pre-trained YOLO model to output a list of detection boxes containing the detected object category, object location, and confidence level; The target tracking module performs target ID matching based on the output of the target detection module, thereby achieving target tracking; The data processing module decomposes the monocular RGB image and the scene depth map obtained by the depth estimation module into several RGB image and depth map sub-block pairs based on the target detection results, thereby reducing the complexity of the input data of the foreground segmentation module and accelerating the semantic segmentation process. The foreground segmentation module receives the output of the target detection module and quickly generates the target mask corresponding to each RGB sub-block. The foreground segmentation module uses the generated mask to separate the target depth information in the scene depth map and calculate the real-time target ranging result; The system output module labels the tracking results and target distance on a monocular RGB image, and then horizontally stitches the RGB image with the scene depth pseudo-color map to construct the output image.
2. The target ranging system based on a self-supervised monocular depth estimation method as described in claim 1, characterized in that, The depth estimation module is implemented using a self-supervised monocular depth estimation model, which includes a depth prediction network and a pose estimation network. During the training of the self-supervised monocular depth estimation model, the outputs of the depth prediction network and the pose estimation network work together to construct the network supervision signal and guide the training of the model. The depth prediction network takes a monocular RGB image as input and outputs a single-channel scene depth map. The network adopts an encoder-decoder architecture. The input to the pose estimation network is two adjacent frames in the training dataset, and the output of the pose estimation network is the pose vector between the two adjacent frames. The feature extraction part of the pose network is the same as the encoder of the deep prediction network. The high-dimensional feature map output by the feature extraction part is then convolved to finally output 6-dimensional data containing 3D rotation vectors and 3D translation vectors. The self-supervised monocular depth estimation model constructs a supervision signal using the image reconstruction process between adjacent frames. The image reconstruction process between adjacent frames, assuming both frames are static scenes, involves mapping the original pixels of the reference frame to the projected pixels of the target frame using depth information and pose information between adjacent frames. After obtaining the coordinates of the projected pixels, bilinear interpolation is used to obtain the grayscale values of the projected pixels. The reconstruction of the reference frame can then be completed using these grayscale values. The mapping process is as follows: p n For frame I n A pixel on, p n in I n The pixel coordinates in (x) are r ,y r ), p n-1 For p n In frame I n-1 The mapping method for the projection points on the surface consists of five steps: (1) Based on the predicted depth value, the reference frame pixel point p n Two-dimensional coordinates (x) r ,y r Projected onto 3D coordinates based on the reference frame; (2) Synthesize the three-dimensional rotation matrix and the three-dimensional translation matrix by utilizing the relative attitude between the reference frame and the target frame; (3) Project the three-dimensional coordinates based on the reference frame onto the three-dimensional coordinate system based on the target frame according to the rotation matrix and the translation matrix; (4) Project the predicted depth values onto the two-dimensional coordinates of the target frame to obtain frame I. n-1 p n The projection point p n-1 ; (5) Using bilinear interpolation, the four pixels closest to the mapping point are used. The grayscale values of the mapping points are calculated to realize the reconstruction of the reference frame.
3. The target ranging system based on a self-supervised monocular depth estimation method as described in claim 2, characterized in that, The encoder of the deep prediction network is built based on ResNet-18, wherein: Input is a monocular three-channel RGB image. The Conv layer is an image preprocessing layer consisting of a convolutional layer with a kernel size of 7×7, a stride of 2, and padding of 3, and a Batch Normalization (BN) layer. MaxPooling is a max pooling layer with a pooling window of 3×3 and a stride of 2. Conv_Block is a downsampling layer consisting of a convolutional layer with a kernel size of 3×3, a stride of 2, and padding of 1, and a BN layer. The encoder output is a list consisting of five feature maps at different levels; The decoder uses multi-scale fusion to perceive feature information at multiple levels; the image upsampling process is implemented using 3×3 padding convolution and bilinear interpolation, denoted as Deconv; in order to fuse feature maps from different levels, the model uses the skip-connection method to fuse feature maps from the encoder that are the same size as the upsampling result of the decoder as the input of the next layer of the decoder, denoted as Concat; the output of the decoder is a 64-channel depth map with the same size as the input image, and the last channel is taken as the predicted depth of the model, i.e., Output in the figure.
4. The target ranging system based on a self-supervised monocular depth estimation method as described in claim 2, characterized in that, The loss function in the training of the self-supervised monocular depth estimation model consists of the photometric loss L between the reconstructed image and the reference image. ssim Geometric consistency loss L G and smoothing loss L S It consists of three weighted parts; The loss function L used in model training is shown in Equation 2, where α, β, and γ are weight parameters set before training: L=αL ssim +βL S +γL G (2)。 5. The target ranging system based on a self-supervised monocular depth estimation method as described in claim 4, characterized in that, The L ssim The accuracy of image reconstruction between adjacent frames is characterized by calculating the photometric error between the reconstructed image and the reference image. A static mask generation method is used to improve the reliability of the photometric error. Specifically, a semantic segmentation model is first used to filter out vehicles and pedestrians in the dataset to generate a static background mask. An effective pixel set V is constructed based on the pixels in the background mask that have successfully mapped to adjacent frames. The weighted loss function during training is calculated based on this effective pixel set. Since dynamic points have a greater photometric loss than static points, the model uses the L1 loss method, which is robust to outliers, to calculate the photometric loss L of set V. ssim ; The L G The consistency of scene depth generated across different frames is characterized by using a reference frame (I). a ) and target frame (I b Relative attitudes (P) between ab ) and the predicted depth map of the reference frame (D a Synthesized target frame depth map Based on the synthesized depth map of the target frame With predicted depth map (D b The depth difference between the two values is used to construct the geometric consistency loss L. G The actual geometric consistency loss is calculated as shown in Equation 1, where p represents any pixel in the set of valid points V.