A semantic feature guided scene depth estimation method for fisheye camera
By designing a depth estimation method guided by semantic features, utilizing semantic information perception and camera pose information, and optimizing network parameters, the distortion error problem in fisheye camera depth estimation is solved, achieving high-precision depth estimation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHUHAI BOJAY ELECTRONICS
- Filing Date
- 2023-06-25
- Publication Date
- 2026-07-21
AI Technical Summary
Existing deep learning methods, when using fisheye cameras for depth estimation, are affected by fisheye image distortion, resulting in large errors in predicted depth values and a lack of depth sample information, making it difficult to achieve accurate depth estimation.
We design a semantic feature-guided scene depth estimation method. By combining a semantic information perception module, a scene depth perception module, and a camera pose perception module with a loss function to train the model, optimize the network parameters, and use semantic information to guide depth feature perception, we can solve the fisheye image distortion problem.
It improves the accuracy of depth estimation and perception results from fisheye cameras, solves the problem of uneven depth caused by fisheye image distortion, and meets the requirements of scene depth estimation.
Smart Images
Figure CN116862964B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of scene perception, and in particular to a semantic feature-guided scene depth estimation method for fisheye cameras. Background Technology
[0002] Scene depth estimation is an important task in the field of vehicle visual perception. By estimating the distance from each pixel in an image to the camera, it provides depth information for 3D scene reconstruction and distance perception. Compared with depth measurement devices (depth cameras, LiDAR), image-based depth estimation has a wider range of applications.
[0003] With the rapid development of artificial intelligence technology, deep learning, with its powerful self-learning capabilities, has provided new ideas for scene depth estimation of vehicles. Current deep learning methods mostly focus on using images from traditional cameras and labeled scene depth map samples to complete the depth estimation task. However, many vehicle cameras often use fisheye cameras with a larger field of view. Fisheye image distortion can easily lead to large errors in predicted depth values, and fisheye image datasets contain relatively little depth sample information, making it difficult to support direct supervised depth estimation methods for fisheye images. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a semantic feature-guided scene depth estimation method for fisheye cameras that has a wide observation range, high prediction depth accuracy, accurate perception results and can meet the requirements of scene depth estimation.
[0005] The technical solution adopted in this invention is: a semantic feature-guided scene depth estimation method for fisheye cameras, the method of which is as follows: First, a semantic information perception module is designed to extract semantic information from the input distorted fisheye image; then, a scene depth perception module is established to guide the perception of depth features using semantic information; next, a camera pose perception module is established to obtain the camera pose information; finally, the image is reconstructed using scene depth map information and camera pose information, a loss function is designed to train the model, and the network parameters are optimized. Specifically, it includes the following steps:
[0006] Step 1: Design the semantic information perception module
[0007] Sub-step 1.1: Design the semantic encoder
[0008] The semantic encoder consists of a two-dimensional convolutional layer, three residual modules, and a feature fusion module. Each residual module contains a two-dimensional convolutional layer. The convolutional layer processes the input image to obtain feature map F1; residual module 1 processes feature map F1 to obtain feature map F2; residual module 2 uses an expanded convolutional layer to increase the receptive field of the image and processes feature map F2 to obtain feature map F3; residual module 3 adds two additional channels to the convolutional layer to learn pixel coordinate features and camera parameter features, extracting image distortion features and processing feature map F3 to obtain feature map F4; finally, feature maps F2, F3, and F4 are input together into the feature fusion module to obtain feature map F5.
[0009] Sub-step 1.2: Design a semantic decoder
[0010] The semantic decoder consists of three deconvolutional layers, three feature fusion modules, and one convolutional layer. Feature map F5 is input to the deconvolutional layer, upsampled to restore feature map F4' to the size of feature map F4, and then F4 and F4' are fused to generate feature map F4”. Next, F4” is input, upsampled to restore feature map F3' to the size of feature map F3, and then F3 and F3' are fused to generate feature map F3”. Then, F3” is input, upsampled to restore feature map F2' to the size of feature map F2, and then F2 and F2' are fused to generate feature map F2”. Finally, a convolutional layer is used to generate the semantic segmentation information map of the scene.
[0011] Step 2: Design the scene depth perception module
[0012] Sub-step 2.1: Design the depth encoder
[0013] The deep encoder uses the same backbone network as the semantic encoder, and the input image is processed by the encoder to obtain the feature map F5.
[0014] Sub-step 2.2: Design a semantically guided deep decoder
[0015] The deep decoder consists of four deconvolutional layers, three feature fusion modules, and three convolutional layers for semantic feature mapping. The formula for semantic feature mapping is shown in equation (1):
[0016] (1);
[0017] Where f represents semantic features, K is the convolution kernel, W is the convolution weight, v represents the convolutional input information, and p is the pixel coordinate.
[0018] First, feature map F5 is input into a deconvolutional layer and upsampled to restore feature map D4 to the size of feature map F4. F4 and D4 are then fused to generate feature map D4'. Simultaneously, the semantic feature map F4” obtained from the semantic decoder is input into a convolutional layer for semantic feature mapping, and then fused with D4' to generate depth map D4”. Next, D4” is used as input and upsampled to restore feature map D3 to the size of feature map F3. F3 and D3 are then fused to generate feature map D3'. Simultaneously, the semantic feature map F3” obtained from the semantic decoder is input into a convolutional layer for semantic feature mapping, and then fused with D4' to generate depth map D3”. Next, D3” is used as input and upsampled to restore feature map D2 to the size of feature map F2. F2 and D2 are then fused to generate feature map D2'. Simultaneously, the semantic feature map F2” obtained from the semantic decoder is input into a convolutional layer for semantic feature mapping, and then fused with D2' to generate depth map D2”. Finally, the result is input into a convolutional layer to obtain the predicted depth map of the scene.
[0019] Step 3: Design the model's loss function and train the model.
[0020] The loss function of the depth estimation network in this scene consists of two parts: the pixel cross-entropy loss for semantic segmentation as shown in Equation (2) and the depth smoothing loss as shown in Equation (3).
[0021] (2);
[0022] in, It is the predicted probability that a pixel belongs to category s. It is the true probability that a pixel belongs to category s;
[0023] (3);
[0024] in and These represent the vertical and horizontal gradients, respectively. The depth smoothing loss ensures that the depth changes are large at locations such as object outlines in the scene depth map, while the scene depth image is as smooth as possible at other locations, thus promoting the smoothness of the scene depth map.
[0025] In summary, the complete loss of the designed model is shown in equation (4), where It controls the weight of the semantic segmentation module's loss in the total loss. The size is 0.3; The weighting factor controls the weights of the deep smoothing loss. Set it to 0.1.
[0026] (4);
[0027] The WoodScape fisheye image dataset and its semantic labels were used as the training set. The initial learning rate was set to 0.0002, which automatically decreased during training. The parameters β1 and β2 of the Adam iterator were set to 0.9 and 0.999, respectively, and the batch size was set to 8. The aforementioned loss function was used for feedback to train the model. After each epoch, the training loss and validation loss were calculated, and the validation loss for each epoch was compared. The model parameters with the minimum validation loss were saved.
[0028] The beneficial effects of this invention are: This invention uses a semantic information-guided depth estimation method, which can solve the problem that the depth estimation of the same object is not smooth due to large image distortion when fisheye cameras perform depth estimation, and can effectively improve the accuracy of depth estimation when using fisheye cameras. Attached Figure Description
[0029] Figure 1 This is a schematic diagram of the model structure;
[0030] Figure 2 This is a network structure diagram of the semantic information perception module and the depth perception module. Detailed Implementation
[0031] Currently, scene depth estimation is a crucial task in the field of vehicle visual perception. By estimating the distance from each pixel in an image to the camera, it provides depth information for 3D scene reconstruction and distance perception. Compared to depth measurement devices (depth cameras, LiDAR), image-based depth estimation has a wider range of applications. With the rapid development of artificial intelligence technology, deep learning, with its powerful self-learning capabilities, offers new approaches to scene depth estimation for vehicles.
[0032] Current deep learning methods mostly focus on using images from traditional cameras and labeled scene depth map samples to perform depth estimation tasks. However, many vehicle cameras often use fisheye cameras with a larger field of view. But the distortion of fisheye images can easily lead to large errors in the predicted depth values, and fisheye image datasets contain relatively little depth sample information, making it difficult to support supervised depth estimation methods directly using fisheye images.
[0033] This invention discloses a semantically guided scene depth estimation method for fisheye cameras. The method employs an end-to-end neural network structure, utilizing the perception results of semantic segmentation to guide scene depth estimation of fisheye images. This method uses a fisheye camera, which has a wide observation range; the semantically guided depth estimation method employed features high prediction accuracy and accurate perception results, thus meeting the requirements for scene depth estimation.
[0034] This invention designs a scene depth estimation method based on semantic guidance for fisheye cameras. This method uses fisheye cameras, which have the advantage of a wide observation range. The semantic information-guided depth estimation method has the advantages of high prediction accuracy and accurate perception results.
[0035] The technical solution adopted in this invention is as follows: A semantically guided fisheye camera scene depth estimation method, characterized by: firstly, designing a semantic information perception module to extract semantic information from the input distorted fisheye image; then establishing a scene depth perception module to guide the perception of depth features using semantic information; next, establishing a camera pose perception module to obtain the camera's pose information; finally, reconstructing the image using scene depth map information and camera pose information, designing a loss function to train the model, and optimizing the network parameters. The overall model architecture is as follows: Figure 1 As shown, the specific steps include:
[0036] Step 1: Design the semantic information perception module
[0037] Sub-step 1.1: Design the semantic encoder
[0038] The semantic encoder consists of a two-dimensional convolutional layer, three residual modules, and a feature fusion module. Each residual module contains a two-dimensional convolutional layer. The convolutional layer processes the input image to obtain feature map F1; residual module 1 processes feature map F1 to obtain feature map F2; residual module 2 uses an expanded convolutional layer to increase the receptive field of the image and processes feature map F2 to obtain feature map F3; residual module 3 adds two additional channels to the convolutional layer to learn pixel coordinate features and camera parameter features, extracting image distortion features and processing feature map F3 to obtain feature map F4; finally, feature maps F2, F3, and F4 are input together into the feature fusion module to obtain feature map F5.
[0039] Sub-step 1.2: Design a semantic decoder
[0040] The semantic decoder consists of four deconvolutional layers, three feature fusion modules, and one convolutional layer. Feature map F5 is input to the deconvolutional layer, upsampled to restore feature map F4' to the size of feature map F4, and then F4 and F4' are fused to generate feature map F4”. Next, F4” is used as input, upsampled to restore feature map F3' to the size of feature map F3, and then F3 and F3' are fused to generate feature map F3”. Then, F3” is used as input, upsampled to restore feature map F2' to the size of feature map F2, and then F2 and F2' are fused to generate feature map F2”. Finally, a convolutional layer is used to generate the semantic segmentation information map of the scene.
[0041] Step 2: Design the scene depth perception module
[0042] Sub-step 2.1: Design the depth decoder
[0043] The deep encoder uses the same backbone network as the semantic encoder, and the input image is processed by the encoder to obtain the feature map F5.
[0044] Sub-step 2.2: Design a semantically guided deep decoder
[0045] The deep decoder consists of four deconvolutional layers, three feature fusion modules, and three convolutional layers for semantic feature mapping. The semantic feature mapping formula is shown in Equation (1). Here, f represents the semantic feature, K is the convolutional kernel, and W is the convolutional weight.
[0046] v represents the input information for convolution, and p is the pixel coordinate.
[0047] (1);
[0048] First, feature map F5 is input into a deconvolutional layer and upsampled to restore feature map D4 to the size of feature map F4. F4 and D4 are then fused to generate feature map D4'. Simultaneously, the semantic feature map F4” obtained from the semantic decoder is input into a convolutional layer for semantic feature mapping, and then fused with D4' to generate depth map D4”. Next, D4” is used as input and upsampled to restore feature map D3 to the size of feature map F3. F3 and D3 are then fused to generate feature map D3'. Simultaneously, the semantic feature map F3” obtained from the semantic decoder is input into a convolutional layer for semantic feature mapping, and then fused with D4' to generate depth map D3”. Then, D3” is used as input and upsampled to restore feature map D2 to the size of feature map F2. F2 and D2 are then fused to generate feature map D2'. Simultaneously, the semantic feature map F2” obtained from the semantic decoder is input into a convolutional layer for semantic feature mapping, and then fused with D2' to generate depth map D2”. This yields the predicted depth map of the scene. The structure diagram of the scene depth perception module is shown below. Figure 2 As shown.
[0049] Step 3: Design the model's loss function and train the model.
[0050] The loss function of the depth estimation network in this scene consists of two parts: the pixel cross-entropy loss for semantic segmentation. As shown in equation (2) and the depth smoothing loss As shown in equation (3).
[0051] (2);
[0052] in, It is the predicted probability that a pixel belongs to category s. It is the true probability that a pixel belongs to category s.
[0053] (3);
[0054] in and These represent the vertical and horizontal gradients, respectively. The depth smoothing loss ensures that the depth changes are large at locations such as object outlines in the scene depth map, while the scene depth image is as smooth as possible at other locations, thus promoting the smoothness of the scene depth map.
[0055] In summary, the complete loss of the designed model is shown in equation (4), where It controls the weight of the semantic segmentation module's loss in the total loss. The size is 0.3. The weighting factor controls the weights of the deep smoothing loss. Set it to 0.1.
[0056] (4);
[0057] The WoodScape fisheye image dataset and its semantic labels were used as the training set. The initial learning rate was set to 0.0002, which automatically decreased during training. The parameters β1 and β2 of the Adam iterator were set to 0.9 and 0.999, respectively, with a batch size of 8. The aforementioned loss function was used for feedback to train the model. After each epoch, the training loss and validation loss were calculated, and the validation loss for each epoch was compared. The model parameters with the minimum validation loss were saved.
[0058] Advantages and significant effects of the present invention: The method uses semantic information-guided depth estimation, which can solve the problem that the depth estimation of the same object is not smooth due to large image distortion when fisheye camera performs depth estimation, and can effectively improve the accuracy of depth estimation when using fisheye camera.
[0059] Although the embodiments of the present invention are described with reference to actual solutions, they do not constitute a limitation on the meaning of the present invention. Modifications to the embodiments and combinations with other solutions based on this specification will be obvious to those skilled in the art.
Claims
1. A semantic feature-guided scene depth estimation method for fisheye cameras, characterized in that: First, a semantic information perception module is designed to extract semantic information from the input distorted fisheye image. Then, a scene depth perception module is established to guide the perception of depth features using semantic information. Next, a camera pose perception module is established to obtain the camera's pose information. Finally, the image is reconstructed using scene depth map information and camera pose information, a loss function is designed to train the model, and the network parameters are optimized. Specifically, the following steps are included: Step 1: Design the semantic information perception module Sub-step 1.1: Design the semantic encoder The semantic encoder consists of a two-dimensional convolutional layer, three residual modules, and a feature fusion module. Each residual module contains a two-dimensional convolutional layer. The convolutional layers process the input image to obtain feature map F1. Residual module 1 processes feature map F1 to obtain feature map F2. Residual module 2 uses an expanded convolutional layer to increase the receptive field of the image and processes feature map F2 to obtain feature map F3. The convolutional layer of residual module 3 adds two additional channels to learn pixel coordinate features and camera parameter features, extracts image distortion features, and processes feature map F3 to obtain feature map F4. Finally, feature maps F2, F3, and F4 are input together into the feature fusion module to obtain feature map F5. Sub-step 1.2: Design a semantic decoder The semantic decoder consists of three deconvolutional layers, three feature fusion modules, and one convolutional layer. Feature map F5 is input to the deconvolutional layer, upsampled to restore feature map F4' to the size of feature map F4, and then F4 and F4' are fused to generate feature map F4”. Then, F4” is used as input, upsampled to restore feature map F3' to the size of feature map F3, and then F3 and F3' are fused to generate feature map F3”. Next, F3” is used as input, upsampled to restore feature map F2' to the size of feature map F2, and then F2 and F2' are fused to generate feature map F2”. Finally, a convolutional layer is used to generate the semantic segmentation information map of the scene. Step 2: Design the scene depth perception module Sub-step 2.1: Design the depth encoder The deep encoder uses the same backbone network as the semantic encoder, and the input image is processed by the encoder to obtain the feature map F5; Sub-step 2.2: Design a semantically guided deep decoder The deep decoder consists of four deconvolutional layers, three feature fusion modules, and three convolutional layers for semantic feature mapping; the semantic feature mapping formula is shown in Equation (1): (1); Where f represents semantic features, K is the convolution kernel, W is the convolution weight, v represents the convolutional input information, and p is the pixel coordinate; First, feature map F5 is input into a deconvolutional layer and upsampled to restore feature map D4 to the size of feature map F4. F4 and D4 are then fused to generate feature map D4'. Simultaneously, the semantic feature map F4” obtained from the semantic decoder is input into a convolutional layer for semantic feature mapping, and then fused with D4' to generate depth map D4”. Next, D4” is used as input and upsampled to restore feature map D3 to the size of feature map F3. F3 and D3 are then fused to generate feature map D3'. Simultaneously, the semantic feature map F3” obtained from the semantic decoder is input into a convolutional layer for semantic feature mapping, and then fused with D4' to generate depth map D3”. Finally, D3” is used as input and upsampled to restore feature map D2 to the size of feature map F2. F2 and D2 are then fused to generate feature map D2'. Simultaneously, the semantic feature map F2” obtained from the semantic decoder is input into a convolutional layer for semantic feature mapping, and then fused with D2' to generate depth map D2”. This yields the predicted depth map of the scene. Step 3: Design the model's loss function and train the model. The loss function of the depth perception module in this scene consists of two parts: the pixel cross-entropy loss for semantic segmentation as shown in Equation (2) and the depth smoothing loss as shown in Equation (3). (2); in, It is the predicted probability that a pixel belongs to category s. It is the true probability that a pixel belongs to category s; (3)； in and These represent the vertical and horizontal gradients, respectively. The depth smoothing loss ensures that the depth changes are large at the locations of object contours in the scene depth map, while the scene depth image is as smooth as possible at other locations, thus promoting the smoothness of the scene depth map. In summary, the complete loss of the designed model is shown in equation (4), where It controls the weight of the semantic segmentation module's loss in the total loss. The size is 0.3; The weighting factor controls the weights of the deep smoothing loss. Set it to 0.
1. (4); The WoodScape fisheye image dataset and WoodScape fisheye image semantic labels were used as the training set. The initial learning rate was set to 0.0002, and the learning rate was automatically decreased during training. The parameters β1 and β2 of the Adam iterator were set to 0.9 and 0.999, respectively, and the batch size was set to 8. The above loss function was used for feedback to train the model. After each epoch, the training loss and validation loss were calculated, the validation loss of each epoch was compared, and the model parameters with the minimum validation loss were saved.