A method for scene depth estimation of fisheye camera based on unsupervised learning
Through the combination of unsupervised learning and generative adversarial network, the problem of image distortion and insufficient depth samples of fisheye cameras is solved, low-cost and efficient scene depth estimation under fisheye cameras is achieved, and the depth perception needs in intelligent traffic is met.
Patent Information
- Application Number
- CN202210698689.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-20
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-06-20
AI Technical Summary
Due to the short focal length and large viewing angle, the image distortion is severe. It is directly used for vehicle visual perception, which will lead to perception result errors. In addition, existing deep learning methods rely mostly on labeled traditional camera data, and the depth sample information of the fisheye image dataset. Fewer, it is difficult to effectively perform scene depth estimation.
Design a fisheye camera airport depth estimation method based on unsupervised learning, using a generative adversarial network for image distortion correction, and combining depth perception and pose perception modules, extract depth directly from fisheye images through end-to-end neural network structure. Information, output scene depth map.
It realizes low-cost and efficient scene depth estimation under fisheye cameras, has a wide perception range and accurate perception results, and meets the depth perception needs in intelligent transportation.
Smart Images

Figure CN115063463B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a scene depth estimation method for a fisheye camera, and in particular to a scene depth estimation method for a fisheye camera based on unsupervised learning, and belongs to the field of scene perception. Background Art
[0002] Transportation is the foundation and lifeline of my country's national economy. The level of transportation development is an important indicator of a country's overall strength and scientific and technological level. With the advent of a new round of scientific and technological revolution, the development of intelligent transportation has become an inevitable trend in transportation development. This has put forward new requirements for the perception of smart cars.
[0003] At present, the visual perception of vehicles still mainly relies on on-board high-definition cameras. The on-board cameras have a small observation range and certain blind spots, and the road conditions obtained are few. However, fisheye cameras with fisheye lenses have a large observation range, and the viewing angle of the fisheye lens is equal to or greater than 180 degrees, and the depth of field obtained is far. However, the shorter the focal length and the larger the viewing angle, the stronger the deformation caused by optical principles. Therefore, fisheye cameras have the problem of image distortion. If the fisheye image is directly applied to the field of visual perception of the vehicle, it will cause certain errors in the perception results.
[0004] In addition, scene depth estimation is an important task in the field of vehicle visual perception. By estimating the distance from each pixel in the image to the camera, depth information is provided for three-dimensional reconstruction and distance perception of the scene. With the rapid development of artificial intelligence technology, deep learning has a strong self-learning ability, which provides a new idea for scene depth estimation of vehicles. The current deep learning methods are mostly focused on images based on traditional cameras, using labeled scene depth map samples to complete the task of depth estimation. In view of the above-mentioned phenomenon of fisheye image distortion and the problem that the current fisheye image data set contains less depth sample information, the present invention discloses an unsupervised scene depth estimation method for fisheye images. The method disclosed in the present invention adopts an end-to-end neural network structure to directly perceive depth information from the original fisheye image obtained by the fisheye camera, that is, input the original image and finally output the scene depth map. The method uses a fisheye camera, which has the characteristics of a wide observation range; the unsupervised learning method adopted has the characteristics of easy implementation and low cost, and conveniently and quickly meets the depth estimation requirements of the scene. Summary of the invention
[0005] The present invention proposes a fisheye camera scene depth estimation method based on unsupervised learning. The method uses a fisheye camera, designs a network architecture that takes image distortion into consideration, and perceives a scene depth map in combination with the camera pose estimation information of the image. The method has the characteristics of low cost, wide perception range, and accurate perception results.
[0006] The technical solution adopted by the present invention is as follows: a method for scene depth estimation of a fisheye camera based on unsupervised learning, characterized in that: first, a generative adversarial network architecture is established to perform distortion correction processing on the input fisheye image; then, a scene depth estimation network architecture is established to extract depth features and posture features of the image; then, the image is reconstructed using scene depth map information and camera posture information, a loss function training model is designed, and network parameters are optimized; finally, the trained network is used to perform depth estimation to obtain a scene depth map of the image, specifically comprising the following steps:
[0007] Step 1: Design a Generative Adversarial Network
[0008] A generative adversarial network is designed to correct the distortion of fisheye images. The generative adversarial network consists of two generators and two discriminators. First, the original fisheye image X and the undistorted image dataset Y are constructed as training sets. X is first converted into an undistorted image in the Y domain through the generator G, and then reconstructed back to the original distorted image input in the X domain through the generator F. The image in the Y domain is first converted into a distorted fisheye image in the X domain through the generator F, and then reconstructed back to the original image input in the Y domain through the generator G. Generator G and discriminator D Y Related, D Y Distinguish Y from generated data G(X), generator F and discriminator D X Related, D X Distinguish X from generated data F(Y). Generator G and generator F use the following generation network structure, discriminator D X and the discriminator D Y The discriminant network structure described below is adopted. Specifically, it includes the following sub-steps:
[0009] Sub-step 1.1: Design and generate network architecture
[0010] The generative network consists of an encoding structure and a decoding structure. The encoder uses downsampling to extract features from the image, and the decoder uses upsampling to reconstruct the image. The encoder network consists of an input layer, three convolutional layers, and a fully connected layer. The sizes of the three layers of filters between the input layer and the fully connected layer are 7, 5, and 3, respectively, with step sizes of 2, 2, and 4, respectively. The number of feature maps is 64, 128, and 256. The number of nodes in the fully connected layer is 256, and a scalar is output; the decoder network consists of three sequentially connected deconvolutional layers and an output layer. The sizes of the three layers of filters between the first deconvolutional layer and the output layer are 5, 3, and 3, respectively, with step sizes of 4, 2, and 2, respectively. The number of feature maps is 128, 64, and 3, respectively. First, a 256 × 256 × 3 fisheye image is used as the input tensor. The encoder downsamples the image, and the convolutional layer extracts features from the input. The activation function uses Relu. The decoder uses the upsampling method to map the features into an image, uses the activation function Leaky Relu to activate the upsampling layer, and finally outputs an image of 256 × 256 × 3. The instance normalization layer is specially used in the generative network to accelerate the convergence of the model while maintaining the independence of each image instance.
[0011] Sub-step 1.2: Designing the adversarial network architecture
[0012] The discriminant network adopts the idea of PatchGANs structure and consists of four convolutional layers. The instance normalization layer is not used in the first and last layers, but in the rest of the layers. The activation function uses Leaky Relu, and the output is a 30×30 matrix. The mean of the matrix is used as the true or false judgment output.
[0013] Sub-step 1.3: Designing the loss function for the generative adversarial network
[0014] The loss function of the generative adversarial network consists of three parts: the losses of the two generator-discriminator pairs, as shown in equations (1) and (2), and the cycle-consistent loss, as shown in equation (3).
[0015] L GAN (G,D Y )=E Y~data(Y) [log(D Y (Y))]+E X~data(X) [log(1-D Y (G(X)))] (1)
[0016] L GAN (F,D X )=E X~data(X) [log(D X (X))]+E Y~data(Y) [log(1-D X(F(Y)))] (2)
[0017] L cyc l e (G,F)=E X~data(X) |F(G(X))-X|+E Y~data(Y) |G(F(Y))-Y| (3)
[0018] Among them, E X~data(X) , E Y~data(Y) Both represent mathematical expectations.
[0019] In summary, the complete loss of the designed generative adversarial neural network is shown in formula (4), where λ c is the weighting factor, controlling the weight of the cycle-consistent loss, λ c Set to 10.
[0020] L GAN (G,F,D X ,D Y )=L GAN (G,D Y )+L GAN (F,D X )+λ c L cycle (G,F) (4)
[0021] Step 2: Design scene depth estimation network
[0022] The scene depth estimation network consists of a depth perception module and a pose perception module. It performs depth estimation on the distortion-corrected image. It includes the following sub-steps:
[0023] Sub-step 2.1: Design the depth perception module
[0024] The depth perception module inputs the distortion-corrected target image I t , output the scene depth map D of the image t , whose structure consists of an encoder and a decoder.
[0025] (1) The coding structure is as follows:
[0026] Convolutional layer 1_1: convolution is performed with a 7×7 receptive field and a 256×256×3 input tensor with a step size of 2, resulting in a feature map F1 with a dimension of 128×128×64.
[0027] Convolutional layer 1_2: Use the 5×5 receptive field to perform convolution with the feature map output by convolutional layer 1_1, with a step size of 2, to obtain a feature map F2 with a dimension of 64×64×128;
[0028] Convolutional layer 1_3: Use the 5×5 receptive field to convolve with the feature map output by convolutional layer 1_2, with a step size of 2, to obtain a feature map F3 with a dimension of 32×32×256;
[0029] Convolutional layer 1_4: Use the 3×3 receptive field to convolve with the feature map output by convolutional layer 1_3, with a step size of 2, to obtain a feature map F4 with a dimension of 16×16×512;
[0030] (2) In the decoding structure, a deconvolution layer is used to input feature map F4, which is upsampled to restore to feature map F4' of the size of feature map F3, and then feature map F43 is generated by fusion of F3 and F4'; F43 is then used as input and upsampled to restore to feature map F3' of the size of feature map F2, and then feature map F32 is generated by fusion of F2 and F3'; FM32 is then used as input and upsampled to restore to feature map F2' of the size of feature map F1, and then feature map F21 is generated by fusion of F1 and F2'; finally, a convolution layer is input to generate a scene depth map.
[0031] Sub-step 2.2: Designing the pose perception module
[0032] The pose perception module inputs the distortion-corrected target image I t and the source image I at adjacent moments s For these two frames, the output source image is relative to the target image. t→s Its structure consists of seven 3×3 convolutional layers and one 1×1 convolutional layer. The first seven convolutional layers use the activation function Relu to extract image features, and then use 1×1 convolution to set the number of feature map channels to 6 to output the feature map. Finally, the dimension of the feature map is transformed to generate a six-dimensional vector, which is the relative camera pose change between adjacent frames.
[0033] Sub-step 2.3: Reconstruct the target image
[0034] According to the scene depth map and pose information obtained in the above steps, the target image I can be reconstructed t '. Given that the camera intrinsic parameter matrix is K, we can perform the mapping shown in equation (5):
[0035] p s ~KT t→s D t (p t )K -1 p t (5)
[0036] Among them, p t is the target image I t The homogeneous coordinates of a pixel in the source image I can be obtained by formula (5)s The projection coordinates p on s , I in the reconstructed image t '(p t )=I s '(p s ). At this time, the p s It is a continuous value, and the pixel value of the point is calculated using the bilinear sampling method, that is, I s (p s ) is obtained by s Upsampling p s The result of bilinear interpolation of the surrounding four pixels is shown in formula (6).
[0037] I t '(p t )=I s (p s )=∑ i∈{top,bottom},j∈{left,right} ω ij I s (p s ij ) (6)
[0038] Among them, ω ij With p s It is linearly proportional to the spatial distance of the four pixels around it and ∑ i,j ω ij =1. By transforming the target image I t All pixel points of are projected and sampled to reconstruct the target image I t '.
[0039] Sub-step 2.4: Design the loss function of the scene depth estimation network
[0040] The loss function of the scene depth estimation network consists of two parts: the reconstruction target image loss L rec As shown in formula (7), and the deep smoothing loss L smooth As shown in formula (8).
[0041]
[0042]
[0043] in and They represent the vertical and horizontal gradients respectively. The depth smoothing loss can ensure that in the scene depth map, the depth changes are large at locations such as object contours, and the scene depth image at other locations is as smooth as possible, thereby promoting the smoothness of the scene depth map.
[0044] In summary, the complete loss of the designed scene depth estimation network is shown in formula (9), where λs is the weighting factor, controlling the weight of the deep smoothing loss, λ s Set to 0.1.
[0045] L depth =L rec +λ s L smooth (9)
[0046] Step 3: Design the model’s loss function and train the model
[0047] In summary, the final loss function of the model designed by the method of the present invention is shown in formula (10), where λ controls the weight of the loss of the scene depth estimation network in the total loss, and the size of λ is 10.
[0048] L final =L GAN (G,F,D X ,D Y )+λL depth (10)
[0049] The WoodScape fisheye image dataset is used as the training set, and the initial learning rate is set to 0.0002. The learning rate decreases automatically during the training process. The parameters β1 and β2 of the Adam iterator are set to 0.9 and 0.999 respectively, and the batch size is set to 8. The above loss function is used for feedback to train the model. The training loss and validation loss are calculated after each epoch, and the validation loss of each epoch is compared. The model parameters with the smallest validation loss are saved.
[0050] Step 4: Use the trained network for depth estimation
[0051] The image sequence acquired by the fisheye camera is sent as input to the trained model to obtain the depth estimation result of the image.
[0052] Advantages and significant effects of the present invention: The method of the present invention uses a fisheye camera with a wide perception range; the designed unsupervised learning method takes into account the correction of image distortion and combines the camera pose estimation information to realize the perception of scene depth, and has the characteristics of low training cost and accurate perception results. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 It is a schematic diagram of the model structure;
[0054] Figure 2 It is a diagram of the architecture of a generative adversarial network;
[0055] Figure 3 Generate a network structure diagram;
[0056] Figure 4It is the discriminant network structure diagram;
[0057] Figure 5 It is a structural diagram of the depth perception module;
[0058] Figure 6 This is the structure diagram of the posture perception module. DETAILED DESCRIPTION
[0059] Transportation is the foundation and lifeline of my country's national economy. The level of transportation development is an important indicator of a country's overall strength and scientific and technological level. With the advent of a new round of scientific and technological revolution, the development of intelligent transportation has become an inevitable trend in transportation development. This has put forward new requirements for the perception of smart cars.
[0060] At present, the visual perception of vehicles still mainly relies on on-board high-definition cameras. The on-board cameras have a small observation range and certain blind spots, and the road conditions obtained are few. However, fisheye cameras with fisheye lenses have a large observation range, and the viewing angle of the fisheye lens is equal to or greater than 180 degrees, and the depth of field obtained is far. However, the shorter the focal length and the larger the viewing angle, the stronger the deformation caused by optical principles. Therefore, fisheye cameras have the problem of image distortion. If the fisheye image is directly applied to the field of visual perception of the vehicle, it will cause certain errors in the perception results.
[0061] In addition, scene depth estimation is an important task in the field of vehicle visual perception. By estimating the distance from each pixel in the image to the camera, depth information is provided for three-dimensional reconstruction and distance perception of the scene. With the rapid development of artificial intelligence technology, deep learning has a strong self-learning ability, which provides a new idea for scene depth estimation of vehicles. The current deep learning methods are mostly focused on images based on traditional cameras, using labeled scene depth map samples to complete the task of depth estimation. In view of the above-mentioned phenomenon of fisheye image distortion and the problem that the current fisheye image data set contains less depth sample information, the present invention discloses an unsupervised scene depth estimation method for fisheye images. The method disclosed in the present invention adopts an end-to-end neural network structure to directly perceive depth information from the original fisheye image obtained by the fisheye camera, that is, input the original image and finally output the scene depth map. The method uses a fisheye camera, which has the characteristics of a wide observation range; the unsupervised learning method adopted has the characteristics of easy implementation and low cost, and conveniently and quickly meets the depth estimation requirements of the scene.
[0062] The present invention proposes a fisheye camera scene depth estimation method based on unsupervised learning. The method uses a fisheye camera, designs a network architecture that takes image distortion into consideration, and perceives a scene depth map in combination with the camera pose estimation information of the image. The method has the characteristics of low cost, wide perception range, and accurate perception results.
[0063] The technical solution adopted by the present invention is as follows: a method for scene depth estimation of a fisheye camera based on unsupervised learning, characterized in that: first, a generative adversarial network architecture is established to perform distortion correction processing on the input fisheye image; then, a scene depth estimation network architecture is established to extract depth features and posture features of the image; then, the image is reconstructed using scene depth map information and camera posture information, a loss function training model is designed, and network parameters are optimized; finally, the trained network is used to perform depth estimation to obtain a scene depth map of the image. The model structure of the present invention is as follows Figure 1 As shown, the specific steps include:
[0064] Step 1: Design a Generative Adversarial Network
[0065] A generative adversarial network is designed to correct the distortion of fisheye images. The generative adversarial network consists of two generators and two discriminators. First, the original fisheye image X and the undistorted image dataset Y are constructed as training sets. X is first converted into an undistorted image in the Y domain through the generator G, and then reconstructed back to the original distorted image input in the X domain through the generator F. The image in the Y domain is first converted into a distorted fisheye image in the X domain through the generator F, and then reconstructed back to the original image input in the Y domain through the generator G. Generator G and discriminator D Y Related, D Y Distinguish Y from generated data G(X), generator F and discriminator D X Related, D X Distinguish X from generated data F(Y). Generator G and generator F use the following generation network structure, discriminator D X and the discriminator D Y The discriminant network structure described below is adopted. The designed network architecture is as follows Figure 2 As shown, it specifically includes the following sub-steps:
[0066] Sub-step 1.1: Design and generate network architecture
[0067] The structure of the generated network is given by Figure 3As shown in the figure, it consists of an encoding structure and a decoding structure. The encoder uses downsampling to extract features from the image, and the decoder uses upsampling to reconstruct the image. The encoder network consists of an input layer, three convolutional layers, and a fully connected layer. The sizes of the three layers of filters between the input layer and the fully connected layer are 7, 5, and 3, respectively, with step sizes of 2, 2, and 4, respectively. The number of feature maps is 64, 128, and 256. The number of nodes in the fully connected layer is 256, and a scalar is output; the decoder network consists of three deconvolutional layers and an output layer connected in sequence. The sizes of the three layers of filters between the first deconvolutional layer and the output layer are 5, 3, and 3, respectively, with step sizes of 4, 2, and 2, respectively. The number of feature maps is 128, 64, and 3, respectively. First, a 256×256×3 fisheye image is used as the input tensor. The encoder downsamples the image, and the convolutional layer extracts features from the input. The activation function uses Relu. The decoder uses the upsampling method to map the features to an image, uses the activation function LeakyRelu to activate the upsampling layer, and finally outputs an image of 256×256×3. The instance normalization layer is specifically used in the generative network to accelerate the convergence of the model while maintaining the independence of each image instance.
[0068] Sub-step 1.2: Designing the adversarial network architecture
[0069] The discriminant network structure is composed of Figure 4 As shown in the figure, the idea of PatchGANs structure is adopted, which consists of four convolutional layers. The instance normalization layer is not used in the first and last layers, but in the rest of the layers. The activation function uses Leaky Relu, and the output is a 30×30 matrix. The mean of the matrix is used as the true or false judgment output.
[0070] Sub-step 1.3: Designing the loss function for the generative adversarial network
[0071] The loss function of the generative adversarial network consists of three parts: the losses of the two generator-discriminator pairs, as shown in equations (1) and (2), and the cycle-consistent loss, as shown in equation (3).
[0072] L GAN (G,D Y )=E Y~data(Y) [log(D Y (Y))]+E X~data(X) [log(1-D Y (G(X)))] (1)
[0073] L GAN (F,D X )=E X~data(X) [log(D X (X))]+E Y~data(Y)[log(1-D X (F(Y)))] (2)
[0074] L cycle (G,F)=E X~data(X) |F(G(X))-X|+E Y~data(Y) |G(F(Y))-Y| (3)
[0075] Among them, E X~data(X) , E Y~data(Y) Both represent mathematical expectations.
[0076] In summary, the complete loss of the designed generative adversarial neural network is shown in formula (4), where λ c is the weighting factor, controlling the weight of the cycle-consistent loss, λ c Set to 10.
[0077] L GAN (G,F,D X ,D Y )=L GAN (G,D Y )+L GAN (F,D X )+λ c L cycle (G,F) (4)
[0078] Step 2: Design scene depth estimation network
[0079] The structure of the scene depth estimation network is as follows Figure 1 As shown in the figure, it consists of a depth perception module and a posture perception module. It performs depth estimation on the image after distortion correction. It specifically includes the following sub-steps:
[0080] Sub-step 2.1: Design the depth perception module
[0081] The structure of the depth perception module is as follows Figure 5 As shown, the input target image I after distortion correction t , output the scene depth map D of the image t , whose structure consists of an encoder and a decoder.
[0082] (1) The coding structure is as follows:
[0083] Convolutional layer 1_1: convolution is performed with a 7×7 receptive field and a 256×256×3 input tensor with a step size of 2, resulting in a feature map F1 with a dimension of 128×128×64.
[0084] Convolutional layer 1_2: Use the 5×5 receptive field to perform convolution with the feature map output by convolutional layer 1_1, with a step size of 2, to obtain a feature map F2 with a dimension of 64×64×128;
[0085] Convolutional layer 1_3: Use the 5×5 receptive field to convolve with the feature map output by convolutional layer 1_2, with a step size of 2, to obtain a feature map F3 with a dimension of 32×32×256;
[0086] Convolutional layer 1_4: Use the 3×3 receptive field to convolve with the feature map output by convolutional layer 1_3, with a step size of 2, to obtain a feature map F4 with a dimension of 16×16×512;
[0087] (2) In the decoding structure, a deconvolution layer is used to input feature map F4, which is upsampled to restore to feature map F4' of the size of feature map F3, and then feature map F43 is generated by fusion of F3 and F4'; F43 is then used as input and upsampled to restore to feature map F3' of the size of feature map F2, and then feature map F32 is generated by fusion of F2 and F3'; FM32 is then used as input and upsampled to restore to feature map F2' of the size of feature map F1, and then feature map F21 is generated by fusion of F1 and F2'; finally, a convolution layer is input to generate a scene depth map.
[0088] Sub-step 2.2: Designing the pose perception module
[0089] The structure of the posture perception module is as follows Figure 6 As shown, the input target image I after distortion correction t and the source image I at adjacent moments s For these two frames, the output source image is relative to the target image. t→s Its structure consists of seven 3×3 convolutional layers and one 1×1 convolutional layer. The first seven convolutional layers use the activation function Relu to extract image features, and then use 1×1 convolution to set the number of feature map channels to 6 to output the feature map. Finally, the dimension of the feature map is transformed to generate a six-dimensional vector, which is the relative camera pose change between adjacent frames.
[0090] Sub-step 2.3: Reconstruct the target image
[0091] According to the scene depth map and pose information obtained in the above steps, the target image I can be reconstructed t '. Given that the camera intrinsic parameter matrix is K, we can perform the mapping shown in equation (5):
[0092] p s ~KT t→s D t (p t )K -1 p t (5)
[0093] Among them, p t is the target image It The homogeneous coordinates of a pixel in the source image I can be obtained by formula (5) s The projection coordinates p on s , I in the reconstructed image t '(p t )=I s '(p s ). At this time, the p s It is a continuous value, and the pixel value of the point is calculated using the bilinear sampling method, that is, I s (p s ) is obtained by s Upsampling p s The result of bilinear interpolation of the surrounding four pixels is shown in formula (6).
[0094] I t '(p t )=I s (p s )=∑ i∈{top,bottom},j∈{left,right} ω ij I s (p s ij ) (6)
[0095] Among them, ω ij With p s It is linearly proportional to the spatial distance of the four pixels around it and ∑ i,j ω ij =1. By transforming the target image I t All pixel points of are projected and sampled to reconstruct the target image I t '.
[0096] Sub-step 2.4: Design the loss function of the scene depth estimation network
[0097] The loss function of the scene depth estimation network consists of two parts: the reconstruction target image loss L rec As shown in formula (7), and the deep smoothing loss L smooth As shown in formula (8).
[0098]
[0099]
[0100] in and They represent the vertical and horizontal gradients respectively. The depth smoothing loss can ensure that in the scene depth map, the depth changes are large at locations such as object contours, and the scene depth image at other locations is as smooth as possible, thereby promoting the smoothness of the scene depth map.
[0101] In summary, the complete loss of the designed scene depth estimation network is shown in formula (9), where λ s is the weighting factor, controlling the weight of the deep smoothing loss, λ s Set to 0.1.
[0102] L depth =L rec +λ s L smooth (9)
[0103] Step 3: Design the model’s loss function and train the model
[0104] In summary, the final loss function of the model designed by the method of the present invention is shown in formula (10), where λ controls the weight of the loss of the scene depth estimation network in the total loss, and the size of λ is 10.
[0105] L final =L GAN (G,F,D X ,D Y )+λL depth (10)
[0106] The WoodScape fisheye image dataset is used as the training set, and the initial learning rate is set to 0.0002. The learning rate decreases automatically during the training process. The parameters β1 and β2 of the Adam iterator are set to 0.9 and 0.999 respectively, and the batch size is set to 8. The above loss function is used for feedback to train the model. The training loss and validation loss are calculated after each epoch, and the validation loss of each epoch is compared. The model parameters with the smallest validation loss are saved.
[0107] Step 4: Use the trained network for depth estimation
[0108] The image sequence acquired by the fisheye camera is sent as input to the trained model to obtain the depth estimation result of the image.
[0109] The method of the present invention uses a fisheye camera with a wide perception range; the designed unsupervised learning method takes into account the correction of image distortion and combines the camera pose estimation information to realize the perception of scene depth, and has the characteristics of low training cost and accurate perception results.
Claims
1. A fisheye camera scene depth estimation method based on unsupervised learning, characterized by: First, a generative adversarial network architecture is established to perform distortion correction on the input fisheye image; then a scene depth estimation network architecture is established to extract depth features and pose features of the image; then the image is reconstructed using the scene depth map information and camera pose information, a loss function training model is designed, and network parameters are optimized; finally, the trained network is used for depth estimation to obtain the scene depth map of the image, which specifically includes the following steps: Step 1: Design a Generative Adversarial Network A generative adversarial network is designed to correct the distortion of fisheye images. The generative adversarial network consists of two generators and two discriminators. First, the original fisheye image X and the undistorted image dataset Y are constructed as training sets. X is first converted into an undistorted image in the Y domain through the generator G, and then reconstructed back to the original distorted image input in the X domain through the generator F. The image in the Y domain is first converted into a distorted fisheye image in the X domain through the generator F, and then reconstructed back to the original image input in the Y domain through the generator G. The generator G and the discriminator D Y Related, D Y Distinguish Y from generated data G(X), generator F and discriminator D X Related, D X Distinguish X from generated data F(Y); Generator G and Generator F use the following generation network structure, discriminator D X and the discriminator D Y The discriminant network structure described below is adopted; specifically, it includes the following sub-steps: Sub-step 1.1: Design and generate network architecture The generator network consists of an encoder network and a decoder network. The encoder in the encoder network extracts features from the image by downsampling, and the decoder in the decoder network reconstructs the image by upsampling. The encoder network consists of an input layer, three convolutional layers and a fully connected layer. The sizes of the three-layer filters between the input layer and the fully connected layer are 7, 5, and 3, respectively, with step sizes of 2, 2, and 4, respectively. The number of feature maps is 64, 128, and 256, and the number of nodes in the fully connected layer is 256, and a scalar is output. The decoder network consists of three sequentially connected deconvolutional layers and an output layer. The sizes of the three-layer filters between the first deconvolutional layer and the output layer are 5, 3, and 3, respectively, with step sizes of 4, 2, and 2, respectively, and the number of feature maps is 128, 64, and 3, respectively. First, a 256×256×3 fisheye image is used as the input tensor. The encoder downsamples the image, and the convolutional layer extracts features from the input. The activation function uses Relu. The decoder uses the upsampling method to map features to images, and uses the activation function Leaky Relu activates the upsampling layer and finally outputs a 256×256×3 image. The instance normalization layer is specially used in the generative network to accelerate the convergence of the model while maintaining the independence of each image instance. Sub-step 1.2: Designing the adversarial network architecture The discriminant network adopts the idea of PatchGANs structure and consists of four convolutional layers. The instance normalization layer is not used in the first and last layers, but in the remaining layers. The activation function uses Leaky Relu, and the output is a 30×30 matrix. The mean of the matrix is used as the true or false judgment output. Sub-step 1.3: Designing the loss function for the generative adversarial network The loss function of the generative adversarial network consists of three parts: the loss of the two generator-discriminator pairs, as shown in equations (1) and (2); and the cycle-consistent loss, as shown in equation (3); L GAN (G,D Y )=E Y~data(Y) [log(D Y (Y))]+E X~data(X) [log(1-D Y (G(X)))] (1) L GAN (F,D X )=E X~data(X) [log(D X (X))]+E Y~data(Y) [log(1-D X (F(Y)))] (2) L cycle (G,F)=E X~data(X) |F(G(X))-X|+E Y~data(Y) |G(F(Y))-Y| (3) Among them, E X~data(X) , E Y~data(Y) All represent mathematical expectations; In summary, the complete loss of the designed generative adversarial neural network is shown in formula (4), where λ c is the weighting factor, controlling the weight of the cycle-consistent loss, λ c Set to 10; L GAN (G,F,D X ,D Y )=L GAN (G,D Y )+L GAN (F,D X )+λ c L cycle (G,F) (4) Step 2: Design scene depth estimation network The scene depth estimation network consists of a depth perception module and a pose perception module; it performs depth estimation on the distortion-corrected image; specifically includes the following sub-steps: Sub-step 2.1: Design the depth perception module The depth perception module inputs the distortion-corrected target image I t , output the scene depth map D of the image t , whose structure consists of an encoder and a decoder; (1) The encoder structure is as follows: Convolutional layer 1_1: convolution is performed with a 7×7 receptive field and a 256×256×3 input tensor with a step size of 2, resulting in a feature map F1 with a dimension of 128×128×64. Convolutional layer 1_2: Use the 5×5 receptive field to perform convolution with the feature map output by convolutional layer 1_1, with a step size of 2, to obtain a feature map F2 with a dimension of 64×64×128; Convolutional layer 1_3: Use the 5×5 receptive field to convolve with the feature map output by convolutional layer 1_2, with a step size of 2, to obtain a feature map F3 with a dimension of 32×32×256; Convolutional layer 1_4: Use the 3×3 receptive field to convolve with the feature map output by convolutional layer 1_3, with a step size of 2, to obtain a feature map F4 with a dimension of 16×16×512; (2) In the decoder structure, a deconvolution layer is used to input feature map F4, which is upsampled to feature map F4' of the size of feature map F3, and then feature map F43 is generated by fusion of F3 and F4'; F43 is then used as input and upsampled to feature map F3' of the size of feature map F2, and then feature map F32 is generated by fusion of F2 and F3'; FM32 is then used as input and upsampled to feature map F2' of the size of feature map F1, and then feature map F21 is generated by fusion of F1 and F2'; finally, a convolution layer is input to generate a scene depth map; Sub-step 2.2: Designing the pose perception module The pose perception module inputs the distortion-corrected target image I t and the source image I at adjacent moments s For these two frames, the output source image is relative to the target image. t→s ; Its structure consists of seven 3×3 convolutional layers and one 1×1 convolutional layer. The first seven convolutional layers use the activation function Relu to extract image features, and then use 1×1 convolution to set the number of feature map channels to 6 to output the feature map. Finally, the dimension of the feature map is transformed to generate a six-dimensional vector, which is the relative camera pose change between adjacent frames. Sub-step 2.3: Reconstruct the target image According to the scene depth map and pose information obtained in the above steps, the target image I is reconstructed. t '; Knowing that the camera intrinsic parameter matrix is K, make the mapping as shown in formula (5): p s ~KT t→s D t (p t )K -1 p t (5) Among them, p t is the target image I t The homogeneous coordinates of a pixel in the source image I can be obtained by formula (5) s The projection coordinates p on s , I in the reconstructed image t '(p t )=I s '(p s ); At this time, the p s It is a continuous value, and the pixel value of the point is calculated using the bilinear sampling method, that is, I s (p s ) is obtained by s Upsampling p s The result of bilinear interpolation of the surrounding four pixels is shown in formula (6); I t '(p t )=I s (p s )=∑ i∈{top,bottom},j∈{left,right} ωijI s (p s ij ) (6) Among them, ω ij With p s It is linearly proportional to the spatial distance of the four pixels around it and ∑ i,j ω ij =1; by taking the target image I t All pixel points of are projected and sampled to reconstruct the target image I t '; Sub-step 2.4: Design the loss function of the scene depth estimation network The loss function of the scene depth estimation network consists of two parts: the reconstruction target image loss L rec As shown in formula (7), and the deep smoothing loss L smooth As shown in formula (8); in and Representing the vertical and horizontal gradients respectively, the depth smoothing loss can ensure that in the scene depth map, the depth changes are large at the object contour position, and the scene depth image is smooth at other positions, thus promoting the smoothness of the scene depth map; In summary, the complete loss of the designed scene depth estimation network is shown in formula (9), where λ s is the weighting factor, controlling the weight of the deep smoothing loss, λ s Set to 0.1; L depth =L rec +λ s L smooth (9) Step 3: Design the model’s loss function and train the model In summary, the final loss function of the model designed by the method of the present invention is shown in formula (10), where λ controls the weight of the loss of the scene depth estimation network in the total loss, and the value of λ is 10; L final =L GAN (G,F,D X ,D Y )+λL depth (10) Use the WoodScape fisheye image dataset as the training set, set the initial learning rate to 0.0002, and the learning rate automatically decreases during the training process. Set the parameters β1 and β2 of the Adam iterator to 0.9 and 0.999 respectively, and set the batch size to 8. Use the above loss function for feedback to train the model. Calculate the training loss and validation loss after each epoch, compare the validation loss of each epoch, and save the model parameters with the smallest validation loss. Step 4: Use the trained network for depth estimation The image sequence acquired by the fisheye camera is sent as input to the trained model to obtain the depth estimation result of the image.
