Event camera depth estimation method based on unsupervised domain adaptation

Abstract

The application discloses an event camera depth estimation method based on unsupervised domain adaptation, and is suitable for the technical field of event camera depth estimation in the computer vision field. The method comprehensively analyzes the differences between the image domain and the event domain, and strengthens the connection between the two modes by changing the image style. This method can make the features in the implicit space more easily aligned, and reduce the gap in the explicit space. Training in the labeled image field, then transferring the knowledge to the unlabeled event data; using cross-domain feature alignment and depth alignment to transfer knowledge from images to events, using appearance for style transfer, and using edge invariant consistency to minimize the gap between different fields. Through unsupervised domain adaptation, event monocular depth estimation is carried out only by using the depth label of the image.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of event camera depth estimation in the field of computer vision, and particularly relates to an event camera depth estimation method based on unsupervised domain adaptation. BACKGROUND

[0002] With the rapid development of emerging applications such as autonomous driving and agile robot navigation, visual perception in high-speed motion scenes has become a serious challenge. Especially in variable outdoor conditions, these applications require intelligent devices to adapt to rapidly changing environments. As a new type of visual sensor, event cameras have been proven to have the advantages of low latency, robustness to light changes, and reduction of motion blur. Compared with frame-based cameras, event cameras capture asynchronous data sequences, only reporting pixel-level light changes. These asynchronous data can provide efficient scene perception in low light and high-speed scenes. Therefore, research on these event-based perception tasks is crucial for various applications in high-speed motion scenes.

[0003] Among numerous event-based perception tasks, event-based depth estimation is an important visual perception task that can perceive the depth information of a scene to improve the safety and stability of autonomous systems in high-speed environments. Event camera depth estimation, namely, replacing the dense image frame input in traditional depth estimation methods with sparse events generated by event cameras. Event cameras work differently from traditional cameras. Event cameras generate events by receiving information about changes in brightness in a scene, and they are not limited by a fixed frame rate. Events are output as long as there are changes in brightness in the scene. The events generated by the event camera can be represented as e=(x,y,p,t), where x and y represent the horizontal and vertical coordinates of the pixel in the camera coordinate system where the brightness change occurs; p represents the trend of the brightness change, that is, p takes +1 when the brightness changes from low to high, and p takes -1 when the brightness changes from high to low; t represents the time when the brightness change occurs. Event camera depth estimation is to input this four-dimensional data into the model for pixel-by-pixel depth estimation.

[0004] However, the unique working principle of event cameras, while ensuring their robustness to environmental changes, also presents some challenges. Instead of providing pixel information about a scene, event cameras provide sparse and asynchronous perception of pixel changes, and it is difficult to estimate the dense depth of a scene with only sparse input. Therefore, many researchers have proposed a number of event camera depth estimation methods tailored to the characteristics of event cameras, which can be broadly divided into three categories: camera parameter estimation-based methods, supervised learning-based methods, and unsupervised learning-based methods.

[0005] 1. Camera parameter estimation-based methods

[0006] Early scholars estimate depth by model-based methods, that is, using the parameters of the camera (including the position information and motion information of the camera) to estimate the optical flow information and depth information of the objects in the scene. A contrast maximization framework for event camera motion, depth and optical flow estimation solves event vision tasks by maximizing the contrast of warped event images to construct a model. Real-time 3D reconstruction based on multi-view event cameras estimates the three-dimensional structure of the scene from event data and camera pose estimation. Focus is everything: event vision loss function based on event alignment in dynamic scenes, and event vision optimization function is developed. However, such methods require a large number of camera parameters, and the accuracy of the estimated results is not accurate due to the precision of parameter measurement and model optimization problems.

[0007] 2. Supervised learning-based method

[0008] With the development of deep learning, many scholars have studied the application of deep learning in event cameras. The supervised learning-based depth estimation method refers to using a labeled data set to train a deep learning model. Learning monocular dense depth from events (MDDE) shows how to generate dense depth maps using monocular vision, overcoming the sparsity of event streams. Transfer learning for event-based task prediction through event-to-image conversion estimates the depth of the event domain with the help of the experience of the image domain. These works all use CNN as the backbone network for depth estimation, the disadvantage is that the temporal information of the event camera is ignored. Many researchers have taken advantage of the modeling of temporal information by RNN and Transformer to further improve the performance of depth estimation. At the same time, with the pursuit of accuracy, the length and number of input event sequences are constantly increasing, which also increases the redundancy of information and the burden on hardware. Now there are many works on restoring the sparsity of events based on previous research, reducing the inference time of the model while maintaining the accuracy requirements of the previous. Then, all these researches based on supervised learning need a large number of depth estimation labels for training. At present, large-scale high-quality event camera data sets are rare, and the preprocessed depth labels increase the difficulty, which brings certain challenges to the promotion and use of the model.

[0009] 3. Unsupervised learning-based method

[0010] Supervised methods rely on high-quality depth labels, while unsupervised methods do not require depth labels for training. Some unsupervised works also utilize stereo data and reconstruction cues for unsupervised learning. Unsupervised event-based depth stereo graph-based depth estimation utilizes stereo event frames for depth estimation and warps stereo images to assist depth estimation. Unsupervised event camera optical flow, depth, and motion estimation reconstructs the current frame from the previous and next event frames in a monocular event video sequence. It is challenging to obtain large-scale stereo data and video sequences. The large-scale stereo data and video sequences utilized by these methods also limit the use of some event datasets.

[0011] Although these existing methods have achieved success in event-based depth estimation, there are two drawbacks. First, the event input complexity is high. Large-scale event sequences and detailed labels are used for supervised training. Some stereo sequences and a large number of images corresponding to event frames can also be used for training. Therefore, this method limits the generalization ability of the model. Second, the camera setup requirement is high. Some self-supervised methods utilize camera parameters to train the model, which limits the use of datasets and brings challenges to practical applications.

[0012] With the success of unsupervised domain adaptation methods in image depth estimation and some attempts in event camera semantic segmentation, such as Figure 1 As shown in the present application, an unsupervised domain adaptation event camera depth estimation method is proposed to reduce the complexity of event data input using unsupervised domain adaptation technology, thereby realizing depth estimation based only on monocular events. This method greatly reduces the requirements for data and alleviates the problem of label scarcity in event camera research. SUMMARY

[0013] The present application proposes an unsupervised domain adaptation event camera depth estimation method. The goal of the present application is to generate high-density and high-quality depth estimation maps in an unsupervised manner. This method achieves this goal through two main tasks. First, this method conducts research by comprehensively analyzing the differences between the image domain and the event domain. The goal is to strengthen the connection between the two modes by changing the image style. This method can make the features in the implicit space more easily aligned and reduce the gap in the explicit space. Second, this method uses an unsupervised domain adaptation method to train in the labeled image domain and then transfer knowledge to the unlabeled event data. Using these techniques, this method aims to provide a novel unsupervised depth estimation method. The present method alleviates the impact of the lack of labels on event camera depth estimation models, and mainly solves the technical problems including the gap between the image domain and the event domain, the alignment between the features of the two domains, and the improvement of the accuracy of depth estimation.

[0014] The event camera depth estimation method based on unsupervised domain adaptation can effectively alleviate the dependence on labels in supervised learning and the data set limitation of previous unsupervised learning. By introducing a style transfer module, the gap between the image domain and the event domain is effectively reduced, so that the performance reduction in domain adaptation is reduced; by introducing an implicit explicit alignment module, the two domain knowledge is better aligned in the feature, and the performance of the network is improved; by the edge invariant loss, the network better fuses the edge contour information of the event frame, and the accuracy of the depth estimation of the model is improved. Figure 2 The overall structure diagram of the event camera depth estimation based on unsupervised domain adaptation is given.

[0015] The present application is realized by the following technical solutions:

[0016] The input of the model proposed by the present application includes an image domain and an event domain (such as Figure 1 ), first, the image domain and the event domain are sent into the respective encoders to generate features, then the features of different domains are constrained by using a loss function, and then input into a shared decoder to output a depth estimation map for training, and then only part of the event domain is used during testing, that is, the event is input into an event encoder to generate features, and then input into a shared decoder to output a depth map. The specific steps are as follows:

[0017] Step one: constructing an event camera depth estimation network model based on unsupervised domain adaptation

[0018] 1. Training data set input and overall structure design

[0019] The training data set includes the source domain KITTI and the target domain MVSEC Outdoor_day2, DENSE Town1 to Town5. Next, the input process will be described in detail. The source domain is set as a set of dense images I i and corresponding depth maps D i , and the target domain is an event stream of M events, where respectively represent the position, polarity and timestamp of the jth event. When inputting event data into the network, we use the method of stacking event frames to voxelize a fixed number of events e j into E k ∈R C×H×W . Our goal is to transfer the knowledge of the source image domain with depth labels to the target event domain without labels through domain adaptation, and we design three new modules to obtain better event domain depth estimation results. As shown in Figure 2 , the training and implementation of the present application are divided into several steps. In order to reduce the domain gap in the source domain, we first perform style conversion on the source image while retaining the original depth label, and obtain Subsequently, the processed image is sent into the image feature branch:

[0020]

[0021] The feature f img is then input into the subsequent image supervised learning branch, into the shared decoder D share to obtain the depth map, i.e.

[0022] D img = D share (f img ) (2)

[0023] It is then trained using the depth label. In the target domain, the event voxel grid E is sent into the image reconstruction network to generate the reconstructed image, i.e.

[0024] I recon = E2VID(E) (3)

[0025] The I recon is then sent into the image feature branch to obtain the reconstructed image feature, i.e.

[0026] f recon = E img (I recon ) (4)

[0027] The event is also sent into the event feature branch to obtain the event feature, i.e.

[0028] f event = E event (E) (5)

[0029] Finally, the features from different modalities are input into the shared depth estimation head D share , which utilizes the implicit explicit consistency module and the edge invariant module to transfer the depth estimation knowledge of the image domain. We will introduce each module in detail below.

[0030] 2. Design of style transfer module

[0031] To improve the domain adaptation performance, we employ the event-to-image reconstruction network E2VID to make the two domains as close as possible. However, due to the sparsity of events, the reconstructed images can suffer from color imbalance and blurring. Reducing these areas to improve the reconstruction quality can have a positive impact on the transfer between different domains. However, this is part of the event-to-image reconstruction task and is irrelevant to depth estimation. We find that this blurring often appears on the sides of the image or inside the objects. Therefore, instead of focusing on optimizing the quality of the generated images, we consider blurring images as a kind of specific image style. At the same time, we use the style transfer network to apply this style to the image domain. This way, we can reduce the impact of images of different styles on depth estimation. Since this is an unpaired image style transfer, we choose the image transfer network CycleGAN to transfer the style by understanding the similarities and differences between the two domains. We train a style transfer network G and use the pre-trained model G to participate in the depth estimation training in the present invention. The following are several steps of style conversion training. First, input the event stream E into E2VID to get the reconstructed image I recon as the source domain. Second, input the source image I i as the target domain. Then, according to the pre-trained model G obtained by the training process of CycleGAN, perform style conversion of the source image domain.

[0032] 3. Designing an implicit-explicit consistency module

[0033] Our domain adaptation method includes inputting data from different modalities into different encoders and using a shared decoder for depth estimation. By training the encoders and decoders in the image domain with depth labels, we can obtain fine-grained information from the scene and generate higher-quality depth maps. Therefore, our goal is to keep the feature maps and depth maps of the event modality and the image modality consistent. In this way, the knowledge of the image domain can be implicitly and explicitly learned through a series of events and generated images. Therefore, as Figure 2 shown, we design an implicit-explicit consistency module. We propose the following implicit consistency loss to keep the embeddings of events and reconstructed images close:

[0034] Two embeddings f event and f image are obtained from the encoders of different domains, and the L1 distance is used to ensure that the embeddings are implicitly aligned:

[0035]

[0036] Similarly, we propose an explicit consistency loss to explicitly align the depth maps. That is:

[0037]

[0038] 4. Design edge-invariant module

[0039] We hope that the present invention is robust to input perturbations or noise in the inference phase. When perturbations occur, the network can stably perform the prediction task. When events of different scales are input to the network, the resulting features and reconstructed images are different, but the gradient of the overall image will only change slightly. Therefore, when inputting event features to the decoder to generate a depth map, we use edge loss to constrain the depth map and image to help the network generate edge-invariant estimates. That is:

[0040]

[0041] where D img , I recon are considered as functions of x and y, and represent the horizontal and vertical derivatives of the above functions, respectively. Using edge loss helps to preserve more details and structural information at the edges of objects, while reducing details in smooth areas.

[0042] Step two: adjust network parameters, train and test the neural network

[0043] 1. Parameter setting

[0044] The present invention uses Pytorch (1.12.1) architecture, implemented on a single NVIDIA RTX3080ti GPU, using Adam optimizer to realize gradient descent. The batch size of the training is set to 8, and the learning rate is set to 10 -4 . We use ResNet18 as the image encoder to extract three scale feature maps for U-connection with the decoder. The depth estimation decoder consists of three convolutional layers of different scales, a convolutional layer for feature fusion of different scales, and a final depth estimation head.

[0045] 2. Input test data set

[0046] For the MVSEC dataset, Outdoor_day1 and Outdoor_night1 are used as the test dataset; for the DENSE dataset, Town10 is used as the test set. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is the main flowchart of unsupervised domain adaptation event camera depth estimation.

[0048] Figure 2 ​is an overall framework diagram based on unsupervised domain adaptation event camera depth estimation.

[0049] Figure 3 is a high-quality result illustration. DETAILED DESCRIPTION

[0050] The present invention experimentally verifies the above method, and evaluates the effect of depth estimation on two data sets, MVSEC and DENSE.

[0051] 1. Qualitative evaluation

[0052] 1) Cross-domain performance evaluation: We compared the present invention with several depth estimation methods that perform well in outdoor scenes to demonstrate our cross-domain performance. We selected several different types of methods that can produce excellent performance in depth estimation, including domain adaptation methods: 3DPL, GASDA; supervised methods: NeWCRFS; self-supervised methods: NeWCRFS; self-supervised methods: Litemono and generalization methods: S2R-DepthNet. We verified the performance of these methods using a similar training method to ours, which includes training on a labeled image domain and then testing on the event domain. In order to solve the adaptability of the model to different modal inputs, we use E2VID to complete the reconstruction from events to images. In the testing stage, we input events and then pass them to the encoder and depth estimation decoder to generate depth maps. Table 1 lists our comprehensive evaluation results. These results show that our designed method narrows the gap between different domains. In event monocular depth estimation, domain adaptability has been significantly improved.

[0053] 2) Depth estimation evaluation on the MVSEC dataset: In order to demonstrate the effectiveness of the present invention, we compared it with several event depth estimation methods on MVSEC. At present, there are very limited open-source unsupervised event depth. E2VID can complete the conversion from the event domain to the image domain, making it possible to conduct depth estimation in an unsupervised setting. ESS as an unsupervised semantic segmentation model and depth estimation also has a related role. Table 2 shows that our method has achieved excellent results on all indicators compared with other methods. These results show that even without using event labels, we can obtain competitive results. Figure 3 shows high-quality results. Qualitative results show that we perform well in object contours and overall depth representation of the scene in an unsupervised environment.

[0054] 3) Depth Estimation Evaluation on the DENSE Dataset: To further validate the effectiveness of our proposed method, we present additional results on the DENSE dataset. When training on the DENSE dataset, we fine-tuned E2VID using L1 loss on DENSE. All other design modules and hyperparameters remained unchanged. Table 3 shows the relevant performance comparisons. The results demonstrate that our metrics are competitive across different datasets in the unsupervised setting of event monocular depth estimation. This indicates that our method can adapt to different datasets even without corresponding depth labels.

[0055] Table 1: Performance comparison of cross-domain depth estimation with other cross-domain methods in MVSEC

[0056]

[0057]

[0058] Table 2: Performance comparison of depth estimation with other depth estimation methods in MVSEC

[0059]

[0060] Table 3: Performance comparison of depth estimation with other depth estimation methods in DENSE

[0061]

[0062] 2. Subjective effect

[0063] Figure 3 The results of this invention on MVSEC and tests with other methods are shown. In the images marked with white circles, it is clear that this invention is superior to other methods in terms of scene object outlines and depth range. For example, in the fourth row, the reconstructed image clearly shows the vehicle on the left and the railing on the right, which are relatively blurry in other methods, while the depth of the vehicle and the railing can be clearly seen in the prediction of this invention.

Claims

1. An event camera depth estimation method based on unsupervised domain adaptation, characterized by: Step 1: First, design an unsupervised domain adaptation framework; The style transfer module is used to perform style transfer on the image domain and the event domain. Then, the processed image domain and event domain are fed into their respective encoders to generate features. The implicit and explicit consistency modules and the edge invariant module are used to constrain the features of different domains. Finally, the features are input into a shared decoder to output a depth estimation map for training. Step 2: During testing, only the event domain portion is used, that is, the event is input into the event encoder to generate features, and then input into the shared decoder to output a depth map; 1) Unsupervised domain adaptation to the overall framework: setting the source domain For dense image I i and corresponding depth map D i The set of targets, allowing the target domain to... For an event stream containing M events, where These represent the position, polarity, and timestamp of the j-th event, respectively; when inputting event data into the network, a method of stacking event frames is used to combine a fixed number of events e. j Voxelization to E k ∈R C×H×W ; Three new modules—a style transfer module, an implicit / explicit consistency module, and an edge invariance module—are introduced to obtain better event domain depth estimation results. In the source domain, to reduce domain discrepancies, the source image is first style-transformed while retaining the original depth labels, resulting in… Subsequently, the processed image Feed into the image feature branch: Then feature f img The input to the subsequent image-supervised learning branch is fed into the shared decoder D. share The depth map is obtained, that is: D img =D share (f img ) (2) Then, it is trained using depth labels; in the target domain, the event voxel grid E is fed into the image reconstruction network to generate a reconstructed image, i.e.: I recon =E2VID(E) (3) Then I recon The image features are fed into the image feature branch to obtain the reconstructed image features, namely: f recon =E img (I recon ) (4) The event is also fed into the event feature branch, resulting in the event features, namely: f event =And event (E) (5) Finally, features from different modalities are input into the shared depth estimation head D. share It utilizes implicit and explicit consistency modules and edge-invariant modules to convey depth estimation knowledge in the image domain; 2) Style transfer module; The event-to-image reconstruction network E2VID is used, and the image transfer network CycleGAN is selected to transfer styles by understanding the similarities and differences between the two domains. Train a style transfer network G and use the pre-trained model G in the depth estimation training; The following are the style transfer training steps: First, input the event stream E into E2VID, and obtain the reconstructed image I. recon Used as a source domain; Secondly, the source image I i The source image domain is used as the target domain; then, style transfer of the source image domain is performed based on the pre-trained model G obtained from the CycleGAN training process. 3) Implicit and explicit consistency modules; Domain adaptation methods involve inputting data from different modalities into different encoders and using a shared decoder for depth estimation; By training an encoder and decoder in the image domain using depth labels, fine-grained information is extracted from the scene to generate higher-quality depth maps. An implicit-explicit consistency module is designed. The following implicit consistency loss is proposed to maintain the proximity between the event embeddings and the reconstructed image embeddings: Two embeddings f are obtained from encoders in different domains. event and f image Using L1 distance ensures implicit alignment of the embedding: An explicit consistency loss is proposed to explicitly align depth maps; that is: 4) Edge-invariant module; When inputting event features into the decoder to generate a depth map, edge loss is used. Using edge loss To constrain the depth map and image, thus helping the network generate edge-invariant estimates; that is: in and Let D represent the derivatives in the horizontal and vertical directions, respectively; img I recon It can be considered as a function of x and y; Each item retains more detail and structural information at the edges of the object, while reducing detail in smooth areas.

2. The event camera depth estimation method based on unsupervised domain adaptation according to claim 1, characterized in that: Steps 1 and 2 each include the following steps: Step 1.1: Process the datasets input from the image domain and event domain, and design an unsupervised domain to adapt to the overall framework; Step 1.2: Design a style transfer module and perform style transfer on the data in the image domain and event domain, retaining the data after style transfer; Step 1.3: Design an implicit and explicit consistency module, and input the style-transferred data into the unsupervised domain adaptation framework, using the implicit and explicit consistency module to constrain the generated features; Step 1.4: Design an edge-invariant module and use the edge-invariant module to constrain the depth map generated by the unsupervised domain adaptation framework; Step 2.1: Set up the training environment and training parameters for training; Step 2.2: Input the test dataset for testing.

3. The event camera depth estimation method based on unsupervised domain adaptation according to claim 1, characterized in that: Implemented using PyTorch (1.12.1) architecture on a single NVIDIA RTX 3080 Ti GPU, gradient descent is achieved using the Adam optimizer; the batch size for training is set to 8, and the learning rate is set to 10. -4 ; ResNet18 is used as the image encoder to extract feature maps at three scales for U-shaped connections with the decoder; the depth estimation decoder consists of three convolutional layers at different scales, a convolutional layer for feature fusion at different scales, and a final depth estimation head.

Images