Event camera depth estimation method based on unsupervised domain adaptation

Abstract

The invention discloses an event camera depth estimation method based on unsupervised domain adaptation, and is suitable for the technical field of event camera depth estimation in the field of computer vision. According to the method, the relation between the two modes is enhanced by comprehensively analyzing the difference between the image domain and the event domain and changing the image style. According to the method, the features in the implicit space can be more easily aligned, and the gap in the explicit space is reduced. Training is carried out in the marked image field, and then knowledge is transferred to unmarked event data; according to the method, knowledge is transferred from an image to an event by utilizing cross-domain feature alignment and depth alignment, style transfer is carried out by utilizing appearance, and the difference between different fields is minimized by utilizing edge invariant consistency. Through unsupervised domain adaptation, event monocular depth estimation is carried out only by using a depth label of an image.

Description

Technical Field

[0001] The present invention is applicable to the technical field of event camera depth estimation in the field of computer vision, and in particular to an event camera depth estimation method based on unsupervised domain adaptation. Background Art

[0002] With the rapid development of emerging applications such as autonomous driving and agile robot navigation, visual perception in high-speed motion scenes has become a serious challenge. Especially in variable outdoor conditions, these applications require smart devices to adapt to rapidly changing environments. As a new type of visual sensor, event cameras have been shown to have the advantages of low latency, robustness to illumination changes, and reduced motion blur. Compared with frame-based cameras, event cameras capture asynchronous data sequences and only report illumination changes at the pixel level. These asynchronous data can provide efficient scene perception in low-light and high-speed scenes. Therefore, studying these event-based perception tasks is crucial for various applications in high-speed motion scenes.

[0003] Among many event-based perception tasks, event-based depth estimation is an important visual perception task. It can perceive the depth information of the scene to improve the safety and stability of the automatic system in a high-speed environment. The so-called event camera depth estimation is to replace the dense image frame input with the sparse events generated by the event camera based on the traditional depth estimation method. The working principle of the event camera is different from that of the traditional camera. The event camera generates events by receiving information about brightness changes in the scene. It is not limited by a fixed frame rate and can output events as long as there is a brightness change in the scene. The event generated by the event camera can be expressed as e = (x, y, p, t), where x and y represent the horizontal and vertical coordinates of the pixels with brightness changes in the camera coordinate system; p represents the trend of brightness change, that is, p takes +1 when the brightness changes from low to high, and p takes -1 when the brightness changes from high to low; t represents the time when the brightness changes. Event camera depth estimation is to input this four-dimensional data into the model for pixel-by-pixel depth estimation.

[0004] However, the unique working principle of event cameras, while ensuring their robustness to environmental changes, also brings some challenges. Event cameras do not provide pixel information about the scene, but sparse and asynchronous perception of pixel changes, and it is difficult to estimate the dense depth of the scene based on sparse input alone. Therefore, a large number of researchers have proposed many methods for event camera depth estimation based on the characteristics of event cameras, which can be generally divided into the following three categories: methods based on camera parameter estimation, methods based on supervised learning, and methods based on unsupervised learning.

[0005] 1. Method based on camera parameter estimation

[0006] Early scholars used a model-based approach to perform depth estimation, that is, using the camera's parameters (including the camera's position and motion information) to estimate the optical flow and depth information of objects in the scene. A contrast maximization framework for event camera motion, depth, and optical flow estimation builds a model to solve event vision tasks by maximizing the contrast of distorted event images. Real-time 3D reconstruction based on multi-view event cameras estimates the three-dimensional structure of the scene based on event data and camera poses. Focus is everything: the loss function based on event vision analyzes event alignment in dynamic scenes and develops an optimization function for event vision. However, this type of method requires a large number of camera parameters, and the estimation results will be inaccurate due to the accuracy of parameter measurement and model optimization issues.

[0007] 2. Supervised learning-based methods

[0008] With the development of deep learning, many scholars have studied the application of deep learning in event cameras. The depth estimation method based on supervised learning refers to the use of labeled data sets to train deep learning models. Learning Monocular Dense Depth from Events (MDDE) shows how to use monocular vision to generate dense depth maps, overcoming the sparsity of event streams. Transfer learning for event-based task prediction through event-to-image conversion uses experience in the image domain to estimate the depth of the event domain. These works all use CNN as the backbone network for depth estimation, but the disadvantage is that the temporal information of the event camera is ignored. Many researchers have made new improvements in depth estimation performance by taking advantage of the modeling advantages of RNN and Transformer for temporal information. At the same time, with the pursuit of accuracy, the length and number of input event sequences are constantly increasing, which also increases information redundancy and hardware burden. Now there are many works that have conducted research on restoring event sparsity based on the previous ones, while reducing the model inference time while maintaining the previous accuracy requirements. Then, all these studies based on supervised learning require a large number of depth estimation labels for training. However, large-scale, high-quality event camera datasets are scarce, and the addition of pre-processed deep labels further increases the difficulty, which brings certain challenges to the promotion and use of the model.

[0009] 3. Methods based on unsupervised learning

[0010] Supervised methods rely on high-quality depth labels, while unsupervised methods do not require depth labels for training. Some unsupervised work also uses stereo data and reconstruction cues for unsupervised learning. Depth estimation based on unsupervised event depth stereo images uses stereo event frames for depth estimation and warps the stereo images to assist depth estimation. Unsupervised event camera optical flow, depth, and motion estimation reconstruct the current frame from the previous and next event frames in a monocular event video sequence. Obtaining large-scale stereo data and video sequences is often challenging. The large-scale stereo data and video sequences used by these methods also limit the use of some event datasets.

[0011] Although these existing methods have been successful in event-based depth estimation, they also have two drawbacks. First, the event input complexity is high. Large-scale event sequences with detailed labels are used for supervised training. In addition, some stereo sequences and a large number of images corresponding to event frames may also be used for training. Therefore, this method limits the generalization ability of the model. Second, the camera setting requirements are high. Some self-supervised methods use camera parameters to train the model, which limits the use of datasets and brings challenges to practical applications.

[0012] With the success of unsupervised domain adaptation methods in image depth estimation, as well as some attempts in the field of event camera semantic segmentation, such as Figure 1 As shown, the present invention proposes an unsupervised domain adaptation event camera depth estimation method that uses unsupervised domain adaptation technology to reduce the complexity of event data input, thereby achieving depth estimation based only on monocular events. This method greatly reduces the requirements for data and alleviates the problem of label scarcity in event camera research. Summary of the invention

[0013] This paper proposes an event camera depth estimation method based on unsupervised domain adaptation. The goal of the present invention is to generate high-density and high-quality depth estimation maps in an unsupervised manner. The method achieves this goal through two main tasks. First, the method starts with a comprehensive analysis of the differences between the image domain and the event domain. The goal is to strengthen the connection between the two modes by changing the image style. This method can make it easier to align features in the implicit space and narrow the gap in the explicit space. Secondly, the method uses an unsupervised domain adaptation method to train in the labeled image domain and then transfer knowledge to unlabeled event data. Using these techniques, the method aims to provide a novel unsupervised depth estimation method. The proposed method alleviates the impact of the lack of labels on the event camera depth estimation model. The main technical problems solved include the gap between the image domain and the event domain, and the alignment between the features of the two domains to improve the accuracy of depth estimation.

[0014] The event camera depth estimation method based on unsupervised domain adaptation proposed in the present invention can effectively alleviate the dependence on labels in supervised learning and the data set limitations of previous unsupervised learning. By introducing the style transfer module, the gap between the image domain and the event domain is effectively reduced to ensure that the performance degradation in domain adaptation is reduced; by introducing the implicit and explicit alignment module, the two domain knowledge is better aligned in features, improving the performance of the network; through the edge-invariant loss, the network can better integrate the edge contour information of the event frame, improving the accuracy of the model's depth estimation. Figure 2 The overall architecture diagram of unsupervised domain adaptation event camera depth estimation is given.

[0015] The present invention is achieved through the following technical solutions:

[0016] The input of the model proposed in this invention includes image domain and event domain (such as Figure 1 ), first send the image domain and event domain to their respective encoders to generate features, then use the loss function to constrain the features of different domains, and then input them into the shared decoder to output the depth estimation map for training. Then, only the event domain part is used during testing, that is, the event is input into the event encoder to generate features, and then input into the shared decoder to output the depth map. The specific steps are as follows:

[0017] Step 1: Build an event camera depth estimation network model based on unsupervised domain adaptation

[0018] 1. Training data set input and overall structure design

[0019] The training dataset includes the source domain KITTI and the target domain Outdoor_day2 in MVSEC, Town1 to Town5 in DENSE. The input process will be described in detail below. is a dense image I i and the corresponding depth map D i The target domain is an event stream of M events, where denote the position, polarity and timestamp of the jth event respectively. When inputting event data into the network, we use the stacking event frame method to stack a fixed number of events e j Voxelization to E k ∈R C×H×W Our goal is to transfer the knowledge of the source image domain with depth labels to the target event domain without labels through domain adaptation. We design three new modules to obtain better event domain depth estimation results. Figure 2 As shown, the training and implementation of the present invention is divided into several steps. In the source domain, in order to reduce the domain gap, we first perform style transfer on the source image while retaining the original depth label, and obtain Then, the processed image Sent into the image feature branch:

[0020]

[0021] Then the feature f img Input the subsequent image supervision learning branch and send it to the shared decoder D share Get the depth map, that is:

[0022] D img =D share (f img ) (2)

[0023] It is then trained using the deep labels. In the target domain, the event voxel grid E is fed into the image reconstruction network to generate a reconstructed image, namely:

[0024] I recon =E2VID(E) (3)

[0025] Then I recon Send it into the image feature branch to obtain the reconstructed image features, namely:

[0026] f recon =E img (I recon ) (4)

[0027] The event is also sent to the event feature branch to obtain the event feature, namely:

[0028] f event =E event (E) (5)

[0029] Finally, features from different modalities are fed into a shared depth estimation head D share , using implicit explicit consistency modules and edge invariant modules to transfer depth estimation knowledge in the image domain. Below we will introduce each module in detail.

[0030] 2. Design style migration module

[0031] To improve the domain adaptation performance, we adopted the event-to-image reconstruction network E2VID to make the two domains as close as possible. However, due to the sparsity of events, the reconstructed image may have problems such as color imbalance and blur. Reducing these areas to improve the reconstruction quality may have a positive impact on the migration between different domains. However, this is part of the event-to-image reconstruction task and has nothing to do with depth estimation. We found that this blur often appears on both sides of the image or inside the object. Therefore, we do not focus on optimizing the quality of the generated image, but consider the blurred image as a specific image style. At the same time, we use the style transfer network to apply this style to the image domain. This can reduce the impact of images of different styles on depth estimation. Since this is an unpaired image style transfer, we chose the image transfer network CycleGAN to transfer the style by understanding the similarities and differences between the two domains. We train a style transfer network G and use the pre-trained model G to participate in the depth estimation training in the method of the present invention. The following are several steps of style transfer training. First, the event stream E is input into E2VID, and the reconstructed image I recon Second, the source image I i As the target domain, the style of the source image domain is then transferred based on the pre-trained model G obtained through the training process of CycleGAN.

[0032] 3. Design implicit and explicit consistency modules

[0033] Our domain adaptation approach consists of feeding data from different modalities into different encoders and using a shared decoder for depth estimation. By training the encoder and decoder in the image domain with depth labels, we can obtain fine-grained information from the scene and generate higher quality depth maps. Therefore, our goal is to align the event modality with the feature maps and depth maps of the image modality. In this way, knowledge from the image domain can be learned implicitly and explicitly through a sequence of events and generated images. Figure 2 As shown in Figure 2, we design an implicit explicit consistency module. We propose the following implicit consistency loss to keep the embedding of the event and the embedding of the reconstructed image close:

[0034] Two embeddings f are obtained from encoders in different domains event and f image , using L1 distance to ensure that the embeddings are implicitly aligned:

[0035]

[0036] Similarly, we propose an explicit consistency loss to explicitly align depth maps. That is:

[0037]

[0038] 4. Design edge-invariant module

[0039] We hope that the present invention is robust to input perturbations or noise during the inference phase. When perturbations occur, the network can stably perform the prediction task. When events of different scales are input to the network, the resulting features and reconstructed images are different, but the gradient of the overall image will only change slightly. Therefore, when the event features are input to the decoder to generate the depth map, we use the edge loss We use the margin loss to constrain the depth map and image to help the network generate edge-invariant estimates. That is:

[0040]

[0041] Where D img , I recon is considered as a function of x and y, and Represents the horizontal and vertical derivatives of the above function respectively. Using edge loss It helps preserve more details and structural information at the edges of objects, while reducing details in smooth areas.

[0042] Step 2: Adjust network parameters, train and test the neural network

[0043] 1. Parameter settings

[0044] This paper adopts Pytorch (1.12.1) architecture, implemented on a single NVIDIARTX3080ti GPU, and uses Adam optimizer to implement gradient descent. The batch size of training is set to 8, and the learning rate is set to 10 -4 We use ResNet18 as the image encoder, from which we extract feature maps of three scales for U-connection with the decoder. The depth estimation decoder consists of three convolutional layers of different scales, a convolutional layer for feature fusion of different scales, and a final depth estimation head.

[0045] 2. Input test data set

[0046] For the MVSEC dataset, Outdoor_day1 and Outdoor_night1 are used as test datasets; for the DENSE dataset, Town10 is used as the test set. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is the main flow chart of unsupervised domain adaptation event camera depth estimation.

[0048] Figure 2This is the overall framework diagram of depth estimation based on unsupervised domain adaptation event camera.

[0049] Figure 3 is a high-quality schematic diagram of the results. DETAILED DESCRIPTION

[0050] The present invention experimentally verifies the above method and evaluates the effect of depth estimation on two datasets, MVSEC and DENSE.

[0051] 1. Qualitative evaluation

[0052] 1) Cross-domain performance evaluation: We compare our invention with several depth estimation methods that perform well in outdoor scenes to demonstrate our cross-domain performance. We select different types of methods that can produce excellent performance in depth estimation, including domain adaptation methods: 3DPL, GASDA; supervised methods: NewCRFS; self-supervised methods: NewCRFS; self-supervised methods: Litemono and generalization methods: S2R-DepthNet. We verify the performance of these methods using a training method similar to ours, which includes training on the labeled image domain and then testing on the event domain. To address the model's adaptability to different modality inputs, we use E2VID to complete the reconstruction from events to images. In the test phase, we input the event and then pass it to the encoder and depth estimation decoder to generate a depth map. Table 1 lists our comprehensive evaluation results. These results show that our designed method narrows the gap between different domains. In event monocular depth estimation, domain adaptability has been significantly improved.

[0053] 2) Evaluation of depth estimation on MVSEC dataset: To demonstrate the effectiveness of our method, we compared it with several event depth estimation methods on MVSEC. Currently, open source unsupervised event depth is very limited. E2VID can complete the conversion from event domain to image domain, and it is possible to perform depth estimation in an unsupervised setting. ESS as an unsupervised semantic segmentation model and depth estimation also have relevant roles. Table 2 shows that our method achieves excellent results in all indicators compared to other methods. These results show that we can obtain competitive results even without using event labels. Figure 3 Qualitative results show that we achieve superior performance on object outlines and overall deep representation of scenes in an unsupervised setting.

[0054] 3) Depth Estimation Evaluation on DENSE Dataset: To further verify the effectiveness of our proposed method, we show more results on the DENSE dataset. When training on the DENSE dataset, we fine-tune E2VID on DENSE using L1 loss. All other design modules and hyperparameters remain unchanged. Table 3 shows the relevant performance comparison. The results show that in the unsupervised setting of event monocular depth estimation, our metrics are competitive on different datasets. This shows that our method can adapt to different datasets even without using the corresponding depth labels.

[0055] Table 1: Comparison of cross-domain depth estimation performance in MVSEC with other cross-domain methods

[0056]

[0057]

[0058] Table 2: Depth estimation performance comparison with other depth estimation methods in MVSEC

[0059]

[0060] Table 3: Comparison of depth estimation performance in DENSE with other depth estimation methods

[0061]

[0062] 2. Subjective effect

[0063] Figure 3 The figure shows the comparison between the test results of the present invention and other methods on MVSEC. It can be clearly seen from the white circles marked in the image that the present invention is superior to other methods in terms of scene object contours and depth range. For example, in the fourth row, the left vehicle and the right railing can be clearly seen in the reconstructed image, which are relatively blurred in other methods, but the depth of the vehicle and the railing can be clearly seen in the prediction of the present invention.

Claims

1. An event camera depth estimation method based on unsupervised domain adaptation, characterized by: Step 1: First, use the style transfer module to transfer the style of the image domain and event domain, then send the processed image domain and event domain into their respective encoders to generate features, then use the implicit and explicit consistency module and the edge invariant module to constrain the features of different domains, and then input them into the shared decoder to output the depth estimation map for training; Step 2: During testing, only the event domain is used, that is, the event is input into the event encoder to generate features, and then input into the shared decoder to output the depth map.

2. The event camera depth estimation method based on unsupervised domain adaptation according to claim 1, characterized in that: The steps 1 and 2 respectively include the following steps: Step 1.1: Process the datasets of image domain and event domain inputs and design the overall framework of unsupervised domain adaptation; Step 1.2: Design a style transfer module, transfer the style of the data in the image domain and the event domain, and retain the data after style transfer; Step 1.3: Design an implicit and explicit consistency module, input the style-transferred data into the unsupervised domain adaptation framework, and use the implicit and explicit consistency module to constrain the generated features. Step 1.4, design an edge-invariant module, and use the edge-invariant module to constrain the depth map generated by the unsupervised domain adaptation framework; Step 2.1, set the training environment and training parameters for training; Step 2.2: Input the test data set for testing.

3. The method for estimating human posture based on a conditional dual-branch diffusion model according to claim 1, characterized in that: 1) Unsupervised Adaptation Framework: Setting the Source Domain is a dense image I i and the corresponding depth map D i The target domain is an event stream of M events, where They represent the position, polarity and timestamp of the jth event respectively; when inputting event data into the network, the stacked event frame method is used to stack a fixed number of events e j Voxelization to E k ∈R C×H×W ; Design three new modules to obtain better event domain depth estimation results; In the source domain, in order to reduce the domain gap, first perform style transfer on the source image while retaining the original depth label, and obtain Then, the processed image Sent into the image feature branch: Then the feature f img Input the subsequent image supervision learning branch and send it to the shared decoder D share Get the depth map, that is: D img =D share (f img ) (2) It is then trained using the deep labels; in the target domain, the event voxel grid E is fed into the image reconstruction network to generate a reconstructed image, namely: I recon =E2VID(E) (3) Then I recon Send it into the image feature branch to obtain the reconstructed image features, namely: f recon =E img (I recon ) (4) The event is also sent to the event feature branch to obtain the event feature, namely: f event =And event (E) (5) Finally, features from different modalities are fed into a shared depth estimation head D share , using implicit and explicit consistency modules and edge invariant modules to transfer depth estimation knowledge in the image domain; 2) Style transfer module; We use the event-to-image reconstruction network E2VID and the image transfer network CycleGAN to transfer the style by understanding the similarities and differences between the two domains. Train a style transfer network G and use the pre-trained model G to participate in depth estimation training; The following are the steps for style transfer training: First, the event stream E is input into E2VID, and the reconstructed image I is obtained. recon Used as source domain; Secondly, the source image I i As the target domain; then perform style conversion on the source image domain based on the pre-trained model G obtained through the training process of CycleGAN; 3) Implicit and explicit consistency module; Domain adaptation methods involve feeding data from different modalities into different encoders and using a shared decoder for depth estimation; By using depth labels to train the encoder and decoder in the image domain, we can obtain fine-grained information from the scene and generate higher quality depth maps. We design an implicit explicit consistency module and propose the following implicit consistency loss to keep the embedding of the event and the embedding of the reconstructed image close: Two embeddings f are obtained from encoders in different domains event and f image , using L1 distance to ensure that the embeddings are implicitly aligned: An explicit consistency loss is proposed to explicitly align the depth map; namely: 4) Edge invariant module; When inputting event features into the decoder to generate depth maps, the edge loss is used Using edge loss to constrain the depth map and image to help the network generate edge-invariant estimates; namely: in and Represent the derivatives in the horizontal and vertical directions respectively; D img , I recon Considered as a function of x and y; Each term of preserves more details and structural information at the edges of objects, while reducing details in smooth areas.

4. The method for 3D hand posture estimation based on spatial perception and structural modeling according to claim 1, characterized in that: The Pytorch (1.12.1) architecture is used and implemented on a single NVIDIA RTX3080ti GPU. The Adam optimizer is used for gradient descent. The batch size of the training is set to 8 and the learning rate is set to 10. -4 ; ResNet18 is used as the image encoder, from which feature maps of three scales are extracted for U-connection with the decoder; the depth estimation decoder consists of three convolutional layers of different scales, a convolutional layer for fusion of features of different scales, and a final depth estimation head.

Images