Model training method and related equipment

Abstract

The embodiment of the invention provides a model training method and related equipment, and relates to the field of AI. In the method, a sample set is obtained, the sample set comprises a first image and event data stream data, the event data stream data is obtained according to an event data stream shot by an event camera within a preset time length, and the first image is shot by a common camera at an end moment of the preset time length. And training the initial depth estimation model according to the first image and the event data stream data to obtain a target depth estimation model. In the scheme, the self-supervised training of the depth estimation model is realized by using the image shot by the common camera and the event data stream of the event camera, the model training can be completed without the help of a truth value depth map with high cost, and the training cost of the depth estimation model can be effectively reduced.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of images, and in particular to a model training method and related equipment. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0003] Image analysis is one of the important tasks in computer vision, among which image depth can be obtained through image analysis. Image depth refers to the depth of the scene in the image, that is, the vertical distance from each pixel in the scene to the camera imaging plane. Obtaining depth information in the scene can help better understand the scene. Therefore, image depth estimation has a wide range of applications in virtual reality, augmented reality, semantic segmentation, autonomous driving, and 3D reconstruction.

[0004] In the image depth estimation method based on deep learning, a recurrent convolutional neural network is used for supervised learning, which requires the help of a true depth map. At this time, a lidar is needed to obtain the true depth map, but the cost of the lidar is very high. Summary of the invention

[0005] The present application provides a model training method and related equipment, which can complete model training without the need for a true depth map obtained by a lidar, and can effectively reduce the training cost of the depth estimation model.

[0006] In a first aspect, a model training method is provided, which can be executed by a model training device or by a chip in the model training device.

[0007] The model training method comprises the following steps: obtaining a sample set. The sample set comprises a first image and event data stream data, wherein the event data stream data is obtained according to a representation matrix of an event data stream captured by an event camera for a preset time length, and the first image is captured by an ordinary camera at the end of the preset time length. The initial depth estimation model is trained according to the first image and the event data stream data to obtain a target depth estimation model.

[0008] In this scheme, the images taken by an ordinary camera and the event data stream of the event camera are used to realize self-supervised training of the depth estimation model, wherein the event data stream of a preset time length is represented as event data stream data for model training. The model training can be completed without the aid of expensive true depth maps, which can effectively reduce the training cost of the depth estimation model.

[0009] In a possible implementation of the first aspect, the above-mentioned training of the initial depth estimation model according to the first image and the event data stream data to obtain the target depth estimation model specifically includes the following steps: processing the event data stream data through the initial depth estimation model to obtain the first depth estimation data. Determining a first loss value of the first image according to the first image and the first depth estimation data. Adjusting and optimizing the initial depth estimation model according to the first loss value to obtain the target depth estimation model.

[0010] Exemplarily, the first loss value is calculated based on the minimum photometric loss function and the mask algorithm. When calculating the first loss value, the influence of the occluded area is weakened by the minimum photometric loss function, and then the relatively static pixels are removed by the mask algorithm. The optimization direction of the initial depth estimation model is determined based on the first loss value to obtain the target depth estimation model.

[0011] In a possible implementation of the first aspect, the first loss value of the first image is determined according to the first image and the first depth estimation data, specifically comprising the following steps: calculating the second loss value according to the first photometric error and the second photometric error. The first photometric error is calculated according to the first image, the previous frame of the first image, and the first depth estimation data; the second photometric error is calculated according to the first image, the next frame of the first image, and the first depth estimation data. The mask of the first image is calculated according to the first photometric error, the second photometric error, the third photometric error, and the fourth photometric error. The third photometric error is obtained according to the first image and the previous frame of the first image; the fourth photometric error is obtained according to the first image and the next frame of the first image. The first loss value is determined according to the second loss value and the mask.

[0012] In this solution, the photometric error between the first image and its previous frame image, i.e., the first photometric error, is determined based on the first depth estimation data, and the photometric error between the first image and its subsequent frame image, i.e., the second photometric error, is determined based on the first depth estimation data. The second loss value can be obtained using the first photometric error and the second photometric error. In addition, the mask of the first image is determined using the third photometric error between the first image and its previous frame image and the fourth photometric error between the first image and its subsequent frame image; and then the first loss value is obtained based on the second loss value and the above mask.

[0013] In a possible implementation of the first aspect, the first loss value of the first image is determined based on the first image and the first depth estimation data, and further includes the following steps: determining a first photometric error based on the first image, the first pose estimation data, and the first depth estimation data. The first pose estimation data is the camera pose transformation data between the first image and the previous frame of the first image. The second photometric error is determined based on the first image, the second pose estimation data, and the first depth estimation data. The second pose estimation data is the camera pose transformation data between the first image and the next frame of the first image.

[0014] In a possible implementation of the first aspect, the determining of the first photometric error according to the first image, the first pose estimation data, and the first depth estimation data specifically includes the following steps: determining a reconstructed image according to the first image, the first pose estimation data, and the first depth estimation data. Determining the first photometric error according to the first image, the reconstructed image, and the similarity between the first image and the reconstructed image.

[0015] It can be seen that in this solution, the first image, the first pose estimation data and the first depth estimation data are first used to reconstruct the first image to obtain a reconstructed image, and then the first photometric error is determined according to the first image, the reconstructed image and the similarity between the first image and the reconstructed image.

[0016] In a possible implementation of the first aspect, the above-mentioned processing of the event data stream data by the initial depth estimation model to obtain the first depth estimation data specifically includes the following steps: extracting features from the event data stream data to obtain feature maps of i scales. The above-mentioned i is an integer greater than or equal to three. The feature maps of the i scales are fused and decoded to obtain a depth map of at least one scale. Determine the depth estimation data of at least one scale of the event data stream data based on the depth map of at least one scale. The scale is the size.

[0017] It can be seen that in this solution, multi-scale feature extraction is performed on the event data stream data to obtain feature maps of multiple scales, and the first depth estimation data can be obtained based on the feature maps of multiple scales. Among them, the feature maps of multiple scales are equivalent to feature maps of multiple resolutions. In this way, feature information under different receptive fields can be obtained, which helps to improve the accuracy of depth estimation.

[0018] In a possible implementation manner of the first aspect, when the depth map of the at least one scale is a depth map of i-1 scales, the fusing and decoding processing of the feature maps of the i scales to obtain the depth map of the at least one scale specifically includes the following steps:

[0019] The feature maps of i scales are fused i times to obtain fused feature maps of i scales. In the i-th fusion process, the fused feature map obtained by the i-1 fusion process is upsampled to obtain the fused feature map of the i-th fusion process. In the i-1 fusion process, the fused feature map obtained by the i-2 fusion process and the first feature map are fused to obtain the fused feature map of the i-1 fusion process. The first feature map is the feature map with the highest resolution among the feature maps of i scales. In the i-2 fusion process, the fused feature map, the first feature map, and the second feature map obtained by the i-3 fusion process are fused to obtain the fused feature map of the i-2 fusion process. The second feature map is the feature map with the second highest resolution among the feature maps of i scales. And so on, until in the first fusion process, the feature maps of i scales are fused to obtain the fused feature map of the first fusion process.

[0020] The fused feature maps of i-1 scales are decoded respectively to obtain depth maps of i-1 scales. The fused feature maps of i-1 scales are the fused feature maps of the fused feature maps of i scales except the fused feature map with the smallest feature map scale.

[0021] It can be seen that in this scheme, i-time fusion processing is used to realize the fusion of multi-scale feature maps to obtain i-scale fused feature maps, which is beneficial to improving the accuracy of the depth map obtained based on the fused feature map.

[0022] In a possible implementation of the first aspect, in the above-mentioned i-1th fusion process, the fusion feature map obtained by the i-2th fusion process and the first feature map are fused to obtain the fusion feature map of the i-1th fusion process, which specifically includes the following steps: upsampling the fusion feature map obtained by the i-2th fusion process to obtain the first upsampled feature map. Concatenating the first upsampled feature map and the first feature map to obtain the fusion feature map of the i-1th fusion process.

[0023] It can be seen that in this scheme, since the scale of the fused feature map obtained by the i-2th fusion processing is smaller than the scale of the first feature map, it is necessary to first upsample the fused feature map obtained by the i-2th fusion processing to obtain the first upsampled feature map, and then splice the first upsampled feature map and the first feature map to obtain the fused feature map of the i-1th fusion processing.

[0024] In a possible implementation of the first aspect, in the above-mentioned i-2nd fusion process, the fusion feature map obtained by the i-3rd fusion process, the first feature map, and the second feature map are fused to obtain the fusion feature map of the i-2nd fusion process, which specifically includes the following steps: upsampling the fusion feature map obtained by the i-3rd fusion process to obtain a second upsampled feature map. Downsampling the first feature map to obtain a downsampled feature map. Concatenating the second upsampled feature map, the downsampled feature map, and the second feature map to obtain the fusion feature map of the i-2nd fusion process.

[0025] It can be seen that in this solution, since the scale of the fused feature map obtained by the i-3 fusion process is smaller than the scale of the second feature map, the fused feature map obtained by the i-3 fusion process is upsampled to obtain the second upsampled feature map. In addition, since the scale of the first feature map is larger than the scale of the second feature map, the first feature map is downsampled to obtain the downsampled feature map, and then the second upsampled feature map, the first feature map and the downsampled feature map are spliced ​​to obtain the fused feature map of the i-2 fusion process.

[0026] In a second aspect, the present application further provides a depth estimation method, comprising the following steps: obtaining event data stream data, the event data stream data being obtained based on an event data stream captured by an event camera for a preset time length. Processing the event data stream data through a target depth estimation model to obtain depth estimation data for the event data stream data, the target depth estimation model being a model trained based on the model training method of the first aspect.

[0027] It can be seen that this solution can obtain depth estimation data by using the event data stream of the preset time length and the target depth estimation model. Among them, the target depth estimation model is trained based on the model training method of the first aspect, and the model training cost is low, which is conducive to reducing the cost of depth estimation.

[0028] In a possible implementation of the second aspect, the above-mentioned event data stream data is processed by a target depth estimation model to obtain depth estimation data of the above-mentioned event data stream data, which specifically includes the following steps: processing the above-mentioned event data stream data by a target depth estimation model to obtain a depth map of at least one scale of the event data stream data; determining the above-mentioned depth estimation data based on the depth map of at least one scale.

[0029] It can be seen that in this solution, specific depth estimation data can be obtained based on the depth map to achieve depth estimation.

[0030] In a third aspect, the present application also provides a model training device, including an acquisition module and a training module.

[0031] The acquisition module is used to acquire a sample set. The sample set includes a first image and event data stream data, wherein the event data stream data is obtained according to an event data stream captured by an event camera for a preset time length, and the first image is captured by a common camera at the end of the preset time length.

[0032] A training module is used to train an initial depth estimation model according to the first image and the event data stream data to obtain a target depth estimation model.

[0033] In this scheme, the model training device uses images taken by an ordinary camera and the event data stream of an event camera to realize self-supervised training of the depth estimation model, wherein the event data stream of a preset time length is represented as event data stream data for model training; model training can be completed without the aid of costly true depth maps, which can effectively reduce the training cost of the depth estimation model.

[0034] In a fourth aspect, the present application also provides a depth estimation device, including an acquisition module and a processing module.

[0035] The acquisition module is used to acquire event data stream data, wherein the event data stream data is obtained based on the event data stream captured by the event camera with a preset time length.

[0036] The processing module is used to process the event data stream data through the target depth estimation model to obtain depth estimation data of the event data stream data. The target depth estimation model is a model trained based on the model training method of the first aspect.

[0037] It can be seen that the depth estimation device of the present solution can obtain depth estimation data by using the event data stream of the preset time length and the target depth estimation model. Among them, the target depth estimation model is trained based on the model training method of the first aspect, and the model training cost is low, which is conducive to reducing the cost of depth estimation.

[0038] In a fifth aspect, the present application also provides a perception system, comprising the depth estimation device as described in the fourth aspect.

[0039] In a sixth aspect, the present application also provides a vehicle, comprising the depth estimation device as described in the fourth aspect or the perception system as described in the fifth aspect.

[0040] In a seventh aspect, the present application also provides an electronic device, comprising a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute the method described in the first aspect or the second aspect.

[0041] In an eighth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method as described in the first aspect or the second aspect.

[0042] In a ninth aspect, the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect or the second aspect.

[0043] In the tenth aspect, the present application also provides a chip, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method described in the first aspect or the second aspect.

[0044] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method described in the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The following is an introduction to the drawings used in the embodiments of the present application.

[0046] Figure 1A is a schematic diagram of a calibration plate provided in an embodiment of the present application;

[0047] Figure 1B This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0048] Figure 1C It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0049] Figure 1D is a schematic diagram of a depth estimation model provided in an embodiment of the present application;

[0050] Figure 1Eis another schematic diagram of a depth estimation model provided in an embodiment of the present application;

[0051] Figure 2 It is a flowchart of a model training method provided in an embodiment of the present application;

[0052] Figure 3 is a schematic diagram of aligning a first image and event data stream data provided by an embodiment of the present application;

[0053] Figure 4A is a schematic diagram of a process for generating first depth estimation data provided by an embodiment of the present application;

[0054] Figure 4B is another schematic diagram of a process for generating first depth estimation data provided by an embodiment of the present application;

[0055] Figure 4C is another schematic diagram of a process for generating first depth estimation data provided by an embodiment of the present application;

[0056] Figure 5A is a structural schematic diagram of a second fusion module provided in an embodiment of the present application;

[0057] Figure 5B is a structural schematic diagram of a third fusion module provided in an embodiment of the present application;

[0058] Figure 5C is a schematic diagram of the structure of another third fusion module provided in an embodiment of the present application;

[0059] Figure 5D is a structural schematic diagram of a first fusion module provided in an embodiment of the present application;

[0060] Figure 5E is a structural diagram of a first decoding module provided in an embodiment of the present application;

[0061] Fig. 6A is a schematic diagram of posture estimation provided by an embodiment of the present application;

[0062] Figure 6B is a schematic diagram of image reconstruction provided by an embodiment of the present application;

[0063] Figure 6C is a schematic diagram of a first image provided in an embodiment of the present application;

[0064] Fig.6D It is a schematic diagram of the effect of a mask algorithm provided in an embodiment of the present application;

[0065] Figure 7 is a flowchart of a depth estimation method provided in an embodiment of the present application;

[0066] Figure 8 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0067] Fig. 9 is a structural schematic diagram of a depth estimation device provided in an embodiment of the present application;

[0068] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0070] Due to the applications involved in the embodiments of the present application, in order to facilitate understanding, the relevant terms and other related concepts involved in the embodiments of the present application are first introduced below.

[0071] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0072] The "at least one" mentioned in the embodiments of the present application refers to one or more, and "multiple" refers to two or more. "At least one of the following" or its similar expression refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, c can be single or multiple. "And / or" describes the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can be represented by: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are an "or" relationship. The serial numbers of the steps of the embodiments of the present application (such as step S1, step S21, etc.) are only for distinguishing different steps, and do not limit the order of execution between the steps.

[0073] Furthermore, unless otherwise specified, the ordinal numbers such as "first" and "second" used in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects. For example, the first device and the second device are only for the convenience of description, and do not represent the difference in structure, importance, etc. between the first device and the second device. In some embodiments, the first device and the second device can also be the same device.

[0074] In the above embodiments, the term "when..." can be interpreted as meaning "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modification, equivalent substitution, improvement, etc. made within the concept and principle of the present application shall be included in the protection scope of the present application.

[0075] (1) Neural Network

[0076] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be:

[0077]

[0078] Where, s = 1, 2, ... d, d is a natural number greater than 1, W s For x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a Sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0079] (2) Deep Neural Networks

[0080] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0081] In summary, the coefficients from the nth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0082] (3) Convolutional Neural Network

[0083] Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (Feature Map). Convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a certain part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0084] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0085] (4) Loss function

[0086] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (Loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this Loss as much as possible.

[0087] (5) Event Camera

[0088] Event cameras are bio-inspired visual sensors that work in a completely different way from standard cameras. Instead of outputting intensity image frames at a constant rate, event cameras only output information about local pixel-level brightness changes. When these pixel-level brightness changes (called events) exceed a set threshold, the event camera timestamps them with microsecond resolution and outputs an asynchronous event stream.

[0089] Specifically, each individual event e i =(u i ,t i ,p i ) records the brightness change of a pixel, including the pixel coordinates u i , timestamp t i , brightness change polarity p i .

[0090] (6) Spatiotemporal Voxel

[0091] The space-time voxel is used to represent the event data stream {e0,e1,...,e i ,...}, that is, the space-time voxel is a representation matrix of the event data stream captured by the event camera with a preset time length. Specifically, the event stream within the preset time length ΔT is represented by a space-time voxel E with a size of B×H×W kIt is represented by converting the event data stream of the preset time length into a matrix of fixed size. Among them, B is the number of channels, which means that the preset time length ΔT is divided into B time periods; for example, the preset time length ΔT is 50 milliseconds, B is 5, then 0 to 10 milliseconds is one time period, 10 milliseconds to 20 milliseconds is another time period, and so on, 5 time periods can be obtained. H is the image height; W is the image width. Spatiotemporal voxel E k The value of each position on can be expressed as:

[0092]

[0093] The above formula (1) represents the event flow of a pixel point in each time period in the space-time voxel E k The value represented by u k Represents the pixel coordinates of a pixel point in the H×W image plane, for example, the coordinates (0, 0). n ∈{0,1,...,B-1} represents the first dimension of the space-time voxel, that is, the time period mentioned above, for example, t n When it is 0, then equation (1) represents the event data flow of the pixel point coordinate (0, 0) in the time period from 0 to 10 milliseconds in the space-time voxel E k The value in p i is the brightness change polarity, It is event i t0 is the normalized timestamp of the space-time voxel, and t0 is the initial timestamp of the space-time voxel.

[0094] (7) Camera calibration

[0095] In the image measurement process and machine vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are camera parameters. Distortion coefficient is one of the camera parameters, corresponding to the camera distortion phenomenon. Under most conditions, these camera parameters must be obtained through experiments and calculations. The process of solving the parameters is called camera calibration (or camera calibration).

[0096] When calibrating the camera parameters, Figure 1A The calibration plate shown is placed in the camera field of view, and the camera position is changed to capture images from multiple perspectives (for ordinary cameras, it is sufficient to capture images in a stationary state, and for event cameras, it is necessary to capture images when the camera is moving). Since the grid size of the calibration plate is known, the optimal camera intrinsic parameters can be estimated by detecting feature points. The calibration process can be completed by the camera calibration toolbox of Matrix & Laboratory (MATLAB).

[0097] Camera distortion includes radial distortion and tangential distortion. Radial distortion is caused by the shape of the lens; more specifically, in the pinhole model, a straight line is still a straight line when projected onto the pixel plane. However, in actual photos, the camera lens often turns a straight line in the real environment into a curve in the picture. This distortion is called radial distortion.

[0098] Tangential distortion is caused by the failure to make the lens and the imaging surface strictly parallel during the camera assembly process. Due to the inconsistency between the actual object and the image projected onto the 2D plane caused by light projection, this inconsistency is stable and can be corrected for subsequent images by calculating the distortion parameters through camera calibration.

[0099] (8) Depth Estimation

[0100] Depth estimation generally refers to the process of using a 2D image projected from a 3D object to determine the actual distance between each pixel in the 2D image and the camera, that is, the process of determining the depth estimation information of each pixel. In addition, depth estimation can also refer to the process of determining the depth map corresponding to the 2D image, and the depth map refers to the depth feature map. Furthermore, the depth estimation information of each pixel can be obtained based on the depth map.

[0101] Depth estimation methods include direct acquisition method, multi-lens multi-image estimation method, binocular depth estimation method, monocular depth estimation method, etc. Among them, the direct acquisition method uses the reflection of laser radar or structured light on the surface of the object to obtain the depth point cloud, which can directly obtain the depth information of each pixel. The multi-lens multi-image estimation method refers to the use of stereo matching technology, using triangulation to convert the parallax information between two images into depth information, and estimating the scene depth information from the image. The binocular depth estimation method refers to the use of two image sensor-based cameras or two images to estimate the depth information. The monocular depth estimation method refers to the use of a single image sensor-based camera or a single image to estimate the depth information. In addition, with the development of deep learning technology, depth estimation methods also include methods for achieving depth estimation based on deep learning.

[0102] In the image depth estimation method based on deep learning, a recurrent convolutional neural network is used for supervised learning, which requires the help of a true depth map. At this time, a lidar is needed to obtain the true depth map, but the cost of the lidar is very high.

[0103] Therefore, the embodiment of the present application provides a model training method, which can realize self-supervised training of the depth estimation model based on the images of ordinary cameras and the event data streams of event cameras. The model training can be completed without the aid of expensive true depth maps, which can effectively reduce the cost of model training.

[0104] The model training method of the embodiment of the present application can be executed by a model training device, or by a chip in the model training device.

[0105] Further, after the depth estimation model is obtained based on the model training method of the embodiment of the present application, the event data stream or event data stream data is input into the depth estimation model to obtain the corresponding depth map or depth estimation data. Therefore, the present application also provides a depth estimation method that can achieve depth estimation. The depth estimation method of the present application can be applied to any scene that requires depth estimation, such as assisted driving scenes, autonomous driving scenes, AR, VR, robots or drones, etc., without limitation.

[0106] For example, reference Figure 1B , Figure 1B It is a schematic diagram of an application scenario provided by an embodiment of the present application, wherein, when the depth estimation method of the embodiment of the present application is applied to an autonomous driving scenario, it is specifically applied to a sensor perception module in an autonomous driving system. Figure 1B In the intelligent driving vehicle system, the sensor perception module receives environmental information and performs information perception. The sensor perception module sends the perceived information (depth map or depth estimation data) to the vehicle perception and positioning module to achieve vehicle perception and positioning. Subsequently, the scene understanding and behavior prediction modules recognize the driving scene and predict the behavior based on the vehicle perception and positioning results, and input the recognition and prediction results into the path planning module for processing to obtain the path planning results. Finally, the vehicle bottom control module realizes the final vehicle control based on the above path planning results.

[0107] The depth estimation method of the embodiment of the present application is applied to the sensor perception module, that is, the module can obtain a dense monocular depth map or depth estimation data through the perception layer information (that is, the event data stream output by the event camera) and the depth estimation model.

[0108] The following introduces a system architecture provided by an embodiment of the present application.

[0109] See attached Figure 1C , an embodiment of the present application provides a system architecture 100. As shown in the system architecture 100, a data acquisition device 160 is used to collect training data for a depth estimation model. In this embodiment, the training data is an image and an event data stream. The data acquisition device 160 stores the training data in a database 130. The image and the event data stream have the same shooting object.

[0110] The training device 120 can train the initial depth estimation model based on the training data maintained in the database 130 to obtain the depth estimation model 101. The specific model training process can refer to the following model training method (such as Figure 2 ) is not described in detail here. Exemplarily, in an embodiment of the present application, after completing the model training, the depth estimation model 101 can obtain the ability to perform depth estimation based on the event data stream or the event data stream data. The training device 120 can be a server or a cloud service device, etc., and can also be a mobile phone terminal, a tablet computer, a laptop computer, AR / VR, a vehicle-mounted terminal, a monitoring device, a vehicle-mounted automatic driving system, a drone and other devices.

[0111] It should be noted that, in actual applications, the training data maintained in the database 130 may not all come from the data acquisition device 160, but may also be received from other devices, such as cameras based on image sensors, event cameras, etc. It should also be noted that the training device 120 may not necessarily perform model training entirely based on the training data maintained by the database 130, but may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0112] The depth estimation model 101 obtained after processing by the training device 120 can be applied to a system or device deployed in a terminal, such as Figure 1C The terminal device 110 shown in FIG. 1 may process the event data stream data based on the depth estimation model 101 to obtain a corresponding depth map or depth estimation data (eg, Figure 1D Alternatively, the terminal device 110 may process the event data stream based on the depth estimation model 101 to obtain a corresponding depth map or depth estimation data (such as Figure 1E The terminal device 110 may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, AR / VR, a vehicle terminal, a vehicle computing platform, a vehicle domain controller, a monitoring device, a vehicle automatic driving system, a drone, etc. It may also be a server or a cloud device, etc. The training device 120 and the terminal device 110 may be the same device.

[0113] In the attached Figure 1C In the embodiment, the terminal device 110 is configured with an I / O interface 112 for data interaction with external devices. The user can input data to the I / O interface 112 through the client device 140. In this embodiment, the input data is an event data stream, which can be input by the user or from the database 130; wherein the client device 140 can be an image acquisition device, such as an event camera, an ordinary camera, etc.

[0114] Optionally, the preprocessing module 113 is used to preprocess the input data received by the I / O interface 112. In the embodiment of the present application, the preprocessing module 113 is used to preprocess the input data received by the I / O interface 112, and the preprocessed data enters the calculation module 111. In the embodiment of the present application, the input data may be an event data stream, and the preprocessing module 113 may be used to process the event data stream obtained by the event camera for a preset time length to obtain corresponding event data stream data.

[0115] When the terminal device 110 preprocesses input data, or when the computing module 111 of the terminal device 110 performs calculations and other related processing, the terminal device 110 can call data, codes, etc. in the data storage system 150 for corresponding processing, and can also store data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0116] Finally, the I / O interface 112 returns the model processing result of the input data, that is, the depth map or depth estimation data, to the client device 140 to provide it to the user. In this case, the client device 140 may be a display.

[0117] In the attached Figure 1C In the case shown in , the user can manually input data, which can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data, it needs to obtain the user's authorization, and the user can set the corresponding permissions in the client device 140. The user can view the results output by the terminal device 110 on the client device 140, and the specific presentation form can be display, sound, action, etc. The client device 140 can also be used as a data collection terminal to collect data such as Figure 1C The input data of the input I / O interface 112 and the output result of the output I / O interface 112 are taken as new sample data and stored in the database 130. Of course, it is also possible to collect data without going through the client device 140, but directly by the I / O interface 112. Figure 1C The input data of the input I / O interface 112 and the output result of the output I / O interface 112 are stored in the database 130 as new sample data.

[0118] It is worth noting that Figure 1C This is only a schematic diagram of a system architecture provided by an embodiment of the present application. Figure 1C The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 1CIn the embodiment, the data storage system 150 is an external memory relative to the terminal device 110. In other cases, the data storage system 150 may also be placed in the terminal device 110.

[0119] The above model training method is described in detail below.

[0120] refer to Figure 2 , Figure 2 200 is a flow chart of a model training method provided in an embodiment of the present application. In this embodiment, the execution subject takes a model training device as an example, and the model training method 200 includes the following steps:

[0121] 201. A model training device obtains a sample set, where the sample set includes a first image and event data stream data.

[0122] Specifically, the first image is an image taken by an ordinary camera, and an ordinary camera refers to a camera based on an image sensor. The event camera outputs an event data stream, and the event data stream data is obtained according to the event data stream captured by the event camera for a preset time length, and the first image is captured by the ordinary camera at the end of the preset time length. The specific time length of the preset time length can be set according to actual conditions and is not limited. The first image and the event data stream capture the same scene.

[0123] Exemplarily, in the embodiment of the present application, the event data stream data may be the event data stream itself or the space-time voxel of a preset time length.

[0124] Exemplarily, the specific process of obtaining a sample set is described below by taking the event data stream data as a spatiotemporal voxel as an example:

[0125] For the first image, in one example, the camera parameters (including internal parameters and external parameters) are first calibrated, and then the first image is obtained by using the camera, and then the first image is dedistorted according to the camera parameters to obtain the first image finally used for model training. In another example, the first image is first obtained by using the camera, and then the camera parameters are calibrated, and finally the first image is dedistorted according to the camera parameters to obtain the first image finally used for model training. In the following, the example of first obtaining the first image, then calibrating the camera parameters and dedistortion processing is taken.

[0126] The model training device first obtains the first image sequence and event data stream in multiple scenes as a data set, and then divides the data set into a training set and a test set according to the first ratio and the second ratio. The training set is used to train the initial depth estimation model, and the test set is used to test the target depth estimation model to evaluate the training effect of the model. The sum of the first ratio and the second ratio is 100%. The specific values ​​of the first ratio and the second ratio can be set according to actual conditions, for example, the first ratio is 80% and the second ratio is 20%. The above-mentioned scene can be any depth estimation scene, such as various scenes of intelligent driving, such as high-speed driving, urban driving and other scenes.

[0127] In one example, the model training device may classify the first image sequence obtained in each scene into a training set and a test set according to a first ratio and a second ratio, respectively. Similarly, the model training device classifies the event data stream obtained in each scene into a training set and a test set according to a first ratio and a second ratio, respectively. The image sequence and the event data stream correspond to each other when the data is divided.

[0128] For example, the first ratio is 80% and the second ratio is 20%. Scene A is photographed within a time interval to obtain an event data stream and a first image sequence arranged in time. The first image sequence corresponding to the first 80% of the time interval can be included in the training set, and the first image sequence corresponding to the last 20% of the time interval can be included in the test set. The event data stream corresponding to the first 80% of the time interval is included in the training set, and the event data stream corresponding to the last 20% of the time interval is included in the test set.

[0129] In order to ensure the generalization of the depth estimation model, the scenes of the training set and the test set can be set to different scenes. Therefore, in another example, the first image sequence and event data stream corresponding to the first scale of scenes in multiple scenes can be classified into the training set, and the first image sequence and event data stream corresponding to the remaining second scale of scenes can be classified into the test set.

[0130] For example, the first ratio is 80%, the second ratio is 20%, and the plurality of scenes is 10 scenes, then the first image sequences and event data streams of 80% of the scenes (8 scenes) are classified into the training set, and the first image sequences and event data streams of the remaining 20% ​​of the scenes (2 scenes) are classified into the test set.

[0131] In one example, an event camera that can simultaneously output pixel-by-pixel aligned normal images and event data streams is used to obtain the first image sequences and event data streams of the above-mentioned multiple scenes.

[0132] In another example, the event camera can only output event streams. In this case, a camera based on an image sensor can be used to obtain the first image sequence of the above multiple scenes; and the event camera can be used to obtain the event data streams of the above multiple scenes.

[0133] Next, the first image is dedistorted according to the camera parameters.

[0134] For radial distortion, quadratic and higher-order polynomial functions related to the distance r from the center are used for correction:

[0135]

[0136] Where P = [x, y] T are the coordinates of the uncorrected point, [x corrected ,y corrected ] T are the coordinates of the corrected point, k1, k2, k3 are the radial distortion coefficients of the camera, and r is the distance between point P and the origin of the coordinate system.

[0137] For tangential distortion, two other parameters p1 and p2 can be used for correction:

[0138]

[0139] Where P = [x, y] T are the coordinates of the uncorrected point, [x corrected ,y corrected ] T are the coordinates of the corrected point, p1 and p2 are the tangential distortion coefficients of the camera, and r is the distance between point P and the origin of the coordinate system.

[0140] For an event data stream of a preset time length, the number of events therein is not fixed. In order to input this event data stream into a neural network, in an embodiment of the present application, the event data stream of a preset time length is represented as an event data stream data.

[0141] In order to satisfy the subsequent calculation of photometric error, such as Figure 3 As shown in Figure 1, the event data stream data of the event camera and the image frame of the ordinary camera are aligned in the signal end time. Specifically, from time k-1 to time k as a preset time length, the event data stream data E corresponding to the preset time length can be obtained. k 301, accordingly, a first image I is obtained by shooting at time k k 302.

[0142] In order to facilitate the calculation of the subsequent loss function, the camera parameters are used to perform pixel alignment on the ordinary image and the event data stream data, that is, the corresponding coordinates of each pixel in the event data stream data in the ordinary image are determined and obtained through coordinate transformation.

[0143] For example, when the event camera can simultaneously output a normal image and an event data stream that are aligned pixel by pixel, there is no need to align the normal image and the event data stream.

[0144] As another example, when the event camera can only output event streams, an additional ordinary camera can be installed so that the external parameters of the two cameras are only the displacement L on the x-axis. Then, according to the following coordinate transformation relationship, the image of the ordinary camera can be projected onto the pixel plane of the event camera through inverse warp. After this alignment, the internal parameters of the event camera are used as the internal parameters of the ordinary image and event stream.

[0145]

[0146] Among them, (x, y, z) are the point coordinates in three-dimensional space, (u1, v1) are the pixel coordinates of the ordinary camera, (u2, v2) are the pixel coordinates of the event camera, and f x1 、f y1 、c x1 、c y1 is the intrinsic parameter of a normal camera, f x2 、f y2 、c x2 、c y2 is the internal parameter of the event camera. The above approximate calculation is only valid when z>>f x2 The alignment accuracy can only be guaranteed on the pixel points of L (for example, in outdoor scenes of autonomous driving, most pixels will meet this condition).

[0147] The supervisory signal of the self-supervised depth estimation in the embodiment of the present application comes from the camera motion in a static scene, so the static frames and moving objects are a kind of interference to the training. In order to reduce this interference, the image frames in which the camera is stationary or there are moving objects in the scene are screened out in the training set. Among them, the specific number of moving objects can be determined according to the actual situation without specific limitation, for example, the number of moving objects is greater than or equal to 1, or the number of moving objects is greater than or equal to 5.

[0148] 202. The model training device trains the initial depth estimation model according to the first image and the event data stream data to obtain a target depth estimation model.

[0149] In an embodiment of the present application, self-supervised training of a depth estimation model is implemented using images taken by an ordinary camera and event data streams from an event camera, wherein an event data stream of a preset time length is represented as event data stream data for model training. Model training can be completed without the aid of costly true depth maps, which can effectively reduce the training cost of the depth estimation model.

[0150] The first image sequence and event data stream data obtained after the above step 201 are classified into a sample set, and the initial depth estimation model is trained based on the sample set to obtain a target depth estimation model. During training, each time, the corresponding first loss value is calculated based on an event data stream data and a first image with pixel alignment and signal end time alignment, and then the initial depth estimation model is adjusted and optimized based on the first loss value. The specific process is described as follows.

[0151] In a possible implementation manner, the above step 202 specifically includes the following steps:

[0152] 221. The model training device processes the event data stream data through the initial depth estimation model to obtain first depth estimation data.

[0153] 222. The model training device determines a first loss value of the first image based on the first image and the first depth estimation data.

[0154] Exemplarily, the first loss value is calculated based on a minimum photometric loss function and a mask algorithm.

[0155] 223. The model training device adjusts and optimizes the initial depth estimation model according to the first loss value to obtain a target depth estimation model.

[0156] In an embodiment of the present application, when the model training device calculates the first loss value, the influence of the occluded area is weakened by the minimum photometric loss function, and then the relatively static pixels are removed by the mask algorithm. The optimization direction of the initial depth estimation model is determined based on the first loss value to obtain the target depth estimation model.

[0157] In a possible implementation manner, the above step 221 specifically includes the following steps:

[0158] A1. Extract features from the event data stream to obtain feature graphs of i scales, where i is an integer greater than or equal to three.

[0159] A2. Fuse and decode the feature maps of i scales to obtain a depth map of at least one scale.

[0160] A3. Determine depth estimation data of at least one scale of the event data stream data based on the depth map of at least one scale.

[0161] Specifically, when a depth map of a certain scale is obtained in step A2, depth estimation data may be obtained based on the depth map, and the depth estimation data is the first depth estimation data.

[0162] When the depth map of two scales or more is obtained in step A2, the depth estimation data of the corresponding scale is obtained based on the depth map of each scale, and the depth estimation data of two scales or more can be obtained. The depth estimation data of two scales or more is the first depth estimation data. Outputting depth estimation data of multiple scales during training has stronger constraints on the network, faster network convergence speed, and higher accuracy of depth estimation.

[0163] More specifically, depth estimation data can be obtained based on the depth map, and each pixel in the depth map can be substituted into the depth estimation calculation formula to obtain the depth estimation value corresponding to each pixel. Therefore, based on a depth map of at least one scale, corresponding depth estimation data of at least one scale can be obtained. Exemplarily, the depth estimation calculation formula can be: D = 1 / (aσ+b), where σ represents the pixel value of the pixel in the depth map, a and b are preset parameters, and the specific sizes of a and b can be set according to actual conditions without limitation.

[0164] In an embodiment of the present application, the initial depth estimation model performs multi-scale feature extraction on the event data stream data to obtain feature maps of multiple scales, and the first depth estimation data can be obtained based on the feature maps of multiple scales. Among them, the feature maps of multiple scales are equivalent to feature maps of multiple resolutions. In this way, feature information under different receptive fields (Receptive Field) can be obtained, which helps to improve the accuracy of depth estimation. The above-mentioned receptive field is the size of the area on the input image mapped back to the pixel points on the feature map output by each layer of the convolutional neural network. That is, the size of a point on the feature map relative to the original image is also the area of ​​the input image that the convolutional neural network feature can "see". In one possible implementation, when the depth map of at least one scale is a depth map of i-1 scales, the above-mentioned step A2 specifically includes the following steps:

[0165] A21. Perform i-time fusion processing on the feature maps of i scales to obtain fused feature maps of i scales.

[0166] In the i-th fusion process, the fusion feature map obtained in the i-1-th fusion process is upsampled to obtain the fusion feature map of the i-th fusion process.

[0167] In the i-1th fusion process, the fused feature map obtained in the i-2th fusion process and the first feature map are fused to obtain the fused feature map of the i-1th fusion process. The first feature map is the feature map with the largest resolution among the feature maps of the i-scales.

[0168] In the i-2 fusion process, the fusion feature map obtained in the i-3 fusion process, the first feature map, and the second feature map are fused to obtain the fusion feature map of the i-2 fusion process. The second feature map is the feature map with the second largest resolution among the feature maps of the i scales.

[0169] And so on, until the first fusion process, the feature maps of i scales are fused to obtain the fused feature map of the first fusion process.

[0170] A22. Decode the fused feature maps of i-1 scales respectively to obtain depth maps of i-1 scales. The fused feature maps of i-1 scales are the fused feature maps of the fused feature maps of i scales except the fused feature map with the smallest feature map scale.

[0171] In the embodiment of the present application, the fusion of multi-scale feature maps is realized by i-fold fusion processing to obtain fused feature maps of i scales, which is conducive to improving the accuracy of the depth map obtained based on the fused feature maps. The use of multi-scale feature jump connections reduces the information loss of features, such as small-scale features, which can directly participate in the feature fusion process, thereby improving the performance of the depth estimation model.

[0172] In an example, when i is 3, at least one of the depth maps of two scales is a depth map of two scales. Take the initial depth estimation model to process the input event data stream data E to obtain the corresponding depth map as an example, refer to Figure 4A , Figure 4A It is a flow chart of generating first depth estimation data provided by an embodiment of the present application; the initial depth estimation model at this time includes a coding network, a first fusion module X0, a second fusion module X1, a third fusion module X2, a first decoding module and a second decoding module. The coding network is used to obtain feature maps of three scales according to the event data stream data E, which are f1, f2 and f3. Among them, the feature map with a larger subscript has a larger number of channels, a smaller scale and a smaller resolution. The scale of feature map f1 is greater than the scale of feature map f2, and the scale of feature map f2 is greater than the scale of feature map f3. The resolution of feature map f1 is greater than the resolution of feature map f2, and the resolution of feature map f2 is greater than the resolution of feature map f3.

[0173] In this embodiment, three fusion processes are performed based on feature maps of three scales. The third fusion module X2 is used to implement the first fusion process. The second fusion module X1 is used to implement the second fusion process. The first fusion module X0 is used to implement the third fusion process.

[0174] In the first fusion process, the feature maps of three scales (f1, f2 and f3) are fused to obtain the fused feature map of the first fusion process.

[0175] In the second fusion process, the fused feature map obtained in the first fusion process and the first feature map are fused to obtain a fused feature map of the second fusion process. The first feature map is the feature map with the largest resolution among the feature maps of the three scales, namely, feature map f1.

[0176] In the third fusion process, the fusion feature map obtained by the second fusion process is upsampled to obtain the fusion feature map of the third fusion process.

[0177] The scale of the fused feature map in the third fusion process is larger than that in the second fusion process. The scale of the fused feature map in the second fusion process is larger than that in the first fusion process.

[0178] The first decoding module is used to decode the fused feature map of the third fusion process to obtain a first depth map. The second decoding module is used to decode the fused feature map of the second fusion process to obtain a second depth map. Based on the first depth map, depth estimation data of one scale can be obtained. Based on the second depth map, depth estimation data of another scale can be obtained.

[0179] In another example, when i is 4, at least one of the depth maps of three scales is used. For example, the initial depth estimation model processes the input event data stream data E to obtain the corresponding depth map. Figure 4B , Figure 4B It is another flow diagram of generating first depth estimation data provided by an embodiment of the present application; the initial depth estimation model at this time includes an encoding network, a first fusion module X0, a second fusion module X1, a third fusion module X2, a fourth fusion module X3, a first decoding module, a second decoding module and a third decoding module. The encoding network is used to obtain feature maps of four scales according to the event data stream data, namely f1, f2, f3 and f4. Among them, the feature map with a larger subscript has a larger number of channels, a smaller scale and a smaller resolution.

[0180] In this embodiment, four fusion processes are performed based on feature maps of four scales. The fourth fusion module X3 is used to implement the first fusion process. The third fusion module X2 is used to implement the second fusion process. The second fusion module X1 is used to implement the third fusion process. The first fusion module X0 is used to implement the fourth fusion process.

[0181] In the first fusion process, the feature maps of four scales (f1, f2, f3 and f4) are fused to obtain the fused feature map of the first fusion process.

[0182] In the second fusion process, the fused feature map obtained in the first fusion process, the first feature map, and the second feature map are fused to obtain a fused feature map of the second fusion process. The second feature map is the feature map with the second largest resolution among the feature maps of the four scales, that is, feature map f2.

[0183] In the third fusion process, the fused feature map obtained in the second fusion process and the first feature map are fused to obtain a fused feature map of the third fusion process. The first feature map is the feature map with the largest resolution among the feature maps of the four scales, namely, feature map f1.

[0184] In the fourth fusion process, the fusion feature map obtained in the third fusion process is upsampled to obtain the fusion feature map of the fourth fusion process.

[0185] The scale of the fused feature map of the 4th fusion process is larger than the scale of the fused feature map of the 3rd fusion process. The scale of the fused feature map of the 3rd fusion process is larger than the scale of the fused feature map of the 2nd fusion process. The scale of the fused feature map of the 2nd fusion process is larger than the scale of the fused feature map of the 1st fusion process.

[0186] The first decoding module is used to decode the fused feature map of the fourth fusion process to obtain a first depth map. The second decoding module is used to decode the fused feature map of the third fusion process to obtain a second depth map. The third decoding module is used to decode the fused feature map of the second fusion process to obtain a third depth map. Based on the first depth map, depth estimation data of a first scale can be obtained. Based on the second depth map, depth estimation data of a second scale can be obtained. Based on the third depth map, depth estimation data of a third scale can be obtained.

[0187] In another example, when i is 5, at least one of the depth maps of four scales is used. Figure 4C , Figure 4C It is another flow diagram of generating first depth estimation data provided by an embodiment of the present application; the initial depth estimation model at this time includes a coding network, a first fusion module X0, a second fusion module X1, a third fusion module X2, a fourth fusion module X3, a fifth fusion module X4, a first decoding module, a second decoding module, a third decoding module and a fourth decoding module. The coding network is used to obtain feature maps of five scales according to the event data stream data, namely f1, f2, f3, f4 and f5. Among them, the feature map with a larger subscript has a larger number of channels, a smaller scale and a smaller resolution.

[0188] In this embodiment, five fusion processes are performed based on the feature maps of five scales. The fifth fusion module X4 is used to implement the first fusion process. The fourth fusion module X3 is used to implement the second fusion process. The third fusion module X2 is used to implement the third fusion process. The second fusion module X1 is used to implement the fourth fusion process. The first fusion module X0 is used to implement the fifth fusion process.

[0189] In the first fusion process, the feature maps of five scales (f1, f2, f3, f4 and f5) are fused to obtain the fused feature map of the first fusion process.

[0190] In the second fusion process, the fused feature map obtained in the first fusion process, the first feature map, the second feature map, and the third feature map are fused to obtain a fused feature map of the second fusion process. The third feature map is a feature map with the third largest resolution among the feature maps of the five scales, namely, feature map f3.

[0191] In the third fusion process, the fused feature map obtained in the second fusion process, the first feature map, and the second feature map are fused to obtain a fused feature map of the third fusion process. The second feature map is the feature map with the second largest resolution among the feature maps of the five scales, that is, feature map f2.

[0192] In the fourth fusion process, the fused feature map obtained in the third fusion process and the first feature map are fused to obtain a fused feature map of the fourth fusion process. The first feature map is the feature map with the largest resolution among the feature maps of the five scales, namely, feature map f1.

[0193] In the fifth fusion process, the fusion feature map obtained in the fourth fusion process is upsampled to obtain the fusion feature map of the fifth fusion process.

[0194] The scale of the fused feature map of the 5th fusion process is larger than the scale of the fused feature map of the 4th fusion process. The scale of the fused feature map of the 4th fusion process is larger than the scale of the fused feature map of the 3rd fusion process. The scale of the fused feature map of the 3rd fusion process is larger than the scale of the fused feature map of the 2nd fusion process. The scale of the fused feature map of the 2nd fusion process is larger than the scale of the fused feature map of the 1st fusion process.

[0195] The first decoding module is used to decode the fused feature map of the fifth fusion process to obtain a first depth map. The second decoding module is used to decode the fused feature map of the fourth fusion process to obtain a second depth map. The third decoding module is used to decode the fused feature map of the third fusion process to obtain a third depth map. The fourth decoding module is used to decode the fused feature map of the second fusion process to obtain a fourth depth map. Based on the first depth map, depth estimation data of a first scale can be obtained. Based on the second depth map, depth estimation data of a second scale can be obtained. Based on the third depth map, depth estimation data of a third scale can be obtained. Based on the fourth depth map, depth estimation data of a fourth scale can be obtained.

[0196] For example, reference Figure 4A , Figure 4B and Figure 4C Although the initial depth estimation model in the embodiment of the present application can output depth maps of i-1 scales to obtain depth estimation data of i-1 scales; however, the initial depth estimation model in the embodiment of the present application can also output a depth map of one scale. Correspondingly, at this time, the initial depth estimation model obtains depth estimation data of one scale based on the depth map of one scale.

[0197] The initial depth estimation model outputs a depth map of a certain scale by using Figure 4A , Figure 4B or Figure 4C When outputting a depth map, a fusion feature of one scale is retained from the fusion features of i-1 scales, and the fusion feature is decoded to obtain a depth map of one scale. For example, Figure 4A In the example, only the output features of the second fusion module X1 are decoded to obtain the second depth map. At this time, the initial depth estimation model does not need to set the first fusion module X0 and the first decoding module. For another example, Figure 4B In the example, only the output features of the first fusion module X0 are decoded to obtain the first depth map, that is, only the depth map with the highest resolution is retained. At this time, the initial depth estimation model does not need to set the second fusion module X1, the third fusion module X2, the second decoding module and the third decoding module.

[0198] For example, Figure 4A , Figure 4B and Figure 4C The encoding network in can be implemented by using multiple small encoding networks, that is, each small encoding network obtains a feature map of one scale. Alternatively, the encoding network can be implemented by using an overall network, using one encoding network to output feature maps of i scales. For example, the ResNet-18 encoding network.

[0199] In a possible implementation, in the above-mentioned i-1th fusion process, the fusion feature map obtained by the i-2th fusion process and the first feature map are fused to obtain the fusion feature map of the i-1th fusion process, which specifically includes the following steps:

[0200] The fused feature map obtained by the (i-2) fusion process is upsampled to obtain a first upsampled feature map.

[0201] Concatenate the first upsampled feature map and the first feature map to obtain a fused feature map of the i-1th fusion processing.

[0202] In the embodiment of the present application, since the scale of the fused feature map obtained by the i-2th fusion processing is smaller than the scale of the first feature map, it is necessary to first upsample the fused feature map obtained by the i-2th fusion processing to obtain the first upsampled feature map, and then splice the first upsampled feature map and the first feature map to obtain the fused feature map for the i-1th fusion processing.

[0203] For example, Figure 4C The structure of the second fusion module X1 in is as follows Figure 5A As shown, it includes an upsampling unit and a splicing unit. The upsampling unit is used to upsample the fused feature map obtained by the third fusion process (such as by bilinear interpolation) to obtain a first upsampled feature map. The splicing unit is used to splice the first upsampled feature map and the first feature map to obtain a fused feature map for the fourth fusion process. The splicing unit includes a splicing module and a convolution module, and the convolution module may include more than or equal to one convolution layer.

[0204] In a possible implementation, in the above-mentioned i-2 fusion process, the fusion feature map, the first feature map, and the second feature map obtained by the i-3 fusion process are fused to obtain the fusion feature map of the i-2 fusion process, which specifically includes the following steps:

[0205] The fused feature map obtained by the i-3th fusion process is upsampled to obtain a second upsampled feature map.

[0206] Downsampling is performed on the first feature map to obtain a downsampled feature map.

[0207] The second up-sampled feature map, the down-sampled feature map, and the second feature map are concatenated to obtain a fused feature map of the i-2th fusion process.

[0208] In the embodiment of the present application, since the scale of the fused feature map obtained by the i-3 fusion process is smaller than the scale of the second feature map, the fused feature map obtained by the i-3 fusion process is upsampled to obtain the second upsampled feature map. In addition, since the scale of the first feature map is larger than the scale of the second feature map, the first feature map is downsampled to obtain the downsampled feature map, and then the second upsampled feature map, the first feature map and the downsampled feature map are spliced ​​to obtain the fused feature map of the i-2 fusion process.

[0209] For example, Figure 4C The structure of the third fusion module X2 in is as follows Figure 5B As shown, it includes an upsampling unit, a downsampling unit and a splicing unit. The upsampling unit is used to perform upsampling processing (such as bilinear interpolation) on the fused feature map obtained by the second fusion process to obtain a second upsampling feature map. The downsampling unit is used to perform downsampling processing (such as maximum pooling) on ​​the first feature map to obtain a downsampled feature map. The splicing unit is used to splice the second upsampling feature map, the downsampling feature map and the second feature map to obtain a fused feature map for the third fusion process. Among them, the splicing unit includes a splicing module and a convolution module, and the above-mentioned convolution module may include greater than or equal to one convolution layer. In addition, Figure 4C Compared with the third fusion module X2, the fourth fusion module X3 in the downsampling unit has the second feature map in addition to the first feature map. Similarly, Figure 4C Compared with the fourth fusion module X3, the fifth fusion module X4 in the downsampling unit has a third feature map as input in addition to the first feature map and the second feature map.

[0210] Also illustratively, Figure 4C The structure of the third fusion module X2 in is as follows Figure 5C As shown, it includes a first convolution unit and a second convolution unit, the first convolution unit is used to splice the first feature map, the second feature map and the fused feature map obtained by the second fusion process to obtain a fused feature. The second convolution unit is used to process the fused feature to obtain a fused feature map of the third fusion process. The above-mentioned first convolution unit or the second convolution unit may include more than or equal to one convolution layer.

[0211] In a possible implementation, in the above-mentioned i-th fusion process, upsampling is performed on the fusion feature map obtained by the i-1-th fusion process to obtain the fusion feature map of the i-th fusion process, which specifically includes the following steps:

[0212] The fused feature map obtained by the i-1th fusion process is upsampled to obtain the fused feature map of the i-th fusion process.

[0213] For example, Figure 4CThe structure of the first fusion module x0 in is as follows Figure 5D As shown, an upsampling unit is included, and the upsampling unit is used to upsample the fused feature map obtained by the fourth fusion process (such as through bilinear interpolation) to obtain the fused feature map of the fifth fusion process.

[0214] For example, Figure 4C The structure of the first decoding module in is as follows Figure 5E As shown, the first decoding module includes a first convolution unit and an activation unit. The first convolution unit may include a convolution layer greater than or equal to one, and the size of the convolution kernel in the convolution layer is m×m, where m is an integer, for example, m is 3. The activation unit may be implemented using an activation function such as a Sigmoid activation function or a ReLU function. Using the first decoding module, the fused feature map of the fifth fusion process is converted into a first depth map.

[0215] In a possible implementation manner, the above step 222 specifically includes the following steps:

[0216] B1. The model training device calculates a second loss value based on the first photometric error and the second photometric error.

[0217] The first photometric error is calculated based on the first image, the previous frame of the first image, and the first depth estimation data. The second photometric error is calculated based on the first image, the next frame of the first image, and the first depth estimation data. In this embodiment, the model training device calculates the second loss value based on the first photometric error, the second photometric error, and the minimum photometric loss function.

[0218] B2. The model training device calculates a mask of the first image according to the first photometric error, the second photometric error, the third photometric error, and the fourth photometric error. The third photometric error is obtained according to the first image and a frame image before the first image. The fourth photometric error is obtained according to the first image and a frame image after the first image.

[0219] In this embodiment, the model training device calculates the mask of the first image according to the first photometric error, the second photometric error, the third photometric error, the fourth photometric error and a mask algorithm.

[0220] B3. The model training device determines the first loss value based on the second loss value and the mask.

[0221] In an embodiment of the present application, the photometric error between the first image and its previous frame image, i.e., the first photometric error, is determined based on the first depth estimation data, and the photometric error between the first image and its subsequent frame image, i.e., the second photometric error, is determined based on the first depth estimation data. The second loss value can be obtained using the minimum photometric loss function, the first photometric error, and the second photometric error. In addition, the mask of the first image is determined using a mask algorithm, a third photometric error between the first image and its previous frame image, and a fourth photometric error between the first image and its subsequent frame image; and then the first loss value is obtained based on the second loss value and the above-mentioned mask.

[0222] In a possible implementation manner, the above step 222 further includes the following steps:

[0223] B4. The model training device determines a first photometric error according to the first image, the first pose estimation data and the first depth estimation data. The first pose estimation data is the camera pose transformation data between the first image and a frame image before the first image.

[0224] B5. The model training device determines a second photometric error according to the first image, the second pose estimation data and the first depth estimation data. The second pose estimation data is the camera pose transformation data between the first image and a frame image subsequent to the first image.

[0225] Specifically, since the event data stream data is relatively sparse, and since the event information is highly correlated with the camera motion, the events of associated pixels may have opposite polarities and very different normalized timestamps, so it is difficult to associate events of adjacent preset time lengths. In order to overcome the above defects, in this embodiment, image frames aligned with the event data stream data are used as input to the pose estimation network.

[0226] Furthermore, in the embodiment of the present application, a pose estimation network is used to obtain camera pose transformation data. Fig. 6A , the first image and the previous frame of the first image are input into the pose estimation network, and the first pose estimation data can be obtained. Exemplarily, the output of the pose estimation network is the camera pose transformation matrix [R|t] between the two frames of images, where R is the rotation and t is the translation. The function of this matrix is ​​to give the coordinates of a point in a coordinate system, and multiplying this matrix by the coordinates of the point in another coordinate system can obtain.

[0227] In a possible implementation manner, the above step B4 specifically includes the following steps:

[0228] B41. The model training device determines a reconstructed image based on the first image, the first pose estimation data and the first depth estimation data.

[0229] B42. The model training device determines a first photometric error based on the first image, the reconstructed image, and the similarity between the first image and the reconstructed image.

[0230] Among them, the similarity between the first image and the reconstructed image can be calculated by a variety of methods. For example, the image structure similarity between the first image and the reconstructed image is used as the similarity between the first image and the reconstructed image. For another example, the cosine distance between the representation vector of the first image and the representation vector of the reconstructed image is used as the similarity between the first image and the reconstructed image. In an embodiment of the present application, the first image, the first pose estimation data, and the first depth estimation data are first used to reconstruct the first image to obtain a reconstructed image. Then, the first photometric error is determined based on the first image, the reconstructed image, and the similarity between the first image and the reconstructed image.

[0231] Exemplarily, the model training device can determine the first photometric error based on the first image, the reconstructed image, the first weight, and the similarity between the first image and the reconstructed image, wherein the specific value of the first weight can be set according to actual conditions and is not limited.

[0232] In step B41, taking the first pose estimation data as an example, for the reconstructed image For each pixel in I, the first pose estimation data and the first depth estimation data are used to inversely warp the pixels in I k-1 The above is obtained by bilinear interpolation. The specific process can be used Figure 6B Demonstration: Using the first image I k The pixel point p in k The depth estimation value (obtained based on the first depth estimation data), I k The previous frame image I of the first image k-1 The pose transformation between (based on the first pose estimation data), the pixel point p k Project the coordinates of to image I k-1 Get pixel point p k-1 Pixel p k-1 Then the pixel coordinate p is obtained through bilinear interpolation k-1 in I k The value on is then assigned to the (Warp) pixel p k In reconstructing the image The value in .

[0233] In step B42, the photometric error pe between adjacent image frames is calculated by the following formula (2):

[0234]

[0235] In the above formula, Ik is the first image, is the reconstructed image. α is the weight coefficient, which can be set according to the actual situation. For example, α is 0.85. SSIM() is a function used to evaluate the structural similarity of images. It is calculated in a 3×3 sliding pixel window. Specifically:

[0236]

[0237] μ X , μ Y Respectively represent the mean of images X and Y in the sliding window, σ X , σ Y Respectively represent the standard deviation of image X, Y in the sliding window, σ XY Represents the covariance of images X and Y in the sliding window. C1 and C2 are constants to avoid the denominator being 0. In the embodiment of the present application, C1 and C2 are set to 0.0001 and 0.0009 respectively. The more similar the structures of X and Y are, the closer SSIM(X,Y) is to 1, and vice versa. In the above formula, image X is the first image, and image Y is the reconstructed image.

[0238] In step B5, the second photometric error can be calculated by using the same calculation method as step B4. The difference is that the second pose estimation data is used to determine the reconstructed image.

[0239] In step B1, since the photometric error pe is based on the grayscale consistency assumption, the scene is static and there is no occlusion, such an assumption is difficult to meet in real scenes. In order to reduce the impact of the widely existing occlusion area, the minimum photometric loss function L is used in the embodiment of the present application. p To process:

[0240]

[0241] Among them, the set {k-1, k+1} refers to k-1, k, k+1; L p The first image I is calculated k and the previous frame image I k-1 The first photometric error between k and the next frame image I k+1 The second photometric error between p A minimum value is determined from the first photometric error and the second photometric difference as the second loss value.

[0242] In step B2, in order to remove relatively static and low-texture pixels, the model performance of the depth estimation model is further improved. In this embodiment, a mask algorithm is used to calculate the mask M ns :

[0243]

[0244] Among them, [] is an Iverson bracket, which means that when the condition in the bracket is met, the position of the pixel in the mask is 1, otherwise it is 0. In other words, the second loss value And the third loss value min k ; pe(I k ,I k-1 ), the values ​​of each corresponding position of the two are compared. When the conditions in the brackets are met, the value of the position in the mask is 1, otherwise it is 0. The third loss value min k ; pe(I k ,I k-1 ) is the minimum value of the third photometric error and the fourth photometric error, wherein the third photometric error and the fourth photometric error are directly calculated according to formula (2). Different from the first photometric error, the third photometric error and the fourth photometric error do not require image reconstruction. The SSIM() function calculates the image structure similarity between the first image and the previous frame image of the first image, and the image structure similarity between the first image and the next frame image of the first image.

[0245] refer to Figure 6C , Figure 6C is a schematic diagram of a first image provided in an embodiment of the present application, Figure 6C is a single frame of an ordinary camera image. Fig.6D , Fig.6D is a schematic diagram of the effect of a mask algorithm provided in an embodiment of the present application, Fig.6D For Figure 6C Calculate the mask M ns The result after that, Fig.6D Black pixels represent 0, and white pixels represent 1.

[0246] In step B3, in the embodiment of the present application, the minimum photometric error loss function and the mask algorithm are combined, and then averaged at each pixel and each scale to obtain the final loss function L:

[0247]

[0248] Among them, s represents the total number of scales, p i represents the pixel coordinates, and T represents the total number of pixels on the image. According to the above formula, the first loss value corresponding to the first image can be calculated.

[0249] When the first loss value corresponding to the first image is obtained, the parameters of the initial depth estimation model can be adjusted and optimized according to the first loss value. Exemplarily, the Adam optimizer is used, the optimizer parameters are kept at the default values, the total number of training rounds is 10, the initial learning rate of the first 8 rounds is set to 0.0001, and the learning rate of the last 2 rounds is set to 0.00001. In addition, when training on other data sets, the optimizer, learning rate and number of training rounds can be adjusted by yourself.

[0250] Exemplarily, when the initial depth estimation model outputs depth estimation data of a certain scale, the first loss value corresponding to the first image can be calculated according to the above-mentioned method of calculating the first loss value, and then the initial depth estimation model can be adjusted and optimized according to the first loss value.

[0251] Exemplarily, when the initial depth estimation model outputs depth estimation data of i-1 scales, for each scale of depth estimation data, i-1 first loss values ​​can be calculated according to the above-mentioned method of calculating the first loss value, and then the average value is calculated based on the i-1 first loss values ​​to obtain the final first loss value, and the initial depth estimation model is adjusted and optimized based on the final first loss value.

[0252] The depth estimation method according to the embodiment of the present application may be executed by a depth estimation device, or by a chip in the depth estimation device.

[0253] refer to Figure 7 , Figure 7 700 is a flowchart of a depth estimation method provided in an embodiment of the present application; taking the execution subject as a depth estimation device as an example, the depth estimation method 700 includes the following steps:

[0254] 701. A depth estimation device obtains event data stream data.

[0255] Specifically, the event data stream data is obtained according to the event data stream captured by the event camera with a preset time length. The specific length of the preset time length can be set according to actual conditions, such as 60 milliseconds, 50 milliseconds or 30 milliseconds.

[0256] 702. The depth estimation device processes the event data stream data by using a target depth estimation model to obtain depth estimation data of the event data stream data.

[0257] Specifically, the target depth estimation model is a model trained based on the model training method shown in any of the above embodiments. Figure 1D and Figure 1E, the depth estimation device may first process the event data stream of a preset time length through the target depth estimation model to obtain event data stream data, and then process the event data stream data to obtain depth estimation data. Alternatively, the depth estimation device may first pre-process the event data stream of a preset time length to obtain event data stream data, and then input the event data stream data into the target depth estimation model for processing to obtain depth estimation data.

[0258] In the embodiment of the present application, depth estimation data can be obtained by using an event data stream of a preset time length and a target depth estimation model. The target depth estimation model is trained based on the model training method shown in any of the above embodiments, and the model training cost is low, which is conducive to reducing the cost of depth estimation.

[0259] In a possible implementation manner, the above step 702 specifically includes the following steps:

[0260] The depth estimation device processes the event data stream data through a target depth estimation model to obtain a depth map of at least one scale of the event data stream data; the depth estimation device determines the depth estimation data based on the depth map of at least one scale.

[0261] In the embodiment of the present application, specific depth estimation data can be obtained based on the depth map to achieve depth estimation. The depth estimation method in the embodiment of the present application can estimate a dense monocular depth map from sparse event data, which can effectively improve the depth estimation accuracy.

[0262] For example, the target depth estimation model may output a depth map of i-1 scales, and the acquisition method may refer to the above Figure 4A , Figure 4B and Figure 4C When the target depth estimation model outputs depth maps of i-1 scales, for each depth map of one scale, depth estimation data of i-1 scales can be calculated according to the depth estimation calculation formula and the depth map, and then the average value is calculated based on the depth estimation data of i-1 scales to obtain the final depth estimation data.

[0263] As another example, the target depth estimation model may output a depth map of a scale, and the depth map of the scale may be one of the i-1 scale depth maps. For example, the depth map with the highest resolution is determined from the i-1 scale depth maps as the depth map output by the target depth estimation model.

[0264] Exemplarily, when the target depth estimation model outputs a depth map of a certain scale, the corresponding depth estimation data can be calculated according to the above depth estimation calculation formula and the depth map of the certain scale. For example, the above depth map can be the depth map with the highest resolution among the depth maps of i-1 scales.

[0265] The method of the embodiment of the present application is described in detail above, and the device provided by the embodiment of the present application is introduced below.

[0266] Figure 8 and Fig. 9 A schematic diagram of a possible device structure provided in an embodiment of the present application. Figure 8 The model training device shown can be used to implement the functions of the above-mentioned model training method embodiment, and thus can also achieve the beneficial effects possessed by the above-mentioned model training method embodiment. In the embodiment of the present application, the model training device can be an electronic device, or a module (such as a chip) applied to an electronic device. Fig. 9 The depth estimation device shown can be used to implement the functions of the above-mentioned depth estimation method embodiment, and thus can also achieve the beneficial effects of the above-mentioned depth estimation method embodiment. In the embodiment of the present application, the depth estimation device can be an electronic device, or a module (such as a chip) applied to an electronic device.

[0267] like Figure 8 As shown, the model training device 800 includes an acquisition module 801 and a training module 802. The model training device 800 is used to implement the above Figure 2 Alternatively, the model training device 800 may include a device for implementing the above Figure 2 A module for any function or operation in the model training method embodiment shown in , which module can be implemented in whole or in part by software, hardware, firmware or any combination thereof.

[0268] When the model training device 800 is used to implement Figure 2 In the function of the method embodiment shown, an acquisition module 801 is used to acquire a sample set. The sample set includes a first image and event data stream data, wherein the event data stream data is obtained according to an event data stream captured by an event camera for a preset time length, and the first image is captured by a common camera at the end of the preset time length. A training module 802 is used to train an initial depth estimation model according to the first image and the event data stream data to obtain a target depth estimation model.

[0269] In an embodiment of the present application, the model training device 800 uses images taken by an ordinary camera and the event data stream of an event camera to implement self-supervised training of a depth estimation model, wherein the event data stream of a preset time length is represented as event data stream data for model training; model training can be completed without the aid of costly true depth maps, thereby effectively reducing the training cost of the depth estimation model.

[0270] In a possible implementation, the training module 802 is specifically used for:

[0271] The event data stream data is processed by an initial depth estimation model to obtain first depth estimation data.

[0272] A first loss value for the first image is determined based on the first image and the first depth estimation data.

[0273] The initial depth estimation model is adjusted and optimized according to the first loss value to obtain a target depth estimation model.

[0274] In a possible implementation manner, the training module 802 is specifically configured to:

[0275] The second loss value is calculated according to the first photometric error and the second photometric error. The first photometric error is calculated according to the first image, a frame image before the first image, and the first depth estimation data; the second photometric error is calculated according to the first image, a frame image after the first image, and the first depth estimation data.

[0276] The mask of the first image is calculated according to the first photometric error, the second photometric error, the third photometric error and the fourth photometric error. The third photometric error is obtained according to the first image and the previous frame of the first image; the fourth photometric error is obtained according to the first image and the next frame of the first image.

[0277] A first loss value is determined based on the second loss value and the mask.

[0278] In a possible implementation manner, the training module 802 is specifically configured to:

[0279] A first photometric error is determined based on the first image, the first pose estimation data, and the first depth estimation data. The first pose estimation data is the camera pose transformation data between the first image and a frame image before the first image.

[0280] The second photometric error is determined according to the first image, the second pose estimation data and the first depth estimation data. The second pose estimation data is the camera pose transformation data between the first image and a frame image subsequent to the first image.

[0281] In a possible implementation, the training module 802 is specifically used to determine the first photometric error according to the first image, the first pose estimation data, and the first depth estimation data:

[0282] A reconstructed image is determined according to the first image, the first pose estimation data, and the first depth estimation data.

[0283] A first photometric error is determined based on the first image, the reconstructed image, and a similarity between the first image and the reconstructed image.

[0284] In a possible implementation manner, the training module 802 processes the event data stream data through the initial depth estimation model to obtain the first depth estimation data, specifically for:

[0285] Feature extraction is performed on the event data stream data to obtain feature graphs of i scales, where i is an integer greater than or equal to three.

[0286] The feature maps of i scales are fused and decoded to obtain a depth map of at least one scale.

[0287] Depth estimation data at at least one scale of the event data stream data is determined based on the depth map at at least one scale.

[0288] In a possible implementation, when the depth map of the at least one scale is a depth map of i-1 scales, the training module is specifically used to fuse and decode the feature maps of i scales to obtain the depth map of at least one scale:

[0289] The feature maps of i scales are fused i times to obtain fused feature maps of i scales. In the i-th fusion process, the fused feature map obtained by the i-1 fusion process is upsampled to obtain the fused feature map of the i-th fusion process. In the i-1 fusion process, the fused feature map obtained by the i-2 fusion process and the first feature map are fused to obtain the fused feature map of the i-1 fusion process. The first feature map is the feature map with the highest resolution among the feature maps of i scales. In the i-2 fusion process, the fused feature map, the first feature map, and the second feature map obtained by the i-3 fusion process are fused to obtain the fused feature map of the i-2 fusion process. The second feature map is the feature map with the second highest resolution among the feature maps of i scales. And so on, until in the first fusion process, the feature maps of i scales are fused to obtain the fused feature map of the first fusion process.

[0290] The fused feature maps of i-1 scales are decoded respectively to obtain depth maps of i-1 scales. The fused feature maps of i-1 scales are the fused feature maps of the fused feature maps of i scales except the fused feature map with the smallest feature map scale.

[0291] In a possible implementation, the training module 802 performs a fusion process on the fusion feature map obtained in the i-2 fusion process and the first feature map in the i-1 fusion process to obtain the fusion feature map of the i-1 fusion process, which is specifically used for:

[0292] The fused feature map obtained by the (i-2) fusion process is upsampled to obtain a first upsampled feature map.

[0293] The first up-sampled feature map and the first feature map are concatenated to obtain a fused feature map of the i-1th fusion processing.

[0294] In a possible implementation, the training module performs a fusion process on the fused feature map, the first feature map, and the second feature map obtained in the i-3 fusion process in the i-2 fusion process to obtain a fused feature map of the i-2 fusion process, specifically for:

[0295] The fused feature map obtained by the i-3th fusion process is upsampled to obtain a second upsampled feature map.

[0296] Downsampling is performed on the first feature map to obtain a downsampled feature map.

[0297] The second up-sampled feature map, the down-sampled feature map, and the second feature map are concatenated to obtain a fused feature map of the i-2th fusion process.

[0298] like Fig. 9 As shown, the depth estimation device 900 includes an acquisition module 901 and a processing module 902. The depth estimation device 900 is used to implement the above Figure 7 Alternatively, the depth estimation apparatus 900 may include a device for implementing the above-mentioned Figure 7 A module for any function or operation in the depth estimation method embodiment shown in the figure may be implemented in whole or in part by software, hardware, firmware or any combination thereof.

[0299] When the depth estimation device 900 is used to implement Figure 7 In the function of the method embodiment shown, the acquisition module 901 is used to acquire event data stream data, and the above event data stream data is obtained according to the event data stream captured by the event camera for a preset time length. The processing module 902 is used to process the event data stream data through the target depth estimation model to obtain depth estimation data of the event data stream data. The above target depth estimation model is a model trained based on the model training method shown in any of the above embodiments.

[0300] The depth estimation device 900 in the embodiment of the present application can obtain depth estimation data by using the event data stream of a preset time length and the target depth estimation model. The target depth estimation model is trained based on the model training method shown in any of the above embodiments, and the model training cost is low, which is conducive to reducing the cost of depth estimation.

[0301] In a possible implementation manner, the processing module 902 is specifically configured to:

[0302] The event data stream data is processed by the target depth estimation model to obtain a depth map of at least one scale of the event data stream data; and the depth estimation data is determined based on the depth map of at least one scale. The depth estimation device in the embodiment of the present application can obtain specific depth estimation data based on the depth map to achieve depth estimation.

[0303] An embodiment of the present application also provides a perception system, comprising the above-mentioned depth estimation device.

[0304] An embodiment of the present application also provides a vehicle, comprising the depth estimation device or the perception system as described above.

[0305] The present application also provides an electronic device, referring to Fig.10 , Fig.101 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 1000 includes a memory 1001, a processor 1002, a communication interface 1004 and a bus 1003. The memory 1001, the processor 1002 and the communication interface 1004 are connected to each other through the bus 1003.

[0306] The memory 1001 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1001 may store a program, and when the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 and the communication interface 1004 are used to execute the various steps of the model training method or the depth estimation method of any embodiment of the present application.

[0307] Processor 1002 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the model training device or depth estimation device of any embodiment of the present application, or to execute the model training method or depth estimation method of any embodiment of the present application.

[0308] Processor 1002 can also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the model training method or depth estimation method of any embodiment of the present application can be completed by the hardware integrated logic circuit or software instruction in the processor 1002. The above-mentioned processor 1002 can also be a general-purpose processor, a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the model training method or depth estimation method in combination with any embodiment of the present application can be directly embodied as a hardware processor execution, or the hardware and software modules in the processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1001, and the processor 1002 reads the information in the memory 1001, and combines its hardware to complete the functions required to be performed by the units included in the model training device or depth estimation device of any embodiment of the present application, or executes the model training method or depth estimation method of any embodiment of the present application.

[0309] The communication interface 1004 uses a transceiver such as, but not limited to, a transceiver to implement communication between the electronic device 1000 and other devices or a communication network. For example, an event data stream may be acquired through the communication interface 1004 .

[0310] The bus 1003 may include a path for transmitting information between various components of the electronic device 1000 (eg, the memory 1001 , the processor 1002 , and the communication interface 1004 ).

[0311] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0312] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0313] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0314] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0315] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: include: Acquire a sample set, the sample set including a first image and event data stream data, the event data stream data is obtained according to an event data stream captured by an event camera for a preset time length, and the first image is captured by a common camera at the end of the preset time length; An initial depth estimation model is trained according to the first image and the event data stream data to obtain a target depth estimation model.

2. The method according to claim 1, characterized in that The training of the initial depth estimation model according to the first image and the event data stream data to obtain a target depth estimation model includes: Processing the event data stream data by using the initial depth estimation model to obtain first depth estimation data; determining a first loss value of the first image based on the first image and the first depth estimation data; The initial depth estimation model is adjusted and optimized according to the first loss value to obtain the target depth estimation model.

3. The method according to claim 2, characterized in that The determining a first loss value of the first image according to the first image and the first depth estimation data comprises: Calculating a second loss value according to a first photometric error and a second photometric error; the first photometric error is calculated according to the first image, a frame image before the first image, and the first depth estimation data; and the second photometric error is calculated according to the first image, a frame image after the first image, and the first depth estimation data; Calculating a mask of the first image according to the first photometric error, the second photometric error, the third photometric error, and the fourth photometric error; the third photometric error is obtained according to the first image and a frame image before the first image, and the fourth photometric error is obtained according to the first image and a frame image after the first image; The first loss value is determined according to the second loss value and the mask.

4. The method according to claim 3, characterized in that The determining a first loss value of the first image according to the first image and the first depth estimation data further includes: determining the first photometric error according to the first image, first pose estimation data, and the first depth estimation data, wherein the first pose estimation data is camera pose transformation data between the first image and a frame image preceding the first image; The second photometric error is determined according to the first image, second pose estimation data, and the first depth estimation data, where the second pose estimation data is camera pose transformation data between the first image and a frame image subsequent to the first image.

5. The method according to claim 4, characterized in that The determining the first photometric error according to the first image, the first pose estimation data and the first depth estimation data comprises: Determine a reconstructed image according to the first image, the first pose estimation data, and the first depth estimation data; The first photometric error is determined based on the first image, the reconstructed image, and a similarity between the first image and the reconstructed image.

6. The method according to any one of claims 2 to 5, characterized in that: The processing of the event data stream data by using the initial depth estimation model to obtain first depth estimation data includes: Performing feature extraction on the event data stream data to obtain feature graphs of i scales, where i is an integer greater than or equal to three; Fusing and decoding the feature maps of the i scales to obtain a depth map of at least one scale; Depth estimation data at at least one scale of the event data stream data is determined based on the depth map at the at least one scale.

7. The method according to claim 6, characterized in that When the depth map of at least one scale is a depth map of i-1 scales, fusing and decoding the feature maps of the i scales to obtain a depth map of at least one scale includes: Performing i-fold fusion processing on the feature maps of the i-scales to obtain fused feature maps of the i-scales; In the i-th fusion process, the fusion feature map obtained by the i-1-th fusion process is upsampled to obtain the fusion feature map of the i-th fusion process; In the i-1th fusion process, the fused feature map obtained in the i-2th fusion process and the first feature map are fused to obtain the fused feature map of the i-1th fusion process, wherein the first feature map is the feature map with the largest resolution among the feature maps of the i-scales; In the (i-2) fusion process, the fusion feature map obtained in the (i-3) fusion process, the first feature map, and the second feature map are fused to obtain the fusion feature map of the (i-2) fusion process, wherein the second feature map is a feature map with the second largest resolution among the feature maps of the i-scales; And so on, until the first fusion process, the feature maps of the i scales are fused to obtain the fused feature map of the first fusion process; The fused feature maps of i-1 scales are decoded respectively to obtain the depth maps of the i-1 scales, and the fused feature maps of the i-1 scales are the fused feature maps of the i scales except the fused feature map with the smallest feature map scale.

8. The method according to claim 7, characterized in that In the i-1th fusion process, the fusion feature map obtained in the i-2th fusion process and the first feature map are fused to obtain the fusion feature map of the i-1th fusion process, including: Performing upsampling processing on the fused feature map obtained by the (i-2)th fusion processing to obtain a first upsampling feature map; The first up-sampled feature map and the first feature map are concatenated to obtain a fused feature map of the i-1th fusion processing.

9. The method according to claim 7 or 8, characterized in that: In the (i-2) fusion process, the fusion feature map obtained in the (i-3) fusion process, the first feature map, and the second feature map are fused to obtain the fusion feature map of the (i-2) fusion process, including: Performing upsampling processing on the fused feature map obtained by the (i-3) fusion processing to obtain a second upsampling feature map; Downsampling the first feature map to obtain a downsampled feature map; The second up-sampled feature map, the down-sampled feature map, and the second feature map are concatenated to obtain a fused feature map of the i-2th fusion processing.

10. A depth estimation method, characterized in that: include: Acquire event data stream data, wherein the event data stream data is obtained according to an event data stream captured by an event camera for a preset time length; The event data stream data is processed by a target depth estimation model to obtain depth estimation data of the event data stream data, wherein the target depth estimation model is a model trained based on the model training method according to any one of claims 1 to 9.

11. The method according to claim 10, characterized in that The processing of the event data stream data by using a target depth estimation model to obtain depth estimation data of the event data stream data includes: Processing the event data stream data by the target depth estimation model to obtain a depth map of at least one scale of the event data stream data; Depth estimation data for the event data stream data is determined based on the depth map at the at least one scale.

12. A model training device, characterized in that: include: an acquisition module, configured to acquire a sample set, wherein the sample set includes a first image and event data stream data, wherein the event data stream data is obtained according to an event data stream captured by an event camera for a preset time length, and the first image is captured by a common camera at the end of the preset time length; A training module is used to train an initial depth estimation model according to the first image and the event data stream data to obtain a target depth estimation model.

13. A depth estimation device, characterized in that: include: An acquisition module, used for acquiring event data stream data, wherein the event data stream data is obtained according to an event data stream captured by an event camera for a preset time length; A processing module is used to process the event data stream data through a target depth estimation model to obtain depth estimation data of the event data stream data, wherein the target depth estimation model is a model trained based on the model training method described in any one of claims 1 to 9.

14. A perception system, characterized in that: Comprising the depth estimation device as claimed in claim 13.

15. A vehicle, characterized in that: Comprising the depth estimation device as claimed in claim 13 or the perception system as claimed in claim 14.

16. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program codes, and the processor is used to call the program codes to execute the method according to any one of claims 1 to 11.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 11.

Images