Self-supervised based millimeter wave radar and camera fusion dense depth estimation method

Abstract

The embodiment of the application discloses a kind of based on self-supervision's millimeter wave radar and camera fusion dense depth estimation method, it is related to scene perception technical field.The method includes: the image sequence of camera in time alignment is input into the dense depth estimation model trained with the point cloud sequence of 4D millimeter wave radar, the convolution feature and uncertainty are obtained by the uncertainty branch processing image and point cloud pair;The depth code corresponding to the current frame is obtained, and the dense depth estimation branch is input into the convolution feature and the depth code, and the dense depth of each pixel unit on the image corresponding frame is decoded to obtain the dense depth estimation graph.The model is trained by the self-supervision method of the application, and the training cost caused by artificial labeling dense depth value is avoided;By constructing the geometric optimization problem between multiple image and point cloud pairs, the depth code is adjusted in the prediction process, so that it can have good depth estimation precision in complex application scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of scene perception, and in particular to a self-supervised millimeter wave radar and camera fusion dense depth estimation method. BACKGROUND

[0002] Depth estimation can provide key three-dimensional perception capabilities for positioning and navigation systems. In application scenarios such as autonomous driving and unmanned aerial vehicle navigation, depth estimation can help the system accurately understand the three-dimensional structure of the environment, including obstacle, road, and other information, so as to realize accurate positioning and path planning, and improve the safety and reliability of the system.

[0003] Currently, depth estimation can be performed by methods based on image data or fusion of image and lidar. However, environmental factors such as rain, snow, fog, and dust may affect the performance of traditional sensors. In harsh environmental conditions, sensors may be severely affected and unable to accurately identify obstacles. Since the millimeter wave has a longer wavelength, the 4D millimeter wave radar can penetrate smoke and dust and is not affected by light, so the method of fusing 4D millimeter wave radar point cloud and image data is more suitable for depth estimation in all-weather conditions. However, compared with cameras and lidars, the point cloud obtained by 4D millimeter wave radar is sparse, noisy, and has a narrow detection field of view (generally within 120°), which makes it difficult to accurately reconstruct the details of the scene. The current 4D millimeter wave radar camera fusion dense depth estimation method has high computational consumption and relatively slow inference speed, which may limit the performance of real-time applications. Moreover, the calculation by the deep learning model often relies on real dense depth values, which need to be manually labeled, resulting in high labor costs and difficulty in obtaining. Related technologies for fusing 4D millimeter wave radar point cloud and image data for dense depth estimation are usually limited to processing of single-frame radar point cloud and camera image pairs, without effectively utilizing the time-series co-visibility information. Therefore, there is still a lack of an efficient and accurate depth estimation method suitable for complex scenes. SUMMARY

[0004] The embodiments of the present application provide a self-supervised millimeter wave radar and camera fusion dense depth estimation method to solve the defects in the related art. The technical solution is as follows:

[0005] In a first aspect, the embodiments of the present application provide a self-supervised millimeter wave radar and camera fusion dense depth estimation method, comprising:

[0006] obtaining an image sequence of a camera and a point cloud sequence of a 4D millimeter wave radar;

[0007] align the image sequence and the point cloud sequence, output an aligned image sequence and an aligned point cloud sequence;

[0008] input the aligned image sequence and the aligned point cloud sequence into an uncertainty branch of the trained dense depth estimation model, encode features of each frame of the aligned image sequence and a corresponding frame of the aligned point cloud sequence to obtain convolutional features through the uncertainty branch, and decode the convolutional features to obtain uncertainty of each pixel unit in the corresponding frame of the image;

[0009] obtain a depth code corresponding to the current frame, input the depth code and the convolutional features into a dense depth estimation branch of the dense depth estimation model, combine the depth code and the convolutional features through the dense depth estimation branch, decode to obtain dense depth of each pixel unit on the corresponding frame of the image, and output a dense depth estimation map.

[0010] In an optional implementation of the first aspect, obtaining the depth code corresponding to the current frame comprises:

[0011] if the current frame is the first frame in the image sequence, obtaining a preset initial depth code;

[0012] otherwise, optimizing the depth code, comprising:

[0013] obtaining uncertainty, a depth code and a dense depth estimation map corresponding to a plurality of key frame images before the current frame in the image sequence;

[0014] constructing a target function for optimizing the depth code based on a difference between the current frame and each key frame image before the current frame; the target function comprises photometric error of each key frame image before the current frame, re-projection error of matching feature points, and re-projection error of the millimeter wave radar;

[0015] combining the target function and a prior distribution constraint of the depth code through a least square solver to calculate camera poses of the current frame and each key frame image before the current frame and the optimized depth code.

[0016] In an optional implementation of the first aspect, the constructing the target function for optimizing the depth code comprises:

[0017] projecting the current frame onto each key frame image before the current frame respectively, calculating light intensity difference on corresponding pixel units, and constructing photometric error;

[0018] After projecting the current frame onto each key frame image before the current frame respectively, at least one pair of matching feature points between two frames is selected, a difference value of pixel coordinates of the matching feature points is calculated, and a re-projection error of the matching feature points is constructed.

[0019] The radar sparse points of the current frame are projected onto the corresponding dense depth estimation map of each key frame image before the current frame, the depth value of the pixel unit where the radar sparse point is located is compared with the depth value of the corresponding pixel unit on the dense depth estimation map, and a re-projection error of the millimeter wave radar is constructed.

[0020] In an optional implementation of the first aspect, the aligning the image sequence and the point cloud sequence and inputting the aligned image sequence and point cloud sequence into the trained dense depth estimation model comprises:

[0021] The timestamps of the image sequence and the point cloud sequence are acquired, and the time sequence of the image sequence and the point cloud sequence is aligned according to the timestamps;

[0022] The external parameters of the camera and the 4D millimeter wave radar are acquired, the point cloud of the corresponding frame in the point cloud sequence is subjected to external parameter transformation according to the external parameters, and a point cloud depth map in the camera coordinate system is obtained, so as to perform spatial alignment on the image sequence and the point cloud sequence;

[0023] The point cloud sequence after the time sequence alignment and the spatial alignment is cropped, and the size of each frame of image and the point cloud of the corresponding frame is unified;

[0024] The cropped image sequence and point cloud sequence are input into the trained dense depth estimation model;

[0025] Each pixel unit on each frame of point cloud can correspond to a pixel unit on the corresponding frame of image, and it is noted that some pixel units of the point cloud map do not have effective depth and can be marked by Inf or 0.

[0026] In an optional implementation of the first aspect, the dense depth estimation model is trained by the following steps, comprising:

[0027] The teacher network model is trained, specifically comprising:

[0028] The sample image sequence of the camera and the sample point cloud sequence of the 4D millimeter wave radar are used as training samples, the sparse depth map of each frame of point cloud in the sample point cloud sequence is used as a supervision sample of the teacher network model, the teacher network model is trained, so that the teacher network model learns the relationship between the depth information on each pixel unit in each frame of sample image in the sample image sequence and the image features, and outputs a trained teacher network model;

[0029] training the student network model, specifically comprising:

[0030] using a sample image sequence of a camera and a sample point cloud sequence of a 4D millimeter wave radar as training samples, using a teacher dense depth estimation map of each sample image predicted by the trained teacher network model as a supervision sample of the student network model, and training the student network model;

[0031] outputting the trained student network model as the dense depth estimation model.

[0032] In an optional implementation of the first aspect, the loss function for training the teacher network model comprises:

[0033] obtaining the teacher dense depth estimation map of each sample image predicted by the teacher network model, obtaining a sparse depth map of a point cloud corresponding to the sample image in the sample point cloud sequence, and constructing a sparse depth loss function based on a difference between an effective depth value on each pixel unit in the sparse depth map and a depth value of a corresponding pixel unit in the teacher dense depth estimation map of each sample image;

[0034] transforming a sample image of a next adjacent frame of the current frame to the current frame based on a preset pose transformation and a dense depth estimation map of the sample image of the current frame to obtain a transformed image of the sample image of the current frame, and constructing a photometric consistency loss function based on the transformed image and the sample image of the current frame;

[0035] training the teacher network model based on the sparse depth loss and the photometric consistency loss, and outputting the trained teacher network model.

[0036] In an optional implementation of the first aspect, training the student network model and outputting the trained student network model as the dense depth estimation model comprises:

[0037] obtaining a sample depth code of each sample image in the sample image sequence obtained by the dense depth estimation branch of the student network model through feature encoding, and constructing a KL divergence loss function based on a difference between the sample depth code corresponding to each sample image and a standard normal distribution;

[0038] obtaining a student dense depth estimation map output by the dense depth estimation branch of the student network model, and constructing a dense depth loss function of all pixel units based on a difference between a depth value on each pixel unit in the student dense depth estimation map output by the student network model and a depth value on a corresponding pixel unit in the teacher dense depth estimation map;

[0039] obtain a sparse depth map of the point cloud corresponding to the student dense depth estimation map in the sequence of sample point clouds, and construct a sparse depth loss function based on a difference between a depth value on each pixel unit in the sparse depth map and a depth value of a corresponding pixel unit in the student dense depth estimation map;

[0040] train the student network model based on the KL divergence loss, the dense depth prediction loss and the sparse depth loss, and output the converged student network model as the dense depth estimation model.

[0041] In a second aspect, the embodiments of the present application further provide a device for self-supervised millimeter wave radar and camera fusion dense depth estimation, comprising:

[0042] a data module configured to obtain a sequence of images of a camera and a sequence of point clouds of a 4D millimeter wave radar;

[0043] The data module is further configured to align the sequence of images and the sequence of point clouds, output the aligned sequence of images and the sequence of point clouds, and input the aligned sequence of images and the sequence of point clouds into an uncertainty branch of the trained dense depth estimation model;

[0044] an uncertainty calculation module configured to perform feature coding on each frame of the aligned sequence of images and a point cloud of a corresponding frame in the aligned sequence of point clouds through the uncertainty branch of the dense depth estimation model, so as to obtain convolutional features, and then calculate an uncertainty of each pixel in a corresponding image frame by decoding the convolutional features;

[0045] a dense depth estimation module configured to obtain a depth code corresponding to a current frame, input the depth code and the convolutional features into a dense depth estimation branch of the dense depth estimation model, decode each pixel unit on a corresponding image frame to obtain a dense depth of each pixel unit on the corresponding image frame through the dense depth estimation branch in combination with the depth code and the convolutional features, and output a dense depth estimation map.

[0046] a depth code optimization module configured to utilize multi-frame co-visibility information to construct a least square loss function to optimize depth codes of a historical key frame and a current frame, so as to provide more accurate depth codes for the dense depth estimation module. The loss function includes photometric errors between the current frame and a plurality of previous key frames, re-projection errors of matching feature points, re-projection errors of the millimeter wave radar, and a Gaussian prior factor of the depth code.

[0047] In a third aspect, the embodiments of the present application further provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the method provided by the first aspect or any one of the implementation manners of the first aspect.

[0048] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method provided in the first aspect of the embodiments of the present application or any of the implementation manners of the first aspect.

[0049] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0050] The millimeter wave radar and camera fusion dense depth estimation method based on self-supervision provided by the embodiments of the present application can realize the fusion of sparse depth and image information dense depth prediction by combining image sequences and point cloud sequences. In view of the problem that the existing depth estimation method needs dense depth supervision, the input data self-supervision method is used to train the dense depth estimation model, the teacher network model is trained by the sparse depth information collected by the 4D millimeter wave radar in the limited training samples, and the student network model is trained by the predicted dense depth output by the teacher network model, thereby avoiding the defect that the related art needs manual annotation of dense depth. On the other hand, based on the depth code least square optimization method of digital and analog joint, the spatial common view constraint can be formed by using the multi-frame information in time sequence to optimize the estimation of the depth code, so as to obtain more accurate dense depth estimation of the current frame. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical scheme in the present application or related art, the drawings needed to be used in the embodiments or related art description will be briefly introduced as follows. Obviously, the drawings in the following description are the drawings of the embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creating any creative labor.

[0052] Figure 1 is a flowchart of a millimeter wave radar and camera fusion dense depth estimation method based on self-supervision according to an embodiment of the present application;

[0053] Figure 2 is a structure diagram of a dense depth estimation student network model according to an embodiment of the present application;

[0054] Figure 3 is a structure diagram of a dense network estimation teacher network model according to an embodiment of the present application;

[0055] Figure 4 is a teacher network self-supervision training flowchart according to an embodiment of the present application;

[0056] Figure 5 is a student network self-supervision training flowchart according to an embodiment of the present application;

[0057] Figure 6FIG. 1 is a structural schematic diagram of a device for dense depth estimation based on self-supervised fusion of millimeter wave radar and camera according to an embodiment of the present application.

[0058] Figure 7 FIG. 2 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] For the purpose of clarity, technical solutions and advantages of the present application will be described in detail below with reference to the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0060] The terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above-described drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, but can optionally include other steps or modules not listed, or can optionally include other steps or modules inherent to the process, method, product or device.

[0061] Related deep network-based technologies generally need to rely on real dense depth values to train dense depth estimation models. However, obtaining real dense depth values corresponding to each pixel unit on an image often requires a large amount of manual work, and it is difficult to generate enough supervised training samples in a short time, thereby limiting the implementation of large-scale data training. Although reducing the number of samples or reducing the number of pixel units of the image can reduce the workload, this will limit the training process of the dense depth estimation model, and the prediction accuracy of the dense depth estimation model obtained by training is difficult to achieve the expected result.

[0062] In addition, in the training process of the current dense depth estimation model, only the information of the current frame is considered, i.e., only one pair of radar point cloud and camera frame is considered, without considering the correlation information between each frame and adjacent frames in a multi-frame sequence. In a complex scene, for example, when an autonomous vehicle enters a tunnel from the outside, the light intensity changes greatly. If only the information of the current frame is considered without considering the correlation information between each frame and adjacent frames, the dense depth estimation model will output results with large differences between the current frame and adjacent frames. Therefore, the current dense depth estimation method has poor robustness in complex application scenarios, and the accuracy of the output results is limited.

[0063] The present application will be described in detail below with reference to specific embodiments.

[0064] Next, the present application will be described in detailFigure 1 and Figure 2 , the self-supervised based millimeter wave radar and camera fusion dense depth estimation method provided by the embodiments of the present application is introduced. For details, please refer to Figure 2 , Figure 2 The dense depth estimation student network model structure diagram provided by the embodiments of the present application is shown. As Figure 1 shown, the method comprises the following steps:

[0065] S101, acquiring an image sequence of a camera and a point cloud sequence of a 4D millimeter wave radar;

[0066] S102, aligning the image sequence and the point cloud sequence, and outputting the aligned image sequence and point cloud sequence;

[0067] S103, inputting the aligned image sequence and point cloud sequence into an uncertainty branch of a trained dense depth estimation model, performing feature encoding on each frame of image in the aligned image sequence and the corresponding frame of point cloud in the aligned point cloud sequence through the uncertainty branch to obtain convolution features, and decoding the convolution features to obtain the uncertainty of each pixel unit in the corresponding frame of image;

[0068] S104, acquiring a depth code corresponding to the current frame, inputting the depth code and the convolution features into a dense depth estimation branch of the dense depth estimation model, decoding to obtain the dense depth of each pixel unit on the corresponding frame of image through the dense depth estimation branch combined with the depth code and the convolution features, and outputting a dense depth estimation map.

[0069] S105, using multi-frame common view information to construct a least squares loss function to optimize the depth codes of the historical key frames and the current frame, so as to provide more accurate depth codes for subsequent frames.

[0070] In the embodiments of the present application, the image sequence of the camera acquired in S101 includes multiple frames of images of the camera shot on a preset shooting time on a to-be-estimated region, and the point cloud sequence of the 4D millimeter wave radar includes multiple frames of point clouds output by the 4D millimeter wave radar after detecting the same to-be-estimated region at a preset detection frequency.

[0071] It can be understood that, in the data acquisition process, due to the differences in the acquisition start time, acquisition end time, acquisition frequency and other parameters of the camera and the 4D millimeter wave radar, therefore, before predicting the dense depth by fusing the image and the corresponding radar point cloud frame, the step of S102 needs to be performed to perform consistency preprocessing on the sequences acquired by the two sensors in time and space.

[0072] In the embodiments of the present application, in S102, the timestamps of the image sequence and the point cloud sequence can be acquired, and the time sequences of the image sequence and the point cloud sequence are aligned according to the timestamps, so as to ensure that each frame of image and each frame of point cloud are aligned in the acquisition time.

[0073] In the embodiments of the present application, in some cases, the acquisition frequencies of the image sequence and the point cloud sequence can be different, and there can be images and point clouds that are not aligned at some time points. Motion compensation calculation can be performed on the missing point cloud data to fill in the missing images or point clouds at the corresponding time points.

[0074] In the embodiments of the present application, in S102, the external parameters of the camera and the 4D millimeter wave radar can be acquired, the point cloud of the corresponding frame in the point cloud sequence is subjected to external parameter transformation according to the external parameters, and is projected to the camera coordinate system, and then is cropped, so as to align the image sequence and the point cloud sequence in space, that is, the spatially aligned image sequence and point cloud sequence are obtained, so that each pixel unit on the projection map of each frame of point cloud can correspond to the pixel unit on the corresponding frame of image. Since the millimeter wave radar point cloud is sparse, some depth values on the point cloud projection map can be invalid values.

[0075] Exemplarily, the size-unified 224x224 image and the point cloud of the millimeter wave radar can be obtained, wherein the point cloud of the millimeter wave radar is stored in the format of sparse depth map.

[0076] It should be noted that the point cloud of the millimeter wave radar gives the sparse depth map of the corresponding frame, that is, the depth detection values of some pixel units are given. For the pixel units that do not have depth detection values at the corresponding pixel unit positions in the sparse depth map, the depth values can be set to 0 or Inf.

[0077] Further, the aligned image sequence and point cloud sequence are input into the trained dense depth estimation model to perform the steps of S103, which specifically include:

[0078] Exemplarily, the trained dense depth estimation model is as shown in Figure 2 Taking the image type in the input image sequence as an RGB image as an example, the RGB image sequence and the point cloud sequence are input into the uncertainty branch of the dense depth estimation model.

[0079] Specifically, one frame of image of the RGB image sequence and the point cloud of the same frame in the point cloud sequence are input into the uncertainty branch of the dense depth estimation model in pairs, and the aligned data is first processed by an encoding-decoding network, wherein the encoder adopts a lightweight convolutional network MobileNet to perform feature encoding on the image frame from the camera and the sparse depth data (i.e., the point cloud corresponding to the image frame) from the millimeter wave radar, extracts convolutional features, and transmits the information in the encoder to the decoder through a skip connection to determine the uncertainty of each pixel unit in the corresponding frame of image.

[0080] As can be understood, the uncertainty is used to represent the accuracy of the depth value on each pixel unit in the image, and the higher the uncertainty, the lower the accuracy of the depth value corresponding to the pixel unit.

[0081] Further, the RGB image sequence and the convolutional features extracted by the uncertainty branch are input into the dense depth estimation branch of the dense depth estimation model to perform the steps of S104, specifically including:

[0082] According to the depth code corresponding to the image of the current frame, in combination with the convolutional features of the image of the current frame extracted by the uncertainty branch, the depth of each pixel unit on the corresponding frame of image is decoded by the decoder of the dense depth estimation branch, and a dense depth estimation image is output.

[0083] As can be understood, the dense depth estimation model learns the mapping relationship between the input image and the point cloud and the depth information and the probability distribution thereof in the training process, thereby predicting the depth value on each pixel unit.

[0084] It should be noted that the dense depth estimation model converts the input camera image and radar data into a low-dimensional latent variable representation, i.e., a depth code, through the encoder part, such as a lightweight convolutional network MobileNet. The depth code can be regarded as a compression or abstraction of the input data, and contains features related to depth information, i.e., the dense depth is a function of the depth code.

[0085] It should be noted that for the first frame of image in the image sequence to be processed, a preset initial depth code is selected. The initial depth code can be obtained in the training process of the dense depth estimation model, or can be set according to the actual scene, and the embodiments of the present application do not limit this.

[0086] In order to further improve the accuracy of depth estimation, the embodiments of the present application can further optimize the depth code using the time sequence frame information, thereby realizing the dense depth estimation under the double driving of data and model.

[0087] Therefore, after S104, the following steps can also be performed:

[0088] S105, construct a least square loss function using the multi-frame co-visibility information to optimize the depth code of the historical key frames and the current frame.

[0089] Specifically, for the images after the first frame in the image sequence to be processed, the depth code can be optimized according to the differences between the image of the current frame and the images of the previous partial key frames in terms of depth code, dense depth estimation map, etc., and the optimization specifically includes:

[0090] obtaining the uncertainty, depth code and dense depth estimation map corresponding to each key frame image before the current frame in the image sequence;

[0091] constructing a target function for optimizing the depth code and camera pose based on the differences between the current frame and each key frame image before the current frame;

[0092] calculating the optimized camera pose of the current frame and each key frame image before the current frame and the optimized depth code by combining the target function and the prior distribution constraint of the depth code through a least square solver.

[0093] Specifically, the geometric shape of the scene can be represented by the depth map D m of the camera key frame, and the set G is expressed as G={D m-k ,D m-k+1 ,..,D m}, the key frame can be understood as a number of image frames containing scene information extracted from the image sequence, and the depth map corresponding to the key frame can be generated by the dense depth estimation model and the depth code to be optimized, thus the variables of the multi-frame dense depth optimization problem only include the poses and related depth codes c i of k+1 camera key frames.

[0094] Among them, the key frame can be obtained according to some rules, for example, if the camera moves or rotates more than a certain limit value, a key frame is selected. For example, if the distance of the current frame relative to the last key frame is more than 0.3m or the angle of rotation is more than 20°, the current frame is selected as a new key frame.

[0095] Among them, the target function for optimizing the depth code includes the following items:

[0096] The photometric error of each key frame image before the current frame, the re-projection error of the matching feature points, the re-projection error of the millimeter wave radar, and the Gaussian prior factor of the depth code.

[0097] Specifically, the photometric error includes:

[0098] The current frame is projected onto each key frame image before the current frame respectively, the light intensity difference on the corresponding pixel unit is calculated, and a target function of photometric error is constructed. The formula is:

[0099]

[0100] The photometric re-projection error is The light intensity difference between the pixel I i of the current frame image and the pixel I j of the target image transformed to the frame i is constrained, and Ω i is the pixel set on the i-th frame image.

[0101]

[0102] Wherein, ω ji is the transformation function of the pixel point x in the i-th frame to the pixel point in the j-th frame, π and π -1 are the camera projection and inverse projection functions respectively, is the dense depth estimation map decoded from the depth code c i , T ji is the pose transformation from the i-th frame to the j-th frame, and c i is the depth code of the i-th frame. It is noted that D i (x) also depends on the convolution features of the image I i and the sparse depth map of the millimeter wave radar.

[0103] Specifically, the re-projection error of the matching feature points includes:

[0104] The current frame is projected onto each key frame image before the current frame respectively, at least one pair of matching feature points between two frames is selected, the difference value of the pixel coordinates of the matching feature points is calculated, and a target function of the re-projection error of the matching feature points is constructed. The re-projection error measures the difference between the feature points observed by the j-th frame image and the pixel coordinates predicted from the i-th frame image. The formula is:

[0105]

[0106] Wherein, M ij is the set of matching feature point pairs in the i-th frame and the j-th frame image, and y is the matching feature point coordinates in the previous j-th frame image.

[0107] It should be noted that the matching feature points can be determined by the BRISK algorithm.

[0108] Specifically, the re-projection error of the millimeter wave radar includes:

[0109] For the millimeter wave radar depth information​ frame i and frame j containing the network predicted depth map D j frame j, pixel x in frame i corresponds to pixel Sparse depth reprojection error, i.e., the difference between the millimeter wave radar observed depth and the network predicted depth of the two corresponding pixels:

[0110] Project the radar sparse points of the current frame onto the dense depth estimation map corresponding to each key frame image before the current frame, compare the depth values of the pixel units where the radar sparse points are located with the depth values of the corresponding pixel units on the dense depth estimation map, and construct a target function of the millimeter wave radar reprojection error, apply the formula:

[0111]

[0112] wherein, is the sparse depth reprojection error, [x] z represents the z component of the x vector.

[0113] Specifically, the Gaussian prior factor of the depth code ensures that the depth code distribution of each millimeter wave radar point cloud and image frame pair i is close to a Gaussian distribution:

[0114]

[0115] Based on the above constraints, the optimized camera pose and depth code of the current frame and the historical key frames can be calculated.

[0116] The decoder of the dense depth estimation model in the embodiment of the application can use the updated depth code to more accurately predict the dense depth of the current frame.

[0117] In the embodiment of the application, the dense depth estimation model can be trained by setting a teacher network model and a student network model, specifically including:

[0118] Training the teacher network model, using the sample image sequence of the camera and the sample point cloud sequence of the 4D millimeter wave radar as training samples, using the sample point cloud sequence as a supervision sample, training the teacher network model, and outputting the trained teacher network model;

[0119] Training the student network model, using the sample image sequence of the camera and the sample point cloud sequence of the 4D millimeter wave radar as training samples, using the teacher dense depth estimation map output by the trained teacher network model as a supervision sample, training the student network model, and outputting the trained student network model as the dense depth estimation model.

[0120] Specifically, the teacher network model is used to predict the dense depth as the dense depth supervision signal for training the student network, and the teacher network model is as follows:Figure 3 As shown, this teacher network employs an encoder-decoder structure. The encoder consists of ResNet-34 residual blocks, which progressively reduce the spatial resolution of the feature maps through convolution. The decoder, on the other hand, progressively increases the spatial resolution of the feature maps through transposed convolutional layers. The input sparse depth and RGB images are initially convolved to obtain shallow features, which are then concatenated into a single tensor as the input to the ResNet-34 residual blocks. The output of the encoder layer is passed to the corresponding decoder layer via skip connections for fusion to prevent the loss of shallow information.

[0121] Specifically, using, such as Figure 4 The self-supervised framework shown is used to train the teacher network model. This framework uses aligned camera sample image sequences and 4D millimeter-wave radar sample point cloud sequences as input training samples, and the sample point cloud sequences as supervision samples. That is, the sparse depth information provided by the point cloud is used as supervision information to train the teacher network model, enabling the teacher network model to learn the relationship between depth information and image features at each pixel unit in each frame of the sample image sequence, and outputting the trained teacher network model.

[0122] Specifically, the loss function used for self-supervised training of the teacher network model consists of three parts: sparse depth prediction loss, edge-aware smoothing loss, and neighboring RGB image reprojection luminosity consistency loss.

[0123] Specifically, the dense depth estimation map of each sample image predicted by the teacher network model is obtained, and the sparse depth map of the point cloud corresponding to the sample image in the sample point cloud sequence is obtained. A sparse depth loss function is constructed based on the difference between the depth value of each pixel unit in the sparse depth map and the depth value of the corresponding pixel unit in the dense depth estimation map of the teacher.

[0124]

[0125] in, This indicates that only depth-valid points are considered. It is an L2 norm.

[0126] To smooth the generated inverse depth map, the edge-aware smoothing loss is defined as:

[0127]

[0128] in Indicated by mean Normalized inverse depth to avoid shrinkage in estimated depth.

[0129] Specifically, based on the preset pose transformation T 1→2and the dense depth estimation map pred1 of the sample image of the current frame 1 transforms the sample image of the next adjacent frame 2 of the current frame 1 to the current frame, specifically, given the intrinsic matrix K of the camera, any pixel p1 in the current frame 1 has a corresponding projection in the next adjacent frame 2:

[0130] p2=KT 1→2 pred1(p1)K -1 p1;

[0131] Therefore, by using the four adjacent pixels of p2 for bilinear interpolation, a corresponding RGB image of the current frame 1 can be synthesized to obtain the transformed image of the current frame 1:

[0132] warped1(p1)=bilinear(RGB2(KT 1→2 pred1(p1)K -1 p1));

[0133] Based on the transformed image and the sample image of the current frame 1, a photometric consistency loss function is constructed, when the environment remains static and the occlusion caused by the change of the view angle is limited, the image warped1 obtained by reverse transformation from the next adjacent frame 2 is similar to the current frame 1, and the photometric consistency loss is:

[0134]

[0135] wherein S is a scale factor of all pixels, (·) (s) indicates that the image is scaled by a factor s, and it should be noted that the photometric loss is only considered for the points without depth supervision, i.e., only for the pixel units {d=0} without sparse depth information.

[0136] Specifically, the model can be trained according to the sparse depth loss, the smoothness loss and the photometric consistency loss to obtain an optimized teacher network model.

[0137] In the embodiments of the present application, the teacher dense depth estimation map of each frame of sample image predicted by the trained teacher network model is taken as a supervision sample to supervise and train the student network model, including:

[0138] Specifically, as Figure 5As shown, the sample image sequence of the camera and the sample point cloud sequence of the 4D millimeter wave radar are input into the self-supervised depth reconstruction network as training sample inputs, the self-supervised depth reconstruction network is a teacher network model, and a teacher dense depth estimation map predicted by the teacher network model is obtained; the sample image sequence of the camera and the sample point cloud sequence of the 4D millimeter wave radar and the teacher dense depth estimation map are input into the student network model as training sample inputs, and a training loss function is constructed based on the output of the student network model and the output of the teacher network model, including a KL divergence loss, a dense depth prediction loss and a sparse depth prediction loss.

[0139] In the embodiments of the present application, the dense depth estimation branch of the student network model can obtain a sample depth code by performing feature encoding on each frame of sample image in the sample image sequence, and a KL divergence loss function is constructed based on the difference between the sample depth code corresponding to each frame of sample image and the standard normal distribution. The KL divergence is used to reduce the difference between the depth code distribution and the standard normal distribution. The formula is as follows:

[0140]

[0141] Wherein, n is the dimension of the depth code, μ and σ represent the mean and variance of the depth code, μ j and σ j represent the jth element in the two vectors.

[0142] In the embodiments of the present application, the uncertainty output by the uncertainty branch of the student network model can be obtained, the student dense depth estimation map output by the dense depth estimation branch of the student network model can be obtained, and based on the uncertainty and the difference between the depth value of each pixel unit on the teacher dense depth estimation map and the depth value of the corresponding pixel unit on the student dense depth estimation map, a dense depth prediction loss function of all pixel units is constructed. The formula is as follows:

[0143]

[0144] Wherein, x represents a pixel in the output feature map, D(·) is the value of the teacher dense depth estimation map, is the logarithm of the inverse depth value predicted from the student network model, is the logarithm of the depth uncertainty.

[0145] It should be noted that in the training, the loss value of each batch is averaged, and the predicted depth value is also thresholded, so as to prevent the appearance of infinite logarithmic value.

[0146] In the embodiments of the present application, a sparse depth map of the point cloud corresponding to the student dense depth estimation map in the sample point cloud sequence is obtained, a sparse depth loss function is constructed based on the difference between the depth value of each pixel unit in the sparse depth map and the depth value of the corresponding pixel unit in the student dense depth estimation map, and the formula is applied:

[0147]

[0148] wherein, D R is the sparse depth value of the millimeter wave radar point cloud, is the predicted depth value output by the student network model at the same pixel unit position.

[0149] Further, based on the KL divergence loss function, the dense depth prediction loss function and the sparse depth loss function, the student network model is trained by an optimization method such as Adam, and the output is a dense depth estimation model.

[0150] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0151] Next, please refer to Figure 6 , the structure diagram of the self-supervised millimeter wave radar and camera fusion dense depth estimation device provided by an exemplary embodiment of the present application. The device can be realized by software, hardware or a combination of the two to become all or part of the terminal, and can also be integrated as an independent module on the server. The self-supervised millimeter wave radar and camera fusion dense depth estimation device 60 in the embodiments of the present application includes a data module 601, an uncertainty calculation module 602, a dense depth estimation module 603 and a depth code optimization module 604, wherein:

[0152] The data module 601 is used to obtain the image sequence of the camera and the point cloud sequence of the 4D millimeter wave radar;

[0153] The data module 601 is also used to align the image sequence and the point cloud sequence, output the aligned image sequence and point cloud sequence, and input the aligned image sequence and point cloud sequence into the uncertainty branch of the trained dense depth estimation model;

[0154] The uncertainty calculation module 602 obtains the convolution features of each frame of image in the aligned image sequence and the corresponding frame of point cloud in the aligned point cloud sequence by feature encoding through the uncertainty branch of the dense depth estimation model, and obtains the uncertainty of each pixel unit in the corresponding frame of image by decoding the convolution features;

[0155] The dense depth estimation module 603 is configured to obtain a depth code corresponding to a current frame, input the depth code and the convolutional feature into a dense depth estimation branch of the dense depth estimation model, decode dense depth of each pixel unit on an image of the corresponding frame by combining the depth code and the convolutional feature through the dense depth estimation branch, and output a dense depth estimation image.

[0156] The depth code optimization module 604 is configured to construct a least square loss function by using multi-frame common view information, and optimize the depth code of the historical key frame and the current frame, so as to provide more accurate depth code for the dense depth estimation module.

[0157] It should be noted that the apparatus 60 provided in the above embodiments is only used as an example to divide the above functions, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above functions. In addition, the apparatus provided in the above embodiments and the method for self-supervised millimeter wave radar and camera fusion dense depth estimation belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be described here.

[0158] The embodiments of the present application also provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method of any one of the above embodiments when executing the program.

[0159] Please refer to Figure 7 , which is a structural block diagram of an electronic device provided in the embodiments of the present application.

[0160] As shown in Figure 7 , the electronic device 700 includes a processor 701 and a memory 702.

[0161] In the embodiments of the present application, the processor 701 is the control center of the computer system, which can be a processor of a physical machine or a processor of a virtual machine. The processor 701 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 can be implemented in at least one of the following hardware forms: a DSP (Digital Signal Processing), a FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array).

[0162] The processor 701 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state.

[0163] The memory 702 can include one or more computer-readable storage media that can be non-transitory. The memory 702 can also include high-speed random access memory and nonvolatile memory such as one or more disk storage devices, flash storage devices. In the embodiments of the present application, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one instruction for being executed by the processor 701 to implement the method in the embodiments of the present application.

[0164] In the embodiments of the present application, the electronic device 700 further includes a peripheral device interface 703 and at least one peripheral device 704. The processor 701, the memory 702 and the peripheral device interface 703 can be connected through a bus or a signal line. Each peripheral device 704 can be connected to the peripheral device interface 703 through a bus, a signal line or a circuit board. Specifically, the peripheral device interface 703 in the display screen, the camera and the millimeter wave radar can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 701 and the memory 702.

[0165] In the embodiments of the present application, the processor 701, the memory 702 and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 701, the memory 702 and the peripheral device interface 703 can be implemented on a separate chip or circuit board. The embodiments of the present application do not make specific limitations on this.

[0166] The electronic device structure block diagram shown in the embodiments of the present application does not constitute a limitation on the electronic device 700. The electronic device 700 can include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0167] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The program is executed by a processor to implement the steps of the method of any of the preceding embodiments. The computer readable storage medium can include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nano system (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0168] Those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary universal hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions or the part of the related art which is essentially contributed can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the method described in each embodiment or some part of the embodiment.

[0169] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A dense depth estimation method based on self-supervised fusion of millimeter-wave radar and camera, characterized in that, include: Acquire image sequences from the camera and point cloud sequences from the 4D millimeter-wave radar; Find image frames and point cloud frames that are close in time to the image sequence and the point cloud sequence, and transform the millimeter-wave radar point cloud to the image coordinate system based on extrinsic parameters to obtain an aligned sequence of image and point cloud frame pairs; The aligned image sequence and point cloud sequence are input into the uncertainty branch of the trained dense depth estimation model. The uncertainty branch is used to encode the features of each frame in the aligned image sequence and the corresponding frame in the aligned point cloud sequence to obtain convolutional features. The uncertainty of each pixel unit in the corresponding frame image is obtained by decoding the convolutional features. Obtain the depth code corresponding to the current frame, input the depth code and the decoded convolutional features into the dense depth estimation branch of the dense depth estimation model, decode to obtain the dense depth of each pixel unit in the image of the corresponding frame, and output the dense depth estimation map; The step of obtaining the depth code corresponding to the current frame includes: If the current frame is the first frame in the image sequence, then a preset initial depth code is obtained; Otherwise, optimize the depth code, including: Obtain the uncertainty, depth code, and dense depth estimation map corresponding to multiple keyframe images preceding the current frame in the image sequence; An objective function for optimizing the depth code is constructed based on the differences between the current frame and each keyframe image preceding the current frame; the objective function includes the photometric error of each keyframe image preceding the current frame, the reprojection error of matching feature points, and the reprojection error of the millimeter-wave radar; By combining the objective function and the prior distribution constraints of the depth code with the least squares solver, the optimized camera pose and optimized depth code of the current frame and each previous keyframe image are calculated.

2. The dense depth estimation method based on self-supervised millimeter-wave radar and camera fusion according to claim 1, characterized in that, The construction of the objective function for optimizing depth codes includes: The current frame is projected onto each keyframe image preceding the current frame, and the light intensity difference on the corresponding pixel unit is calculated to construct the photometric error. After projecting the current frame onto each keyframe image preceding the current frame, at least one pair of matching feature points between the two frames is selected, the difference in pixel coordinates of the matching feature points is calculated, and the reprojection error of the matching feature points is constructed. The radar sparse points of the current frame are projected onto the dense depth estimation map corresponding to each keyframe image before the current frame. The depth value of the pixel unit where the radar sparse point is located is compared with the depth value of the corresponding pixel unit on the dense depth estimation map to construct the reprojection error of the millimeter-wave radar.

3. The dense depth estimation method based on self-supervised millimeter-wave radar and camera fusion according to claim 1, characterized in that, Find temporally similar image frames and point cloud frames in the image sequence and the point cloud sequence, transform the point cloud to the image coordinate system, and then input the aligned image and point cloud pairs into a trained dense depth estimation model, including: Obtain the timestamps of the image sequence and the point cloud sequence, and align the temporal sequence of the image sequence and the point cloud sequence according to the timestamps; The external parameters of the camera and the 4D millimeter-wave radar are obtained, and the point cloud in the point cloud sequence is transformed according to the external parameters and projected onto the camera coordinate system to spatially align the image sequence and the point cloud sequence. The point cloud projection images after temporal alignment and spatial alignment are cropped to unify the size of each frame image and the point cloud of the corresponding frame. The cropped image sequence and point cloud sequence are input into the trained dense depth estimation model; In this system, each pixel unit in each frame of the point cloud corresponds to a pixel unit in the corresponding frame of the image.

4. The dense depth estimation method based on self-supervised millimeter-wave radar and camera fusion according to claim 1, characterized in that, The dense depth estimation model is trained through the following steps: Training the teacher network model specifically includes: Using sample image sequences from a camera and sample point cloud sequences from a 4D millimeter-wave radar as training samples, and using the sparse depth map of each frame of the point cloud in the sample point cloud sequence as the supervision sample for the teacher network model, the teacher network model is trained so that it learns the relationship between depth information and image features in each pixel unit of each sample image in the sample image sequence, and outputs the trained teacher network model. Training the student network model specifically includes: Using the sample image sequence from the camera and the sample point cloud sequence from the 4D millimeter-wave radar as training samples, the dense depth estimation map of each sample image predicted by the trained teacher network model is used as the supervision sample for training the student network model. The trained student network model is output as the dense depth estimation model.

5. The dense depth estimation method based on self-supervised millimeter-wave radar and camera fusion according to claim 4, characterized in that, The steps for training the teacher network model include: Obtain the dense depth estimation map of each sample image predicted by the teacher network model, obtain the sparse depth map of the point cloud corresponding to the sample image in the sample point cloud sequence, and construct a sparse depth loss function based on the difference between the depth value of each effective pixel unit in the sparse depth map and the depth value of the corresponding pixel unit in the dense depth estimation map of the teacher. Based on the preset pose transformation and the dense depth estimation map of the sample image of the current frame, the sample image of the next adjacent frame of the current frame is transformed to the current frame to obtain the transformed image of the adjacent frame; a photometric consistency loss function is constructed based on the transformed image and the sample image of the current frame. The teacher network model is trained by minimizing the sparse depth loss and photometric consistency loss, and the trained teacher network model is output.

6. The dense depth estimation method based on self-supervised millimeter-wave radar and camera fusion according to claim 4, characterized in that, The student network model is trained, and the trained student network model is output as the dense depth estimation model, including: The dense depth estimation branch of the student network model is used to perform feature encoding on each frame of the sample image sequence to obtain the sample depth code. The KL divergence loss function is constructed based on the difference between the sample depth code corresponding to each frame of the sample image and the standard normal distribution. Obtain the uncertainty of the uncertainty branch output of the student network model, obtain the dense depth estimation map of the student network model output by the dense depth estimation branch, and construct the dense depth loss function for all pixel units based on the uncertainty of the student network model output and the difference between the depth value of each pixel unit on the dense depth estimation map of the student network model and the depth value of the corresponding pixel unit on the dense depth estimation map of the teacher. Obtain the sparse depth map of the point cloud corresponding to the dense depth estimation map of the student in the sample point cloud sequence, and construct the sparse depth loss function based on the difference between the depth value of each pixel unit in the sparse depth map and the depth value of the corresponding pixel unit in the dense depth estimation map of the student. The student network model is trained by minimizing the KL divergence loss, the dense depth loss, and the sparse depth loss, and the converged student network model is output as the dense depth estimation model.

7. An apparatus based on the self-supervised millimeter-wave radar and camera fusion dense depth estimation method according to any one of claims 1-6, characterized in that, include: The data module is used to acquire image sequences from the camera and point cloud sequences from the 4D millimeter-wave radar. The data module is also used to align the image sequence and the point cloud sequence, output the aligned image sequence and point cloud sequence, and input the aligned image sequence and point cloud sequence into a trained dense depth estimation network and a depth code optimization method that combines digital modeling; The uncertainty calculation module performs feature encoding on each frame of the aligned image sequence and the corresponding point cloud frame in the aligned point cloud sequence through the uncertainty branch in the dense depth estimation model, thereby obtaining convolutional features; then, it decodes the convolutional features and calculates the uncertainty of the depth of each pixel in the corresponding frame image. The dense depth estimation module is used to obtain the depth code corresponding to the current frame, input the depth code and the convolutional features into the dense depth estimation branch of the dense depth estimation model, decode the depth code and the convolutional features through the dense depth estimation branch to obtain the dense depth of each pixel unit in the image of the corresponding frame, and output the dense depth estimation map. The depth code optimization module, which combines digital and analog models, is used to construct a least-squares loss function using multi-frame common-view information to optimize the depth code of historical keyframes and the current frame, thereby providing the optimized depth code for the dense depth estimation module. The least-squares loss function includes photometric errors between the current frame and multiple previous keyframe images, reprojection errors of matching feature points, reprojection errors of millimeter-wave radar, and Gaussian prior factors of the depth code.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images