Depth Map Generation Method and Apparatus

Abstract

A depth map generation method and apparatus are provided. The depth map generation method includes: generating a first depth map based on a stereo image; acquiring a second depth map captured by a depth camera; and generating a final depth map by fusing the first depth map and the second depth map.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically, to a method and apparatus for generating depth maps. Background Technology

[0002] Recently, many technologies, such as autonomous driving, robot navigation, and augmented reality, require accurate and dense depth estimation to solve high-level tasks. Depth can be acquired by sensors such as radar (LiDAR), structure light (SL), and time-of-flight (TOF), or generated from binocular images through depth estimation based on stereo matching algorithms.

[0003] Specifically, LiDAR is a sensor capable of accurately measuring long distances, but it can only provide sparse depth maps and has a small vertical imaging field of view. Furthermore, high-precision LiDAR devices are expensive and bulky, limiting their application scenarios. SL and TOF sensors have limited measurement distances, and their accuracy gradually decreases with increasing distance. This limitation restricts their measurement range, resulting in poor performance in outdoor environments. Additionally, the acquired depth maps suffer from high noise and low resolution. Depth estimation based on stereo matching algorithms can obtain dense depth maps. Currently, most high-accuracy methods are based on deep learning networks. However, depth estimation accuracy decreases when encountering ill-conditioned regions such as textureless areas, reflective areas, and occluded areas. Furthermore, it suffers from domain adaptation problems; that is, network models trained on specific datasets (such as synthetic datasets) experience accuracy degradation on different datasets (such as real-world datasets), making them unsuitable for new environments. Summary of the Invention

[0004] According to an exemplary embodiment of the present invention, a depth map generation method is provided, comprising: generating a first depth map based on a binocular image; acquiring a second depth map acquired by a depth camera; and generating a final depth map by fusing the first depth map and the second depth map.

[0005] The steps for obtaining the second depth map may include: extracting depth features from the second depth map; and generating a third depth map with depth completion by performing a similarity matrix-guided filter on the extracted depth features, wherein the similarity matrix is ​​generated based on the stereo image.

[0006] The steps for generating a third depth map with depth completion may include: obtaining the similarity between the current pixel and its neighboring pixels in the depth map based on a similarity matrix, and filtering the depth features of the current pixel in a predetermined direction based on the depth features of the neighboring pixels and the similarity.

[0007] The predetermined direction may include at least one of the following: a top-down direction, a bottom-up direction, a left-to-right direction, and a right-to-left direction.

[0008] The steps for generating the first depth map may include: obtaining a first feature map of each monocular image included in the stereo image; obtaining a matching cost matrix based on the first feature map of the monocular image; performing cost aggregation on the matching cost matrix; and generating a depth map of each monocular image based on the cost-aggregated matching cost matrix.

[0009] The step of generating the first depth map may further include: performing a consistency check on the depth map of each monocular image to obtain a reliable depth map for each monocular image.

[0010] The step of obtaining the first feature map may include performing feature extraction on the binocular image using a neural network based on an attention mechanism.

[0011] The steps to obtain the matching cost matrix may include: obtaining an initial matching cost matrix based on the first feature map of the monocular image, and inputting the initial matching cost matrix into a predetermined neural network for convolution to obtain the matching cost matrix.

[0012] The steps for cost aggregation may include performing cost aggregation on the matching cost matrix based on a graph neural network (GNN).

[0013] The steps of cost aggregation may include: obtaining the neighboring nodes of each center node as elements of the matching cost matrix based on the distance, and updating the matching cost of each center node based on the weight of each neighboring node and the matching cost of the neighboring nodes to obtain the matching cost matrix of the first cost aggregation.

[0014] The cost aggregation step may also include using a second feature map to guide the cost aggregation.

[0015] The second feature map can be generated based on a semantic segmentation neural network, or the second feature map can be the same as the first feature map.

[0016] The steps for cost aggregation may include: obtaining the neighboring pixels of each pixel in the second feature map based on the distance; finding the neighboring nodes of each center node as an element of the matching cost matrix according to the pixel correspondence between the second feature map and the matching cost; updating the matching cost of each center node based on the weight of the neighboring nodes and the matching cost to obtain the matching cost matrix of the second cost aggregation; and aggregating the matching cost matrix of the first cost aggregation with the matching cost matrix of the second cost aggregation to obtain the matching cost matrix of the third cost aggregation.

[0017] The distance can be spatial distance, feature vector distance, or a weighted sum of spatial distance and feature vector distance.

[0018] The steps of cost aggregation may include: obtaining a temporary matching cost matrix based on multiple matching cost matrices of different scales, obtaining the probability that each element of the temporary matching cost matrix corresponds to a predetermined disparity, and obtaining a fourth matching cost matrix based on the probability and the temporary matching cost matrix.

[0019] The cost aggregation step may include obtaining a fifth matching cost matrix based on the similarity matrix and the matching cost matrix, wherein the similarity matrix is ​​generated based on the stereo images.

[0020] The steps for generating the first depth map may include: inputting a stereo image into a predetermined neural network for feature extraction to obtain a first feature map of each monocular image included in the stereo image; obtaining a matching cost matrix based on the first feature map of the monocular image; calculating the probability that each element of the matching cost matrix corresponds to a predetermined disparity to convert the matching cost matrix into a matching probability matrix; and generating a depth map of each monocular image based on the matching probability matrix.

[0021] The steps for obtaining a reliable depth map may include: comparing pixels at corresponding positions between depth maps of various monocular images; determining the pixels at corresponding positions as unreliable pixels when the depth difference between pixels at corresponding positions is greater than or equal to a predetermined threshold; and determining the pixels at corresponding positions as reliable pixels when the depth difference between pixels at corresponding positions is less than the predetermined threshold, thereby obtaining a reliable depth map.

[0022] The steps to obtain a reliable depth map may include: processing the first feature map, the initial matching cost matrix, and the matching cost matrix based on the reliable depth map.

[0023] According to an exemplary embodiment of the present invention, a depth map generation apparatus is provided, comprising: a first acquisition unit for generating a first depth map based on a binocular image; a second acquisition unit for acquiring a second depth map acquired by a depth camera; and a depth synthesis unit for generating a final depth map by fusing the first depth map and the second depth map.

[0024] The second acquisition unit may include: a depth feature extraction unit, which extracts depth features from the second depth map; and a filtering unit, which generates a depth-completed third depth map by performing filtering guided by a similarity matrix on the extracted depth features, wherein the similarity matrix is ​​generated by the first acquisition unit based on the stereo image.

[0025] The filtering unit can obtain the similarity between the current pixel and its neighboring pixels in the depth map based on the similarity matrix, and filter the depth features of the current pixel in a predetermined direction based on the depth features of the neighboring pixels and the similarity.

[0026] The predetermined direction may include at least one of the following: a top-down direction, a bottom-up direction, a left-to-right direction, and a right-to-left direction.

[0027] The first acquisition unit may include: a feature extraction unit, which obtains a first feature map of each monocular image included in the binocular image; a matching cost acquisition unit, which obtains a matching cost matrix based on the first feature map of the monocular image; a cost aggregation unit, which performs cost aggregation on the matching cost matrix; and a depth map generation unit, which generates a depth map of each monocular image based on the cost-aggregated matching cost matrix.

[0028] The first acquisition unit may further include: a reliable depth acquisition unit, which performs consistency detection on the depth map of each monocular image to obtain a reliable depth map of each monocular image.

[0029] The feature extraction unit can extract features from binocular images based on a neural network with an attention mechanism.

[0030] The matching cost acquisition unit can obtain an initial matching cost matrix based on the first feature map of the monocular image, and input the initial matching cost matrix into a predetermined neural network for convolution to obtain the matching cost matrix.

[0031] The cost aggregation unit can perform cost aggregation on the matching cost matrix based on a graph neural network (GNN).

[0032] The cost aggregation unit can obtain the neighboring nodes of each center node as elements of the matching cost matrix based on the distance, and update the matching cost of each center node based on the weight of each neighboring node and the matching cost of the neighboring nodes to obtain the matching cost matrix of the first cost aggregation.

[0033] The cost aggregation unit can also use a second feature map to guide cost aggregation.

[0034] The second feature map can be generated based on a semantic segmentation neural network, or the second feature map can be the same as the first feature map.

[0035] The cost aggregation unit can obtain the neighboring pixels of each pixel in the second feature map based on the distance. According to the pixel correspondence between the second feature map and the matching cost, it finds the neighboring nodes of each center node that is an element of the matching cost matrix. Based on the weight of the neighboring nodes and the matching cost, it updates the matching cost of each center node to obtain the matching cost matrix of the second cost aggregation. Then, it aggregates the matching cost matrix of the first cost aggregation with the matching cost matrix of the second cost aggregation to obtain the matching cost matrix of the third cost aggregation.

[0036] The distance can be spatial distance, feature vector distance, or a weighted sum of spatial distance and feature vector distance.

[0037] The cost aggregation unit can obtain a temporary matching cost matrix based on multiple matching cost matrices of different scales. Each element of the temporary matching cost matrix corresponds to a probability of a predetermined disparity. A fourth matching cost matrix is ​​obtained based on the probability and the temporary matching cost matrix.

[0038] The cost aggregation unit can obtain a fifth matching cost matrix based on the similarity matrix and the matching cost matrix, wherein the similarity matrix is ​​generated by the first acquisition unit based on the stereo image.

[0039] The first acquisition unit may include: a feature extraction unit, which inputs the stereo image into a predetermined neural network to extract features and obtain a first feature map of each monocular image included in the stereo image; a matching cost acquisition unit, which obtains a matching cost matrix based on the first feature map of the monocular image; a matching probability acquisition unit, which calculates the probability of each element of the matching cost matrix corresponding to a predetermined disparity to convert the matching cost matrix into a matching probability matrix; and a depth map generation unit, which generates a depth map of each monocular image based on the matching probability matrix.

[0040] The reliable depth acquisition unit can compare pixels at corresponding positions between depth maps of various monocular images. When the depth difference between pixels at corresponding positions is greater than or equal to a predetermined threshold, the pixel at the corresponding position is determined as an unreliable pixel. When the depth difference between pixels at corresponding positions is less than the predetermined threshold, the pixel at the corresponding position is determined as a reliable pixel, thereby obtaining a reliable depth map.

[0041] The trusted depth acquisition unit can process the first feature map, the initial matching cost matrix, and the matching cost matrix based on the trusted depth map.

[0042] According to an exemplary embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores computer-executable instructions, and when the instructions are executed by the processor, the aforementioned method is performed.

[0043] According to an exemplary embodiment of the present invention, a computer-readable medium is provided having computer-executable instructions stored thereon, which, when executed, perform the aforementioned method.

[0044] Further aspects and / or advantages of the general concept of the invention will be set forth in part in the description which follows, and in part will be obvious from the description or may be learned by practice of the general concept of the invention. Attached Figure Description

[0045] The above and other objects and features of the present invention will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0046] Figure 1This is a block diagram illustrating a depth map generation apparatus according to an exemplary embodiment of the present invention;

[0047] Figure 2 This is a detailed block diagram illustrating a depth map generation apparatus according to an exemplary embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram illustrating a depth map generation apparatus according to an exemplary embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram illustrating depth filtering guided by the obtained similarity matrix according to an exemplary embodiment of the present invention.

[0050] Figure 5 This is a schematic diagram illustrating feature extraction based on an attention mechanism according to an exemplary embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram illustrating a stacked hourglass structure of a propagation module based on an attention mechanism according to an exemplary embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram illustrating the inter-pixel influence according to an exemplary embodiment of the present invention;

[0053] Figure 8 This is a schematic diagram illustrating the propagation of matching costs from top to bottom in the height dimension according to an exemplary embodiment of the present invention;

[0054] Figure 9 This is a flowchart illustrating a cost aggregation unit based on a graph neural network (GNN) performing cost aggregation according to an exemplary embodiment of the present invention;

[0055] Figure 10 This is a schematic diagram illustrating a cost aggregation unit based on a graph neural network performing cost aggregation according to an exemplary embodiment of the present invention;

[0056] Figure 11 This is a schematic diagram illustrating the operation of an aggregation function according to an embodiment of the present invention;

[0057] Figure 12 This is a schematic diagram illustrating the feature / matching cost improvement operation performed by the trusted depth acquisition unit according to an exemplary embodiment of the present invention;

[0058] Figure 13 This is a schematic diagram illustrating a geometric consistency optimization operation according to an exemplary embodiment of the present invention;

[0059] Figure 14 This is a flowchart illustrating a depth map generation method according to an exemplary embodiment of the present invention. Detailed Implementation

[0060] The present invention will now be described in detail with reference to embodiments thereof, examples of which are illustrated in the accompanying drawings, wherein the same reference numerals refer to the same parts throughout. The embodiments will be described below with reference to the accompanying drawings in order to explain the present invention.

[0061] In summary, this invention obtains high-quality depth maps by simultaneously optimizing both the depth maps obtained from sensors and the depth maps obtained from stereo matching algorithms. On one hand, addressing the issues of noise, sparsity, and low resolution in depth maps obtained from sensors, a deep learning network guided by binocular images is proposed to complete the sensor-obtained depth maps, resulting in a completed depth map A. On the other hand, addressing the problem of poor accuracy in ill-conditioned regions for depth estimation based on deep learning methods, a deep learning network based on attention mechanisms and graph neural networks is proposed to extract robust features and transfer these features to ill-conditioned regions, resulting in a depth map B. Finally, through depth optimization, the completed depth map A and the depth map B obtained from the stereo matching algorithm are jointly optimized, and corresponding normal vector maps are obtained from depth maps A and B respectively. Figure 1 Consistency constraints generate high-quality depth maps.

[0062] Figure 1 This is a block diagram illustrating a depth map generation apparatus according to an exemplary embodiment of the present invention.

[0063] Reference Figure 1 According to an exemplary embodiment of the present invention, the depth map generation apparatus 100 may include a first acquisition unit 110, a second acquisition unit 120, and a depth synthesis unit 130.

[0064] The first acquisition unit 110 can generate a first depth map based on the stereo images. Furthermore, the first acquisition unit 110 can also generate a similarity matrix based on the stereo images. Here, the first acquisition unit 110 can arbitrarily select one monocular image (left or right image, for example only, not a limitation) from the stereo images, and use a neural network (convolutional neural network, for example only, not a limitation) to calculate the similarity between image pixels to obtain a similarity matrix. Furthermore, for example only, the stereo images can be acquired by a stereo camera. It should also be understood that the left image / right image, left disparity map / right disparity map, left depth map / right depth map, etc., described in this invention are merely examples of stereo images and not limitations. This invention is also applicable to other types of stereo images, such as top-bottom stereo images.

[0065] The second acquisition unit 120 can acquire a second depth map captured by a depth camera. More specifically, the second acquisition unit 120 can generate a second depth map by performing depth completion on the depth map based on a similarity matrix. More specifically, the second acquisition unit 120 can first acquire the similarity matrix generated by the first acquisition unit 110, and then input the depth map into a deep learning network composed of the second acquisition unit 120 to obtain the second depth map. Alternatively, an existing deep learning network can be used to directly input the depth map and output a second depth map. However, this invention obtains the similarity between pixels in the depth map through a similarity matrix, thereby guiding the network to assign higher weights to adjacent pixels with high similarity during depth completion, thus obtaining a more accurate completed depth map. Here, this is only an example and not a limitation; the depth map can be acquired by a depth camera such as an SL or TOF sensor.

[0066] The depth synthesis unit 130 can generate a final depth map by fusing the first depth map with the second depth map. Here, by way of example only and not limitation, the depth synthesis unit 130 can perform the fusion in a weighted fusion manner to output a high-quality depth map.

[0067] The following will refer to Figure 2 and Figure 3 The depth map generation apparatus 100 according to an exemplary embodiment of the present invention will be described in more detail. Figure 2 This is a detailed block diagram illustrating a depth map generation apparatus according to an exemplary embodiment of the present invention. Figure 3 This is a schematic diagram illustrating a depth map generation apparatus according to an exemplary embodiment of the present invention.

[0068] Reference Figure 2 The second acquisition unit 120 may include a depth feature extraction unit 211 and a filtering unit 212, and the first acquisition unit 110 may include a feature extraction unit 221, a matching cost acquisition unit 222, a cost aggregation unit 223 and a depth map generation unit 224.

[0069] The depth feature extraction unit 211 can extract depth features from the second depth map. More specifically, the depth feature extraction unit 211 can input the depth map into a neural network (by way of example only and not a limitation, the neural network can be a convolutional neural network) to extract depth features.

[0070] Subsequently, in order to fill the holes in the depth map and improve its resolution, the depth map needs to be filtered. This invention uses a similarity matrix to guide the filtering, which preserves boundary information and ensures the smoothness of the object's interior. Figure 4 This is a schematic diagram illustrating guided filtering based on the obtained similarity matrix according to an exemplary embodiment of the present invention. Figure 4As shown, the first acquisition unit 110 can generate a similarity matrix based on the stereo image and use it for filtering by the second acquisition unit 120. Here, the filtering unit 212 can generate a third depth map with depth completion by filtering the extracted depth features guided by the similarity matrix. At this time, the depth synthesis unit 130 can generate the final depth map by fusing the first depth map and the third depth map. The role of the similarity matrix can be to calculate pixel similarity to guide the adaptive aggregation of subsequent features, which is more conducive to propagation in areas with less texture. More specifically, the filtering unit 212 can obtain the similarity between the current pixel of the depth map and the neighboring pixels of the current pixel based on the similarity matrix, and use the depth features based on the neighboring pixels and the similarity to filter the depth features of the current pixel in a predetermined direction. As an example and not a limitation, assuming that the depth features of the current pixel of the obtained depth map are F (dimensions H*W*C, where H represents height, W represents width, and C represents the number of channels), the filtering unit 212 can slice it along the dimensions to obtain C depth feature slices (dimensions H*W). The process of propagation within each depth feature slice includes updating in a predetermined direction. Here, by way of example only and not limitation, the predetermined direction may include at least one of the following: a top-down direction, a bottom-up direction, a left-to-right direction, and a right-to-left direction. Furthermore, as an example and not a limitation, for the top-down and left-to-right directions, assuming the depth feature value of the neighboring pixel q of the current pixel p is d′(q), the filtering unit 212 can obtain the similarity w between the current pixel p and the neighboring pixel q through the similarity matrix, and filter the depth feature value d(p) of the current pixel p according to the following equation (1) to obtain the updated depth feature value d′(p) of the current pixel p; for the bottom-up and right-to-left directions, assuming the unupdated depth feature value of the neighboring pixel q of the current pixel p is d(q) and the updated depth feature value is d′(q), the filtering unit 212 can obtain the similarity w between the current pixel p and the neighboring pixel q through the similarity matrix, and filter the depth feature value d(p) of the current pixel p according to the following equation (2) to obtain the updated depth feature value d″(p) of the current pixel p. The filtering unit 212 can then convolve the filtered depth features through a neural network such as a convolutional neural network to generate a third depth map with depth completion.

[0071] d′(p)=d(p)+w*d′(q) (1)

[0072] d″(p)=w*d(q)+(1-w 2 )*d′(q) (2)

[0073] Using multi-view geometry theory, pixel depth can be calculated from the disparity of two monocular images (left and right images, for example only, not a limitation) of a binocular image. Here, the disparity map can be generated using a stereo matching algorithm, i.e., finding the correspondence between pixels in the left and right images and generating the disparity map by minimizing the matching cost. Currently, with the application of deep learning networks in stereo matching algorithms, the accuracy of disparity maps has been greatly improved. However, in ill-conditioned regions, such as textureless regions, occluded regions, and reflective regions, the generated disparity maps still have significant errors. To address these challenging issues, this invention improves three steps in the stereo matching algorithm: matching cost calculation, cost aggregation, and disparity calculation. In the matching cost calculation, an attention mechanism can be used for feature extraction to obtain robust depth features (such as object boundaries) to reduce erroneous matching between pixels in the binocular image, making cost matching more accurate in ill-conditioned regions. In addition, the learned left and right depth maps can be used to determine the reliable depth and feed it back to the feature extraction network to improve the feature / matching cost, which will be described later. In the cost aggregation, a graph neural network can be used to aggregate costs, capturing a long range of dependencies between pixels, and the extracted robust features can be transferred to ill-conditioned regions. In the disparity calculation, in order to obtain a more accurate disparity map, left and right consistency constraints can be applied to pixels without label information.

[0074] The feature extraction unit 221 can obtain a first feature map of each monocular image included in the stereo image. More specifically, the feature extraction unit 221 can perform feature extraction on the stereo image based on a neural network with an attention mechanism. In addition, the similarity matrix can be obtained by the feature extraction unit 221 based on the stereo image, or it can be obtained by setting another unit in the first acquisition unit 110.

[0075] Figure 5 This is a schematic diagram illustrating feature extraction based on an attention mechanism according to an exemplary embodiment of the present invention. Before adding the attention mechanism, features are aggregated through convolution, and the size of the receptive field is determined by the size of the convolution kernel; however, after adding the attention mechanism, different weights are assigned to different features, thus the range of feature aggregation is dynamic, such as... Figure 5 As shown in the diagram, the left side represents the feature map, and the right side represents the attention mechanism. The attention mechanism learns the weights corresponding to the features. If the weights of the features used in the convolution operation of the attention-based neural network are small, then... Figure 5 The arrow pointing left on the right indicates the amount of information that is blocked from propagating from that feature to the central feature of the convolution, while... Figure 5The arrow pointing to the right on the right side indicates the amount of information contributed by surrounding features to the central feature during the convolution operation. If the attention mechanism captures the boundary information of the object, features belonging to the same object can be aggregated, while features not belonging to the same object will be blocked.

[0076] More specifically, the feature extraction unit 221 can first perform feature extraction on the stereo image to obtain a feature map F (with dimensions H*W*C, where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map) for each monocular image included in the stereo image, and then take the feature map f of each channel. i (with dimensions H*W) and perform max pooling operation MaxPool(f) on it. i That is, to calculate the feature map f i The largest element m i Since there are a total of C channels, C max pooling operations are performed, ultimately yielding C largest elements m1, m2, ..., mn. c Then m1, m2, ..., m c The input is fed into a neural network to obtain attention parameters a1, a2, ..., a c Finally, the attention parameters a1, a2, ..., a c Feature maps f1, f2, ..., f of different channels respectively c Element-wise multiplication yields a new feature map F'.

[0077] As another example, an attention mechanism based on the feature space can also be used, whereby the feature extraction unit 221 can take the feature map f of each pixel of the feature map F. j (Dimension: C) and perform max pooling operation on it MaxPool(f) j That is, to calculate the feature map f j The largest element m j Since there are a total of H*W pixels, H*W max pooling operations are performed, ultimately yielding H*W largest elements m1, m2, ..., mn. H*W Then m1, m2, ..., m H*W The inputs are fed into the neural network to obtain attention parameters b1, b2, ..., b H*W Finally, the attention parameters b1, b2, ..., b H*W Feature maps f1, f2, ..., f of different pixels respectively H*W Element-wise multiplication yields a new feature map F. Therefore, attention mechanisms can be applied to feature extraction from the left and right images, and global information can be learned through attention to obtain robust depth features (object boundaries, etc.), thereby reducing erroneous matching between pixels in binocular images and making matching in ill-conditioned regions more accurate.

[0078] Robust and reliable stereo matching is crucial for scene understanding in multimedia applications. Strong visual cues can be detected and matched well with current feature descriptors, but they are often too sparse to propagate the matching cost. However, propagating the cost from strong cells to weak cells is key to dense disparity map generation. Considering this, an attention-based propagation module can be designed to propagate the reliable matching cost from one cell to its neighboring regions, especially ill-conditioned regions. This attention-based propagation module can consist of two parts: an attention component and a propagation component.

[0079] Specifically, robust matching beliefs and reduced false-positive matching beliefs can be enhanced by embedding spatial attention layers into the stacked hourglass module of existing convolutional neural network models. After compressing the four-dimensional matching cost into three dimensions, the three-dimensional matching cost can be sliced, and disparity belief propagation can be performed from one side to the other for each slice. Unlike the stacked hourglass module structure, explicit disparity belief propagation can be employed within the matching cost to achieve more efficient belief propagation.

[0080] Figure 6 This is a schematic diagram illustrating a stacked hourglass structure of a propagation module based on an attention mechanism according to an exemplary embodiment of the present invention. Figure 6 As shown, an existing convolutional neural network model can be used as the backbone, and the stacked hourglass module structure can be improved by using a propagation module based on an attention mechanism. Since the information transmission of each unit to its neighborhood is similar to that of a shared convolutional kernel, a spatial attention mechanism can be introduced to amplify the influence of robust pixels, while a channel attention mechanism can be introduced to more effectively extract robust features. Figure 6 The propagation module in the present invention is an attention-based propagation module according to an exemplary embodiment of the present invention, wherein the stacked hourglass structure of the attention-based propagation module has the same meaning as the stacked hourglass with embedded attention mechanism.

[0081] The stacked hourglass module structure of convolutional neural network models can transfer information from one unit to its neighbors through different scale designs, and the range of each unit can be expanded by upsampling and downsampling the matching cost. However, this information exchange is sometimes insufficient because the influence of one pixel on another is closely related to the distance between them. Figure 7 This is a schematic diagram illustrating the inter-pixel influence according to an exemplary embodiment of the present invention. For example... Figure 7As shown, black units represent units with strong disparity beliefs, i.e., robust matching, while white units represent units with weak disparity beliefs. Generally, disparity beliefs can be propagated from black units to the other two white units. However, using a typical convolutional layer kernel size is limited, requiring stacking many convolutional layers or larger kernels to achieve this belief propagation. Therefore, a matching cost propagation module based on disparity belief propagation can be employed. The output of the stacked hourglass module structure of the convolutional neural network model contains four dimensions: channels, disparity, height, and width. The channel size can be compressed to obtain a three-dimensional matching cost: disparity, height, and width. To propagate disparity beliefs from one side to the other, the matching cost can be partitioned along the height or width dimension. Figure 8 This is a schematic diagram illustrating the propagation of matching costs from top to bottom in the height dimension according to an exemplary embodiment of the present invention. Figure 8 As shown, the propagation in the height dimension from top to bottom is described. When obtaining a slice of shape "disparity * 1 * width", it can be copied i times, and a two-dimensional convolutional layer with a kernel size of 1 * kw can be added, whose input and output channels are both D, i.e., the disparity dimension in the current matching cost. In addition, a batch normalization (BN) layer can be added before nonlinear activation, and then another convolutional and BN layer (without activation) can be added. In this way, the disparity propagation module according to the exemplary embodiment of the present invention can prevent the propagation of units with low disparity belief globally. The matching cost acquisition unit 222 can obtain the matching cost matrix based on the first feature map of the monocular image. That is, the matching cost acquisition unit 222 can obtain an initial matching cost matrix based on the first feature map of the monocular image, and input the initial matching cost matrix into a neural network (only as an example and not a limitation, the neural network can be an Hourglass convolutional network or a multi-layer 3D convolutional neural network) for convolution to obtain the matching cost matrix. The matching cost acquisition unit 222 can obtain the matching cost matrix by taking a left feature map F with feature dimensions of H * W * C. l and right feature map F rThe initial matching cost matrix is ​​obtained by concatenating the features along the disparity direction. More specifically, when the disparity is 0, the matching cost acquisition unit 222 can directly concatenate the left and right feature maps to obtain a matrix of dimension H*W*2C; when the disparity is d, the matching cost acquisition unit 222 can first translate the right feature map along a predetermined direction (horizontal direction only, not a limitation) by d columns (where d is a positive integer), and then concatenate the left and right feature maps to obtain a matrix of dimension H*W*2C; assuming the disparity value d ranges from 0 to DD (where DD is a positive integer), the matching cost acquisition unit 222 can sequentially translate the right feature map according to the magnitude of the disparity and concatenate the left and right feature maps to obtain a matrix of dimension H*W*2C*D, i.e., the initial matching cost matrix. Subsequently, the matching cost acquisition unit 222 can input the initial matching cost matrix into a neural network such as the Hourglass convolutional network or a multi-layer 3D convolutional neural network to obtain the matching cost matrix (dimension H*W*D). Furthermore, the matching cost acquisition unit 222 can obtain multiple matching cost matrices of different scales by inputting the initial matching cost matrix into a neural network such as the Hourglass convolutional network or a multi-layer 3D convolutional neural network.

[0082] Cost aggregation unit 223 can perform cost aggregation on the matching cost matrix. Here, cost aggregation unit 223 can perform cost aggregation on the matching cost matrix based on a graph neural network (GNN). More specifically, cost aggregation unit 223 can obtain the neighboring nodes of each center node that is an element of the matching cost matrix based on distance, and update the matching cost of each center node based on the weight of each neighboring node and the matching cost of the neighboring nodes to obtain a first cost-aggregated matching cost matrix. The following will refer to... Figure 9 and Figure 10 This will be described. Figure 9 This is a flowchart illustrating the cost aggregation unit 223 performing cost aggregation based on a graph neural network (GNN) according to an exemplary embodiment of the present invention. Figure 10 This is a schematic diagram illustrating cost aggregation unit 223 performing cost aggregation based on a graph neural network according to an exemplary embodiment of the present invention. Figure 9 As shown, in step S910, the cost aggregation unit 223 can construct a graph based on the matching cost matrix. More specifically, each element of the matching cost matrix is ​​used as a center node, resulting in a total of H*W center nodes. Each center node consists of location information and feature information (f, dimension D). The cost aggregation unit 223 can obtain a specific number k (k can be set by those skilled in the art according to actual conditions) of other center nodes (n1, n2, ..., n) that are closest to each center node. k(i.e., neighboring nodes), and connect each central node with its corresponding k nearest other central nodes to form edges, thereby constructing a graph G. The distance can be spatial distance, feature vector distance, or a weighted sum of spatial distance and feature vector distance; more specifically, it can be calculated using Euclidean distance. In step S920, the cost aggregation unit 223 can connect the matching costs of the k other central nodes (also called k neighboring nodes) corresponding to each central node along the feature dimension (i.e., the feature dimension of each nearest neighbor node is D, and the total dimension after connection is k*D) and input it into a neural network (only as an example and not a limitation; this neural network can be a multilayer perceptron network) to obtain the weights (w1, w2, ..., w) of the edges between each central node and its corresponding k nearest other central nodes. k The matching cost matrix of the first cost aggregation is obtained by updating the matching cost of each center node based on the weight and the matching cost of the k nearest other center nodes. More specifically, the cost aggregation unit 223 can update the matching cost matrix by the following equation (3).

[0083]

[0084] Among them, w i It is the weight of the edge corresponding to the k nearest other center nodes, f i It is the matching cost of the k nearest other center nodes, f before1 It is the matching cost before the central node is updated, f current1 σ is the matching cost after the center node is updated, a1 and b1 are predetermined constants, and k is a positive integer. Through the above operations, feature propagation can be performed between pixels. After multiple iterations, a long range of dependencies between pixels can be captured, thereby transferring the extracted robust features to ill-conditioned regions.

[0085] Furthermore, the cost aggregation unit 223 can also use the second feature map F seg Guided cost aggregation. Here, the second feature map is generated based on a semantic segmentation neural network (such as deeplabV3), or the second feature map is the same as the first feature map. More specifically, the cost aggregation unit 223 can obtain the neighboring pixels of each pixel in the second feature map based on the distance, find the neighboring nodes of each center node as an element of the matching cost matrix according to the pixel correspondence between the second feature map and the matching cost, update the matching cost of each center node based on the weight of the neighboring nodes and the matching cost to obtain the matching cost matrix of the second cost aggregation, and aggregate the matching cost matrix of the first cost aggregation with the matching cost matrix of the second cost aggregation to obtain the matching cost matrix of the third cost aggregation. The cost aggregation unit 223 can first determine the matching cost matrix based on F segA graph is constructed where each element of the matching cost matrix is ​​used as a center node, resulting in a total of H*W center nodes. Each center node consists of location information and feature information (f, dimension D). Since each center node of the matching cost matrix is ​​related to F... seg Each pixel has a one-to-one correspondence, so the cost aggregation unit 223 can obtain the value of F. seg The cost aggregation unit 223 obtains the k nearest other center nodes (i.e., neighboring pixels) to each center node by averaging the distance of each pixel to a specific number of k (k can be set by those skilled in the art according to the actual situation) other pixels (i.e., neighboring pixels). Then, the cost aggregation unit 223 can obtain the F value of each center node. seg The k (k is a positive integer) nearest other center nodes (n1, n2, ..., n) between pixels k The graph G is constructed by connecting each central node with its k nearest other central nodes to form edges. The distance can be spatial distance, feature vector distance, or a weighted sum of spatial and feature vector distances. More specifically, it can be calculated using Euclidean distance or Hamming distance. Subsequently, the cost aggregation unit 223 connects the matching costs of the k nearest other central nodes for each central node along the feature dimension and inputs them into the neural network, such as a multilayer perceptron network, to obtain the weights (w1, w2, ..., w) of the edges between each central node and its k nearest other central nodes. k Based on the weights and the matching costs of the k nearest other center nodes, the matching cost of each center node is updated to obtain a second cost aggregation matching cost matrix. The first cost aggregation matching cost matrix is ​​then aggregated with the second cost aggregation matching cost matrix to obtain a third cost aggregation matching cost matrix. More specifically, the cost aggregation unit 223 can update the matching cost matrix using the following equation (4):

[0086]

[0087] Among them, w i It is the weight of the edge corresponding to the k nearest other center nodes, f i It is the matching cost of the k nearest other center nodes, f before2 It is the matching cost before the central node is updated, f current1 σ is the matching cost after the central node is updated, a2 and b2 are predetermined constants, and k is a positive integer.

[0088] The resulting third cost aggregation matching cost matrix not only includes matching cost aggregation guided by depth estimation itself, but also matching cost aggregation guided by semantic segmentation, thus improving accuracy.

[0089] In addition, other implementation methods can be adopted, as follows:

[0090] Stereo image matching, capable of providing convincing dense depth maps without depth sensors, has garnered significant attention in modern applications. Efficient matching cost aggregation is crucial for stereo matching tasks. While traditional methods and deep neural networks have developed some effective cost aggregation techniques, finding correct matching relationships in ill-conditioned regions remains challenging. Utilizing long-term contextual information can alleviate this difficulty. Compared to previous works employing multi-layer 3D convolutions, a novel Guided Graph Convolutional Module (GGCM) can address this issue. GGCM constructs a graph based on the pixel similarity of the guide image. By stacking multiple GGCM modules, global background information is learned, with the vertices of the graph being pixels, and it is constructed as a K-nearest neighbor graph based on similarity distances. Furthermore, a simple and efficient distance calculation method is defined and compared with existing GCN methods. This distance comprises two parts: a spatial distance for modeling spatial relationships and a part defined in the feature space of the guide image, which explores non-local pixel similarity and preserves boundary information. This method is non-local because, after multiple convolution iterations, each node receives information from all other nodes in the graph.

[0091] Current stereo matching methods typically employ downsampling and upsampling operations in 3D convolutional layers for cost aggregation, which reduces resolution and leads to blurred disparity map edges. Traditional guided image filtering methods have demonstrated that image-guided filtering can preserve the edges of depth images. Therefore, a graph can be automatically constructed based on the guided image and applied to graph convolution operations to adaptively aggregate cost features.

[0092] Specifically, given a feature map of the guiding image (dimension H*W*C), and an input matching cost matrix C (dimension H*W*D), each element of the matching cost matrix can be considered as a center node, resulting in a total of H*W center nodes. Each center node consists of feature information (dimension D).

[0093] First, construct a graph G (N vertices, each with k neighbors and k edges) based on similarity distance. The vertices (dimension N = H * W) are pixels in a tensor, and edges (dimension k * N) are generated between their k neighboring pixels. A simple and efficient distance function d can be defined. ij As shown in equation (i) below.

[0094]

[0095] This distance can consist of two parts: one part is the spatial distance pixel coordinates used for spatial relationship modeling, represented as (x... i ,y i), (x j ,y j The first term in the equation is the Euclidean distance between two pixel coordinates, and the second term is defined in the feature space of the guiding image to utilize non-local pixel similarity and preserve depth edge information; for feature map I, each pixel has a feature vector f. i (There are C dimensions, each dimension is represented by f) ic ) or f j (There are C dimensions, each dimension is represented by f) jc The distance in the feature space represents the Euclidean distance between two feature vectors, which is the second term in the formula; l represents the l-th layer of the convolution.

[0096] Secondly, the matching cost units are adaptively aggregated by weighted summation of k neighboring pixels. Figure 11 This is a schematic diagram illustrating the operation of an aggregation function according to an embodiment of the present invention. For example... Figure 11 As shown, the aggregation weights are estimated by the edge filtering network, using a cost-unit-dependent, data-driven approach rather than predefined parameterized operations. The aggregation function is shown in equation (ii) below.

[0097]

[0098] in, Representing vertex information, the edge filtering network F l Edge information L can be used l (i,j) is the input, and the output is a weight matrix θ. ij W l Represents the learnable parameters, u j l Indicates v i l The neighboring nodes, E l Let N(i) represent the edge attention function, N(i) represent the number of neighboring nodes, l represent the l-th layer of the convolution, and γ represent the edge attention function. ij This represents the edge-attention parameter.

[0099] Finally, the aggregated vertex information can be used. And the original vertex information Update the matching cost matrix as shown in equation (iii) below. Here, w is a fixed parameter.

[0100]

[0101] Furthermore, the cost aggregation unit 223 can obtain a temporary matching cost matrix based on multiple matching cost matrices of different scales, obtain the probability that each element of the temporary matching cost matrix corresponds to a predetermined disparity, and obtain a fourth matching cost matrix based on the probability and the temporary matching cost matrix. More specifically, the cost aggregation unit 223 can transform (e.g., upsample) multiple matching cost matrices of different scales (cost1, cost2, ..., costk) to the same size (dimension H*W*D) and concatenate them along the disparity dimension as shown in the following equation (5) to obtain a temporary matching cost matrix (dimension H*W*kD). Subsequently, the cost aggregation unit 223 can obtain the probability that each element of the temporary matching cost matrix corresponds to a predetermined disparity based on the temporary matching cost matrix. Here, the cost aggregation unit 223 can use the softmax function as shown in the following equation (6) to obtain the probability P that each element of the temporary matching cost matrix corresponds to a predetermined disparity. Finally, the cost aggregation unit 223 can multiply the probability P with the temporary matching cost matrix as shown in the following equation (7) to obtain the fourth matching cost matrix. Experimental verification shows that using this strategy can effectively improve the accuracy of generating disparity maps.

[0102] cost temp =concat(cost1, cost2,..., costk) (5)

[0103] P = F softmax (cost temp (6)

[0104] cost new =cost temp *P (7)

[0105] Where cost1, cost2, ..., costk represent k matching cost matrices of different scales, and cost... temp Let P represent the temporary matching cost matrix, where P represents the temporary matching cost matrix cost. temp Each element corresponds to the probability of a predetermined disparity, cost new Let k represent the newly obtained matching cost matrix (i.e., the matching cost matrix of the sixth cost aggregation), where k is a positive integer.

[0106] Furthermore, the cost aggregation unit 223 can also guide matching cost aggregation based on the similarity matrix, that is, it can obtain the fifth matching cost matrix based on the similarity matrix and the matching cost matrix. More specifically, the cost aggregation unit 223 can perform a convolution operation on the matching cost matrix obtained by the matching cost acquisition unit 222 using the similarity matrix (dimension: HW*HW). The parameters in the convolution operation are determined by the similarity matrix. Specifically, the matching cost matrix (H*W*D) is converted into a two-dimensional matching cost matrix (HW*D), and the multiplication of the similarity matrix and the two-dimensional matching cost matrix (HW*HW)*(HW*D) is performed to obtain a new matrix HW*D. Then, the dimensions are transformed to obtain the fifth matching cost matrix (H*W*D).

[0107] It should be understood that the cost aggregation unit 223 can be added as a separate part to an existing stereo matching network or depth estimation network to improve the performance of the existing network.

[0108] In stereo matching, using contextual information for cost aggregation is highly effective in improving matching accuracy. However, deep learning networks often perform cost aggregation on a four-dimensional matching cost matrix, where the four dimensions are features, disparity, image height, and image width. Generating a disparity map from a four-dimensional matching cost matrix requires compressing the feature and disparity dimensions. Existing techniques often use simple fully connected layers for pixel-by-pixel disparity calculation, which does not consider contextual information, leading to errors in disparity generation. According to an embodiment of the present invention, after compressing the feature dimension, a context-based matching confidence propagation can be performed on the three-dimensional matching cost matrix to improve the accuracy of disparity calculation for each pixel during dimensionality compression. More specifically, the first acquisition unit 110 may include a matching probability acquisition unit for calculating the probability Pd (by way of example only, not a limitation, that each element (x, y) of the matching cost matrix obtained by the matching cost acquisition unit 222 corresponds to a predetermined disparity D) to convert the three-dimensional matching cost matrix into a three-dimensional matching probability matrix C(H*W*D). Subsequently, a confidence propagation algorithm can be used to optimize the matching probability matrix, and the confidence propagation algorithm can be simplified to a layer-by-layer convolution process within the matching probability matrix, that is, performing a convolution operation on the matching probability matrix (H*W*D). In other words, the matching probability acquisition unit that performs the above matching probability calculation can replace the cost aggregation unit 223, and the depth map generation unit 224 can generate a depth map for each monocular image based on the matching probability matrix. According to an embodiment of the present invention, when implementing a differentiable confidence propagation algorithm within the matching probability matrix, the values ​​within the matching probability matrix are transformed from probability space to logarithmic space to fit the confidence propagation algorithm using a convolution process. In this process, a guide map describing the similarity between adjacent pixels is needed to participate in the calculation; specifically, the convolution kernel parameters during the convolution process are determined by the pixel values ​​at the corresponding positions in the guide map. This guide map can be transformed from some feature maps obtained during the convolution process, or it can be transformed from the optical flow and semantic segmentation results of the image.

[0109] The depth map generation unit 224 can generate a depth map for each monocular image based on the cost aggregation matching cost matrix. Here, the depth map generation unit 224 can use any one of the first cost aggregation matching cost matrix, the third cost aggregation matching cost matrix, the fourth matching cost matrix, and the fifth matching cost matrix obtained by the cost aggregation unit 223 to generate the depth map for each monocular image. According to an exemplary embodiment of the present invention, the depth map generation unit 224 can first generate a disparity map disp(H*W) for each monocular image based on the cost aggregation matching cost matrix or the matching probability matrix, and then generate a depth map for each monocular image based on the disparity map disp. More specifically, the depth map generation unit 224 can use the softmax function to calculate each element C of the aggregated matching cost matrix or the matching probability matrix. d (x i ,y i The probability P corresponding to the disparity dimension D. di Then, all disparity values ​​di in the range of 0 to DD are compared with their corresponding probabilities P. di Multiply and sum to obtain the disparity value of the current pixel in the monocular image, as shown in equation (8) below.

[0110]

[0111] Where di represents the disparity value, c d These represent the elements of the matching cost matrix or matching probability matrix used for aggregation. This represents the disparity value of the current pixel, and DD represents the maximum disparity value.

[0112] Subsequently, the depth map generation unit 224 can... Calculate the depth map for each monocular image, where b represents the distance between the optical centers of the two cameras used to acquire the two monocular images, f represents the focal length of the two cameras used to acquire the two monocular images, and d is the disparity value of the obtained disparity map.

[0113] Furthermore, according to an exemplary embodiment of the present invention, the depth map generation unit 224 may further include a reliable depth acquisition unit. The reliable depth acquisition unit can compare pixels at corresponding positions between depth maps of various monocular images. When the depth difference between pixels at corresponding positions is greater than or equal to a predetermined threshold, the pixel at the corresponding position is determined as an unreliable pixel; when the depth difference between pixels at corresponding positions is less than the predetermined threshold, the pixel at the corresponding position is determined as a reliable pixel, thereby obtaining a reliable depth map. More specifically, the reliable depth acquisition unit can set the pixel value of unreliable pixels to 0 and the pixel value of reliable pixels to 1, thereby obtaining a reliable depth map. Subsequently, the reliable depth acquisition unit can process the first feature map, the initial matching cost matrix, and the matching cost matrix based on the reliable depth map. More specifically, the reliable depth acquisition unit can improve the feature / matching cost by multiplying the reliable depth map with the first feature map obtained by the feature extraction unit 221 or the initial matching cost matrix or matching cost matrix obtained by the matching cost acquisition unit 222 to remove occluded areas and then pass them to surrounding pixels, thereby preserving reliable features or matching costs and improving the accuracy of the generated depth map. Here, after the trusted depth acquisition unit multiplies the trusted depth map with the first feature map obtained by the feature extraction unit 221 or the initial matching cost matrix or matching cost matrix obtained by the matching cost acquisition unit 222, the matching cost acquisition unit 222 can obtain the initial matching cost matrix based on the product of the trusted depth map and the first feature map, and can obtain the matching cost matrix based on the product of the trusted depth map and the initial matching cost matrix. The cost aggregation unit 223 can perform cost aggregation on the product of the trusted depth map and the matching cost matrix. Figure 12 This is a schematic diagram illustrating a feature / matching cost enhancement operation performed by a trusted depth acquisition unit according to an exemplary embodiment of the present invention. Figure 12 As shown, the trusted depth acquisition unit can pass the product of the obtained trusted depth map and the feature map or the matching cost matrix to the surrounding pixels in subsequent processing.

[0114] Furthermore, according to an exemplary embodiment of the present invention, in the training of the neural network constituted by the first acquisition unit 110, the error between the calculated disparity value and the labeled disparity value can be used as a first loss function for backpropagation to train the neural network. Real-world scene datasets often use sparse LiDAR data as annotations, resulting in a large amount of unlabeled data in an image. During training, sparse annotations can only constrain a portion of pixels, lacking effective constraints on unlabeled pixels. According to an exemplary embodiment of the present invention, a left-right consistency detection method can be used to constrain unlabeled pixels. That is, for unlabeled pixels, the calculated disparity map of the monocular image is mapped to the disparity map of the corresponding monocular image, and the difference between the calculated disparity map of the corresponding monocular image and the mapped disparity map of the corresponding monocular image is used as the loss function for unlabeled pixels for backpropagation. Furthermore, the loss function for unlabeled pixels can be used together with the loss function for labeled pixels as a first loss function for backpropagation to train the neural network constituted by the first acquisition unit 110. More specifically, the disparity map of the calculated monocular image can be mapped to the disparity map of the corresponding other monocular image as shown in equation (9) below (by way of example only, and not a limitation, the right disparity map can be mapped to the left disparity map) to generate a new disparity map of the other monocular image (by way of example only, and not a limitation, the left disparity map), and then the difference L between the calculated disparity map of the monocular image and the mapped disparity map of the corresponding other monocular image can be obtained as shown in equation (10) below. warp_consistance And as shown in equation (11), it is combined with the loss function L of the marked pixels. smooth Together they are used for training the neural network formed by the first acquisition unit 110.

[0115]

[0116]

[0117] L = L smooth +L warp_consistance (11)

[0118] in, D represents the disparity map of the new left image generated from the right image. l (x l (x, y) represents pixel (x) l The value of (x, y), l -D l (x l ,y) represents the pixel (x) l ,y) shift to the left by D l (x l ,y) to obtain the new pixel coordinates, I R (x l -Dl (x l ,y) represents pixel (x) r -D l (x l The pixel values ​​corresponding to y) in the right figure, L warp_consistance L represents the consistency loss function for unlabeled pixels. smooth Let L represent the loss function for the labeled pixels, and L represent the first loss function. Furthermore, it should be noted that equations (9) to (11), as well as the left image, right image, and leftward image mentioned in this description, are merely examples. Those skilled in the art can perform corresponding processing based on the actual situation of the binocular image.

[0119] Figure 13 This is a schematic diagram illustrating a geometric consistency optimization operation according to an exemplary embodiment of the present invention. Figure 13 As shown, during the training of the neural network composed of the first acquisition unit 110 and the second acquisition unit 120, the first depth map can be input into one neural network (by way of example only, not a limitation; the neural network can be a convolutional neural network) and the second depth map can be input into another neural network (by way of example only, not a limitation; the other neural network can be a convolutional neural network) to obtain the first normal vector map respectively. Second normal vector diagram And through the normal vector Figure 1 Consistency constraints generate high-quality depth maps, more specifically, the first normal vector map. Second normal vector diagram The differences between them can be used as a second loss function for backpropagation to train the neural network composed of the first acquisition unit 110 and the second acquisition unit 120. According to an exemplary embodiment of the present invention, the first normal vector map can be calculated using the following equation (12). Second normal vector diagram The differences between them. Here, existing traditional methods or deep learning networks can also be used to optimize depth maps, but this invention utilizes depth data obtained from multiple sensors. Figure 1 The optimization process utilizes the geometric constraints between them to obtain a more consistent depth map, thereby achieving a high-quality depth map.

[0120]

[0121] Furthermore, training models typically requires a large amount of labeled data to supervise the network's learning of specific tasks, such as the depth estimation step in this invention. However, labeling data in practice requires significant human and material resources. To save resources, synthetic and real data can be used as training sets, with testing performed on real data. In this case, the network structure composed of the depth map generation device can be modified, transforming the regression network structure into a network structure for two tasks: regression and classification. These two tasks largely share parameters. The classification branch requires the network to correctly classify synthetic and real data, minimizing the binary classification error, while the regression branch requires the network to minimize the regression error, obtaining a correct depth estimate. To achieve domain adaptation, the network structure composed of the depth map generation device can be divided into three parts: a shared encoding module, a regression module, and a classification module. To train the network composed of the depth map generation device simultaneously without ambiguity, a gradient inversion layer can be used. Before this layer, the network parameters regress towards encoding the data from both domains into similar distributions to obtain the optimal depth estimate. After this layer, the network parameters are optimized towards reducing the domain classification loss.

[0122] Figure 14 This is a flowchart illustrating a depth map generation method according to an exemplary embodiment of the present invention.

[0123] like Figure 14 As shown, in step S1410, a first depth map can be generated based on the binocular image.

[0124] In step S1420, a second depth map acquired by the depth camera can be obtained.

[0125] In step S1430, a final depth map can be generated by fusing the first depth map with the second depth map.

[0126] The steps for obtaining the second depth map may include: extracting depth features from the second depth map; and generating a third depth map with depth completion by performing a similarity matrix-guided filter on the extracted depth features, wherein the similarity matrix is ​​generated based on the stereo image.

[0127] The steps for generating a third depth map with depth completion may include: obtaining the similarity between the current pixel and its neighboring pixels in the depth map based on a similarity matrix, and filtering the depth features of the current pixel in a predetermined direction based on the depth features of the neighboring pixels and the similarity.

[0128] The predetermined direction may include at least one of the following: a top-down direction, a bottom-up direction, a left-to-right direction, and a right-to-left direction.

[0129] The steps for generating the first depth map may include: obtaining a first feature map of each monocular image included in the stereo image; obtaining a matching cost matrix based on the first feature map of the monocular image; performing cost aggregation on the matching cost matrix; and generating a depth map of each monocular image based on the cost-aggregated matching cost matrix.

[0130] The step of generating the first depth map may further include: performing a consistency check on the depth map of each monocular image to obtain a reliable depth map for each monocular image.

[0131] The step of obtaining the first feature map may include performing feature extraction on the binocular image using a neural network based on an attention mechanism.

[0132] The steps to obtain the matching cost matrix may include: obtaining an initial matching cost matrix based on the first feature map of the monocular image, and inputting the initial matching cost matrix into a predetermined neural network for convolution to obtain the matching cost matrix.

[0133] The steps for cost aggregation may include performing cost aggregation on the matching cost matrix based on a graph neural network (GNN).

[0134] The steps of cost aggregation may include: obtaining the neighboring nodes of each center node as elements of the matching cost matrix based on the distance, and updating the matching cost of each center node based on the weight of each neighboring node and the matching cost of the neighboring nodes to obtain the matching cost matrix of the first cost aggregation.

[0135] The cost aggregation step may also include using a second feature map to guide the cost aggregation.

[0136] The second feature map can be generated based on a semantic segmentation neural network, or the second feature map can be the same as the first feature map.

[0137] The steps for cost aggregation may include: obtaining the neighboring pixels of each pixel in the second feature map based on the distance; finding the neighboring nodes of each center node as an element of the matching cost matrix according to the pixel correspondence between the second feature map and the matching cost; updating the matching cost of each center node based on the weight of the neighboring nodes and the matching cost to obtain the matching cost matrix of the second cost aggregation; and aggregating the matching cost matrix of the first cost aggregation with the matching cost matrix of the second cost aggregation to obtain the matching cost matrix of the third cost aggregation.

[0138] The distance can be spatial distance, feature vector distance, or a weighted sum of spatial distance and feature vector distance.

[0139] The steps of cost aggregation may include: obtaining a temporary matching cost matrix based on multiple matching cost matrices of different scales, obtaining the probability that each element of the temporary matching cost matrix corresponds to a predetermined disparity, and obtaining a fourth matching cost matrix based on the probability and the temporary matching cost matrix.

[0140] The cost aggregation step may include obtaining a fifth matching cost matrix based on the similarity matrix and the matching cost matrix, wherein the similarity matrix is ​​generated based on the stereo images.

[0141] The steps for generating the first depth map may include: inputting a stereo image into a predetermined neural network for feature extraction to obtain a first feature map of each monocular image included in the stereo image; obtaining a matching cost matrix based on the first feature map of the monocular image; calculating the probability that each element of the matching cost matrix corresponds to a predetermined disparity to convert the matching cost matrix into a matching probability matrix; and generating a depth map of each monocular image based on the matching probability matrix.

[0142] The steps for obtaining a reliable depth map may include: comparing pixels at corresponding positions between depth maps of various monocular images; determining the pixels at corresponding positions as unreliable pixels when the depth difference between pixels at corresponding positions is greater than or equal to a predetermined threshold; and determining the pixels at corresponding positions as reliable pixels when the depth difference between pixels at corresponding positions is less than the predetermined threshold, thereby obtaining a reliable depth map.

[0143] The steps to obtain a reliable depth map may include: processing the first feature map, the initial matching cost matrix, and the matching cost matrix based on the reliable depth map.

[0144] Furthermore, according to exemplary embodiments of the present invention, an electronic device may also be provided, including a memory and a processor, wherein the memory stores computer-executable instructions, and when the instructions are executed by the processor, the aforementioned method is performed. According to exemplary embodiments of the present invention, a computer-readable medium may also be provided, having stored computer-executable instructions thereon, and when the instructions are executed, the aforementioned method is performed.

[0145] According to embodiments of the present invention, high-precision depth maps can be generated by optimizing the depth maps obtained from sensors, which suffer from noise, sparsity, and low resolution, as well as the poor accuracy of depth estimation based on deep learning methods in ill-conditioned regions. Furthermore, by filtering the depth maps obtained from sensors using binocular images, a completed high-resolution depth map can be generated. Additionally, by extracting robust features using a stereo matching deep learning network and transferring these features to ill-conditioned regions, a high-precision depth map can be generated. Furthermore, by using a depth optimization network to jointly optimize the first depth map and the completed second depth map through consistency constraints, a high-quality depth map can be generated. Moreover, by employing a multi-sensor-based high-quality depth map generation method, the advantages of each depth sensor can be utilized to obtain high-quality depth maps, making it applicable even in challenging scenarios.

[0146] Although the invention has been shown and described with reference to specific exemplary embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the claims and their equivalents.

Claims

1. A method for generating a depth map, comprising: obtaining a first depth map based on binocular images; obtaining a second depth map using a depth camera; obtaining a final depth map by fusing the first depth map and the second depth map; wherein the obtaining the first depth map comprises: obtaining a first feature map of each monocular image included in the binocular images, obtaining a matching cost matrix based on the first feature map of the monocular image, performing cost aggregation on the matching cost matrix, and obtaining a depth map of each monocular image based on the cost aggregated matching cost matrix, wherein the performing cost aggregation comprises: obtaining neighbor nodes of each center node as an element of the matching cost matrix according to a distance, and updating a matching cost of each center node based on a weight of each neighbor node and a matching cost of the neighbor node to obtain a first cost aggregated matching cost matrix.

2. The depth map generation method of claim 1, wherein, the obtaining the second depth map comprises: extracting depth features from a third depth map, the third depth map being captured by the depth camera; performing depth completion by filtering the extracted depth features guided by a similarity matrix to obtain the second depth map; wherein the similarity matrix is generated based on the binocular images.

3. The depth map generation method of claim 2, wherein, the performing depth completion to obtain the second depth map comprises: obtaining a similarity between a current pixel of the depth map and a neighbor pixel of the current pixel based on the similarity matrix, and filtering a depth feature of the current pixel in a predetermined direction based on a depth feature of the neighbor pixel and the similarity.

4. The depth map generation method of claim 3, wherein, the predetermined direction comprises at least one of a direction from top to bottom, a direction from bottom to top, a direction from left to right, and a direction from right to left.

5. The depth map generation method of claim 1, wherein, the obtaining the first depth map further comprises: performing consistency detection on the depth map of each monocular image to obtain a trusted depth map of each monocular image.

6. The depth map generation method of claim 1, wherein, the obtaining the first feature map comprises feature extraction on the binocular images based on a neural network of an attention mechanism.

7. The depth map generation method of claim 6, wherein, the feature extraction on the binocular images comprises: performing a max-pooling operation on a feature map of each channel of the binocular images to obtain an element; obtaining an attention parameter by inputting the obtained element into the neural network; obtaining a new feature map based on the obtained attention parameter.

8. The depth map generation method of claim 1, wherein, the obtaining the matching cost matrix comprises: obtaining an initial matching cost matrix based on the first feature map of each monocular image, and obtaining the matching cost matrix by performing convolution on the initial matching cost matrix through a predetermined neural network.

9. The depth map generation method of claim 1, wherein, the performing cost aggregation comprises performing cost aggregation on the matching cost matrix based on a graph neural network (GNN).

10. The depth map generation method of claim 9, wherein, the performing cost aggregation on the matching cost matrix comprises: performing cost aggregation on the matching cost matrix by constructing a graph according to the matching cost matrix, and connecting and inputting a matching cost of a neighbor node corresponding to each center node in the graph along a feature dimension into the graph neural network to obtain a weight of an edge between each center node and a neighbor node closest to the center node in terms of distance.

11. The depth map generation method of claim 1, wherein, the performing cost aggregation further comprises guiding the cost aggregation using a second feature map.

12. The depth map generation method of claim 11, wherein, the performing cost aggregation comprises: obtaining neighbor nodes of each pixel of the second feature map according to a distance; obtaining neighbor nodes of each center node as an element of the matching cost matrix according to a pixel correspondence between the second feature map and the matching cost; The matching cost matrix of the second cost aggregation is obtained by updating the matching cost of each center node based on the weight of the neighbor node and the matching cost. The matching cost matrix of the third cost aggregation is obtained by aggregating the matching cost matrix of the first cost aggregation and the matching cost matrix of the second cost aggregation.

13. The depth map generation method of claim 12, wherein, The second feature map is generated based on a semantic segmentation neural network, or the second feature map is the same as the first feature map.

14. The depth map generation method of claim 1, wherein, The distance is a spatial distance, a feature vector distance, or a weighted sum of the spatial distance and the feature vector distance.

15. The depth map generation method of claim 1, wherein, The cost aggregation includes: obtaining a temporary matching cost matrix according to a plurality of matching cost matrices of different scales, each element of the temporary matching cost matrix corresponding to a probability of a predetermined disparity, and obtaining a fourth matching cost matrix according to the probability and the temporary matching cost matrix.

16. The depth map generation method of claim 8, wherein, The cost aggregation includes obtaining a fifth matching cost matrix based on a similarity matrix and a matching cost matrix, The similarity matrix is obtained based on monocular images included in the binocular images.

17. The depth map generation method of any one of claims 2-4, wherein, The similarity matrix is obtained based on monocular images included in the binocular images.

18. An electronic device comprising a memory and a processor, the memory having stored thereon computer executable instructions which, when executed by the processor, perform the method of any one of claims 1-17.

19. The electronic device of claim 18, wherein, The electronic device further comprises a binocular camera and a depth camera.

20. The electronic device of claim 18 or 19, wherein, The depth camera comprises a structured light sensor (SL) or a time-of-flight sensor (TOF).

21. A computer readable medium having stored thereon computer executable instructions which, when executed, perform the method of any one of claims 1-17.

22. A method for generating a depth map, comprising: selecting monocular images from binocular images; obtaining a first depth map based on the binocular images; obtaining a similarity matrix between image pixels of the selected monocular images using a first neural network; performing feature extraction on a depth image obtained using a depth camera using a second neural network to obtain depth features; performing filtering on the obtained depth features based on the obtained similarity matrix to obtain a second depth map; obtaining a final depth map by performing image fusion on the obtained first depth map and the obtained second depth map; wherein obtaining the first depth map comprises: obtaining a first feature map of each monocular image included in the binocular images, obtaining a matching cost matrix based on the first feature map of the monocular image, performing cost aggregation on the matching cost matrix, and obtaining a depth map of each monocular image based on the matching cost matrix of the cost aggregation, wherein the cost aggregation comprises: obtaining a neighbor node of each center node as an element of the matching cost matrix according to a distance, and updating the matching cost of each center node based on the weight of each neighbor node and the matching cost of the neighbor node to obtain a matching cost matrix of the first cost aggregation; or obtaining a first feature map of each monocular image included in the binocular images, obtaining a matching cost matrix based on the first feature map of the monocular image, calculating a probability that each element of the matching cost matrix corresponds to a predetermined disparity to convert the matching cost matrix into a matching probability matrix, and obtaining a depth map of each monocular image based on the matching probability matrix.

Images