System for disparity estimation and method for disparity estimation in the system

Abstract

A system for disparity estimation and a method for disparity estimation of the system are disclosed. The system for disparity estimation includes: one or more feature extractor modules configured to extract one or more feature maps from one or more input images; and one or more semantic information modules connected to one or more outputs of the one or more feature extractor modules, wherein the one or more semantic information modules are configured to: generate one or more foreground semantic information to be provided to the one or more feature extractor modules for disparity estimation in the next training period.

Description

[0001] This application claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 62 / 768,055, filed on November 15, 2018, and entitled “Foreground-Background Aware Dilated Multi-Scale Network for Disparity Estimation,” the entire contents of which are expressly incorporated herein by reference. Technical Field

[0002] One or more aspects of an embodiment according to the present disclosure relate to a foreground-background aware atrous multi-scale network (FBA-AMNet) for disparity estimation. Background Art

[0003] Depth estimation is a fundamental computer vision problem that aims to predict a measurement of the distance to each point in a captured scene. This has many applications, such as the ability to separate foreground (near) objects from background (far) objects. Accurate depth estimation allows the background in a scene to be separated from foreground objects of interest, and allows images from non-professional photographers or cameras with smaller lenses to be processed to obtain more aesthetically pleasing images that are focused on the subject.

[0004] The above information in the Background Art section is only for enhancement of understanding of the background of the technology and therefore it should not be construed as an admission of the existence or relevance of prior art. Summary of the invention

[0005] This summary is provided to introduce a selection of features and concepts of embodiments of the present disclosure that will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. One or more of the described features can be combined with one or more other described features to provide a feasible device.

[0006] Aspects of example embodiments of the present disclosure relate to a foreground-background aware dilated multi-scale network for disparity estimation.

[0007] According to some example embodiments of the present disclosure, a system for disparity estimation includes: one or more feature extractor modules, configured to extract one or more feature maps from one or more input images; and one or more semantic information modules, connected at one or more outputs of the one or more feature extractor modules, wherein the one or more semantic information modules are configured to generate one or more foreground semantic information to be provided to the one or more feature extractor modules for disparity estimation in the next training period.

[0008] In some embodiments, the system further includes: an extended cost volume (ECV) module connected at the one or more outputs of the one or more feature extractor modules, the ECV module being configured to calculate matching cost information between the one or more feature maps; a stacked hole multi-scale (AM) module connected at the output of the ECV module and configured to process the matching cost information between the one or more feature maps from the ECV module to aggregate multi-scale context information, the stacked AM module comprising a plurality of AM modules stacked together; and a regression module connected at the output of the stacked AM module and configured to estimate the disparity of the system based on the aggregated multi-scale context information from the stacked AM module and one or more foreground semantic information.

[0009] In some embodiments, the ECV module includes: a disparity level feature distance sub-volume module configured to determine a pixel-by-pixel absolute difference between a first feature map and a second feature map; a disparity level depth correlation sub-volume module configured to determine a correlation between the first feature map and the second feature map; and a disparity level feature cascade sub-volume module configured to cascade the first feature map and the second feature map offset by d at each disparity level d. In some embodiments, the size of the disparity level feature distance sub-volume module is H×W×(D+1)×C, where H, W, and C represent height, width, and feature size, and D represents the maximum disparity that the system can predict; the size of the disparity level depth correlation sub-volume module is H×W×(D+1)×C; and the size of the disparity level feature cascade sub-volume module is H×W×(D+1)×2C.

[0010] In some embodiments, the system is further configured to determine the size of the ECV module by cascading the disparity level features of the distance sub-volume module, the disparity level depth correlation sub-volume module, and the disparity level feature cascade sub-volume module along the depth dimension, wherein the size of the ECV module is H×W×(D+1)×4C. In some embodiments, the stacked AM module includes a plurality of AM modules stacked together using shortcut connections within the stacked AM module, wherein the AM modules of the plurality of AM modules of the stacked AM module are configured to process the matching cost information between the one or more feature maps from the ECV using k pairs of 3×3 dilated convolutional layers and two 1×1 convolutional layers. In some embodiments, the k pairs of 3×3 dilated convolutional layers have dilation factors [1, 2, 2, 4, 4, …, k / 2, k / 2, k], wherein two 1×1 convolutional layers with dilation factor one are added at the end of the AM module of the plurality of AM modules for feature refinement and feature size adjustment.

[0011] In some embodiments, the one or more feature extractor modules include: a first deep separable residual network (D-ResNet) module configured to receive a first input image and first foreground semantic information; a second D-ResNet module configured to receive a second input image and second foreground semantic information; a first AM module connected at the output of the first D-ResNet module; and a second AM module connected at the output of the second D-ResNet module. In some embodiments, the first D-ResNet module and the second D-ResNet module have shared weights, the first AM module and the second AM module have shared weights, wherein each of the first AM module and the second AM module is configured as a scene understanding module for capturing deep global context information and local details, wherein the ECV module is connected at the output of the first AM module and the output of the second AM module.

[0012] In some embodiments, the one or more semantic information modules include: a first semantic information module connected at the output of the first AM module, wherein the first semantic information module is configured to generate first foreground semantic information, wherein the first foreground semantic information is provided to the first D-ResNet module via a first feedback loop as an additional input to the system for the next training period of the system; and a second semantic information module connected at the output of the second AM module, wherein the second semantic information module is configured to generate second foreground semantic information, wherein the second foreground semantic information is provided to the second D-ResNet module via a second feedback loop as an additional input to the system for the next training period of the system.

[0013] In some embodiments, the first semantic information module includes: a first convolutional neural network (CNN) module connected to the output of the first AM module; a first upsampling module connected to the output of the first CNN module; and a first prediction module connected to the output of the first upsampling module and configured to generate first foreground semantic information. In some embodiments, the second semantic information module includes: a second convolutional neural network (CNN) module connected to the output of the second AM module; a second upsampling module connected to the output of the second CNN module; and a second prediction module connected to the output of the second upsampling module and configured to generate second foreground semantic information. In some embodiments, the system is a multi-task module configured to perform two tasks, wherein the two tasks are disparity estimation and foreground semantic information generation, wherein the loss of the system is a weighted sum of two losses from the two tasks.

[0014] According to some example embodiments of the present disclosure, a method for disparity estimation of a system, the system comprising one or more feature extractor modules, one or more semantic information modules, an extended cost volume (ECV) module, a stacked hole multi-scale (AM) module and a regression module, the method comprising: extracting one or more feature maps from one or more input images through the one or more feature extractor modules; generating one or more foreground semantic information through the one or more semantic information modules connected at one or more outputs of the one or more feature extractor modules, wherein the one or more foreground semantic information are provided to the one or more feature extractor modules; calculating matching cost information between the one or more feature maps through the ECV module connected at the one or more outputs of the one or more feature extractor modules; processing the matching cost information between the one or more feature maps from the ECV module through the stacked AM module connected at the output of the ECV module to aggregate multi-scale context information for disparity regression; estimating the disparity of the system based on the aggregated multi-scale context information and foreground semantic information through the regression module connected at the output of the stacked AM module; and recursively training the system using the one or more feature maps and the one or more foreground semantic information until convergence.

[0015] In some embodiments, one or more foreground semantic information of the current period is calculated by the one or more semantic information modules in the previous period, wherein the one or more input images include a first input image and a second input image, wherein the one or more feature maps extracted from the one or more input images include a first feature map extracted from the first input image and a second feature map extracted from the second input image, and wherein the method further includes: determining the pixel-by-pixel absolute difference between the first feature map and the second feature map through a disparity level feature distance sub-volume module of the ECV module; determining the correlation between the first feature map and the second feature map through a disparity level depth correlation sub-volume module of the ECV module; and cascading the first feature map and the second feature map with an offset d at each disparity level d through a disparity level feature cascade sub-volume module.

[0016] In some embodiments, the size of the disparity level feature distance sub-volume module is H×W×(D+1)×C, where H, W and C represent height, width and feature size, and D represents the maximum disparity that the system can predict; the size of the disparity level depth correlation sub-volume module is H×W×(D+1)×C; and the size of the disparity level feature cascade sub-volume module is H×W×(D+1)×2C.

[0017] In some embodiments, the method further comprises: determining the size of the ECV module by cascading the disparity level feature of the distance sub-volume module, the disparity level depth correlation sub-volume module and the disparity level feature cascade sub-volume module along the depth dimension, wherein the size of the ECV module is H×W×(D+1)×4C.

[0018] In some embodiments, the method also includes: generating first foreground semantic information through a first semantic information module of the one or more semantic information modules; receiving a first input image and first foreground semantic information through a first deep separable residual network (D-ResNet) module of the one or more feature extractor modules, wherein the first foreground semantic information is provided to the first D-ResNet module via a first feedback loop as an additional input for the next training period of the system; generating second foreground semantic information through a second semantic information module of the one or more semantic information modules; receiving a second input image and second foreground semantic information through a second D-ResNet module of the one or more feature extractor modules, wherein the second foreground semantic information is provided to the second D-ResNet module via a second feedback loop as an additional input for the next training period of the system; and capturing deep global context information and local details for scene understanding through the first AM module and the second AM module of the one or more feature extractor modules.

[0019] In some embodiments, the stacked AM module includes multiple AM ​​modules stacked together using shortcut connections within the stacked AM module, wherein the method further includes: processing the matching cost information between the one or more feature maps from the ECV using k pairs of 3×3 dilated convolution layers and two 1×1 convolution layers through the AM module of the multiple AM ​​modules of the stacked AM module, wherein the k pairs of 3×3 dilated convolution layers have dilation factors [1, 2, 2, 4, 4,…, k / 2, k / 2, k], and wherein two 1×1 convolution layers with dilation factor one are added at the end of the AM module for feature refinement and feature size adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] These and other features of some example embodiments of the present disclosure will be appreciated and understood with reference to the specification, claims, and drawings, in which:

[0021] Figure 1 A block diagram showing an atrous multiscale network (AMNet) according to some embodiments of the present disclosure;

[0022] Figure 2AA block diagram showing a residual network (ResNet) block according to some embodiments of the present disclosure;

[0023] Figure 2B A block diagram showing a depthwise separable ResNet (D-ResNet) block according to some embodiments of the present disclosure;

[0024] Figure 3 shows the development of an extended cost volume (ECV) module according to some embodiments of the present disclosure;

[0025] Figure 4 shows the structure and specifications of a void multiscale (AM) module and a stacked AM module according to some embodiments of the present disclosure;

[0026] Figure 5A illustrates a multi-tasking network according to some embodiments of the present disclosure;

[0027] Figure 5B Another multi-tasking network according to some embodiments of the present disclosure is shown;

[0028] Figure 6 The FBA-AMNet system according to some embodiments of the present disclosure is shown;

[0029] Figure 7 A method for disparity estimation for an FBA-AMNet system according to some embodiments of the present disclosure is shown;

[0030] Figure 8 The disparity estimation results of two foreground objects by AMNet and FBA-AMNet according to some embodiments of the present disclosure are shown; and

[0031] Fig. 9 An image and coarse-to-fine foreground-background segmentation results generated by FBA-AMNet according to some embodiments of the present disclosure are shown. DETAILED DESCRIPTION

[0032] The specific embodiments described below in conjunction with the accompanying drawings are intended as a description of some example embodiments of the foreground-background perception hole multi-scale network for disparity estimation provided according to the present disclosure, and are not intended to represent the only form in which the present disclosure can be interpreted or utilized. This description describes the features of the present disclosure in conjunction with the illustrated embodiments. However, it should be understood that the same or equivalent functions and structures can be implemented by different embodiments, which are also intended to be included in the scope of the disclosure. As shown elsewhere herein, the same element numbers are intended to indicate the same elements or features.

[0033] Depth estimation is a fundamental computer vision problem that aims to predict the measurement of the distance of each point in a captured scene. There is a recent interest in the estimation of the real-world depth of elements in a captured scene. This has many applications, such as the ability to separate foreground (near) objects from background (far) objects. Accurate depth estimation allows the background in a scene to be separated from the foreground object of interest. Accurate foreground-background separation allows the captured image to be processed to simulate effects such as bokeh effects. Bokeh is a soft out-of-focus blur of the background, which is mastered by using the right settings in an expensive camera with a fast lens and a wide aperture and bringing the camera closer to the subject and moving the subject away from the background to simulate a shallow depth of field. Therefore, accurate depth estimation allows processing images from non-professional photographers or cameras with smaller lenses (such as mobile phone cameras) to obtain more aesthetically pleasing images with bokeh effects that are focused on the subject. Other applications of accurate depth estimation include three-dimensional (3D) object reconstruction and virtual reality applications, where it is desirable to change the background or subject and render them according to the desired virtual reality. Other applications of accurate depth estimation from captured scenes are in automotive automation, surveillance cameras, autonomous driving applications, and improved safety by improving object inspection accuracy and estimating the distance of an object from the camera using only the camera or depth estimation from the camera input and from multiple sensors.

[0034] Given a rectified stereo image pair, the depth estimate can be converted to a disparity estimate using camera calibration. For each pixel in one image, the disparity estimate finds the offset between a pixel and its corresponding pixel in the other image on the same horizontal line, such that the two pixels are projections of similar 3D positions.

[0035] Some embodiments of the present disclosure provide a system and method for estimating the real-world depth of elements in a scene captured by two stereo cameras. Two stereo rectified images are captured and processed to accurately calculate the disparity between pixels in the two images. The pipeline of the stereo matching disparity estimation system includes three parts: feature extraction from left and right images captured by stereo cameras, matching cost calculation between left feature maps and right feature maps, and post-processing and disparity regression through a scene understanding module.

[0036] Some embodiments of the present disclosure provide designs for components of a disparity estimation system. For example, an extended cost volume module for matching cost calculation is provided. The extended cost volume module includes an original cost volume module, a disparity level correlation module, and a disparity level difference module. A stacked hole convolution module with a skip connection inside is also provided for post-processing. Soft classification-based regression is used for disparity regression. Some embodiments of the present disclosure may be applicable to object segmentation.

[0037] Some embodiments may utilize semantic information (specifically, using semantic segmentation or semantic boundary information of foreground objects) to enhance the disparity estimation task. Foreground semantic information can be used in two ways (e.g., adding one or more tasks to the disparity estimation network for foreground semantic segmentation / boundary prediction, and using foreground semantic segmentation or boundary maps as additional input features (RGB-S input) in addition to red, green, and blue (RGB) images). The two methods can also be combined together in one network. A multi-task network with RGB-S input is recursively trained, where the input semantic map of the current epoch is calculated by the multi-task network in the previous epoch.

[0038] Some embodiments of the present disclosure provide a deep learning architecture for disparity estimation with a design of the first three components (e.g., feature extractor, cost body, second-level processing module). In some embodiments, the example network adopts a feature extractor based on deep separable convolution, a cost body for calculating matching costs using different similarity metrics, and a scene understanding module for aggregating rich multi-scale contextual information using dilated convolution. Figure 3 A cost body for calculating matching costs using different similarity metrics is described in detail. Some embodiments of the present disclosure also provide a multi-task network that utilizes foreground-background segmentation information to enhance itself with better foreground-background perception. In some embodiments, the example network is trained end-to-end using an iterative training method. In some embodiments, the training method outperforms prior art methods by a significant margin and achieves the highest level on the three most popular disparity estimation benchmarks (KITTI stereo 2015, KITTI stereo 2012, and Sceneflow).

[0039] A disparity estimation system based on a convolutional neural network (CNN) may include feature extraction, matching cost estimation, second-level processing, and disparity regression. The feature extractor extracts distinguishable high-level feature maps from the left and right input images, and the choice may be a residual network. The cost body calculates the matching cost between the left feature map and the right feature map. Correlation, pixel-by-pixel difference, or simple cascading are used together for cost body calculation. The post-processing module uses a collection of convolutional layers or a scene understanding module to further process and refine the output from the cost body. Finally, the disparity regressor or disparity classifier performs pixel-by-pixel disparity prediction.

[0040] For example, in a CNN-based disparity estimation system, first, deep features are extracted from the rectified left and right images using a CNN-based feature extractor (such as ResNet-50 or VGG-16). Then, a cost volume (CV) is formed by measuring the matching cost between the left feature map and the right feature map. The selection of matching costs may include correlation, absolute distance, and / or feature concatenation. The CV is further processed and refined by a second-level processing module for disparity regression. In addition, information from other low-level vision tasks (such as semantic segmentation or edge detection) can be used to enhance the disparity estimation system.

[0041] Figure 1 A block diagram of a dilated multi-scale network (AMNet) 100 is shown according to some embodiments of the present disclosure. AMNet 100 is a CNN-based disparity estimation system. Figure 1 The AMNet 100 includes a first depthwise separable ResNet (D-ResNet) 106, a second D-ResNet 108, a first dilated multi-scale (AM) module 110, a second AM module 112, an extended cost volume (ECV) module 114, a stacked AM module 118, an upscale module 120, and a regression module 122.

[0042] exist Figure 1 In the example embodiment, the standard ResNet-50 backbone is modified to D-ResNet (e.g., 106, 108) as the backbone of AMNet 100. Figure 1 In the AMNet 100 of FIG. 1 , the first D-ResNet 106 receives the first input image 102 as input, and the second D-ResNet 108 receives the second input image 104 as input. The first D-ResNet 106 and the second D-ResNet 108 may have shared weights. The first AM module 110 is connected to the output of the first D-ResNet 106, and the second AM module 112 is connected to the output of the second D-ResNet 108. The AM module 110 and the AM module 112 may also have shared weights.

[0043] Each of the AM modules 110 and 112 is designed as a scene understanding module that captures deep global context information as well as local details. In some embodiments, the D-ResNet modules (e.g., first D-ResNet 106, second D-ResNet 108) following the AM modules (e.g., 110, 112) can be used as feature extractors. Therefore, the combination of the first D-ResNet 106 and the first AM module 110 can be used to extract features of the first input image 102, and the combination of the second D-ResNet 108 and the second AM module 112 can be used to extract features of the second input image 104.

[0044] The ECV module 114 is connected to the outputs of the AM modules 110 and 112. The ECV module 114 is a combination of a disparity level depthwise correlation sub-volume, a disparity level feature distance sub-volume, and a disparity level feature concatenation sub-volume. The ECV module 114 carries rich information about the matching costs under different similarity metrics. The ECV module 114 can be processed by a stacked AM module 118 (connected to the output of the ECV module 114) and can predict disparity using a flexible argmin-based disparity regression. In some embodiments, argmin is an argument that minimizes the cost function.

[0045] In some embodiments, foreground-background segmentation information can be used to enhance disparity estimation. AMNet 100 can be extended to a foreground-background aware AM-Net (FBA-AMNet) that utilizes foreground-background segmentation to improve disparity estimation. The foreground-background segmentation map can be provided to AMNet 100 as an additional input feature (RGB-S input) to AMNet 100, and AMNet 100 is extended to a multi-task network in which the main task is disparity estimation and the auxiliary task is foreground-background segmentation. In some embodiments, the multi-task network is referred to as FBA-AMNet. The auxiliary tasks help the network (e.g., FBA-AMNet) have better foreground-background perception, thereby further improving disparity estimation.

[0046] Aspects of the various components of AMNet 100 will now be described in more detail in the following sections.

[0047] Figure 2A shows a block diagram of a ResNet 201 block according to some embodiments of the present disclosure, Figure 2B A block diagram of a D-ResNet 203 block according to some embodiments of the present disclosure is shown. The D-ResNet 203 block may be Figure 1 The first D-ResNet 106 or the second D-ResNet 108 of the AMNet 100.

[0048] Depthwise separable convolutions decompose standard convolutions into depthwise convolutions (e.g., 206 or 210) followed by 1×1 convolutions (e.g., 208 or 212). Depthwise separable convolutions have shown great potential in image classification tasks and have been further developed as network backbones for other computer vision tasks. In the example embodiment of D-ResNet 203, in a standard ResNet-50 (e.g., ResNet 201) backbone network, standard convolutions are replaced with customized depthwise separable convolutions.

[0049] The standard 3×3 convolutional layer in the ResNet-50 (e.g., ResNet 201) backbone contains 9×D in ×D out parameters, while the depthwise separable convolutional layer (e.g., D-ResNet 203) contains D in ×(9+D out ) parameters, among which D in and D out Represents the size of the input feature map and the output feature map respectively. Because D in the ResNet model out It can be, for example, 32 or larger, so directly replacing a standard convolution (e.g., 202 or 204) with a depthwise separable convolution (e.g., a depthwise convolution 206 followed by a 1×1 convolution 208 or a depthwise convolution 210 followed by a 1×1 convolution 212) will result in a model with very small complexity. In the example embodiment of D-ResNet 203, in the depthwise separable convolution layer (the depthwise convolution 206 followed by a 1×1 convolution 208 or the depthwise convolution 210 followed by a 1×1 convolution 212), D out It has been further increased so that the number of parameters in D-ResNet 203 is close to that of ResNet-50 (e.g., ResNet 201).

[0050] For each D-ResNet block (e.g., 203) modified from a ResNet block (e.g., 201), a 1×1 convolution (e.g., 214) is implemented on the input feature map in a shortcut connection for feature size matching.

[0051] Figure 2A and Figure 2B A comparison between a standard ResNet block (e.g., ResNet 201) and a D-ResNet block (e.g., D-ResNet 203) is shown. The network specifications of the D-ResNet (e.g., D-ResNet 203) backbone are listed in Table 1. Rectified Linear Units (ReLU) and batch normalization are used after each layer. After the D-ResNet (e.g., D-ResNet 203) backbone, the shape of the feature map is 1 / 4 of the shape of the input image.

[0052] In some embodiments, a 50-layer residual network (e.g., ResNet 201) can be modified in PSMNet to a feature extractor consisting of 4 groups of residual blocks, where each residual block consists of two convolutional layers with a 3×3 convolution kernel. The number of residual blocks in the 4 groups is {3; 16; 3; 3} respectively. In the ResNet of PSMNet, for the four residual groups, the number of output feature maps is D out ={32; 64; 128; 128}, where for all residual blocks, D in =D out . Due to D out is 32 or larger, so directly replacing the standard convolution with depthwise separable convolution will result in a model with a very small number of parameters. However, in the current D-ResNet (e.g., D-ResNet 203), for the depthwise separable convolution layers in the four residual blocks, D out Increase to D out ={96; 256; 256; 256}, where for the first residual block, D in =32, so that the number of parameters in the current D-ResNet (e.g., D-ResNet 203) is close to the number of parameters of PSMNet. Therefore, the current D-ResNet (e.g., D-ResNet 203) can learn more deep features than ResNet (e.g., ResNet 201) while having similar complexity. Since the depthwise separable residual blocks (e.g., 206, 210) may not have the same number of input features and output features, D out Point-wise 1×1 projection filters can be deployed on shortcut (residual) connections to transform D in Input features are projected to D out Characteristics (e.g. Figure 2A and Figure 2B Shows the standard ResNet( Figure 2A ) and D-ResNet( Figure 2B). ReLU and batch normalization are used after each layer. After the D-ResNet backbone, the width and height of the output feature map are 1 / 4 of the width and height of the input image. The network specifications of the D-ResNet backbone are listed in Table 1.

[0053]

[0054]

[0055] Table 1: Detailed layer specifications for D-ResNet. "Repeat" indicates that the current layer or block is repeated a specific number of times. "Str." and "Dil." represent the stride and dilation factor (e.g., in some embodiments, the stride is the step size or amount of movement when the filter slides, and the dilation factor is the ratio of the filter upscaling or upsampling after inserting zeros), and "Sepconv" represents separable convolution.

[0056] Figure 3 The development of an ECV module for calculating the matching cost between feature maps of a pair of stereo images according to some embodiments of the present disclosure is shown. The ECV module may be Figure 1 The ECV module 114 is provided.

[0057] The ECV module 114 is developed to perform left feature map (F l ) and the right feature map (F r ) are summarized by using three different matching costs between the two. The ECV module 114 may include three sub-volumes, for example, a disparity level feature distance sub-volume, a disparity level depth correlation sub-volume, and a disparity level feature cascade sub-volume.

[0058] Given an AMNet (e.g., AMNet 100), the maximum disparity that can be predicted is D. A disparity level represents a disparity value d from 0 to D. At a disparity value d, F r Shift right by d pixels to match F with necessary trimming and zero padding l For example, for a disparity level d, first the right feature map F r Shift d pixels to the right (e.g., 301), and then add the right feature map F r With the left feature map F l Cascade (e.g., 302). Remove two feature maps F l and F r The out-of-bounds portion (e.g., 302) of the left feature map F is removed, and zero padding (e.g., 303) is performed to fill the new feature map (e.g., 304) with the same pixel value as the original left feature map F. l The same width as the width of the (for example, 303). Figure 3 This process is shown in FIG.

[0059] In some embodiments, in the disparity level feature distance subvolume, F is calculated across all disparity levels. l With F r The point-by-point (e.g., pixel-by-pixel) absolute difference between l The size of is H×W×C, where H, W, and C represent the height, width, and feature size, and all D+1 difference maps are packed together to form a sub-volume of size H×W×(D+1)×C. In some embodiments, in (D+1), 1 is the left feature F without any offset l With characteristic right F r The method is repeated D times until the maximum disparity offset D of the left image relative to the right image is obtained.

[0060] For example, at each disparity level d, the right feature map F is shifted by d r and the left feature map F l After alignment, the pixel-by-pixel absolute difference is calculated. All (D+1) difference maps are packed together to form a sub-volume of size H×W×(D+1)×C.

[0061] In the disparity level depth dependency subvolume, F is defined for a square block of size 2t+1 (eg, “t” is a parameter) l The x in 1 The center block p l With F r The x in 2 The center block p 2 The correlation between them is as shown in Equation 1:

[0062] c(x 1 , x 2 )=∑ O∈[-t，t]×[-t，t] <F l (x 1 +o), F r (x 2 +o)>. (1)

[0063] Therefore, instead of calculating p l With x 1 The correlation between all other blocks within a neighborhood of size D (e.g., extending along the horizontal line) centered at the value p across all disparity levels (e.g., extending along the disparity level) is l Aligned with F r p lThe correlation between corresponding blocks of is calculated. This results in a sub-volume of size H×W×(D+1)×C. In order to make the size of the output feature map comparable to other sub-volumes, depth correlation can be implemented. At each disparity level, the depth correlation of two aligned blocks is calculated and packed across all depth channels, as shown in Equations 2 and 3 (below).

[0064]

[0065] c(x 1 , x 1 )=[c 0 (x 1 , x 1 ), c 1 (x 1 , x 1 ), ..., c C (x 1 , x 1 )], (3)

[0066] The depth correlation is calculated for all blocks across all disparity levels, which results in a sub-volume of size H×W×(D+1)×C.

[0067] In the disparity level feature cascade subbody, at each disparity level d, the right feature map F with offset d r With the left feature map F l Direct concatenation. All D+1 concatenated feature maps are packed together to form a sub-volume of size H×W×(D+1)×2C. In some embodiments, for the left feature map F l For each shift of the left feature in , compute the difference map.

[0068] Finally, all the children of ECV are concatenated along the depth dimension, which results in an ECV of size H×W×(D+1)×4C. ECV can provide rich information to subsequent modules (e.g., stacked AM modules, regression modules) in the system (e.g., AMNet 100), which is used to describe the left feature map F at all disparity levels. l and right feature map F r In some embodiments, 3D convolution is implemented in subsequent convolutional layers (e.g., stacked AM modules 118) in a network (e.g., AMNet 100) by introducing the disparity dimension into an ECV module (e.g., ECV module 114).

[0069] Figure 4 The structure and specification of the AM module and the stacked AM module according to some embodiments of the present disclosure are shown. Figure 4 In , “Dil.” represents the dilation factor in each atrous convolutional layer.

[0070] Figure 4 The stacked AM module 402 is a second-level processing module connected to the output of the ECV module. The stacked AM module 402 processes the output from the ECV module (e.g., the left feature map F at all disparity levels). l With the right feature map F r ) to aggregate multi-scale contextual information for disparity regression.

[0071] In CNN-based low-level vision systems, the atrous convolution module and the scene understanding module can be used to aggregate multi-scale contextual information for dense disparity prediction. Figure 4 The stacked AM modules can be used as scene understanding modules. Figure 4 The stacked AM module 402 includes three AM blocks 404, 406, and 408. The three AM blocks 404, 406, and 408 are stacked together using internal shortcut connections to form the stacked AM module 402. In some embodiments, the shortcut connection is a connection between neural network layers that skips some intermediate layers.

[0072] The AM block is a block of convolutional layers with an encoder-decoder structure. The AM block can process the input feature map using k pairs of 2-stride convolution layers and 1-stride convolution layers as the encoder part. The choice of k can be, for example, 3, 4, or 5. Then, a set of deconvolution layers is implemented as the decoder part for further processing the feature map and upscaling the feature map to its original size.

[0073] In some embodiments, in the stacked AM module 402, all convolutional layers in the encoder of each of the AM blocks (e.g., 404, 406, and 408) are dilated convolutional layers (e.g., Figure 4, only AM block 406 is expanded, however, all convolutional layers in each of AM blocks 404 and 408 are also dilated convolutional layers similar to AM block 406). The dilation factor in each AM block (e.g., 406) can be increased by a multiple of 2. Because dilated convolutions naturally carry more information and maintain the size of feature maps, the decoder portion can be removed and two additional convolutional layers can be added at the end for feature refinement. In one example, an AM block (e.g., 404, 406, or 408) utilizes k pairs of 3×3 dilated convolutional layers and two 1×1 convolutional layers to process the matching cost information between feature maps from the ECV module, where k is an integer power of 2 and greater than 0. For example, the stacked AM block 406 is designed as a collection of 3×3 dilated convolutions with different dilation factors [1, 2, 2, 4, 4, …, k / 2, k / 2, k]. Without losing spatial resolution, the dilation factor increases as the AM block 406 becomes deeper to capture dense multi-scale contextual information. In the AM block 406, two 1×1 convolutions with dilation factor 1 are added at the end for feature refinement and feature resizing.

[0074] To aggregate more coarse-to-fine context information, the cascading of three AM modules (eg, 404 , 406 , and 408 ) is implemented using internal shortcut connections to form a stacked AM module 402 .

[0075] If targeted Figure 1 As discussed, the AM modules (e.g., 110, 112) after the D-ResNet (e.g., 106, 108) backbone network can be used to form a feature extractor, and the stacked AM modules (e.g., 118) after the ECV (e.g., 114) can be used as a second-level processing module. 3D convolution is implemented in the stacked AM modules (e.g., 118).

[0076] In some embodiments, the upscaling module 120 may be an upsampling module that renders input features to a higher resolution.

[0077] In some embodiments, a soft argmin operation may be employed for disparity regression (eg, in regression module 122). For one output layer, the expectation of D+1 disparities is calculated as the final disparity prediction, as shown in Equation 4:

[0078]

[0079] in, is the flexible maximum probability of disparity j at pixel i, and D is the maximum disparity value.

[0080] For example, the predicted disparity d iThe stacked AM module (e.g., 118, 402) may be based on the aggregated multi-scale context information from the stacked AM module (e.g., 118, 402). In some embodiments, the stacked AM module (e.g., 118, 402) generates higher-level features based on the cascaded feature maps in the ECV module (e.g., 114), which are then used to estimate the disparity d i . This is called cost aggregation and disparity calculation.

[0081] Smoothing loss L 1 Can be used to measure the predicted disparity d i Disparity with ground truth The loss is calculated as the average smoothing loss L over all labeled pixels 1 During training, three losses are calculated separately from the three AM blocks (e.g., 404, 406, and 408) in the stacked AM module (e.g., 402), and the three losses are summed to form the final loss, as shown in Equations (5) and (6).

[0082]

[0083]

[0084] Where N is the total number of labeled pixels. In some embodiments, the final loss Train AMNet (e.g., AMNet 100), AMNet (e.g., AMNet 100) calculates the final disparity d i Or knowledge is achieved from such training when generating the disparity map 124 .

[0085] During testing, only the output from the final AM module (e.g., 408) is used for disparity regression. Based on the disparity regression, the final disparity d i may be predicted (eg, a disparity map 124 may be generated).

[0086] Some embodiments of the present disclosure provide a system that takes disparity estimation as the main task and uses semantic information to assist in disparity estimation. Semantic information (such as semantic segmentation maps and semantic boundaries) defines the category and position of each object in an image. It helps the system to better understand the image and further assist in disparity estimation. For the disparity estimation network, the goal is to predict the disparity values ​​of all pixels. Given the fact that the disparity changes dramatically at the location where the foreground object appears, adding prior information describing the foreground object will help the system to perform accurate disparity estimation. In addition, better perception of foreground objects can lead to better disparity estimation. In some embodiments, in outdoor driving scenes (such as KITTI), foreground objects are defined as vehicles and people. In some embodiments, a foreground-background segmentation map can be used to improve disparity estimation. In some embodiments, only foreground pixels and background pixels are distinguished. The exact category of the foreground object is not concerned. In addition, in some embodiments, only the semantic information of the foreground object is considered, and the exact category of the foreground object is not considered.

[0087] There are two general approaches to utilize foreground-background segmentation information or foreground semantic information into a disparity estimation system (eg, AMNet 100).

[0088] The first approach is to extend the network into a multi-task network, in which the main task is disparity estimation and the sub-task is foreground segmentation or foreground semantic boundary prediction. Figure 5A A multi-task network 501 according to some embodiments of the present disclosure is shown. The multi-task network 501 is designed to have a shared base and different heads for two tasks. For example, in the multi-task network 501, the two tasks share the same root at the beginning of the network structure and have separate heads at the end of the network structure. The shared root may include input images (e.g., Img1 and Img2) 502 and 504, feature extractors 506 and 508, 1×1 convolution modules 510 and 512, ECV module 514, stacked AM module 518, upsampling module 520, and regression module 522. In the multi-task network 501, the shared root may be Figure 1 For example, feature extractors 506 and 508 may be D-ResNet 106 and 108, 1×1 convolution modules 510 and 512 may be AM modules 110 and 112, ECV module 514 may be ECV module 114, stacked AM module 518 may be stacked AM module 118, upsampling module 520 may be upscaling module 120, and regression module 522 may be Figure 1The shared root may be used to generate a disparity output 524 (e.g., a disparity estimate). A CNN module 526 (connected at the output of the 1×1 convolution module 510) followed by an upsampling module 528 and a prediction module 530 may be used to generate a semantic output 532 (e.g., a foreground segmentation or a foreground semantic boundary prediction).

[0089] The total loss function of the multi-task network 501 is a weighted sum of the losses of the two tasks. Due to the data imbalance in the sub-tasks (e.g., foreground segmentation or foreground semantic boundary prediction), the weight of the foreground pixels or foreground semantic boundaries in the total loss function increases. By optimizing the multi-task network 501 for both tasks, the shared root is trained to implicitly have a better perception of the foreground object, which leads to better disparity estimation.

[0090] The second approach is to directly provide additional foreground-background segmentation information as additional input besides the RGB image (RGB-S input) to guide the disparity estimation network. This requires accurate segmentation maps in both the training phase and the testing phase. Figure 5B Another multi-tasking network 503 according to some embodiments of the present disclosure for the second method is shown. The network 503 may be Figure 1 100 with additional foreground-background segmentation information as additional input. For example, feature extractors 506 and 508 may be D-ResNet 106 and 108, 1×1 convolution modules 510 and 512 may be AM modules 110 and 112, ECV module 514 may be ECV module 114, stacked AM module 518 may be stacked AM module 118, upsampling module 520 may be upscaling module 120, and regression module 522 may be Figure 1 The regression module 122 of AMNet 100 of FIG. 502 is a graph of the foreground segmentation map or the foreground semantic boundary map. In some embodiments, the foreground segmentation map or the foreground semantic boundary map can be used as another input feature (e.g., 534, 536) to the network 503 in addition to the RGB image (e.g., 502, 504). This forms the RGB-S input to the network 503. The additional input signal (e.g., 534, 536) sends prior knowledge to the network 503 for better image understanding. In this way, features for better image understanding come from the additional input (534, 536). The network 503 is not trained to better understand images.

[0091] Because the two technologies (such as Figure 5A and Figure 5B As discussed in , both improve the performance of the disparity estimation task, so the two techniques can be combined together in a single system.

[0092] Figure 6An FBA-AMNet 600 system is shown according to some embodiments of the present disclosure. Figure 1 AMNet 100 can be extended to FBA-AMNet 600. FBA-AMNet 600 is designed as a multi-task network with RGB-S input (e.g. Figure 5A and Figure 5B as discussed).

[0093] The FBA-AMNet 600 includes a first D-ResNet 606 configured to receive a first input image 602 and first foreground semantic information 632a, and a second D-ResNet 608 configured to receive a second input image 604 and second foreground semantic information 632b. The FBA-AMNet 600 also includes a first AM module 610 connected to the output of the first D-ResNet 606, a second AM module 612 connected to the output of the second D-ResNet 608, an ECV module 614, a stacked AM module 618, an upscaling module 620, and a regression module 622.

[0094] The first D-ResNet 606 and the second D-ResNet 608 may have shared weights. The first AM module 610 and the second AM module 612 may also have shared weights. Each of the first AM module 610 and the second AM module 612 is designed as a scene understanding module that captures deep global context information as well as local details. In some embodiments, the combination of the first D-ResNet 606 and the first AM module 610 can be used to extract features of the first input image 602, and the combination of the second D-ResNet 608 and the second AM module 612 can be used to extract features of the second input image 604.

[0095] The ECV module 614 is connected to the output of the first AM module 610 and the second AM module 612. The ECV module 614 is a combination of the disparity level depth correlation sub-volume, the disparity level feature distance sub-volume, and the disparity level feature cascade sub-volume. The ECV module 614 carries rich information about the matching cost under different similarity metrics. The stacked AM module 618 is a second level processing module connected at the output of the ECV module 614. The stacked AM module 618 processes the output from the ECV module 614 (e.g., the left feature map F at all disparity levels) l With the right feature map F r The matching cost information between d and d is used to aggregate the multi-scale context information for disparity regression. The disparity regression can be calculated in the regression module 622, and based on the disparity regression, the final disparity d can be predicted. i (For example, disparity map 624 may be generated).

[0096] The first CNN module 626a (connected at the output of the first AM module 610) following the first upsampling module 628a and the first prediction module 630a can be used to generate a first semantic output 632a (e.g., a foreground-background segmentation map or foreground semantic information), and the second CNN module 626b (connected at the output of the second AM module 612) following the second upsampling module 628b and the second prediction module 630b can be used to generate a second semantic output 632b (e.g., a foreground-background segmentation map or foreground semantic information). In some embodiments, the CNN modules 626a and 626b can be multi-layer neural networks. In some embodiments, the first upsampling module 628a and the second upsampling module 628b can be used to present the input features as a higher resolution.

[0097] The FBA-AMNet 600 is iteratively (or recursively) trained by providing semantic outputs 632a and 632b (e.g., foreground-background segmentation maps) at the current epoch via feedback loops 616a and 616b as additional segmentation inputs to the FBA-AMNet 600 at the next epoch.

[0098] For example, at epoch 0, the input foreground semantic information to the multi-task network is initialized to zero.

[0099] At epoch K, the input foreground semantic information (e.g., 632a, 632b) to the multi-task network for one image is the output from the multi-task network for the image at epoch K-1.

[0100] This process is repeated until convergence.

[0101] In some embodiments, during the inference phase (e.g., in the first prediction module 630a and the second prediction module 630b), the segmentation task (e.g., foreground-background segmentation) may be ignored, and a zero map is provided as an additional input representing the foreground-background segmentation map (e.g., 632a and 632b). Although the segmentation task (e.g., foreground-background segmentation) is ignored, the performance of the FBA-AMNet system 600 has been improved because the multi-task network (e.g., including the first CNN module 626a followed by the first upsampling module 628a and the first prediction module 630a or the second CNN module 626b followed by the second upsampling module 628b and the second prediction module 630b) estimates the foreground-background segmentation map (e.g., 632a and 632b) and implicitly learns the foreground object boundary.

[0102] In some embodiments, two inference iterations are run, where a zero map is provided in a first iteration and the foreground-background segmentation maps (eg, 632a and 632b) output from the first iteration are used as additional input to the second iteration.

[0103] All layers in the feature extractor (e.g., 606, 608, 610, and 612) are shared between two tasks (e.g., disparity estimation 624 and foreground-background segmentation prediction (e.g., foreground semantic information 632a and 632b)). In addition to the feature extractor, binary classification layers (626a, 626b), upsampling layers (628a, 628b), and softmax layers (630a, 630b) are added for foreground-background segmentation (e.g., 632a, 632b).

[0104] During training, FBA-AMNet 600 continuously refines and exploits its foreground-background segmentation predictions (e.g., 632a, 632b) to learn better perception of foreground objects. The loss of FBA-AMNet 600 is calculated as a weighted sum of two losses from two tasks:

[0105] L=L disp +λL seg , where λ is L seg The weight of L disp represents the loss in the disparity estimate (e.g., 624), and L seg represents the loss in foreground-background segmentation prediction (e.g., 632a, 632b). During testing, the segmentation task is ignored and the zero map is used as an additional input.

[0106] Due to the data imbalance in the secondary task (e.g., foreground-background segmentation prediction), the weight of foreground pixels or foreground semantic boundaries in the loss function increases. By optimizing the multi-task FBA-AMNet 600 for two tasks (e.g., foreground-background segmentation and disparity prediction), the shared roots (606, 608, 610, 612, 614, 618, 620, 622) are trained to implicitly have better perception of foreground objects, which leads to better disparity estimation.

[0107] Figure 7 A method for disparity estimation for an FBA-AMNet system according to some embodiments of the present disclosure is shown. Figure 6 The FBA-AMNet system 600 is implemented Figure 7 Method 700.

[0108] At 702 , input images 602 and 604 are received at D-ResNet 606 and D-ResNet 608 .

[0109] At 704 , a feature map is extracted from an input image (eg, 602 , 604 ) by a feature extractor module (eg, a combination of 606 and 610 , and a combination of 608 and 612 ).

[0110] At 706, foreground semantic information (e.g., 632a, 632b) is generated by connecting the semantic information module (e.g., a combination of 626a, 628a, and 630a and a combination of 626b, 628b, and 630b) at the output of the feature extractor module (specifically, the AM module 610, the AM module 612 of the feature extractor module). The foreground semantic information (e.g., 632a, 632b) is provided to one or more feature extractor modules (specifically, the D-ResNet 606 and the D-ResNet 608 of the one or more feature extractor modules) for estimated disparity in the next training epoch.

[0111] At 708, matching cost information between one or more feature maps is calculated by an ECV module (eg, 614) connected at the output of the feature extractor module (specifically, the AM module 610, 612 of the feature extractor module).

[0112] At 710 , matching cost information between one or more feature maps from the ECV module 614 is processed by the stacked AM module 618 connected at the output of the ECV module 614 to aggregate multi-scale context information for disparity regression.

[0113] At 712 , the disparity of the FBA-AMNet system 600 is estimated based on the aggregated multi-scale contextual information and foreground semantic information (eg, 632 a , 632 b ) by the regression module 622 connected at the output of the stacked AM module 618 .

[0114] The FBA-AMNet system 600 is recursively trained using one or more feature maps and one or more foreground semantic information (eg, 632a, 632b) until convergence.

[0115] The method 700 may be evaluated on three of the most popular disparity estimation benchmarks—KITTI Stereo 2015, KITTI Stereo 2012, and Scene Flow.

[0116] The KITTI benchmark provides images of size 376×1248 captured by a pair of stereo cameras in real-world driving scenes. KITTI Stereo 2015 includes 200 training stereo image pairs and 200 testing stereo image pairs. Sparse ground truth disparity maps can be provided with the training data. D1-all error can be used as the main evaluation metric to calculate the percentage of pixels whose estimated error is ≥ 3px and ≥ 5% of their ground truth disparity.

[0117] KITTI Stereo 2012 includes 194 training stereo image pairs and 195 testing stereo image pairs. Out-Noc error can be used as the main evaluation metric to calculate the percentage of pixels with an estimation error ≥ 3px for all unoccluded pixels.

[0118] The Scene Flow benchmark is a synthetic dataset of approximately 39,000 stereo image pairs of size 540×960 rendered from various synthetic sequences. Three subsets of approximately 35,000 stereo image pairs are available for training (Flyingthings3D train, Monkka, and Driving), and one subset of approximately 4,000 stereo image pairs is available for testing (Flyingthings3D test). Scene Flow provides complete ground truth disparity maps for all images. End-point-error (EPE) is used as an evaluation metric.

[0119] In some embodiments, first, AMNet-8 and AMNet-32 are trained from scratch on the scene flow training set. For both models, the dilation factors of the atrous convolutional layers in the AM module are set to [1, 2, 2, 4, 4, 8, 1, 1] and [1, 2, 2, 4, 4, 8, 8, 16, 16, 32, 1, 1], respectively. The maximum disparity D is set to 192. The parameter t in ECV is set to 0. The weight λ of the segmentation loss is set to 0.5. For a pair of input images, two blocks of size 256×512 at the same random position are cropped as input to the network. All pixels with a ground truth disparity greater than D are excluded from training. The models (e.g., 600) are end-to-end trained with a block size of 16 for 15 epochs using the Adam optimizer. The learning rate is initially set to 10 -3 , and decreases to 10 after 10 epochs. -4 FBA-AMNet is not trained on scene flow due to the fact that segmentation labels in scene flow are not consistent across scenes or objects.

[0120] In some embodiments, four models (e.g., AMNet-8, AMNet-32 (e.g., AMNet 100), FBA-AMNet-8, and FBA-AMNet-32 (e.g., FBA-AMNet 600)) on KITTI from pre-trained AMNet-8 and AMNet-32 (e.g., AMNet 100) models are then fine-tuned. To train the FBA-AMNet model, the first layer in the AMNet model is modified to have 4 channels for RGB-S input, and a binary classification layer, a bilinear upsampling layer, and a softmax layer for the foreground-background segmentation task may be added. The training is performed using the Adam optimizer with a batch size of 12 for 1000 epochs. Figure 6 and Figure 7 The model was trained using the iterative training method described in . The learning rate was initially set to 10 -3 , which drops to 10 after 600 epochs. -4 For the new layers, the learning rate can be increased by a factor of 10. Other settings are the same as the scene flow training process. The foreground-background segmentation map is initialized to zero at epoch 0.

[0121] All models are implemented using PyTorch on NVIDIA Titan-Xp GPU.

[0122]

[0123] Table 2: Performance comparison of the model of the example embodiment and the most published methods on the KITTI Stereo 2015 test set. D1-bg means evaluation on static background pixels. D1-fg means evaluation on dynamic foreground pixels. D1-all means evaluation on all pixels.

[0124]

[0125]

[0126] Table 3: Performance comparison on the KITTI Stereo 2012 test set. The error threshold is set to 3.

[0127] method GC-Net

[15] DispNetC

[20] PSMNet[2] AMNet-8 AMNet-32 EPE 2.51 1.68 1.09 0.81 0.74

[0128] Table 4: Performance comparison on the scene flow test set. All results are reported in EPE.

[0129] Results on the KITTI Stereo 2015 test set: Four models (AMNet-8, AMNet-32 (e.g., AMNet 100), FBA-AMNet-8, and FBA-AMNet-32 (e.g., FBA-AMNet 600)) are compared on the KITTI Stereo 2015 test set using all published methods on all evaluation settings. The results are shown in Table 2. All four models outperform previous methods on D1-all by a significant margin. Compared to the previous best result at 2.25%, the FBA-AMNet-32 (e.g., FBA-AMNet 600) model pushes D1-all (all pixels) to 1.93% with a relative gain of 14.2%.

[0130] Results on KITTI Stereo 2012 Test Set: Performance comparison on the KITTI Stereo 2012 test set is shown in Table 3. Consistent with KITTI Stereo 2015, the four models significantly outperform all other published methods on all evaluation settings. The FBA-AMNet-32 (e.g., FBA-AMNet 600) model reduces Out-Noc to 1.32% with a relative gain of 11.4%, compared to the previous best result reported at 1.49%. Only results for error threshold 3 are reported here. Results for other error thresholds are consistent with those for error threshold 3.

[0131] Results on the scene flow test set: The AMNet-8 model and the AMNet-32 (e.g., AMNet 100) model are compared on the scene flow test set using all published methods. Both models outperform the other methods by a large margin. The results reported in EPE are shown in Table 4. Compared to the previous best result at 1.09, the AMNet-32 (e.g., AMNet 100) model pushes E PE to 0.74 with a relative gain of 32.1%.

[0132] The following sections analyze the effectiveness of each component of the proposed architecture in detail. Since KITTI only allows a limited number of evaluations on the test set per month, most of the analysis is performed on the scene flow test set.

[0133] This section explores how modifications to the network backbone from the standard ResNet-50 to D-ResNet change performance and complexity. The following three models are compared: an AMNet-32 (e.g., AMNet100) model using ResNet-50 as the network backbone, an AMNet-32 (e.g., AMNet 100) model using a modified ResNet-50 as the network backbone by directly replacing standard convolutions with depthwise separable convolutions, and an example AMNet-32 (e.g., AMNet 100) model. The results on the scene flow test set and the number of parameters in each model are shown in Table 5. As a network backbone with a smaller number of parameters, D-ResNet outperforms the standard ResNet-50.

[0134]

[0135] Table 5: Performance and complexity comparison of three models using different network backbones. Results are reported on the scene flow test set.

[0136] An ablation study was performed for ECV with seven models modified from the AMNet-32 (e.g., AMNet 100) model using different combinations of three subsets (disparity-level feature distance subset, disparity-level depth correlation subset, and disparity-level feature cascade subset). The result comparison and feature size comparison on the scene flow test set are shown in Table 6. The results show that the disparity-level feature distance subset is more effective than the other two, and the combination of the three subsets (e.g., ECV module 614) leads to the best performance.

[0137]

[0138] Table 6: Performance comparison and feature size comparison of models using different cost volumes. “Dist.”, “Corr.”, and “FC” represent disparity-level feature distance, disparity-level depth correlation, and disparity-level feature cascade, respectively. All results are reported in EPE on the scene flow test set.

[0139]

[0140] Table 7: Performance comparison and running time comparison for each image. All results are reported on the scene flow test set. The size of the test image is 540×960.

[0141] In some embodiments, a deeper structure allows the AM module (e.g., 610, 612) to aggregate more multi-scale contextual information that leads to finer feature representations, while being more computationally expensive. The effects of different structures of the AM module on the performance and speed of the AMNet-32 (e.g., AMNet 100) model can be analyzed by setting the maximum dilation factor k of the AM module to 4, 8, 16, and 32. The performance comparison and speed comparison of the four models on the scene flow test set are shown in Table 7. The size of all test images is 540×960.

[0142] Compared to AMNet, FBA-AMNet (e.g., 600) is designed and trained to generate smoother and more accurate shapes for foreground objects, which leads to more refined disparity maps. Figure 8 Shown are the disparity estimation results of the AMNet model (e.g., 100) and the FBA-AMNet (600) model for two challenging foreground objects from the KITTI test images. Figure 8 The visualization in supports the fact that FBA-AMNet (e.g., 600) is able to generate finer details for foreground objects.

[0143] Fig. 9 An image from the KITTI Stereo 2015 test set and the coarse-to-fine foreground-background segmentation results generated by the FBA-AMNet-32 (e.g., FBA-AMNet 600) model at training epochs 10, 300, 600, and 1000 are shown. The visualization shows that during the training process, the multi-task network gradually learns a better perception of the foreground object. Because the optimization process of the multi-task network is biased towards the disparity estimation task, the segmentation task of generating a suitable segmentation map is expected.

[0144] Example embodiments of the present disclosure provide an end-to-end deep learning architecture with a design for each major component for disparity estimation. The model (e.g., 600) is capable of extracting deep features and discriminative features, using three different similarity metrics to calculate rich matching costs, and aggregating multi-scale contextual information for dense disparity estimation. How each component contributes to the final result is analyzed and visualized in detail. On the three most popular disparity estimation benchmarks, the example FBA-AMNet (e.g., 600) outperforms all other published methods.

[0145] It will be understood that although the terms "first", "second", and "third", etc. may be used herein to describe various elements, components, regions, layers, and / or portions, these elements, components, regions, layers, and / or portions should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer, or portion from another element, component, region, layer, or portion. Therefore, without departing from the spirit and scope of the inventive concept, the first element, component, region, layer, or portion discussed herein may also be referred to as a second element, component, region, layer, or portion.

[0146] For ease of description, spatially relative terms such as "under", "below", "below", "below", "above", "above", etc. may be used herein to describe the relationship of one element or feature to another element or feature as shown in the accompanying drawings. It will be understood that such spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation depicted in the accompanying drawings. For example, if the device in the accompanying drawings is turned over, the element described as "under" or "below" or "below" other elements or features will then be positioned as "above" the other elements or features. Therefore, the example terms "under" and "below" may include both above and below orientations. The device may be positioned otherwise (e.g., rotated 90 degrees or at other orientations), and the spatially relative descriptions used herein should be interpreted accordingly. In addition, it will be understood that when a layer is referred to as "between" two layers, it can be the only layer between the two layers, or one or more intermediate layers may also be present.

[0147] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the inventive concept. As used herein, the terms "substantially," "about," and similar terms are used as terms of approximation rather than as terms of degree, and are intended to account for the inherent deviations of measured or calculated values ​​that would be recognized by one of ordinary skill in the art.

[0148] As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. It will also be understood that when the terms "include" and / or "comprise" are used in this specification, it indicates that there are stated features, wholes, steps, operations, elements and / or components, but does not exclude the existence or increase of one or more other features, wholes, steps, operations, elements, components and / or their groups. As used herein, the term "and / or" includes any combination and all combinations of one or more of the associated listed items. When a statement such as "at least one of..." is after a column of elements, the statement modifies the entire column of elements rather than a single element in the column. In addition, the use of "may" when describing an embodiment of the inventive concept represents "one or more embodiments of the present disclosure". In addition, the term "exemplary" is intended to represent an example or description. As used herein, the terms "use" and "used" may be considered to be synonymous with the terms "utilize" and "utilize", respectively.

[0149] It will be understood that when an element or layer is referred to as being “on,” “connected to,” “bound to,” or “adjacent to” another element or layer, the element or layer may be directly on, directly connected to, bound to, or directly adjacent to the other element or layer, or one or more intervening elements or layers may be present. In contrast, when an element or layer is referred to as being “directly on,” “directly connected to,” “directly bound to,” or “immediately adjacent to” another element or layer, there are no intervening elements or layers.

[0150] Any numerical range listed here is intended to include all sub-ranges of the same numerical precision contained in the listed range. For example, the range of "1.0 to 10.0" is intended to include all sub-ranges between the listed minimum value 1.0 and the listed maximum value 10.0 (and including the minimum value 1.0 and the maximum value 10.0), that is, all sub-ranges with a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0 (such as, for example, 2.4 to 7.6). Any maximum numerical limit listed here is intended to include all lower numerical limits contained therein, and any minimum numerical limit listed in this specification is intended to include all higher numerical limits contained therein.

[0151] In some embodiments, one or more outputs of different embodiments of the methods and systems of the present disclosure may be sent to an electronic device that is coupled to a display device or has a display device for displaying one or more outputs of different embodiments of the methods and systems of the present disclosure or information about one or more outputs.

[0152] The electronic or electrical devices and / or any other related devices or components according to the embodiments of the present disclosure described herein may be implemented using any suitable hardware, firmware (e.g., application specific integrated circuits), software, or a combination of software, firmware, and hardware. For example, the various components of these devices may be formed on an integrated circuit (IC) chip or on separate IC chips. In addition, the various components of these devices may be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or formed on a substrate. In addition, the various components of these devices may be processes or threads running on one or more processors that execute computer program instructions in one or more computing devices and interact with other system components for performing the various functions described herein. The computer program instructions are stored in a memory, which may be implemented in a computing device using a standard memory device (such as, for example, a random access memory (RAM)). The computer program instructions may also be stored in other non-temporary computer-readable media (such as, for example, a CD-ROM, a flash drive, etc.). In addition, those skilled in the art should recognize that without departing from the spirit and scope of the exemplary embodiments of the present disclosure, the functions of various computing devices may be combined or integrated into a single computing device, or the functions of a specific computing device may be distributed on one or more other computing devices.

[0153] Although exemplary embodiments of a foreground-background aware hole multi-scale network for disparity estimation have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Therefore, it will be appreciated that a foreground-background aware hole multi-scale network for disparity estimation constructed according to the principles of the present disclosure may be implemented other than as specifically described herein. The inventive concepts are also defined in the claims and their equivalents.

Claims

1. A system for disparity estimation, the system comprising: one or more feature extractor modules configured to extract one or more feature maps from one or more input images, wherein the one or more input images include a first input image and a second input image, wherein the one or more feature maps extracted from the one or more input images include a first feature map extracted from the first input image and a second feature map extracted from the second input image; One or more semantic information modules, connected to one or more outputs of the one or more feature extractor modules, wherein the one or more semantic information modules are configured to: generate one or more foreground semantic information to be provided to the one or more feature extractor modules for disparity estimation in the next training period; an extended cost volume module, connected to the one or more outputs of the one or more feature extractor modules, the extended cost volume module is configured to calculate matching cost information between the first feature map and the second feature map; a stacked atrous multi-scale module connected at an output of the extended cost volume module and configured to process matching cost information between the first feature map and the second feature map from the extended cost volume module to aggregate multi-scale context information; and A regression module is connected at an output of the stacked atrous multi-scale module and is configured to estimate disparity based on the aggregated multi-scale context information from the stacked atrous multi-scale module and the one or more foreground semantic information.

2. The system according to claim 1, wherein: The stacked atrous multi-scale module includes multiple atrous multi-scale modules stacked together.

3. The system according to claim 1, wherein: The extended cost module includes: a disparity level feature distance sub-volume module configured to determine a pixel-by-pixel absolute difference between the first feature map and the second feature map; a disparity level depth correlation submodule configured to determine a correlation between the first feature map and the second feature map; and The disparity level feature cascade submodule is configured to d Cascade offset d The first feature map and the second feature map of .

4. The system of claim 3, wherein: The size of the disparity level feature distance sub-volume module is H×W×( D +1)×C, where H, W, and C represent the height, width, and feature size, D represents the maximum disparity that the system can predict; The size of the parallax level depth correlation sub-volume module is H×W×( D +1) × C; and The size of the disparity level feature cascade submodule is H×W×( D +1) × 2C.

5. The system according to claim 4, wherein: The size of the extended cost volume module is determined by cascading the disparity level feature distance sub-volume module, the disparity level depth correlation sub-volume module, and the disparity level feature cascade sub-volume module along the depth dimension, where the size of the extended cost volume module is H×W×( D +1) × 4C.

6. The system according to claim 3, wherein: The multiple dilated multi-scale modules are stacked together using shortcut connections within the stacked dilated multi-scale modules, wherein the dilated multi-scale modules of the multiple dilated multi-scale modules of the stacked dilated multi-scale modules are configured to process matching cost information between a first feature map and a second feature map from an extended cost volume module using k pairs of 3×3 dilated convolutional layers and two 1×1 convolutional layers, wherein k is an integer power of 2 and greater than 0.

7. The system according to claim 6, wherein: k pairs of 3×3 dilated convolutional layers have dilation factors [1, 2, 2, 4,4, …, k / 2, k / 2, k], wherein two 1×1 convolutional layers with dilation factor one are added at the end of the dilated multi-scale module of the multiple dilated multi-scale modules for feature refinement and feature size adjustment.

8. The system according to claim 1, wherein: The one or more feature extractor modules include: A first depthwise separable residual network module is configured to receive a first input image and first foreground semantic information; A second depthwise separable residual network module is configured to receive a second input image and second foreground semantic information; A first atrous multi-scale module connected to the output of the first depthwise separable residual network module; and The second atrous multi-scale module is connected to the output of the second depthwise separable residual network module.

9. The system according to claim 8, wherein: The first depth-wise separable residual network module and the second depth-wise separable residual network module have shared weights, the first dilated multi-scale module and the second dilated multi-scale module have shared weights, wherein each of the first dilated multi-scale module and the second dilated multi-scale module is configured as a scene understanding module for capturing deep global context information and local details, and wherein the extended cost volume module is connected to the output of the first dilated multi-scale module and the output of the second dilated multi-scale module.

10. The system according to claim 8, wherein: The one or more semantic information modules include: a first semantic information module connected at an output of the first dilated multi-scale module, wherein the first semantic information module is configured to generate first foreground semantic information, wherein the first foreground semantic information is provided to the first deep separable residual network module via a first feedback loop as an additional input to the system for a next training epoch of the system; and A second semantic information module is connected at the output of the second atrous multi-scale module, wherein the second semantic information module is configured to generate second foreground semantic information, wherein the second foreground semantic information is provided to the second deep separable residual network module via a second feedback loop as an additional input to the system for a next training epoch of the system.

11. The system according to claim 10, wherein: The first semantic information module includes: A first convolutional neural network module, connected to the output of the first atrous multi-scale module; A first upsampling module connected to the output of the first convolutional neural network module; and The first prediction module is connected to the output of the first upsampling module and is configured to generate first foreground semantic information.

12. The system according to claim 10, wherein: The second semantic information module includes: A second convolutional neural network module, connected to the output of the second atrous multi-scale module; A second upsampling module connected to the output of the second convolutional neural network module; and The second prediction module is connected to the output of the second upsampling module and is configured to generate second foreground semantic information.

13. The system of claim 1, wherein: The system is a multi-task module configured to perform two tasks, wherein the two tasks are disparity estimation and foreground semantic information generation, wherein the loss of the system is a weighted sum of two losses from the two tasks.

14. A method for disparity estimation of a system, the system comprising one or more feature extractor modules, one or more semantic information modules, an extended cost volume module, a stacked hole multi-scale module and a regression module, the method comprising: extracting, by the one or more feature extractor modules, one or more feature maps from one or more input images; generating one or more foreground semantic information by the one or more semantic information modules connected at one or more outputs of the one or more feature extractor modules, wherein the one or more foreground semantic information are provided to the one or more feature extractor modules; calculating matching cost information between the one or more feature maps by an extended cost volume module connected to the one or more outputs of the one or more feature extractor modules; Processing matching cost information between the one or more feature maps from the extended cost volume module through a stacked atrous multi-scale module connected at the output of the extended cost volume module to aggregate multi-scale context information for disparity regression; estimating disparity based on the aggregated multi-scale contextual information and the one or more foreground semantic information by a regression module connected at the output of the stacked atrous multi-scale module; and The system is recursively trained using the one or more feature maps and the one or more foreground semantic information until convergence.

15. The method according to claim 14, wherein: The one or more foreground semantic information of the current epoch is calculated by the one or more semantic information modules in the previous epoch, wherein the one or more input images include a first input image and a second input image, wherein the one or more feature maps extracted from the one or more input images include a first feature map extracted from the first input image and a second feature map extracted from the second input image, and wherein the method further comprises: Determine a pixel-by-pixel absolute difference between the first feature map and the second feature map by extending the disparity level feature distance sub-volume module of the cost volume module; Determining a correlation between the first feature map and the second feature map by extending the disparity level depth correlation sub-volume module of the cost volume module; and By extending the disparity level features of the cost volume module and cascading the sub-volume modules, at each disparity level d Cascade offset d The first feature map and the second feature map of .

16. The method of claim 15, wherein: The size of the disparity level feature distance sub-volume module is H×W×( D +1)×C, where H, W, and C represent the height, width, and feature size, D represents the maximum disparity that the system can predict; The size of the parallax level depth correlation sub-volume module is H×W×( D +1) × C; and The size of the disparity level feature cascade submodule is H×W×( D +1) × 2C.

17. The method according to claim 16, further comprising: The size of the extended cost volume module is determined by cascading the disparity level feature distance sub-volume module, the disparity level depth correlation sub-volume module and the disparity level feature cascade sub-volume module along the depth dimension, wherein the size of the extended cost volume module is H×W×( D +1) × 4C.

18. The method according to claim 16, further comprising: generating first foreground semantic information through a first semantic information module of the one or more semantic information modules; receiving, by a first deep separable residual network module of the one or more feature extractor modules, a first input image and first foreground semantic information, wherein the first foreground semantic information is provided to the first deep separable residual network module via a first feedback loop as an additional input for a next training epoch of the system; generating second foreground semantic information through a second semantic information module of the one or more semantic information modules; receiving, by a second deep separable residual network module of the one or more feature extractor modules, a second input image and second foreground semantic information, wherein the second foreground semantic information is provided to the second deep separable residual network module via a second feedback loop as an additional input for a next training epoch of the system; and Through the first atrous multi-scale module and the second atrous multi-scale module of the one or more feature extractor modules, deep global context information and local details are captured for scene understanding.

19. The method according to claim 15, wherein: The stacked dilated multi-scale module includes a plurality of dilated multi-scale modules stacked together using shortcut connections within the stacked dilated multi-scale module, wherein the method further includes: processing the matching cost information between the one or more feature maps from the extended cost volume module using k pairs of 3×3 dilated convolutional layers and two 1×1 convolutional layers through the dilated multi-scale module of the plurality of dilated multi-scale modules of the stacked dilated multi-scale module, wherein the k pairs of 3×3 dilated convolutional layers have dilation factors [1, 2, 2, 4, 4, …, k / 2, k / 2, k], wherein two 1×1 convolutional layers with dilation factor one are added at the end of the dilated multi-scale module for feature refinement and feature size adjustment, wherein k is an integer power of 2 and greater than 0.

Images