Two-stage depth estimation machine learning algorithm for equirectangular projection stereo matching and spherical warping layers

Abstract

A system and method having an end-to-end two-stage deep learning framework for depth estimation is disclosed that takes in a spherical color image and estimates a dense spherical depth map. The envisioned framework can include view synthesis (stage 1) and multi-view stereo matching (stage 2). The combination of the two-stage process can provide the benefits of geometric constraints from stereo matching to improve depth map quality without requiring additional input data. It is also envisioned that for the multi-view spherical stereo matching stage, a spherical warping layer can be used to integrate multiple spherical feature volumes into one cost volume with uniformly sampled inverse depth. The two-stage spherical depth estimation system and method can be used in various applications including virtual reality, autonomous driving, and robotics.

Description

Technical Field

[0001] This disclosure relates to a system and method having an end-to-end two-stage depth estimation machine learning framework for isorectangular projection stereo matching and a spherical twist layer. Background Technology

[0002] Three-dimensional (3D) scene understanding can be relevant to applications such as virtual reality (VR), augmented reality (AR), autonomous driving, or robotics. For example, rapid and reliable acquisition of omnidirectional 3D data is considered a useful building block for such applications, enabling user interaction with the digital environment. Summary of the Invention

[0003] A system and method for employing a two-stage depth estimation deep machine learning algorithm are disclosed. This algorithm includes a first stage that provides an equal rectangular projection (ERP) image to a coarse monocular depth estimation machine learning algorithm operable to estimate a coarse depth map. The first stage also includes a differentiable depth image-based rendering (DIBR) algorithm that receives the coarse depth map and generates one or more synthetic images.

[0004] The second stage may feed the ERP image and the one or more synthesized images to a multi-view stereo matching machine learning algorithm, which includes two cascaded stages for disparity prediction across territories ranging from coarse to fine. The multi-view stereo matching machine learning algorithm may also be operable to generate a final spherical depth map.

[0005] The second stage may include a spherical feature extraction machine learning algorithm that pairs the one or more synthetic images with the ERP image and generates one or more feature maps. The second stage may also include a cost volume construction algorithm that aggregates the spherical features extracted from the ERP image and the one or more synthetic images. The cost volume construction algorithm may also use a uniformly sampled depth assumption to generate the cost volume. It is envisioned that the cost volume construction algorithm can use one or more feature maps to generate the cost volume. It is also envisioned that a new interval can be used to uniformly sample the depth assumption at a specified level.

[0006] The second stage may further include a cost aggregation machine learning algorithm, operable to aggregate cost volumes using one or more 3D convolutional neural networks. This cost aggregation machine learning algorithm can be implemented using an hourglass encoding and decoding process. The second stage may also include a regression algorithm that regresses disparity values ​​pixel-by-pixel for each specified level. Imagine that a baseline ground truth depth map could be used to supervise both the coarse depth map and the final spherical depth map.

[0007] It is also envisioned that a coarse monocular depth estimation machine learning algorithm could be implemented as a lightweight machine learning network that utilizes coordinate convolution to perform 360-degree perception. The DIBR algorithm is also operable to transform a first set of pixels from the ERP image into a second set of pixels on the target image in a fully differentiable manner. Finally, the DIBR algorithm is operable to splat the first set of pixels onto the target image. The DIBR algorithm also includes a soft z-buffering algorithm to handle occlusion, and it can generate a final prediction that is a weighted average of the points splattered onto the individual pixels from the second set of pixels. Attached Figure Description

[0008] Figure 1 An exemplary computing system that can be used by the disclosed embodiments is described.

[0009] Figure 2 An exemplary embodiment of an end-to-end two-stage depth estimation deep learning framework is illustrated.

[0010] Figure 3 An exemplary embodiment of a multi-view omnidirectional stereo matching network is illustrated.

[0011] Figure 4 An exemplary machine learning convolutional neural network (CNN) is illustrated.

[0012] Figure 5 This is an example of a computer system being used to control an automated personal assistant.

[0013] Figure 6 The illustration shows a computing system for controlling a robot that is at least partially autonomous.

[0014] Figure 7 This is an example in which the computing system can be used to control and monitor the system. Detailed Implementation

[0015] Embodiments of this disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various forms and alternative forms. The figures are not necessarily to scale; some features may be enlarged or minimized to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those skilled in the art, the various features illustrated and described with reference to any figure may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described. The combinations of illustrated features provide representative embodiments of typical applications. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of this disclosure may be desired.

[0016] Machine learning algorithms (e.g., CNNs or DNNs) employed on spherical images are becoming increasingly widely used. For example, machine learning algorithms are emerging in applications related to deep learning on 360-degree images. Similarly, isometric projection images (which can be continuous contextual representations including slight to severe distortion) are a known representation of omnidirectional images.

[0017] To improve the effectiveness of convolutional kernels used in machine learning algorithms, omnidirectional depth can be used to introduce a series of rectangular filter banks with different kernel sizes to account for equirectangular distortion. It is also envisioned that distortion-aware spherical kernels derived from traditional convolutional kernels could be employed. Such kernels could be designed to operate in conjunction with CNNs as special layers, which is commonly used in the feature extraction stage of stereo matching networks. However, another approach is envisioned that could include spherical warp layers applied in cost volume construction.

[0018] It should be understood that machine learning algorithms can also be used to solve depth estimation by modeling the mapping between monocular images and depth maps. Multi-task machine learning algorithms can also be used to jointly predict semantic labels, normals, and depth simultaneously. For example, the known "pano-popup" algorithm can be used to simultaneously predict dense depth, surface normals, and planar boundaries from a single equirectangular image.

[0019] Unsupervised deep learning methods for monocular depth estimation can also employ self-constraints without direct depth supervision. A known framework based on view synthesis and a loss computed when warped from a nearby view to the target can be used; however, such methods may output a reconstruction loss in image construction that appears visually good but has low-quality depth. To overcome these known problems, geometric constraints can be employed. For example, left-right consistency and epipolar constraints can be used. Furthermore, for spherical images, cubemaps from 360-degree video can be used as supervisory signals along with photometric and pose consistency. Unsupervised methods are also understood to reduce the need for ground-valued baseline data; however, these existing techniques may not produce satisfactory high-quality depth maps.

[0020] Furthermore, omnidirectional depthwise and mapping convolutions can leverage specialized kernels designed for spherical images to improve geometric learning accuracy. However, directly learning depth using machine learning algorithms may require a large amount of training data from 2D image space. t Deep learning implicit mapping. Even though 360-degree cameras are accessible at a reasonable cost, building a comprehensive 360-degree dataset with benchmark ground truth depth and labels is not always feasible or cost-effective.

[0021] While previous stereo matching methods may perform well on perspective stereo images, they may not always be directly applicable to 360-degree stereo images due to the nature of spherical geometry. For example, a known algorithm called "360SD-Net" utilizes CNN-based techniques on 360-degree stereo images. A drawback of this known algorithm is that it employs a learnable shift layer for costly volumetric assumption plane sampling, which often requires additional training resources. It is envisioned that the spherical warp layer disclosed in this application could be a closed-form machine learning solution that does not require trainable parameters. The disclosed spherical warp layer could also be differentiable, allowing it to be used in end-to-end stereo matching networks.

[0022] Furthermore, existing known systems can employ learnable shift layers for costly volumetric assumption planar sampling. In contrast, the disclosed two-stage system and method can provide a self-refinement mechanism to achieve better and more accurate disparity results for 360-degree stereo images. Moreover, RGB-D spherical data and spherical stereo pairs can be expensive resources to acquire, but the disclosed two-stage machine learning algorithm overcomes this overhead due to the fact that it can be trained on smaller datasets.

[0023] Three-dimensional (3D) scene understanding may also be relevant to applications such as virtual reality (VR), augmented reality (AR), autonomous driving, or robotics. For example, rapid and reliable acquisition of omnidirectional 3D data is considered a useful building block for such applications, enabling user interaction with the digital environment.

[0024] Therefore, it is envisioned that devices such as omnidirectional LiDAR could be used for applications like autonomous vehicles or indoor 3D scanning to obtain high-quality omnidirectional 3D information. However, LiDAR can be expensive or may produce sparse 3D scans. Compared to LiDAR, more conventional cameras are likely cheaper and are more commonly used to capture the visual appearance of a scene. Therefore, it is envisioned that significant cost savings could be achieved by generating high-quality omnidirectional 3D scans using images from conventional cameras.

[0025] To achieve technical efficiency and cost savings using conventional camera images, an end-to-end two-stage deep learning framework for depth estimation (i.e., the PanoDepth framework) can be used. The first stage can be view synthesis, and the second stage, stereo matching, is used for monocular omnidirectional depth estimation. The PanoDepth framework is envisioned to take an isometric projection (ERP) image as input in the first stage to generate a synthesized view. The first stage can then pass the original image and one or more synthesized views to the subsequent stereo matching stage to generate a more refined depth map. In the second stage of stereo matching, an omnidirectional multi-view stereo matching network can be used to process omnidirectional stereo pairs.

[0026] Figure 1 An exemplary system 100 that can be used to implement a panoramic depth frame is depicted. System 100 may include at least one computing device 102. Computing system 102 may include at least one processor 104 operatively connected to memory unit 108. Processor 104 may be one or more integrated circuits that implement the functions of a central processing unit (CPU) 106. It should be understood that CPU 106 may also be one or more integrated circuits that implement the functions of a general-purpose processing unit or a special-purpose processing unit (e.g., a graphics processing unit, ASIC, FPGA).

[0027] CPU 106 may be a commercially available processing unit that implements an instruction set such as x86, ARM, Power, or MIPS instruction set families. During operation, CPU 106 may execute stored program instructions retrieved from memory unit 108. The stored program instructions may include software that controls the operation of CPU 106 to perform the operations described herein. In some examples, processor 104 may be a system-on-a-chip (SoC) that integrates the functionality of CPU 106, memory unit 108, network interface, and input / output interface into a single integrated device. Computing system 102 may implement an operating system for managing various aspects of operation.

[0028] Memory cell 108 may include volatile and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 102 is deactivated or powered off. Volatile memory may include static and dynamic random access memory (RAM) for storing program instructions and data. For example, memory cell 108 may store machine learning model 110 or algorithm, training dataset 112 of machine learning model 110, and / or raw source data 115.

[0029] The computing system 102 may include a network interface device 122 configured to provide communication with external systems and devices. For example, the network interface device 122 may include wired and / or wireless Ethernet interfaces as defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface device 122 may include a cellular communication interface for communicating with cellular networks (e.g., 3G, 4G, 5G). The network interface device 122 may be further configured to provide a communication interface to an external network 124 or the cloud.

[0030] External network 124 may be referred to as the World Wide Web or the Internet. External network 124 can establish standard communication protocols between computing devices. External network 124 can allow information and data to be easily exchanged between computing devices and the network. One or more servers 130 can communicate with external network 124.

[0031] The computing system 102 may include an input / output (I / O) interface 120, which may be configured to provide digital and / or analog inputs and outputs. The I / O interface 120 may include an additional serial interface (e.g., a Universal Serial Bus (USB) interface) for communicating with external devices.

[0032] The computing system 102 may include a human-machine interface (HMI) device 118, which may include any device that enables the system 100 to receive control input. Examples of input devices may include HMI inputs such as a keyboard, mouse, touchscreen, voice input device, and other similar devices. The computing system 102 may include a display device 132. The computing system 102 may include hardware and software for outputting graphical and textual information to the display device 132. The display device 132 may include an electronic display screen, projector, printer, or other suitable device for displaying information to a user or operator. The computing system 102 may be further configured to allow interaction with remote HMIs and remote display devices via a network interface device 122.

[0033] System 100 can be implemented using one or more computing systems. While this example depicts a single computing system 102 implementing all the described features, the aim is to allow various features and functions to be decoupled and implemented by multiple computing units that communicate with each other. The chosen system architecture may depend on a variety of factors.

[0034] System 100 can implement a machine learning algorithm 110 configured to analyze raw source data 115. Raw source data 115 may include raw or unprocessed sensor data, which may represent the input dataset for the machine learning system. Raw source data 115 may include video, video clips, images, and raw or partially processed sensor data (e.g., image data received from a camera 114, which may include a digital camera or LiDAR). In some examples, machine learning algorithm 110 may be a neural network algorithm designed to perform a predetermined function. For example, a neural network algorithm may be configured in automotive applications to identify objects (e.g., pedestrians) from images provided by a digital camera and / or from depth maps from a LiDAR sensor.

[0035] System 100 may store a training dataset 112 for machine learning algorithm 110. Training dataset 112 may represent a previously constructed dataset used to train machine learning algorithm 110. Machine learning algorithm 110 can use training dataset 112 to learn weighting factors associated with the neural network algorithm. Training dataset 112 may include source datasets having corresponding consequences or results that machine learning algorithm 110 attempts to replicate via the learning process. In one example, training dataset 112 may include source images and depth maps from various scenes that can identify objects (e.g., pedestrians).

[0036] Machine learning algorithm 110 can operate in learning mode using training dataset 112 as input. Machine learning algorithm 110 can be executed in multiple iterations using data from training dataset 112. With each iteration, machine learning algorithm 110 can update its internal weighting factors based on the obtained results. For example, machine learning algorithm 110 can compare its output with results included in training dataset 112. Since training dataset 112 includes expected results, machine learning algorithm 110 can determine when performance is acceptable. After machine learning algorithm 110 reaches a predetermined performance level (e.g., 100% consistency with results associated with training dataset 112), machine learning algorithm 110 can be executed using data not in training dataset 112. The trained machine learning algorithm 110 can be applied to new datasets to generate annotated data.

[0037] Machine learning algorithm 110 can also be configured to identify features in raw source data 115. Raw source data 115 may include multiple instances or input datasets for which it expects annotation results. For example, machine learning algorithm 110 may be configured to identify the presence of pedestrians in an image and annotate the occurrence event. Machine learning algorithm 110 may be programmed to process raw source data 115 to identify the presence of features. Machine learning algorithm 110 may be configured to identify features in raw source data 115 as predetermined features. Raw source data 115 may be derived from various sources. For example, raw source data 115 may be actual input data collected by a machine learning system. Raw source data 115 may be machine-generated for testing the system. As an example, raw source data 115 may include raw digital images from a camera.

[0038] In this example, machine learning algorithm 110 can process the raw source data 115 and generate output. Machine learning algorithm 110 can generate confidence levels or factors for each generated output. For example, a confidence value exceeding a predetermined high confidence threshold can indicate that machine learning algorithm 110 is confident that the identified feature corresponds to the specific feature. A confidence value below a low confidence threshold can indicate that machine learning algorithm 110 has some uncertainty regarding the existence of the specific feature.

[0039] Figure 2 Block diagram 200 illustrates an embodiment of a two-stage framework architecture (i.e., panoramic depth) that can receive a single ERP image as input and generate one or more high-quality omnidirectional depth maps. As shown, block diagram 200 may include a coarse depth estimation network 202 (i.e., the first stage) and a multi-view stereo matching network 204 (i.e., the second stage).

[0040] In box 206, network 202 can receive a single ERP image from camera 114. For example, the ERP image could be a single 360-degree image of a given scene. The ERP image can then be fed to coarse depth estimation network 208, which is operable to estimate an initial depth map, which is then provided to a differentiable depth image-based rendering (DIBR) module 210. Module 210 can then use the provided depth map to synthesize a novel view with a predefined baseline.

[0041] Imagine that in module 208, lightweight networks can be employed to generate synthetically produced, high-quality novel views, medium-quality depth maps, or even coarse multi-planar depth maps. For example, module 208 could employ a known lightweight network called "CoordNet" for coarse depth estimation. CoordNet can be used to implement 360-degree perception of ERP images using coordinate convolutions. However, other known networks can be employed for coarse depth estimation. The estimated coarse depth map and ERP image can then be provided to DIBR module 210, which renders multiple synthetic views of a predefined baseline. Imagine that a vertical baseline can be selected and used on top of a horizontal baseline.

[0042] By employing CoordNet, a single 360-degree view image can be passed through module 208 to generate a coarse depth estimate. The generated coarse depth map can support rendering the input 360-degree image using DIBR module 210, which transforms pixels from the source image to pixels on the target image in a fully differentiable manner. It is envisioned that the transformation operation performed by the DIBR module can be adopted using hierarchical 3D scene inference. The pixels from the source image can then be distributed across an empty target image. Occlusion can then be handled using a soft z-buffer, and the final prediction can be a weighted average of the points distributed to the same pixel.

[0043] Then, a multi-view stereo matching network 204 (i.e., the second stage) can be used to generate accurate and robust omnidirectional depth estimates. First, one or more synthetic images 212a-212b provided by the DIPR module 210, along with the input ERP image 206, can be passed to the stereo matching network 214 to generate the final depth map 216. It is envisioned that the stereo matching network may include two cascaded stages for disparity prediction in coarse to fine manor territories.

[0044] Supervision from the ground truth depth image 218 can also be applied to the depth generated from the coarse monocular depth estimation network 208 and the disparity generated from the stereo matching network 214. For example, the ground truth depth image 218 can also be used to supervise and train the coarse depth prediction 220 generated by the coarse depth estimation network 208 and the final depth prediction 216 generated by the stereo matching network 204 in an end-to-end manner.

[0045] In summary, block diagram 200 illustrates a two-stage network comprising: (1) a coarse depth estimation network 208 followed by a differentiable DIBR module 210 for novel view synthesis, and (2) a multi-view stereo matching network 204 with a differentiable spherical twist layer and a cascaded mechanism for efficient and high-quality depth estimation. An ERP image 206 can initially be fed into the coarse depth estimation network 208 to estimate an initial depth map for the DIBR module 210 to synthesize a novel view with a predefined baseline. The original ERP image 206 and the synthesized images 212a-212b can then be fed into the multi-view stereo matching network 204 to generate a final depth map 216. Both networks can be trained end-to-end, and both are supervised using a ground truth depth 218.

[0046] Figure 3 An embodiment of the multi-view stereo matching network 204 (i.e., the second stage) is illustrated. As shown, network 204 may include one or more input stages 308a-308b. However, it is contemplated that only one input stage (e.g., 308a) may be used. Each of the input stages 308a-308b may further include spherical feature extraction networks 310a-310b, spherical twist layers (SWL) 312a-312b, cost volume building modules 314a-314b, cost aggregation modules 316a-316b, and depth prediction modules 318a-318b.

[0047] As shown in the figure, one or more generated synthetic views (i.e., 212a-212b) can be paired with the input ERP image 206 and passed through a weight-sharing feature extraction network 304. It is envisioned that network 304 can be constructed using one or more known convolutional neural networks with multiple layers stacked together.

[0048] For example, Figure 4 An exemplary CNN 400 is illustrated, which can be implemented as part of network 204 (or as part of another network employed within network 202 or network 204). As shown, CNN 400 may include one or more convolutional layers 440-440; one or more pooling layers 450-470; one or more fully connected layers 460; and a softmax layer 470. It is envisioned that, depending on the given application, CNN 400 may alternatively be implemented using known DNNs or decision trees.

[0049] CNN 400 can receive data 410 (e.g., input 204 and / or synthesized views 212a-212b). It is envisioned that data 410 can be slightly processed before being fed to CNN 400. Convolutional layers 440-440 can be designed to extract features from data 410. For example, convolutional layers 440-440 can perform filtering operations (e.g., kernels) before passing the results to the next layer of CNN 400. Filtering operations can include image labeling, edge detection, and image sharpening when the received data 410 is an image.

[0050] CNN 400 may also include one or more pooling layers 450-470 that receive convolutional data from corresponding convolutional layers 440-440. Pooling layers 450-470 may include one or more pooling layer units that apply a pooling function to the outputs of one or more convolutional layers computed using the pooling function at different frequency bands. For example, pooling layer 450 may apply a pooling function to the kernel output received from convolutional layer 440. The pooling function implemented by pooling layers 450-470 may be an averaging or maximizing function, or any other function that aggregates multiple values ​​into a single value.

[0051] Next, one or more fully connected layers 480 may attempt to learn a non-linear combination of high-level features from the output data received from convolutional layers 440-440 and pooling layers 450-470. Finally, CNN 400 may include a softmax layer 490, which uses a softmax function to combine the outputs of the fully connected layers 480. CNN 400 may employ batch normalization layers, max pooling layers, and dropout layers. It is envisioned that CNN 400 may employ spatial pyramid pooling layers to extract multi-scale contextual information. It is also envisioned that CNN 400 may include one or more pooling layers of different sizes.

[0052] Back Figure 3 Multi-scale contextual information can then be input into the spherical feature networks 310a-310b. Imagine that for each input in a stereo pair with a resolution of 3×Z×k, the feature extraction networks 310a-310b can output information with… Feature maps of resolution. It is also envisioned that for networks 310a-310b, each cascade stage (the...) l (one) of them l >1, the output feature map can have The resolution is 32. However, it is envisioned that the feature map can have a resolution greater than or less than 32.

[0053] Similarly, the VR and AR market has experienced significant growth in recent years. As a result, an increasing number of 360-degree cameras have been developed and used as content generation sources for VR and AR applications. It is envisioned that stereo matching techniques could be used to generate 3D information by pairing two adjacent 360-degree images. However, previous stereo matching methods (e.g., conventional and deep learning machine algorithms) may not be suitable for ERP stereo pairs because equal rectangular projection (ERP) introduces distortion in the images (e.g., image 206).

[0054] It is further hypothesized that for perspective stereo images, disparity can also be proportional to inverse depth. Discrete disparity assumptions suggest that planes can be uniformly sampled and later merged based on estimated probabilities. However, for spherical stereos, disparity may be related to both inverse depth and latitude values ​​in the image. Due to the irregular distortion introduced by the spherical geometry, uniform assumption plane sampling is not ideal for pairs of equirectangular stereos. As discussed above, known algorithms attempt to employ trainable shift filters to select the optimal assumption step, but these methods require significant computational overhead and converge at an unacceptably slow rate.

[0055] Therefore, network 204 may include one or more spherical twist layers (SWLs) 312a-312b that operatively transform uniformly sampled disparity or inverse depth into spherical coordinates during spherical cost volume construction. It is envisioned that SWLs 312a-312b may be differentiable and may be used within network 204 (e.g., as a plug-in) for an end-to-end depth stereo matching network. SWLs 312a-312b may provide a stereo matching method for perspective-projected images suitable for ERP stereo pairs. It is envisioned that SWLs 312a-312b may deterministically transform uniformly sampled inverse depth or disparity into spherical displacement to reduce processing requirements or memory usage. Therefore, SWLs 312a-312b may be used instead of learnable layers to construct the cost volume.

[0056] For example, SWL 312a-312b can be used as a portion of the inverse depth that is uniformly sampled to cover the entire depth range, as represented by Equation 1 below:

[0057] Equation (1)

[0058] in D This assumes the total number of planes. d j It is the first j A depth plane, d min and d max These are the minimum and maximum values ​​of depth, and vIt is a planar interval. Imagine SWL 312a-312b could allow for depth assumptions. d j Transform into a spherical region C j The displacement is used to map pixels from the composite view to the original view. C j Then we can define it using the following equation 2:

[0059] Equation (2)

[0060] in Refers to the dimension values ​​across the image. b Indicates the baseline, and H f It is the height of the feature map. Unlike known learnable shift filters, the SWL 312a-312b is designed as a closed-form solution that does not require additional training.

[0061] Imagine that, utilizing the displacement φ from SWL 312a-312b, spherical features extracted from the φ view can be aggregated to construct a cost volume with a uniform sampling depth assumption. Known fusion implementations, including variance-based cost volume formation methods or cascaded designs, can also be applied to improve accuracy. Imagine that SWL 312a-312b could be designed to work seamlessly with such known cascaded designs. Furthermore, at each level... l The depth assumption can be uniformly sampled using a new interval based on the predictions calculated in level l+1. The corresponding displacements can then be calculated using the same spherical coordinate mapping process.

[0062] Network 204 may further include cost volume construction modules 314a-314b, which can construct the cost volume using extracted feature maps. In implementation, feature maps can be provided from spherical feature extraction modules 310a-310b or from spherical twisting layers 312a-312b.

[0063] Following the construction of the cost volume by modules 316a-316b, cost aggregation modules 316a-316b can be used to aggregate spatial context information at different levels using an hourglass-shaped encoding and decoding process. Modules 316a-316b can be designed using one or more multi-scale 3D CNN machine learning algorithms. It is also envisioned that modules 316a-316b can help regularize noise in blurred regions caused by occlusion or textureless surfaces to help improve the final prediction quality. Modules 316a-316b can then be applied to each stage... l The disparity value is regressed pixel by pixel, as shown in Equation 3 below:

[0064] Equation (3)

[0065] Where 𝑘 is the weighted sum of each plane level l by its normalized probability, as shown in Equation 4 below:

[0066] Equation (4)

[0067] in This represents the softmax function, and 𝑝 𝑗 Indicates the probability of the nth plane value. 𝑗,l It is the interval of level l.

[0068] Imagine that network 200 can be trained in an end-to-end manner, where supervision can be applied to two sub-processes (i.e., stages 202 and 204). The final loss function of the entire network can be defined as... loss = 𝐿 粗略 + 𝐿 立体 ,in and These are the weights of the coarse depth estimation loss and the stereo matching loss, respectively. A combination of smoothed L1 loss and a smoothing term is also envisioned for depth estimation of non-empty pixels labeled 𝑝, where the depth is denoted in Equation 5 below. D :

[0069] L 粗略 Equation (5)

[0070] Where 𝑀(𝑝) is the binary mask used to mask the missing regions, and 𝛼 and 𝛽 are the weights of the L1 loss and the smoothing term. It is also envisioned that network 200 can compute the berHu loss over all outputs from each level, and then compute the weighted sum. The same binary mask 𝑀 can be applied to the entire network 200, and the stereo matching loss can be defined using Equation 6 below:

[0071] L 立体 = Equation (6)

[0072] in It is the loss weight of the l-th level.

[0073] Figure 5-7 The diagram illustrates various applications that can be used to implement two-stage networks (i.e., networks 202 and 204). For example, Figure 5An embodiment is illustrated in which computing system 540 can be used to control at least partially autonomous robots, such as at least partially autonomous vehicle 500. Computing system 540 can be similar to... Figure 1 The system 100 described herein. Sensor 530 may include one or more video / camera sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR sensors and / or one or more positioning sensors (such as GPS). Some or all of these sensors are preferred, but not necessarily integrated into vehicle 500.

[0074] Alternatively, sensor 530 may include an information system for determining the state of the actuator system. Sensor 530 may collect sensor data or other information to be used by computing system 540. An example of such an information system is a weather information system for determining the current or future state of weather in an environment. For example, using an input signal x, a classifier may, for example, detect objects near a robot that is at least partially autonomous. The output signal y may include information characterizing that the object is near a robot that is at least partially autonomous. A control command A may then be determined based on this information, for example, to avoid a collision with the detected object.

[0075] The actuator 510, which can be integrated into the vehicle 500, can be provided by the vehicle 500's brakes, propulsion system, engine, powertrain, or steering. An actuator control command can be determined to control the actuator (or multiple actuators) 510 so that the vehicle 400 avoids collisions with the detected objects. The detected objects can also be classified according to what a classifier considers them most likely to be (e.g., pedestrians or trees), and the actuator control command A can be determined based on this classification.

[0076] Figure 6 An embodiment is illustrated, in which a computer system 640 controls an automated personal assistant 650. Sensor 630 may be an optical sensor, such as a video image of a gesture from user 649. Alternatively, sensor 630 may also be an audio sensor, such as a sensor for receiving voice commands from user 649.

[0077] The control system 640 then determines an actuator control command A for controlling the automated personal assistant 650. The actuator control command A is determined based on a sensor signal S from sensor 630. Sensor signal S is transmitted to the control system 640. For example, a classifier may be configured to implement a gesture recognition algorithm to identify gestures made by user 649. The control system 640 can then determine the actuator control command A to be transmitted to the automated personal assistant 650. It then transmits the actuator control command A to the automated personal assistant 650.

[0078] For example, actuator control command A can be determined based on the identified user gestures recognized by a classifier. It can then include information that causes the automated personal assistant 650 to retrieve information from a database and output that retrieved information in a format suitable for the user 649 to receive.

[0079] In another embodiment, instead of the automated personal assistant 650, the control system 640 may control a household appliance (not shown) controlled according to an identified user gesture. The household appliance may be a washing machine, stove, oven, microwave oven, or dishwasher.

[0080] Figure 6 An embodiment of a computing system control access control system 600 is illustrated. The access control system can be designed to physically control access. For example, it may include a door 601. A sensor 630 is configured to detect a scene relevant to determining whether access is authorized. The sensor 630 may collect image or video data related to the scene. For example, it may be an optical sensor used to provide image or video data for detecting a person's face. A classifier can be configured to interpret the image or video data, for example, by matching the identity with known persons stored in a database, thereby determining the identity of the person. Then, depending on the classifier's interpretation, for example, based on the determined identity, an actuator control signal A can be determined. The actuator 610 may be a lock that authorizes or disauthorizes access based on the actuator control signal A. Non-physical, logical access control is also possible.

[0081] Figure 7 An embodiment of the computing system 740 controlling the monitoring system 700 is shown. This embodiment is largely consistent with... Figure 5 The embodiments shown are identical. Therefore, only the differences will be described in detail. Sensor 730 is configured to detect the supervised scene. Sensor 730 may collect image or video data related to the scene. The computing system does not necessarily control actuator 710, but rather controls display 710a. For example, a machine learning system may determine the classification of the scene, such as whether the scene detected by optical sensor 730 is suspicious. Actuator control signal A transmitted to display 710a may then be configured, for example, to cause display 710a to adjust the displayed content based on the determined classification, for example, to highlight objects deemed suspicious by the machine learning system.

[0082] The processes, methods, or algorithms disclosed herein may be deliverable to / implemented by a processing device, controller, or computer, which may include any existing programmable electronic control unit or dedicated electronic control unit. Similarly, the processes, methods, or algorithms may be stored in various forms as data and instructions executable by a controller or computer, including but not limited to information permanently stored on non-writable storage media such as ROM devices and information changeably stored on writable storage media such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media. The processes, methods, or algorithms may also be implemented in a software executable object. Alternatively, suitable hardware components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or combinations of hardware, software, and firmware components, may be used to embody the processes, methods, or algorithms wholly or partially.

[0083] While exemplary embodiments have been described above, they are not intended to describe all possible forms covered by the claims. The terms used in this specification are descriptive and not limiting, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously described, features of various embodiments may be combined to form further embodiments of the invention that may not be explicitly described or illustrated. While various embodiments may have been described as providing advantages over other embodiments or prior art implementations in one or more desired features, or being preferred over other embodiments or prior art implementations, those skilled in the art will recognize that one or more features or characteristics may be compromised depending on the specific application and implementation to achieve desired overall system properties. These properties may include, but are not limited to, cost, strength, durability, lifecycle cost, merchantability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Accordingly, while any embodiment may be described as less desirable than other embodiments or prior art implementations in one or more features, these embodiments are not outside the scope of this disclosure and may be desirable for a particular application.

Claims

1. A method for employing a spherical twist layer machine learning algorithm within a spherical stereo matching network, comprising: Receive one or more rectangular projection ERP images from the sensor; Feature maps are received from the spherical feature extraction machine learning algorithm of the spherical feature extraction module of the spherical stereo matching network; and One or more spherical distortion layers of the spherical stereo matching network convert the disparity of the inverse depth assumption or uniform sampling into a displacement in the spherical coordinate domain, wherein the displacement is generated using a cosine angle of one or more latitude values ​​across the one or more ERP images, one or more baseline values, and the height of the feature map.

2. The method of claim 1, wherein the inverse depth assumption covers the entire depth range.

3. The method according to claim 1, further comprising: Map one or more pixels from one or more composite images to the ERP image.

4. The method of claim 3, wherein the spherical stereo matching network comprises two cascaded stages for parallax prediction in a coarse-to-fine manner, wherein the multi-view stereo matching machine learning algorithm is operable to generate a final spherical depth map.

5. The method of claim 4, wherein the spherical stereo matching network further comprises a spherical feature extraction algorithm operable to pair the one or more synthetic images with the ERP image.

6. The method of claim 5, wherein the spherical stereo matching network generates one or more feature maps.

7. The method of claim 6, wherein the one or more feature maps have a resolution of 32 times one-quarter of the width of the ERP image and one-quarter of the second height of the ERP image.

8. The method of claim 6, wherein the spherical stereo matching network further comprises a cost volume construction algorithm that aggregates spherical features extracted from the ERP image and the one or more synthetic images.

9. The method according to claim 8, wherein, The cost volume construction algorithm uses the depth assumption of uniform sampling to generate the cost volume.

10. The method according to claim 9, wherein, The cost volume is generated by the cost volume construction algorithm using one or more feature maps.

11. The method according to claim 9, wherein, Use the new interval to uniformly sample the depth hypothesis at the specified level.

12. The method of claim 10, wherein the spherical stereo matching network further comprises a cost aggregation machine learning algorithm operable to aggregate cost volumes using one or more 3D convolutional neural networks.

13. The method according to claim 12, wherein, The cost aggregation machine learning algorithm is implemented using an hourglass encoding and decoding process.

14. The method of claim 12, wherein the spherical stereo matching network further comprises a regression algorithm that regresses disparity values ​​pixel-by-pixel for each specified level.

15. The method according to claim 1, comprising: Configure the physical system to operate using displacements in the spherical coordinate domain.

16. The method of claim 15, further comprising: The inverse depth assumption is converted into displacement in the spherical coordinate domain, wherein the inverse depth assumption covers the entire depth range.

17. The method of claim 15, further comprising: Map one or more pixels from one or more composite images to the ERP image.

18. A system for employing a spherical twist layer machine learning algorithm within a spherical stereo matching network, comprising: Sensor, operable to capture one or more rectangular projected ERP images; Controller, operable for: Feature maps are received from the spherical feature extraction machine learning algorithm of the spherical feature extraction module of the spherical stereo matching network; and One or more spherical distortion layers of the spherical stereo matching network convert the disparity of the inverse depth assumption or uniform sampling into displacement in the spherical coordinate domain, wherein the displacement is generated using the cosine angle of one or more latitude values ​​across the one or more ERP images, one or more baseline values, and the height of the feature map.

19. The system of claim 18, wherein the inverse depth assumption covers the entire depth range.

20. The system of claim 18, further comprising: Map one or more pixels from one or more composite images to the ERP image.

Images