Direct depth prediction

Abstract

An exemplary system and technique for training a machine learning model is described. The system includes a memory configured to store image data captured by multiple cameras, and one or more processors communicatively coupled to the memory. One or more processors are configured to run a machine learning model on the image data, the machine learning model comprising multiple layers. One or more processors are configured to apply a nonlinear mapping function to the output of one of the layers in order to generate depth data. One or more processors are configured to train a machine learning model on the depth data in order to generate a trained machine learning model.

Description

[Technical Field]

[0001]

[0001] This application claims priority to U.S. Patent Application No. 18 / 467,804, filed on 15 September 2023, which is incorporated herein by reference in its entirety.

[0002]

[0002] This disclosure relates to depth prediction for machine learning models. [Background technology]

[0003]

[0003] Accurate depth estimation can be important for a variety of use cases. For example, accurate depth estimation (e.g., distance to an object) can be important for obstacle avoidance. Obstacle avoidance can be associated with autonomous driving, assistance and / or warehouse robots, augmented and / or virtual scene construction, etc. Accurate depth estimation can also be important for 3D construction of environments, spatial scene understanding (e.g., for image editing), and other use cases. [Overview of the project]

[0004]

[0004] This disclosure generally relates to the use of a nonlinear mapping function to derive depth predictions from the output of a layer of a machine learning model. The layer may be the output layer of the machine learning model so that the nonlinear mapping function is applied to the output of the machine learning model. Alternatively, the layer may be an intermediate layer, such as the last intermediate layer, so that the nonlinear mapping function is applied at the output layer of the machine learning model. The nonlinear mapping function may be selected based on a specific use case or error of interest. The system may use the depth data generated by the application of the nonlinear mapping function to train a machine learning model to produce a trained machine learning model. The trained machine learning model may be used to control the behavior of devices such as vehicles and robots in an environment, for purposes such as steering to avoid obstacles.

[0005]

[0005] In one example, the present disclosure describes a system comprising a memory configured to store image data captured by a plurality of cameras, and one or more processors communicatively coupled to the memory, wherein the one or more processors are configured to run a machine learning model on the image data, the machine learning model comprising a plurality of layers, to apply a nonlinear mapping function to the output of one of the plurality of layers in order to generate depth data, and to train the machine learning model on the depth data in order to generate a trained machine learning model.

[0006]

[0006] In another example, the present disclosure describes a method which includes running a machine learning model on image data captured by a plurality of cameras, wherein the machine learning model comprises a plurality of layers; applying a nonlinear mapping function to the output of one of the plurality of layers in order to generate depth data; and training the machine learning model on the depth data in order to generate a trained machine learning model.

[0007]

[0007] In another example, the present disclosure describes a non-temporary computer-readable storage medium containing instructions, which, when executed, causes one or more processors to run a machine learning model, comprising multiple layers, on image data captured by multiple cameras, to apply a nonlinear mapping function to the output of one of the multiple layers to generate depth data, and to train the machine learning model on the depth data to generate a trained machine learning model.

[0008]

[0008] In another example, the present disclosure describes a system which includes means for running a machine learning model on image data captured by a plurality of cameras, the machine learning model comprising a plurality of layers; means for applying a nonlinear mapping function to the output of one of the plurality of layers in order to generate depth data; and means for training the machine learning model on the depth data in order to generate a trained machine learning model.

[0009]

[0009] Details of one or more examples are described in the accompanying drawings and the following description. Other features, purposes, and advantages will become apparent from the specification, drawings and claims. [Brief explanation of the drawing]

[0010] [Figure 1]

[0010] This is a block diagram illustrating an exemplary vehicle according to one or more aspects of the present disclosure. [Figure 2]

[0011] This is a conceptual diagram illustrating an exemplary architecture of a system for providing depth prediction based on image data from multiple cameras, according to one or more aspects of the present disclosure. [Figure 3]

[0012] This is a conceptual diagram illustrating an example of a decoder-side architecture associated with a single camera among multiple cameras, according to one or more aspects of the present disclosure. [Figure 4]

[0013] This is a conceptual diagram illustrating an exemplary neural network according to one or more aspects of the present disclosure. [Figure 5]

[0014] This is a graph illustrating an exemplary nonlinear function according to one or more aspects of the present disclosure. [Figure 6]

[0015] This graph illustrates an example of a depth distribution according to one or more aspects of the present disclosure. [Figure 7]

[0016] This graph illustrates an example of a nonlinear function with different values ​​of alpha. [Figure 8]

[0017] This flowchart illustrates a depth prediction technique according to one or more aspects of the present disclosure. [Modes for carrying out the invention]

[0011]

[0018] In conventional stereo camera systems that utilize machine learning models such as neural networks, the neural network outputs disparity data between two cameras, which is then converted into depth data. For example, depth can be determined using triangulation based on the point correspondence between images captured by each of the two cameras. Disparity can be the displacement of a point between images captured by the two cameras. However, vehicles and / or robots may have a multi-camera setup rather than a stereo camera setup. Unlike conventional stereo camera systems, a multi-camera setup may include multiple cameras, each with a different field of view. Some or all of the multiple cameras may have overlapping fields of view with one or more other cameras in the multiple. Therefore, in a multi-camera setup, disparity data is not necessarily correlated with depth data.

[0012]

[0019] According to the technique of this disclosure, instead of outputting disparity data and converting the disparity data to depth data, the system may apply a nonlinear mapping function to the outputs of layers of a machine learning model (e.g., to the outputs of the output layer or the outputs of the hidden layers) to generate depth data, and then use the depth data to train a machine learning model to generate a trained machine learning model that can be used to operate vehicles, robots, augmented reality devices, virtual reality devices, etc.

[0013]

[0020] Figure 1 is a block diagram illustrating an exemplary vehicle according to one or more aspects of the present disclosure. The device 100 may include an autonomous driving vehicle, a driving assistance vehicle (e.g., a vehicle having an advanced driver-assistance system (ADAS)), a robot, an augmented reality system, a virtual reality system, and the like. In some examples, the device 100 may be referred to as an ego vehicle. In other examples, the device 100 may include more or fewer components than those shown in Figure 1.

[0014]

[0021] The device 100 may include a LiDAR system 102, a camera 104, a controller 106, one or more sensors 108, input / output device(s) 120, a wireless connection component 130, and a memory 160. The LiDAR system 102 may include one or more light emitters and one or more light sensors. The LiDAR system 102 may be deployed within or around the device 100. For example, the LiDAR system 102 may be mounted on the roof of the device 100, in the bumper of the device 100, and / or at other locations of the device 100. The LiDAR system 102 may be configured to emit light pulses and sense light pulses reflected from objects in the environment of the device 100. The LiDAR system 102 (and / or the processor(s) 110) may determine the distance to such objects based on the time between the emission of the light pulses and the sensing of the reflections of the light pulses. The LiDAR system 102 may emit such pulses within a 360-degree field around the device 100 to detect objects within the 360-degree field, such as objects in front of, behind, or to the side of the device 100. Although described herein as including the LiDAR system 102, it should be understood that another distance or depth sensing system may be used in place of the LiDAR system 102.

[0015]

[0022] Dense (or sparse) depth estimation can be relatively important for various applications such as autonomous driving, assistive robots, augmented reality, virtual reality scene construction, and image editing. For example, in an autonomous driving scenario, depth estimation may provide an estimated distance from one vehicle (e.g., a first vehicle) to another vehicle (e.g., a second vehicle), which may be important for the motion systems of the first vehicle, such as acceleration, braking, steering, etc.

[0016]

[0023] Depth estimation may be implemented by utilizing any of several different principles, such as the principles associated with the camera 104. For example, multi-camera information and the like may be used for depth estimation.

[0017]

[0024] The camera 104 may be a plurality of camera sensors (also referred to as cameras) located at positions in or on the vehicle, such as in or on mirrors, bumpers, and / or other locations of the device 100. The camera 104 may be configured to capture video or image data within the environment 195 surrounding the device 100. The controller 106 may use information from the camera 104 to determine depth information for objects that may be within the field of view of one or more of the cameras 104. In some examples, the cameras 104 may include or be referred to as a multi-view camera system.

[0018]

[0025] Different techniques exist for depth estimation or prediction. For example, depth estimation may be performed based on calibrated stereo images, multi-view stereo images, learning from monocular video (e.g., Structure from Motion (SfM)), and the like.

[0019]

[0026] When using a calibrated stereo setup, the most traditional way to perform this task is by using a known baseline. The system can perform disparity calculations between pixels of two corresponding images captured by a pair of horizontally offset cameras. The system can then calculate depth directly from the results of the disparity calculation by knowing the focal length and the stereo setup baseline (e.g., the distance between the two cameras). Consequently, a deep learning network that conventionally works on stereo camera input images can first predict the disparity and then convert the disparity to depth in order to predict the depth.

[0020]

[0027] Depth estimation or prediction can be performed in several different ways in monocular or multi-camera setups. Using self-monitoring, the system may estimate depth using the time axis. Using this technique, the system may require reliable attitude information between image frames from either a positioning engine or an attitude estimation network. The system may use self-monitoring to utilize collaborative learning of depth and self-motion (e.g., the motion of a vehicle or robot including the system). The use of self-monitoring may leverage the Structure-from-Motion (SfM) principle to perform view synthesis as the self-monitoring signal. The use of self-monitoring may also eliminate the need for depth ground truth, which can sometimes be difficult to construct. Furthermore, the richness of video data makes self-monitoring an attractive option.

[0021]

[0028] With complete monitoring, a system can learn to map images to dense depth maps. However, obtaining high-quality depth annotations in the real world is extremely difficult.

[0022]

[0029] In partial monitoring, the system relies primarily on self-monitoring, while full monitoring can be applied to a small set of pixels. Such techniques can help resolve the ambiguity of scale in self-monitoring.

[0023]

[0030] For vehicles with a single (forward-facing) camera, depth estimation using data from a single camera may not be sufficient for a complete scene understanding. For example, there is no information about the sides and rear of the autonomous vehicle. However, autonomous vehicles may need a representation of their surroundings to navigate or maneuver successfully, for example, to avoid obstacles.

[0024]

[0031] Multi-camera setups inside or on vehicles are ubiquitous in modern autonomous vehicles. The use of multiple cameras, such as camera 104, can capture the entire surrounding environment of the autonomous vehicle. Therefore, multi-camera depth estimation can be particularly useful in constructing a complete 3D representation of the surrounding environment.

[0025]

[0032] If the system uses calibrated stereo techniques, a clear baseline may exist, making the relationship between parallax and depth meaningful, and thus predicting parallax can provide some insight. For other techniques such as monocular, uncalibrated stereo, and multiview, the relationship between parallax and depth is typically weak. This weak relationship can lead to inherent scale ambiguity.

[0026]

[0033] Regardless of the technique used, depth is typically the information ultimately sought, and predicting disparity has been the dominant approach, as most depth estimation efforts have started with the use of stereo techniques. Additionally, in such conventional cases, the output of the neural network is disparity, so the system may need to employ floating-point division (which can be costly to implement in hardware) to derive depth. The system may also need to convert the disparity data into a form that is more easily visualized or used for inference.

[0027]

[0034] According to the technique disclosed herein, the output of machine learning models such as neural networks can be directly interpreted as depth information. In other words, the technique takes the disparity output from the neural network and does not convert the disparity to depth.

[0028]

[0035] If a system, such as a machine learning model, normalizes depth to a value within the range [0,1], multiplies the depth by (max_depth - min_depth), and adds min_depth, which may be a linear function, the result may be smaller than desired. max_depth may represent the maximum depth in the scene or image, and min_depth may represent the minimum depth in the scene or image. For example, the weights of a neural network may saturate due to the sigmoid properties of the output layer. For example, the input value must be very large in order to make the value 1.

[0029]

[0036] The controller 106 may be an autonomous or driver assistance controller (e.g., ADAS) configured to control the operation of device 100. For example, the controller 106 may control the acceleration, braking, and / or navigation of device 100 through the environment surrounding device 100. The controller 106 may include one or more processors, e.g., one or more processors 110. The processors may include one or more central processing units (CPUs), such as single-core or multi-core CPUs, graphics processing units (GPUs), digital signal processors (DSPs), neural processing units (NPUs), and multimedia processing units. Instructions executed by the processors 110 may be read from memory 160, for example, to cause the processors 110 to perform operations attributed to the processors 110 in this disclosure. In some examples, one or more processors 110 may be based on the ARM or RISC-V instruction set.

[0030]

[0037] An NPU is generally a specialized circuit configured to implement all the necessary control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), kernel methods, and so on. An NPU may also be referred to as a neural signal processor (NSP), tensor processing unit (TPU), neural network processor (NNP), intelligence processing unit (IPU), or vision processing unit (VPU).

[0031]

[0038] The processor(s) 110 may be configured to accelerate performance on common machine learning tasks such as image classification, machine translation, object detection, and various other tasks. In some examples, multiple processors 110 may be instantiated on a single chip, such as a system on a chip (SoC), while in other examples, one or more processors 110 may be part of a dedicated machine learning accelerator device.

[0032]

[0039] In some examples, one or more processors 110 can be optimized for training or inference, or in some cases, configured to balance performance between both. With respect to a processor(s) 110 capable of performing both training and inference, the two tasks can still generally be performed independently.

[0033]

[0040] In some examples, a processor(s) 110 designed to accelerate training is generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation involving inputting an existing dataset (often labeled or tagged), iterating through that dataset, and then tuning model parameters 184 such as weights and biases to improve model performance. Generally, optimization based on incorrect predictions involves backpropagating through layers of the model to determine gradients to reduce prediction errors. In some examples, some or all of the tuning of model parameters 184 may be performed outside of device 100, such as within a cloud computing environment.

[0034]

[0041] In some examples, a processor(s) 110 designed to accelerate inference is generally configured to operate on a complete model. Thus, such a processor(s) 110 may be configured to take new data as input and process the data rapidly through a model that has already been trained to produce model outputs (e.g., inferences).

[0035]

[0042] In some examples, processor(s) 110 may operate on predictive models such as artificial neural networks (ANNs) or random forests (RFs). An ANN may include hardware and / or software components, including several connected nodes (e.g., artificial neurons), roughly corresponding to neurons in the human brain. Each connection or edge transmits a signal from one node to another (like a physical synapse in the brain). When a node receives a signal, it processes it and then transmits the processed signal to other connected nodes. In some cases, the signals between nodes contain real numbers, and the output of each node is calculated as a function of the sum of its inputs. Each node and edge may be associated with one or more node weights that determine how the signal is processed and transmitted. During the training process, these weights are adjusted to improve the accuracy of the results (i.e., by minimizing a loss function that somehow corresponds to the difference between the current result and the target result). Edge weights increase or decrease the intensity of the signal transmitted between nodes. In some cases, nodes have thresholds below which no signal is transmitted at all. In some examples, nodes are aggregated into layers. Different layers perform different transformations on their inputs. The first layer is known as the input layer, and the last layer is known as the output layer. In some cases, a signal traverses a particular layer multiple times.

[0036]

[0043] A convolutional neural network (CNN) is a class of neural networks commonly used in computer vision or image classification systems. In some cases, CNNs can enable the processing of digital images with minimal preprocessing. CNNs can be characterized by the use of convolutional (or cross-correlation) hidden layers. These layers apply convolutional operations to the input before signaling the results to the next layer. Each convolutional node can process data from a limited field (i.e., the receptive field) of the input. During the forward pass of a CNN, the filters in each layer can be convolved across the input volume, and the dot product between the filter and the input is computed. During the training process, filters can be modified to activate when they detect specific features in the input.

[0037]

[0044] The term "loss function" refers to a function that influences how a machine learning model is trained in a supervised learning model. Specifically, during each training iteration, the model's output is compared to known annotation information in the training data. The loss function provides a value indicating how close the predicted annotation data is to the actual annotation data. After calculating the loss function, the model's parameters are updated accordingly, and a new set of predictions is made during the next iteration.

[0038]

[0045] In some embodiments, the wireless connection 130 components may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connection 130 processing components are further connected to one or more antennas 135.

[0039]

[0046] The processor(s) 110 may also include one or more sensor processing units associated with the LiDAR system 102, the camera 104, and / or the sensor(s) 108. For example, the processor(s) 110 may include one or more image signal processors associated with the camera 104 and / or the sensor(s) 108, and / or a navigation processor associated with the sensor(s) 108, the navigation processor may include satellite-based positioning system components (e.g., GPS or GLONASS), and inertial positioning system components.

[0040]

[0047] Device 100 may also include one or more input and / or output devices 120, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, or a microphone.

[0041]

[0048] Device 100 also includes memory 160, which represents one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this example, memory 160 includes computer executable components that can be run by one or more of the aforementioned components of device 100.

[0042]

[0049] Examples of memory 160 include random access memory (RAM), read-only memory (ROM), or hard disks. Examples of memory 160 also include solid-state memory and hard disk drives. In some examples, memory 160 is used to store computer-readable, computer-executable software, which, when executed, includes instructions that cause the processor to perform various functions described herein. In some cases, memory 160 includes a basic input / output system (BIOS) that controls basic hardware or software operations, such as, among other things, interaction with peripheral components or peripheral devices. In some cases, a memory controller operates the memory cells. For example, the memory controller may include a row decoder, a column decoder, or both. In some cases, the memory cells in memory 160 store information in the form of logical states.

[0043]

[0050] In particular, in this example, memory 160 contains model parameters 184 (e.g., weights, biases, and other machine learning model parameters 184). One or more of the illustrated components, and other components not illustrated, may be configured to implement various aspects of the techniques described herein.

[0044]

[0051] In general, device 100 and / or its components may be configured to carry out the techniques described herein. Note that device 100 in Figure 1 is merely an example, and in other examples, alternative processing systems 100 having more components, fewer components, and / or different components may be used.

[0045]

[0052] In some embodiments, the camera 104 and / or sensor(s) 108 may include optical equipment (e.g., an image sensor, camera, etc.) for recording or capturing images, such as those that can be stored locally and transmitted elsewhere. For example, an image sensor may capture visual information using one or more photosensitive elements that can be tuned for sensitivity to the visible spectrum of electromagnetic radiation. The resolution of such visual information may be measured in pixels, and each pixel may associate independently captured information. Thus, in some cases, each pixel may correspond to, for example, one component of the two-dimensional (2D) Fourier transform of an image. A computational method may use the pixel information to reconstruct the image captured by the device. In a camera, the image sensor may convert light incident on the camera lens into an analog or digital signal. The electronic device may then display the image on a display panel based on the digital signal. Image sensors are commonly incorporated into electronic devices such as smartphones, tablet personal computers (PCs), laptop PCs, and wearable devices.

[0046]

[0053] In some embodiments, the sensor(s) 108 may include a direct depth-sensing sensor capable of determining the depth of an object or the distance to an object in the environment surrounding the device 100. Data from such a depth-sensing sensor may be used to supplement the depth mapping techniques discussed herein.

[0047]

[0054] An input / output device(s) 120 (which may include, for example, I / O ports) may manage the input and output signals of device 100. In some cases, an input / output device(s) 120 may represent a physical connection or port to an external peripheral. In some cases, an input / output device(s) 120 may utilize an operating system. In other cases, an input / output device(s) 120 may represent or interact with a modem, keyboard, mouse, touchscreen, or similar device. In some cases, an input / output device(s) 120 may be implemented as part of a processor (for example, a processor in processor(s) 110). In some cases, a user may interact with the device via an input / output device(s) 120 or via a hardware component controlled by an input / output device(s) 120.

[0048]

[0055] In one embodiment, the memory 160 includes depth output 165 (which may include a depth map), depth loss 170, training information 175, photometric loss 180, smoothing loss 182, model parameters 184, inference 186, pull loss 188, and one or more nonlinear functions 190.

[0049]

[0056] Figure 2 is a conceptual diagram illustrating an exemplary architecture of a system for providing depth prediction based on image data from multiple cameras, according to one or more aspects of this disclosure. Each of the image data 202A, 202B-202N (collectively referred to as image data 202) represents image data captured by each of the cameras 104. Image data 202A is set at time t(I t Image data 202B may include frames of image data captured by the first camera of camera 104 at time t(I t Image data 202N may include frames of image data captured by the second camera of camera 104 at time t-1, time t-2, and so on.t ), may include frames of image data captured by the Nth camera among the cameras 104 at time t-1, time t-2, and so on.

[0050]

[0057] Image data 202 can be input to a machine learning model 250. The machine learning model 250 may include encoders 203A-203N, a cross-attention function 240, and / or decoders 204A-204N. In some examples, instead of a single machine learning model 250 as shown, there may be separate machine learning models, such as each machine learning model associated with each image data (e.g., image data 202A, image data 202B, image data 202N) of image data 202. The cross-attenuation unit 240 may be configured to allow features in images captured from different cameras to interact and cooperate to improve the overall quality of features extracted from images captured across all of the cameras 104.

[0051]

[0058] Machine learning model 250 processes image data 202 and outputs depth output.

[0052]

number

[0053] This generates a mapping, and then mapping units 206A to 206N (collectively referred to as mapping unit 206), for example,

[0054]

number

[0055] This applies. Output

[0056]

number

[0057] This can take different forms, such as a depth map that directly shows the estimated depth of each pixel. The mapping unit 206 uses a nonlinear mapping function as the output of the machine learning model 250.

[0058]

number

[0059] It can be applied to the output of machine learning model 250. While Figure 2 shows it being applied to the output of machine learning model 250, in some examples, mapping 206 can be applied as part of machine learning model 250 itself, for example, as the output layer of machine learning model 250.

[0060]

[0059] Decoder-side architectures 230A, 230B to 230N are shown in an abbreviated form for the purpose of simplification. Each of the decoder-side architectures 230 may be associated with each camera of camera 104 so that each decoder-side architecture 230 may receive encoded image data 202 associated with its respective camera. For example, decoder-side architecture 230A may be associated with camera 104A, decoder-side architecture 230B may be associated with camera 104B, and so on.

[0061]

[0060] Each output of the decoder-side architecture 230 may be input to the final loss unit 205. The final loss unit 205 may provide the final loss to generate depth data such as depth estimation. In some examples, this depth estimation may take the form of a depth map.

[0062]

[0061] Figure 3 is a conceptual diagram illustrating an example of a decoder-side architecture associated with a single camera among a plurality of cameras, according to one or more aspects of the present disclosure. In the example of Figure 3, a decoder-side architecture 230A associated with camera 104A is shown. Each of the decoder-side architectures 230B to 230N may be similar to the decoder-side architecture 230A.

[0063]

[0062] As discussed with respect to Figure 2, the output of decoder 204A, which may be the output of machine learning model 250, is the depth output

[0064]

number

[0065] It may include. The mapping unit 204A is a nonlinear mapping function

[0066]

number

[0067] output depth

[0068]

number

[0069] Apply to,

[0070]

number

[0071] It can generate more accurate depth estimates.

[0072]

[0063] The output of the mapping unit 204A can be input to the view synthesis unit 218A. The view synthesis unit 218 accepts one or more context frames (I) as input. S The target frame is reconstructed by further obtaining attitude estimates from 216A and relative attitude unit 220A.

[0073]

number

[0074] 222A is generated. For example, the view synthesis unit 218A may perform interpolation such as bilinear interpolation based on the relative pose from the relative pose unit 220A and using the output of the mapping unit 206A.

[0075]

[0064] The relative attitude unit 220 is generally configured to determine the relative attitude. Such a relative attitude may be determined based on data such as global positioning satellite (GPS) data. Such a relative attitude may change, for example, from frame to frame of image data 204A when the device 100 is moving.

[0076]

[0065] Reconstructed target frame

[0077]

number

[0078] 222A may be input to the smoothing loss unit 208, which, for example, reconstructs the target frame

[0079]

number

[0080] The loss is determined based on the "smoothing" of 222A or the output of the mapping unit 206A. In one embodiment, smoothing is performed on the reconstructed target frame

[0081]

number

[0082] The depth can be measured by the gradient (or average gradient) between adjacent pixels across 222A or by the output of the mapping unit 206A. For example, an image of a simple scene with a few objects may have a very smooth depth map, but an image of a complex scene with many objects may have a less smooth depth map because the gradient between adjacent pixels changes frequently and significantly due to the reflection of many objects.

[0083]

[0066] The smoothing loss unit 208A provides the smoothing loss component to the final loss unit 205. Although not shown in Figure 3, the smoothing loss component may be associated with hyperparameters (e.g., weights) in the final loss unit 205, which may vary the effect of the smoothing loss on the final loss unit 205.

[0084]

[0067] The context frame 216A can generally be a frame close to the target frame 202A. For example, the context frame 216A can be several frames or time steps of any size on the target frame 202A, such as t+ / -1 (adjacent frame), t+ / -2 (non-adjacent frame), etc. While these examples are symmetric with respect to the target frame 202A, the context frame 216A can be located asymmetrically, such as t-1 and t+3.

[0085]

[0068] The reconstructed target frame 222A may be compared with the target frame 202A by the photometric loss unit 224A to generate a photometric loss, which is another component used by the final loss unit 205. As discussed above, although not shown in the figure, the photometric loss component may be associated with hyperparameters (e.g., weights) in the final loss unit 205, which changes the effect of the photometric loss on the final loss unit 205.

[0086]

[0069] The output of the relative attitude unit 220A may be input to the adaptive depth monitoring loss unit 212A, which takes a partial depth ground truth 210A of the target frame 202A as a further input to generate the adaptive depth monitoring loss. In some embodiments, the adaptive depth monitoring loss function 212A has or uses only estimated depth ground truth values ​​for a portion of the scene in the target frame 202, and therefore this step may be referred to as “partial monitoring”. In other words, the machine learning model 250 provides a depth output for each pixel in the target frame 202A, but the partial depth ground truth 210A may include only estimated ground truth values ​​for a subset of pixels in the target frame 202A.

[0087]

[0070] The partial depth ground truth 210A is determined through one of a variety of different techniques. In one embodiment, a sensor fusion function (or module) uses one or more sensors to directly sense depth information from a portion of the target frame.

[0088]

[0071] The pull loss unit 226A may calculate a pull loss that represents the degree to which the corners of an object are accurately joined in the depth map. The pull loss unit 226A may receive data representing the input shape in order to calculate the pull loss. The pull loss unit 226A may provide the pull loss to the final loss unit 205. The pull loss may function as a prior value for depth values ​​to keep the depth values ​​within a given set, which may be useful in areas where the data may not be easily understood, such as open sky. As discussed above, although not shown in the figure, the pull loss component may be associated with hyperparameters (e.g., weights) in the final loss unit 205, which will change the effect of the pull loss on the final loss unit 205.

[0089]

[0072] In one embodiment, the final or total (multi-component) loss generated by the final loss unit 205 (which may be generated based on the smoothing loss generated by the smoothing loss unit 208A, the adaptive depth monitoring loss generated by the adaptive depth monitoring loss unit 212A, the photometric loss generated by the photometric loss unit 224A, and / or the pull loss generated by the pull loss unit 226A) is used to update or improve the machine learning model 250. For example, using gradient descent and / or backpropagation, one or more parameters of the depth model 204 may be improved or updated based on the total loss generated for a given target frame 202.

[0090]

[0073] In some embodiments, this update may be performed independently and / or sequentially for a set of target frames 202A (for example, by using stochastic gradient descent to sequentially update the parameters of the machine learning model 250 based on each target frame 202A), and / or on a batch of target frames 202A (for example, by using batch gradient descent).

[0091]

[0074] Using the architectures of Figures 2 and 3, the machine learning model 250 is trained to thereby produce improved and more accurate depth estimates. During runtime inference, the trained model 250 calculates the depth output for the input target frame, for example, target frame 202A.

[0092]

number

[0093] This can be used to generate depth output, which can then be used for various purposes such as autonomous driving and / or driver assistance, as discussed above. In some embodiments, at runtime, the machine learning model 250 may be used without considering or using other embodiments of the architecture in Figures 2 and 3, such as context frame, view synthesis unit, relative attitude unit, reconstructed target frame, photometric loss unit, smoothing loss unit, adaptive depth monitoring loss unit, pull loss unit, and / or final loss unit.

[0094]

[0075] Figure 4 is a conceptual diagram illustrating an exemplary neural network according to one or more aspects of the present disclosure. The neural network 400 may be an example of the machine learning model 250 or a part thereof.

[0095]

[0076] The neural network 400 may include an input layer 402, one or more hidden layers 404, and an output layer 406. The input layer 402 may represent an input to the neural network 400. For example, such an input may include image data 202. The hidden layers 404, also called hidden layers, may apply various functions to the data provided by the preceding layer. The output layer 406 may apply a function to the output of the last layer of the hidden layers 404, and may output an estimate or prediction from the neural network 400.

[0096]

[0077] Each input data is input to each node of the first layer of the hidden layer(s) 402. Each node of the hidden layer(s) may apply weights to the input data to the node as part of providing a function for each layer of the hidden layer(s) 402. For example, a node may apply weights according to an activation function that a given layer may contain. When the neural network 400 is being trained (for example, using the architectures of Figures 2 and 3), the node weights may be adjusted based on, for example, the final loss (and / or any of the other losses considered in relation to Figure 3) determined by the final loss unit 205. Thus, the relationship between the input image data and the depth estimate may be adjusted to improve the depth estimate.

[0097]

[0078] Note that the number of hidden layers, inputs, outputs, and nodes of the neural network 400 is provided as an example only. The neural network 400 may include any number of hidden layers, inputs, outputs, and / or nodes.

[0098]

[0079] As discussed above, in some examples, the mapping unit 206 may apply a nonlinear mapping function (one or more) to the output of the machine learning model 250. For example, mapping may be performed on the output 408 of the neural network 400. In other examples, the mapping unit 206 may be part of the output layer 406. In other words, mapping may be performed as a function of the output layer 406. For example, the output layer 406 may perform a nonlinear mapping function on the output from the last layer of the hidden layer (one or more) 404. In such examples, the mapping unit 206 is part of the neural network 400 and does not need to exist separately, nor does it need to apply a nonlinear mapping function to the output 408.

[0099]

[0080] In some examples, the neural network 400 may apply one or more activation functions to the image data 202. An activation function may be a function applied by the node(s) of the neural network to compute the output of the node(s). In some examples, all of the hidden layers(s) 404 may apply an activation function to the data from the previous layer. In some examples, the output layer 406 may apply an activation function. For example, different experiments were conducted using activation functions that focused mainly on two places, namely, 1) all of the hidden layers(s) 404 and 2) the output layer 406. For example, the potential activation functions of the rectified linear unit (ReLU), parametric rectified linear unit (PReLU), and exponential linear unit (ELU) were tested for all of the hidden layers(s) 404. The latent activation functions ReLU, PReLU, ELU, sigmoid, α×sigmoid (avoiding pushing values ​​to +infinity to obtain max_depth), and soft plus were tested for the output layer 406. Experiments showed that in-place ELU for the output layer 406 achieved higher accuracy and better training stability than the other activation functions. For the hidden layer(s) 404, PReLU performed better than the other tested alternatives.

[0100]

[0081] The output of the machine learning model 250 may be in the range [0,1], and this output can be uniform within that range. However, different levels of precision exist for different depth values, and relative error may be used as the metric. As a result, the output of the machine learning model 250 may not necessarily map linearly to the output range [min_depth,max_depth]. In fact, it may be desirable to map the output of the machine learning model 250 non-linearly to the range [min_depth,max_depth].

[0101]

[0082] Next, the potential mapping of the output of the machine learning model 250 to the output range is considered. In the example of an autonomous driving scenario, it may be desirable to select a nonlinear function such that the output of the nonlinear function is very accurate within the first 20 meters from the ego vehicle. For example, in an autonomous driving scenario, it may be more important that depth information about objects relatively close to the ego vehicle is more accurate than depth information about objects relatively farther away from the ego vehicle.

[0102]

[0083] Figure 5 is a graph illustrating an exemplary nonlinear function according to one or more aspects of the present disclosure. In the example of Figure 5, the x-axis is the input depth, which may be in the range of 0 to 1 (for example, input to the mapping unit 206A in Figure 2).

[0103]

number

[0104] The x-axis represents the output depth (e.g., within the range of min_depth~max_depth) of a nonlinear function (e.g., the output of mapping unit 206A). Since it may be desirable to have more accuracy in depth information associated with objects relatively close to device 100, there may be more pixels with lower depth values, and since the accuracy metric may be more sensitive to errors at small depth values, a nonlinear mapping function may be selected to allocate a larger x-range to the lower half of the y-range. To achieve this, several possible functions may be used, such as logarithmic (power) functions, square root (square) functions, and negative inverse functions. As can be seen in Figure 5, approximately 0.7 units of the x-axis of function 500 are allocated to the first 0.5 units of the y-axis.

[0105]

[0084] For example, the slope of the nonlinear mapping function may be selected to be based on the probability distribution of the loss function, the input depth of the nonlinear mapping function, and / or the output depth of the nonlinear mapping function. For example, the nonlinear mapping function may be associated with a mean absolute relative error loss function, and the slope of the nonlinear mapping function may increase with the input depth of the nonlinear mapping function and / or decrease with a higher probability density of the output depth of the nonlinear mapping function.

[0106]

[0085] Figure 6 is a graph illustrating an example of depth distribution according to one or more aspects of the present disclosure. The x-axis in Figure 6 represents the depth of pixels in the image data 204, and the y-axis in Figure 6 represents the number of pixels of the image data 204 within the depth represented by the x-axis. As can be seen, there are many points in the range of 3 to 8 m, a few points in the immediately following range, and then another number of points in the range greater than 100 m (e.g., the sky, the horizon). The distribution of points in the pixel data can vary considerably depending on the environment in which the device 100 is operating. For example, pixel data in an urban environment may be significantly different from pixel data in a rural environment.

[0107]

[0086] A nonlinear function can be derived to satisfy the error of any data or problem. For example, in the case of ego vehicles, one nonlinear function may be appropriate for highway driving objects that are relatively far apart from each other, while a different nonlinear function may be appropriate for urban driving where objects may be closer together and further apart. As such a system, device 100 may select or determine a nonlinear mapping function based on the environment in which device 100 is operating. For example, device 100 may include a plurality of nonlinear mapping functions 162, from which device 100 may select one for use in mapping 204 based on the surrounding environment detected by device 100 through data from a LiDAR system 102, a camera 104, and / or one or more sensors 108, user input, etc.

[0108]

[0087] A large class of strictly monotonic functions may be used for mapping. Such mappings may map a portion of a curve to a reasonable range around 0, e.g., the range [0,1]. Simple normalization may be used. Some of the potential nonlinear mapping functions may work better than others, depending on the exact distribution of pixel depth. For example, the square root function may work better than the logarithmic function.

[0109]

[0088] In some examples, an optimal nonlinear mapping function that can be used to map the output of a depth estimation neural network to an output range can be derived theoretically or empirically. Both theoretical and empirical techniques were used to determine this function as described herein.

[0110]

[0089] With regard to theoretical techniques, both discretization approaches and computation-based approaches were analyzed. For example, formal techniques for determining this mapping function may include using computation of variation and using numerical techniques with discretized ranges. Such techniques may result in lookup-based techniques that may include curve fitting to the discretized version.

[0111]

[0090] For example, one could start with a proxy optimization function configured to maintain uniform error characteristics by equalizing the amount of error in each epsilon interval. This can ensure that any small perturbation to the input results in a small perturbation to the output, rather than a small perturbation to the input resulting in a large perturbation to the output.

[0112]

[0091] FIG. 7 is a graph diagram illustrating examples of nonlinear functions having different values of alpha. Consider discretized analysis here. When the discretized analysis is applied to an output range (e.g., of the output 408 in FIG. 4), a min_depth of 1.0 and a uniform distribution of pixel depth values may be assumed. Accordingly, a sequence that may be a discrete representation of the output 4080 will have the following properties. When alpha is equal to 1, the sequence may be 1, 2, 4, 7, (11), (16), .... Such a sequence may be referred to as the lazy caterer's sequence (quadratic form), which describes the maximum number of pieces of a single disk (e.g., a pancake or pizza, which is commonly used to illustrate the situation) that can be produced by a given number of straight cuts. For other values of alpha, the sequence may be exponential (aₙ=αaₙ₋₁+(n-1)α+1). Accordingly, depending on the value of α, different shapes may be defined for this function, as shown in FIG. 7.

[0113]

[0092] A non-uniform distribution of depth values, for example,

[0114]

Equation

[0115] When a distribution in the form of is assumed, the sequence may be 1, 2, 6, 30, 270, (4590), ....

[0116]

[0093] The increase may be, for example, in the form of x x . For example, when alpha is equal to 1: The nth term of the sequence is given by the following formula T(n)=n n +n Here, using these relational expressions, the following series can be easily obtained T(1)=1 1 +1=2 T(2)=2 2 +2=6 T(3)=3 3 +3=30 T(4)=44 +4=260 Similarly, T(5) = 5 5 +5=3130

[0117] For other values ​​of alpha, the discretized formula may become unworkable.

[0118]

[0094] Next, we consider using variational calculus to derive the mapping function.

[0119]

[0095] As discussed above, logarithmic functions, square root functions, and x 1 / 3 Various functions, including functions, were tested empirically.

[0120]

[0096] For example, the nonlinear mapping function applied by mapping 206 could be y=f(x), which is f:[0,1]→[d min d max [0,1] is defined from ]. [0,1] may be the output of the decoder 204 of the machine learning model 250. The decoder 204 may output values ​​from 0 to 1, or -1 to 1, or -0.5 to 0.5, which are typically in a small range, but in contrast to this, d min d max This refers to a wider range, such as 1-100m or 1-300m.

[0121]

[0097] The nonlinear mapping function f may be a monotonically non-decreasing function (non-decreasing with respect to increase in order to capture the aspect in which the function may be flat over a certain range). Some examples of the slope of the function based on different considerations are as follows: For a general cost function that provides the same penalty for all depths,

[0122]

number

[0123] And in the equation, η is a constant. In this case, the function y(x) is linear. However, the cost is

[0124]

number

[0125] In this form, errors at greater depths are weighted less (y in the denominator). * (caused by) In this example, y * This is the true depth from the ground truth, such as partial depth ground truth 210A. Exemplary slopes can take the following forms. Note that the steepness increases with depth, as shown below.

[0126]

number

[0127]

[0098] In addition, given a scenario in which the probability distribution p(.) of the ground truth data is provided, some exemplary slopes of the nonlinear mapping function are provided below. The slope is lower when p(y) is high or has a higher density.

[0128]

number

[0129]

[0099] Furthermore, for example, if the cost function is different, for example,

[0130]

number

[0131] If this is the case, this further weights the effect of errors at greater distances, and further possibilities may arise, as described below. In such cases, the slope increases much more steeply with distance.

[0132]

number

[0133]

[0100] From the slope, the actual nonlinear mapping function f(x) can be obtained via numerical integration or solved in a closed form. The above are just some exemplary implementations, and other implementations are possible, for example, the overall guideline for the steepness of the nonlinear mapping function depends on the depth cost function and the depth probability distribution.

[0134]

[0101] For example, if the output of Qdepth is x, then the function

[0135]

number

[0136] To find a specific loss function

[0137]

number

[0138] In some cases, optimization is desirable. Considerations for optimization may include the fact that only small perturbations exist in the loss under various considerations such as mean loss, scaled loss with respect to depth distribution, and uniform loss over all x, for small perturbations in the input x.

[0139]

[0102] Figure 8 is a flowchart illustrating a depth prediction technique according to one or more aspects of the present disclosure. A processor(s) 110 may run a machine learning model 250 on image data 202, the machine learning model 250 comprising several layers (e.g., an input layer 402, an intermediate layer(s) 404, and an output layer 406) (800). For example, a processor(s) 110 may run a machine learning model 250 using image data 202 from a camera 104 as input.

[0140]

[0103] The processor(s) 110 may apply a nonlinear mapping function to the output of one of the layers in order to generate depth data (802). For example, the processor(s) 110 may apply one of the nonlinear mapping functions 190 to either the output of the machine learning model 250 (e.g., the output of the output layer 406) or the output of the last layer of the hidden layer(s) 404 (in which case the nonlinear mapping function is applied at the output layer 406).

[0141]

[0104] The processor(s) 110 may train the machine learning model 250 based on depth data in order to generate a trained machine learning model 250. For example, the processor(s) 110 may train the machine learning model 250 using depth data to better estimate depth. In the process of training the machine learning model 250, the processor(s) 110 may modify model parameters 184, such as weights applied by the nodes of the machine learning model 250, in order to improve depth estimation such as inference 186.

[0142]

[0105] In some examples, the slope of the nonlinear mapping function is based on at least one of the loss function, the input depth of the nonlinear mapping function, or the probability distribution of the output depth of the nonlinear mapping function. In some examples, the nonlinear mapping function is associated with a mean absolute relative error loss function, and the slope of the nonlinear mapping function is at least one of a) increasing with the input depth of the nonlinear mapping function, or b) decreasing with a higher probability density of the output depth of the nonlinear mapping function.

[0143]

[0106] In some examples, the processor(s) 110 may be further configured to control the operation of a device (e.g., device 100) based on a trained machine learning model 250. In some examples, the device includes a vehicle or a robot. In some examples, as part of controlling the operation of the vehicle or robot, the processor(s) 110 may be configured to navigate the vehicle or robot within the environment 195.

[0144]

[0107] In some examples, the system includes a camera 104, which includes at least three cameras (e.g., a multi-camera setup), where at least three cameras are configured to capture image data 202, and each of the at least three cameras has a different field of view. In some examples, each of the cameras 104 has its own field of view, and at least two of each field of view overlap. In some examples, one layer is an output layer 406, where one or more processors are configured to apply a nonlinear mapping function to the output of the machine learning model 250 as part of applying a nonlinear mapping function. In some examples, one layer is a hidden layer (e.g., one or more hidden layers 404), where one or more processors are configured to apply a nonlinear mapping function to the output layer 406 of the machine learning model 250 as part of applying a nonlinear mapping function.

[0145]

[0108] In some examples, multiple layers include at least one activation function layer configured to apply an activation function to the output of each preceding layer. In some examples, at least one activation function layer includes at least one hidden layer (e.g., of hidden layer 404), and the activation function of at least one hidden layer includes at least one of normalized linear unit (ReLU), parametric normalized linear unit (PReLU), or exponential linear unit (ELU). In some examples, the activation function of at least one hidden layer includes PReLU. Note that the activation function may be stored as model parameter 184.

[0146]

[0109] In some examples, at least one activation function layer includes an output layer 406, the activation function of the output layer 406 includes at least one of the normalized linear unit (ReLU), parametric normalized linear unit (PReLU), and exponential linear unit (ELU), sigmoid, α×sigmoid, or soft plus. In some examples, the activation function of the output layer 406 includes PReLU. In some examples, the output of one layer does not include disparity data, and the processor(s) 110 may be configured to train a machine learning model 250 without determining the disparity data.

[0147]

[0110] In some examples, the nonlinear mapping function is the first nonlinear mapping function among a plurality of nonlinear mapping functions (e.g., one or more nonlinear functions 190), and the depth data is the first depth data. In such examples, the processor(s) 110 may be further configured to determine changes in the environment 195 and, based on the changes in the environment 195, apply a second nonlinear mapping function among the plurality of nonlinear mapping functions to the output of one of the plurality of layers in order to generate second depth data. The processor(s) 110 may be configured to further train the trained machine learning model based on the second depth data. In some examples, as part of applying a second nonlinear mapping function among the plurality of nonlinear mapping functions to the output of one of the plurality of layers, the processor(s) 110 may be configured to stop applying the first nonlinear mapping function among the plurality of nonlinear mapping functions.

[0148]

[0111] In some examples, the processor(s) 110 may run a machine learning model 252 on image data 202, the machine learning model 250 comprising several layers (e.g., an input layer 402, an intermediate layer(s) 404, and an output layer 406). The processor(s) 110 may apply a nonlinear mapping function (e.g., a nonlinear mapping function 190) to the output of one of the layers to predict depth data. In some examples, this machine learning model may be a trained machine learning model. The processor(s) 110 may be further configured to use depth data in at least one of an advanced driver assistance system, a robot, an augmented reality device, or a virtual reality device. For example, a device 100 which may include an advanced driver assistance system, a robot, an augmented reality device, or a virtual reality device may use depth data in an operation performed by the device 100, such as navigation.

[0149]

[0112] The embodiments of the various aspects of this disclosure may be used individually or in any combination.

[0150]

[0113] This disclosure includes the following clauses:

[0151]

[0114] Clause 1. A system comprising: a memory configured to store image data captured by a plurality of cameras; and one or more processors communicatively coupled to the memory, wherein the one or more processors are configured to run a machine learning model on the image data, the machine learning model comprising a plurality of layers; to apply a nonlinear mapping function to the output of one of the plurality of layers in order to generate depth data; and to train the machine learning model on the depth data in order to generate a trained machine learning model.

[0152]

[0115] Clause 2. The system according to Clause 1, wherein the slope of the nonlinear mapping function is based on at least one of the loss function, the input depth of the nonlinear mapping function, or the probability distribution of the output depth of the nonlinear mapping function.

[0153]

[0116] Clause 3. The system according to Clause 2, wherein the nonlinear mapping function is associated with a mean absolute relative error loss function, and the slope of the nonlinear mapping function is at least one of the following: a) increases with the input depth of the nonlinear mapping function, or b) decreases with the higher probability density of the output depth of the nonlinear mapping function.

[0154]

[0117] Clause 4. The system described in any of Clauses 1 to 3, wherein one or more processors are further configured to control the operation of the device based on a trained machine learning model.

[0155]

[0118] Clause 5. The system described in Clause 4, wherein the device includes a vehicle or robot, and one or more processors are configured to navigate the vehicle or robot in an environment as part of controlling the operation of the vehicle or robot.

[0156]

[0119] Clause 6. The system according to any one of Clauses 1 to 5, further comprising a plurality of cameras, wherein the plurality of cameras includes at least three cameras, the at least three cameras are configured to capture image data, and each of the at least three cameras has a different field of view.

[0157]

[0120] Clause 7. The system according to any one of Clauses 1 to 6, wherein one layer is an output layer, and one or more processors are configured to apply a nonlinear mapping function to the output of a machine learning model as part of applying a nonlinear mapping function.

[0158]

[0121] Clause 8. The system according to any one of Clauses 1 to 6, wherein one layer is an intermediate layer and, as part of applying a nonlinear mapping function, one or more processors are configured to apply a nonlinear mapping function in the output layer of a machine learning model.

[0159]

[0122] Clause 9. The system according to any one of Clauses 1 to 8, wherein multiple layers include at least one activation function layer configured to apply an activation function to the output of each preceding layer.

[0160]

[0123] Clause 10. The system according to Clause 9, wherein at least one activation function layer comprises at least one intermediate layer, and the activation function of at least one intermediate layer comprises at least one of a normalized linear unit (ReLU), a parametric normalized linear unit (PReLU), or an exponential linear unit (ELU).

[0161]

[0124] Clause 11. The system according to Clause 10, wherein the activation function of at least one intermediate layer includes PReLU.

[0162]

[0125] Clause 12. The system according to any one of Clauses 9 to 11, wherein at least one activation function layer includes an output layer, and the activation function of the output layer includes at least one of a normalized linear unit (ReLU), a parametric normalized linear unit (PReLU), and an exponential linear unit (ELU), a sigmoid, an α×sigmoid, or a soft plus.

[0163]

[0126] Clause 13. The system according to Clause 12, wherein the activation function of the output layer includes PReLU.

[0164]

[0127] Clause 14. The system described in any of Clauses 1 to 13, wherein the output of one layer does not include disparity data, and one or more processors are configured to train a machine learning model without determining the disparity data.

[0165]

[0128] Clause 15. The system according to any one of Clauses 1 to 14, wherein the nonlinear mapping function is a first nonlinear mapping function among a plurality of nonlinear mapping functions, the depth data is the first depth data, and one or more processors are further configured to determine changes in the environment and, based on the changes in the environment, apply a second nonlinear mapping function among the plurality of nonlinear mapping functions to the output of one of the plurality of layers in order to generate second depth data, and to further train a trained machine learning model based on the second depth data.

[0166]

[0129] Clause 16. The system according to Clause 15, wherein one or more processors are configured to stop applying a first nonlinear mapping function among the multiple nonlinear mapping functions as part of applying a second nonlinear mapping function among the multiple nonlinear mapping functions to the output of one of the multiple layers.

[0167]

[0130] Clause 17. A method comprising: running a machine learning model, comprising multiple layers, on image data captured by multiple cameras; applying a nonlinear mapping function to the output of one of the multiple layers in order to generate depth data; and training a machine learning model on the depth data in order to generate a trained machine learning model.

[0168]

[0131] Clause 18. The method according to Clause 17, wherein the slope of the nonlinear mapping function is based on at least one of the loss function, the input depth of the nonlinear mapping function, or the probability distribution of the output depth of the nonlinear mapping function.

[0169]

[0132] Clause 19. The method according to Clause 18, wherein the nonlinear mapping function is associated with a mean absolute relative error loss function, and the slope of the nonlinear mapping function is at least one of the following: a) increases with the input depth of the nonlinear mapping function, or b) decreases with the higher probability density of the output depth of the nonlinear mapping function.

[0170]

[0133] Clause 20. The method of any of Clauses 17 to 19, further comprising controlling the operation of a device based on a trained machine learning model.

[0171]

[0134] Clause 21. The method of Clause 20, wherein the device includes a vehicle or robot, and controlling the operation of the vehicle or robot includes navigating the vehicle or robot within an environment.

[0172]

[0135] Clause 22. The method according to any one of Clauses 17 to 21, wherein the plurality of cameras includes at least three cameras, and each of the at least three cameras has a different field of view.

[0173]

[0136] Clause 23. The method according to any of Clauses 17 to 22, wherein one layer is an output layer and the application of a nonlinear mapping function is applied to the output of a machine learning model.

[0174]

[0137] Clause 24. The method according to any of Clauses 17 to 22, wherein one layer is an intermediate layer and the application of a nonlinear mapping function is applied to the output layer of the machine learning model.

[0175]

[0138] Clause 25. The method according to any one of Clauses 17 to 24, wherein the multiple layers include at least one activation function layer configured to apply an activation function to the output of each preceding layer.

[0176]

[0139] Clause 26. The method according to Clause 25, wherein at least one activation function layer comprises at least one intermediate layer, and the activation function of at least one intermediate layer comprises at least one of a normalized linear unit (ReLU), a parametric normalized linear unit (PReLU), or an exponential linear unit (ELU).

[0177]

[0140] Clause 27. The method according to Clause 26, wherein the activation function of at least one intermediate layer includes PReLU.

[0178]

[0141] Clause 28. The method according to any one of Clauses 25 to 27, wherein at least one activation function layer includes an output layer, and the activation function of the output layer includes at least one of a normalized linear unit (ReLU), a parametric normalized linear unit (PReLU), and an exponential linear unit (ELU), a sigmoid, an α×sigmoid, or a soft plus.

[0179]

[0142] Clause 29. The method according to Clause 28, wherein the activation function of the output layer includes PReLU.

[0180]

[0143] Clause 30. The method of any of Clauses 17 to 29, wherein the output of one layer does not include disparity data, and training a machine learning model based on depth data is trained without determining disparity data.

[0181]

[0144] Clause 31. The method according to any one of Clauses 17 to 30, further comprising: a nonlinear mapping function being a first nonlinear mapping function among a plurality of nonlinear mapping functions; depth data being the first depth data; and a method being the first depth data; and further training a trained machine learning model based on the second depth data.

[0182]

[0145] Clause 32. The method according to Clause 31, wherein applying a second nonlinear mapping function among a plurality of nonlinear mapping functions to the output of one of a plurality of layers is equivalent to stopping the application of a first nonlinear mapping function among a plurality of nonlinear mapping functions.

[0183]

[0146] Clause 33. A non-temporary computer-readable storage medium including instructions, wherein, when the instructions are executed, one or more processors cause one or more processors to run a machine learning model, comprising multiple layers, on image data captured by multiple cameras; to apply a nonlinear mapping function to the output of one of the multiple layers in order to generate depth data; and to train the machine learning model on the depth data in order to generate a trained machine learning model.

[0184]

[0147] Clause 34. A system comprising: means for running a machine learning model, which includes multiple layers, on image data captured by multiple cameras; means for applying a nonlinear mapping function to the output of one of the multiple layers in order to generate depth data; and means for training a machine learning model on the depth data in order to generate a trained machine learning model.

[0185]

[0148] Clause 35. A system comprising a memory configured to store image data captured by a plurality of cameras, and one or more processors communicatively coupled to the memory, wherein one or more processors are configured to run a machine learning model on the image data, the machine learning model comprising a plurality of layers, and to apply a nonlinear mapping function to the output of one of the plurality of layers in order to predict depth data.

[0186]

[0149] Clause 36. The system according to Clause 35, wherein one or more processors are further configured to use depth data in at least one of an advanced driver assistance system, a robot, an augmented reality device, or a virtual reality device.

[0187]

[0150] In some cases, it should be noted that any particular action or event of any of the techniques described herein may be performed in different sequences, added, merged, or completely excluded (for example, not all actions or events described may be necessary for the practice of the technique). Furthermore, in some particular cases, the actions or events may not be performed sequentially, but rather concurrently, for example, through multithreading, interrupt handling, or across multiple processors.

[0188]

[0151] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium, or transmitted through a computer-readable medium, or executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium including any medium that facilitates the transfer of computer programs from one place to another according to a communication protocol, for example. Thus, the computer-readable medium may generally correspond to (1) a non-transient tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described herein. A computer program product may include a computer-readable medium.

[0189]

[0152] As an example, and not an limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is also appropriately referred to as computer-readable media. For example, if instructions are transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but instead refer to non-temporary tangible storage media. The terms "disk" and "disc" as used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using a laser. Combinations of the above should also be included within the scope of computer-readable media.

[0190]

[0153] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the terms “processor” and “processing circuit” as used herein may refer to any of the structures described above or any other structure suitable for implementing the techniques described herein. In addition, in some embodiments, the functionality described herein may be provided in dedicated hardware modules and / or software modules configured for encoding and decoding, or incorporated into a composite codec. It is also possible to fully implement these techniques in one or more circuits or logic elements.

[0191]

[0154] The techniques of the disclosed herein can be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units have been described in this disclosure to highlight the functional aspects of devices configured to perform the disclosed techniques, but these do not necessarily require implementation by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit, or provided by a collection of interoperable hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0192]

[0155] Various embodiments have been described. These and other embodiments are within the scope of the following claims.

Claims

1. It is a system, A memory configured to store image data captured by multiple cameras, The memory comprises one or more processors that are communicably coupled to the memory, and the one or more processors A machine learning model, comprising multiple layers, is executed on the aforementioned image data. To generate depth data, a nonlinear mapping function is applied to the output of one of the layers mentioned above. A system configured to train a machine learning model based on the depth data in order to generate a trained machine learning model.

2. The system according to claim 1, wherein the slope of the nonlinear mapping function is based on at least one of the loss function, the input depth of the nonlinear mapping function, or the probability distribution of the output depth of the nonlinear mapping function.

3. The system according to claim 2, wherein the nonlinear mapping function is associated with a mean absolute relative error loss function, and the slope of the nonlinear mapping function is at least one of the following: a) increases with the input depth of the nonlinear mapping function, or b) decreases with the higher probability density of the output depth of the nonlinear mapping function.

4. The system according to claim 1, wherein the one or more processors are further configured to control the operation of the device based on the trained machine learning model.

5. The system according to claim 4, wherein the device includes a vehicle or robot, and as part of controlling the operation of the vehicle or robot, the one or more processors are configured to navigate the vehicle or robot in an environment.

6. The system according to claim 1, further comprising the plurality of cameras, wherein the plurality of cameras includes at least three cameras, the at least three cameras are configured to capture the image data, and each of the at least three cameras has a different field of view.

7. The system according to claim 1, wherein one of the layers is an output layer, and as part of applying the nonlinear mapping function, one or more processors are configured to apply the nonlinear mapping function to the output of the machine learning model.

8. The system according to claim 1, wherein the one layer is an intermediate layer, and as part of applying the nonlinear mapping function, the one or more processors are configured to apply the nonlinear mapping function in the output layer of the machine learning model.

9. The system according to claim 1, wherein the plurality of layers include at least one activation function layer configured to apply an activation function to the output of each preceding layer.

10. The system according to claim 9, wherein the at least one activation function layer includes at least one intermediate layer, and the activation function of the at least one intermediate layer includes at least one of a normalized linear unit (ReLU), a parametric normalized linear unit (PReLU), or an exponential linear unit (ELU).

11. The system according to claim 10, wherein the activation function of the at least one intermediate layer includes PReLU.

12. The system according to claim 11, wherein the at least one activation function layer includes an output layer, and the activation function of the output layer includes at least one of a normalized linear unit (ReLU), a parametric normalized linear unit (PReLU), an exponential linear unit (ELU), a sigmoid, an α×sigmoid, or a soft plus.

13. The system according to claim 12, wherein the activation function of the output layer includes a PReLU.

14. The system according to claim 1, wherein the output of one layer does not include disparity data, and the one or more processors are configured to train the machine learning model without determining the disparity data.

15. The nonlinear mapping function is the first nonlinear mapping function among a plurality of nonlinear mapping functions, the depth data is the first depth data, and the one or more processors are Determine environmental changes, Based on the changes in the environment, in order to generate second depth data, the second nonlinear mapping function among the plurality of nonlinear mapping functions is applied to the output of one of the plurality of layers, The system according to claim 1, further configured to further train the trained machine learning model based on the second depth data.

16. The system according to claim 15, wherein, as part of applying the second nonlinear mapping function among the plurality of nonlinear mapping functions to the output of one of the plurality of layers, one or more processors are configured to stop applying the first nonlinear mapping function among the plurality of nonlinear mapping functions.

17. It is a method, This involves running a machine learning model, which includes multiple layers, on image data captured by multiple cameras. To generate depth data, a nonlinear mapping function is applied to the output of one of the multiple layers, A method comprising training a machine learning model on depth data in order to generate a trained machine learning model.

18. The method according to claim 17, wherein the slope of the nonlinear mapping function is based on at least one of the loss function, the input depth of the nonlinear mapping function, or the probability distribution of the output depth of the nonlinear mapping function.

19. The method according to claim 18, wherein the nonlinear mapping function is associated with a mean absolute relative error loss function, and the slope of the nonlinear mapping function is at least one of the following: a) increases with the input depth of the nonlinear mapping function, or b) decreases with the higher probability density of the output depth of the nonlinear mapping function.

20. The method according to claim 17, further comprising controlling the operation of the device based on the trained machine learning model.

21. The method according to claim 20, wherein the device includes a vehicle or a robot, and controlling the operation of the vehicle or the robot includes navigating the vehicle or the robot in an environment.

22. The method according to claim 17, wherein the plurality of cameras includes at least three cameras, and each of the at least three cameras has a different field of view.

23. The method according to claim 17, wherein the one layer is an output layer, and applying the nonlinear mapping function includes applying the nonlinear mapping function to the output of the machine learning model.

24. The method according to claim 17, wherein the one layer is an intermediate layer, and applying the nonlinear mapping function includes applying the nonlinear mapping function in the output layer of the machine learning model.

25. The method according to claim 17, wherein the plurality of layers include at least one activation function layer configured to apply an activation function to the output of each preceding layer.

26. The method according to claim 25, wherein the at least one activation function layer includes at least one intermediate layer, and the activation function of the at least one intermediate layer includes at least one of a normalized linear unit (ReLU), a parametric normalized linear unit (PReLU), or an exponential linear unit (ELU).

27. The method according to claim 26, wherein the activation function of the at least one intermediate layer includes PReLU.

28. The method according to claim 25, wherein the at least one activation function layer includes an output layer, and the activation function of the output layer includes at least one of a normalized linear unit (ReLU), a parametric normalized linear unit (PReLU), an exponential linear unit (ELU), a sigmoid, an α×sigmoid, or a soft plus.

29. The method according to claim 25, wherein the activation function of the output layer includes PReLU.

30. The method according to claim 17, wherein the output of one layer does not include disparity data, and training the machine learning model based on the depth data includes training the machine learning model without determining the disparity data.

31. The nonlinear mapping function is the first nonlinear mapping function among a plurality of nonlinear mapping functions, the depth data is the first depth data, and the method is Determining environmental changes, Based on the changes in the environment, in order to generate second depth data, the second nonlinear mapping function among the plurality of nonlinear mapping functions is applied to the output of one of the plurality of layers, The method according to claim 17, further comprising training the trained machine learning model based on the second depth data.

32. The method according to claim 31, wherein applying the second nonlinear mapping function among the plurality of nonlinear mapping functions to the output of one of the plurality of layers includes stopping the application of the first nonlinear mapping function among the plurality of nonlinear mapping functions.

33. It is a system, A machine learning model, comprising multiple layers, is applied to image data captured by multiple cameras, and the means for executing the machine learning model are provided. Means for applying a nonlinear mapping function to the output of one of the multiple layers in order to generate depth data, A system comprising means for training a machine learning model based on depth data in order to generate a trained machine learning model.

34. It is a system, A memory configured to store image data captured by multiple cameras, The memory comprises one or more processors that are communicably coupled to the memory, and the one or more processors A machine learning model, comprising multiple layers, is executed on the aforementioned image data. A system configured to apply a nonlinear mapping function to the output of one of the multiple layers in order to predict depth data.

35. The system according to claim 34, wherein one or more processors are further configured to use the depth data in at least one of an advanced driver assistance system, a robot, an augmented reality device, or a virtual reality device.

Images