Deep learning model for high resolution prediction

Abstract

Systems and techniques for deep learning model optimization are provided. An example process can include generating a plurality of prediction outputs associated with a plurality of output channels corresponding to a multi-channel prediction target based on processing data using a neural network model; determining that a prediction output from the plurality of prediction outputs has a value outside a quantization range and one or more remaining prediction outputs from the plurality of prediction outputs have respective values within the quantization range; clamping the predicted output based on the quantization range; and generating a single channel output based on the clamped prediction output and the one or more remaining prediction outputs.

Description

Technical Field

[0001] The present disclosure as a whole relates to neural network models. For example, aspects of the present disclosure relate to systems and techniques for deep learning models for high-resolution prediction. Background Art

[0002] Deep learning models can be used to perform a variety of tasks, such as detection and / or recognition (e.g., scene or object detection and / or recognition), depth estimation, pose estimation, image reconstruction, classification, three-dimensional (3D) modeling, dense regression tasks, data compression and / or decompression, image processing, etc. Deep learning models can be general and can achieve high-quality results in a variety of tasks. However, although deep learning models can be general and accurate, these models are typically large and slow, and often have high memory requirements and computational costs. In many cases, the computational complexity of these models can be high, and these models can be difficult to train. Summary of the invention

[0003] Systems and techniques are described herein that provide deep learning models (eg, with ordered multi-channel output arrangements) for high-resolution prediction. In one illustrative example, a process for wireless communication at a user equipment (UE) is provided.

[0004] According to at least one example, a method for implementing a deep learning model is provided. The method implemented by the processor includes: based on processing data using a neural network model, generating multiple prediction outputs associated with multiple output channels corresponding to a multi-channel prediction target; determining that a prediction output from the multiple prediction outputs has a value outside a quantization range, and one or more remaining prediction outputs from the multiple prediction outputs have corresponding values ​​within the quantization range; clamping the prediction output based on the quantization range; and generating a single channel output based on the clamped prediction output and the one or more remaining prediction outputs.

[0005] In another example, a device for implementing a deep learning model is provided, the device comprising at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: generate a plurality of prediction outputs associated with a plurality of output channels corresponding to a multi-channel prediction target based on processing data using a neural network model; determine that a prediction output from the plurality of prediction outputs has a value outside a quantization range, and one or more remaining prediction outputs from the plurality of prediction outputs have corresponding values ​​within the quantization range; clamp the prediction output based on the quantization range; and generate a single channel output based on the clamped prediction output and the one or more remaining prediction outputs.

[0006] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which instructions, when executed by one or more processors, cause the one or more processors to: generate a plurality of prediction outputs associated with a plurality of output channels corresponding to a multi-channel prediction target based on processing data using a neural network model; determine that a prediction output from the plurality of prediction outputs has a value outside a quantization range and one or more remaining prediction outputs from the plurality of prediction outputs have corresponding values ​​within the quantization range; clamp the prediction output based on the quantization range; and generate a single channel output based on the clamped prediction output and the one or more remaining prediction outputs.

[0007] In another example, a device for implementing a deep learning model is provided. The device includes: a component for generating multiple prediction outputs associated with multiple output channels corresponding to a multi-channel prediction target based on processing data using a neural network model; a component for determining that a prediction output from the multiple prediction outputs has a value outside a quantization range, and one or more remaining prediction outputs from the multiple prediction outputs have corresponding values ​​within the quantization range; a component for clamping the prediction output based on the quantization range; and a component for generating a single channel output based on the clamped prediction output and the one or more remaining prediction outputs.

[0008] In some aspects, an apparatus may be a mobile device (e.g., a mobile phone or so-called "smart phone" or other mobile device), a camera (e.g., a digital camera, an Internet Protocol (IP) camera, a camera of a mobile device), a smart wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a multi-camera system, a vehicle or a computing device or system of a vehicle, or other device, or may be a portion thereof. In some aspects, the apparatus includes one or more cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus described above may include one or more sensors.

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] The foregoing and other features and embodiments will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Exemplary embodiments of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1 is a simplified block diagram illustrating an example electronic device according to some examples of the present disclosure;

[0013] Figure 2 is a flow chart illustrating an example process for optimizing a deep learning model according to some examples of the present disclosure;

[0014] Figure 3 is a table illustrating example multi-channel outputs derived from target outputs in a deep learning model according to some examples of the present disclosure;

[0015] Figure 4A is a diagram illustrating example representations of digits according to some examples of the present disclosure;

[0016] Figure 4B is a diagram illustrating an example of an ordered multi-channel output representation according to some examples of the present disclosure;

[0017] Figure 5A is a chart illustrating example predicted values ​​and ground truth values ​​across multiple channels according to some examples of the present disclosure;

[0018] Figure 5B is a graph illustrating examples of clamped integral values ​​determined for a primary loss function according to some examples of the present disclosure;

[0019] Figure 6 illustrates example depth estimation output generated by a deep learning model according to some examples of the present disclosure;

[0020] Figure 7 is a flowchart illustrating an example process for optimizing a deep learning model according to some examples of the present disclosure; and

[0021] Figure 8 is a diagram illustrating an example system architecture for implementing certain aspects described herein. DETAILED DESCRIPTION

[0022] Some aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes in order to provide a thorough understanding of each embodiment of the application. However, it will be apparent that various embodiments can be put into practice without these specific details. Each drawing and description are not intended to be restrictive.

[0023] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the subsequent description of the exemplary embodiments will provide an enabling description for implementing the exemplary embodiments to those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the scope of the present application as set forth in the appended claims.

[0024] As described above, deep learning models can be used to perform a variety of tasks, such as but not limited to detection and / or recognition (e.g., scene or object detection and / or recognition, facial detection and / or recognition, etc.), depth estimation, pose estimation, image reconstruction, classification, three-dimensional (3D) modeling, dense regression tasks, data compression and / or decompression, and image processing. In addition, deep learning models can be general and can achieve high-quality results in a variety of tasks. However, although deep learning models can be general and accurate, these models are typically large, slow, and typically have high memory requirements and computational costs. In many cases, the computational complexity of these models may be high, and these models may be difficult to train.

[0025] In some cases, 8-bit quantization can be used for deep learning model optimization. For example, 8-bit quantization can be used to quantize the weights and / or parameters (e.g., feature maps, etc.) of the deep learning model in order to reduce the size and inference time of the model. However, the prediction resolution of the deep learning model optimized by 8-bit quantization is generally not enough for dense regression tasks, such as but not limited to depth estimation, posture estimation, etc. For example, the prediction resolution (e.g., the resolution of the predicted value or feature map) of the deep learning model optimized by 8-bit quantization may be limited to 256 values, which may be too low for dense regression tasks. In some cases, the deep learning model can be optimized by using two 8-bit numbers to represent 16-bit numbers. In such cases, one 8-bit number in the 8-bit number can be used for higher bits, and another 8-bit number can be used for lower bits.

[0026] In some cases, this approach may result in discontinuous behavior on the lower bit when carrying (which may be difficult to model by a deep learning model), may conflict with the inference of the deep learning model (e.g., the model may have difficulty following this pattern during prediction), and may increase the difficulty of training the model. For example, when two 8-bit numbers are used to represent a 16-bit number, the value associated with the lower bit (e.g., the value in the vector corresponding to the 8-bit number for the lower bit) may change from zero to its maximum value and quickly drop back to zero (and at some points, the target value associated with the lower bit may be very close to the target value associated with the higher bit). The maximum value associated with the lower bit does not always increase or continue to increase as the target value associated with the lower bit increases. This discontinuous behavior is difficult to implement by a deep learning model during inference.

[0027] Systems, devices, processes (also referred to as methods), and computer-readable media (collectively, "systems and techniques") for deep learning model optimization (e.g., optimization of a neural network or multiple neural networks) are described herein. In some examples, the systems and techniques described herein may implement an ordered multi-channel output arrangement for deep learning model optimization (e.g., multiple outputs (such as predicted outputs) may be distributed to multiple channels therein). The multi-channel outputs may be used to optimize a deep learning model (e.g., one or more neural networks). For example, each multi-channel output may include a value vector generated based on the number of channels and a target output associated with the multi-channel output. The values ​​included in the value vector may be distributed across different channels associated with the multi-channel output.

[0028] In some cases, the systems and techniques described herein can implement an 8-bit quantized deep learning model with an ordered multi-channel output arrangement. The ordered multi-channel output arrangement can increase the output resolution of the 8-bit quantized model using a deep learning model trained on floating point numbers. During model training, real data predictions can be decomposed into multi-channel representations for determining supervised losses. Then, clamped integrals can be used to recover high-precision outputs.

[0029] For example, during model training, model predictions (e.g., multi-channel outputs) may be quantized and compared to quantized real data predictions to determine a supervised loss. The quantized model predictions may be clamped to prevent the quantized prediction values ​​from falling outside the quantization range. In some cases, when the quantized prediction values ​​are outside the quantization range, there may not be any additional gradients, so the quantized prediction values ​​may be clamped to ensure that the quantized prediction values ​​are within the quantization range and improve the performance of the model. The clamped values ​​may then be compared to the quantized real data predictions to determine a supervised loss for the clamped values. The supervised loss may be used to train a deep learning model, and the clamped values ​​may be used to recover high precision and / or resolution output for a single channel.

[0030] In some cases, the systems and techniques described herein may use the following Equation 1 to perform a clamped integration for each number (eg, each prediction from a multi-channel output):

[0031]

[0032] where v i is the value for channel i, and N is the number of channels. In some aspects, the ordered multi-channel output arrangement described herein can split or decompose each target output (which can be a floating point number) from a deep learning model into multiple 8-bit numbers. In some examples, multiple 8-bit numbers can represent floating point outputs. The target outputs can be distributed to each channel from 0 to 1 in ascending order.

[0033] Examples of systems and techniques for processing data described herein are Figures 1 to 8 exemplified in and described below.

[0034] Figure 1 is a diagram of an example electronic device 100 illustrating some examples according to the present disclosure. The electronic device 100 may implement the systems and techniques disclosed herein. For example, the electronic device 100 may perform the deep learning model training and optimization systems and techniques described herein. The electronic device 100 may perform various tasks and operations, such as, but not limited to, extended reality tasks and operations (e.g., tracking, mapping, positioning, content rendering, pose estimation, object detection / recognition, etc.), image / video processing and / or post-processing, data processing and / or post-processing, computer graphics, machine vision, depth estimation, neural network model training, neural network model optimization, object modeling, multimedia rendering and / or synthesis, object detection, object recognition, positioning, scene recognition, and / or any other data processing tasks, effects, and / or calculations.

[0035] In some examples, the electronic device 100 may perform supervised training for a deep learning model. The deep learning model may be configured to perform one or more tasks, such as, but not limited to, dense regression tasks (e.g., depth estimation, pose estimation, etc.). The deep learning model may be based on a model architecture, such as a convolutional neural network (CNN) architecture, a recurrent neural network (RNN) model architecture, and / or any other neural network architecture. In some examples, the electronic device 100 may optimize the deep learning model, as further described herein. For example, in some cases, the electronic device 100 may optimize the deep learning model by implementing an ordered multi-channel output arrangement, as further described herein.

[0036] exist Figure 1 In the example shown, the electronic device 100 includes one or more image sensors 102, one or more inertial sensors 104 (e.g., one or more inertial measurement units), one or more other sensors 106 (e.g., one or more radio detection and ranging (radar) sensors, light detection and ranging (LIDAR) sensors, acoustic / sound sensors, infrared (IR) sensors, magnetometers, touch sensors, laser rangefinders, light sensors, proximity sensors, motion sensors, active pixel sensors, machine vision sensors, ultrasonic sensors, etc.), storage 108, computing components 110, and machine learning engines 120. In some cases, the electronic device 100 may include additional software and / or software engines, such as extended reality (e.g., virtual reality, augmented reality, mixed reality, etc.) engines, image processing engines, rendering engines, depth estimation engines, pose estimation engines, etc.

[0037] Figure 1 The components 102 to 120 shown are non-limiting examples provided for purposes of illustration and explanation. In other examples, the electronic device 100 may include more than Figure 1 For example, in some cases, electronic device 100 may include more, fewer, and / or different components than those shown in FIG. Figure 1 One or more display devices, one or more other processing engines, one or more receivers (e.g., global positioning system, global navigation satellite system, etc.), one or more communication devices (e.g., radio frequency (RF) interface and / or any other wireless / wired communication receiver / transmitter), one or more other hardware components, and / or one or more other software and / or hardware components not shown. Figure 8 An example architecture and example hardware components that may be implemented by electronic device 100 are further described.

[0038] Figure 1 The one or more image sensors 102 in may include any number of image sensors. For example, in some cases, the one or more image sensors 102 may include a single image sensor. In other cases, the one or more image sensors 102 may include two image sensors in a dual-camera implementation, or more than two image sensors in other multi-camera implementations. The electronic device 100 may be part of or implemented by a single computing device or multiple computing devices. In some examples, the electronic device 100 may be part of an electronic device (or multiple electronic devices), such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), a phone system (e.g., a smart phone, a cellular phone, a conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a smart TV, a display device, a game console, a video streaming device, an IoT (Internet of Things) device, a smart wearable device (e.g., a head-mounted display (HMD), smart glasses, etc.), or any other suitable electronic device.

[0039] In some implementations, one or more image sensors 102, one or more inertial sensors 104, other sensors 106, storage 108, computing components 110, and machine learning engine 120 may be part of the same computing device. For example, in some cases, one or more image sensors 102, one or more inertial sensors 104, one or more other sensors 106, storage 108, computing components 110, and machine learning engine 120 may be integrated into a smartphone, a laptop, a tablet, a smart wearable device, a game console, an advanced driver assistance system (ADAS) for a vehicle, a server, a smart TV, a camera system, and / or any other computing device. In other implementations, one or more image sensors 102, one or more inertial sensors 104, other sensors 106, storage 108, computing components 110, and machine learning engine 120 may be part of two or more separate computing devices. For example, in some cases, some of components 102 to 120 may be part of or implemented by one computing device, and the remaining components may be part of or implemented by one or more other computing devices.

[0040] The one or more image sensors 102 may include any image and / or video sensor or capture device, such as a digital camera sensor, a video camera sensor, a smartphone camera sensor, an image / video capture device on an electronic device (such as a television or computer), a camera, etc. In some cases, the one or more image sensors 102 may be part of a multi-camera system or computing device (such as an extended reality (XR) device (e.g., HMD, smart glasses, etc.), a digital camera system, a smartphone, a smart television, a gaming console, etc.). The one or more image sensors 102 may capture image and / or video content (e.g., raw image and / or video data) that may be processed by the computing component 110.

[0041] In some examples, one or more image sensors 102 may capture image data and generate frames based on the image data, and / or provide the image data or frames to computing component 110 for processing. A frame may include a video frame or a static image in a video sequence. A frame may include an array of pixels representing a scene. For example, a frame may be: a red-green-blue (RGB) frame having red, green, and blue color components per pixel; a luminance, redness, blueness (YCbCr) frame having a luminance component and two chrominance (color) components (redness and blueness) per pixel; or any other suitable type of color or monochrome picture.

[0042] The electronic device 100 may optionally include one or more inertial sensors 104. The one or more inertial sensors 104 may detect certain forces and angular rates of the electronic device 100. In some cases, the one or more inertial sensors 104 may detect the orientation of the electronic device 100. The one or more inertial sensors 104 may generate linear acceleration measurements, rotation rate measurements, and / or heading measurements. In some examples, the one or more inertial sensors 104 may be used to measure the pitch, roll, and yaw of the electronic device 100.

[0043] The electronic device 100 may optionally include one or more other sensors 106. In some examples, the one or more other sensors 106 may detect and generate other measurements used by the electronic device 100. In some cases, the computing component 110 may use data and / or measurements from the one or more image sensors 102, the one or more inertial sensors 104, and / or the one or more other sensors 106 to perform various tasks, such as, but not limited to, locating the electronic device 100 in three-dimensional (3D) space, tracking the posture of the electronic device 100, etc. In other examples, the electronic device 100 may also include other sensors, such as magnetometers, acoustic / sound sensors, IR sensors, machine vision sensors, smart scene sensors, radar sensors, LIDAR sensors, light sensors, etc.

[0044] The storage 108 may be any storage device for storing data. In addition, the storage 108 may store data from any of the components of the electronic device 100. For example, the storage 108 may store data from one or more image sensors 102 (e.g., image or video data), data from one or more inertial sensors 104 (e.g., measurements), data from one or more other sensors 106 (e.g., measurements), data from the computing component 110 (e.g., processing parameters, timestamps, preferences, virtual content, rendered content, scene graphs, tracking and positioning data, object detection data, configurations, XR application data, recognition data, synchronization data, outputs, deep learning models and / or associated data and / or parameters, etc.), and / or data from the machine learning engine 120. In some examples, the storage 108 may include a buffer for storing frames and / or other camera data for processing by the computing component 110.

[0045] The one or more computing components 110 may include a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, a digital signal processor (DSP) 116, and / or an image signal processor (ISP) 118. The computing component 110 may perform various operations such as image enhancement, computer vision, depth estimation, pose estimation, graphics rendering, extended reality (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, etc.), image / video processing, sensor processing, detection (e.g., face detection, object detection, feature detection, pattern detection, scene detection or scene change detection, occlusion detection, etc.), recognition (e.g., text recognition, face recognition, object recognition, feature recognition, pattern recognition, scene recognition, occlusion detection, etc.), machine learning, filtering, object detection, and any of the various operations described herein.

[0046] exist Figure 1 In the example shown, computing component 110 may implement machine learning engine 120. For example, operations for machine learning engine 120 may be implemented by any computing component in computing component 110. Machine learning engine 120 may include one or more neural network models, such as deep learning models, such as the supervised deep learning models described herein. In some examples, computing component 110 may also implement one or more other processing engines.

[0047] Although electronic device 100 is shown as including certain components, those skilled in the art will appreciate that electronic device 100 may include more than Figure 1 For example, in some instances, the electronic device 100 may further include one or more memory devices (e.g., RAM, ROM, cache, etc.), one or more network interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, and / or Figure 1 Other hardware or processing devices not shown. Figure 8 Illustrative examples of computing devices and hardware components that may be implemented using electronic device 100 are described.

[0048] Furthermore, reference to any of the components of the electronic device 100 in the singular should not be interpreted as limiting the number of such components implemented by the electronic device 100 to one. For example, reference to a processor in the singular should not be interpreted as limiting the number of processors implemented by the electronic device 100 to one. Similarly, reference to an image sensor or camera in the singular should not be interpreted as limiting the number of image sensors or cameras implemented by the electronic device 100 to one.

[0049] Figure 2is a flow chart illustrating an example process 200 for optimizing a deep learning model. In this example, the process 200 can be used to implement a quantized model that performs higher accuracy and / or resolution predictions. For example, the process 200 can implement an 8-bit quantized deep learning model with an ordered multi-channel output arrangement that achieves prediction results with a higher resolution than other 8-bit quantized deep learning models.

[0050] At box 204, the electronic device 100 may decompose each target output from the deep learning model 202 into a multi-channel output. In an illustrative example, the deep learning model 202 may include a neural network or multiple neural networks (e.g., CNN, RNN, or other types of neural networks). In some examples, each multi-channel output may include N channels. The N channels may include two or more channels. For example, in some cases, the N channels may include 8 channels. In some examples, each target output may include a floating point number. In some cases, the electronic device 100 may decompose each target output from the deep learning model 202 into multiple 8-bit numbers.

[0051] At block 206, the electronic device 100 may apply a clamped integration to each output / prediction from the multi-channel output. In some examples, the electronic device 100 may perform the clamped integration using equation 1 described above. For example, in some cases, the electronic device 100 may perform the clamped integration for each output from the multi-channel output as follows: where v i is the value for channel i, and N is the number of channels.

[0052] At block 208, the electronic device 100 may restore the single channel output based on the clamped integral. For example, the electronic device 100 may use the clamped integral to ensure that the output value from the multi-channel output is within the quantization range. Here, if the output value from the multi-channel output is outside the quantization range, the electronic device 100 may clamp the output value so that the output value falls within the quantization range. For illustration, if the quantization range is from 0 to 1 and the output value is 1.1, the clamped integral may reduce the output value to 1 so that the output value falls within the range of 0 to 1. Similarly, if the quantization range is from 0 to 1 and the output value is -0.1, the clamped integral may adjust the output value to 0 so that the output value falls within the range of 0 to 1.

[0053] The single-channel output may include a single-channel representation of the multi-channel output. The single-channel output may provide a single-channel output with higher accuracy and / or resolution than a single-channel output derived from an 8-bit quantized deep learning model that does not implement a multi-channel output as described herein. In some examples, the electronic device 100 may use clamped integration to recover the single-channel representation of the multi-channel output. For example, in some cases, the electronic device 100 may use the clamped output derived during the clamped integration to generate a single-channel representation of the multi-channel output. In some examples, the electronic device 100 may determine the single-channel representation of the multi-channel output based on the average or mean of the clamped output derived from the multi-channel output using the clamped integration.

[0054] Figure 3 is an example of learning from deep learning models such as Figure 2 202) in the deep learning model 202 shown in the figure. The multi-channel outputs can be used to optimize the deep learning model, as further described herein. In some examples, each multi-channel output may include a value vector generated based on the number of channels and the target output associated with the multi-channel output, as further explained below. The values ​​in the value vector may be distributed across different channels (e.g., channels 304-318) associated with the multi-channel output. The value vector may include a prediction vector that can be used to recover the target value used to generate the value vector. For example, the values ​​in the value vector may be added and divided by the number of channels to recover the target value associated with the value vector.

[0055] The value vector may be determined by multiplying the target value associated with the value vector by the number of channels and dividing the result by the number of channels to determine the multi-channel target distribution number. In some examples, the multi-channel target distribution number may include a floating point number. In some cases, if the multi-channel target distribution number is an integer, the value vector may be determined by splitting the multi-channel target distribution number into a binary digit vector that, when added, equals the multi-channel target distribution number. The number of binary digits in the value vector may be equal to the number of channels determined for the multi-channel output. In addition, the sequence of the distribution of binary digits across channels may include a binary digit having a "1" value followed by a binary digit having a "0" value. For example, if the multi-channel target distribution number is equal to 5 and the number of channels is 8, the value vector may include the following sequence of binary digits: 1, 1, 1, 1, 1, 0, 0, and 0. As shown in this example, the combination of binary digits equals 5 (e.g., 1+1+1+1+1+0+0+0=5), and the sequence of binary digits begins with a "1" value followed by a "0" value.

[0056] If the multi-channel target distribution number used to generate the value vector is not an integer, the value vector may be determined by splitting the multi-channel target distribution number into a vector including one or more binary digits corresponding to the integer portion of the multi-channel target distribution number and a floating point number corresponding to the decimal portion of the multi-channel target distribution number. In addition, the sum of the binary digits and the floating point numbers in the value vector may be equal to the multi-channel target distribution number. For example, if the multi-channel target distribution number is equal to 6.4 and the number of channels is 8, the value vector may include the following sequence of values ​​having binary digits and floating point numbers: 1, 1, 1, 1, 1, 0.4, and 0. As shown in this example, the combination of the binary digits and the floating point numbers is equal to 6.4 (e.g., 1+1+1+1+1+1+0.4+0=6.4). The sequence of values ​​begins with any "1" value, followed by a floating point number "0.4", and followed by any decimal floating point number "0" value.

[0057] The sequence of values ​​may be distributed across channels such that binary digits having a value of "1" are distributed to the first channel or the first few channels, followed by any binary digits having a value of "0". Thus, if the number of channels is 8 and the sequence of values ​​includes binary digits 1, 1, 1, 1, 1, 0, 0, and 0, binary digits having a value of "1" may be distributed to the first five channels, and the remaining binary digits having a value of "0" may be distributed to the remaining last few channels. If the sequence of values ​​includes floating point numbers, the sequence of values ​​may be distributed across channels such that binary digits having a value of "1" are distributed to the first channel or the first few channels, floating point numbers are distributed to the next channel, and binary digits having a value of "0" are distributed to the remaining channels (if any). Thus, in the previous example where the multi-channel target distribution channels equal 6.4, the first six channels are associated with a binary value of "1", the seventh channel is associated with a "0.4" floating point number, and the last channel is associated with a binary value of "0" in the sequence of values.

[0058] A multi-channel output may include a number n of channels, where n represents two or more integer values. Figure 3 In the example shown, each multichannel output includes 8 channels, so n represents the value 8. In addition, Figure 3 The table in includes a target value column 302 as a column of target values ​​and channel columns 304-318 representing columns of channel values. The channel values ​​in the channel columns 304-318 may include quantized values ​​from 0 to 1 for different channels associated with the channel columns 304-318 and may be used to recover the target values ​​in the target value column 302. The quantized value of each multi-channel output may be derived based on the number of channels associated with the multi-channel output and the corresponding target value in the target value column 302. Figure 3In the example shown, the target values ​​in target value column 302 include values ​​from 0 to 0.125, and the channel values ​​include quantized values ​​from 0 to 1. The quantized value of the channel associated with a particular target value from target value column 302 represents the particular target value from target value column 302 and can be used to restore target value column 302.

[0059] As previously described, the quantized values ​​for the channels in channel columns 304-318 may be determined by multiplying the corresponding target values ​​by the number of channels, which in this example is 8. The result of multiplying the corresponding target values ​​by the number of channels may represent a total multi-channel target value for the channel, which may be an integer or a floating point number. The total multi-channel target value for the channel may be used to determine the quantized value for the channel and the sequence of quantized values ​​for distribution across the channels.

[0060] If the total multi-channel target value is not an integer (e.g., if the total multi-channel target value is a floating point number), in order to determine the quantized value for the channel associated with the target value from the target value column 302, the total multi-channel target value can be split into binary digits and floating point numbers. The binary digits and floating point numbers can represent the quantized values ​​for different channels of the target value, and can include values ​​for each channel, and can be summed up to the total multi-channel target value. For example, if the total multi-channel target value is 6.4, the quantized values ​​can include 1, 1, 1, 1, 1, 1, 0.4, and 0. Here, the first six binary digits (e.g., 1, 1, 1, 1, and 1) and the floating point number (e.g., 0.4) represent and sum up to the total multi-channel target value (e.g., 6.4). The remaining binary digits include the value 0 because the first six binary digits and the floating point number as the seventh value sum up to the total multi-channel target value 6.4. Values ​​in a sequence of values ​​including binary digits and floating point numbers (e.g., 1, 1, 1, 1, 1, 1, 0.4, and 0) may be distributed across channels associated with channel columns 304-318, with binary digits equal to 1 being distributed to the first few channels until each binary digit equal to 1 is assigned to a channel, floating point numbers being distributed to the next channel after the last channel associated with binary digit value 1, and the last binary digit equal to 0 being distributed to the remaining channels (e.g., the last channel).

[0061] If the total multi-channel target value is an integer, then in order to determine the quantized value for the channel, the total multi-channel target value can be broken down into binary digits. The binary digits may include a binary digit for each channel and may be summed up to the total multi-channel target value. For example, if the total multi-channel target value is 5, the binary digits may include 1, 1, 1, 1, 1, 0, 0, and 0. Here, the first five binary digits (e.g., 1, 1, 1, 1, and 1) represent and sum up to the total multi-channel target value. The remaining binary digits include the value 0 because the first five binary digits sum up to the total multi-channel target value of 5. The binary digits in the sequence of binary digits (e.g., 1, 1, 1, 1, 1, 0, 0, and 0) may be distributed across the channels associated with the channel columns 304-318, wherein the binary digits equal to 1 are distributed to the first few channels until each binary digit equal to 1 is assigned to a channel, and the binary digits equal to 0 are distributed to the remaining channels.

[0062] For example, the target value is 320 (which is Figure 3 Quantized channel values ​​1, 0, 0, 0, 0, 0, 0, and 0 for the channels corresponding to channel columns 304-318 may be determined based on target value 320 (which is 0.125 as described above in this example) and the number of channels (which is 8 in this example).

[0063] For illustration, quantized channel values ​​1, 0, 0, 0, 0, 0, 0, and 0 may be determined by multiplying the target value 0.125 by 8 channels (equal to 1). The result of multiplying the target value by the number of channels (which is 1 in this example) is itself a binary number. Therefore, to determine the quantized value for the channel associated with the channel columns 304-318, the binary number resulting from multiplying the target value by the number of channels is combined with the seven binary numbers having a value of 0 and corresponding to the remaining eight channels. The quantized channel values ​​are then distributed across the corresponding eight channels in the channel columns 304-318 starting with the first channel (e.g., the channel associated with the channel column 304) and associated with the target value 320. Each quantized value equal to "1" from the quantized channel values ​​is distributed first (e.g., across one or more corresponding channels associated with one or more of the channel columns 304-318). Each quantized value equal to "0" from the quantized channel values ​​is distributed subsequently (e.g., across one or more corresponding channels associated with one or more of the channel columns 304-318).

[0064] For example, a total multi-channel target value as a result of multiplying the target value of 0.125 by 8 channels may be split into binary digits having a value of 1, and then each binary digit having a value of 1 is distributed across the associated channels from channel columns 304-318 starting from the first channel (e.g., the channel associated with channel column 304) until all values ​​of 1 have been distributed. Once all values ​​of 1 have been distributed, if there are any remaining channels without a value, the remaining channels may be assigned a binary digit having a value of 0. If the result of multiplying the target value of 0.125 by 8 channels (e.g., the total multi-channel target value) includes a floating point number, the fractional portion of the floating point number is added to the next channel after the last channel that receives the value of 1. Therefore, since the result of multiplying the target value of 0.125 by 8 channels is 1 in this example, the first channel (e.g., the channel associated with channel column 304) includes a value of 1 and the remaining channels (e.g., the channels associated with channel columns 306-318) include a value of 0.

[0065] The target value 322 (which is 0.7 in this example) is represented by a value of 1 in a first channel (e.g., channel 0 associated with channel column 304) (e.g., channel 0), a value of 1 in a second channel (e.g., channel 1 associated with channel column 306), a value of 1 in a third channel (e.g., channel 2 associated with channel column 308), a value of 1 in a fourth channel (e.g., channel 3 associated with channel column 310), a value of 0 in a fifth channel (e.g., channel 4 associated with channel column 312), a value of 0 in a sixth channel (e.g., channel 5 associated with channel column 314), a value of 0 in a seventh channel (e.g., channel 6 associated with channel column 316), and a value of 0 in an eighth channel (e.g., channel 7 associated with channel column 318). The values ​​of 1, 1, 1, 1, 0, 0, 0, and 0 for the channels corresponding to channel columns 304-318 may be determined based on the target value 322 (e.g., 0.5) and the number of channels (which is 8 in this example). For example, the values ​​1, 1, 1, 1, 0, 0, 0, and 0 can be determined by multiplying the target value 0.5 by 8 channels (equal to 4). The result of multiplying the target value 0.5 by 8 channels is split into binary digits equal to the result of multiplying the target value 0.5 by 8 channels. The binary digits are then distributed across the 8 channels starting from the first channel (e.g., channel 0 associated with channel column 304).

[0066] As previously explained, the result of multiplying the target value of 0.5 by 8 channels is split into binary digits having a value of 1, and each value of 1 is then distributed across channels 304-318 starting from the first channel (e.g., the channel associated with channel column 304) until there are no more values ​​of 1. If the result of multiplying the target value of 0.5 by 8 channels includes a floating point number, the fractional portion of the floating point number is associated with the next channel after the last channel that received the value of 1. Therefore, since in this example, the result of multiplying the target value of 0.5 by 8 channels is 4, the first four channels (e.g., the channels associated with channel columns 304-310) include a value of 1, which makes the four values ​​of 1 total 4 (e.g., the result of multiplying the target value by the number of channels is totaled). Therefore, the remaining channels (e.g., the channels associated with channel columns 312-318) include a value of 0.

[0067] The target value 324 (which is 0.5 in this example) is represented by a binary digit having a value of 1 in a first channel (e.g., channel 0 associated with channel column 304), a binary digit having a value of 1 in a second channel (e.g., channel 1 associated with channel column 306), a binary digit having a value of 1 in a third channel (e.g., channel 2 associated with channel column 308), a binary digit having a value of 1 in a fourth channel (e.g., channel 3 associated with channel column 310), a binary digit having a value of 1 in a fifth channel (e.g., channel 4 associated with channel column 312), a floating point number having a value of 0.6 in a sixth channel (e.g., channel 5 associated with channel column 314), a binary digit having a value of 0 in a seventh channel (e.g., channel 6 associated with channel column 316), and a binary digit having a value of 0 in an eighth channel (e.g., channel 7 associated with channel column 318). The values ​​1, 1, 1, 1, 1, 0.6, 0, and 0 for the channels associated with channel columns 304-318 may be determined based on the target value 324 (e.g., 0.7) and the number of channels, which in this example is 8. For example, the sequence of quantized target values ​​1, 1, 1, 1, 1, 0.6, 0, and 0 may be determined by multiplying the target value 0.7 by 8 channels (equal to 5.6) and splitting the value 5.6 into five binary digits with a value of 1, a floating point number with a value of 0.6, and two binary digits with a value of 0. The sequence of quantized target values ​​is then distributed across the 8 channels starting from the first channel (e.g., the channel associated with channel column 304) until the last channel (e.g., the channel associated with channel column 318).

[0068] To illustrate, the result of multiplying the target value 0.7 by 8 channels (e.g., 5.6) is split into five binary digits, each of which has a value of 1, and then each value of 1 is distributed across the first five channels (e.g., channels associated with channel columns 304-312) until there are no more values ​​of 1. Since, in this example, the result of multiplying the target value 0.7 by 8 channels includes a floating point number (e.g., 5.6), the fractional portion of the floating point number (e.g., 0.6) is added to the subsequent channel after the last channel (e.g., the channel associated with channel column 314) that receives the value of 1. Therefore, since, in this example, the result of multiplying the target value 0.7 by 8 channels is 5.6, the first five channels (e.g., channels 0-4 associated with channel columns 304-312) include values ​​of 1 that add up to 5. The remaining 0.6 value from the 5.6 result is then added to the next channel (which in this example is the channel associated with channel column 314), and then a binary number with a value of 0 is added to the remaining channels, which in this example include the channels associated with channel columns 316-318.

[0069] The target value 326 (which is 0.8 in this example) is represented by a binary digit having a value of 1 in a first channel (e.g., channel 0 associated with channel column 304), a binary digit having a value of 1 in a second channel (e.g., channel 1 associated with channel column 306), a binary digit having a value of 1 in a third channel (e.g., channel 2 associated with channel column 308), a binary digit having a value of 1 in a fourth channel (e.g., channel 3 associated with channel column 310), a binary digit having a value of 1 in a fifth channel (e.g., channel 4 associated with channel column 312), a binary digit having a value of 1 in a sixth channel (e.g., channel 5 associated with channel column 314), a floating point number having a value of 0.4 in a seventh channel (e.g., channel 6 associated with channel column 316), and a binary digit having a value of 0 in an eighth channel (e.g., channel 7 associated with channel column 318). A sequence of quantized target values ​​1, 1, 1, 1, 1, 1, 0.4, and 0 for channels associated with channel columns 304-318 may be determined based on target value 326 (e.g., 0.8) and the number of channels (e.g., 8). For example, a sequence of quantized target values ​​1, 1, 1, 1, 1, 1, 0.4, and 0 may be determined by multiplying the target value 0.8 by 8 channels (equal to 6.4) and splitting the resulting value 6.4 into six binary digits with a value of 1 (equal to 6), a floating point number with a value of 0.6, and two binary digits with a value of 0. The six binary digits with a value of 1, the floating point number with a value of 0.4, and the two binary digits with a value of 0 together equal 6.4. The sequence of quantized target values ​​is then distributed across 8 channels starting from the first channel (e.g., channel 0 associated with channel column 304) until the last channel (e.g., channel 7 associated with channel column 318).

[0070] One way to represent a 16-bit number by two 8-bit numbers is to multiply one 8-bit number by 256 and add the result to the other 8-bit number. This solution may result in discontinuous behavior for the lower 8-bit number when the target value is near the carry bit. Figure 4A 4 is a diagram illustrating an example representation 400 of a 16-bit number. In particular, the representation 400 includes two 8-bit numbers that together represent a 16-bit number. In this example, the representation 400 includes an upper representation 410 and a lower representation 412. The upper representation 410 represents the upper 8 bits, and the lower representation 412 represents the lower 8 bits.

[0071] Figure 4A The graph in FIG. 4 includes an original value 402 predicted by a deep learning model and a target value 404 determined for the deep learning model. In some examples, the target value V may be calculated and / or represented by Equation 2 as follows: V 目标 =(V 高×256+V 低 )÷256 2 Here, V 高 represents the maximum target value, and V 低 Indicates the minimum target value. Figure 4A As shown, the lower bit representation 412 has discontinuous behavior, which may be difficult to model by a deep learning model and difficult to train a model on.

[0072] For example, the vector corresponding to the lower bit representation 412 goes from zero to a maximum value and quickly drops back to zero, such as when a carry is needed for the higher bit (and at some points, the target value associated with the lower bit representation 412 may be very close to the target value associated with the higher bit representation 410). Furthermore, the maximum value associated with the lower bit representation 412 does not always increase or continue to increase as the target value associated with the lower bit representation 412 increases. This discontinuous behavior is difficult to implement by a deep learning model during inference (and therefore difficult to model by a neural network).

[0073] Figure 4B 4 is a diagram illustrating an example of an ordered multi-channel output representation 420. The ordered multi-channel output representation 420 may allow 8-bit numbers to be represented across different channels and may allow for higher resolution output. In addition, the ordered multi-channel output representation 420 may have a more continuous behavior that may be more easily implemented by a deep learning model and simplifies the training of the deep learning model. In some examples, the ordered multi-channel output representation 420 may allow the size of the deep learning model to be reduced and the output speed (e.g., prediction speed) of the deep learning model may be increased.

[0074] In this example, representation 420 includes a multi-channel output. The multi-channel output includes channel outputs 430-438 corresponding to channels 0 through 7. Figure 4B The graph in FIG. 4 includes an original value 422 predicted by a deep learning model and a target value 424 determined for the deep learning model. In some examples, the target value V may be calculated and / or represented by Equation 2 as follows: V 目标 =mean(V0+V1+V2+V3+V4). Here, V 高 represents the maximum target value, and V 低 Indicates the minimum target value. Figure 4BAs shown, the value of each channel may start at zero and may saturate at the maximum value of the value of each channel as the target value increases. In addition, multi-channel output leads to more continuous behavior, which is easier to model (and therefore easier to implement by a deep learning model), and can simplify the training of deep learning models. Multi-channel output can also increase the resolution of the output to a resolution that exceeds that originally achieved by representing 8-bit numbers with a single channel. For example, the ordered multi-channel output representation 420 can utilize an 8-bit quantized deep learning model to achieve higher resolution predictions.

[0075] Figure 5A is a chart illustrating example predicted values ​​and true data values ​​across multiple channels that can be used to determine loss. As shown, the table includes rows 502 of predicted values ​​across channels 510-516 and rows 504 of true data values ​​across channels 510-516. The predicted values ​​in rows 502 for channels 510-516 and the true data values ​​in rows 504 for channels 510-516 can be used to determine a loss 520 used to train a deep learning model.

[0076] For example, in some cases, the predicted values ​​in row 502 for channels 510-516 can be compared to the true data values ​​in row 504 for channels 510-516. The difference between the predicted values ​​in row 502 for channels 510-516 and the true data values ​​in row 504 for channels 510-516 can be used to determine a loss 520. The loss 520 can be used to train a deep learning model to obtain higher accuracy predictions / outputs. In some examples, the loss 520 can be applied to the original model predictions to improve the predictions and the model.

[0077] Figure 5B is a chart illustrating an example of a clamped integral value determined for a main loss function. As shown, the chart includes a row 530 of clamped prediction values ​​for channels 510-516. Although the deep learning model for determining the prediction can be trained by supervised training as previously described, the original model output / prediction may still exceed the desired range of [0,1], which may further reduce the effective resolution representability. The clamped integral can be used to ensure that the predicted values ​​for multiple channels (e.g., channels 510-516) are within the quantization range, which is [0,1] in this example. Therefore, if the predicted value is outside the quantization range, the predicted value can be clamped so that the predicted value falls within the quantization range.

[0078] For illustration, in Figure 5B In the example shown, the quantization range is from 0 to 1. However, Figure 5AAs shown, the predicted values ​​for channels 510 and 516 are outside the range of [0,1]. In particular, the predicted value for channel 510 is 1.1 and thus exceeds the upper limit of the quantization range [0,1], and the predicted value for channel 516 is -0.1 and thus falls below the lower limit of the quantization range [0,1]. On the other hand, Figure 5B , the predicted value for channel 510 has been reduced to 1 by the clamped integral (e.g., from 1.1) and is therefore now within the upper limit of the quantization range, and the predicted value for channel 516 has been increased to 0 by the clamped integral (e.g., from -0.1) and is therefore now within the lower limit of the quantization range. Figure 5B As shown, the clamped integral can adjust the prediction value outside the quantization range so that the adjusted prediction value is within the quantization range.

[0079] In some examples, the clamped integration may be performed using Equation 1 described above. For example, in some cases, the electronic device 100 may perform the clamped integration for one or more predicted values ​​from the multi-channel prediction as follows: where v i is the value for channel i, and N is the number of channels. Clamped integration ensures that the predicted values ​​for channels 510-516 are all within the quantization range, as explained previously.

[0080] The clamped integral value in row 530 may be used to restore a single channel output (e.g., restored value 540). The single channel output may include a single channel representation of a multi-channel prediction value. The single channel output may provide a single channel output with higher accuracy and / or resolution than a single channel output derived from an 8-bit quantized deep learning model that does not implement a multi-channel output as described herein. In some examples, restored value 540 may be used to determine loss 542. In some cases, loss 542 may include a high level of loss based on restored value 540.

[0081] As previously described, the clamped integration value derived during the clamped integration may be used to generate a restored value 540, which in some examples may provide a single-channel representation of the multi-channel prediction. In some examples, the single-channel representation of the multi-channel prediction (e.g., restored value 540) may be determined based on an average or mean of the clamped prediction values ​​in row 530. In other examples, the single-channel representation of the multi-channel prediction (e.g., restored value 540) may be determined based on a different calculation or equation.

[0082] Figure 6Example depth estimation outputs generated by a deep learning model are illustrated. In this example, the depth estimation outputs include a three-dimensional (3D) depth map 602 depicting a face and another 3D depth map 604 depicting a face. 3D depth map 602 represents the depth estimation output by a deep learning model that was not trained using clamped integration. On the other hand, 3D depth map 604 represents the depth estimation output by a deep learning model that was trained using clamped integration.

[0083] like Figure 6 As shown, the 3D depth map 604 generated by the deep learning model trained with clamped integration has a higher resolution than the 3D depth map 602 generated by the deep learning model trained without clamped integration. Therefore, clamped integration can significantly improve the output quality of the deep learning model after quantization of model parameters (e.g., weights and / or feature maps).

[0084] In some examples, the 3D depth map 602 and the 3D depth map 604 may be generated based on the ordered multi-channel output representation as previously described. In some cases, the process for determining the ordered multi-channel output representation may involve only additional decomposition and integration steps for the training phase of the deep learning model. Therefore, the process for determining the ordered multi-channel output representation may be performed during training and easily deployed for any task. In some cases, when used for single image depth reconstruction, the ordered multi-channel output representation process may produce results with higher resolution using an 8-bit quantized deep learning model.

[0085] Figure 7 700 for optimizing a deep learning model. At block 702, process 700 may include generating a plurality of prediction outputs associated with a plurality of output channels corresponding to a multi-channel prediction target based on processing data using a neural network model (e.g., deep learning model 202). In some cases, generating the plurality of prediction outputs may include distributing the plurality of prediction outputs across the plurality of output channels. In some examples, the neural network model includes an 8-bit quantization model. In some cases, the neural network model is a CNN or RNN model. In other cases, the neural network model may have any other neural network architecture.

[0086] In some examples, the plurality of prediction outputs include quantized prediction values. In some cases, the plurality of output channels include 8 channels. In some examples, the quantized prediction output associated with one or more channels from the plurality of output channels represents one or more values ​​within a target range. In some aspects, process 700 may include a single channel associated with a prediction target into the plurality of output channels.

[0087] At block 704, process 700 may include determining that a prediction output from the plurality of prediction outputs has a value outside of a quantization range and that one or more remaining prediction outputs from the plurality of prediction outputs have corresponding values ​​within a quantization range. In some examples, the quantization range may be [0, 1]. For example, in some cases, a maximum value of the quantization range is 1 and a minimum value of the quantization range is 0.

[0088] At block 706, process 700 may include clamping the predicted output based on the quantization range. In some cases, clamping the predicted output based on the quantization range may include modifying the value of the predicted output to fall within the quantization range. For example, clamping the predicted output may include reducing or increasing the value of the predicted output so that the value is within the quantization range. In one illustrative example, if the predicted output is 1.1 and the quantization range is [0, 1], process 700 may include reducing the predicted output (e.g., from 1.1) to 1. As another illustrative example, if the predicted output is -0.1 and the quantization range is [0, 1], process 700 may include increasing the predicted output (e.g., from -0.1) to 0.

[0089] At block 708, process 700 may include generating a single channel output based on the clamped prediction output and the one or more remaining prediction outputs. In some aspects, the clamped prediction output and the one or more remaining prediction outputs may include a quantized prediction value. In some cases, the quantized prediction value includes a sequence of values ​​between 0 and 1. In some examples, the quantized prediction value represents a multi-channel prediction target. In some aspects, process 700 may include restoring the multi-channel prediction target using the quantized prediction value. In some examples, the data may include image data. In some aspects, process 700 may include reconstructing a depth map associated with the image data based at least in part on the single channel output.

[0090] In some aspects, process 700 may include multiplying the multi-channel prediction target by the number of channels in the plurality of output channels. Process 700 may include dividing the result of multiplying the multi-channel prediction target by the number of channels into binary digits. Process 700 may also include distributing binary digits across at least a portion of the plurality of output channels. In some examples, at least the first binary digit in the binary digits has a value of 1, and at least a portion of the plurality of predicted outputs includes binary digits. In some aspects, distributing binary digits across at least a portion of the plurality of output channels may include distributing one or more binary digits with a value of 1 to one or more output channels before distributing one or more additional binary digits with a value of 0 to one or more additional output channels. In some cases, the plurality of predicted outputs may include one or more binary digits and one or more additional binary digits. In some cases, the first binary digit is distributed to the first channel from the plurality of output channels.

[0091] In some aspects, process 700 may include generating a multichannel target distribution value based on the product of the number of channels in the multichannel prediction target and the multiple output channels. In some examples, the multichannel target distribution value may include a floating point number having an integer portion and a fractional portion. In some aspects, process 700 may include dividing the integer portion of the multichannel target distribution value into a binary digit set. Process 700 may include associating the binary digit set with a channel set from multiple output channels. Process 700 may also include associating the value of the fractional portion of the floating point number with the channel from multiple output channels. In some cases, multiple predicted outputs may include the value of the fractional portion of the binary digit set and the floating point number.

[0092] In some aspects, process 700 may include associating one or more remaining channels from the plurality of output channels with one or more additional binary digits. In some cases, each of the one or more additional binary digits may include a value of 0, and each of the binary digits in the set of binary digits may include a value of 1. In some examples, the plurality of predicted outputs include the one or more additional binary digits.

[0093] In some aspects, process 700 may include comparing the plurality of predicted outputs to a plurality of real data predictions associated with the plurality of output channels, and determining a loss associated with the neural network model based on the comparison of the plurality of predicted outputs to the plurality of real data predictions. Process 700 may include training the neural network model based on the loss.

[0094] In some aspects, process 700 may include recovering a multi-channel prediction target based on the single channel output, the clamped prediction output, and / or one or more remaining prediction outputs. Process 700 may include determining a loss associated with the neural network model based on the recovered multi-channel prediction target. Process 700 may also include training the neural network model based on the loss.

[0095] In some examples, the processes described herein (e.g., process 200, process 700, and / or any other process described herein) can be performed by a computing device or apparatus. In one example, process 200 and / or process 700 can be performed by Figure 1 In another example, process 200 and / or process 700 may be performed by an electronic device 100 having Figure 8 The computing system of the computing device architecture 800 shown in FIG. Figure 8 The computing device of the computing device architecture 800 shown may implement Figure 7 The operation and / or this article about Figures 1 to 7 The components and / or operations described by any of .

[0096] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smart watch or other wearable device), a server computer, a self-driving vehicle or a computing device for a self-driving vehicle, a robotic device, a laptop, a smart TV, a camera, and / or any other computing device with resource capabilities for performing the processes described herein (including process 200, process 700, and / or other processes described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0097] Components of a computing device may be implemented in circuits. For example, a component may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0098] Process 200 and process 700 are illustrated as logic flow diagrams, the operations of which represent sequences of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0099] Additionally, process 200, process 700, and / or any other process described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0100] Figure 8 is a diagram illustrating an example of a system for implementing certain aspects of the present disclosure. Specifically, Figure 8 An example of a computing system 800 is illustrated, which may be, for example, any computing device making up a computing system, a camera system, or any components thereof, where the components of the system communicate with each other using a connection 805. The connection 805 may be a physical connection using a bus, or a direct connection into the processor 810, such as in a chipset architecture. The connection 805 may also be a virtual connection, a networked connection, or a logical connection.

[0101] In some examples, computing system 800 is a distributed system, where the functionality described in the present disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some examples, one or more of the described system components represent a number of such components, each performing some or all of the functionality for which the component is described. In some examples, each component may be a physical or virtual device.

[0102] The example system 800 includes at least one processing unit (CPU or processor) 810 and connections 805 that couple various system components including system memory 815, such as read-only memory (ROM) 820 and random access memory (RAM) 825, to the processor 810. The computing system 800 may include a cache 812 of high-speed memory directly connected to, close to, or integrated as part of the processor 810.

[0103] Processor 810 may include any general purpose processor and hardware or software services, such as services 832, 834, and 836 stored in storage device 830, that are configured to control processor 810 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 810 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0104] To enable user interaction, the computing system 800 includes an input device 845 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The computing system 800 may also include an output device 835, which may be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 800. The computing system 800 may include a communication interface 840, which may generally govern and manage user input and system output.

[0105] The communication interface can perform or facilitate receiving and / or sending wired or wireless communications using wired and / or wireless transceivers, including using audio jacks / plugs, microphone jacks / plugs, universal serial bus (USB) ports / plugs, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transmission, Low power (BLE) wireless signal transmission, Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof.

[0106] The communication interface 840 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 800 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the Global Positioning System (GPS) of the United States, the Global Navigation Satellite System (GLONASS) of Russia, the BeiDou Navigation Satellite System (BDS) of China, and the Galileo GNSS of Europe. There is no limitation to operating on any particular hardware arrangement, and thus the underlying features herein may be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0107] The storage device 830 may be a non-volatile and / or non-transitory and / or computer-readable memory device, and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0108] The storage device 830 may include software services, servers, services, etc., which, when the code defining such software is executed by the processor 810, causes the system to perform a function. In some examples, a hardware service that performs a specific function may include a software component stored in a computer-readable medium connected to the necessary hardware components (such as the processor 810, the connection 805, the output device 835, etc.) to perform the function. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media that can store data and does not include carrier waves and / or transient electronic signals that propagate wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer readable media may store thereon code and / or machine executable instructions, which may represent a process, function, subprogram, program, routine, subroutine, module, software package, category, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by transmitting and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be transmitted, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0109] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals including bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0110] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those skilled in the art that embodiments may be practiced without these specific details. For the sake of clarity of explanation, in some cases, the present technology may be presented as including a separate functional block, which includes a device, device assembly, step or routine in a method embodied in a combination of software or hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid obscuring these embodiments in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown without necessary details to avoid confusing each embodiment.

[0111] Individual embodiments may be described above as processes or methods depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams or block diagrams. Although flow charts may describe operations as sequential processes, many operations in the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process may be terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. The process may correspond to a method, function, process, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to a calling function or a main function.

[0112] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. Such instructions may include, for example, instructions and data that enable or configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions in other ways. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions such as assembly language, firmware, source code. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0113] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or plug-in cards. By way of additional examples, such functions may also be implemented on circuit boards between different chips or different processes executed on a single device.

[0114] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0115] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader essence and scope of this specification, the embodiments can be used in any number of environments and applications beyond the environment and application described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in a different order than described.

[0116] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0117] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0118] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0119] Claim language or other language in this disclosure that states "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that states "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language that states "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that states "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0120] Various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above in terms of their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. The technician may implement the described functions in different ways for each specific application, but such specific implementation decisions should not be interpreted as making it depart from the scope of the application.

[0121] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices with multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium including program code, which includes instructions for executing one or more of the methods, algorithms, and / or operations described above when executed. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0122] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein.

[0123] Illustrative aspects of the present disclosure include:

[0124] Aspect 1. A device for implementing a deep learning model, the device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory, the at least one processor being configured to: generate multiple prediction outputs associated with multiple output channels corresponding to a multi-channel prediction target based on processing data using a neural network model; determine that a prediction output from the multiple prediction outputs has a value outside a quantization range, and one or more remaining prediction outputs from the multiple prediction outputs have a corresponding value within the quantization range; clamp the prediction output based on the quantization range; and generate a single-channel output based on the clamped prediction output and the one or more remaining prediction outputs.

[0125] Aspect 2. An apparatus according to Aspect 1, wherein the neural network model comprises an 8-bit quantized model.

[0126] Aspect 3. An apparatus according to any one of aspects 1 to 2, wherein the plurality of prediction outputs comprises quantized prediction values.

[0127] Aspect 4. An apparatus according to any one of Aspects 1 to 3, wherein the at least one processor is configured to: compare the multiple predicted outputs with multiple real data predictions associated with the multiple output channels; determine a loss associated with the neural network model based on the comparison of the multiple predicted outputs with the multiple real data predictions; and train the neural network model based on the loss.

[0128] Aspect 5. An apparatus according to any one of Aspects 1 to 4, wherein the at least one processor is configured to: restore the multi-channel prediction target based on the single-channel output, the clamped prediction output, and at least one of the one or more remaining prediction outputs; determine the loss associated with the neural network model based on the restored multi-channel prediction target; and train the neural network model based on the loss.

[0129] Aspect 6. The apparatus according to any one of aspects 1 to 5, wherein the maximum value of the quantization range is 1, and the minimum value of the quantization range is 0.

[0130] Aspect 7. An apparatus according to any one of Aspects 1 to 6, wherein the clamped prediction output and the one or more remaining prediction outputs include quantized prediction values, wherein the quantized prediction values ​​include a sequence of values ​​between 0 and 1, and wherein the quantized prediction values ​​represent the multi-channel prediction target.

[0131] Aspect 8. The apparatus according to aspect 7, wherein the at least one processor is configured to: restore the multi-channel prediction target using the quantized prediction value.

[0132] Aspect 9. The device according to any one of aspects 1 to 8, wherein the plurality of output channels comprises 8 channels.

[0133] Aspect 10. An apparatus according to any one of aspects 1 to 9, wherein the quantized prediction output associated with one or more channels from the plurality of output channels represents one or more values ​​within a target range.

[0134] Aspect 11. An apparatus according to any one of aspects 1 to 10, wherein to generate the plurality of prediction outputs, the at least one processor is configured to distribute the plurality of prediction outputs across the plurality of output channels.

[0135] Aspect 12. An apparatus according to any one of Aspects 1 to 11, wherein the at least one processor is configured to: multiply the multi-channel prediction target by the number of channels in the multiple output channels; divide the result of multiplying the multi-channel prediction target by the number of channels into binary digits, wherein at least a first binary digit among the binary digits includes a value of 1, and wherein at least a portion of the multiple predicted outputs includes the binary digits; and distribute the binary digits across at least a portion of the multiple output channels, wherein the first binary digit is distributed to a first channel from the multiple output channels.

[0136] Aspect 13. An apparatus according to Aspect 12, wherein in order to distribute the binary digits across at least a portion of the multiple output channels, the at least one processor is configured to distribute one or more binary digits having a value of 1 to one or more output channels before distributing one or more additional binary digits having a value of 0 to one or more additional output channels, wherein the multiple predicted outputs also include the one or more binary digits and the one or more additional binary digits.

[0137] Aspect 14. An apparatus according to any one of Aspects 1 to 13, wherein the at least one processor is configured to: generate a multi-channel target distribution value based on the product of the multi-channel prediction target and the number of channels in the multiple output channels, wherein the multi-channel target distribution value includes a floating-point number having an integer part and a decimal part; divide the integer part of the multi-channel target distribution value into a set of binary digits; associate the set of binary digits with a set of channels from the multiple output channels; and associate the value of the decimal part of the floating-point number with a channel from the multiple output channels, wherein the multiple predicted outputs include the set of binary digits and the value of the decimal part of the floating-point number.

[0138] Aspect 15. An apparatus according to Aspect 14, wherein the at least one processor is configured to: associate one or more remaining channels from the plurality of output channels with one or more additional binary digits, wherein each of the one or more additional binary digits includes a value of 0, wherein each binary digit in the set of binary digits includes a value of 1, and wherein the plurality of predicted outputs also include the one or more additional binary digits.

[0139] Aspect 16. An apparatus according to any one of Aspects 1 to 15, wherein the data comprises image data, and wherein the at least one processor is configured to reconstruct a depth map associated with the image data based at least in part on the single channel output.

[0140] Aspect 17. An apparatus according to any one of aspects 1 to 16, wherein in order to clamp the prediction output based on the quantization range, the at least one processor is configured to modify the value of the prediction output to fall within the quantization range.

[0141] Aspect 18. An apparatus according to any one of aspects 1 to 17, wherein the apparatus comprises a mobile device.

[0142] Clause 19. The apparatus of clause 18, wherein the mobile device comprises a camera.

[0143] Aspect 20. A method for implementing a processor of a deep learning model, the method comprising: generating multiple prediction outputs associated with multiple output channels corresponding to a multi-channel prediction target based on processing data using a neural network model; determining that a prediction output from the multiple prediction outputs has a value outside a quantization range, and one or more remaining prediction outputs from the multiple prediction outputs have a corresponding value within the quantization range; clamping the prediction output based on the quantization range; and generating a single channel output based on the clamped prediction output and the one or more remaining prediction outputs.

[0144] Aspect 21. The processor-implemented method of aspect 20, wherein the neural network model comprises an 8-bit quantized model.

[0145] Aspect 22. A processor-implemented method according to any one of aspects 20 to 21, wherein the plurality of prediction outputs comprises quantized prediction values.

[0146] Aspect 23. A processor-implemented method according to any one of Aspects 20 to 22, the method further comprising: comparing the multiple predicted outputs with multiple real data predictions associated with the multiple output channels; determining a loss associated with the neural network model based on the comparison of the multiple predicted outputs with the multiple real data predictions; and training the neural network model based on the loss.

[0147] Aspect 24. A processor-implemented method according to any one of Aspects 20 to 23, the method further comprising: restoring the multi-channel prediction target based on the single-channel output, the clamped prediction output, and at least one of the one or more remaining prediction outputs; determining a loss associated with the neural network model based on the restored multi-channel prediction target; and training the neural network model based on the loss.

[0148] Aspect 25. A processor-implemented method according to any one of aspects 20 to 24, wherein a maximum value of the quantization range is 1 and a minimum value of the quantization range is 0.

[0149] Aspect 26. A processor-implemented method according to any one of Aspects 20 to 25, wherein the clamped prediction output and the one or more remaining prediction outputs include quantized prediction values, wherein the quantized prediction values ​​include a sequence of values ​​between 0 and 1, and wherein the quantized prediction values ​​represent the multi-channel prediction target.

[0150] Clause 27. The processor-implemented method of clause 26, further comprising: restoring the multi-channel prediction target using the quantized prediction value.

[0151] Aspect 28. A processor-implemented method according to any one of aspects 20 to 27, wherein the plurality of output channels comprises 8 channels.

[0152] Aspect 29. A processor-implemented method according to any one of aspects 20 to 28, wherein the quantized prediction output associated with one or more channels from the plurality of output channels represents one or more values ​​within a target range.

[0153] Aspect 30. The processor-implemented method of any one of aspects 20 to 29, wherein generating the plurality of prediction outputs comprises distributing the plurality of prediction outputs across the plurality of output channels.

[0154] Aspect 31. A method implemented by a processor according to any one of Aspects 20 to 30, the method further comprising: multiplying the multi-channel prediction target by the number of channels in the multiple output channels; dividing the result of multiplying the multi-channel prediction target by the number of channels into binary digits, wherein at least a first binary digit among the binary digits includes a value of 1, and wherein at least a portion of the multiple prediction outputs includes the binary digits; and distributing the binary digits across at least a portion of the multiple output channels, wherein the first binary digit is distributed to a first channel from the multiple output channels.

[0155] Aspect 32. A processor-implemented method according to Aspect 31, wherein distributing the binary digits across at least a portion of the multiple output channels includes distributing one or more binary digits having a value of 1 to one or more output channels before distributing one or more additional binary digits having a value of 0 to one or more additional output channels, wherein the multiple predicted outputs also include the one or more binary digits and the one or more additional binary digits.

[0156] Aspect 33. A processor-implemented method according to any one of Aspects 20 to 32, the method further comprising: generating a multi-channel target distribution value based on the product of the multi-channel prediction target and the number of channels in the multiple output channels, wherein the multi-channel target distribution value comprises a floating-point number having an integer part and a fractional part; dividing the integer part of the multi-channel target distribution value into a set of binary digits; associating the set of binary digits with a set of channels from the multiple output channels; and associating the value of the fractional part of the floating-point number with a channel from the multiple output channels, wherein the multiple predicted outputs comprise the set of binary digits and the value of the fractional part of the floating-point number.

[0157] Aspect 34. A processor-implemented method according to Aspect 33, the method further comprising: associating one or more remaining channels from the plurality of output channels with one or more additional binary digits, wherein each of the one or more additional binary digits comprises a value of 0, wherein each binary digit in the set of binary digits comprises a value of 1, and wherein the plurality of predicted outputs further comprises the one or more additional binary digits.

[0158] Clause 35. A processor-implemented method according to any one of Clauses 20 to 34, wherein the data comprises image data, and the method further comprises reconstructing a depth map associated with the image data based at least in part on the single-channel output.

[0159] Aspect 36. A non-transitory computer-readable medium having stored thereon instructions, which when executed by one or more processors cause the one or more processors to perform the operations according to any one of aspects 19 to 35.

[0160] Aspect 37. An apparatus comprising means for performing the operations according to any one of aspects 19 to 35.

Claims

1. A device for implementing a deep learning model, the device comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory, the at least one processor configured to: generating a plurality of prediction outputs associated with a plurality of output channels corresponding to a multi-channel prediction target based on processing the data using the neural network model; determining that a predicted output from the plurality of predicted outputs has a value outside of a quantization range and that one or more remaining predicted outputs from the plurality of predicted outputs have corresponding values ​​within the quantization range; clamping the prediction output based on the quantization range; as well as A single channel output is generated based on the clamped prediction output and the one or more remaining prediction outputs.

2. The apparatus of claim 1, wherein the neural network model comprises an 8-bit quantized model. The apparatus of claim 1 , wherein the plurality of prediction outputs comprises quantized prediction values.

4. The apparatus of claim 1 , wherein the at least one processor is configured to: comparing the plurality of predicted outputs to a plurality of real data predictions associated with the plurality of output channels; determining a loss associated with the neural network model based on a comparison of the plurality of predicted outputs to the plurality of real data predictions; as well as The neural network model is trained based on the loss.

5. The apparatus of claim 1 , wherein the at least one processor is configured to: restoring the multi-channel prediction target based on at least one of the single channel output, the clamped prediction output, and the one or more remaining prediction outputs; Determining a loss associated with the neural network model based on the recovered multi-channel prediction target; and The neural network model is trained based on the loss. The apparatus according to claim 1 , wherein a maximum value of the quantization range is 1, and a minimum value of the quantization range is 0. 7 .

7. The apparatus of claim 1, wherein the clamped prediction output and the one or more remaining prediction outputs comprise quantized prediction values, wherein the quantized prediction values ​​comprise a sequence of values ​​between 0 and 1, and wherein the quantized prediction values ​​represent the multi-channel prediction target.

8. The apparatus of claim 7, wherein the at least one processor is configured to: The multi-channel prediction target is restored using the quantized prediction value.

9. The apparatus of claim 1, wherein the plurality of output channels comprises 8 channels.

10. The apparatus of claim 1, wherein the quantized prediction output associated with one or more channels from the plurality of output channels represents one or more values ​​within a target range.

11. The apparatus of claim 1, wherein to generate the plurality of prediction outputs, the at least one processor is configured to distribute the plurality of prediction outputs across the plurality of output channels.

12. The apparatus of claim 1, wherein the at least one processor is configured to: multiplying the multi-channel prediction target by the number of channels in the plurality of output channels; dividing a result of multiplying the multi-channel prediction target by the number of channels into binary digits, wherein at least a first binary digit of the binary digits comprises a value of 1, and wherein at least a portion of the plurality of prediction outputs comprises the binary digits; and The binary digits are distributed across at least a portion of the plurality of output channels, wherein the first binary digit is distributed to a first channel from the plurality of output channels.

13. An apparatus according to claim 12, wherein in order to distribute the binary digits across at least a portion of the plurality of output channels, the at least one processor is configured to distribute one or more binary digits having a value of 1 to one or more output channels before distributing one or more additional binary digits having a value of 0 to one or more additional output channels, wherein the plurality of predicted outputs also includes the one or more binary digits and the one or more additional binary digits.

14. The apparatus of claim 1, wherein the at least one processor is configured to: generating a multi-channel target distribution value based on a product of the multi-channel prediction target and a number of channels in the plurality of output channels, wherein the multi-channel target distribution value comprises a floating point number having an integer portion and a fractional portion; dividing the integer portion of the multi-channel target distribution value into a set of binary digits; associating the set of binary digits with a set of channels from the plurality of output channels; as well as The value of the fractional portion of the floating point number is associated with a channel from the plurality of output channels, wherein the plurality of predicted outputs includes the set of binary digits and the value of the fractional portion of the floating point number.

15. The apparatus of claim 14, wherein the at least one processor is configured to: Associating one or more remaining channels from the plurality of output channels with one or more additional binary digits, wherein each of the one or more additional binary digits comprises a value of 0, wherein each binary digit in the set of binary digits comprises a value of 1, and wherein the plurality of predicted outputs also comprises the one or more additional binary digits.

16. The apparatus of claim 1, wherein the data comprises image data, and wherein the at least one processor is configured to reconstruct a depth map associated with the image data based at least in part on the single channel output. 17 . The apparatus of claim 1 , wherein to clamp the prediction output based on the quantization range, the at least one processor is configured to modify a value of the prediction output to fall within the quantization range.

18. The apparatus of claim 1, wherein the apparatus comprises a mobile device.

19. The apparatus of claim 18, wherein the mobile device comprises a camera.

20. A method for implementing a processor of a deep learning model, the method comprising: generating a plurality of prediction outputs associated with a plurality of output channels corresponding to a multi-channel prediction target based on processing the data using the neural network model; determining that a predicted output from the plurality of predicted outputs has a value outside of a quantization range and that one or more remaining predicted outputs from the plurality of predicted outputs have corresponding values ​​within the quantization range; clamping the prediction output based on the quantization range; as well as A single channel output is generated based on the clamped prediction output and the one or more remaining prediction outputs.

21. The processor-implemented method of claim 20, wherein the neural network model comprises an 8-bit quantized model, and wherein the plurality of output channels comprises 8 channels.

22. The processor-implemented method of claim 20, wherein the plurality of prediction outputs comprises quantized prediction values.

23. The processor-implemented method of claim 20, further comprising: comparing the plurality of predicted outputs to a plurality of real data predictions associated with the plurality of output channels; determining a loss associated with the neural network model based on a comparison of the plurality of predicted outputs to the plurality of real data predictions; as well as The neural network model is trained based on the loss.

24. The processor-implemented method of claim 20, further comprising: restoring the multi-channel prediction target based on at least one of the single channel output, the clamped prediction output, and the one or more remaining prediction outputs; determining a loss associated with the neural network model based on the recovered multi-channel prediction target; as well as The neural network model is trained based on the loss.

25. The processor-implemented method of claim 20, wherein a maximum value of the quantization range is 1 and a minimum value of the quantization range is 0.

26. The processor-implemented method of claim 20, wherein the clamped prediction output and the one or more remaining prediction outputs comprise quantized prediction values, wherein the quantized prediction values ​​comprise a sequence of values ​​between 0 and 1, and wherein the quantized prediction values ​​represent the multi-channel prediction target, and the method further comprises: The multi-channel prediction target is restored using the quantized prediction value.

27. The processor-implemented method of claim 20, wherein quantized prediction outputs associated with one or more channels from the plurality of output channels represent one or more values ​​within a target range.

28. The processor-implemented method of claim 20, wherein generating the plurality of prediction outputs comprises distributing the plurality of prediction outputs across the plurality of output channels.

29. The processor-implemented method of claim 20, further comprising: multiplying the multi-channel prediction target by the number of channels in the plurality of output channels; dividing a result of multiplying the multi-channel prediction target by the number of channels into binary digits, wherein at least a first binary digit of the binary digits comprises a value of 1, and wherein at least a portion of the plurality of prediction outputs comprises the binary digits; as well as The binary digits are distributed across at least a portion of the plurality of output channels, wherein the first binary digit is distributed to a first channel from the plurality of output channels.

30. The processor-implemented method of claim 20, wherein clamping the predicted output based on the quantization range comprises modifying a value of the predicted output to fall within the quantization range.

Images