Image processing method and system within an image signal processing pipeline for a high dynamic range sensor

Abstract

The present disclosure relates to an image signal processing pipeline for a high dynamic range sensor. Apparatus, systems, and techniques are provided for: receiving, at one or more processors associated with an image signal processing (ISP) pipeline, a compressed image generated by an image sensor, where the compressed image is captured at a first bit depth associated with the image sensor and compressed to a second bit depth lower than the first bit depth, and where the ISP is associated with a third bit depth lower than the first bit depth and higher than the second bit depth; and decompressing the compressed image according to a power curve to generate a partially decompressed image having the third bit depth, where multiple regions of the partially decompressed image are decompressed by individual decompression amounts based on corresponding pixel values of each of the multiple regions.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the compression and decompression of images, including the decompression of images within an Image Signal Processing (ISP) pipeline. For example, at least one embodiment relates to a processor or computing system for decompressing a compressed image according to a power curve to generate a partially decompressed image according to various new techniques described herein. Background Art

[0002] High Dynamic Range (HDR) image sensors capture data with a large dynamic range, typically exceeding the memory, hardware, and data bandwidth available for processing them. Thus, the raw image data at the image sensor is typically compressed to reduce the bandwidth requirements during transmission to the ISP pipeline. The data is then decompressed at the ISP for processing. Modern image sensors (e.g., automotive sensors) are typically HDR image sensors that support high bit-depth images in order to provide a large dynamic illumination range when capturing images (e.g., at 24-bit resolution). On the other hand, the ISP hardware may not support such a high bit-depth as the image sensor, resulting in a possible degradation of image quality after the image is restored at the ISP due to the difference in bit-depth resolution between the sensor and the ISP. Brief Description of the Drawings

[0003] Figure 1 Shows a computer system for hosting and executing a camera image compression component according to at least one embodiment;

[0004] Figure 2 Is a flowchart of a process for performing compression and decompression algorithms for HDR images in an ISP pipeline according to at least one embodiment;

[0005] Figure 3 Is a diagram of a computing environment 300 for compressing and decompressing images from a camera sensor using a power curve according to at least one embodiment;

[0006] Figure 4 Shows a power curve having a linear segment for compressing and decompressing HDR images from a camera without affecting the SNR and color accuracy of the images according to at least one embodiment;

[0007] Figure 5 Shows the impact of using PWL to compress and decompress HDR row images from a camera and HDR-processed images from the ISP according to at least one embodiment;

[0008] Figure 6 Shows an example data center where at least one embodiment can be used;

[0009] Figure 7AShows an example of an autonomous vehicle according to at least one embodiment;

[0010] Figure 7B is a diagram according to at least one embodiment showing a system for communication between one or more cloud-based servers and Figure 7A an autonomous vehicle;

[0011] Figure 8 is a block diagram showing a computer system according to at least one embodiment;

[0012] Figure 9 is a block diagram showing a computer system according to at least one embodiment; and

[0013] Figure 10 shows at least a portion of a graphics processor according to one or more embodiments. DETAILED DESCRIPTION

[0014] Compression and decompression schemes for an image signal processing pipeline

[0015] In at least one embodiment, a method is provided for implementing a new compression and / or decompression scheme for processing high dynamic range (HDR) images in an image signal processing (ISP) pipeline to achieve high performance without color ratio or signal-to-noise ratio (SNR) degradation. In certain embodiments, the compression and / or decompression scheme is used to process data generated from a camera sensor at an initial high bit depth such that the data can be decompressed at the ISP to a lower bit depth than the initial high bit depth while preserving SNR and color accuracy. Piecewise linear (PWL) spline functions can be used to perform sensor compression and ISP decompression, which are implemented based on a power curve using curve fitting techniques. The power curve can be selected to compress the HDR image from the sensor bit depth to the ISP bit depth, where the compression level of a region of the image is based on the pixel values of the region according to the power curve. In one or more embodiments, the power curve can be characterized by two parameters (a constant and a power) to allow an initial linear approximation in the embodiments without degrading image quality. The power of the power curve can be based on a target compression ratio (e.g., the ratio of the input bit depth to the output bit depth). As an example, the pixel value of an image region can be the luminance value of the image region and / or the radiance value of the image region. The luminance value (in lux) and the radiance value (in watts) represent the amount of light from each surface in the scene within the image. In certain embodiments, a portion of the power curve can be approximated by a linear segment. In one embodiment, the linear segment is used to compress / decompress a region of the image having low pixel values (e.g., to compress / decompress a region of the image where the pixel values are below a certain threshold). In such a case, using the linear segment to compress / decompress the low pixel value (e.g., low signal) region of the input image can preserve SNR in the low light regions of the image, where the increase in noise due to quantization can be most critical.

[0016] Embodiments enable a system to combine an image sensor with a high bit depth (e.g., an HDR camera) with an ISP having a lower bit depth without degrading the SNR or color ratio in the processed image. The decompression function and optional compression function can be customized to provide non-linear decompression and / or compression that minimizes SNR and color ratio degradation. As a result, embodiments of the present disclosure enable system designers to select image sensors and ISP pipeline components with greater flexibility, which in turn enables them to reduce costs without degrading functionality. In an embodiment, the image sensor and ISP pipeline can be configured to capture an entire scene of interest in a single exposure. Additionally, in an embodiment, the image sensor and ISP pipeline can be configured to operate using a single mode that can function in both night and day scenes, thereby eliminating the use of separate night and day modes. Embodiments enable the settings of the image sensor and / or ISP to be adjusted to a superset of a dual-mode (e.g., night and day mode) exposure system, regardless of the bit depth of the ISP hardware. Thus, in an embodiment, the image sensor can be simplified by eliminating one or more operating modes and mode transitions between operating modes and removing transition artifacts. Additionally, this results in a simplified software that lacks an auto-exposure algorithm (commonly used to control the sensor settings of an image sensor). Further, embodiments enable the image sensor to be configured with fixed settings, which makes the system more secure and results in a simplified sensor driver, thereby saving a significant amount of engineering and quality analysis effort.

[0017] The present disclosure provides techniques and methods for compressing and decompressing HDR images between a sensor of a camera with a high bit depth and an ISP with a lower bit depth than the camera sensor while maintaining the SNR and color accuracy of the image. Compression and / or decompression can be configured based on the camera bit depth, the transmission bit depth (for transmission between the camera and the ISP), and / or the ISP bit depth. A power curve for compression at the camera and / or for decompression at the ISP can be determined based on the camera bit depth, the transmission bit depth, and / or the ISP bit depth. For example, a first power curve can be determined for compression and a second power curve can be determined for decompression. By configuring the level of compression and / or decompression according to the power curve, the size of the compressed image and / or the decompressed image (e.g., a partially decompressed image decompressed to the bit depth of the ISP) can be optimized. Decompression and / or compression performed according to the power curve can cause less compression to be performed on regions of the image with low luminance values and more compression to be performed on regions of the image with high luminance values. In some embodiments, a linear segment is determined as an alternative to a portion of the power curve. The linear segment can be an approximation of the power curve for low pixel values (e.g., low signals) having values less than a threshold, thereby preserving SNR and color accuracy for darker regions of underexposed images. Additionally, using a linear segment for low pixel value regions of the compressed / decompressed image can result in compression and / or decompression of the image without significantly degrading image attributes.

[0018] Figure 1 A system for hosting and executing a camera image compression / decompression component 115A - 115B is shown in accordance with at least one embodiment. In at least one embodiment, the computer system 100 can be a server, a system-on-chip (SoC), a desktop computer, a laptop computer, a mobile computing device, a cloud computing environment, and / or any other computer system. In at least one embodiment, the computer system 100 can include, but is not limited to, one or more processors 120 representative of one or more graphics processing units (GPUs), central processing units (CPUs), and / or any other processor. The computer system 100 can also include a cache 113, a data store 116, and / or other components and features not shown. The computer system 100 can include an ISP 140, which can be implemented using one or more processors 120, a camera sensor 130, and optionally one or more other components. In an embodiment, the camera sensor 130 and the computer system 100 can be components of a device such as a mobile phone, an autonomous vehicle, a non-autonomous vehicle, a video surveillance system, a laptop computer, a desktop computer, a quality analysis (QA) inspection system, or other systems.

[0019] In at least one embodiment, computer system 100 may include any number of caches 113, including those described herein. For example, in at least one embodiment, cache 113 may include a level three (“L3”) cache and / or a level two (“L2”) cache that is available to both the CPU and GPU of computer system 100. In at least one embodiment, cache 113 may include a write-back cache that may track the state of lines, for example, by using a cache coherence protocol (such as MEI, MESI, MSI, etc.). In at least one embodiment, the L3 cache may include 4MB of memory or more, depending on the embodiment, although smaller cache sizes may be used.

[0020] In at least one embodiment, computer system 100 may include data storage 116 (e.g., memory). In at least one embodiment, data storage 116 may be on-chip memory of computer system 100, which may store a neural network, one or more components of an image processing pipeline, etc., for execution on the GPU of computer system 100. In at least one embodiment, data storage 116 may include an L2 or L3 cache.

[0021] In at least one embodiment, processor 120 may include an embedded processor. In at least one embodiment, processor 120 may include a boot and power management processor, which may be a dedicated processor and subsystem to handle boot power and management functions as well as related security implementations. In at least one embodiment, the boot and power management processor may be part of the boot sequence of system 100 and may provide runtime power management services. In at least one embodiment, the boot power and management processor may provide clock and voltage programming, assistance with system low power state transitions, management of system 100 thermal and temperature sensors, and / or management of the power state of system 100. In at least one embodiment, each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and system 100 may use the ring oscillator to detect the temperature of the CPU, GPU, and / or the accelerator of system 100.

[0022] In at least one embodiment, processor 120 may further include a set of embedded processors that may be used as an audio processing engine, which may be an audio subsystem capable of providing full hardware support for multi-channel audio through multiple interfaces, as well as a wide and flexible range of audio I / O interfaces. In at least one embodiment, the audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

[0023] In at least one embodiment, the processor 120 may further include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. In at least one embodiment, the always-on processor engine may include, but is not limited to, processor cores, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0024] In at least one embodiment, the processor 120 may further include a security cluster engine that includes, but is not limited to, a dedicated processor subsystem for handling automotive application security management. In at least one embodiment, the security cluster engine may include, but is not limited to, two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In the secure mode, in at least one embodiment, two or more cores may operate in a lockstep mode and function as a single core with comparison logic to detect any differences between their operations. In at least one embodiment, the processor 120 may further include a real-time camera engine that includes, but is not limited to, a dedicated processor subsystem for handling real-time camera management. In at least one embodiment, the processor 120 may further include a signal processor, such as a high-dynamic range signal processor, which may include, but is not limited to, an image signal processor as a hardware engine that is part of the ISP 140. The processor 120 may further interact with the camera sensor 130 (also referred to as an image sensor) to receive and process images from the camera sensor 130.

[0025] In at least one embodiment, the processor 120 may include a video image synthesizer that may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required for a video playback application to produce the final image of the player window. In at least one embodiment, the video image synthesizer may perform lens distortion correction on the wide-angle camera 120, surround camera 120, and / or in-vehicle monitoring camera sensor.

[0026] The camera sensor 130 may be an image sensor or imager that detects and transmits information for making an image from the camera by converting the variable attenuation of light waves reflected by objects within the image into small bursts of current that serve as transmitted information. These waves may be light or other electromagnetic radiation. The camera sensor 130 can be used in analog and digital types of electronic imaging devices, including digital cameras, camera modules, camera phones, optical mouse devices, medical imaging devices, night vision devices, radar, sonar, and / or other imaging devices.

[0027] The ISP 140 can perform an intermediate set of digital image processing operations to process an image for rendering such that the processed image is optimized for chroma accuracy, minimal hardware cost, and / or low CPU utilization. In at least one embodiment, the ISP 140 (or at least components of the ISP 400) consists of the computer system 100 and / or the processor 120 of the computer system 100. In at least one embodiment, the ISP 140 (or at least components of the ISP) can be a component within a camera responsible for performing image processing before outputting an image from the camera. In at least one embodiment, the ISP 140 can be a remote and / or separate component that can process images from a camera connected to the ISP 140. In such a case, the remote ISP 140 can be connected to any number of compatible cameras to process images from each camera. In at least one embodiment, the operations performed by the ISP 140 can include applying a Bayer filter, performing noise reduction, performing shadow correction, performing image scaling, performing gamma correction, performing image enhancement, performing color space conversion, performing linearization, applying demosaicing techniques, performing frame rate conversion, performing image compression and / or decompression, and / or performing any combination of data transfer. The ISP 140 can include components used between an image source (e.g., the camera sensor 130 or scanner) and an image renderer (e.g., a television, printer, or computer screen) to perform the operations. The ISP 140 can be implemented as computer software, in a digital signal processor, on a general-purpose processor, on a dedicated processor, on a field-programmable gate array (FPGA), in analog circuitry, or as a fixed-function application-specific integrated circuit (ASIC). The ISP 140 can be or include a digital signal processor (DSP) that can be executed by a digital CPU with a dedicated hardware processing unit optimized for certain types of computations for processing images from the camera sensor 130.

[0028] According to one or more embodiments, the camera image compression / decompression components 115A-B can be used to perform compression and / or decompression of images. In at least one embodiment, the camera image compression / decompression components 115A-B can provide new compression and / or decompression schemes for high-dynamic range (HDR) images in an image signal processing (ISP) pipeline to achieve high performance without color ratio or signal-to-noise ratio (SNR) degradation. In some embodiments, the camera image compression / decompression components 115A-B can provide compression and / or decompression schemes to process data generated at an initial high bit depth from the camera sensor 130 such that the data can be decompressed at the ISP 140 to a lower bit depth than the initial high bit depth while preserving SNR and color accuracy. The camera image compression / decompression components 115A-B can use piecewise linear (PWL) spline functions to perform compression and / or decompression. In one embodiment, the image compression / decompression component 115A in the camera sensor 130 performs compression using a first PWL spline function, and the camera image compression / decompression component 115B performs decompression and / or compression using a second PWL spline function. As described below, the second PWL spline function can be determined based on the first PWL spline function and a determined power curve. A piecewise linear function can refer to a function defined on a real number interval such that there exists a set of such intervals: on each interval, the PWL function represents an affine transformation. An affine transformation can refer to a geometric transformation that preserves lines and parallelism, but may or may not preserve distances and angles.

[0029] In some embodiments, the camera image compression / decompression components 115A-B perform compression and / or decompression algorithms according to a power curve. The power curve can be selected to compress an HDR image from the sensor bit depth of the camera sensor (e.g., 24 bits) to the ISP bit depth (which may be lower than the sensor bit depth, e.g., 20 bits), where the compression level of a region of the image is based on the pixel values of the region according to the power curve. As an example, the pixel values of an image region can be the luminance value of the image region and / or the radiance value of the image region. The luminance value (in lux) and the radiance value (in watts) represent the amount of light from each surface in the scene within the image. By controlling the compression and / or decompression level of a region of the HDR image based on the region's luminance condition according to the power curve, the camera image compression / decompression components 115A-B can optimize the size of the compressed image, performing less compression on regions of the image with low luminance values and more compression on regions of the image with high luminance values, thereby preserving SNR and color accuracy for darker regions in the image with less or insufficient light.

[0030] In some embodiments, the camera image compression / decompression component 115 uses a compression / decompression power curve. The power curve can include a linear segment for compressing / decompressing regions of the image having low pixel values (e.g., pixels having a luminance or radiance value below a specific threshold for compressing / decompressing the image). In such a case, using the linear segment to compress / decompress the low pixel value regions of the input image can preserve the SNR in the low light regions of the image, where the noise added due to quantization can be most critical. In one embodiment, pixel values below a specific threshold may not be compressed or may be compressed according to a linear function rather than a non-linear function. Additionally, using a linear segment for compressing / decompressing the low pixel value regions of the image (e.g., which may not apply compression) can result in compression and / or decompression of the image without significantly degrading the image attributes, as described in more detail below.

[0031] In one embodiment, the camera image compression / decompression component 115B can receive, at the processor 120, a compressed image generated by the camera sensor 130 and compressed by the camera image compression / decompression component 115A for processing at the ISP 140. In some embodiments, the camera sensor 130 captures the compressed image at the bit depth of the camera sensor 130, and then the camera image compression / decompression component 115A compresses the captured image to a different bit depth lower than the camera sensor bit depth. The bit depth of an image can represent the amount of color information stored in the image, such that the higher the bit depth of the image, the more colors can be stored. For example, the simplest image may be a 1-bit image, which can only display two colors, black and white. In one embodiment, the camera image compression / decompression component 115A compresses the captured image from the camera sensor 130 by applying different amounts or levels of compression to different regions of the captured image according to a power curve based on the corresponding pixel values of each region, such that the SNR and color accuracy of the darker regions of the image are not degraded during the compression / decompression process. The power curve can be a curve with two degrees of freedom. In an embodiment, a portion of the power curve associated with pixel values below a threshold is approximated with a linear segment. As an example, the camera image compression / decompression component 115A can apply a first compression level to a first region of the captured image having a first pixel value, and then apply a second compression level to a second region of the captured image having a higher second pixel value, such that the first compression level is lower than the second compression level.

[0032] In one implementation, the camera image compression / decompression component 115 checks to determine whether the bit depth of the camera sensor 130 is equal to the bit depth of the ISP 140. If the bit depth of the camera sensor 130 is higher than the bit depth of the ISP 140 and the bit depth of the ISP 140 is higher than the bit depth of the compressed image, then the camera image compression / decompression component 115B performs decompression of the image from the bit depth of the compressed image to the bit depth of the ISP 140 using a power curve such that the decompressed image can be processed by the ISP 140 at the bit depth of the ISP 140, as described in more detail below with respect to Figure 3 More detailed description.

[0033] In one implementation, the camera image compression / decompression component 115B uses a power curve that at least partially corresponds to a piecewise linear (PWL) spline function to decompress the image from the bit depth of the compressed version of the image to the bit depth of the ISP 140. The PWL spline function includes a set of inflection points. The camera image compression / decompression component 115B or another computer system can use curve fitting techniques to determine the power curve based on the set of inflection points of the PWL function. Curve fitting techniques can refer to the process of approximating the inflection points of the PWL function as a known curve by sampling the curve and performing linear interpolation between points. The technique can also use an algorithm to calculate the most important inflection points in the curve, subject to a certain error tolerance. In some implementations, the power curve includes a linear segment that extends to a pixel value threshold for linearly processing image pixel values below the pixel value threshold to preserve the SNR and color accuracy of the low pixel value regions of the image. In this case, the camera image compression / decompression component 115B can avoid compressing (e.g., can fully decompress) image pixel values below the pixel value threshold to preserve the signal-to-noise ratio (SNR) of the image pixel values below the threshold. Alternatively, compression and / or decompression of pixel values below the threshold can be performed according to a linear function rather than a power function.

[0034] In some implementations, the camera image compression / decompression component 115B decompresses regions of the image with different decompression amounts based on the corresponding pixel values of each region of the image. As an example, the camera image compression / decompression component 115B can apply a first decompression amount to a first region of the compressed image having a specific pixel value; and can apply a second decompression amount to a second region of the compressed image having a pixel value higher than the pixel value of the first region such that after decompression the second region has a greater residual compression than the first region. In some embodiments, the first decompression amount is lower than the second decompression amount. In some embodiments, the second decompression amount is greater than the first decompression amount. In certain implementations, the pixel value of a given region of the image corresponds to the luminance value of that region, which indicates the amount of light in the captured region of the image.

[0035] Figure 2 FIG. 200 is a flowchart of a process for performing compression and decompression algorithms for HDR images in an ISP pipeline, according to at least one embodiment. In at least one embodiment, the processing logic implementing process 200 may provide a compression and / or decompression scheme to process data generated at an initial high bit depth from a camera sensor such that the data can be decompressed at an image signal processor (ISP) to a bit depth lower than the initial high bit depth while preserving SNR and color accuracy.

[0036] In operation 202, an image sensor captures an image at a first bit depth. The image sensor may be, for example, an HDR image sensor that captures images at a high bit depth (e.g., a 24-bit bit depth). The image sensor may be an HDR sensor having a bit depth of, for example, 20 bits, 24 bits, 26 bits, 32 bits, etc. In operation 204, the image sensor (e.g., a processing device at the image sensor) compresses the image from the first bit depth to a lower second bit depth (e.g., from a 24-bit bit depth to a 12-bit bit depth). The bit depth to which the image is compressed may be based on the connection bandwidth between the image sensor and the ISP. The second bit depth may be, for example, 8 bits, 10 bits, 12 bits, 14 bits, 16 bits, 18 bits, etc. In some embodiments, PWL compression is used to perform the compression. Alternatively, other compression techniques may be used, such as cubic compression, lookup table compression, etc. In certain implementations, a power curve that may or may not include linear segments is used to compress the image. The power curve may be determined based on the characteristics of the image sensor and may be selected such that an image compressed according to the power curve and then subsequently decompressed (to the original or a different bit depth) maintains a color ratio and SNR close to the original. In one embodiment, a PWL compression function is determined based on the determined power curve. In operation 206, the image sensor transmits the compressed image to the ISP pipeline.

[0037] In operation 210, processing logic receives, at one or more processors associated with an image signal processing (ISP) pipeline, the compressed image generated by the image sensor, where the ISP is associated with a third bit depth that is lower than the first bit depth and higher than the second bit depth (e.g., having a bit depth of 16 - 22 bits, e.g., 20 bits). The ISP may have the same bit depth as the image sensor or a lower bit depth than the image sensor. For example, the ISP may have a bit depth of 10 bits, 12 bits, 14 bits, 16 bits, 18 bits, 20 bits, 22 bits, 24 bits, etc.

[0038] At operation 220, the processing logic decompresses the compressed image according to a power curve (or different power curves) to generate a partially decompressed image with a third bit depth. Multiple regions of the partially decompressed image can be decompressed by separate decompression amounts according to the power curve, based on one or more corresponding pixel values for each of the multiple regions. In one embodiment, the decompression is performed using a PWL function generated using curve fitting techniques based on the power curve. In an implementation, the pixel value of a region of the image can refer to the luminance value of that region of the image, as described in more detail herein.

[0039] In one embodiment, at operation 230, the processing logic applies a first decompression amount to a first region of the compressed image having a first pixel value according to the power curve. In an implementation, the first pixel value of the first region of the image can refer to the luminance value of the first region of the image, as described in more detail herein.

[0040] In one embodiment, at operation 240, the processing logic applies a second decompression amount to a second region of the compressed image having a higher second pixel value according to the power curve. In an embodiment, after decompression, the remaining compression level of the second region is greater than the remaining compression level of the first region. In one embodiment, the first decompression amount is lower than the second decompression amount. In one embodiment, the second decompression amount is lower than the first decompression amount. In an implementation, the second pixel value of the second region of the image can refer to the luminance value of the second region of the image, which is higher than the first luminance value of the first region.

[0041] Figure 3 is a diagram of a computing environment 300 for compressing and decompressing an image from a camera sensor using a power curve (e.g., using a PWL function adapted to the power curve) according to at least one embodiment. In at least one embodiment, the processing logic can utilize a piecewise linear function (PWL) to perform compression and decompression of image data based on a power curve corresponding to the PWL function. In this case, the captured image is first compressed by the camera sensor 312 using the PWL function and then decompressed in the ISP 314 using another PWL function (e.g., an approximate inverse PWL function of the PWL function used for compression) to achieve the bit depth associated with the ISP 314. The PWL function for decompression can be selected such that when applied to the compressed image, the resulting partially compressed image has a compression state corresponding to the power curve in the embodiment. The computing environment 300 can be the same as or similar to Figure 1 the computer system 100, the sensor 312 can be the same as or similar to Figure 1 the camera sensor 130, and the ISP 314 can be the same as or similar to Figure 1 the ISP 140.

[0042] At operation 310, the processing logic executes a compression algorithm at sensor 312 to compress the captured image from a first bit depth (e.g., 24 bits) that is the bit depth of sensor 312 to a second bit depth (e.g., 12 bits) using a PWL function 316, which may or may not be based on a power curve. The processing logic can compress the original image (e.g., 24 bits) into a compressed image (e.g., 12 bits) to enable faster and more efficient transmission of the image data from sensor 312 to ISP 314. In one embodiment, the processing logic can utilize the PWL compression equation at sensor 312 to compress 24-bit linear data (X) into 12-bit data (Y) using the following equation:

[0043] Y = PWL sensor (X),

[0044] where PWL sensor represents the piecewise linear equation used by sensor 312 to compress data (X).

[0045] At operation 320, the processing logic transmits the compressed image to ISP 314. In an embodiment, ISP 314 can perform image processing and correction functions on the received image, including linearization (LIN), black level correction (BLC), lens shading correction (LSC), and white balance correction (WBC). In an embodiment, when the processing logic determines that the bit depth of ISP 314 is a third bit depth that is lower than the original bit depth of the original image but higher than the compressed bit depth (e.g., an intermediate bit depth of 20 bits), the processing logic applies PWL decompression to the data (Y) from sensor 312 to generate partially decompressed data (e.g., 20-bit data) (Z) using a PWL function 318, which may or may be based on a power curve, and the PWL function 318 generates an image represented by the data (Z) that can be processed by the remainder of ISP 314. The final image can have a new compression state (e.g., a partially compressed state) that has less compression than the initial compression state. In the new compression state, the amount of compression for each pixel can be based on the luminance or radiance value of that pixel. The new compression state can correspond to the power curve 322.

[0046] The processing logic can generate data (Z) using the following equation:

[0047] Z = PWL isp (Y),

[0048] where PWL ispRepresents the piecewise linear equation used by ISP 314 to decompress data (Y). Thus, by combining the above two equations, the processing logic uses the following equation to generate an image of a new bit depth (e.g., 20-bit image) for ISP 314 from the original bit depth image (e.g., 24-bit image) generated by sensor 312. The following equation uses the power curve 322 to convert the original bit depth image (e.g., 24-bit image) into an image of a new bit depth (e.g., 20-bit image):

[0049] Z = PWL isp (PWL sensor (X))

[0050] The PWL processing can be performed in a manner that approximates the following power function:

[0051] Z ≈ X α

[0052] , where α can be the value of the power defining the curve. As described below, the PWL processing can be performed in a manner that approximates the following generalized power function:

[0053] Z ≈ βX α

[0054] , where α and β are parameters (e.g., numerical values) defining the power curve.

[0055] Thus, the final compression state of the image processed by the rest of ISP 314 has a form that approximates the power curve 322. The power curve 322 shows the final target compression state of the image regions according to the relevant luminance values of the image regions (e.g., where a higher compression level is applied to those regions with higher luminance values). The arrangement of the power curve 322 of the final compression state of this image is achieved based on the combination of PWL compression (e.g., using PWL function 316) and PWL decompression (e.g., using PWL function 318). In fact, the final compression state shown by the power curve 322 can be achieved by multiplying the PWL function 316 by the PWL function 318. During the design of the compression and / or decompression stage, the compression and / or decompression stage can be solved by multiplying or dividing the final compression state (e.g., according to the power curve 322) by either the compression or decompression PWL function to achieve the other of the compression or decompression functions. Thus, the processing logic can accurately determine what each inflection point of the PWL function for compression and / or decompression should be.

[0056] In one embodiment, it is assumed that the PWL processing is determined based on the inflection points of the sensor PWL compression and the inflection points of the ISP PWL decompression function based on the curve fitting technique of the corresponding power curve. These compression and decompression algorithms may result in high quantization noise for low signal levels (e.g., low pixel values). High quantization noise has a negative impact on the SNR and color accuracy of low pixel values. In this case, to preserve the SNR and color accuracy, the processing logic can use a modified power curve that includes a linear segment at the beginning so that compression and decompression can be prevented for low power values within the linear segment of the power curve. The modified power function for generating data (Z) can be expressed as:

[0057] Z = βX α

[0058] , where β is a constant added to the power curve to allow for a linear region at the beginning of the curve. Therefore, the following equation can be used to implement 24-bit to 20-bit compression:

[0059] 2 20 = β(2 24 ) α

[0060] , and the following equation can be used to implement the linear segment in the low signal region, e.g., power values of 512 or below:

[0061]

[0062] Similar equations using different starting and target bit depth values (e.g., other than 24 bits and 20 bits) can also be used to solve for different power curves with customized linear segments.

[0063] In some embodiments, the automatic white balance (AWB) operation and / or the lens shading correction (LSC) operation have the form of gain * X and the power is scaled according to the gain that preserves the color ratio. α scaled.

[0064] Figure 4shows a power curve with a linear segment that can be used to compress and / or decompress an image from a camera (e.g., such as an HDR image) without affecting the SNR and color accuracy of the image, according to at least one embodiment. In at least one embodiment, the processing logic implementing process 400 can utilize a power curve 410 that includes a linear segment 412 to compress and / or decompress an HDR image based on the pixel values of an image region. In certain implementations, the linear segment 412 in the power curve 410 can be used for pixel values at 512 or below 512. Using the linear segment for pixel values below 512 can prevent the compression and / or decompression of regions having these pixel values, and thus prevent a decrease in SNR for low pixel values (e.g., low signal levels) without causing color inaccuracies.

[0065] As Figure 4 shown, power curve 420 represents a larger scale view of linear segment 412, which illustrates that pixel values of a 24-bit image up to a value of 512 can be linearly converted to 20-bit pixel values, optionally preventing any actual compression or decompression of the image, as described in more detail herein.

[0066] Figure 5 shows the impact of PWL compression and decompression using an HDR raw image from a camera and an HDR-processed image from an ISP, according to at least one embodiment. As Figure 5 shown, image 510 is generated using a 20-bit camera sensor and processed using a 20-bit ISP. Image 520 shows the same image as image 510, but image 520 is captured by a 24-bit sensor and processed by a 20-bit ISP using one or more of the disclosed embodiments. As Figure 5 shown, compared to image 510, image 520 shows less saturation at high power values, even though the final bit depth shown is the same.

[0067] Image 530 represents the raw image generated using a 20-bit camera sensor before processing. Image 540 shows the same image as image 530, but image 540 is the raw image captured by a 24-bit camera sensor. As shown, both raw images 530, 540 show much higher saturation than the processed images processed according to the embodiments described herein.

[0068] Data center

[0069] Figure 6FIG. 600 shows an example data center in which at least one embodiment may be used. In an embodiment, the example data center 600 may be used to determine an optimal power curve and PWL function for compression and / or decompression. For example, the data center 600 may perform curve fitting operations to determine one or more PWL functions for compression and / or decompression that map to a power curve and / or cause a compressed image to be compressed according to and / or corresponding to the power curve. In at least one embodiment, the data center 600 includes a data center infrastructure layer 610, a framework layer 620, a software layer 630, and an application layer 640.

[0070] In at least one embodiment, as Figure 6 shown, the data center infrastructure layer 610 may include a resource coordinator 612, grouped computing resources 614, and node computing resources (“node C.R.s”) 616(1)-616(N), where “N” represents a positive integer (which may be a different integer “N” than the integer used in other figures). In at least one embodiment, the node C.R.s 616(1)-616(N) may include, but are not limited to, any number of central processing units (“CPU”) or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors, etc.), memory storage devices 618(1)-618(N) (such as dynamic read-only memory, solid state drives, or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules, and cooling modules, etc. In at least one embodiment, one or more of the node C.R.s 616(1)-616(N) may be servers having one or more of the above computing resources.

[0071] In at least one embodiment, the grouped computing resources 614 may include separate groupings (not shown) of node C.R.s housed within one or more racks, or many racks (also not shown) housed within a data center at various geographical locations. In at least one embodiment, the separate groupings of node C.R.s within the grouped computing resources 614 may include grouped computing, network, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s including CPUs or processors may be grouped within one or more racks to provide computing resources to support one or more workloads. In at least one embodiment, one or more racks may also include any number of power modules, cooling modules, and network switches, in any combination.

[0072] In at least one embodiment, the resource coordinator 612 may configure or otherwise control one or more node C.R.s 616(1)-616(N) and / or the grouped computing resources 614. In at least one embodiment, the resource coordinator 612 may include a software design infrastructure (“SDI”) management entity for the data center 600. In at least one embodiment, the resource coordinator 112 may include hardware, software, or some combination thereof.

[0073] In at least one embodiment, as Figure 6 shown, the framework layer 620 includes a job scheduler 622, a configuration manager 624, a resource manager 626, and a distributed file system 628. In at least one embodiment, the framework layer 620 may include a framework for the software 632 of the support software layer 630 and / or one or more applications 642 of the application layer 640. In at least one embodiment, the software 632 or the application 642 may respectively include web-based service software or applications, such as the services or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 620 may be, but is not limited to, a free and open-source software web application framework, such as Apache Spark that may utilize the distributed file system 628 for large-scale data processing (e.g., “big data”) TM (hereinafter referred to as “Spark”). In at least one embodiment, the job scheduler 632 may include a Spark driver to facilitate scheduling of the workloads supported by the various layers of the data center 600. In at least one embodiment, the configuration manager 624 may be able to configure the different layers, such as the software layer 630 and the framework layer 620 including Spark and the distributed file system 628 for supporting large-scale data processing. In at least one embodiment, the resource manager 626 is capable of managing the clusters or grouped computing resources mapped to or allocated for supporting the distributed file system 628 and the job scheduler 622. In at least one embodiment, the clusters or grouped computing resources may include the grouped computing resources 614 on the data center infrastructure layer 610. In at least one embodiment, the resource manager 626 may coordinate with the resource coordinator 612 to manage these mapped or allocated computing resources.

[0074] In at least one embodiment, the software 632 included in the software layer 630 may include software used by at least a portion of the node C.R.s 616(1)-616(N), the grouped computing resources 614, and / or the distributed file system 1228 of the framework layer 620. In at least one embodiment, one or more types of software may include, but are not limited to, Internet web search software, email virus scanning software, database software, and streaming video content software.

[0075] In at least one embodiment, one or more applications 642 included in the application layer 640 may include one or more types of applications used by at least a portion of the node C.R.s 1216(1)-1216(N), the grouped computing resources 614, and / or the distributed file system 628 of the framework layer 620. In at least one embodiment, one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing applications, and machine learning applications, including training or inference software, machine learning framework software (such as PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0076] According to one or more embodiments, the camera image compression / decompression component 115 may be used to provide a compression and / or decompression solution for high dynamic range (HDR) images in an image signal processing (ISP) pipeline. Details regarding the camera image compression / decompression component 115 are provided herein in connection with Figure 1 provided. In at least one embodiment, the camera image compression / decompression component 115 may be used in Figure 6 to perform the compression and decompression of HDR images as described herein.

[0077] Autonomous vehicle

[0078] Although not limited to autonomous vehicles, embodiments of the present disclosure can be particularly effective when applied to images generated by cameras of autonomous vehicles. It is expected that cameras on vehicles can generate accurate images both in direct sunlight (e.g., having a measured value of hundreds to thousands of lux) and on moonless and starless nights (e.g., having a measured value of a fraction of a lux). Thus, it can be expected that cameras of vehicles can accurately generate images in a variety of environments, the luminance values (in lux) of which vary by up to approximately eight orders of magnitude between extreme environments. Traditionally, cameras have been able to take accurate images across such a wide range of environments by performing automatic white balance and automatically switching between day and night modes, each with different camera settings. This enables the camera and the ISP to remain at a relatively low bit depth, where the camera and the ISP may have the same bit depth. In an embodiment, an HDR camera can be used with a lower bit depth ISP. The HDR camera may compress images, which may be partially decompressed at the ISP. The compression and / or decompression can be performed according to a power curve (optionally having a linear segment), and the color ratio and SNR can be preserved. By implementing embodiments of the present disclosure, the image sensor and the ISP can work together to generate a final processed image that is not saturated in bright light and does not lose data in dim light, while minimizing the bandwidth requirements for transferring data between the image sensor and the ISP and keeping the ISP at a lower bit depth than the image sensor.

[0079] Figure 7A An example of an autonomous vehicle 700 according to at least one embodiment is shown. In at least one embodiment, the autonomous vehicle 700 (alternatively referred to herein as "vehicle 700") can be, but is not limited to, a passenger vehicle, such as a car, truck, bus, and / or another type of vehicle that can accommodate one or more passengers. In at least one embodiment, the vehicle 700 can be a semi-trailer truck for hauling cargo. In at least one embodiment, the vehicle 700 can be an aircraft, a robotic vehicle, or another type of vehicle.

[0080] Autonomous vehicles can be described according to the levels of automation defined by the National Highway Traffic Safety Administration (“NHTSA”) under the United States Department of Transportation and the Society of Automotive Engineers (“SAE”) in “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (e.g., Standard No. J3016 - 201806 issued on June 15, 2018, Standard No. J3016 - 201609 issued on September 30, 2016, and previous and future versions of this standard). In at least one embodiment, vehicle 700 may be capable of functioning according to one or more of Levels 1 to 5 of the automation levels. For example, in at least one embodiment, according to the embodiment, vehicle 700 may be capable of conditional automation (Level 3), highly automated (Level 4), and / or fully automated (Level 5).

[0081] In at least one embodiment, vehicle 700 may include, but is not limited to, components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of the vehicle. In at least one embodiment, vehicle 1300 may include, but is not limited to, a propulsion system 750, such as an internal combustion engine, a hybrid device, a fully electric motor, and / or another type of propulsion system. In at least one embodiment, propulsion system 750 may be connected to the driveline of vehicle 700, which may include, but is not limited to, a transmission, to enable the propulsion of vehicle 700. In at least one embodiment, a signal may be received from throttle / accelerator 752 to control propulsion system 750.

[0082] In at least one embodiment, when propulsion system 750 is operating (e.g., when vehicle 700 is moving), a steering system 754 (which may include, but is not limited to, a steering wheel) is used to steer vehicle 700 (e.g., along a desired path or route). In at least one embodiment, steering system 754 may receive a signal from a steering actuator 756. In at least one embodiment, the steering wheel may be optional for fully automated (Level 5) functions. In at least one embodiment, a brake sensor system 746 may be used to operate vehicle brakes in response to signals received from a brake actuator 748 and / or a brake sensor.

[0083] In at least one embodiment, controller 736 may include, but is not limited to, one or more system - on - chips (“SoC”)( Figure 7A(not shown) and / or a graphics processing unit (“GPU”) provides signals (e.g., representing commands) to one or more components and / or systems of the vehicle 700. For example, in at least one embodiment, the controller 736 may send signals to operate the vehicle brakes via the brake actuator 748, operate the steering system 754 via one or more steering actuators 756, and operate the propulsion system 750 via one or more throttles / accelerators 752. In at least one embodiment, one or more controllers 736 may include one or more on-board (e.g., integrated) computing devices that process sensor signals and output operation commands (e.g., signals representing commands) to achieve autonomous driving and / or assist the driver in driving the vehicle 700. In at least one embodiment, one or more controllers 736 may include a first controller for autonomous driving functions, a second controller for functional safety functions, a third controller for artificial intelligence functions (e.g., computer vision), a fourth controller for infotainment functions, a fifth controller for redundancy in emergency situations, and / or other controllers. In at least one embodiment, a single controller may handle two or more of the above functions, and two or more controllers may handle a single function and / or any combination thereof.

[0084] In at least one embodiment, one or more controllers 736 provide signals for controlling one or more components and / or systems of the vehicle 700 in response to sensor data received from one or more sensors (e.g., sensor inputs). In at least one embodiment, the sensor data may be received from sensors of sensor types such as but not limited to one or more global navigation satellite system (“GNSS”) sensors 758 (e.g., one or more global positioning system sensors), one or more RADAR sensors 760, one or more ultrasonic sensors 762, one or more LIDAR sensors 764, one or more inertial measurement unit (IMU) sensors 766 (e.g., one or more accelerometers, one or more gyroscopes, one or more magnetic compasses, one or more magnetometers, etc.), one or more microphones 1396, one or more stereo cameras 768, one or more wide-angle cameras 770 (e.g., fisheye cameras), one or more infrared cameras 772, one or more surround cameras 774 (e.g., 360-degree cameras), remote cameras ( Figure 7A (not shown), mid-range cameras ( Figure 7A (not shown), one or more speed sensors 744 (e.g., for measuring the speed of the vehicle 700), one or more vibration sensors 742, one or more steering sensors 740, one or more brake sensors (e.g., as part of the brake sensor system 746), and / or other sensor types.

[0085] In at least one embodiment, one or more controllers 736 may receive inputs (e.g., represented by input data) from the instrument panel 732 of the vehicle 700 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (“HMI”) display 734, a sound annunciator, a speaker, and / or other components of the vehicle 700. In at least one embodiment, the output may include information such as vehicle speed, speed, time, map data (e.g., high-definition map ( Figure 7A not shown in the figure), location data (e.g., the location of the vehicle 700, e.g., on a map), direction, the locations of other vehicles (e.g., occupancy raster), information about objects, and the status of objects sensed by one or more controllers 736, etc. For example, in at least one embodiment, the HMI display 734 may display information about the presence of one or more objects (e.g., road signs, warning signs, traffic signal changes, etc.) and / or information about driving operations that the vehicle has made, is making, or will make (e.g., changing lanes now, exiting at Exit 34B within two miles, etc.).

[0086] In at least one embodiment, the vehicle 700 further includes a network interface 724, which may communicate via one or more networks using one or more wireless antennas 726 and / or one or more modems. For example, in at least one embodiment, the network interface 724 may be capable of communicating via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”) networks, etc. In at least one embodiment, one or more wireless antennas 726 may also enable communication between objects in the environment (e.g., vehicles, mobile devices) using one or more local area networks (e.g., Bluetooth, Bluetooth Low Energy (LE), Z-Wave, ZigBee, etc.) and / or one or more low-power wide area networks (hereinafter referred to as “LPWAN”) (e.g., LoRaWAN, SigFox, etc. protocols).

[0087] According to one or more embodiments, the camera image compression / decompression component 115 may be used to provide compression and / or decompression solutions for high dynamic range (HDR) images in image signal processing (ISP). The camera image compression / decompression component 115 may be executed on a processor of the vehicle 700, e.g., on the CPU, GPU, SoC, etc. of the vehicle 700. Details regarding the camera image compression / decompression component 115 are described herein in connection with Figure 1Provided. In at least one embodiment, the camera image compression / decompression component 115 can be used in a system Figure 7A of the system to perform compression and / or decompression of HDR images as described herein.

[0088] Figure 7B illustrates an example system architecture of an Figure 7A autonomous vehicle 700 according to at least one embodiment. In at least one embodiment, Figure 7B each of one or more components, one or more features, and one or more systems in the vehicle 700 in is shown connected via a bus 702. In at least one embodiment, the bus 702 can include, but is not limited to, a CAN data interface (alternatively referred to herein as the "CAN bus"). In at least one embodiment, CAN can be a network inside the vehicle 700 that helps control various features and functions of the vehicle 700, such as actuation of brakes, acceleration, braking, steering, windshield wipers, etc. In one embodiment, the bus 702 can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). In at least one embodiment, the bus 702 can be read to find the steering wheel angle, ground speed, engine revolutions per minute ("RPM"), button positions, and / or other vehicle state indicators. In at least one embodiment, the bus 702 can be an ASIL B-compliant CAN bus.

[0089] In at least one embodiment, in addition to or from CAN, FlexRay and / or Ethernet protocols can be used. In at least one embodiment, there can be any number of formed buses 702, which can include, but are not limited to, zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and / or zero or more other types of buses using other protocols. In at least one embodiment, two or more buses can be used to perform different functions, and / or can be used for redundancy. For example, a first bus can be used for collision avoidance functions, and a second bus can be used for actuation control. In at least one embodiment, each bus of the bus 702 can communicate with any component of the vehicle 700, and two or more of the bus 702 can communicate with corresponding components. In at least one embodiment, each of any number of system-on-chips ("SoC") 704 (e.g., SoC 704(A) and SoC 704(B)), each of one or more controllers 736, and / or each computer in the vehicle can access the same input data (e.g., input from sensors of the vehicle 700) and can be connected to a common bus, such as a CAN bus.

[0090] In at least one embodiment, vehicle 700 may include one or more controllers 736, such as those described herein with respect to Figure 7A those described. In at least one embodiment, controller 736 may be used for a variety of functions. In at least one embodiment, controller 736 may be coupled to any of the various other components and systems of vehicle 700 and may be used to control vehicle 700, the artificial intelligence of vehicle 700, the infotainment of vehicle 700, and / or other functions.

[0091] In at least one embodiment, vehicle 700 may include any number of SoCs 704. In at least one embodiment, each of the SoCs 704 may include, but is not limited to, a central processing unit (“one or more CPUs”) 706, a graphics processing unit (“one or more GPUs”) 708, one or more processors 710, one or more caches 712, one or more accelerators 714, one or more data stores 716, and / or other components and features not shown. In at least one embodiment, one or more of the SoCs 704 may be used to control vehicle 700 in various platforms and systems. For example, in at least one embodiment, one or more of the SoCs 704 may be combined with a high-definition (“HD”) map 722 in a system (e.g., the system of vehicle 700), and the high-definition map 722 may obtain map refreshes and / or updates from one or more servers ( Figure 7B not shown in) via network interface 724.

[0092] In at least one embodiment, one or more of the CPUs 706 may include a CPU cluster or CPU complex (alternatively referred to herein as “CCPLEX”). In at least one embodiment, one or more of the CPUs 706 may include multiple cores and / or a secondary (“L2”) cache. For example, in at least one embodiment, one or more of the CPUs 706 may include eight cores in a multiprocessor configuration coupled to each other. In at least one embodiment, one or more of the CPUs 706 may include four dual-core clusters, each of which has a dedicated L2 cache (e.g., a 2MB L2 cache). In at least one embodiment, one or more of the CPUs 706 (e.g., CCPLEX) may be configured to support simultaneous cluster operations such that any combination of the clusters of one or more of the CPUs 706 may be active at any given time.

[0093] In at least one embodiment, one or more CPUs 706 may implement power management functions, which may include, but are not limited to, one or more of the following features: automatically clock-gating individual hardware modules during idle to save dynamic power; clock-gating each core clock when the core is not actively executing instructions due to executing a wait for interrupt ("WFI") / wait for event ("WFE") instruction; each core may be independently powered; each core cluster may be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster may be independently power-gated when all cores are power-gated. In at least one embodiment, one or more CPUs 706 may further implement an enhanced algorithm for managing power states, where allowed power states and expected wake-up times are specified, and the hardware / microcode determines the optimal power state for core, cluster, and CCPLEX inputs. In at least one embodiment, the processing core may support a simplified power state input sequence in software, where the work is shared with the microcode.

[0094] In at least one embodiment, one or more GPUs 708 may include an integrated GPU (referred to herein as an "iGPU"). In at least one embodiment, one or more GPUs 708 may be programmable and may be effective for parallel workloads. In at least one embodiment, one or more GPUs 708 may use an enhanced tensor instruction set. In at least one embodiment, one or more GPUs 708 may include one or more streaming microprocessors, where each streaming microprocessor may include a level 1 ("L1") cache (e.g., an L1 cache with a storage capacity of at least 96 KB), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache with a storage capacity of 512 KB). In at least one embodiment, one or more GPUs 708 may include at least eight streaming microprocessors. In at least one embodiment, one or more GPUs 708 may use a compute application programming interface (API). In at least one embodiment, one or more GPUs 708 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA model).

[0095] In at least one embodiment, one or more GPUs 708 may be power-optimized to achieve optimal performance in automotive and embedded use cases. For example, in at least one embodiment, one or more GPUs 708 may be fabricated on fin field-effect transistor (“FinFET”) circuitry. In at least one embodiment, each streaming microprocessor may include multiple mixed-precision processing cores divided into multiple blocks. For example but not limited to, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In at least one embodiment, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA tensor cores for deep learning matrix arithmetic, a level zero (“L0”) instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In at least one embodiment, the streaming microprocessor may include independent parallel integer and floating-point data paths to provide efficient execution of workloads mixing computational and addressing operations. In at least one embodiment, the streaming microprocessor may include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation between parallel threads. In at least one embodiment, the streaming microprocessor may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0096] In at least one embodiment, one or more GPUs 708 may include high bandwidth memory (“HBM”) and / or a 16GB high bandwidth memory second generation (“HBM2”) memory subsystem to provide a peak memory bandwidth of approximately 900GB / second in some examples. In at least one embodiment, synchronous graphics random access memory (“SGRAM”), such as graphics double data rate type five synchronous random access memory (“GDDR5”), may be used in addition to or in place of HBM memory.

[0097] In at least one embodiment, one or more GPUs 708 may include unified memory technology. In at least one embodiment, address translation service (“ATS”) support may be used to allow one or more GPUs 708 to directly access the page tables of one or more CPUs 706. In at least one embodiment, when a memory management unit (“MMU”) of a GPU in one or more GPUs 708 experiences a miss, an address translation request may be sent to one or more CPUs 706. In response, in at least one embodiment, a CPU in one or more CPUs 706 may look up the virtual-physical mapping of the address in its page table and transmit the translation back to one or more GPUs 708. In at least one embodiment, unified memory technology may allow a single unified virtual address space for the memory of both one or more CPUs 706 and one or more GPUs 708, thus simplifying the programming of one or more GPUs 708 and porting applications to one or more GPUs 708.

[0098] In at least one embodiment, one or more GPUs 708 may include any number of access counters, which may track the frequency of access by one or more GPUs 708 to the memory of other processors. In at least one embodiment, one or more access counters may help ensure that memory pages are moved into the physical memory of the processor that most frequently accesses the pages, thereby improving the efficiency of the memory range shared among processors.

[0099] In at least one embodiment, one or more SoCs 704 may include any number of caches 712, including those described herein. For example, in at least one embodiment, one or more caches 712 may include a level three (“L3”) cache that may be used by one or more CPUs 706 and one or more GPUs 708 (e.g., connected to the CPU 706 and GPU 708). In at least one embodiment, one or more caches 712 may include a write-back cache that may track the state of lines, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). In at least one embodiment, although a smaller cache size may be used, according to an embodiment, the L3 cache may include 4MB of memory or more.

[0100] In at least one embodiment, one or more SoCs 704 may include one or more accelerators 714 (e.g., hardware accelerators, software accelerators, or a combination thereof). In at least one embodiment, one or more SoCs 704 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memories. In at least one embodiment, the large on-chip memory (e.g., 4MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. In at least one embodiment, the hardware acceleration cluster may be used to supplement one or more GPUs 708 and offload some tasks of one or more GPUs 708 (e.g., free up more cycles of one or more GPUs 708 to perform other tasks). In at least one embodiment, one or more accelerators 714 may be used for target workloads that are stable enough to withstand acceleration testing (e.g., perception, convolutional neural networks (“CNNs”), recurrent neural networks (“RNNs”), etc.). In at least one embodiment, CNNs may include region-based or region convolutional neural networks (“RCNNs”) and fast RCNNs (e.g., as used for object detection) or other types of CNNs.

[0101] In at least one embodiment, one or more accelerators 714 (e.g., a hardware acceleration cluster) may include one or more deep learning accelerators (“DLAs”). In at least one embodiment, one or more DLAs may include, but are not limited to, one or more Tensor Processing Units (“TPUs”), which may be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. In at least one embodiment, a TPU may be an accelerator configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). In at least one embodiment, one or more DLAs may be further optimized for a particular set of neural network types and floating point operations and inference. In at least one embodiment, the design of one or more DLAs may provide higher performance per millimeter than a typical general-purpose GPU and generally far exceed the performance of a CPU. In at least one embodiment, one or more TPUs may perform several functions, including supporting, for example, INT8, INT16, and FP16 data types for single-instance convolution functions for features and weights and post-processor functions. In at least one embodiment, one or more DLAs may execute a neural network, especially a CNN, quickly and efficiently on processed or unprocessed data for any of a variety of functions, including, for example, but not limited to: a CNN for object recognition and detection using data from a camera sensor; a CNN for distance estimation using data from a camera sensor; a CNN for emergency vehicle detection and identification and detection using data from a microphone; a CNN for face recognition and vehicle owner recognition using data from a camera sensor; and / or a CNN for security and / or safety-related events.

[0102] In at least one embodiment, a DLA may perform any function of one or more GPUs 708, and by using an inference accelerator, for example, a designer may target one or more DLAs or one or more GPUs 708 for any function. For example, in at least one embodiment, a designer may concentrate the processing and floating point operations of a CNN on one or more DLAs and leave other functions to one or more GPUs 708 and / or one or more accelerators 714.

[0103] In at least one embodiment, one or more accelerators 714 may include a programmable vision accelerator (“PVA”), which may alternatively be referred to herein as a computer vision accelerator. In at least one embodiment, one or more PVAs may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (“ADAS”) 738, autonomous driving, augmented reality (“AR”) applications, and / or virtual reality (“VR”) applications. In at least one embodiment, one or more PVAs may strike a balance between performance and flexibility. For example, in at least one embodiment, each of one or more PVAs may include, for example but not limited to, any number of reduced instruction set computer (“RISC”) cores, direct memory access (“DMA”), and / or any number of vector processors.

[0104] In at least one embodiment, the RISC cores may interact with an image sensor (e.g., the image sensor of any of the cameras described herein), an image signal processor, and the like. In at least one embodiment, each RISC core may include any number of memories. In at least one embodiment, according to the embodiment, the RISC cores may use any one of a variety of protocols. In at least one embodiment, the RISC cores may execute a real-time operating system (“RTOS”). In at least one embodiment, one or more integrated circuit devices, application specific integrated circuits (“ASICs”), and / or storage devices may be used to implement the RISC cores. For example, in at least one embodiment, the RISC cores may include an instruction cache and / or tightly coupled RAM.

[0105] In at least one embodiment, the DMA may enable the components of the PVA to access system memory independently of one or more CPUs 706. In at least one embodiment, the DMA may support any number of features for optimizing the supply to the PVA, including but not limited to, supporting multi-dimensional addressing and / or circular addressing. In at least one embodiment, the DMA may support up to six or more dimensions of addressing, which may include but are not limited to block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0106] In at least one embodiment, the vector processor can be a programmable processor that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In at least one embodiment, the PVA can include a PVA core and two vector processing subsystem partitions. In at least one embodiment, the PVA core can include a processor subsystem, DMA engines (e.g., two DMA engines), and / or other peripherals. In at least one embodiment, the vector processing subsystem can serve as the main processing engine of the PVA and can include a vector processing unit (“VPU”), an instruction cache, and / or a vector memory (e.g., “VMEM”). In at least one embodiment, the VPU core can include a digital signal processor, e.g., a single instruction multiple data (“SIMD”), very long instruction word (“VLIW”) digital signal processor. In at least one embodiment, the combination of SIMD and VLIW can improve throughput and speed.

[0107] In at least one embodiment, each vector processor can include an instruction cache and can be coupled to dedicated memory. As a result, in at least one embodiment, each vector processor can be configured to execute independently of other vector processors. In at least one embodiment, the vector processors included in a particular PVA can be configured to employ data parallelism. For example, in at least one embodiment, multiple vector processors included in a single PVA can execute general computer vision algorithms, except on different regions of an image. In at least one embodiment, the vector processors included in a particular PVA can simultaneously execute different computer vision algorithms on one image, or even execute different algorithms on sequential images or partial images. In at least one embodiment, among other things, any number of PVAs can be included in a hardware acceleration cluster, and any number of vector processors can be included in each PVA. In at least one embodiment, the PVA can include additional error correction code (“ECC”) memory to enhance overall system security.

[0108] In at least one embodiment, one or more accelerators 714 may include an on-chip computer vision network and static random access memory (“SRAM”) to provide high-bandwidth, low-latency SRAM for the one or more accelerators 714. In at least one embodiment, the on-chip memory may include at least 4 MB of SRAM, which includes, for example but not limited to, eight field-configurable memory blocks that can be accessed by both the PVA and the DLA. In at least one embodiment, each pair of memory blocks may include an advanced peripheral bus (“APB”) interface, configuration circuitry, a controller, and a multiplexer. In at least one embodiment, any type of memory may be used. In at least one embodiment, the PVA and the DLA may access the memory via a backbone network that provides high-speed access to the memory for the PVA and the DLA. In at least one embodiment, the backbone network may include an on-chip computer vision network that interconnects the PVA and the DLA to the memory (e.g., using the APB).

[0109] In at least one embodiment, the on-chip computer vision network may include an interface that determines that both the PVA and the DLA provide ready and valid signals before transmitting any control signals / address / data. In at least one embodiment, the interface may provide separate phases and separate channels for transmitting control signals / address / data, as well as burst-type communication for continuous data transfer. In at least one embodiment, the interface may conform to the International Organization for Standardization (“ISO”) 26262 or International Electrotechnical Commission (“IEC”) 61508 standards, although other standards and protocols may be used.

[0110] In at least one embodiment, one or more SoCs 704 may include a real-time line-of-sight tracking hardware accelerator. In at least one embodiment, the real-time line-of-sight tracking hardware accelerator may be used to quickly and efficiently determine the position and extent of an object (e.g., within a world model) to generate a real-time visualization simulation for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison with LIDAR data for positioning and / or other functions, and / or for other uses.

[0111] In at least one embodiment, one or more accelerators 714 have a wide range of uses for autonomous driving. In at least one embodiment, PVA can be used in key processing stages in ADAS and autonomous vehicles. In at least one embodiment, the capabilities of PVA at low power and low latency match well with algorithm domains that require predictable processing. In other words, PVA performs well in semi-dense or dense conventional computations, even on small data sets, which may require predictable runtimes with low latency and low power. In at least one embodiment, such as in vehicle 700, PVA may be designed to run classical computer vision algorithms because they can be effective in object detection and integer math operations.

[0112] For example, according to at least one embodiment of the technology, PVA is used to perform computer stereo vision. In at least one embodiment, an algorithm based on semi-global matching may be used in some examples, although this is not meant to be limiting. In at least one embodiment, applications for level 3-5 autonomous driving use dynamic estimation / stereo matching in operation (e.g., structure from motion, pedestrian recognition, lane detection, etc.). In at least one embodiment, PVA can perform computer stereo vision functions on inputs from two monocular cameras.

[0113] In at least one embodiment, PVA can be used to perform dense optical flow. For example, in at least one embodiment, PVA can process raw RADAR data (e.g., using 4D fast Fourier transform) to provide processed RADAR data. In at least one embodiment, for example, by processing raw time-of-flight data to provide processed time-of-flight data, PVA is used for time-of-flight depth processing.

[0114] In at least one embodiment, the DLA can be used to run any type of network to enhance control and driving safety, including, for example but not limited to, a neural network, the output of which is a confidence level for each object detection. In at least one embodiment, the confidence level can be represented or interpreted as a probability, or represented as providing a relative "weight" of each detection relative to other detections. In at least one embodiment, the confidence level measurement enables the system to make further decisions, namely, regarding which detections should be considered true positive detections rather than false positive detections. In at least one embodiment, the system can set a threshold for the confidence level and consider only detections that exceed the threshold as true positive detections. In embodiments using an automatic emergency braking ("AEB") system, false positive detections will cause the vehicle to automatically perform emergency braking, which is clearly undesirable. In at least one embodiment, highly confident detections can be considered as triggers for AEB. In at least one embodiment, the DLA can run a neural network for regressing confidence values. In at least one embodiment, the neural network can take at least some subset of parameters as its input, such as bounding box dimensions, an obtained ground plane estimate (e.g., from another subsystem), and the output of one or more IMU sensors 766 related to the vehicle 700 direction, distance, 3D position estimate of an object obtained from the neural network and / or other sensors (e.g., one or more LIDAR sensors 764 or one or more RADAR sensors 760).

[0115] In at least one embodiment, one or more SoCs 704 can include one or more data storage devices 716 (e.g., memory). In at least one embodiment, one or more data storages 716 can be on-chip memories of one or more SoCs 704, which can store neural networks to be executed on one or more GPUs 708 and / or DLA. In at least one embodiment, one or more data storages 716 can have a large enough capacity to store multiple instances of neural networks for redundancy and safety. In at least one embodiment, one or more data storages 716 can include L2 or L3 caches.

[0116] In at least one embodiment, one or more SoCs 704 may include any number of processors 710 (e.g., embedded processors). In at least one embodiment, one or more processors 710 may include a boot and power management processor, which may be a dedicated processor and subsystem to handle boot power and management functions as well as associated security implementations. In at least one embodiment, the boot and power management processor may be part of the boot sequence of one or more SoCs 704 and may provide runtime power management services. In at least one embodiment, the boot power and management processor may provide clock and voltage programming, assist in system low-power state transitions, manage one or more SoC 704 thermal and temperature sensors, and / or manage the power state of one or more SoCs 704. In at least one embodiment, each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and one or more SoCs 704 may use the ring oscillator to detect the temperature of one or more CPUs 706, one or more GPUs 708, and / or one or more accelerators 714. In at least one embodiment, if it is determined that the temperature exceeds a threshold, the boot and power management processor may enter a temperature fault routine and place one or more SoCs 704 in a lower power state and / or place the vehicle 700 in a safe parking pattern for the driver (e.g., safely park the vehicle 700).

[0117] In at least one embodiment, one or more processors 710 may further include a set of embedded processors that may be used as an audio processing engine, and the audio processing engine may be an audio subsystem that can provide full hardware support for multi-channel audio to the hardware through multiple interfaces and a wide and flexible range of audio I / O interfaces. In at least one embodiment, the audio processing engine is a dedicated processor core that has a digital signal processor with dedicated RAM.

[0118] In at least one embodiment, one or more processors 710 may further include an always-on processor engine that may provide the necessary hardware features to support low-power sensor management and wake-up use cases. In at least one embodiment, the processors on the always-on processor engine may include, but are not limited to, processor cores, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0119] In at least one embodiment, one or more processors 710 may further include a security cluster engine, which includes but is not limited to dedicated processor subsystems for handling security management of automotive applications. In at least one embodiment, the security cluster engine may include but is not limited to two or more processor cores, tightly coupled RAM, supporting peripherals (such as timers, interrupt controllers, etc.) and / or routing logic. In the security mode, in at least one embodiment, two or more cores may operate in a lockstep mode and may be used as a single core with comparison logic for detecting any differences between their operations. In at least one embodiment, one or more processors 710 may further include a real-time camera engine, which may include but is not limited to dedicated processor subsystems for handling real-time camera management. In at least one embodiment, one or more processors 710 may further include a high-dynamic range signal processor, which may include but is not limited to an image signal processor, which is a hardware engine as part of a camera processing pipeline.

[0120] In at least one embodiment, one or more processors 710 may include a video image synthesizer, which may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required for a video playback application to produce a final image for a player window. In at least one embodiment, the video image synthesizer may perform lens distortion correction on one or more wide-angle cameras 770, one or more surround cameras 774, and / or one or more in-cabin monitoring camera sensors. In at least one embodiment, preferably, the in-cabin monitoring camera sensors are monitored by a neural network running on another instance of the SoC 704, which is configured to identify in-cabin events and respond accordingly. In at least one embodiment, the in-cabin system may perform but is not limited to lip reading to activate cellular services and make phone calls, indicate emails, change the destination of the vehicle, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. In at least one embodiment, certain functions are available to the driver when the vehicle is operating in autonomous mode and are otherwise disabled.

[0121] In at least one embodiment, the video image synthesizer may include enhanced temporal noise reduction for simultaneous spatial and temporal noise reduction. For example, in at least one embodiment, in the case where motion occurs in the video, the noise reduction appropriately weights the spatial information, thereby reducing the weight of the information provided by adjacent frames. In at least one embodiment, in the case where an image or a part of the image does not include motion, the temporal noise reduction performed by the video image synthesizer may use information from a previous image to reduce the noise in the current image.

[0122] In at least one embodiment, the video image synthesizer may also be configured to perform stereo correction on the input stereo lens frames. In at least one embodiment, when using an operating system desktop, the video image synthesizer may also be used for user interface synthesis and does not require one or more GPUs 708 to continuously render new surfaces. In at least one embodiment, when powering one or more GPUs 708 and making them actively perform 3D rendering, the video image synthesizer may be used to offload one or more GPUs 708 to improve performance and responsiveness.

[0123] In at least one embodiment, one or more of the SoCs 704 may further include a Mobile Industry Processor Interface ("MIPI") camera serial interface, a high-speed interface, and / or a video input block for receiving video and inputs from cameras, which can be used for camera and related pixel input functions. In at least one embodiment, one or more of the SoCs 704 may further include an input / output controller, which can be software-controlled and can be used to receive I / O signals that are not committed to a specific role.

[0124] In at least one embodiment, one or more of the SoCs 704 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio encoder / decoders ("codecs"), power management, and / or other devices. In at least one embodiment, one or more of the SoCs 704 can be used to process data from (e.g., connected via Gigabit Multimedia Serial Link and Ethernet channels) cameras, sensors (e.g., one or more LIDAR sensors 764, one or more RADAR sensors 760, etc., which can be connected via Ethernet channels), data from bus 702 (e.g., the speed of vehicle 700, steering wheel position, etc.), data from one or more GNSS sensors 758 (e.g., connected via Ethernet bus or CAN bus), etc. In at least one embodiment, one or more of the SoCs 704 may further include a dedicated high-performance mass storage controller, which may include its own DMA engine and can be used to relieve one or more CPUs 706 from conventional data management tasks.

[0125] In at least one embodiment, one or more SoCs 704 can be an end-to-end platform with a flexible architecture that spans automation levels 3 - 5, thus providing an integrated functional safety architecture that leverages and effectively uses computer vision and ADAS technologies to achieve diversity and redundancy, and that provides a platform for a flexible and reliable driving software stack as well as deep learning tools. In at least one embodiment, one or more SoCs 704 can be faster and more reliable than conventional systems, and even more energy - efficient and space - efficient. For example, in at least one embodiment, one or more accelerators 714, when combined with one or more CPUs 706, one or more GPUs 708, and one or more data storage devices 716, can provide a fast and effective platform for level 3 - 5 autonomous vehicles.

[0126] In at least one embodiment, computer vision algorithms can be executed on a CPU, which can be configured using a high - level programming language (such as C) to execute a variety of processing algorithms on a variety of visual data. However, in at least one embodiment, a CPU generally cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In at least one embodiment, many CPUs cannot execute complex object detection algorithms in real time, which are used in in - vehicle ADAS applications and actual level 3 - 5 autonomous vehicles.

[0127] The embodiments described herein allow multiple neural networks to be executed simultaneously and / or sequentially, and allow the results to be combined to achieve level 3 - 5 autonomous driving functions. For example, in at least one embodiment, a CNN executed on a DLA or a discrete GPU (e.g., one or more GPUs 720) can include text and word recognition, thus allowing a supercomputer to read and understand traffic signs, including signs that the neural network has not been specifically trained for. In at least one embodiment, the DLA can also include a neural network that is capable of recognizing, interpreting, and providing semantic understanding of symbols, and passing that semantic understanding to a path - planning module running on a CPU Complex.

[0128] In at least one embodiment, for a level 3, 4, or 5 drive, multiple neural networks can be run simultaneously. For example, in at least one embodiment, a warning sign consisting of a connected electric light with the text "Caution: flashing lights indicate icy conditions" can be interpreted independently or jointly by multiple neural networks. In at least one embodiment, the warning sign itself can be recognized as a traffic sign by a first deployed neural network (e.g., a neural network that has been trained), and the text "flashing lights indicate icy conditions" can be interpreted by a second deployed neural network, which notifies the vehicle's path planning software (preferably executed on the CPU Complex) that when flashing lights are detected, there is an icy condition. In at least one embodiment, a third deployed neural network can be operated on multiple frames to identify the flashing lights and notify the vehicle's path planning software of the presence (or absence) of the flashing lights. In at least one embodiment, all three neural networks can be run simultaneously, e.g., within the DLA and / or on one or more GPUs 708.

[0129] In at least one embodiment, a CNN for face recognition and vehicle owner recognition can use data from a camera sensor to identify the presence of an authorized driver and / or the owner of the vehicle 700. In at least one embodiment, a normally open sensor processor engine can be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and, in a security mode, can be used to disable the vehicle when the owner leaves the vehicle. In this way, one or more SoCs 704 provide protection against theft and / or carjacking.

[0130] In at least one embodiment, the CNN for emergency vehicle detection and recognition can use data from microphone 796 to detect and recognize emergency vehicle sirens. In at least one embodiment, one or more SoCs 704 use the CNN to classify ambient and urban sounds, as well as to classify visual data. In at least one embodiment, the CNN running on the DLA is trained to recognize the relative approach speed of an emergency vehicle (e.g., by using the Doppler effect). In at least one embodiment, the CNN can also be trained to recognize emergency vehicles for the area in which the vehicle is operating, as identified by one or more GNSS sensors 758. In at least one embodiment, when operating in Europe, the CNN will seek to detect European sirens, while in North America, the CNN will seek to recognize only North American sirens. In at least one embodiment, once an emergency vehicle is detected, a control program can be used with the assistance of one or more ultrasonic sensors 762 to execute emergency vehicle safety routines, slow down the vehicle, pull the vehicle over to the side of the road, park, and / or idle the vehicle until the emergency vehicle has passed.

[0131] In at least one embodiment, vehicle 700 can include one or more CPUs 718 (e.g., one or more discrete CPUs or one or more dCPUs), which can be coupled to one or more SoCs 704 via a high-speed interconnect (e.g., PCIe). In at least one embodiment, one or more CPUs 718 can include X86 processors, e.g., one or more CPUs 718 can be used to perform any of a variety of functions, such as including potentially arbitrating inconsistent results between ADAS sensors and one or more SoCs 704, and / or monitoring the status and health of one or more monitoring controllers 736 and / or on-chip information system (“Info SoC”) 730.

[0132] In at least one embodiment, vehicle 700 can include one or more GPUs 720 (e.g., one or more discrete GPUs or one or more dGPUs), which can be coupled to one or more SoCs 704 via a high-speed interconnect (e.g., NVIDIA's NVLINK channels). In at least one embodiment, one or more GPUs 720 can provide additional artificial intelligence capabilities, such as by executing redundant and / or different neural networks, and can be used for training and / or updating neural networks at least in part based on inputs from the sensors of vehicle 700 (e.g., sensor data).

[0133] In at least one embodiment, vehicle 700 may further include a network interface 724, which may include, but is not limited to, one or more wireless antennas 726 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). In at least one embodiment, network interface 724 may be used to enable a wireless connection with other vehicles and / or computing devices (e.g., a passenger's client device) via an Internet cloud service (e.g., using a server and / or other network devices). In at least one embodiment, to communicate with other vehicles, a direct link and / or an indirect link (e.g., via a network and the Internet) may be established between vehicle 70 and another vehicle. In at least one embodiment, a vehicle-to-vehicle communication link may be used to provide the direct link. In at least one embodiment, the vehicle-to-vehicle communication link may provide vehicle 700 with information about vehicles in the vicinity of vehicle 700 (e.g., vehicles in front of, to the side of, and / or behind vehicle 700). In at least one embodiment, the foregoing function may be part of the cooperative adaptive cruise control function of vehicle 700.

[0134] In at least one embodiment, network interface 724 may include a SoC that provides modulation and demodulation functions and enables one or more controllers 736 to communicate via a wireless network. In at least one embodiment, network interface 724 may include a radio frequency front end for upconverting from baseband to radio frequency and downconverting from radio frequency to baseband. In at least one embodiment, frequency conversion may be performed in any technically feasible manner. For example, frequency conversion may be performed by well-known processes and / or using a superheterodyne process. In at least one embodiment, the radio frequency front end function may be provided by a separate chip. In at least one embodiment, the network interface may include wireless functions for communicating via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0135] In at least one embodiment, vehicle 700 may further include one or more data stores 728, which may include, but are not limited to, off-chip (e.g., one or more SoCs 704) storage. In at least one embodiment, one or more data stores 728 may include, but are not limited to, one or more storage elements, including RAM, SRAM, dynamic random access memory (“DRAM”), video random access memory (“VRAM”), flash memory, hard disks, and / or other components and / or devices that can store at least one bit of data.

[0136] In at least one embodiment, vehicle 700 may further include one or more GNSS sensors 758 (e.g., GPS and / or assisted GPS sensors) to assist in mapping, perception, occupancy grid generation, and / or path planning functions. In at least one embodiment, any number of GNSS sensors 758 may be used, including for example but not limited to GPS using a universal serial bus (“USB”) connector with an Ethernet to serial (e.g., RS-232) bridge.

[0137] In at least one embodiment, vehicle 700 may further include one or more RADAR sensors 760. In at least one embodiment, the one or more RADAR sensors 760 may be used by vehicle 700 for remote vehicle detection, even in darkness and / or adverse weather conditions. In at least one embodiment, the RADAR functional safety level may be ASIL B. In at least one embodiment, the one or more RADAR sensors 760 may use a CAN bus and / or bus 1302 (e.g., to transmit data generated by the one or more RADAR sensors 760) for control and access to object tracking data, and in some examples may access an Ethernet channel to access raw data. In at least one embodiment, a variety of RADAR sensor types may be used. For example but not limited to, one or more of the RADAR sensors 760 may be suitable for front, rear, and side RADAR use. In at least one embodiment, the one or more RADAR sensors 760 are pulsed Doppler RADAR sensors.

[0138] In at least one embodiment, one or more RADAR sensors 760 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In at least one embodiment, long-range RADAR may be used for adaptive cruise control functions. In at least one embodiment, the long-range RADAR system may provide a wide field of view achieved through two or more independent scans (e.g., within a range of 250m). In at least one embodiment, one or more RADAR sensors 760 may assist in differentiating between static and moving objects and may be used by the ADAS system 738 for emergency braking assistance and forward collision warning. In at least one embodiment, one or more sensors 760 included in the long-range RADAR system may include, but are not limited to, monostatic multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In at least one embodiment, with six antennas, the central four antennas may create a focused beam pattern designed to record the surrounding environment of the vehicle 700 at a relatively high speed with minimal traffic interference from adjacent lanes. In at least one embodiment, the other two antennas may expand the field of view so that vehicles entering or leaving the lane of the vehicle 700 can be quickly detected.

[0139] In at least one embodiment, by way of example, the mid-range RADAR system may include, for example, a range of up to 160m (front) or 80m (rear), and a field of view of up to 42 degrees (front) or 150 degrees (rear). In at least one embodiment, the short-range RADAR system may include, but is not limited to, any number of RADAR sensors 760 designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, in at least one embodiment, the RADAR sensor system may generate two beams that continuously monitor the rear direction of the vehicle and the nearby blind spots. In at least one embodiment, the short-range RADAR system may be used in the ADAS system 738 for blind spot detection and / or lane change assistance.

[0140] In at least one embodiment, the vehicle 700 may further include one or more ultrasonic sensors 762. In at least one embodiment, one or more ultrasonic sensors 762 that may be positioned at the front, rear, and / or side positions of the vehicle 700 may be used for parking assistance and / or creating and updating occupancy grids. In at least one embodiment, a variety of ultrasonic sensors 762 may be used, and different ultrasonic sensors 762 may be used for different detection ranges (e.g., 2.5m, 4m). In at least one embodiment, the ultrasonic sensors 762 may operate at a functional safety level of ASIL B.

[0141] In at least one embodiment, vehicle 700 may include one or more LIDAR sensors 764. In at least one embodiment, one or more LIDAR sensors 764 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. In at least one embodiment, one or more LIDAR sensors 764 may operate at a functional safety level of ASIL B. In at least one embodiment, vehicle 700 may include multiple (e.g., two, four, six, etc.) LIDAR sensors 764 (e.g., providing data to a gigabit Ethernet switch) that may use Ethernet channels.

[0142] In at least one embodiment, one or more LIDAR sensors 764 may be capable of providing a list of objects and their distances for a 360-degree field of view. In at least one embodiment, commercially available one or more LIDAR sensors 764 may, for example, have an advertised range of approximately 100 m, an accuracy of 2 cm - 3 cm, and support an Ethernet connection of 100 Mbps. In at least one embodiment, one or more non-protruding LIDAR sensors may be used. In such an embodiment, one or more LIDAR sensors 764 may include small devices that can be embedded in the front, rear, sides, and / or corner positions of vehicle 700. In at least one embodiment, one or more LIDAR sensors 764, in such an embodiment, may provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, and have a range of 200 m, even for low-reflectivity objects. In at least one embodiment, the forward one or more LIDAR sensors 764 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0143] In at least one embodiment, LIDAR technology (such as 3D flash LIDAR) may also be used. In at least one embodiment, 3D flash LIDAR uses a laser flash as a transmission source to illuminate approximately 200 m around the vehicle 700. In at least one embodiment, the flash LIDAR unit includes, but is not limited to, a receiver that records the laser pulse propagation time and the reflected light on each pixel, which in turn corresponds to the range from the vehicle 700 to the object. In at least one embodiment, flash LIDAR may allow highly accurate and distortion-free images of the surrounding environment to be generated using each laser flash. In at least one embodiment, four flash LIDAR sensors may be deployed, with one sensor on each side of the vehicle 700. In at least one embodiment, the 3D flash LIDAR system includes, but is not limited to, a solid-state 3D line-of-sight array LIDAR camera that has no moving parts other than a fan (such as a non-scanning LIDAR device). In at least one embodiment, the flash LIDAR device may use Class I (eye-safe) laser pulses of 5 nanoseconds per frame and may capture the reflected laser as 3D range point clouds and co-registered intensity data.

[0144] In at least one embodiment, the vehicle 700 may further include one or more IMU sensors 766. In at least one embodiment, one or more IMU sensors 766 may be located at the center of the rear axle of the vehicle 700. In at least one embodiment, one or more IMU sensors 766 may include, for example but not limited to, one or more accelerometers, one or more magnetometers, one or more gyroscopes, a magnetic compass, multiple magnetic compasses, and / or other sensor types. In at least one embodiment, for example, in a six-axis application, one or more IMU sensors 766 may include, but are not limited to, accelerometers and gyroscopes. In at least one embodiment, for example, in a nine-axis application, one or more IMU sensors 766 may include, but are not limited to, accelerometers, gyroscopes, and magnetometers.

[0145] In at least one embodiment, one or more IMU sensors 766 may be implemented as a miniature high-performance GPS-aided inertial navigation system ("GPS / INS") that combines microelectromechanical system ("MEMS") inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude; in at least one embodiment, one or more IMU sensors 766 may enable the vehicle 700 to estimate heading without input from a magnetic sensor by directly observing and correlating the speed changes from GPS to one or more IMU sensors 766. In at least one embodiment, one or more IMU sensors 766 and one or more GNSS sensors 758 may be combined in a single integrated unit.

[0146] In at least one embodiment, vehicle 700 may include one or more microphones 796 placed within and / or around vehicle 700. In at least one embodiment, additionally, one or more microphones 796 may be used for emergency vehicle detection and identification.

[0147] In at least one embodiment, vehicle 700 may further include any number of camera types, including one or more stereo cameras 768, one or more wide-angle cameras 770, one or more infrared cameras 772, one or more surround cameras 774, one or more long-range cameras 798, one or more mid-range cameras 776, and / or other camera types. In at least one embodiment, the cameras may be used to capture image data around the entire perimeter of vehicle 700. In at least one embodiment, the type of camera used depends on vehicle 700. In at least one embodiment, any combination of camera types may be used to provide the necessary coverage around vehicle 700. In at least one embodiment, the number of cameras deployed may vary according to the embodiment. For example, in at least one embodiment, vehicle 700 may include six cameras, seven cameras, ten cameras, twelve cameras, or other numbers of cameras. In at least one embodiment, the cameras may support, by way of example but not limitation, Gigabit Multimedia Serial Link (“GMSL”) and / or Gigabit Ethernet communications. In at least one embodiment, each camera may be described in more detail previously herein Figure 7A which may be described in more detail.

[0148] In at least one embodiment, vehicle 700 may further include one or more vibration sensors 742. In at least one embodiment, one or more vibration sensors 742 may measure the vibration of components of vehicle 700 (e.g., the axle). For example, in at least one embodiment, a change in vibration may indicate a change in the road surface. In at least one embodiment, when two or more vibration sensors 742 are used, the difference between the vibrations may be used to determine the friction or slippage of the road surface (e.g., when there is a vibration difference between a powered drive axle and a freely rotating axle).

[0149] In at least one embodiment, vehicle 700 may include an ADAS system 738. In at least one embodiment, ADAS system 738 may include, but is not limited to, a SoC. In at least one embodiment, ADAS system 738 may include, but is not limited to, any number of autonomous / adaptive / automatic cruise control (“ACC”) systems, cooperative adaptive cruise control (“CACC”) systems, forward collision warning (“FCW”) systems, automatic emergency braking (“AEB”) systems, lane departure warning (“LDW”) systems, lane keeping assist (“LKA”) systems, blind spot warning (“BSW”) systems, rear cross traffic warning (“RCTW”) systems, collision warning (“CW”) systems, lane centering (“LC”) systems, and / or other systems, features, and / or functions and combinations thereof.

[0150] In at least one embodiment, the ACC system may use one or more RADAR sensors 760, one or more LIDAR sensors 764, and / or any number of cameras. In at least one embodiment, the ACC system may include a longitudinal ACC system and / or a lateral ACC system. In at least one embodiment, the longitudinal ACC system monitors and controls the distance to another vehicle adjacent to vehicle 700 and automatically adjusts the speed of vehicle 700 to maintain a safe distance from the vehicle ahead. In at least one embodiment, the lateral ACC system performs distance keeping and recommends that vehicle 700 change lanes when necessary. In at least one embodiment, the lateral ACC is related to other ADAS applications, such as LC and CW.

[0151] In at least one embodiment, the CACC system uses information from other vehicles, which may be received via a wireless link or indirectly via a network connection (e.g., via the Internet) from other vehicles via a network interface 724 and / or one or more wireless antennas 726. In at least one embodiment, the direct link may be provided by a vehicle-to-vehicle (“V2V”) communication link, while the indirect link may be provided by an infrastructure-to-vehicle (“I2V”) communication link. Generally, V2V communication provides information about the vehicle immediately ahead (e.g., the vehicle immediately in front of vehicle 700 and in the same lane), while I2V communication provides information about traffic further ahead. In at least one embodiment, the CACC system may include one or both of the I2V and V2V information sources. In at least one embodiment, in the case of information about the vehicle ahead of a given vehicle 700, the CACC system may be more reliable and has the potential to improve the smoothness of traffic flow and reduce road congestion.

[0152] In at least one embodiment, the FCW system is designed to warn the driver of a hazard so that the driver can take corrective action. In at least one embodiment, the FCW system uses a forward camera and / or one or more RADAR sensors 760, which are coupled to a dedicated processor, an electronic signal processor (“DSP”), an FPGA, and / or an ASIC, which are electrically coupled to provide driver feedback, such as a display, a speaker, and / or a vibration component. In at least one embodiment, the FCW system can provide warnings, such as in the form of sounds, visual warnings, vibrations, and / or rapid braking pulses.

[0153] In at least one embodiment, the AEB system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. In at least one embodiment, the AEB system can use one or more forward cameras and / or one or more RADAR sensors 760 coupled to a dedicated processor, a DSP, an FPGA, and / or an ASIC. In at least one embodiment, when the AEB system detects a hazard, it generally first warns the driver to take corrective action to avoid the collision, and, if the driver does not take corrective action, the AEB system can automatically apply the brakes in an attempt to prevent or at least mitigate the effects of the predicted collision. In at least one embodiment, the AEB system can include technologies such as dynamic brake support and / or braking for an impending collision.

[0154] In at least one embodiment, when the vehicle 700 crosses a lane marking, the LDW system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to warn the driver. In at least one embodiment, the LDW system is inactive when the driver indicates an intentional lane departure, such as by activating a turn signal. In at least one embodiment, the LDW system can use a front-facing camera coupled to a dedicated processor, a DSP, an FPGA, and / or an ASIC, which is electrically coupled to provide driver feedback such as a display, a speaker, and / or a vibration component. In at least one embodiment, the LKA system is a variant of the LDW system. In at least one embodiment, if the vehicle 700 starts to leave the lane, the LKA system provides steering input or braking to correct the vehicle 700.

[0155] In at least one embodiment, the BSW system detects and warns a vehicle driver in a vehicle blind spot. In at least one embodiment, the BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. In at least one embodiment, the BSW system can provide an additional warning when the driver uses a turn signal. In at least one embodiment, the BSW system can use one or more rear-facing cameras and / or one or more RADAR sensors 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback such as a display, speaker, and / or vibration component.

[0156] In at least one embodiment, when an object is detected outside the rear camera range while the vehicle 700 is in reverse, the RCTW system can provide visual, audible, and / or tactile notifications. In at least one embodiment, the RCTW system includes an AEB system to ensure that the vehicle brakes are applied to avoid a collision. In at least one embodiment, the RCTW system can use one or more rear-facing RADAR sensors 760, which are coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to provide driver feedback such as a display, speaker, and / or vibration component.

[0157] In at least one embodiment, conventional ADAS systems may be prone to producing false positive results, which may annoy and distract the driver, but are generally not catastrophic because conventional ADAS systems warn the driver and allow the driver to decide whether a safety situation truly exists and take appropriate action. In at least one embodiment, in the case of conflicting results, the vehicle 700 itself decides whether to heed the results of the main computer or the secondary computer (e.g., the first controller or the second controller of the controller 736). For example, in at least one embodiment, the ADAS system 738 can be a backup and / or auxiliary computer for providing perception information to a backup computer rationality module. In at least one embodiment, the backup computer rationality monitor can run various software redundantly on hardware components to detect faults in perception and dynamic driving tasks. In at least one embodiment, the output from the ADAS system 738 can be provided to the monitoring MCU. In at least one embodiment, if the output from the main computer and the output from the auxiliary computer conflict, the supervisory MCU decides how to reconcile the conflict to ensure safe operation.

[0158] In at least one embodiment, the host computer may be configured to provide a confidence score to the supervisory MCU to indicate the host computer's confidence in the selected result. In at least one embodiment, if the confidence score exceeds a threshold, the supervisory MCU may follow the instructions of the host computer regardless of whether the secondary computer provides conflicting or inconsistent results. In at least one embodiment, in the case where the confidence score does not meet the threshold and the host computer and the secondary computer indicate different results (e.g., conflict), the supervisory MCU may arbitrate between the computers to determine an appropriate result.

[0159] In at least one embodiment, the supervisory MCU may be configured to run a neural network that is trained and configured to determine the conditions under which the secondary computer provides a false alarm based at least in part on the output from the host computer and the output from the secondary computer. In at least one embodiment, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot be trusted. For example, in at least one embodiment, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system identifies a metallic object that is not actually a hazard, such as a drain grate or manhole cover that would trigger an alarm. In at least one embodiment, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to override the LDW when there is a bicyclist or pedestrian present and lane departure is actually the safest course of action. In at least one embodiment, the supervisory MCU may include at least one of a DLA or GPU suitable for running a neural network with an associated memory. In at least one embodiment, the supervisory MCU may be included and / or be a component of one or more SoCs 704.

[0160] In at least one embodiment, the ADAS system 738 may include a secondary computer that performs ADAS functions using traditional computer vision rules. In at least one embodiment, the secondary computer may use classical computer vision rules (if-then), and the presence of the neural network in the supervisory MCU can improve reliability, safety, and performance. For example, in at least one embodiment, the diverse implementations and intentional non-identity make the overall system more fault-tolerant, especially for faults caused by software (or software-hardware interface) functions. For example, in at least one embodiment, if there is a software vulnerability or error in the software running on the host computer and the different software code running on the secondary computer provides a consistent overall result, the supervisory MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the host computer will not result in a significant error.

[0161] In at least one embodiment, the output of the ADAS system 738 can be input into the perception module of the host computer and / or the dynamic driving task module of the host computer. For example, in at least one embodiment, if the ADAS system 738 indicates a forward collision warning due to an object directly ahead, the perception block can use this information when identifying the object. In at least one embodiment, as described herein, the auxiliary computer can have its own neural network, which is trained to reduce the risk of false alarms.

[0162] In at least one embodiment, the vehicle 700 can further include an infotainment SoC 730 (e.g., in-vehicle infotainment system (IVI)). Although shown and described as an SoC, in at least one embodiment, the infotainment system SoC 730 may not be an SoC and can include, but is not limited to, two or more discrete components. In at least one embodiment, the infotainment SoC 730 can include, but is not limited to, a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movie, streaming, etc.), telephone (e.g., hands-free call), network connection (e.g., LTE, WiFi, etc.), and / or information services (e.g., navigation system, rear parking assist, radio data system, vehicle-related information such as fuel level, total covered distance, brake fuel level, oil level, door open / closed, air filter information, etc.) to the vehicle 700. For example, the infotainment SoC 730 can include a radio, disk player, navigation system, video player, USB and Bluetooth connections, car, in-vehicle entertainment system, WiFi, steering wheel audio control, hands-free voice control, head-up display (“HUD”), HMI display 734, telematics device, control panel (e.g., for controlling various components, features, and / or systems and / or interacting therewith), and / or other components. In at least one embodiment, the infotainment SoC 730 can further be used to provide information (e.g., visual and / or auditory) to the user of the vehicle 700, such as information from the ADAS system 738, autonomous driving information (such as planned vehicle maneuvers), trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0163] In at least one embodiment, the infotainment SoC 730 may include any number and type of GPU functionality. In at least one embodiment, the infotainment SoC 730 may communicate with other devices, systems, and / or components of the vehicle 700 via the bus 702. In at least one embodiment, the infotainment SoC 730 may be coupled to a monitoring MCU such that the GPU of the infotainment system may perform some autonomous driving functions in the event of a failure of the main controller 736 (e.g., the main computer and / or standby computer of the vehicle 700). In at least one embodiment, the infotainment SoC 730 may cause the vehicle 700 to enter the driver-to-safe-stop mode as described herein.

[0164] In at least one embodiment, the vehicle 700 may further include a dashboard 732 (e.g., a digital dashboard, an electronic dashboard, a digital instrument cluster, etc.). In at least one embodiment, the dashboard 732 may include, but is not limited to, a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). In at least one embodiment, the dashboard 732 may include, but is not limited to, any number and combination of a set of gauges, such as a speedometer, a fuel level, an oil pressure, a tachometer, an odometer, a turn indicator, a shift position indicator, one or more seatbelt warning lights, one or more parking brake warning lights, one or more engine fault lights, auxiliary restraint system (e.g., airbag) information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 730 and the dashboard 732. In at least one embodiment, the dashboard 732 may be included as part of the infotainment SoC 730, and vice versa.

[0165] According to one or more embodiments, the camera image compression / decompression component 115 may be used to provide a compression and / or decompression scheme for high dynamic range (HDR) images in image signal processing (ISP). The camera image compression / decompression component 115 may execute on a processor of the vehicle 700, such as on the CPU, GPU, SoC, etc. of the vehicle 700. Details regarding the camera image compression / decompression component 115 are provided herein in connection with Figure 1 provided. In at least one embodiment, the camera image compression / decompression component 115 may be used in Figure 7B a system to perform compression and / or decompression of HDR images as described herein.

[0166] Computer system

[0167] Figure 8is a block diagram showing an exemplary computer system according to at least one embodiment, which exemplary computer system can be a system with interconnected devices and components, a system-on-chip (SOC), or some combination thereof that forms a processor, which processor can include execution units to execute instructions. In an embodiment, the computer system can be used to determine an optimal power curve and PWL function for compression and / or decompression. For example, the computer system can perform curve fitting operations, for example, to determine one or more PWL functions for compression and / or decompression, which map to a power curve and / or cause a compressed image to be compressed according to and / or in correspondence with the power curve. In an embodiment, the computer system can perform one or more operations of an ISP. For example, the computer system can include a camera compression / decompression component of the ISP, which can decompress (e.g., partially decompress) an incoming compressed image according to the embodiments described herein, and optionally perform one or more operations of the ISP, and then optionally render and / or process the image. Once the image is decompressed (e.g., partially decompressed) and fully processed by the ISP, the image can be used for one or more purposes. For example, the image can be input into one or more trained machine learning models, which can perform object determination and / or recognition and / or one or more other operations.

[0168] In at least one embodiment, according to the present disclosure, for example, the embodiments described herein, the computer system 800 can include, but is not limited to, components such as a processor 802, whose execution units include logic to execute algorithms for processing data. In at least one embodiment, the computer system 800 can include a processor, such as those available from Intel Corporation of Santa Clara, California processor families, Xeon TM , XScale TM and / or StrongARM TM , Core TM or Nervana TM microprocessors, although other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, etc.) can also be used. In at least one embodiment, the computer system 800 can execute a version of the WINDOWS operating system available from Microsoft Corporation of Redmond, Wash., although other operating systems (such as UNIX and Linux), embedded software, and / or graphical user interfaces can also be used.

[0169] Embodiments can be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants (“PDAs”), and handheld PCs. In at least one embodiment, the embedded application can include a microcontroller, a DSP, a system on a chip, a network computer (“NetPC”), a set-top box, a network hub, a wide area network (“WAN”) switch, or any other system that can execute one or more instructions according to at least one embodiment.

[0170] In at least one embodiment, computer system 800 can include, but is not limited to, a processor 802, which can include, but is not limited to, one or more execution units 808 to perform machine learning model training and / or inference according to the techniques described herein. In at least one embodiment, computer system 800 is a single-processor desktop or server system, but in another embodiment, computer system 800 can be a multi-processor system. In at least one embodiment, processor 802 can include, but is not limited to, a complex instruction set computer (“CISC”) microprocessor, a reduced instruction set computing (“RISC”) microprocessor, a very long instruction word (“VLIW”) microprocessor, a processor implementing an instruction set combination, or any other processor device, such as a digital signal processor. In at least one embodiment, processor 802 can be coupled to a processor bus 810, which can transfer data signals between processor 802 and other components in computer system 800.

[0171] In at least one embodiment, processor 802 can include, but is not limited to, a level 1 (“L1”) internal cache memory (“cache”) 804. In at least one embodiment, processor 802 can have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory can reside outside of processor 802. Other embodiments can also include a combination of internal and external caches, depending on the specific implementation and requirements. In at least one embodiment, register file 806 can store different types of data in various registers, including, but not limited to, integer registers, floating-point registers, status registers, and instruction pointer registers.

[0172] In at least one embodiment, an execution unit 808, including but not limited to logic for performing integer and floating point operations, is also located in the processor 802. In at least one embodiment, the processor 802 may also include a microcode (“ucode”) read only memory (“ROM”) for storing microcode for certain macroinstructions. In at least one embodiment, the execution unit 808 may include logic for processing a packet instruction set 809. In at least one embodiment, by including the packet instruction set 809 in the instruction set of a general purpose processor, and the associated circuitry for the instructions to be executed, operations used by many multimedia applications can be performed using the packet data in the processor 802. In at least one embodiment, operations can be performed on the packet data by using the full width of the data bus of the processor to accelerate and more efficiently execute many multimedia applications, which may not require transferring smaller data units on the data bus of the processor to perform one or more operations on one data element at a time.

[0173] In at least one embodiment, the execution unit 808 may also be used in a microcontroller, an embedded processor, a graphics device, a DSP, and other types of logic circuits. In at least one embodiment, the computer system 800 may include but not limited to a memory 820. In at least one embodiment, the memory 820 may be a dynamic random access memory (“DRAM”) device, a static random access memory (“SRAM”) device, a flash memory device, or another storage device. In at least one embodiment, the memory 820 may store instructions 819 and / or data 821 represented by data signals that can be executed by the processor 802.

[0174] In at least one embodiment, the system logic chip can be coupled to the processor bus 810 and the memory 820. In at least one embodiment, the system logic chip can include, but is not limited to, a Memory Controller Hub (“MCH”) 816, and the processor 802 can communicate with the MCH 816 via the processor bus 810. In at least one embodiment, the MCH 816 can provide a high-bandwidth memory path 818 to the memory 820 for instruction and data storage and for storage of graphics commands, data, and textures. In at least one embodiment, the MCH 816 can initiate data signals among the processor 802, the memory 820, and other components in the computer system 800, and bridge data signals among the processor bus 810, the memory 820, and the system I / O interface 822. In at least one embodiment, the system logic chip can provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH 816 can be coupled to the memory 820 via the high-bandwidth memory path 818, and the graphics / video card 812 can be coupled to the MCH 816 via an Accelerated Graphics Port (“AGP”) interconnect 814.

[0175] In at least one embodiment, the computer system 800 can use the system I / O interface 822 as a proprietary hub interface bus to couple the MCH 816 to an I / O Controller Hub (“ICH”) 830. In at least one embodiment, the ICH 830 can provide direct connections to certain I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus can include, but is not limited to, a high-speed I / O bus for connecting peripheral devices to the memory 820, the chipset, and the processor 802. Examples can include, but are not limited to, an audio controller 829, a Firmware Hub (“Flash BIOS”) 828, a wireless transceiver 826, a data storage 824, a legacy I / O controller 823 that includes a user input and a keyboard interface, a serial expansion port 827 (such as a USB port), and a network controller 834. In at least one embodiment, the data storage 824 can include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash device, or other mass storage devices.

[0176] In at least one embodiment, Figure 8 a system including interconnected hardware devices or “chips” is shown, while in other embodiments, Figure 8 a SoC can be shown. In at least one embodiment, Figure 8The devices shown can be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of computer system 800 are interconnected using Compute Express Link (CXL) interconnects.

[0177] According to one or more embodiments, the camera image compression / decompression component 115 can be used to provide compression and / or decompression schemes for high dynamic range (HDR) images in image signal processing (ISP). Details regarding the camera image compression / decompression component 115 are provided herein in conjunction with Figure 1 provided. In at least one embodiment, the camera image compression / decompression component 115 can be used in Figure 8 a system to perform compression and / or decompression of HDR images as described herein.

[0178] Figure 9 is a block diagram showing an electronic device 900 for utilizing a processor 910 according to at least one embodiment. In at least one embodiment, the electronic device 900 can be, for example but not limited to, a laptop computer, a tower server, a rack server, a blade server, a notebook computer, a desktop computer, a tablet computer, a mobile device, a phone, an embedded computer, or any other suitable electronic device.

[0179] In at least one embodiment, the electronic device 900 can include, but is not limited to, a processor 910 communicatively coupled to any suitable number or type of components, peripherals, modules, or devices. In at least one embodiment, the processor 910 is coupled using a bus or interface, such as an I 2 2C bus, a system management bus (“SMBus”), a low pin count (LPC) bus, a serial peripheral interface (“SPI”), a high definition audio (“HDA”) bus, a serial advanced technology attachment (“SATA”) bus, a universal serial bus (“USB”) (versions 1, 2, 3, etc.), or a universal asynchronous receiver / transmitter (“UART”) bus. In at least one embodiment, Figure 9 shows a system that includes interconnected hardware devices or “chips,” while in other embodiments, Figure 9 an exemplary SoC can be shown. In at least one embodiment, Figure 9 the devices shown can be interconnected with proprietary interconnect lines, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, Figure 9 one or more components of are interconnected using Compute Express Link (CXL) interconnect lines.

[0180] In at least one embodiment, Figure 9It may include a display 924, a touch screen 925, a touchpad 930, a near field communication unit (“NFC”) 945, a sensor hub 940, a thermal sensor 946, a fast chipset (“EC”) 935, a trusted platform module (“TPM”) 938, a BIOS / firmware / flash (“BIOS, FW Flash”) 922, a DSP 960, a drive 920 (such as a solid state disk (“SSD”) or a hard disk drive (“HDD”)), a wireless local area network unit (“WLAN”) 950, a Bluetooth unit 952, a wireless wide area network unit (“WWAN”) 956, a global positioning system (GPS) unit 955, a camera (“USB 3.0 camera”) 954 (such as a USB 3.0 camera) and / or a low power double data rate (“LPDDR”) memory unit (“LPDDR3”) 915 implemented in accordance with, for example, the LPDDR3 standard. These components may each be implemented in any suitable manner.

[0181] In at least one embodiment, other components may be communicatively coupled to the processor 910 via the components described herein. In at least one embodiment, an accelerometer 941, an ambient light sensor (“ALS”) 942, a compass 943, and a gyroscope 944 may be communicatively coupled to the sensor hub 940. In at least one embodiment, a thermal sensor 939, a fan 937, a keyboard 936, and a touchpad 930 may be communicatively coupled to the EC 935. In at least one embodiment, a speaker 963, headphones 964, and a microphone (“mic”) 965 may be communicatively coupled to an audio unit (“audio codec and class D amplifier”) 962, which may in turn be communicatively coupled to the DSP 960. In at least one embodiment, the audio unit 962 may include, for example but not limited to, an audio encoder / decoder (“codec”) and a class D amplifier. In at least one embodiment, a subscriber identity module (“SIM”) 957 may be communicatively coupled to the WWAN unit 956. In at least one embodiment, components (such as the WLAN unit 950, the Bluetooth unit 952, and the WWAN unit 956) may be implemented in a next generation form factor (NGFF).

[0182] According to one or more embodiments, a camera image compression / decompression component 115 may be used to provide a compression and / or decompression scheme for high dynamic range (HDR) images in image signal processing (ISP). Details regarding the camera image compression / decompression component 115 are provided herein in connection with Figure 1 provided. In at least one embodiment, the camera image compression / decompression component 115 may be used in Figure 9 a system to perform compression and decompression of HDR images as described herein.

[0183] Figure 10It is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, system 1000 includes one or more processors 1002 and one or more graphics processors 1008, and can be a single-processor desktop system, a multi-processor workstation system, or a server system with a large number of processors 1002 or processor cores 1007. In at least one embodiment, system 1000 is a processing platform integrated within a system-on-chip (SoC) integrated circuit for use in mobile, handheld, or embedded devices.

[0184] In at least one embodiment, system 1000 can be included in or incorporated into a server-based gaming platform, including a game console such as a game and media console, a mobile game console, a handheld game console, or an online game console. In at least one embodiment, system 1000 is a mobile phone, a smartphone, a tablet computing device, or a mobile Internet device. In at least one embodiment, the processing system 1000 can also be coupled to or integrated within a wearable device, such as a smartwatch wearable device, a smart glasses device, an augmented reality device, or a virtual reality device. In at least one embodiment, the processing system 1000 is a television or set-top box device having one or more processors 1002 and a graphical interface generated by one or more graphics processors 1008.

[0185] In at least one embodiment, each of the one or more processors 1002 includes one or more processor cores 1007 to process instructions that, when executed, perform operations for system and user software. In at least one embodiment, each of the one or more processor cores 1007 is configured to process a specific instruction sequence 1009. In at least one embodiment, the instruction sequence 1009 can facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or computing via very long instruction word (VLIW). In at least one embodiment, the processor cores 1007 can each process different instruction sequences 1009, which can include instructions that help to emulate other instruction sequences. In at least one embodiment, the processor cores 1007 can also include other processing devices, such as a digital signal processor (DSP).

[0186] In at least one embodiment, the processor 1002 includes a cache memory 1004. In at least one embodiment, the processor 1002 may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory is shared among the various components of the processor 1002. In at least one embodiment, the processor 1002 also uses an external cache (e.g., a level 3 (L3) cache or a last-level cache (LLC)) (not shown), and this external cache can be shared among the processor cores 1007 using known cache coherence techniques. In at least one embodiment, the processor 1002 further includes a register file 1006, and the processor may include different types of registers for storing different types of data (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers). In at least one embodiment, the register file 1006 may include general-purpose registers or other registers.

[0187] In at least one embodiment, one or more processors 1002 are coupled to one or more interface buses 1010 to transfer communication signals, such as address, data, or control signals, between the processor 1002 and other components in the system 1000. In at least one embodiment, the interface bus 1010 may be a processor bus, such as a version of the direct media interface (DMI) bus, in one embodiment. In at least one embodiment, the interface bus 1010 is not limited to the DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), a memory bus, or other types of interface buses. In at least one embodiment, the processor 1002 includes an integrated memory controller 1016 and a platform controller hub 1030. In at least one embodiment, the memory controller 1016 facilitates communication between the memory device and other components of the processing system 1000, while the platform controller hub (PCH) 1030 provides connections to input / output (I / O) devices via a local I / O bus.

[0188] In at least one embodiment, the memory device 1020 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or have suitable performance to be used as a processor memory. In at least one embodiment, the storage device 1020 may be used as the system memory of the processing system 1000 to store data 1022 and instructions 1021 for use when one or more processors 1002 execute an application or process. In at least one embodiment, the memory controller 1016 is also coupled to an optional external graphics processor 1012, which may communicate with one or more of the graphics processors 1008 in the processor 1002 to perform graphics and media operations. In at least one embodiment, the display device 1011 may be connected to the processor 1002. In at least one embodiment, the display device 1011 may include one or more of an internal display device, such as in a mobile electronic device or a laptop device, or an external display device connected via a display interface (such as DisplayPort, etc.). In at least one embodiment, the display device 1011 may include a head-mounted display (HMD), such as a stereoscopic display device for virtual reality (VR) applications or augmented reality (AR) applications.

[0189] In at least one embodiment, the platform controller hub 1030 enables peripheral devices to be connected to the storage device 1020 and the processor 1002 via a high-speed I / O bus. In at least one embodiment, the I / O peripheral devices include, but are not limited to, an audio controller 1046, a network controller 1034, a firmware interface 1028, a wireless transceiver 1026, a touch sensor 1025, a data storage device 1024 (e.g., a hard disk drive, a flash memory, etc.). In at least one embodiment, the data storage device 1024 can be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a Peripheral Component Interconnect bus (e.g., PCI, PCIe). In at least one embodiment, the touch sensor 1025 can include a touch screen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, the wireless transceiver 1026 can be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver, such as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1028 enables communication with the system firmware and can be, for example, a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, the network controller 1034 can enable a network connection to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to the interface bus 1010. In at least one embodiment, the audio controller 1046 is a multi-channel high-definition audio controller. In at least one embodiment, the processing system 1000 includes an optional legacy I / O controller 1040 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system 1000. In at least one embodiment, the platform controller hub 1030 can also be connected to one or more Universal Serial Bus (USB) controllers 1042, which connect input devices, such as a keyboard and mouse 1043 combination, a camera 1044, or other USB input devices.

[0190] In at least one embodiment, instances of the memory controller 1016 and the platform controller hub 1030 can be integrated into a discrete external graphics processor, such as the external graphics processor 1012. In at least one embodiment, the platform controller hub 1030 and / or the memory controller 1016 can be external to one or more processors 1002. For example, in at least one embodiment, the system 1000 can include an external memory controller 1016 and a platform controller hub 1030, which can be configured as a memory controller hub and a peripheral controller hub in a system chipset that communicates with the processor 1002.

[0191] According to one or more embodiments, the camera image compression component 115 can be used to provide compression and / or decompression solutions for high dynamic range (HDR) images in image signal processing (ISP). Details regarding the camera image compression component 115 are provided herein in connection with Figure 1 provided. In at least one embodiment, the camera image compression component 115 can be used in the system 1000 to perform compression and decompression of HDR images as described herein.

[0192] Other variations are within the spirit of the present disclosure. Thus, although the disclosed techniques are susceptible to various modifications and alternative configurations, certain of its illustrated embodiments are shown in the drawings and have been described in detail above. However, it should be understood that the intention is not to limit the disclosure to the one or more specific forms disclosed, but rather, the intention is to cover all modifications, alternative configurations, and equivalents falling within the spirit and scope of the present disclosure as defined by the appended claims.

[0193] Unless otherwise stated or clearly contradicted by the context, the use of the terms "a," "an," "the," and similar referents in the context of describing the disclosed embodiments (especially in the context of the appended claims) should be construed to cover both the singular and the plural, rather than as a definition of the terms. Unless otherwise stated, the terms "comprising," "having," "including," and "containing" should be construed as open-ended terms (meaning "including but not limited to"). The term "connected" (when not otherwise modified, referring to a physical connection) should be construed to mean included in whole or in part, attached to, or joined together, even if there are some intervening elements. Unless otherwise indicated herein, references to numerical ranges in this document are merely intended as a shorthand method of referring separately to each individual value falling within the range, and each individual value is incorporated into the specification as if it were recited herein individually. In at least one embodiment, unless otherwise indicated or clearly contradicted by the context, the use of the term "set" (e.g., "set of items") or "subset" should be construed to mean a non-empty collection including one or more members. Additionally, unless otherwise indicated or clearly contradicted by the context, a "subset" of a corresponding set does not necessarily mean a proper subset of the corresponding set, but rather, the subset and the corresponding set can be equal.

[0194] Unless otherwise expressly indicated or clearly contradicted by context, a conjunctive phrase such as "at least one of A, B, and C" or "at least one of A, B and C" is understood in context to typically mean an item, term, etc., which can be A or B or C, or any non-empty subset of the set A and B and C. For example, in an illustrative example of a set having three members, the conjunctive phrases "at least one of A, B, and C" and "at least one of A, B and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctive language is not generally intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by context, the term "plural" denotes a plural state (e.g., "a plurality of items" means a plurality of items). In at least one embodiment, the number of items in a plurality of items is at least two, but may be more if expressly indicated or indicated by context. Further, unless otherwise stated or clear from context, the phrase "based on" means "at least partially based on" rather than "based solely on".

[0195] Unless otherwise indicated herein or clearly contradicted by context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations and / or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors by hardware or a combination thereof. In at least one embodiment, the code is stored, for example, in the form of a computer program on a computer-readable storage medium, the computer program including a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transitory signals (e.g., propagated transient electrical or electromagnetic transmissions), but includes non-transitory data storage circuits (e.g., buffers, caches, and queues). In at least one embodiment, the code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, the executable instructions, when executed by one or more processors of a computer system (i.e., as a result of being executed), cause the computer system to perform the operations described herein. In at least one embodiment, a set of non-transitory computer-readable storage media includes a plurality of non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media in the plurality of non-transitory computer-readable storage media lack all of the code, but the plurality of non-transitory computer-readable storage media together store all of the code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors, e.g., the non-transitory computer-readable storage medium stores the instructions and a main central processing unit (“CPU”) executes some of the instructions while a graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of a computer system have separate processors and different processors execute different subsets of the instructions.

[0196] Thus, in at least one embodiment, a computer system is configured to implement one or more services that perform, individually or jointly, the operations of the processes described herein, and such a computer system is configured with suitable hardware and / or software enabling the implementation of the operations. Additionally, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment is a distributed computer system that includes a plurality of devices operating in different ways such that the distributed computer system performs the operations described herein and such that a single device does not perform all of the operations.

[0197] Any and all uses of examples or exemplary language (e.g., "such as") provided herein are for illustrative purposes only to better clarify embodiments of the present disclosure and do not limit the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating that any non-claimed element is essential for practicing the disclosure.

[0198] All references cited herein, including publications, patent applications, and patents, are incorporated herein by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and its entire content were set forth herein.

[0199] In the specification and claims, the terms "coupled" and "connected" and their derivatives may be used. It should be understood that these terms are not intended as synonyms for each other. Rather, in a particular example, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other but still cooperate or interact with each other.

[0200] Unless otherwise expressly stated, it is understood that throughout the specification, terms such as "process", "compute", "calculate", "determine", etc., refer to actions and / or processes of a computer or computing system or similar electronic computing device that process and / or transform data represented as a physical quantity (e.g., electrons) in the registers and / or memory of the computing system into other data similarly represented as a physical quantity in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0201] In a similar manner, the term "processor" may refer to any device or portion of a memory that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that may be stored in the registers and / or memory. As a non-limiting example, a "processor" may be a CPU or GPU. A "computing platform" may include one or more processors. As used herein, a "software" process may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Similarly, each process may refer to multiple processes that execute instructions sequentially or in parallel, continuously or intermittently. In at least one embodiment, the terms "system" and "method" may be used interchangeably herein as long as the system can embody one or more methods and the method can be considered a system.

[0202] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data may be accomplished in a variety of ways, such as by receiving data as a parameter of a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data may be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data may be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transferring, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transferring, sending, or presenting analog or digital data may be implemented by transmitting the data as an input or output parameter of a function call, an application programming interface, or a parameter of an interprocess communication mechanism.

[0203] Although the description herein sets forth example embodiments of the described techniques, other architectures may be used to implement the described functionality and are intended to fall within the scope of the present disclosure. Additionally, although specific assignments of responsibilities may have been defined above for purposes of description, the various functions and responsibilities may be assigned and partitioned differently depending on circumstances.

[0204] Moreover, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter claimed in the appended claims need not be limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claims.

Claims

1. A method for image processing, comprising: receiving, at one or more processors associated with an Image Signal Processing (ISP) pipeline, an image generated by an image sensor and captured at a first bit depth associated with the image sensor, wherein the image is compressed to a second bit depth lower than the first bit depth, and wherein the ISP is associated with a third bit depth lower than the first bit depth and higher than the second bit depth; and decompressing the compressed image according to a power curve to generate a partially decompressed image having the third bit depth, wherein multiple regions of the partially decompressed image are decompressed to at least two decompression levels based on corresponding pixel values of each of the multiple regions.

2. The method according to claim 1, wherein the power curve includes a linear segment extending to a pixel value threshold for linearly processing pixel values below the pixel value threshold, and wherein processing pixel values based on the linear segment of the power curve comprises: avoiding the compressed state of the pixel values to preserve the signal-to-noise ratio (SNR) of the pixel values.

3. The method according to claim 1, wherein decompressing the compressed image according to the power curve preserves the color ratio of the compressed image and keeps the signal-to-noise ratio of the compressed image below a signal-to-noise ratio threshold.

4. The method according to claim 1, wherein the power curve corresponds to a piecewise linear Piecewise Linear (PWL) spline function that includes multiple inflection points.

5. The method according to claim 4, further comprising: determining the power curve based on the multiple inflection points using curve fitting techniques.

6. The method according to claim 1, wherein decompressing the compressed image further comprises: applying a first decompression amount to a first region of the compressed image having a first pixel value; and applying a second decompression amount to a second region of the compressed image having a higher second pixel value, wherein after decompression the second region has a greater residual compression than the first region.

7. The method according to claim 6, wherein the first pixel value of the first region corresponds to a first luminance value of the first region, and the second pixel value of the second region corresponds to a second luminance value of the second region, and wherein the second luminance value is higher than the first luminance value.

8. The method according to claim 1, wherein in order to compress the image from the first bit depth to the second bit depth, the image sensor is configured to: apply a first compression amount to a first region of the captured image having a first pixel value according to the power curve or a second power curve; and apply a second compression amount to a second portion of the captured image having a higher second pixel value according to the power curve or the second power curve, wherein the first compression amount is lower than the second compression amount.

9. The method according to claim 8, wherein, in order to compress the image from the first bit depth to the second bit depth, the image sensor is used to use a piecewise linear PWL spline function associated with the power curve or the second power curve, and the PWL spline function includes a plurality of inflection points.

10. The method according to claim 9, wherein the image sensor is used to use a curve fitting technique to determine the power curve or the second power curve based on the plurality of inflection points.

11. A system for image processing, comprising: a memory device; and a processing device operatively coupled to the memory device to: determine a first bit depth associated with a sensor of a camera; determine a second bit depth associated with an image signal processing ISP pipeline associated with the camera; and in response to determining that the first bit depth is different from the second bit depth: determine a third bit depth associated with a compressed image generated by the sensor of the camera, determine a plurality of inflection points corresponding to a piecewise linear PWL spline function of the sensor of the camera based on the third bit depth and the second bit depth of the ISP, and determine a power curve for decompressing the compressed image to generate a partially decompressed image having the second bit depth.

12. The system according to claim 11, wherein based on the corresponding pixel values of each region in a plurality of regions, the plurality of regions of the partially decompressed image are decompressed at separate decompression levels in a manner that preserves the color ratio of the compressed image and keeps the signal-to-noise ratio of the compressed image lower than a signal-to-noise ratio threshold.

13. The system according to claim 11, wherein the power curve includes a linear segment for processing pixel values below a pixel value threshold, and wherein the linear segment preserves the color ratio and the signal-to-noise ratio for low light conditions.

14. The system according to claim 11, wherein the compressed image is captured at the first bit depth associated with the sensor and compressed to the third bit depth lower than the first bit depth, and wherein the second bit depth of the ISP is lower than the first bit depth and higher than the third bit depth.

15. The system according to claim 11, wherein to decompress the compressed image, the processing device is further used to: apply a first decompression amount to a first region of the compressed image having a first pixel value; and apply a second decompression amount to a second region of the compressed image having a higher second pixel value, wherein after decompression the second region has a greater compression than the first region.

16. An electronic device, comprising: a memory; and one or more processors operatively coupled to the memory to perform operations, the operations including: At one or more processors associated with an Image Signal Processing (ISP) pipeline, a compressed image generated by an image sensor is received, where the compressed image is captured at a first bit depth associated with the image sensor and compressed to a second bit depth lower than the first bit depth, and where the ISP is associated with a third bit depth lower than the first bit depth and higher than the second bit depth; and The compressed image is decompressed according to a power curve to generate a partially decompressed image having the third bit depth, where multiple regions of the partially decompressed image are decompressed at separate decompression levels based on corresponding pixel values of each of the multiple regions.

17. The electronic device according to claim 16, wherein the power curve includes a linear segment extending to a pixel value threshold for linearly processing pixel values below the pixel value threshold.

18. The electronic device according to claim 17, wherein processing pixel values based on the linear segment of the power curve comprises: Avoiding the compressed state of the pixel values to preserve the Signal-to-Noise Ratio (SNR) of the pixel values.

19. The electronic device according to claim 16, wherein the power curve corresponds to a piecewise linear Piece-Wise Linear (PWL) spline function that includes multiple inflection points.

20. The electronic device according to claim 19, wherein the one or more processors are configured to perform further operations, and the further operations also comprise: Determining the power curve based on the multiple inflection points using curve fitting techniques.

21. The electronic device according to claim 16, wherein decompressing the compressed image further comprises: Applying a first decompression amount to a first region of the compressed image having a first pixel value; and Applying a second decompression rate to a second region of the compressed image having a higher second pixel value, where after decompression the second region has a greater residual compression than the first region.

22. The electronic device according to claim 21, wherein the first pixel value of the first region corresponds to a first luminance value of the first region, and the second pixel value of the second region corresponds to a second luminance value of the second region, and wherein the second luminance value is higher than the first luminance value.

23. The electronic device according to claim 16, wherein in order to compress the compressed image from the first bit depth to the third bit depth, the image sensor is configured to: Apply a first compression amount to a first region of the captured image having a first pixel value; and Apply a second compression amount to a second portion of the captured image having a higher second pixel value, where the first compression amount is lower than the second compression amount.

24. The electronic device according to claim 16, wherein in order to compress the compressed image from the first bit depth to the third bit depth, the image sensor is configured to use a piecewise linear Piece-Wise Linear (PWL) spline function associated with the power curve or a second power curve, the PWL spline function including multiple inflection points.

25. The electronic device according to claim 24, wherein the image sensor is configured to determine the power curve or the second power curve based on the plurality of inflection points by using a curve fitting technique.

Images