RGB-IR data processing for autonomous systems and applications

Abstract

To provide a system configured to process image data including infrared values and visible light values.SOLUTION: A system determines how IR data and visible light data should be blended together to produce an optimal image according to the current light level. The system calculates a scene detection value for image data based on a comparison between an infrared value and a visible light value. The system can then determine an amount of infrared correction, a color correction factor, a color saturation factor, and the like to apply to the image data. The system then transforms the image data based on the amount of infrared correction, color correction factor, color saturation factor, and the like. The transformed image data contains more information about a low-light scene than previously available information, thereby producing higher quality images.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The embodiments of this disclosure relate to image processing in general, and more particularly to the dynamic control of image generation based on IR data and visible light data (e.g., RGB data) from red, green, and blue (RGB)-infrared (IR) sensors. [Background technology]

[0002] RGB-IR sensors are used in imaging systems, such as those used in driver and occupant monitoring systems (for example, in automobiles). In RGB-IR sensors, the color signals become less useful as the amount of visible light in the scene decreases. Systems using RGB-IR sensors generally have a visible light threshold; above this threshold, data from the RGB sensors is used, and below it, data from the IR sensors is used. RGB-IR systems typically have one-quarter the number of IR sensor pixels as RGB sensor pixels. Therefore, the signals from IR sensor pixels have one-quarter the resolution of the signals from RGB sensor pixels. Such reduced resolution from IR sensor pixels diminishes the quality and usefulness of the images produced in low-light settings (for example, at night). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (eg, Standard No. J3016-201806, published on June 15, 2018, Standard No. J3016-201609, published on September 30, 2016, and previous and future versions of this standard)

[0004] This disclosure will be better understood from the detailed description given below and from the accompanying drawings of various embodiments of this disclosure. However, the drawings should not be taken as limiting this disclosure to any particular embodiment, but are for illustrative and illustrative purposes only. [Brief explanation of the drawing]

[0005] [Figure 1] This figure shows an exemplary computing environment, including techniques for dynamically adjusting image data, including infrared and visible light values, according to some embodiments of the present disclosure. [Figure 2] This figure shows a conceptual IR subtraction factor curve based on several embodiments of the present disclosure. [Figure 3] A lookup table for determining one or more quantities used to adjust an image based on scene detection values, as described in some embodiments of the present disclosure. [Figure 4] This is a flowchart of a method for adjusting image data, including infrared and visible light values, to generate an updated image, according to some embodiments of the present disclosure. [Figure 5] This is a flowchart of a method for processing a transformed image (transformed, for example, according to the method in Figure 4) using a trained machine learning model, according to some embodiments of the present disclosure. [Figure 6] is a flowchart of a method for displaying a transformed image (e.g., transformed according to the method of Figure 4), in accordance with some embodiments of the present disclosure. [Figure 7] is a diagram illustrating an exemplary computing device including a scene analyzer for modifying images, in accordance with some embodiments of the present disclosure. [Figure 8A] is a diagram illustrating an example of an autonomous vehicle, in accordance with at least one embodiment. [Figure 8B] is a diagram illustrating an example of camera locations and fields of view for the autonomous vehicle of Figure 8A, in accordance with at least one embodiment. [Figure 8C] is a block diagram illustrating an exemplary system architecture for the autonomous vehicle of Figure 8A, in accordance with at least one embodiment. [Figure 8D] is a diagram illustrating a system for communication between one or more cloud-based servers and the autonomous vehicle of Figure 8A, in accordance with at least one embodiment. [Figure 9A] is a diagram illustrating inference and / or training logic, in accordance with at least one embodiment. [Figure 9B] is a diagram illustrating inference and / or training logic, in accordance with at least one embodiment. [Figure 10] is a diagram illustrating an exemplary data center system, in accordance with at least one embodiment. [Figure 11] is a diagram illustrating a computer system, in accordance with at least one embodiment. [Figure 12] is a diagram illustrating a computer system, in accordance with at least one embodiment. [Figure 13] is a diagram illustrating at least a portion of a graphics processor, in accordance with one or more embodiments. [Figure 14] is a diagram illustrating at least a portion of a graphics processor, in accordance with one or more embodiments. [Description of Embodiments]

[0006] Occupant monitoring systems generally use monochrome infrared (IR) cameras, and more recently have begun to use color IR cameras (i.e., red, green, blue, infrared (RGB-IR) cameras). The use of IR allows the camera to function at night by using IR illumination to brighten the target area (e.g., the driver in the cabin). The image sensor of an RGB-IR camera is generally configured such that a large portion of the pixel sensors detect visible light, and a smaller portion detect IR (e.g., one quarter of the pixel sensors detect IR). When the scene becomes darker (e.g., at night), the pixel sensors for visible light are less effective, and conventional systems switch to using only IR pixel sensors. This switch from color IR to monochrome IR is abrupt and can cause the image quality and resolution to substantially decrease (e.g., reduce to one quarter of the resolution). The abrupt switching and change in resolution can be particularly problematic for computer vision applications trained using specific image quality and resolution, which may not cope well with changes between brighter scenes and darker scenes.

[0007] Embodiments of the present disclosure include techniques for an Image Signal Processing (ISP) pipeline that better copes with transitions between bright scenes and dark scenes. The present technology can use information from visible light pixel sensors to keep the image resolution for dark scenes higher than was conventionally achievable, and is better at transitions between bright scenes and dark scenes than previous solutions. For example, IR signals measured by non-IR pixels may be used under some lighting conditions, and the amount of IR signal to be used may vary based on the lighting conditions. The present technology can gently adapt the image capture and rendering of the combined near-IR light and visible light signals in response to changes in the mixture of ambient light and emitted near-IR light.

[0008] This technology analyzes images from an RGB-IR camera to determine a scene detection value (which may indicate, for example, the brightness of the scene). In one example, the scene detection value may be based on the ratio of the average intensity of IR (such as that emitted by an IR emitter associated with the RGB-IR camera) to the average intensity of visible light. Generally, the ratio of IR intensity to visible light intensity is lower during the day (e.g., a ratio of 1:100) because the intensity of IR illumination is lower than the intensity of visible light. Conversely, this ratio is generally much higher at night (e.g., a ratio of 1:1) because the intensity of IR illumination is closer to the intensity of visible light. Also, pixels configured to detect visible light generally detect IR. Therefore, generally, at night or in other low-light settings (e.g., in a tunnel) where there is little background ambient light (e.g., from the sun), IR pixels and visible light pixels (e.g., G pixels) may detect the same IR output by the RGB-IR camera system, resulting in an IR to visible light ratio of approximately 1:1.

[0009] An RGB-IR camera may include an RGB-IR color filter array (CFA). An RGB-IR CFA is a variation of a standard Bayer CFA in which some of the positions in the color filter array that normally allow red, green, or blue light to pass through are replaced with IR filters that allow IR wavelengths (e.g., near-IR wavelengths) to pass through and block visible light. For example, this can be done by taking a 4x4 area of ​​the Bayer sensor, replacing two of the red filters and two of the blue filters with IR filters, and creating a repeating pattern across the sensor. In a typical design, the camera would have an IR cut filter with a narrow bandpass that matches the emission spike of the IR emitter (e.g., at 940 nm) of the RGB-IR camera.

[0010] R, G, and B color filters are generally transparent to IR emission. As a result, IR rays leak equally into all channels of the sensor. Traditionally, to achieve high-quality images for human visualization and / or use in machine learning systems operating on image systems, the IR signal measured on an IR pixel is subtracted from the R, G, and B pixels. As visible light in the scene is reduced compared to IR rays (e.g., from an IR emitter), the resulting color signal decreases, becoming noisier and potentially black. Signals from IR-only pixels are generally separated in the image signal processor to provide a 1 / 4 resolution image that can be used for machine vision and / or display to a user. This image is usable in both bright and dark conditions, but the lower resolution reduces potential machine vision performance compared to a monochromatic IR sensor (e.g., where all pixels are IR sensor pixels).

[0011] Systems using RGB-IR cameras (for example, occupant monitoring systems (OMS) and / or driver monitoring systems (DMS) in automobiles) face the problem that as the amount of visible light in the scene decreases, the color signals become less useful. Therefore, conventional systems using RGB-IR cameras apply a visible light threshold, and when the amount of available visible light falls below the threshold, such systems transition to using the IR pixel sensors of the RGB-IR camera. However, signals from IR-only pixels in a typical RGB-IR sensor result in a 1 / 4 resolution image, producing a lower quality image and, consequently, reducing the usefulness of monochromatic IR information. In contrast to conventional systems using RGB-IR cameras, the systems using RGB-IR cameras described in the embodiments herein can maintain full resolution regardless of lighting conditions. Maintaining full resolution rather than fluctuating between full resolution and 1 / 4 resolution increases the amount of data being processed in the resulting image, improves image quality, improves the efficiency of training these systems, saves time, and increases the potential efficiency of the system.

[0012] Traditionally, all intensity for visible light pixels due to IR is filtered out. However, in this embodiment, processing logic dynamically determines, based on a determined scene detection value, how much of the intensity for visible light pixels should be filtered out. By using visible light pixels to detect IR signals, the number of pixels used in low-light conditions increases, and therefore the resolution of the image produced in low-light conditions also increases. As the scene becomes darker, more and more of the signal for visible light pixels due to IR is used. This can result in incorrect white balance, incorrect colors, incorrect color saturation, etc., if predetermined values ​​such as static white balance correction, color correction, and color saturation are used for an RGB-IR camera. Therefore, in addition to using the scene detection value to determine the amount of infrared correction (e.g., how much IR signal should be subtracted from the measurement for visible light pixels), the scene detection value and / or the amount of infrared correction may be used to dynamically determine the white balance correction factor, color correction factor, color saturation factor, etc. Each of these factors will shift depending on how much of the IR signal is filtered out of the visible light pixel value.

[0013] Therefore, once the scene detection value is determined, it can then be used as input to determine the amounts used by different parts or components of the image processing pipeline. For example, the scene detection value can be used to tune the amount of infrared correction, white balance correction, color correction, and saturation. As the scene transitions from bright to dark, the ratio increases (e.g., from 1:100 to 1:1), resulting in a smooth decrease in the IR subtraction rate and a smooth increase in desaturation. The ratio of IR intensity to visible light intensity can be input into a function or used as an index value for a lookup table, with each particular scene detection value (e.g., the ratio) corresponding to a particular IR subtraction value, a particular desaturation rate, and so on.

[0014] Conventional systems using RGB-IR cameras use either the signal from RGB pixels or the signal from the IR camera at a given time. Such systems switch, according to a certain threshold, from using the signal from RGB pixels when the scene is bright (e.g., during the day) to using the signal from IR pixels when the scene is dark (e.g., during the night). This switch from using the signal from RGB pixels to using the signal from IR pixels is an abrupt switch between a higher-resolution color image and a lower-resolution monochrome IR image. For example, there may be a predetermined and static threshold at which a conventional system switches from outputting an RGB image to outputting a monochrome IR image. Embodiments avoid such abrupt switching by using a scene detection value as input to smoothly transition from an output image based more on RGB information in brighter scenes (where visible light is dominant) to an output image based more on IR information in darker scenes (where IR illumination is dominant). For a given scene detection value, the portion of the signal based on IR may be used rather than completely removing a portion of the signal for the RGB pixel sensor that is based on IR. In the embodiment, the technology continues to use RGB information in darker conditions to maintain consistent resolution, without discarding RGB information, and uses only the IR pixel sensor when the scene becomes dark, resulting in an output image with 1 / 4 the information / resolution. Therefore, in the embodiment, a monochrome IR image produced in low-light conditions may have the same resolution as a color image produced in bright-light conditions. In the embodiment, a system using a standard RGB-IR camera can provide full-resolution images with high usability for both machine vision and human vision tasks, provided that image quality can be minimized and gradually degraded in response to a reduction in ambient visible light.

[0015] Various aspects of the above-described techniques are described in detail below herein, not as limitations but as examples. The examples provided below describe techniques incorporated into the pipeline of one or more image signal processors that may be applied after an image has been received from an image sensor and before it is permanently stored. In other examples, the technique may be used to update an image after the image data has been permanently stored.

[0016] Some embodiments are described in relation to cameras in a car cabin. However, it should be understood that the embodiments also cover the use of images produced by RGB-IR cameras in other applications, such as in other types of vehicle cabins (e.g., airplane cockpits, truck cabins, train cabins, etc.), in security cameras, and in outward-facing cameras of vehicles (e.g., imaging the surrounding area outside the vehicle). Therefore, the examples described herein regarding the adjustment of images used in cabin camera systems are merely examples, and the same techniques described in these examples are applicable to any other settings in which RGB-IR cameras are used.

[0017] Several embodiments are described with respect to IR and IR sensors. It should be understood that the term infrared (IR) as used herein may include both conventional IR radiation (e.g., radiation with wavelengths between 780 nm and 1 mm) and near-IR radiation (e.g., radiation with wavelengths between 750 nm and 1300 nm).

[0018] The systems and methods described herein may be used by, but are not limited to, non-autonomous vehicles, semi-autonomous vehicles (for example, in one or more adaptive driver assistance systems (ADAS)), piloted and un-piloted robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, flying vessels, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, submarines, drones, and / or other vehicle types. Furthermore, the systems and methods described herein may, by example, be used for a variety of purposes, including but not limited to, machine control, machine motion, machine operation, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and supervision, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twinning, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, and / or any other suitable applications.

[0019] The disclosed embodiments may include a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aerospace systems, intermediate systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems incorporating one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems at least partially implemented in data centers, systems for performing conversational AI operations, systems for performing optical transport simulations, systems for performing collaborative content creation for 3D assets, systems at least partially implemented using cloud computing resources, and / or other types of systems.

[0020] Figure 1 shows an exemplary computing environment 100 that includes techniques for performing dynamic transformations of one or more images based on calculated scene detection values, according to some embodiments of the present disclosure. The computing environment 100A may include one or more processing units 104 (e.g., an image signal processor (ISP) and / or other processors for the ISP pipeline 101) that operate on incoming image data 102 (e.g., an RGB-IR image) to generate transformed image data 130 (e.g., a transformed color image or a monochrome image).

[0021] The processing unit 104 may be configured to receive image data 102 and perform one or more operations on the image data in order to generate images that can be used in downstream processes (for example, by image analysis algorithms, machine learning models, output to a display, etc.), as described below. The processing unit 104 may be one or more devices capable of processing data for one or more images, or may include such devices. The processing unit 104 may be called an image processor and may be one or more image signal processors (ISPs), digital signal processors (DSPs), graphical processing units (GPUs), central processing units (CPUs), data processing units (DPUs), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), other integrated circuits, or a combination thereof.

[0022] The processing unit 104 may receive image data 102 (e.g., one or more images) from another device (e.g., an RGB-IR camera) as an analog signal, a digital signal, or a combination thereof. The image may be a sequence of images, and each image in the sequence may be a video frame. The processing unit 104 may receive the image data as an encoded image, a raw image, or a combination thereof.

[0023] Image data 102 may correspond to a set of values ​​from visible light sensor pixels and IR sensor pixels, where the set may have a size (e.g., set size), and each value may have a size (e.g., pixel size). The set size may be called resolution and may be measured in terms of the number of pixels, and may be 720×480 (e.g., Standard-Definition (SD)), 1920×1800 (High Definition (HD)), 3840×2160 (Ultra High Definition (4K UHD)), 7680×4320 (8K UHD), or other sizes or ratios. The value size may be called pixel size and may have a range based on the number of bits (e.g., pixel value range). For example, the value size can be 8 bits (e.g., range 0 to 255), 10 bits (e.g., range 0 to 1023), 12 bits (e.g., range 0 to 4K), any other number of bits, or a combination thereof. The value size (e.g., pixel size) may relate to the dynamic range of each image.

[0024] Image data 102 may include intensity values ​​from many sensor pixels of an RGB-IR camera, including intensity values ​​from red (R) sensor pixels, green (G) sensor pixels, blue (B) sensor pixels, and infrared (IR) sensor pixels. Image data 102 may include color information (e.g., from R, G, B sensor pixels) and / or monochromatic information (e.g., from IR sensor pixels) and may correspond to a still image (e.g., a photograph), an image in a sequence of images (e.g., frames of a video), or a combination thereof. In embodiments, image data 102 includes a histogram of the image. Hardware components (not shown) may generate histograms of the red, green, blue, and IR channels for the generated image. Entries in the histogram may have discretized values ​​(e.g., discretized into 256 different values ​​in one embodiment). In one embodiment, there are more possible values ​​for lower signals than for higher signals (e.g., higher resolution for the histogram).

[0025] Image data 102 may be input to a scene analyzer 106 and / or a processing pipeline 101. The scene analyzer 106 performs an analysis of the image data (e.g., an image histogram) to generate a scene detection value 108 for the image data 102. The scene detection value 108 is a metric of the relative power of IR emission to visible light and may be determined based on the intensity (i.e., power) values ​​of IR and the intensity (i.e., power) values ​​of visible light (e.g., R, G, and / or B signals). In an embodiment, the scene detection value 108 is calculated based on a relationship between an IR intensity value and one or more visible light intensity values. This relationship may be, for example, a ratio between an IR intensity value and one or more visible light intensity values. In one embodiment, the ratio is calculated between one or more IR intensity values ​​and one or more G intensity values.

[0026] In one embodiment, the scene analyzer 106 calculates the average IR intensity based on signals from one or more IR pixel sensors and the average visible light intensity based on signals from one or more G pixel sensors. In one embodiment, the scene analyzer 106 then calculates the ratio of the average IR intensity to the average G intensity. This ratio can be used as the scene detection value 108. Alternatively, the scene detection value 108 can be determined directly from this ratio.

[0027] Alternatively or additionally, the scene analyzer 106 may calculate the mean of the G, R, and / or B pixels and use the mean from the G, R, and / or B pixels together with the mean from the IR pixels to determine the scene detection value 108. For example, the scene analyzer 106 may determine the mean of the R, G, and B pixels and compare this summation mean with the mean of the IR pixels to determine the scene detection value 108. Additional or alternatively, other statistics such as median, maximum, minimum, and mean over quartiles may be determined based on the R, G, B, and / or IR pixels, and such values ​​may be used to determine the scene detection value 108. For example, the ratio of the median, maximum, minimum, and mean over quartiles of the G, B, and / or R pixels to the median, maximum, minimum, and mean over quartiles of the IR pixels may be calculated and used for the scene detection value 108.

[0028] In some embodiments, data from the entire image is used to determine the scene detection value 108. For example, the mean or other statistics may be determined based on data from all G pixels and all IR pixels. Alternatively, data from less than the entire image may be used to determine the scene detection value 108. For example, some areas of the scene may be more important than other areas of the scene for later processing and / or determination. One example is an area of ​​the image that shows the occupants and / or driver. Thus, in some embodiments, the scene analyzer 106 may use data only from areas of the image to determine the scene detection value 108 (e.g., a fixed area where the driver is generally located, or a dynamically determined area identified based on object detection and / or classification using a trained machine learning model). In some embodiments, the scene analyzer applies weights to values ​​from different areas / pixels (e.g., to determine a weighted average of IR values ​​and / or visible light values), and some areas / pixels may be weighted more heavily than others for calculation.

[0029] The scene detection value 108 may be used to determine the amount of IR to use in the image and / or to determine how the image should be modified to account for the amount of IR that should be used. Once the scene detection value 108 is calculated, one or more quantities 110 may be determined based on the scene detection value 108. Examples of such quantities 110 include quantity 110A, which is an IR subtraction factor; quantity 110B, which is a white balance correction factor; quantity 110C, which is a color correction factor; and quantity 110D, which is a color saturation factor. Each of the quantities 110 may be used at different stages of the processing pipeline 101 to process the image data 102. These quantities 110 compensate for the side effects caused by the amount of IR signal that is not removed from the color channels.

[0030] Visible light sensor pixels generally detect IR radiation, as well as radiation for specific wavelengths of visible light that the visible light sensor pixel is configured to detect. Conventionally, infrared correction is performed for RGB-IR image data by determining the intensity value of the IR sensor pixel and then completely subtracting those intensity values ​​from the intensity values ​​of surrounding visible light sensor pixels (e.g., R, G, and B sensor pixels). However, embodiments utilize the detection of IR radiation to improve image resolution for medium and low light conditions by selecting that the visible light sensor pixel does not subtract all of its IR contribution to the intensity value of the visible light sensor pixel under certain conditions.

[0031] In one embodiment, an IR subtraction factor (e.g., quantity 110A) is determined based on the scene detection value 108. The IR subtraction factor controls the amount of infrared correction 112 to be performed. Thus, the IR subtraction factor controls the amount of IR contribution to be subtracted from the visible light sensor pixel value for a given image. Based on the IR subtraction factor (quantity 110A), a small amount of the IR contribution to the visible light sensor pixel value may be subtracted for infrared correction 112. In one embodiment, the IR subtraction value is a factor having a value between 0 and 1.

[0032] In some embodiments, once the IR subtraction factor (quantity 110A) is determined, the IR subtraction factor may be used to perform infrared correction 112 according to the following formula. P out (x,y=P in (x,y)-I L *IR SubtractionFactor Here, P in This is the input pixel value for a pixel with coordinates (x,y), and P out This is the output pixel value for the pixel at coordinate (x,y), and IR SubtractionFactor This is the determined IR subtraction factor, and I L This is the local IR average for the pixel with coordinates (x,y).

[0033] In one embodiment, the scene detection value can be anywhere between 0 and 1. For example, a ratio of 1:100 IR intensity to visible light intensity (corresponding to a well-lit or bright scene) may cause the scene detection value to be close to 0 (e.g., 0.01), while a ratio of 1:1 IR intensity to visible light intensity (corresponding to a poorly lit or dark scene) may cause the scene detection value to be 1. In one example, if the scene detection value is close to 0, indicating that the scene is well-lit or in full sunlight, the IR subtraction factor may be 1, and the entire IR contribution may not be subtracted from the visible light pixel sensor value. On the other hand, if the scene detection value is close to 1, indicating that the scene is poorly lit or in darkness, the IR subtraction value may be 0, and the IR contribution may not be subtracted from the visible light pixel sensor value at all. For intermediate scene detection values ​​(for example, approximately 0.2 to 0.8, or approximately 0.3 to 0.7, or approximately 0.4 to 0.6), the IR subtraction factor may have values ​​ranging from approximately 0.1 to 0.9. An inverse relationship may exist between the scene detection value and the IR subtraction factor. Therefore, as the scene detection value increases, the IR subtraction factor may decrease. This inverse relationship between the scene detection value and the IR subtraction factor may be linear or nonlinear.

[0034] Figure 2 is a chart 200 showing a conceptual IR subtraction factor curve 230 according to several embodiments of the present disclosure. The x-axis of the chart corresponds to the scene detection value 205, and the y-axis of the chart corresponds to the IR subtraction factor 210. The IR subtraction factor curve 230 is divided into three conceptual zones, including a natural light dominant zone 215, a transition zone 220, and an IR dominant zone 225. In the natural light dominant zone 215, there is generally a large amount of background light, which causes the intensity values ​​for visible light sensor pixels to be relatively high, resulting in high-quality image data with higher color fidelity than in other areas. Therefore, for the natural light dominant zone, the IR contribution to the intensity values ​​of visible light sensor pixels is not useful, and the IR subtraction factor is 1 (i.e., 100% of the IR contribution is removed from the intensity values ​​for visible light sensor pixels). In one example, the natural light-dominant zone includes scene detection values ​​ranging from 0 to approximately 0.4.

[0035] In the transition zone 220, there is less natural light in the scene than in the natural light-dominant zone 215. As a result, the intensity value for the visible light sensor pixels is lower in the transition zone 220 than in the natural light-dominant zone 215. By utilizing part of the IR contribution to the intensity value for the visible light sensor pixels (for example, not subtracting all of the IR contribution), the overall intensity value can be increased, which can improve the SNR. Therefore, for the scene detection value in the transition zone 220, the IR subtraction factor 210 can steadily decrease as the scene detection value 205 increases.

[0036] RGB-IR cameras emit their own IR rays to illuminate a scene. As natural or background light decreases in the scene (for example, at night), the IR rays emitted by the RGB-IR camera become dominant. In the IR-dominant zone 225, the majority or all of the contribution to the intensity value measured by the visible light sensor pixels comes from IR. Traditionally, when there is little background light (for example, in the IR-dominant zone 225), the RGB-IR camera switches to using only the IR sensor pixel information. However, this reduces the resolution of the resulting image to one-quarter of the resolution achieved under naturally lit conditions.

[0037] In the embodiment, data from visible light sensor pixels is used even in low-light conditions. In the embodiment, in the IR-dominant zone 225, the IR subtraction value is 0 or close to 0. As a result, there is little IR contribution removed from the intensity value of the visible light sensor pixels. Thus, data remains available for most or all of the sensor pixels of the RGB-IR camera, even in low-light settings. This causes the resolution of the image produced in such low-light settings to become the standard resolution achievable during settings with better lighting.

[0038] In an IR-dominant setting, the majority or all of the radiation measured is IR radiation. Since the same wavelength of radiation (i.e., IR radiation) is measured by all sensor pixels, color information can be lost. Therefore, the resulting image may be monochrome. However, these monochrome images have a much higher resolution than what was traditionally achievable using RGB-IR cameras in low-light settings.

[0039] Returning to Figure 1, the adjustment of infrared correction 112 may result in changes to white balance, color correction, color saturation, etc. By not removing all IR contributions to visible light sensor pixel values ​​under all conditions, a single white balance correction, a single color correction, a single color saturation correction, etc., cannot be calculated a priori for an RGB-IR camera. Instead, different adjustments should be made for white balance correction 114, color correction 118, and color saturation 124, depending on the amount of infrared correction performed. Thus, the quantity 110 determined by the scene analyzer 106 may include a white balance correction factor (quantity 110B), a color correction factor (quantity 110C), a color saturation factor (quantity 110D), etc.

[0040] In some embodiments, some or all of the quantities 110 are determined using one or more functions that relate the scene detection value to one or more of the quantities 110. Thus, the calculated scene detection value may be input to a function, and the function may return values ​​for one or more of the quantities 110. In some embodiments, some or all of the quantities 110 are determined using one or more lookup tables. The scene detection value 108 may be used as an index to one or more lookup tables, and the quantized values ​​for one or more of the quantities 110 may be included in the entries of one or more lookup tables.

[0041] In one example, if the scene is detected as a night scene (e.g., a high scene detection value 108), IR is dominant, and there may be an average pixel value ratio of IR to G that is close to 1:1. For such a high scene detection value 108, there is little IR subtraction in infrared correction 112, color saturation 124 may apply desaturation to the color image (the color saturation factor may be low), and the white balance correction factor and color correction factor may be 0 in color correction 118 and color saturation 124 (AWB gain is unity, and the unmodified CCM identity matrix will be used under low-light conditions). The output under such circumstances may be a monochrome image.

[0042] In another example, if the scene is detected as a daytime scene (e.g., a low scene detection value 108), visible light may be dominant, and there may be a low ratio of IR to visible light pixel values ​​(e.g., 1:50, 1:100, etc.). For such a low scene detection value 108, a full IR subtraction may be performed in the infrared correction 112, and the color saturation 124 and color correction 118 may be left as is (e.g., not modified by the correction factor) to preserve saturation information, etc. The output under such circumstances may be a color image. In the embodiment, the rate of IR subtraction and the amount of desaturation may share an inverse relationship depending on the ambient visible light in the scene.

[0043] The example allows for tuning the amount of IR rejection based on lighting conditions, which is a compromise between better SNR and color fidelity for images in medium-light to low-light scenes.

[0044] Figure 3 is an exemplary lookup table 300 that can be used to determine one or more quantities 308 (corresponding, for example, quantity 110 in Figure 1) used to adjust an image based on a scene detection value 305, according to some embodiments of the present disclosure. Once the scene detection value 305 is calculated, it can be used as an index to find an entry (e.g., a row) in the lookup table 300. The entries may include a first value for the IR subtraction factor 310, a second value for the white balance correction factor 315, a third value for the color correction factor 320, a fourth value for the color saturation factor 325, and so on. Note that the exemplary values ​​included in the exemplary lookup table 300 are for illustrative purposes only, and other values ​​may be used.

[0045] Returning to Figure 1, automatic white balance (AWB) 114 may be performed after infrared correction 112. White balance (WB) is the process of removing unrealistic color casts so that objects that appear white to the human eye are rendered as white in the image. White balance is performed by determining the amount of gain to apply to each of the color channels, and each color channel may receive a different amount of gain. The goal of AWB is to ensure that all gray objects in the scene have equal amounts of red, green, and blue intensity. Proper white balance takes into account the "color temperature" of the light source, which refers to the relative warmth or coolness of the white light. While the human eye is very good at determining what is white under different light sources, digital cameras often have great difficulty with auto white balance (AWB), sometimes creating unsightly blue, orange, or even green color casts.

[0046] When IR signals are not removed from the visible light sensor pixel values, it becomes more difficult to perform correct white balance correction 114. The IR signals act as an offset. As a result, when AWB is performed on an image in which IR signals remain, different amounts of gain may be applied to the IR offset in each color channel, which can result in a purple cast. Therefore, a white balance correction factor is applied in the embodiment, and the white balance correction factor is based on the amount of IR signal that has not been removed from the visible light sensor pixel values. In particular, by leaving a portion of the IR signal intact, an offset is introduced into each color channel. This offset may appear as a purple color. In one embodiment, when more IR rays are retained, the system moves away from applying sufficient white balance gain and towards applying equal gain to each color channel. For example, for a pure IR signal (e.g., at night), there is no color signal to correct, so no gain may be applied to any of the color channels. At the opposite end of the spectrum, for a pure color signal (for example, one where all IR signals are removed from the visible light sensor pixel values), a standard correct white balance amount is used (for example, one determined based on system calibration). Between the pure color signal and the pure IR signal, an intermediate white balance factor (for example, between 0 and 1) may be determined.

[0047] White balance correction 114 may be performed on the image data after infrared correction 112, based on the determined white balance correction factor. In some embodiments, the white balance correction factor (quantity 110B) may be used to perform white balance correction 114 according to the following formula.

number

number

Equation

[0048] After white balance correction 114, standard demosaicing 116 may be performed. Demosaicing is a digital image process used to construct a full-color image from incomplete color samples output from an image sensor covered with a color filter array. The RGB-IR camera may include a plurality of sensor pixels each associated with a particular wavelength of radiation (e.g., a modified Bayer pattern including R, G, B and IR sensor pixels). In the demosaicing process, data from surrounding pixels associated with one or more wavelengths different from the subject sensor pixel may be used to estimate (e.g., via interpolation) values (i.e., power or intensity values) of one or more wavelengths for the subject sensor pixel. For example, in the demosaicing process, the content of an IR-only sensor pixel (photosite) may be replaced with color information that would have been present there based on interpolation of the same color data from surrounding color sensor pixels.

[0049] After demosaicing 116, color correction 118 may be performed using the determined color correction factor. Due to the spectral characteristics of the optical lenses and filters used in the camera system, the lighting of the scene, and the spectral characteristics of the color filter of the image sensor, the camera's RGB data may not provide faithful color rendition. In detail, the spectral characteristics of the image sensor may differ from the spectral response of the human eye. This infidelity in color reproduction is largely generated from the difference between the spectral characteristics of the imager color filter array and the spectral response of the human eye, and from the lighting of the scene. Color errors resulting from the lighting of the scene can be controlled by auto white balance. However, auto white balance does not sufficiently reduce the color errors generated by the spectral characteristics of the imager color filter array. Two types of color correction are mainly used to compensate for these color errors. One is a hue and saturation control method that uses Cr and Cb control. The other is a 3x3 color matrix method.

[0050] For a pure IR signal (for example, at night), there is no color data, and therefore no color correction is needed. Thus, the identity matrix can be used for color correction of a pure IR signal. On the other hand, for a pure color signal, the color matrix will have strong terms on the diagonals and negative terms on the off-diagonal values. There may be a correct, known, exact color correction value determined based on calibration, but this correct correction value can only be applied when the IR signal is completely removed from the visible light pixel sensor value. A factor (for example, having a value between 0 and 1) may be calculated based on the amount or IR remaining in the signal and applied to the correct correction value.

[0051] In some embodiments, a color correction factor (quantity 110C) may be used to perform color correction 118 according to the following formula.

number

number

number

[0052] After color correction 118 is performed, global tone mapping 120 may be performed. After global tone mapping 120, local tone mapping 122 may be performed.

[0053] After global tone mapping 120 and local tone mapping 122, color saturation 124 (i.e., desaturation) may be performed using a determined color saturation factor (quantity 110D). In one embodiment, the color saturation factor has a value between 0 and 1. Adjusting the color saturation 124 can directly control the hue, saturation, and / or luminance of the image. Some images may be saturated in color, and performing desaturation on such images may improve image quality. Conversely, some images may benefit from increasing color saturation.

[0054] In some embodiments, the color saturation factor (quantity 110D) may be used to perform color saturation correction 124 in the YUV color space according to the following formula. a. Y out(x,y=Y in (x,y) b. U out (x,y)=U in (x,y)*SAT factor c. V out (x,y=V in (x,y)*SAT factor Here, Y in , U in and V in These are the respective input YUV values ​​for the pixels at coordinates (x,y), and SAT factor This is the color saturation correction factor, Y out , U out and V out These are the respective output YUV values ​​for the pixel at coordinates (x,y).

[0055] Processing pipeline 101 does not represent a complete processing pipeline, and one or more additional operations may be performed in processing pipeline 101 in addition to those specified herein. Such additional operations are standard operations for the ISP pipeline and are not significant to this discussion; therefore, they are omitted for clarity and brevity. However, it should be understood that processing pipeline 101 may include such additional operations that may be performed before, during, and / or after the enumerated operations.

[0056] The processing pipeline 101 outputs converted image data 130, which may include, as appropriate, data from the visible light pixel sensor only, or data from both the visible light pixel sensor and the IR pixel sensor. Depending on the lighting conditions and / or other factors, the converted image data may include a color image or a monochrome image, and may be in the same color space as the input image data 102, or in a different color space. For example, the input image data 102 may be in the RGB color space, and the converted image data 130 may be in the luminance-chrominance (YUV) color space.

[0057] The image output by the processing pipeline 101 can be a full-resolution image regardless of the amount of visible light in the captured scene. The full-resolution image output can be in color in scenes with sufficient natural light, and in scenes with insufficient natural light for color, it smoothly transitions to a full-resolution monochrome image.

[0058] Figure 4 shows a flowchart of one exemplary example of Method 400 for processing image data output by an RGB-IR camera and, based on the processing, outputting a transformed image, according to an embodiment of the present disclosure. For simplicity of explanation, the Method of the present disclosure is shown and described as a series of actions. However, the actions of the present disclosure may be performed in various orders and / or concurrently, and together with other actions not shown and described herein. Furthermore, according to the subject matter disclosed, not all illustrated actions are necessarily required to implement the Method. Furthermore, those skilled in the art will understand and acknowledge that the Method may be alternatively represented as a series of interrelated states via a state diagram or events. Furthermore, it should be acknowledged that the Method disclosed herein may be stored in a product to facilitate transport and transfer of such Method to a computing device. As used herein, the term “product” is intended to encompass computer programs accessible from any computer-readable device or storage medium. In one implementation, Method 400 may be carried out using a processing unit 104 as shown in Figure 1.

[0059] Method 400 may be carried out by the processing logic of a processing device (e.g., processing unit 104) and may begin in operation 410. In operation 410, the processing logic may receive image data including infrared and visible light values. In the embodiment, the image data may be output by an RGB-IR camera. The image may correspond to a still image (e.g., a photograph), an image in a sequence of images (e.g., frames of a video), or a combination thereof. In the embodiment, the image data includes a histogram of pixel values.

[0060] In operation 420, the processing logic may process the image data to calculate a scene detection value. In the embodiment, the processing logic performs a comparison between one or more IR values ​​from the image data and one or more visible light values ​​(e.g., G values) from the image data and determines a scene detection value based on the comparison. The scene detection value may be based on the relationship between the IR values ​​and the visible light values.

[0061] In one embodiment, in block 422, the processing logic calculates a first value representing the intensity of the infrared value. The first value may be calculated based on the mean, median, mean, maximum, average per quartile, or other statistical value of the infrared value of the image (for example, from an image histogram).

[0062] In one embodiment, in block 424, the processing logic calculates a second value representing the intensity of the visible light values. The second value may be calculated based on the mean, median, average, maximum, average per quartile, or other statistical value of the visible light values ​​of the image (for example, from an image histogram). In one embodiment, the second value is determined based on the G value from the image histogram.

[0063] In one embodiment, in block 426, the processing logic determines the relationship between a first value and a second value. In one embodiment, this relationship is the ratio of the first value to the second value. A low ratio (and corresponding low scene detection value) may indicate a scene with abundant ambient light, while a high ratio (and corresponding high scene detection value) may indicate a scene in darkness.

[0064] In operation 430, the processing logic can determine one or more quantities based on the scene detection value. The determined quantities may include, for example, an IR subtraction factor, a white balance correction factor, a color correction factor, and / or a color saturation factor. In the embodiment, the magnitude of the IR subtraction factor varies inversely to the magnitude of the scene detection value. In the embodiment, the magnitude of the color saturation factor varies in direct proportion to the magnitude of the scene detection value.

[0065] In one embodiment, in block 432, the processing logic uses the scene detection value as an index to a lookup table. Entries in the lookup table may correspond to the calculated scene detection value and may include one or more enumerated quantities, such as a specified IR subtraction factor, white balance correction factor, color correction factor, and / or color saturation factor for the scene detection value. Each scene detection value may be associated with a set of quantities.

[0066] In one embodiment, in block 434, the processing logic uses scene detection values ​​as input to one or more functions, which may output quantities (e.g., IR subtraction factor, white balance correction factor, color correction factor, color saturation factor, etc.).

[0067] In block 440, the processing logic transforms the image data based on one or more quantities. This may include subtracting an amount with an IR contribution from the R, G, and B pixel values, performing white balance correction using a determined white balance correction factor, performing color correction using a determined color correction factor, and performing desaturation or saturation using a determined color saturation factor. The transformed image data may result in a full-resolution image that is either a color image or a monochrome image (e.g., color in rich light and monochrome in low light) that determines the lighting of the captured scene. In some embodiments, the output image is in the YUV color space.

[0068] Figure 5 is a flowchart of a method for processing a transformed image (e.g., transformed according to the method in Figure 4) using a trained machine learning model, according to some embodiments of the present disclosure. In block 510 of method 500, the processing logic receives the transformed image data (e.g., processed according to method 400). The transformed image data may include a full-resolution image, which may be a color image or a monochrome image, and determines the lighting conditions.

[0069] In block 520, the processing logic processes the transformed image data to generate an output. In one embodiment, in block 522, the transformed image data is input to a trained machine learning model. The trained machine learning model processes the image data and generates an output, which is received in block 524.

[0070] The trained machine learning model may be, for example, an artificial neural network, a support vector machine, a random forest algorithm, a decision tree, a hidden Markov model (HMM), a Gaussian mixture model (GMM), a regression model, or other types of machine learning models. Since the input image data is for full-resolution images with the best possible SNR, the accuracy of the output of the machine learning model can be increased, in contrast to when a 1 / 4-resolution monochrome image is input to the machine learning model. The machine learning model can be trained to produce outputs that are predictions, classifications, estimates, etc. The output may include a confidence rating, and the confidence rating (and accuracy) of the machine learning model can be maximized by transforming the image using Method 400 before input to the trained machine learning model.

[0071] In a DMS (Driver Monitoring System) example, a machine learning model might output activity classifications about the driver's activities for the vehicle (e.g., indicating whether the driver's hands are on the steering wheel), attention classifications or attention level determinations for the driver (e.g., indicating whether the driver's eyes are on the road). In an OMS (Operational Monitoring System) example, a machine learning model might output an indication that one or more occupants have been detected. This could be used, for example, to alert the driver about a child in the rear seat when the driver turns off the vehicle.

[0072] One type of machine learning model that can be used is artificial neural networks, such as deep neural networks. Artificial neural networks generally include feature representation components that have classifier or regression layers that map features to a desired output space. A convolutional neural network (CNN) hosts, for example, multiple layers of convolutional filters. Pooling may be performed in the lower layers to address nonlinearity, and a multilayer perceptron is usually added on top to map the top layer features extracted by the convolutional layers to decisions (e.g., classification outputs). Deep learning is a class of machine learning algorithms that use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. Deep neural networks can be trained in a supervised (e.g., classification) manner and / or in an unsupervised (e.g., pattern analysis) manner. Deep neural networks consist of layers, where different layers learn different levels of representation corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and complex representation. In an image recognition application, for example, the raw input might be a matrix of pixels; the first representation layer might extract pixels and encode edges; the second layer might synthesize and encode arrays of edges; the third layer might encode higher-level shapes; and the fourth layer might output, for example, a classification or prediction. In particular, the deep learning process can learn on its own which features should be optimally placed at which level. "Deep" in "deep learning" refers to the number of layers to which the data is transformed. More precisely, a deep learning system has a substantial credit assignment path (CAP) depth. A CAP is a chain of transformations from input to output. A CAP explains the latent causal relationships between input and output. In the case of a feedforward neural network, the CAP depth can be the network depth, which can be the number of hidden layers + 1.In the case of a recurrent neural network where a signal can propagate through layers two or more times, the CAP depth is potentially unlimited.

[0073] In one embodiment, one or more machine learning models are recurrent neural networks (RNNs). An RNN is a type of neural network that includes memory to enable it to capture time dependencies. An RNN is capable of learning input-output mappings that depend on both current and past inputs. An RNN deals with past and future scans and makes predictions based on this continuous scan information. An RNN can be trained using a training dataset to produce a fixed number of outputs (for example, to classify time-varying data such as video data as belonging to a fixed number of classes). One type of RNN that can be used is a long short-term memory (LSTM) neural network.

[0074] A common architecture for such tasks is the LSTM (Long Short-Term Memory). Unfortunately, LSTMs are not suitable for images because they do not capture spatial information as well as convolutional networks do. For this purpose, a variant of the LSTM called ConvLSTM can be used, which includes convolutional operations within the LSTM cells. ConvLSTM replaces matrix multiplication with convolutional operations at each gate in the LSTM cells. In doing so, it captures the underlying spatial features by convolutional operations in multidimensional data. The main difference between ConvLSTM and LSTM is the number of input dimensions. Since LSTM input data is one-dimensional, it is not suitable for spatial sequence data such as video, satellite, and radar image data sets. ConvLSTM is designed for 3D data as its input. In one embodiment, a CNN-LSTM machine learning model is used. CNN-LSTM is an integration of CNN (Convolutional Layer) with LSTM. First, the CNN portion of the model processes the data, and the one-dimensional result supplies the LSTM model.

[0075] In block 530, the processing logic performs one or more actions based on the output generated in block 520. These actions may include, for example, generating an alert if the driver's attention is not on the road, or generating an alert when the driver is getting out of a car with a child in the back seat.

[0076] Figure 6 is a flowchart of a method for displaying a converted image (e.g., converted according to the method in Figure 4) according to some embodiments of the present disclosure. In block 610, processing logic receives converted image data (e.g., converted according to method 400). In block 620, processing logic outputs the converted image data to a display. For example, an RGB-IR camera may be an in-cabin camera that produces images of the occupants of a car. The cabin may include a monitor, and the converted image may be output to the monitor. For example, the image may be output to a display in the car's dashboard, showing an image of the car's interior. The image may be a full-resolution image regardless of lighting, a color image when there is ambient visible light (e.g., daytime), and a monochrome image when there is little or no ambient visible light (e.g., nighttime).

[0077] Figure 7 shows an exemplary machine of computer system 700, within which a set of instructions can be executed to cause the machine to perform one or more of the methodologies described herein. In some embodiments, computer system 700 may be a computing device including a processor having a cache controller, a memory controller, or a combination thereof. In alternative embodiments, the machine may be connected to other machines (for example, networked) in a LAN, intranet, extranet, and / or internet. The machine may operate as a server machine or client machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server machine or client machine in a cloud computing infrastructure or environment.

[0078] A machine can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a web appliance, a server, a network router, a switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify the actions to be taken by that machine. Furthermore, although a single machine is shown, the term “machine” shall be understood to also include any set of machines that individually or collectively execute a set (or set) of instructions for carrying out one or more of the methodologies described herein.

[0079] An exemplary computer system 700 includes a processing device 702 (for example, a processor 200), main memory 704 (for example, read-only memory (ROM), flash memory, synchronous dynamic random access memory (DRAM) such as SDRAM or Rambus DRAM (RDRAM), etc.), static memory 706 (for example, flash memory, static random access memory (SRAM), etc.), and a data storage system 718, which communicate with each other via a bus 730.

[0080] The processing device 702 represents one or more general-purpose processing devices, such as a microprocessor or a central processing unit. More specifically, the processing device may be a processor implementing a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or other instruction sets, or a combination of instruction sets. The processing device 702 may also be one or more dedicated processing devices, such as a data processing unit (DPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or a network processor. The processing device 702 is configured to execute instructions 726 for performing the operations and steps described herein. The computer system 700 may further include a network interface device 708 for communication over the network 720.

[0081] The data storage system 718 may include a machine-readable storage medium 724 (also known as a non-temporary computer-readable medium) storing one or more sets of instructions 726 or software that implement any one or more of the methodologies or functions described herein. The instructions 726 may also reside, all or at least partially, in the main memory 704 and / or in the processing device 702 while they are being executed by the computer system 700, and the main memory 704 and the processing device 702 also constitute a machine-readable storage medium.

[0082] In one embodiment, instruction 726 includes instructions for implementing functionality corresponding to the scene analyzer 106 in Figure 1. The instructions may, in addition or alternatively, include instructions for implementing functionality corresponding to infrared correction and / or any other functions of the processing pipeline 101. Although the machine-readable storage medium 724 is shown to be a single medium in one exemplary embodiment, the term “non-temporary machine-readable storage medium” should be taken to include a single medium or more mediums that store one or more sets of instructions. The term “machine-readable storage medium” should also be taken to include any medium capable of storing or encoding a set of instructions for a machine to execute, causing the machine to perform one or more of the methodologies of this disclosure. Accordingly, the term “machine-readable storage medium” should be taken to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0083] In embodiments, the technology described herein applies to driver monitoring systems (DMS) and / or occupant monitoring systems (OMS) of automobiles. The automobiles may be autonomous vehicles, semi-autonomous vehicles, or manually driven vehicles. DMS and OMS systems may operate for both visible and near-iron light. The application of the technology described herein to DMS and OMS systems enables remote conferencing and / or passenger monitoring applications that benefit from color, along with machine vision monitoring applications that use a single camera for cost reduction.

[0084] Figure 8A shows an example of an autonomous vehicle 800 according to at least one embodiment. In at least one embodiment, the autonomous vehicle 800 (alternatively referred to herein as "vehicle 800") may be a passenger vehicle, including, but not limited to, a car, truck, bus, and / or another type of vehicle that accommodates one or more passengers. In at least one embodiment, vehicle 800 may be a semi-tractor-trailer truck used for transporting cargo. In at least one embodiment, vehicle 800 may be an aircraft, a robotic vehicle, or another type of vehicle.

[0085] Autonomous vehicles can be described in terms of automation levels as defined by the National Highway Traffic Safety Administration ("NHTSA"), a division of the U.S. Department of Transportation, and Non-Patent Document 1 of the Society of Automotive Engineers ("SAE"). In at least one embodiment, vehicle 800 may be capable of functioning at one or more of the autonomous driving levels from Level 1 to Level 5. For example, in at least one embodiment, vehicle 800 may be capable of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the embodiment.

[0086] In at least one embodiment, the vehicle 800 may include components such as a chassis, a vehicle body, wheels (e.g., two, four, six, eight, eighteen, etc.), tires, axles, and other components of the vehicle, but are not limited. In at least one embodiment, the vehicle 800 may include a propulsion system 850, such as an internal combustion engine, a hybrid power plant, a fully electric engine, and / or another propulsion system type, but are not limited. In at least one embodiment, the propulsion system 850 may be connected to the drivetrain of the vehicle 800, and the drivetrain may include a transmission to enable the propulsion of the vehicle 800, but are not limited. In at least one embodiment, the propulsion system 850 may be controlled in response to receiving a signal from (one or more) throttles / accelerators 852.

[0087] In at least one embodiment, a steering system 854, which may include a steering wheel, is used to steer the vehicle 800 (for example, along a desired path or route) when the propulsion system 850 is operating (for example, when the vehicle 800 is moving). In at least one embodiment, the steering system 854 may receive signals from (one or more) steering actuators 856. In at least one embodiment, the steering wheel may be optional for fully automated (Level 5) functionality. In at least one embodiment, a brake sensor system 846 may be used to actuate the vehicle brakes in response to receiving signals from (one or more) brake actuators 848 and / or brake sensors.

[0088] In at least one embodiment, but not limited to, one or more controllers 836, which may include one or more system-on-chip ("SoC") (not shown in Figure 8A) and / or one or more graphics processing units ("GPU"), provide signals (for example, representing commands) to one or more components and / or systems of the vehicle 800. For example, in at least one embodiment, one or more controllers 836 may send signals to operate the vehicle brakes via one or more brake actuators 848, signals to operate the steering system 854 via one or more steering actuators 856, and signals to operate the propulsion system 850 via one or more throttle / accelerators 852. In at least one embodiment, the controller 836 (one or more) may include one or more onboard (e.g., integrated) computing devices that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving the vehicle 800. In at least one embodiment, the controller 836 (one or more) may include a first controller for autonomous driving functions, a second controller for functional safety functions, a third controller for artificial intelligence functions (e.g., computer vision), a fourth controller for infotainment functions, a fifth controller for emergency redundancy, and / or other controllers. In at least one embodiment, a single controller may address two or more of the above functions, two or more controllers may address a single function, and / or any combination thereof.

[0089] In at least one embodiment, one or more controllers 836 provide signals for controlling one or more components and / or systems of the vehicle 800 in response to sensor data (e.g., sensor inputs) received from one or more sensors. In at least one embodiment, the sensor data may include, but are not limited to, one or more global navigation satellite system ("GNSS") sensors 858 (e.g., one or more global positioning system sensors), one or more radar sensors 860, one or more ultrasonic sensors 862, one or more lithium-ion radar sensors 864, or one or more inertial measurement units ("IMUs"). (unit) Sensors 866 (e.g., (one or more) accelerometers, (one or more) gyroscopes, (one or more) magnetic compasses, (one or more) magnetometers, etc.), (one or more) microphones 896, (one or more) stereo cameras 868, (one or more) wide-angle cameras 870 (e.g., fisheye cameras), (one or more) infrared cameras 872, (one or more) ambient cameras 874 (e.g., 360-degree cameras), (one or more ) can be received from a cabin camera 875, a long-range camera (not shown in Figure 8A), one or more medium-range cameras (not shown in Figure 8A), one or more speed sensors 844 (for measuring the speed of the vehicle 800), one or more vibration sensors 842, one or more steering sensors 840, one or more brake sensors (for example, as part of a brake sensor system 846), and / or other sensor types.

[0090] In one embodiment, one or more cabin cameras 875 are RGB-IR cameras that generate image data having both visible light and IR components. One or more controllers 836 may process the image data using the techniques described above to generate transformed image data (for example, by performing method 400). The transformed image data may have improved SNR and / or image quality and / or resolution compared to images conventionally generated by the RGB-IR camera. The generated images (i.e., transformed image data) may be input by one or more trained machine learning models by one or more controllers 836 and / or output for observation by the driver or occupant via the HMI display 834.

[0091] In at least one embodiment, one or more of the controllers 836 may receive inputs (represented, for example, by input data) from the instrument cluster 832 of the vehicle 800 and provide outputs (represented, for example, by output data, display data, etc.) via the human-machine interface ("HMI") display 834, an audible annunciator, a loudspeaker, and / or other components of the vehicle 800. In at least one embodiment, the output may include information such as vehicle speed, speed, time, map data (e.g., a high-resolution map (not shown in Figure 8A)), location data (e.g., the location of vehicle 800 on a map), direction, location of other vehicles (e.g., occupying grid), and information about objects and the status of objects sensed by (one or more) controllers 836. For example, in at least one embodiment, the HMI display 834 may display information about the presence of one or more objects (e.g., road signs, warning signs, changes in traffic lights, etc.) and / or information about driving operations that the vehicle has performed, is performing, or will perform (e.g., currently changing lanes, exiting at exit 34B 3.22 km (2 miles) ahead, etc.).

[0092] In at least one embodiment, the vehicle 800 further includes a network interface 824, which may use one or more wireless antennas 826 and / or one or more modems to communicate over one or more networks. For example, in at least one embodiment, the network interface 824 may be capable of communicating over Long-Term Evolution ("LTE"), Wideband Code Division Multiple Access ("WCDMA®"), Universal Mobile Telecommunications System ("UMTS"), Global System for Mobile Communication ("GSM"), IMT-CDMA Multi-Carrier ("CDMA2000") networks, and the like. Furthermore, in at least one embodiment, one or more wireless antennas 826 may enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area network protocols such as Bluetooth, Bluetooth Low Energy ("LE"), Z-wave, ZigBee, and / or one or more low-power wide-area network ("LPWAN") protocols such as LoRaWAN, SigFox.

[0093] In some embodiments, the inference and / or training logic 915 described below is used to perform inference and / or training operations using the transformed image data generated as described above. Further details regarding the inference and / or training logic 915 are provided herein in conjunction with Figures 9A and / or 9B. In at least one embodiment, the inference and / or training logic 915 may be used for inference or prediction operations based at least in part on weight parameters computed using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0094] Figure 8B shows an example of camera locations and fields of view for the autonomous vehicle 800 of Figure 8A, according to at least one embodiment. In at least one embodiment, the cameras and their respective fields of view are exemplary and not limiting. For example, in at least one embodiment, additional and / or alternative cameras may be included, and / or the cameras may be located in different locations on the vehicle 800.

[0095] In at least one embodiment, the camera type for a camera may include, but is not limited to, a digital camera that can be adapted for use with components and / or systems of the vehicle 800. In at least one embodiment, one or more cameras may operate at Automotive Safety Integrity Level ("ASIL") B and / or another ASIL. In at least one embodiment, the camera type may be capable of any image capture rate, depending on the embodiment, such as 60 frames per second (fps), 1220 fps, 240 fps, etc. In at least one embodiment, the camera may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In at least one embodiment, the color filter array may include a red-clear-clear-clear ("RCCC") color filter array, a red-clear-clear-blue ("RCCB") color filter array, a red-clear-clear-blue ("RBGC") color filter array, a Foveon X3 color filter array, a Bayer sensor ("RGGB") color filter array, a monochromatic sensor color filter array, an RGB-IR color filter array, and / or another type of color filter array. In at least one embodiment, a clear pixel camera may be used to increase light sensitivity, such as a camera with an RCCC, RCCB, and / or RBGC color filter array. In one embodiment, one or more of the cameras are RGB-IR cameras, and the image data generated by one or more cameras is processed according to the techniques described above.

[0096] In at least one embodiment, one or more of the cameras may be used to implement advanced driver assistance system ("ADAS") functions (for example, as part of a redundant or fail-safe design). For example, in at least one embodiment, a multi-function mono-camera may be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. In at least one embodiment, one or more of the cameras (for example, all of the cameras) may simultaneously record and provide image data (for example, video).

[0097] In at least one embodiment, one or more cameras may be mounted in a mounting assembly, such as a custom-designed (three-dimensional ("3D") printed) assembly, to eliminate stray light and reflections from within the vehicle (e.g., reflections reflected from the dashboard to the windshield) that could interfere with the camera image data capture ability. Referring to a door mirror mounting assembly, in at least one embodiment, the door mirror assembly may be custom 3D printed so that the camera mounting plate matches the shape of the door mirror. In at least one embodiment, one or more cameras may be integrated into the door mirror. In at least one embodiment, for a side-view camera, one or more cameras may be integrated into the four pillars at each corner of the cabin.

[0098] In at least one embodiment, a camera having a field of view including a portion of the environment in front of the vehicle 800 (e.g., a front camera) may be used for a perimeter view to help identify the path and obstacles ahead and to assist in providing information essential for generating an occupied grid and / or determining a preferred vehicle path, with the help of one or more of the controllers 836 and / or control SoCs. In at least one embodiment, the front camera may be used to implement many ADAS functions similar to LIDAR, including, but not limited to, emergency braking, pedestrian detection, and collision avoidance. In at least one embodiment, the front camera may also be used for ADAS functions and systems including, but not limited to, lane departure warning ("LDW"), autonomous cruise control ("ACC"), and / or traffic sign recognition.

[0099] In at least one embodiment, various cameras, including a monocular camera platform including, for example, a CMOS ("complementary metal oxide semiconductor") color imager, may be used in a front configuration. In at least one embodiment, a wide-angle camera 870 may be used to perceive objects entering the view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although only one wide-angle camera 870 is shown in Figure 8B, in other embodiments, there may be any number of wide-angle cameras (including 0) on the vehicle 800. In at least one embodiment, any number of long-range cameras 898 (e.g., long-view stereo camera pairs) may be used for depth-based object detection, particularly for objects for which the neural network has not yet been trained. In at least one embodiment, the long-range cameras 898 may also be used for object detection and classification, as well as basic object tracking.

[0100] In at least one embodiment, any number of stereo cameras 868 may also be included in the front configuration. In at least one embodiment, one or more of the (one or more) stereo cameras 868 may include an integrated control unit with a scalable processing unit, which may provide a programmable logic ("FPGA") and a multicore microprocessor having an integrated Controller Area Network ("CAN") or Ethernet interface on a single chip. In at least one embodiment, such a unit may be used to generate a 3D map of the environment of the vehicle 800, including distance estimation for all points in the image. In at least one embodiment, one or more of the (one or more) stereo cameras 868 may include, but are not limited to, one or more compact stereo vision sensors, which may include, but are not limited to, two camera lenses (one on the left and one on the right) and an image processing chip capable of measuring the distance from a vehicle 800 to a target object and using the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. In at least one embodiment, other types of (one or more) stereo cameras 868 may be used in addition to or as an alternative to those described herein.

[0101] In at least one embodiment, a camera having a field of view that includes a portion of the environment to the sides of the vehicle 800 (e.g., a side-view camera) may be used for the surrounding view and provide information used to create and update the occupied grid and generate side collision warnings. For example, in at least one embodiment, one or more surrounding cameras 874 (e.g., four surrounding cameras shown in Figure 8B) may be positioned on the vehicle 800. In at least one embodiment, one or more surrounding cameras 874 may include, but are not limited to, any number and combination of wide-angle cameras, one or more fisheye cameras, one or more 360-degree cameras, and / or similar cameras. For example, in at least one embodiment, four fisheye cameras may be positioned in front of, behind, and to the sides of the vehicle 800. In at least one embodiment, the vehicle 800 may use three surrounding cameras 874 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., a front camera) as a fourth surrounding view camera.

[0102] In at least one embodiment, one or more cabin cameras 875 have a field of view that covers the driver and / or passenger area of ​​the vehicle's cabin.

[0103] In at least one embodiment, a camera having a field of view that includes a portion of the environment behind the vehicle 800 (e.g., a rear-view camera) may be used for parking assistance, surrounding view, rear collision warning, and creation and updating of the occupancy grid. In at least one embodiment, a wide variety of cameras may be used, including, but not limited to, cameras that are also suitable as (one or more) front cameras as described herein (e.g., long-range camera 898 and / or (one or more) medium-range cameras 876, (one or more) stereo cameras 868, (one or more) infrared cameras 872, etc.).

[0104] In some embodiments, the inference and / or training logic 915 is used to perform inference and / or training operations using image data processed as described above. Further details regarding the inference and / or training logic 915 are provided herein in conjunction with Figures 9A and / or 9B.

[0105] Figure 8C is a block diagram illustrating an exemplary system architecture for the autonomous vehicle 800 of Figure 8A, according to at least one embodiment. In at least one embodiment, each of the components, features, and systems of the vehicle 800 in Figure 8C is shown as being connected via a bus 802. In at least one embodiment, the bus 802 may include, but is not limited to, a CAN data interface (alternatively referred to herein as the “CAN bus”). In at least one embodiment, the CAN may be an internal network of the vehicle 800 used to assist in the control of various features and functionalities of the vehicle 800, such as brake activation, acceleration, brake control, steering, and windshield wipers. In at least one embodiment, the bus 802 may be configured to have tens or even hundreds of nodes, each having its own unique identifier (e.g., a CAN ID). In at least one embodiment, the bus 802 may be read to find steering angle, ground speed, engine revolutions per minute ("RPM"), button position, and / or other vehicle status indicators. In at least one embodiment, bus 802 may be an ASIL B compliant CAN bus.

[0106] In at least one embodiment, the FlexRay and / or Ethernet protocols may be used in addition to or as an alternative to CAN. In at least one embodiment, there may be any number of buses forming bus 802, which may include, but are not limited to, zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and / or zero or more other types of buses using different protocols. In at least one embodiment, two or more buses may be used to perform different functions and / or for redundancy. For example, a first bus may be used for collision avoidance functionality and a second bus may be used for operation control. In at least one embodiment, each bus of bus 802 may communicate with any of the components of vehicle 800, and two or more buses of bus 802 may communicate with corresponding components. In at least one embodiment, any number of system-on-chip ("SoC") 804 (such as SoC804(A) and SoC804(B), each of the (one or more) controllers 836, and / or each computer in the vehicle may have access to the same input data (e.g., inputs from sensors in the vehicle 800) and may be connected to a common bus such as a CAN bus.

[0107] In at least one embodiment, the vehicle 800 may include one or more controllers 836, such as those described herein with respect to Figure 8A. In at least one embodiment, one or more controllers 836 may be used for a variety of functions. In at least one embodiment, one or more controllers 836 may be coupled to any of the various other components and systems of the vehicle 800 and may be used for controlling the vehicle 800, the artificial intelligence of the vehicle 800, infotainment for the vehicle 800, and / or other functions.

[0108] In at least one embodiment, the vehicle 800 may include any number of SoCs 804. In at least one embodiment, each of the SoCs 804 may include, but not limited to, a central processing unit ("CPU") 806, a graphics processing unit ("GPU") 808, one or more processors 810, one or more caches 812, one or more accelerators 814, one or more data stores 816, and / or other components and features not shown. In at least one embodiment, one or more SoCs 804 may be used to control the vehicle 800 in various platforms and systems. For example, in at least one embodiment, one or more SoCs 804 may be combined in a system (e.g., the system of the vehicle 800) that has a high-definition ("HD") map 822 that can obtain map refreshes and / or updates from one or more servers (not shown in Figure 8C) via a network interface 824.

[0109] In at least one embodiment, one or more CPU 806s may include a CPU cluster or CPU complex (alternatively referred to herein as “CCPLEX”). In at least one embodiment, one or more CPU 806s may include multiple cores and / or Level 2 ("L2") caches. For example, in at least one embodiment, one or more CPU 806s may include eight cores in a coherent multiprocessor configuration. In at least one embodiment, one or more CPU 806s may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2 megabyte (MB) L2 cache). In at least one embodiment, one or more CPU 806s (e.g., CCPLEX) may be configured to support concurrent cluster operation, which allows any combination of clusters of one or more CPU 806s to be active at any given time.

[0110] In at least one embodiment, one or more of the (one or more) CPU806s may implement a power management capability, which includes, but is not limited to, one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to conserve dynamic power; each core clock may be gated when such core is not actively executing instructions by executing an interrupt-wait ("WFI") / event-wait ("WFE") instruction; each core may be power-gated independently; each core cluster may be clock-gated independently when all cores are clock-gated or power-gated; and / or each core cluster may be power-gated independently when all cores are power-gated. In at least one embodiment, one or more CPU806s may further implement an extended algorithm for managing power states, where acceptable power states and expected wake-up times are specified, and the hardware / microcode determines which power state is best to enter for the core, cluster, and CCPLEX. In at least one embodiment, the processing core may support a simple power state entry sequence in software where the work is offloaded to the microcode.

[0111] In at least one embodiment, the GPU808 may include an integrated GPU (alternatively referred to herein as an “iGPU”). In at least one embodiment, the GPU808 may be programmable and efficient for parallel workloads. In at least one embodiment, the GPU808 may use an extended tensor instruction set. In at least one embodiment, the GPU808 may include one or more streaming microprocessors, each streaming microprocessor may include a Level 1 ("L1") cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In at least one embodiment, the GPU808 may include at least eight streaming microprocessors. In at least one embodiment, one or more GPU808s may use one or more compute application programming interfaces (APIs). In at least one embodiment, one or more GPU808s may use one or more parallel computing platforms and / or programming models (for example, NVIDIA's CUDA model).

[0112] In at least one embodiment, one or more of the (one or more) GPU808s may be power-optimized for best performance in automotive and embedded use cases. For example, in at least one embodiment, one or more of the (one or more) GPU808s may be fabricated on Fin field-effect transistor ("FinFET") circuit elements. In at least one embodiment, each streaming microprocessor may incorporate several mixed-precision processing cores divided into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In at least one embodiment, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, 2 mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, a Level 0 ("L0") instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In at least one embodiment, the streaming microprocessor may include independent parallel integer and floating-point data paths to efficiently execute workloads involving a mixture of computation and addressing calculations. In at least one embodiment, the streaming microprocessor may include independent thread scheduling capabilities to enable finer-grained synchronization and coordination between parallel threads. In at least one embodiment, the streaming microprocessor may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0113] In at least one embodiment, one or more of the (one or more) GPU808s may include high-bandwidth memory ("HBM") and / or 16GB of high-bandwidth memory second generation ("HBM2") memory subsystems to provide a peak memory bandwidth of approximately 900 GB / s in some examples. In at least one embodiment, in addition to or as an alternative to HBM memory, synchronous graphics random-access memory ("SGRAM"), such as graphics double data rate type five synchronous random-access memory ("GDDR5").

[0114] In at least one embodiment, one or more GPU808s may include unified memory technology. In at least one embodiment, address translation service (ATS) support may be used to enable one or more GPU808s to directly access the page tables of one or more CPU806s. In at least one embodiment, when a GPU in the GPU808 memory management unit (MMU) encounters a miss, an address translation request may be sent to one or more CPU806s. In at least one embodiment, in response, two of the CPUs of the one or more CPU806s may look up virtual-physical mappings for addresses in their page tables and send the translation back to one or more GPU808s. In at least one embodiment, the unified memory technology enables a single, unified virtual address space for the memory of both the (one or more) CPU806 and the (one or more) GPU808, thereby simplifying the programming of the (one or more) GPU808 and the porting of applications to the (one or more) GPU808.

[0115] In at least one embodiment, one or more GPU808s may include any number of access counters that can track how often one or more GPU808s access the memory of other processors. In at least one embodiment, one or more access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses the page most frequently, thereby improving the efficiency of memory ranges shared between processors.

[0116] In at least one embodiment, one or more of the (one or more) SoC804 may include any number of caches 812, including those described herein. For example, in at least one embodiment, the (one or more) caches 812 may include a Level 3 ("L3") cache (e.g., connected to the (one or more) CPU806 and (one or more) GPU808) that is available to both the (one or more) CPU806 and the (one or more) GPU808. In at least one embodiment, the (one or more) caches 812 may include a write-back cache that can track the state of the line, for example by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). In at least one embodiment, the L3 cache may include 4 MB or more of memory, depending on the embodiment, but smaller cache sizes may be used.

[0117] In at least one embodiment, one or more of the (one or more) SoC804 may include one or more accelerators 814 (e.g., hardware accelerators, software accelerators, or a combination thereof). In at least one embodiment, the (one or more) SoC804 may include a hardware acceleration cluster which may include an optimized hardware accelerator and / or a large on-chip memory. In at least one embodiment, the large on-chip memory (e.g., 4 MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. In at least one embodiment, the hardware acceleration cluster may complement the (one or more) GPU808 and be used to offload some of the tasks of the (one or more) GPU808 (e.g., to free up more cycles of the (one or more) GPU808 to perform other tasks). In at least one embodiment, one or more accelerators 814 may be used for a target workload that is stable enough to accept acceleration (e.g., perception, convolutional neural networks ("CNN"), recurrent neural networks ("RNN"), etc.). In at least one embodiment, the CNN may include region-based, i.e., regional convolutional neural networks ("RCNN"), and fast RCNNs (e.g., those used for object detection), or other types of CNNs.

[0118] In at least one embodiment, one or more accelerators 814 (e.g., a hardware acceleration cluster) may include one or more deep learning accelerators ("DLAs"). In at least one embodiment, one or more DLAs may include, but not limited to, one or more Tensor Processing Units ("TPUs"), which may be configured to provide additional, 10 trillion operations per second for deep learning applications and inference. In at least one embodiment, a TPU may be configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and may be an accelerator optimized for that purpose. In at least one embodiment, one or more DLAs may be further optimized for specific neural network types and sets of floating-point operations, as well as for inference. In at least one embodiment, a design of one or more DLAs may provide more performance per millisecond than a typical general-purpose GPU and generally far exceed the performance of a CPU. In at least one embodiment, one or more TPUs may implement several functions, including, for example, a single-instance convolution function supporting INT8, INT16, and FP16 data types for both features and weights, as well as a post-processing function. In at least one embodiment, one or more DLAs may rapidly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including, but not limited to, CNNs for object recognition and detection using data from camera sensors, CNNs for distance estimation using data from camera sensors, CNNs for emergency vehicle detection and identification and detection using data from microphones, CNNs for face recognition and vehicle owner identification using data from camera sensors, and / or CNNs for security and / or safety-related events.

[0119] In at least one embodiment, one or more DLAs may perform any function of one or more GPU808s, and for example, by using inference accelerators, the designer may target either one or more DLAs or one or more GPU808s for any function. For example, in at least one embodiment, the designer may concentrate CNN and floating-point arithmetic processing on one or more DLAs and offload other functions to one or more GPU808s and / or one or more accelerators 814.

[0120] In at least one embodiment, one or more accelerators 814 may include a programmable vision accelerator ("PVA"), which may be referred to herein as a computer vision accelerator instead. In at least one embodiment, the PVA may be designed and configured to accelerate computer vision algorithms for advanced driver-assistance systems ("ADAS") 838, autonomous driving, augmented reality ("AR") applications, and / or virtual reality ("VR") applications. In at least one embodiment, the PVA may provide a balance between performance and flexibility. For example, in at least one embodiment, each PVA may include, for example, any number of reduced instruction set computer ("RISC") cores, direct memory access ("DMA"), and / or any number of vector processors.

[0121] In at least one embodiment, the RISC core may interact with an image sensor (e.g., an image sensor of any camera described herein), one or more image signal processors, and so on. In at least one embodiment, each RISC core may include any amount of memory. In at least one embodiment, the RISC core may use one of several protocols, depending on the embodiment. In at least one embodiment, the RISC core may run a real-time operating system ("RTOS"). In at least one embodiment, the RISC core may be implemented using one or more integrated circuit devices, application-specific integrated circuits ("ASICs"), and / or memory devices. For example, in at least one embodiment, the RISC core may include an instruction cache and / or tightly coupled RAM.

[0122] In at least one embodiment, the DMA may enable components of the PVA to access system memory independently of one or more CPU 806s. In at least one embodiment, the DMA may support any number of features used to provide optimizations to the PVA, including, but not limited to, support for multidimensional addressing and / or circular addressing. In at least one embodiment, the DMA may support up to six or more addressing dimensions, these addressing dimensions may include, but not limited to, block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0123] In at least one embodiment, a vector processor is a programmable processor that can be designed to efficiently and flexibly perform programming for computer vision algorithms and can provide signal processing capabilities. In at least one embodiment, the PVA may include a PVA core and two vector processing subsystem partitions. In at least one embodiment, the PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. In at least one embodiment, the vector processing subsystem may act as the primary processing engine of the PVA and may include a vector processing unit ("VPU"), an instruction cache, and / or vector memory (e.g., "VMEM"). In at least one embodiment, the VPU core may include a digital signal processor, such as a single instruction, multiple data ("SIMD") or very long instruction word ("VLIW") digital signal processor. In at least one embodiment, a combination of SIMD and VLIW may improve throughput and speed.

[0124] In at least one embodiment, each vector processor may include an instruction cache and be coupled to dedicated memory. As a result, in at least one embodiment, each vector processor may be configured to operate independently of other vector processors. In at least one embodiment, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in at least one embodiment, multiple vector processors included in a single PVA may execute a common computer vision algorithm, but on different regions of an image. In at least one embodiment, the vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on a single image, or even execute different algorithms on a contiguous image or portion of an image. In at least one embodiment, among other things, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In at least one embodiment, the PVA may include additional error correction code ("ECC") memory to improve the overall safety of the system.

[0125] In at least one embodiment, one or more accelerators 814 may include a computer vision network on-chip and static random-access memory ("SRAM") to provide high-bandwidth, low-latency SRAM for one or more accelerators 814. In at least one embodiment, the on-chip memory may include, for example, at least 4 MB of SRAM including, but not limited to, eight field-configurable memory blocks, which may be accessible by both the PVA and DLA. In at least one embodiment, each pair of memory blocks may include an advanced peripheral bus ("APB") interface, configurable circuit elements, a controller, and a multiplexer. In at least one embodiment, any type of memory may be used. In at least one embodiment, the PVA and DLA may access the memory via a backbone that provides high-speed access to the memory. In at least one embodiment, the backbone may include a computer vision network on a chip that interconnects the PVA and DLA to memory (for example, using an APB).

[0126] In at least one embodiment, the computer vision network on chip may include an interface in which both the PVA and DLA determine to provide ready and enable signals before any control signals / addresses / data are transmitted. In at least one embodiment, the interface may provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. In at least one embodiment, the interface may conform to the International Organization for Standardization (ISO) 26262 or the International Electrotechnical Commission (IEC) 61508 standard, but other standards and protocols may be used.

[0127] In at least one embodiment, one or more of the (one or more) SoC804s may include a real-time ray tracing hardware accelerator. In at least one embodiment, the real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and extent of objects (e.g., in a world model) to generate real-time visualization simulations for RADAR signal interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general waveform propagation simulation, comparison with LIDAR data for localization and / or other functions, and / or other uses.

[0128] In at least one embodiment, one or more accelerators 814 can have diverse uses for autonomous driving. In at least one embodiment, PVA can be used for key processing stages in ADAS and autonomous vehicles. In at least one embodiment, the capabilities of PVA are well matched for algorithmic domains requiring predictable processing at low power and low latency. In other words, PVA performs well even on small datasets for semi-dense or dense regular computations that may require predictable runtime along with low latency and low power. In at least one embodiment, such as in a vehicle 800, PVA can be designed to run conventional computer vision algorithms, as they may be efficient in object detection and integer calculations.

[0129] For example, according to at least one embodiment of the technology, PVA is used to implement computer stereo vision. In at least one embodiment, a semi-global matching-based algorithm may be used in some examples, but is not limited to this. In at least one embodiment, an application for Level 3–5 autonomous driving uses motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.) on the fly. In at least one embodiment, PVA may implement computer stereo vision functionality for input from two monocular cameras.

[0130] In at least one embodiment, PVA may be used to perform high-density optical flow. For example, in at least one embodiment, PVA may process raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR data. In at least one embodiment, PVA may be used for time-of-flight depth processing by processing raw time-of-flight data to provide processed time-of-flight data.

[0131] In at least one embodiment, DLA may be used to power any type of network for improving control and driving safety, including, for example, a neural network that outputs a measure of confidence for each object detection. In at least one embodiment, confidence may be expressed or interpreted as the probability of each detection compared to other detections, or as providing its relative “weight.” In at least one embodiment, the confidence measure allows the system to make further determinations about which detections should be considered true positive detections rather than false positive detections. In at least one embodiment, the system may set a threshold for confidence and consider only detections exceeding the threshold as true positive detections. In embodiments where an automatic emergency braking (“AEB”) system is used, a false positive detection would cause the vehicle to automatically apply the emergency brakes, which is obviously undesirable. In at least one embodiment, a highly reliable detection may be considered a trigger for the AEB. In at least one embodiment, DLA may power a neural network to regress confidence values. In at least one embodiment, the neural network may take as its input at least a subset of parameters, including, among other things, the dimensions of the bounding box, ground plane estimates obtained (e.g., from another subsystem), outputs from one or more IMU sensors 866 correlated with the orientation of the vehicle 800, distance, and 3D location estimates of objects obtained from the neural network and / or other sensors (e.g., one or more LiDAR sensors 864 or one or more RADAR sensors 860).

[0132] In at least one embodiment, one or more of the (one or more) SoC804 may include (one or more) data stores 816 (e.g., memory). In at least one embodiment, the (one or more) data stores 816 may be on-chip memory of the (one or more) SoC804, which may store a neural network to be run on the (one or more) GPU808 and / or DLA. In at least one embodiment, the capacity of the (one or more) data stores 816 may be large enough to store multiple instances of the neural network for redundancy and safety. In at least one embodiment, the (one or more) data stores 816 may include (one or more) L2 or L3 caches.

[0133] In at least one embodiment, one or more of the (one or more) SoC804 may include any number of (one or more) processors 810 (e.g., embedded processors). In at least one embodiment, the (one or more) processors 810 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions and related security enforcement. In at least one embodiment, the boot and power management processor may be part of the boot sequence of the (one or more) SoC804 and may provide runtime power management services. In at least one embodiment, the boot power and management processor may provide clock and voltage programming, assistance with system low-power state transitions, management of thermal and temperature sensors of the (one or more) SoC804, and / or management of the power state of the (one or more) SoC804. In at least one embodiment, each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and (one or more) SoC 804 may use the ring oscillator to detect the temperatures of (one or more) CPU 806, (one or more) GPU 808, and / or (one or more) accelerator 814. In at least one embodiment, if it is determined that the temperature exceeds a threshold, the boot and power management processor may enter a temperature fault routine to put (one or more) SoC 804 into a low-power state and / or put the vehicle 800 into chauffeur-to-safe stop mode (e.g., to safely stop the vehicle 800).

[0134] In at least one embodiment, the (one or more) processor 810 may further include a set of embedded processors that can act as an audio processing engine, which may be an audio subsystem enabling full hardware support for multi-channel audio via multiple interfaces and a wide range of flexible audio I / O interfaces. In at least one embodiment, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0135] In at least one embodiment, the (one or more) processors 810 may further include an always-on processor engine that can provide the hardware features necessary to support low-power sensor management and startup use cases. In at least one embodiment, the always-on processor engine may include, but is not limited to, a processor core, tightly coupled RAM, supporting peripherals (e.g., timer and interrupt controllers), various I / O controller peripherals, and routing logic.

[0136] In at least one embodiment, the processor 810 (one or more) may further include a safety cluster engine, which may include, but is not limited to, a dedicated processor subsystem for addressing safety management for automotive applications. In at least one embodiment, the safety cluster engine may include, but is not limited to, two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores may, in at least one embodiment, operate in lockstep mode and function as a single core with comparison logic for detecting any differences between their operations. In at least one embodiment, the processor 810 (one or more) may further include a real-time camera engine, which may include, but is not limited to, a dedicated processor subsystem for addressing real-time camera management. In at least one embodiment, the processor 810 (one or more) may further include a high dynamic range signal processor, which may include, but is not limited to, an image signal processor that is a hardware engine that is part of a camera processing pipeline.

[0137] In at least one embodiment, the (one or more) processor 810 may include a video image synthesizer, which may be a processing block (for example, implemented on a microprocessor) that implements video post-processing functions required by a video playback application to produce a final image for a player window. In at least one embodiment, the video image synthesizer may perform lens distortion correction on the (one or more) wide-angle camera 870, the (one or more) ambient camera 874, and / or the (one or more) in-cabin surveillance camera sensors. In at least one embodiment, the (one or more) in-cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of the SoC 804, configured to identify and respond to events within the cabin. In at least one embodiment, the in-cabin system may, but is not limited, perform lip-reading to activate cellular services, make phone calls, write emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. In at least one embodiment, some functions are available to the driver when the vehicle is operating in autonomous mode, and are unavailable in other cases.

[0138] In at least one embodiment, the video image synthesizer may include extended temporal noise reduction for both spatial and temporal noise reduction. For example, in at least one embodiment, if motion occurs in the video, the noise reduction appropriately weights the spatial information and reduces the weight of the information provided by adjacent frames. In at least one embodiment, if the image or part of the image does not contain motion, the temporal noise reduction performed by the video image synthesizer may use information from previous images to reduce noise in the current image.

[0139] In at least one embodiment, the video image synthesizer may also be configured to perform stereo rectification on the input stereo lens frame. In at least one embodiment, the video image synthesizer may be further used for user interface compositing when the operating system desktop is in use, so that (one or more) GPU808s are not required to continuously render new surfaces. In at least one embodiment, when (one or more) GPU808s are powered on, active, and performing 3D rendering, the video image synthesizer may be used to offload (one or more) GPU808s to improve performance and responsiveness.

[0140] In at least one embodiment, one or more of the SoC804s may further include a Mobile Industry Processor Interface ("MIPI") camera serial interface, a high-speed interface, and / or a video input block that can be used for the camera and associated pixel input functions to receive video and input from the camera. In at least one embodiment, one or more of the SoC804s may further include one or more input / output controllers, which may be controlled by software and may be used to receive I / O signals that are not committed to a specific role.

[0141] In at least one embodiment, one or more of the (one or more) SoC804 may further include a wide range of peripheral interfaces for enabling communication with peripherals, audio encoders / decoders ("codecs"), power management, and / or other devices. In at least one embodiment, the (one or more) SoC804 may be used to process data from cameras (e.g., connected via Gigabit Multimedia Serial Link and Ethernet Channel), data from sensors (e.g., one or more) LiDAR sensors 864, one or more RADAR sensors 860, etc., which may be connected via Ethernet Channel), data from bus 802 (e.g., vehicle speed, steering wheel position, etc.), data from one or more GNSS sensors 858 (e.g., connected via Ethernet Bus or CAN Bus), and the like. In at least one embodiment, one or more of the (one or more) SoCs 804 may further include a dedicated high-performance mass storage controller, which may include its own DMA engine and may be used to free the (one or more) CPUs 806 from routine data management tasks.

[0142] In at least one embodiment, one or more SoC804s may be an end-to-end platform with a flexible architecture spanning automation levels 3–5, thereby providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and can provide a platform for a flexible, reliable driving software stack, along with deep learning tools. In at least one embodiment, one or more SoC804s may be faster, more reliable, and more energy-efficient and space-efficient than conventional systems. For example, in at least one embodiment, one or more accelerators 814, when combined with one or more CPUs 806, one or more GPUs 808, and one or more data stores 816, may provide a fast and efficient platform for Level 3–5 autonomous vehicles.

[0143] In at least one embodiment, a computer vision algorithm may run on a CPU, and this algorithm may be configured using a high-level programming language such as C to execute a wide variety of processing algorithms across a wide variety of visual data. However, in at least one embodiment, the CPU often fails to meet the performance requirements of many computer vision applications, such as requirements related to execution time and power consumption. In at least one embodiment, many CPUs are unable to execute complex object detection algorithms in real time, as used in in-vehicle ADAS applications and in actual Level 3-5 autonomous vehicles.

[0144] The embodiments described herein enable multiple neural networks to be implemented simultaneously and / or sequentially, allowing the results to be combined to enable Level 3–5 autonomous driving functionality. For example, in at least one embodiment, a CNN running on a DLA or a separate GPU (e.g., one or more GPU820s) may include text and word recognition, enabling the neural network to read and understand traffic signs, including signs that the neural network has not been specifically trained on. In at least one embodiment, the DLA may further include a neural network that can identify and interpret signs, provide a semantic understanding of the signs, and pass that semantic understanding to a route planning module running on a CPU complex.

[0145] In at least one embodiment, multiple neural networks may be operating simultaneously with respect to Level 3, 4, or 5 driving. For example, in at least one embodiment, a warning sign with an electric light and the text "Caution: Flashing light indicates icy condition" may be interpreted independently or collectively by several neural networks. In at least one embodiment, such a warning sign itself may be identified as a traffic sign by a first introduced neural network (e.g., a trained neural network), and the text "Flashing light indicates icy condition" may be interpreted by a second introduced neural network, which informs the vehicle's route planning software (preferably running on the CPU complex) that an icy condition is present when a flashing light is detected. In at least one embodiment, the flashing light may be identified by running a third introduced neural network over multiple frames, which informs the vehicle's route planning software of the presence (or absence) of the flashing light. In at least one embodiment, all three neural networks may run simultaneously within the DLA and / or on one or more GPU808s, etc.

[0146] In at least one embodiment, a CNN for facial recognition and vehicle owner identification may use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 800. In at least one embodiment, an always-on sensor processing engine may be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and in security mode to disable such vehicle when the owner leaves such vehicle. In this way, (one or more) SoC804 provides security against theft and / or vehicle hijacking.

[0147] In at least one embodiment, a CNN for emergency vehicle detection and identification may use data from microphone 896 to detect and identify emergency vehicle sirens. In at least one embodiment, one or more SoCs 804 use a CNN to classify environmental and urban sounds, as well as visual data. In at least one embodiment, a CNN operating on DLA is trained to identify the relative speed at which an emergency vehicle is approaching (for example, by using the Doppler effect). In at least one embodiment, a CNN may also be trained to identify emergency vehicles specific to the area in which the vehicle is operating, as identified by one or more GNSS sensors 858. In at least one embodiment, when operating in Europe, the CNN attempts to detect European sirens, and when in North America, the CNN attempts to identify only North American sirens. In at least one embodiment, when an emergency vehicle is detected, a control program may be used to execute an emergency vehicle safety routine, slow down the vehicle, pull over to the side of the road, stop the vehicle, and / or idle the vehicle in conjunction with one or more ultrasonic sensors 862 until the emergency vehicle has passed.

[0148] In at least one embodiment, the vehicle 800 may include one or more CPUs 818 (e.g., one or more individual CPUs, or one or more dCPUs), and the one or more CPUs 818 may be coupled to one or more SoCs 804 via a high-speed interconnect (e.g., PCIe). In at least one embodiment, the one or more CPUs 818 may include, for example, an x86 processor. The one or more CPUs 818 may be used to perform any of a variety of functions, including, for example, mediating potentially inconsistent results between ADAS sensors and one or more SoCs 804, and / or monitoring the status and health of one or more controllers 836 and / or the infotainment system ("Infotainment SoC") 830 on the chip.

[0149] In at least one embodiment, the vehicle 800 may include (one or more) GPUs 820 (e.g., (one or more) individual GPUs, or (one or more) dGPUs), and the (one or more) GPUs 820 may be coupled to the (one or more) SoCs 804 via a high-speed interconnect (e.g., NVIDIA's NVLINK channel). In at least one embodiment, the (one or more) GPUs 820 may provide additional artificial intelligence functionality, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based at least in part on input from the vehicle 800's sensors (e.g., sensor data).

[0150] In at least one embodiment, the vehicle 800 may further include a network interface 824, which may include, but is not limited to, one or more wireless antennas 826 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas and Bluetooth antennas). In at least one embodiment, the network interface 824 may be used to enable wireless connectivity to Internet cloud services (e.g., with one or more servers and / or other network devices) with other vehicles and / or computing devices (e.g., passenger client devices). In at least one embodiment, a direct link may be established between the vehicle 800 and other vehicles for communication with other vehicles, and / or an indirect link may be established (e.g., over a network and via the Internet). In at least one embodiment, the direct link may be provided using an inter-vehicle communication link. In at least one embodiment, the vehicle-to-vehicle communication link may provide vehicle 800 with information about nearby vehicles (e.g., vehicles in front of, to the side of, and / or behind vehicle 800). In at least one embodiment, such aforementioned functionality may be part of the cooperative adaptive driving control functionality of vehicle 800.

[0151] In at least one embodiment, the network interface 824 may include an SoC that provides modulation and demodulation functionality, enabling one or more controllers 836 to communicate over a wireless network. In at least one embodiment, the network interface 824 may include a radio frequency front end for baseband-to-radio frequency up-conversion and radio frequency-to-baseband down-conversion. In at least one embodiment, frequency conversion may be carried out in any technically feasible manner. For example, frequency conversion may be carried out through a well-known process and / or using a superheterodyne process. In at least one embodiment, the radio frequency front end functionality may be provided by a separate chip. In at least one embodiment, the network interface may include wireless functionality for communication over LTE, WCDMA®, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0152] In at least one embodiment, the vehicle 800 may further include one or more data stores 828, which may include, but are not limited to, off-chip (e.g., not on one or more SoCs 804) storage. In at least one embodiment, the data stores 828 may include one or more storage elements, which may include, but are not limited to, RAM, SRAM, dynamic random-access memory ("DRAM"), video random-access memory ("VRAM"), flash memory, hard disks, and / or other components and / or devices capable of storing at least one bit of data.

[0153] In at least one embodiment, the vehicle 800 may further include (one or more) GNSS sensors 858 (e.g., GPS and / or auxiliary GPS sensors) to assist mapping, perception, occupy grid generation, and / or route planning functions. In at least one embodiment, any number of GNSS sensors 858 may be used, including, for example, a GPS using a Universal Serial Bus ("USB") connector with an Ethernet-to-serial (e.g., RS-232) bridge.

[0154] In at least one embodiment, the vehicle 800 may further include one or more RADAR sensors 860. In at least one embodiment, one or more RADAR sensors 860 may be used by the vehicle 800 for long-range vehicle detection, even in darkness and / or severe weather conditions. In at least one embodiment, the functional safety level of the RADAR may be ASIL B. In at least one embodiment, one or more RADAR sensors 860 may, in some examples, use a CAN bus and / or bus 802 for control (e.g., to transmit data generated by one or more RADAR sensors 860) and for accessing object tracking data, along with access to an Ethernet channel for accessing raw data. In at least one embodiment, a wide variety of RADAR sensor types may be used. For example, but not limited to, one or more RADAR sensors 860 may be preferred for forward, rear, and side RADAR use. In at least one embodiment, one or more of the (one or more) RADAR sensors 860 are pulsed Doppler RADAR sensors.

[0155] In at least one embodiment, the (one or more) RADAR sensors 860 may include different configurations, such as narrow-field long-range, wide-field short-range, and short-range lateral coverage. In at least one embodiment, the long-range RADAR may be used for adaptive driving control functionality. In at least one embodiment, the long-range RADAR system may provide a wide field of view achieved by two or more independent scans, such as within a 250 m (meter) range. In at least one embodiment, the (one or more) RADAR sensors 860 may help distinguish between static and moving objects and may be used by the ADAS system 838 for emergency braking assistance and forward collision warning. In at least one embodiment, the (one or more) sensors 860 included in the long-range RADAR system may include, but are not limited to, multiple (e.g., six or more) fixed RADAR antennas, as well as monostatic and multimodal RADARs with high-speed CAN and FlexRay interfaces. In at least one embodiment, if there are six antennas, the four central antennas may create a focused beam pattern designed to record the area around vehicle 800 at a faster speed with minimal interference from traffic in adjacent lanes. In at least one embodiment, two additional antennas may expand the field of view, which may allow for the rapid detection of vehicles entering or leaving the lane of vehicle 800.

[0156] In at least one embodiment, the medium-range RADAR system may, as an example, include a range of up to 160 m (forward) or 80 m (rearward) and a field of view of up to 42 degrees (forward) or 150 degrees (rearward). In at least one embodiment, the short-range RADAR system may include, but not limited to, any number of RADAR sensors 860 designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, in at least one embodiment, the RADAR sensor system may create two beams that constantly monitor blind spots in the rearward direction and adjacent to the vehicle. In at least one embodiment, the short-range RADAR system may be used in an ADAS system 838 for blind spot detection and / or lane change assistance.

[0157] In at least one embodiment, the vehicle 800 may further include one or more ultrasonic sensors 862. In at least one embodiment, one or more ultrasonic sensors 862, which can be positioned in front of, behind, and / or to the side of the vehicle 800, may be used for parking assistance and / or to create and update the occupancy grid. In at least one embodiment, a wide variety of one or more ultrasonic sensors 862 may be used, and different ultrasonic sensors 862 may be used for different detection ranges (e.g., 2.5m, 4m). In at least one embodiment, one or more ultrasonic sensors 862 may operate at functional safety level ASIL B.

[0158] In at least one embodiment, the vehicle 800 may include one or more LiDAR sensors 864. In at least one embodiment, one or more LiDAR sensors 864 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. In at least one embodiment, one or more LiDAR sensors 864 may operate at functional safety level ASIL B. In at least one embodiment, the vehicle 800 may include multiple LiDAR sensors 864 (e.g., two, four, six, etc.), and those LiDAR sensors 864 may use an Ethernet channel (e.g., to provide data to a Gigabit Ethernet switch).

[0159] In at least one embodiment, one or more LiDAR sensors 864 may be capable of providing a list of objects and their distances over a 360-degree field of view. In at least one embodiment, one or more commercially available LiDAR sensors 864 may have an advertised range of approximately 80m, for example, with an accuracy of 2cm to 3cm and support for an 80Mbps Ethernet connection. In at least one embodiment, one or more non-protruding LiDAR sensors may be used. In such embodiments, one or more LiDAR sensors 864 may include small devices that can be incorporated into the front, rear, side, and / or corner locations of a vehicle 800. In at least one embodiment, one or more LiDAR sensors 864 may, in such embodiments, provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees over a range of 200m, even for low-reflectivity objects. In at least one embodiment, one or more front-mounted LiDAR sensors 864 may be configured for a horizontal field of view between 45 and 135 degrees.

[0160] In at least one embodiment, LiDAR technology such as 3D flash LiDAR may also be used. In at least one embodiment, the 3D flash LiDAR uses a flash of a laser as a transmission source to illuminate the area around the vehicle 800 up to approximately 200 m. In at least one embodiment, the flash LiDAR unit includes, but is not limited to, a receptor which records the transit time of the laser pulse and the reflected light on each pixel, which corresponds to the range from the vehicle 800 to the object. In at least one embodiment, the flash LiDAR may enable the generation of highly accurate and distortion-free ambient images with each laser flash. In at least one embodiment, four flash LiDAR sensors may be introduced, one on each side of the vehicle 800. In at least one embodiment, the 3D flash LiDAR system includes, but is not limited to, a solid-state 3D staring array LiDAR camera (e.g., a non-scanning LiDAR device) with no moving parts other than a fan. In at least one embodiment, a flash LiDAR device may use a Class I (eye-safe) laser pulse of 5 nanoseconds per frame and capture reflected laser light as a 3D-range point cloud and position-synchronized (co-registered) intensity data.

[0161] In at least one embodiment, the vehicle 800 may further include one or more IMU sensors 866. In at least one embodiment, one or more IMU sensors 866 may be located centered on the rear axle of the vehicle 800. In at least one embodiment, one or more IMU sensors 866 may include, for example, one or more accelerometers, one or more magnetometers, one or more gyroscopes, magnetic compasses, multiple magnetic compasses, and / or other sensor types. In at least one embodiment, such as in a 6-axis application, one or more IMU sensors 866 may include, for example, an accelerometer and a gyroscope. In at least one embodiment, such as in a 9-axis application, one or more IMU sensors 866 may include, for example, an accelerometer, a gyroscope, and a magnetometer.

[0162] In at least one embodiment, one or more IMU sensors 866 may be implemented as a small, high-performance GPS-Aided Inertial Navigation System ("GPS / INS") that combines micro-electro-mechanical systems ("MEMS") inertial sensors, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. In at least one embodiment, one or more IMU sensors 866 may enable a vehicle 800 to estimate its bearing without requiring input from magnetic sensors by directly observing changes in velocity and correlating them from GPS to one or more IMU sensors 866. In at least one embodiment, one or more IMU sensors 866 and one or more GNSS sensors 858 may be combined in a single integrated unit.

[0163] In at least one embodiment, the vehicle 800 may include one or more microphones 896 placed inside and / or around the vehicle 800. In at least one embodiment, one or more microphones 896 may be used, among other things, for emergency vehicle detection and identification.

[0164] In at least one embodiment, the vehicle 800 may further include any number of camera types, including (one or more) stereo cameras 868, (one or more) wide-angle cameras 870, (one or more) infrared cameras 872, (one or more) ambient cameras 874, (one or more) long-range cameras 898, (one or more) medium-range cameras 876, and / or other camera types. In at least one embodiment, cameras may be used to capture image data around the entire periphery of the vehicle 800. In at least one embodiment, the type of camera used depends on the vehicle 800. In at least one embodiment, any combination of camera types may be used to provide the required coverage around the vehicle 800. In at least one embodiment, the number of cameras introduced may vary depending on the embodiment. For example, in at least one embodiment, the vehicle 800 may include six cameras, seven cameras, ten cameras, twelve cameras, or any other number of cameras. In at least one embodiment, the camera may, as an example, support Gigabit Multimedia Serial Link ("GMSL") and / or Gigabit Ethernet communication, though not limited to these. In at least one embodiment, each camera may be as described more in detail previously herein with respect to Figures 8A and 8B.

[0165] In at least one embodiment, the vehicle 800 may further include one or more vibration sensors 842. In at least one embodiment, one or more vibration sensors 842 may measure vibrations of components of the vehicle 800, such as one or more axles. For example, in at least one embodiment, a change in vibration may indicate a change in the road surface. In at least one embodiment, when two or more vibration sensors 842 are used, the difference in vibration may be used to determine the amount of friction or slip of the road surface (for example, when the difference in vibration is between a power-driven axle and a free-rotating axle).

[0166] In at least one embodiment, the vehicle 800 may include an ADAS system 838. In at least one embodiment, the ADAS system 838 may include a SoC in some examples, but is not limited to. In at least one embodiment, the ADAS system 838 may include, but not limited to, any number and combination of autonomous / adaptive / automatic cruise control ("ACC") systems, cooperative adaptive cruise control ("CACC") systems, forward crash warning ("FCW") systems, automatic emergency braking ("AEB") systems, lane departure warning ("LDW") systems, lane keep assist ("LKA") systems, blind spot warning ("BSW") systems, rear cross-traffic warning ("RCTW") systems, collision warning ("CW") systems, lane centering ("LC") systems, and / or other systems, features, and / or functionalities.

[0167] In at least one embodiment, the ACC system may use (one or more) RADAR sensors 860, (one or more) LIDAR sensors 864, and / or any number of cameras. In at least one embodiment, the ACC system may include a longitudinal ACC system and / or a lateral ACC system. In at least one embodiment, the longitudinal ACC system monitors and controls the distance of vehicle 800 to another vehicle directly in front and automatically adjusts the speed of vehicle 800 to maintain a safe distance from the vehicle ahead. In at least one embodiment, the lateral ACC system implements distance maintenance and advises vehicle 800 to change lanes when necessary. In at least one embodiment, the lateral ACC is related to other ADAS applications such as LC and CW.

[0168] In at least one embodiment, the CACC system uses information from other vehicles, which may be received from other vehicles via a network interface 824 and / or (one or more) wireless antennas 826, either wirelessly or indirectly via a network connection (e.g., via the Internet). In at least one embodiment, direct links may be provided by vehicle-to-vehicle ("V2V") communication links, and indirect links may be provided by infrastructure-to-vehicle ("I2V") communication links. Generally, V2V communication provides information about the immediately preceding vehicle (e.g., a vehicle in the same lane immediately before vehicle 800), and I2V communication provides information about traffic further ahead. In at least one embodiment, the CACC system may include either or both I2V and V2V information sources. In at least one embodiment, having information about the vehicle ahead of vehicle 800 may make the CACC system more reliable, which could improve the smoothness of traffic flow and reduce congestion on the road.

[0169] In at least one embodiment, the FCW system is designed to alert the driver about hazardous materials so that such a driver can take corrective action. In at least one embodiment, the FCW system uses a front camera and / or (one or more) RADAR sensors 860 coupled to a dedicated processor, digital signal processor ("DSP"), FPGA, and / or ASIC, which are electrically coupled to provide driver feedback, such as a display, speaker, and / or vibration components. In at least one embodiment, the FCW system may provide warnings in the form of sound, visual warnings, vibration, and / or quick brake pulses.

[0170] In at least one embodiment, the AEB system may detect an impending forward collision with another vehicle or other object and automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. In at least one embodiment, the AEB system may use one or more front cameras and / or one or more RADAR sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. In at least one embodiment, when the AEB system detects a hazard, the AEB system may first alert the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system may automatically apply the brakes to prevent the anticipated collision or at least mitigate its impact. In at least one embodiment, the AEB system may include techniques such as dynamic brake support and / or pre-crash braking.

[0171] In at least one embodiment, the LDW system provides visual, auditory, and / or tactile warnings, such as vibration of the steering wheel or seat, to alert the driver when the vehicle 800 crosses a lane marker. In at least one embodiment, the LDW system does not activate when the driver indicates an intentional lane departure, such as by activating the turn signal. In at least one embodiment, the LDW system may use a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to provide driver feedback, such as a display, speaker, and / or vibration component. In at least one embodiment, the LKA system is a variation of the LDW system. In at least one embodiment, the LKA system provides steering input or brake control to correct the vehicle 800 if the vehicle 800 begins to move out of its lane.

[0172] In at least one embodiment, the BSW system detects vehicles in the vehicle's blind spot and warns the driver about those vehicles. In at least one embodiment, the BSW system may provide visual, auditory, and / or tactile alerts to indicate that merging or changing lanes is not safe. In at least one embodiment, the BSW system may provide additional warnings when the driver uses the turn signal. In at least one embodiment, the BSW system may use one or more rear-facing cameras and / or one or more RADAR sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0173] In at least one embodiment, the RCTW system may provide visual, auditory, and / or tactile notifications when an object is detected outside the range of the rear camera while the vehicle 800 is reversing. In at least one embodiment, the RCTW system includes an AEB system to ensure that the vehicle brakes are applied to avoid a crash. In at least one embodiment, the RCTW system may use one or more rear-facing radar sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to provide driver feedback, such as a display, speaker, and / or vibration components.

[0174] In at least one embodiment, conventional ADAS systems may be prone to producing false positive results, which can be annoying and distracting to the driver, but this is usually not a major issue because conventional ADAS systems alert the driver, allowing the driver to determine whether safety conditions truly exist and act accordingly. In at least one embodiment, the vehicle 800 itself determines, in the event of conflicting results, whether to follow the result from a primary computer (e.g., a first controller of controller 836) or a secondary computer (e.g., a second controller of controller 836). For example, in at least one embodiment, the ADAS system 838 may be a backup and / or secondary computer for providing perceptual information to a backup computer rationality module. In at least one embodiment, the backup computer rationality monitor may run redundant and varied software on hardware components to detect failures in perceptual and dynamic driving tasks. In at least one embodiment, the output from the ADAS system 838 may be provided to a supervisory MCU. In at least one embodiment, if the output from the primary computer and the output from the secondary computer are inconsistent, the supervising MCU determines how to reconcile the inconsistency to ensure safe operation.

[0175] In at least one embodiment, the primary computer may be configured to provide the supervising MCU with a reliability score indicating the reliability of the primary computer in the selected outcome. In at least one embodiment, if the reliability score exceeds a threshold, the supervising MCU may follow the instructions of the primary computer, regardless of whether the secondary computer provides contradictory or inconsistent results. In at least one embodiment, if the reliability score does not meet the threshold, and the primary and secondary computers produce different results (e.g., contradictory), the supervising MCU may mediate between the computers to determine an appropriate outcome.

[0176] In at least one embodiment, the supervising MCU may be configured to operate one or more neural networks trained and configured to determine, at least in part, the conditions under which a secondary computer provides a false alarm, based on the output from the primary computer and the output from the secondary computer. In at least one embodiment, one or more neural networks in the supervising MCU may learn when the output from the secondary computer can be trusted and when it cannot. For example, in at least one embodiment, when the secondary computer is a radar-based FCW system, one or more neural networks in the supervising MCU may learn when the FCW system identifies a metallic object that is not actually a hazard, such as a drain grate or manhole cover, which triggers an alarm. In at least one embodiment, when the secondary computer is a camera-based LDW system, the neural network in the supervising MCU may learn to disable the LDW when a cyclist or pedestrian is present and lane departure is actually the safest operation. In at least one embodiment, the supervisor MCU may include at least one DLA or GPU suitable for running (one or more) neural networks together with associated memory. In at least one embodiment, the supervisor MCU may comprise and / or be included as a component of (one or more) SoC804.

[0177] In at least one embodiment, the ADAS system 838 may include a secondary computer that implements ADAS functionality using conventional computer vision rules. In at least one embodiment, the secondary computer may use conventional computer vision rules (if-then), and the presence of (one or more) neural networks in the supervising MCU may improve reliability, safety, and performance. For example, in at least one embodiment, diverse implementations and intentional non-identities make the entire system more fault-tolerant, particularly against failures caused by software (or software-hardware interface) functionality. For example, in at least one embodiment, if there is a software bug or error in the software running on the primary computer, and non-identical software code running on the secondary computer provides a consistent overall result, the supervising MCU may have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer has not caused a critical error.

[0178] In at least one embodiment, the output of the ADAS system 838 may be fed to a perception block and / or a dynamic driving task block of the primary computer. For example, in at least one embodiment, if the ADAS system 838 indicates a forward crash warning due to an object immediately preceding, the perception block may use this information when identifying the object. In at least one embodiment, the secondary computer may have its own trained, and therefore false-positive-reducing, neural network, as described herein.

[0179] In at least one embodiment, the vehicle 800 may further include an infotainment SoC 830 (e.g., an in-vehicle infotainment system (IVI)). Although shown and described as an SoC, the infotainment system SoC 830 may not be an SoC in at least one embodiment and may include, but not limited to, two or more separate components. In at least one embodiment, the infotainment SoC 830 may include, but not limited to, a combination of hardware and software which may be used to provide the vehicle 800 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., a navigation system, rear parking assist, wireless data system, vehicle-related information such as fuel level, total mileage, brake fuel level, oil level, door open / closed, air filter information, etc.). For example, the infotainment SoC 830 may include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, car computer, in-car entertainment, Wi-Fi, steering wheel audio control, hands-free voice control, heads-up display ("HUD"), HMI display 834, telematics devices, control panels (for example, for controlling and / or interacting with various components, features, and / or systems), and / or other components. In at least one embodiment, the infotainment SoC 830 may be further used to provide (one or more) users of the vehicle 800 with (for example, visual and / or auditory) information, such as information from the ADAS system 838, autonomous driving information such as planned vehicle operation and trajectory, ambient information (for example, intersection information, vehicle information, road information, etc.), and / or other information.

[0180] In at least one embodiment, the infotainment SoC 830 may include any amount and type of GPU functionality. In at least one embodiment, the infotainment SoC 830 may communicate with other devices, systems, and / or components of the vehicle 800 via bus 802. In at least one embodiment, the infotainment SoC 830 may be coupled to a supervisory MCU so that if one or more primary controllers 836 (e.g., the primary and / or backup computers of the vehicle 800) fail, the GPU of the infotainment system may perform some self-driving functions. In at least one embodiment, the infotainment SoC 830 may put the vehicle 800 into driver-safe stop mode as described herein.

[0181] In at least one embodiment, the vehicle 800 may further include an instrument cluster 832 (e.g., a digital dashboard, electronic instrument cluster, digital instrument panel, etc.). In at least one embodiment, the instrument cluster 832 may include, but is not limited to, a controller and / or supercomputer (e.g., a separate controller or supercomputer). In at least one embodiment, the instrument cluster 832 may include, but is not limited to, a set of instrumentation such as a speedometer, fuel level, oil pressure, tachometer, odometer, direction indicator, shift lever position indicator, (one or more) seat belt warning lights, (one or more) parking brake warning lights, (one or more) engine fault lights, auxiliary restraint system (e.g., airbag) information, lighting control, safety system control, navigation information, etc. In some examples, the information may be displayed and / or shared between the infotainment SoC 830 and the instrument cluster 832. In at least one embodiment, the instrument cluster 832 may be included as part of the infotainment SoC 830, and vice versa.

[0182] In some embodiments, the inference and / or training logic 915 is used to perform inference and / or training operations using the transformed image data that may be generated as described above.

[0183] Figure 8D is a diagram of a system 878 for communication between one or more cloud-based servers and the autonomous vehicle 800 of Figure 8A, according to at least one embodiment. In at least one embodiment, the system 878 may include, but is not limited to, one or more servers 878, one or more networks 890, and any number and types of vehicles, including the vehicle 800. In at least one embodiment, the server 878 may include, but is not limited to, a plurality of GPUs 884(A) to 1084(H) (collectively referred to herein as GPU 884), PCIe switches 882(A) to 882(D) (collectively referred to herein as PCIe switches 882), and / or CPUs 880(A) to 880(B) (collectively referred to herein as CPU 880). In at least one embodiment, the GPU 884, the CPU 880, and the PCIe switch 882 may be interconnected by a high-speed interconnect, such as, for example, an NVLink interface 888 and / or PCIe connection 886 developed by NVIDIA. In at least one embodiment, the GPU 884 is connected via an NVLink and / or NVSwitch SoC, and the GPU 884 and PCIe switch 882 are connected via a PCIe interconnect. Eight GPU 884s, two CPU 880s, and four PCIe switch 882s have been shown, but this is not limited to them. In at least one embodiment, each of the (one or more) servers 878 may include, but not limited to, any number of GPU 884s, CPU 880s, and / or PCIe switch 882s in any combination. For example, in at least one embodiment, each of the (one or more) servers 878 may include 8, 16, 32, and / or more GPUs 884.

[0184] In at least one embodiment, one or more servers 878 may receive from a vehicle via one or more networks 890 image data representing images showing unexpected or altered road conditions, such as recently started road construction. In at least one embodiment, one or more servers 878 may transmit to a vehicle via one or more networks 890 map information 894, which includes updated or unupdated neural networks 892 and / or, but is not limited to, information about traffic and road conditions. In at least one embodiment, updates to map information 894 may include updates to HD map 822, which includes, but is not limited to, information about construction sites, potholes, detours, floods, and / or other obstacles. In at least one embodiment, the neural network 892 and / or the map information 894 may arise from new training and / or experience represented in data received from any number of vehicles in the environment, and / or from training performed in a data center (for example, using one or more servers 878 and / or other servers).

[0185] In at least one embodiment, one or more servers 878 may be used to train a machine learning model (e.g., a neural network) based at least in part on training data. In at least one embodiment, the training data may be generated by the vehicle and / or generated in a simulation (e.g., using a game engine). In at least one embodiment, any amount of training data may be tagged and / or otherwise preprocessed (e.g., if the relevant neural network benefits from supervised learning). In at least one embodiment, any amount of training data may not be tagged and / or preprocessed (e.g., if the relevant neural network does not require supervised learning). In at least one embodiment, once the machine learning model is trained, the machine learning model may be used by the vehicle (e.g., transmitted to the vehicle via one or more networks 890) and / or the machine learning model may be used by one or more servers 878 to remotely monitor the vehicle.

[0186] In at least one embodiment, one or more servers 878 may receive data from a vehicle and apply the data to a state-of-the-art real-time neural network for real-time intelligent inference. In at least one embodiment, one or more servers 878 may include one or more deep learning supercomputers and / or dedicated AI computers powered by GPUs 884, such as DGX and DGX Station Machines developed by NVIDIA. However, in at least one embodiment, one or more servers 878 may include a deep learning infrastructure using a CPU-powered data center.

[0187] In at least one embodiment, the deep learning infrastructure of one or more servers 878 may be capable of high-speed real-time inference and may use that capability to assess and verify the health of the processor, software, and / or associated hardware in the vehicle 800. For example, in at least one embodiment, the deep learning infrastructure may receive periodic updates from the vehicle 800, such as a series of images and / or objects located in that series of images (e.g., via computer vision and / or other machine learning object classification techniques). In at least one embodiment, the deep learning infrastructure may operate its own neural network to identify objects and compare them to objects identified by the vehicle 800. If the results do not match and the deep learning infrastructure concludes that the AI ​​in the vehicle 800 is malfunctioning, the one or more servers 878 may send a signal to the vehicle 800 instructing the vehicle 800's fail-safe computer to take control, notify passengers, and complete a safe parking operation.

[0188] In at least one embodiment, one or more servers 878 may include one or more GPUs 884 and one or more programmable inference accelerators (e.g., NVIDIA TensorRT3 devices). In at least one embodiment, a combination of a GPU-powered server and inference acceleration may enable real-time response. In at least one embodiment, such as when performance is not critical, servers powered by CPUs, FPGAs, and other processors may be used for inference.

[0189] Figure 9A shows an inference and / or training logic (hardware structure) 915 used to perform inference and / or training operations associated with one or more embodiments (for example, using the converted image data as output by Method 400). Further details regarding the inference and / or training logic 915 are provided below in conjunction with Figures 9A and / or 9B.

[0190] In at least one embodiment, the inference and / or training logic 915 may include, but not limited to, code and / or data storage 901 for storing forward and / or output weights and / or input / output data, and / or other parameters, for constituting neurons or layers of a neural network used for training and / or inference in one or more embodiments. In at least one embodiment, the training logic 915 may include, or be coupled to, code and / or data storage 901 for storing graph code or other software for controlling timing and / or sequence, and weight and / or other parameter information should be loaded into the code and / or data storage 901 to constitute logic including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, the code, such as graph code, loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, the code and / or data storage 901 stores the weight parameters and / or input / output data of each layer of the neural network being trained or used in conjunction with one or more embodiments during the forward propagation of input / output data and / or weight parameters during training and / or inference using the embodiments of one or more embodiments. In at least one embodiment, any portion of the code and / or data storage 901 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0191] In at least one embodiment, any portion of the code and / or data storage 901 may be inside or outside one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or code and / or data storage 901 may be cache memory, dynamic randomly addressable memory ("DRAM"), static randomly addressable memory ("SRAM"), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the selection of whether the code and / or code and / or data storage 901 is inside or outside the processor, or whether it consists of DRAM, SRAM, flash, or some other type of storage, may depend on available storage, on-chip vs. off-chip, latency requirements of the training and / or inference functions being performed, batch size of data used in neural network inference and / or training, or any combination of these factors.

[0192] In at least one embodiment, the inference and / or training logic 915 may include code and / or data storage 905 for storing backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network used to train and / or infer in one or more embodiments. In at least one embodiment, the code and / or data storage 905 stores weight parameters and / or input / output data for each layer of the neural network used to train or in conjunction with one or more embodiments during backward propagation of input / output data and / or weight parameters during training and / or inference using embodiments of one or more embodiments. In at least one embodiment, the training logic 915 may include, or be coupled to, code and / or data storage 905 for storing graph code or other software for controlling timing and / or sequence, and weight and / or other parameter information should be loaded into the code and / or data storage 905 to constitute logic including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, the code, such as graph code, loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, any part of the code and / or data storage 905 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any part of the code and / or data storage 905 may be inside or outside one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 905 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage.In at least one embodiment, the selection of whether the code and / or data storage 905 is, for example, internal or external to the processor, or whether it consists of DRAM, SRAM, flash or some other type of storage, may depend on the available storage, on-chip vs. off-chip, latency requirements of the training and / or inference functions being performed, the batch size of the data used in the neural network inference and / or training, or any combination of these factors.

[0193] In at least one embodiment, the code and / or data storage 901 and the code and / or data storage 905 may be separate storage structures. In at least one embodiment, the code and / or data storage 901 and the code and / or data storage 905 may be the same storage structure. In at least one embodiment, the code and / or data storage 901 and the code and / or data storage 905 may be partially the same storage structure and partially separate storage structures. In at least one embodiment, any portion of the code and / or data storage 901 and the code and / or data storage 905 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0194] In at least one embodiment, the inference and / or training logic 915 may include, but not limited to, one or more arithmetic logic units ("ALUs") 910, including integer and / or floating-point units, for performing logical and / or mathematical operations that are at least partially based on or shown by training and / or inference code (e.g., graph code), the result of which activations (e.g., output values ​​from layers or neurons in a neural network) stored in activation storage 920, and these activations are functions of input / output and / or weight parameter data stored in code and / or data storage 901 and / or code and / or data storage 905. In at least one embodiment, the activation stored in the activation storage 920 is generated according to linear algebra and / or matrix-based mathematics performed by (one or more) ALUs 910 in response to the execution of an instruction or other code, and the weight values ​​stored in the code and / or data storage 905 and / or code and / or data storage 901 are used as operands along with other values ​​such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in the code and / or data storage 905 or code and / or data storage 901, or in other on-chip or off-chip storage.

[0195] In at least one embodiment, the (one or more) ALU910 are contained within one or more processors or other hardware logic devices or circuits, but in another embodiment, the (one or more) ALU910 may be outside the processor or other hardware logic devices or circuits (e.g., coprocessors) that use them. In at least one embodiment, the ALU910 may be contained within an execution unit of a processor, or otherwise contained within a bank of ALUs accessible by execution units of a processor, either within the same processor or distributed across different types of processors (e.g., a central processing unit, a graphics processing unit, a fixed-function unit, etc.). In at least one embodiment, the code and / or data storage 901, the code and / or data storage 905, and the activation storage 920 may be on the same processor or other hardware logic devices or circuits, but in another embodiment, they may be in different processors or other hardware logic devices or circuits, or in some combination of the same processor or other hardware logic devices or circuits and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of the activated storage 920 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Furthermore, inference and / or training code may be stored together with other code accessible to the processor or other hardware logic or circuitry, and may be fetched and / or processed using the processor's fetch, decode, schedule, execute, retirement, and / or other logic circuits.

[0196] In at least one embodiment, the activated storage 920 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the activated storage 920 may be entirely or partially located within or outside one or more processors or other logic circuits. In at least one embodiment, the selection of whether the activated storage 920 is, for example, inside or outside the processor, or whether it consists of DRAM, SRAM, flash, or some other type of storage, may depend on available storage, on-chip vs. off-chip, latency requirements of the training and / or inference functions being performed, batch size of data used in neural network inference and / or training, or any combination of these factors. In at least one embodiment, the inference and / or training logic 915 shown in Figure 9A may be used in conjunction with application-specific integrated circuits ("ASICs"), such as a Tensorflow® processing unit from Google, an inference processing unit (IPU) from Graphcore®, or a Nervana® (e.g., "Lake Crest") processor from Intel Corp. In at least one embodiment, the inference and / or training logic 915 shown in Figure 9A may be used in conjunction with other hardware, such as a central processing unit ("CPU") hardware, a graphics processing unit ("GPU") hardware, or a field-programmable gate array ("FPGA").

[0197] Figure 9B shows the inference and / or training logic 915 in at least one or more embodiments. In at least one embodiment, the inference and / or training logic 915 may include, but is not limited to, hardware logic in which computational resources are either dedicated or, otherwise, used only in conjunction with weight values ​​or other information corresponding to one or more layers of neurons in a neural network. In at least one embodiment, the inference and / or training logic 915 shown in Figure 9B may be used in conjunction with application-specific integrated circuits (ASICs), such as a Tensorflow® processing unit from Google, an Inference Processing Unit (IPU) from Graphcore®, or a Nervana® (e.g., "Lake Crest") processor from Intel Corp. In at least one embodiment, the inference and / or training logic 915 shown in Figure 9B may be used in conjunction with other hardware, such as a central processing unit (CPU) hardware, a graphics processing unit (GPU) hardware, or a field-programmable gate array (FPGA). In at least one embodiment, the inference and / or training logic 915 includes, but is not limited to, code and / or data storage 901 and code and / or data storage 905, which may be used to store code (e.g., graph code), weight values, and / or other information including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. In at least one embodiment shown in Figure 9B, each of the code and / or data storage 901 and code and / or data storage 905 is associated with a dedicated computing resource, such as compute hardware 902 and compute hardware 906, respectively. In at least one embodiment, each of the compute hardware 902 and compute hardware 906 comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on the information stored in the code and / or data storage 901 and code and / or data storage 905, respectively, and the results are stored in the activation storage 920.

[0198] In at least one embodiment, each of the code and / or data storages 901 and 905 and the corresponding compute hardware 902 and 906 correspond to different layers of a neural network, thereby the activation resulting from one “storage / compute pair 901 / 902” of the code and / or data storage 901 and compute hardware 902 is provided as input to the “storage / compute pair 905 / 906” of the code and / or data storage 905 and compute hardware 906 in order to mirror the conceptual organization of the neural network. In at least one embodiment, each of the storage / compute pairs 901 / 902 and 905 / 906 may correspond to two or more neural network layers. In at least one embodiment, additional storage / compute pairs (not shown) may be included in the inference and / or training logic 915 after or in parallel with the storage / compute pairs 901 / 902 and 905 / 906.

[0199] Figure 10 shows an exemplary data center 1000 in which at least one embodiment may be used. In at least one embodiment, the data center 1000 includes a data center infrastructure layer 1010, a framework layer 1020, a software layer 1030, and an application layer 1040.

[0200] In at least one embodiment, as shown in Figure 10, the data center infrastructure layer 1010 may include a resource orchestrator 1012, grouped computing resources 1014, and node computing resources ("node CRs") 1016(1) to 1016(N), where "N" represents any positive integer. In at least one embodiment, the node CRs 1016(1) to 1016(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and cooling modules, etc. In at least one embodiment, one or more nodes CR from among nodes CR1016(1) to 1016(N) may be servers having one or more of the computing resources described above.

[0201] In at least one embodiment, the grouped computing resources 1014 may include separate groupings of node CRs housed in one or more racks (not shown), or many racks housed in a data center at various geographical locations (also not shown). A separate grouping of node CRs within the grouped computing resources 1014 may include grouped compute resources, network resources, memory resources, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs, including CPUs or processors, may be grouped in one or more racks to provide compute resources to support one or more workloads. In at least one embodiment, one or more racks may also include any number of power modules, cooling modules, and network switches in any combination.

[0202] In at least one embodiment, the resource orchestrator 1012 may configure or otherwise control one or more nodes CR1016(1) to 1016(N) and / or a grouped computing resource 1014. In at least one embodiment, the resource orchestrator 1012 may include a software design infrastructure ("SDI") management entity for the data center 1000. In at least one embodiment, the resource orchestrator may include hardware, software, or any combination thereof.

[0203] In at least one embodiment, as shown in Figure 10, the framework layer 1020 includes a job scheduler 1022, a configuration manager 1024, a resource manager 1026, and a distributed file system 1028. In at least one embodiment, the framework layer 1020 may include a framework for supporting software 1032 of the software layer 1030, and / or one or more applications 1042 of the application layer 1040. In at least one embodiment, the software 1032 or (one or more) applications 1042 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. In at least one embodiment, the framework layer 1020 may be a type of free and open-source software web application framework, such as Apache Spark® ("Spark"), which can leverage the distributed file system 1028 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 1022 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 1000. In at least one embodiment, the configuration manager 1024 may be able to configure different layers, such as the software layer 1030 and the framework layer 1020, which includes Spark and a distributed file system 1028 to support large-scale data processing. In at least one embodiment, the resource manager 1026 may be able to manage clustered or grouped computing resources that are mapped or allocated to support the distributed file system 1028 and the job scheduler 1022. In at least one embodiment, the clustered or grouped computing resources may include a grouped computing resource 1014 in the data center infrastructure layer 1010.In at least one embodiment, the resource manager 1026 may work in conjunction with the resource orchestrator 1012 to manage these mapped or allocated computing resources.

[0204] In at least one embodiment, the software 1032 contained within the software layer 1030 may include software used by nodes CR1016(1) to 1016(N), grouped computing resources 1014, and / or at least a portion of the distributed file system 1028 of the framework layer 1020. One or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0205] In at least one embodiment, one or more applications 1042 contained within the application layer 1040 may include one or more types of applications used by nodes CR1016(1) to 1016(N), grouped computing resources 1014, and / or at least a portion of the distributed file system 1028 of the framework layer 1020. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive compute, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0206] In at least one embodiment, any of the configuration manager 1024, resource manager 1026, and resource orchestrator 1012 may implement any number and type of self-correcting actions based on any amount and type of data obtained in any technically feasible manner. In at least one embodiment, the self-correcting actions may relieve the data center operator of data center 1000 of the task of determining potentially faulty configurations and potentially avoiding underutilized and / or underperforming portions of the data center.

[0207] In at least one embodiment, the data center 1000 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by computing weight parameters according to a neural network architecture using the software and computing resources described above with respect to the data center 1000. In at least one embodiment, a trained machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to the data center 1000 by using weight parameters computed through one or more training techniques described herein.

[0208] In at least one embodiment, the data center may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, or other hardware to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above may be configured as services that enable users to train information or perform inference on information, such as image recognition, speech recognition, or other artificial intelligence services.

[0209] In some embodiments, the inference and / or training logic 915 is used to perform inference and / or training operations using the transformed image data generated as described above.

[0210] Figure 11 is a block diagram showing an exemplary computer system, which may be a system 1100 having interconnected devices and components, a system-on-a-chip (SOC), or any combination thereof, formed together with a processor that may include an execution unit for executing instructions, according to at least one embodiment. In at least one embodiment, the computer system 1100 may include components such as a processor 1102 for employing an execution unit including logic for implementing algorithms for process data, as described herein, but not limited to the embodiments described herein. In at least one embodiment, the computer system 1100 may include a processor such as the PENTIUM® processor family, Xeon®, Itanium®, XScale®, and / or StrongARM®, Intel® Core®, or Intel® Nervana® microprocessors, available from Intel Corporation in Santa Clara, California, but other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, etc.) may also be used. In at least one embodiment, the computer system 1100 may run a version of the WINDOWS® operating system available from Microsoft Corporation in Redmond, Washington, but other operating systems (e.g., UNIX® and Linux®), embedded software, and / or graphical user interfaces may also be used.

[0211] The embodiments may be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, the embedded application may include a microcontroller, a digital signal processor ("DSP"), a system-on-a-chip, a network computer ("NetPC"), a set-top box, a network hub, a wide area network ("WAN") switch, or any other system capable of implementing one or more instructions according to at least one embodiment.

[0212] In at least one embodiment, the computer system 1100 may include, but is not limited to, a processor 1102, which may include, but is not limited to, one or more execution units 1108 for performing machine learning model training and / or inference by the techniques described herein. In at least one embodiment, the computer system 1100 is a single-processor desktop or server system, while in another embodiment, the computer system 1100 may be a multi-processor system. In at least one embodiment, the processor 1102 may include, but is not limited to, a complex instruction set computer ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, a processor implementing a combination of instruction sets, or any other processor device such as a digital signal processor. In at least one embodiment, the processor 1102 may be coupled to a processor bus 1110, and the processor bus 1110 may transmit data signals between the processor 1102 and other components in the computer system 1100.

[0213] In at least one embodiment, the processor 1102 may include, but is not limited to, a level 1 ("L1") internal cache memory ("cache") 1104. In at least one embodiment, the processor 1102 may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory may reside outside the processor 1102. Other embodiments may also include a combination of both internal and external caches, depending on the specific implementation and requirements. In at least one embodiment, the register file 1106 may store different types of data in various registers, including, but is not limited to, integer registers, floating-point registers, status registers, and instruction pointer registers.

[0214] In at least one embodiment, but not limited to, an execution unit 1108 containing logic for performing integer and floating-point arithmetic is also present in the processor 1102. In at least one embodiment, the processor 1102 may also include a microcode ("u-code") read-only memory ("ROM") for storing microcode for several macro instructions. In at least one embodiment, the execution unit 1108 may include logic for handling a packed instruction set 1109. In at least one embodiment, by including the packed instruction set 1109, along with the associated circuit elements for executing the instructions, in the instruction set of the general-purpose processor 1102, arithmetic used by many multimedia applications can be performed using packed data in the general-purpose processor 1102. In one or more embodiments, many multimedia applications may be accelerated and run more efficiently by using the full width of the processor's data bus to perform arithmetic on packed data, which may eliminate the need to transfer smaller units of data across the processor's data bus to perform one or more arithmetic operations, one data element at a time.

[0215] In at least one embodiment, the execution unit 1108 may also be used in a microcontroller, embedded processor, graphics device, DSP, and other types of logic circuits. In at least one embodiment, the computer system 1100 may include, but is not limited to, memory 1120. In at least one embodiment, memory 1120 may be implemented as a dynamic random access memory ("DRAM") device, a static random access memory ("SRAM") device, a flash memory device, or other memory device. In at least one embodiment, memory 1120 may store (one or more) instructions 1119 and / or data 1121, which are represented by data signals that can be executed by the processor 1102.

[0216] In at least one embodiment, a system logic chip may be coupled to the processor bus 1110 and memory 1120. In at least one embodiment, the system logic chip may include, but is not limited to, a memory controller hub ("MCH") 1116, and the processor 1102 may communicate with the MCH 1116 via the processor bus 1110. In at least one embodiment, the MCH 1116 may provide a high-bandwidth memory path 1118 to memory 1120 for instruction and data storage, as well as for the storage of graphics commands, data, and textures. In at least one embodiment, the MCH 1116 may direct data signals between the processor 1102, memory 1120, and other components in the computer system 1100, and bridge data signals between the processor bus 1110, memory 1120, and system I / O 1122. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH1116 may be coupled to memory 1120 through a high-bandwidth memory path 1118, and the graphics / video card 1112 may be coupled to the MCH1116 through an Accelerated Graphics Port ("AGP") interconnect 1114.

[0217] In at least one embodiment, the computer system 1100 may use a system I / O 1122, which is a proprietary hub interface bus for coupling the MCH 1116 to the I / O controller hub ("ICH") 1130. In at least one embodiment, the ICH 1130 may provide direct connectivity to several I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, but is not limited to, a high-speed I / O bus for connecting peripherals to memory 1120, a chipset, and a processor 1102. Examples may include, but are not limited to, an audio controller 1129, a firmware hub ("Flash BIOS") 1128, a wireless transceiver 1126, data storage 1124, a legacy I / O controller 1123 including a user input and keyboard interface 1125, a serial expansion port 1127 such as a Universal Serial Bus ("USB"), and a network controller 1134. The data storage 1124 may comprise a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0218] In at least one embodiment, Figure 11 shows a system including interconnected hardware devices or “chips,” while in other embodiments, Figure 11 may show an exemplary system-on-a-chip (“SoC”). In at least one embodiment, devices may be interconnected by proprietary interconnects, standard interconnects (e.g., PCIe), or any combination thereof. In at least one embodiment, one or more components of computer system 1100 are interconnected using compute express link (CXL) interconnects.

[0219] In some embodiments, the inference and / or training logic 915 is used to perform inference and / or training operations using the transformed image data generated as described above. Further details regarding the inference and / or training logic 915 are provided below in conjunction with Figures 9A and / or 9B. In at least one embodiment, the inference and / or training logic 915 may be used in the system of Figure 11 for inference or prediction operations, at least in part, based on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0220] Such components can be used to generate synthetic data that mimics failure scenarios in the network training process, which can help improve network performance while limiting the amount of synthetic data to avoid overfitting.

[0221] Figure 12 is a block diagram showing an electronic device 1200 for utilizing the processor 1210 according to at least one embodiment. In at least one embodiment, the electronic device 1200 may be, for example, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop, a tablet, a mobile device, a telephone, an embedded computer, or any other suitable electronic device, for example, but not limited to.

[0222] In at least one embodiment, the system 1200 may include a processor 1210 communicably coupled to any number or type of preferred components, peripherals, modules, or devices, but not limited to these. In at least one embodiment, the processor 1210 is coupled using a bus or interface such as a 1°C bus, a System Management Bus ("SMBus"), a Low Pin Count (LPC) bus, a Serial Peripheral Interface ("SPI"), a High Definition Audio ("HDA") bus, a Serial Advance Technology Attachment ("SATA") bus, a Universal Serial Bus ("USB") (versions 1, 2, or 3), or a Universal Asynchronous Receiver / Transmitter ("UART") bus. In at least one embodiment, Figure 12 shows a system including interconnected hardware devices or “chips,” while in other embodiments, Figure 12 may show an exemplary system-on-a-chip (“SoC”). In at least one embodiment, the devices shown in Figure 12 may be interconnected by proprietary interconnects, standard interconnects (e.g., PCIe), or any combination thereof. In at least one embodiment, one or more components of Figure 12 are interconnected using a Compute Express Link (CXL) interconnect.

[0223] In at least one embodiment, Figure 12 shows a display 1224, a touchscreen 1225, a touchpad 1230, a Near Field Communication ("NFC") unit 1245, a sensor hub 1240, a thermal sensor 1246, an Express Chipset ("EC") 1235, a Trusted Platform Module ("TPM") 1238, a BIOS / firmware / flash memory ("BIOS,FW flash") 1222, a DSP 1260, a drive 1220 such as a Solid State Disk ("SSD") or Hard Disk Drive ("HDD"), a Wireless Local Area Network ("WLAN") unit 1250, a Bluetooth unit 1252, and a Wireless Wide Area Network ("WWAN") unit. The components may include a network (1256), a Global Positioning System (GPS) (1255), a camera such as a USB 3.0 camera ("USB 3.0 camera") (1254), and / or a Low Power Double Data Rate ("LPDDR") memory unit ("LPDDR3") (1215) implemented, for example, in the LPDDR3 standard. Each of these components may be implemented in any preferred manner.

[0224] In at least one embodiment, other components may be communicatively coupled to the processor 1210 through the components described above. In at least one embodiment, the accelerometer 1241, the ambient light sensor ("ALS") 1242, the compass 1243, and the gyroscope 1244 may be communicatively coupled to the sensor hub 1240. In at least one embodiment, the thermal sensor 1239, the fan 1237, the keyboard 1246, and the touchpad 1230 may be communicatively coupled to the EC 1235. In at least one embodiment, the speaker 1263, the headphones 1264, and the microphone ("mic") 1265 may be communicatively coupled to the audio unit ("audio codec and class d amplifier") 1262, and the audio unit 1262 may be communicatively coupled to the DSP 1260. In at least one embodiment, the audio unit 1264 may include, for example, an audio coder / decoder ("codec") and a Class D amplifier. In at least one embodiment, a SIM card ("SIM") 1257 may be communicatively coupled to the WWAN unit 1256. In at least one embodiment, components such as the WLAN unit 1250 and the Bluetooth unit 1252, as well as the WWAN unit 1256, may be implemented in a Next Generation Form Factor ("NGFF").

[0225] The inference and / or training logic 915 is used to perform inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 915 are provided below in conjunction with Figures 9A and / or 9B. In at least one embodiment, the inference and / or training logic 915 may be used in the system of Figure 12 for inference or prediction operations, at least in part, based on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0226] Such components can be used to generate synthetic data that mimics failure scenarios in the network training process, which can help improve network performance while limiting the amount of synthetic data to avoid overfitting.

[0227] Figure 13 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, system 1300 includes one or more processors 1302 and one or more graphics processors 1308, and may be a single-processor desktop system, a multi-processor workstation system, or a server system having a large number of processors 1302 or processor cores 1307. In at least one embodiment, system 1300 is a processing platform embedded in a system-on-a-chip (SoC) integrated circuit for use in a mobile device, handheld device, or embedded device.

[0228] In at least one embodiment, system 1300 may include, or may be incorporated within, a server-based gaming platform, a game console including a game and media console, a mobile gaming console, a handheld game console, or an online game console. In at least one embodiment, system 1300 is a mobile phone, a smartphone, a tablet computing device, or a mobile internet device. In at least one embodiment, processing system 1300 may also include, can be coupled with, or may be incorporated within, wearable devices such as a smartwatch wearable device, a smart eyewear device, an augmented reality device, or a virtual reality device. In at least one embodiment, processing system 1300 is a television or set-top box device having one or more processors 1302 and a graphical interface produced by one or more graphics processors 1308.

[0229] In at least one embodiment, one or more processors 1302 each include one or more processor cores 1307 for processing instructions that perform actions for system and user software when executed. In at least one embodiment, each of the one or more processor cores 1307 is configured to process a particular instruction set 1309. In at least one embodiment, the instruction set 1309 may facilitate computing via composite instruction set computing (CISC), reduced instruction set computing (RISC), or very long instruction words (VLIW). In at least one embodiment, each processor core 1307 may process a different instruction set 1309, and the instruction set 1309 may include instructions to facilitate emulation of other instruction sets. In at least one embodiment, the processor core 1307 may also include other processing devices, such as a digital signal processor (DSP).

[0230] In at least one embodiment, the processor 1302 includes cache memory 1304. In at least one embodiment, the processor 1302 may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory is shared among the various components of the processor 1302. In at least one embodiment, the processor 1302 also uses an external cache (e.g., a Level 3 (L3) cache or a Last Level Cache (LLC)) (not shown), and the external cache may be shared among the processor cores 1307 using known cache coherency techniques. In at least one embodiment, additionally, a register file 1306 is included in the processor 1302, and the register file 1306 may include different types of registers for storing different types of data (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers). In at least one embodiment, the register file 1306 may include general-purpose registers or other registers.

[0231] In at least one embodiment, one or more processors 1302 are coupled with one or more interface buses 1310 to transmit communication signals, such as addresses, data, or control signals, between the processors 1302 and other components in the system 1300. In at least one embodiment, the interface bus 1310 in one embodiment may be a processor bus, such as a version of the Direct Media Interface (DMI) bus. In at least one embodiment, the interface 1310 is not limited to the DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), a memory bus, or other types of interface buses. In at least one embodiment, the (one or more) processors 1302 include an integrated memory controller 1316 and a platform controller hub 1330. In at least one embodiment, the memory controller 1316 facilitates communication between the memory device and other components of the system 1300, and the platform controller hub (PCH) 1330 provides connectivity to I / O devices via the local I / O bus.

[0232] In at least one embodiment, the memory device 1320 may be a dynamic random-access memory (DRAM) device, a static random-access memory (SRAM) device, a flash memory device, a phase-change memory device, or any other memory device having performance suitable for acting as process memory. In at least one embodiment, the memory device 1320 may act as system memory for the system 1300 to store data 1322 and instructions 1321 for use when one or more processors 1302 execute an application or process. In at least one embodiment, the memory controller 1316 may also be coupled to an optional external graphics processor 1312, which may communicate with one or more graphics processors 1308 in the processor 1302 to perform graphics and media operations. In at least one embodiment, a display device 1311 may be connected to one or more processors 1302. In at least one embodiment, the display device 1311 may include one or more internal display devices, such as those found in mobile electronic devices or laptop devices, or external display devices attached via a display interface (e.g., DisplayPort). In at least one embodiment, the display device 1311 may include a head-mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

[0233] In at least one embodiment, the platform controller hub 1330 enables peripherals to connect to the memory device 1320 and processor 1302 via a high-speed I / O bus. In at least one embodiment, the I / O peripherals include, but are not limited to, an audio controller 1346, a network controller 1334, a firmware interface 1328, a wireless transceiver 1326, a touch sensor 1325, and a data storage device 1324 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1324 may be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensor 1325 may include a touch screen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, the wireless transceiver 1326 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or Long-Term Evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1328 enables communication with system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, the network controller 1334 may enable network connectivity to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to the interface bus 1310. In at least one embodiment, the audio controller 1346 is a multi-channel high-definition audio controller.In at least one embodiment, system 1300 includes an optional legacy I / O controller 1340 for connecting legacy devices (e.g., Personal System 2 (PS / 2)) to the system. In at least one embodiment, platform controller hub 1330 can also connect to one or more Universal Serial Bus (USB) controller 1342-connected input devices, such as a keyboard and mouse combination 1343, a camera 1344, or other USB input devices.

[0234] In at least one embodiment, instances of the memory controller 1316 and the platform controller hub 1330 may be integrated into a discreet external graphics processor, such as an external graphics processor 1312. In at least one embodiment, the platform controller hub 1330 and / or the memory controller 1316 may be external to one or more processors 1302. For example, in at least one embodiment, system 1300 may include an external memory controller 1316 and a platform controller hub 1330, which may be configured as a memory controller hub and peripheral controller hub within a system chipset communicating with (one or more) processors 1302.

[0235] The inference and / or training logic 915 is used to perform inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 915 are provided below in conjunction with Figures 9A and / or 9B. In at least one embodiment, part or all of the inference and / or training logic 915 may be incorporated into the graphics processor 1312. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more ALUs embodied in the graphics processor. Furthermore, in at least one embodiment, the inference and / or training operations described herein may be performed using logic other than the logic shown in Figure 9A or Figure 9B. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that constitute the ALUs of the graphics processor for performing one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0236] Such components can be used to generate synthetic data that mimics failure scenarios in the network training process, which can help improve network performance while limiting the amount of synthetic data to avoid overfitting.

[0237] Figure 14 is a block diagram of a processor 1400 having one or more processor cores 1402A to 1202N, an integrated memory controller 1414, and an integrated graphics processor 1408, according to at least one embodiment. In at least one embodiment, the processor 1400 may include additional cores up to additional core 1402N, represented by dashed boxes. In at least one embodiment, each of the processor cores 1402A to 1202N includes one or more internal cache units 1404A to 1404N. In at least one embodiment, each processor core also has access to one or more shared cached units 1406.

[0238] In at least one embodiment, the internal cache units 1404A-1404N and the shared cache unit 1406 represent a cache memory hierarchy within the processor 1400. In at least one embodiment, the cache memory units 1404A-1404N may include at least one level of instruction and data cache within each processor core, and one or more levels of shared intermediate level caches such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, where the highest level cache prior to external memory is classified as LLC. In at least one embodiment, cache coherency logic maintains coherency among the various cache units 1406 and 1404A-1404N.

[0239] In at least one embodiment, the processor 1400 may also include a set of one or more bus controller units 1416 and a system agent core 1410. In at least one embodiment, one or more bus controller units 1416 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1410 provides management functionality for various processor components. In at least one embodiment, the system agent core 1410 includes one or more integrated memory controllers 1414 for managing access to various external memory devices (not shown).

[0240] In at least one embodiment, one or more of the processor cores 1402A to 1402N include support for simultaneous multithreading. In at least one embodiment, the system agent core 1410 includes components for coordinating and operating the cores 1402A to 1402N during multithreaded processing. In at least one embodiment, the system agent core 1410 may additionally include a power control unit (PCU), the PCU including logic and components for regulating the power state of one or more of the processor cores 1402A to 1402N and the graphics processor 1408.

[0241] In at least one embodiment, the processor 1400 additionally includes a graphics processor 1408 for performing graphics processing operations. In at least one embodiment, the graphics processor 1408 is coupled with a system agent core 1410 which includes a shared cache unit 1406 and one or more integrated memory controllers 1414. In at least one embodiment, the system agent core 1410 also includes a display controller 1411 for driving graphics processor outputs to one or more coupled displays. In at least one embodiment, the display controller 1411 may also be a separate module coupled with the graphics processor 1408 via at least one interconnection, or it may be incorporated within the graphics processor 1408.

[0242] In at least one embodiment, a ring-based interconnect unit 1412 is used to connect the internal components of the processor 1400. In at least one embodiment, alternative interconnect units such as point-to-point interconnects, switching interconnects, or other techniques may be used. In at least one embodiment, the graphics processor 1408 is connected to the ring interconnect 1412 via an I / O link 1413.

[0243] In at least one embodiment, I / O link 1413 represents at least one of several types of I / O interconnects, including on-package I / O interconnects that facilitate communication between various processor components and high-performance embedded memory modules 1418, such as eDRAM modules. In at least one embodiment, each of the processor cores 1402A to 1402N and the graphics processor 1408 use the embedded memory module 1418 as a shared last-level cache.

[0244] In at least one embodiment, the processor cores 1402A to 1402N are homogeneous cores that execute a common instruction set architecture. In at least one embodiment, the processor cores 1402A to 1402N are heterogeneous in terms of instruction set architecture (ISA), where one or more of the processor cores 1402A to 1402N execute a common instruction set, and the other cores of one or more of the processor cores 1402A to 1402N execute a subset of the common instruction set or a different instruction set. In at least one embodiment, the processor cores 1402A to 1402N are heterogeneous in terms of microarchitecture, where one or more cores with relatively high power consumption are coupled with one or more power cores with lower power consumption. In at least one embodiment, the processor 1400 may be implemented on one or more chips or as an SoC integrated circuit.

[0245] In some embodiments, the inference and / or training logic 915 is used to perform inference and / or training operations using transformed image data (for example, generated by an RGB-IR camera and then processed based on lighting conditions). Details relating to the inference and / or training logic 915 are provided below in conjunction with Figures 9A and / or 9B. In at least one embodiment, part or all of the inference and / or training logic 915 may be incorporated into the processor 1400. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more of the graphics processor 112, (one or more) graphics cores 1402A-1402N, or other components embodied in Figure 14. Furthermore, in at least one embodiment, the inference and / or training operations described herein may be performed using logic other than the logic shown in Figure 9A or Figure 9B. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that constitute the ALU of a graphics processor for implementing one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0246] Such components can be used to generate synthetic data that mimics failure scenarios in the network training process, which can help improve network performance while limiting the amount of synthetic data to avoid overfitting.

[0247] Other variations are within the scope of this disclosure. Thus, the disclosed techniques can be modified and constructed in various ways, several exemplary embodiments of which are shown in the drawings and described in detail above. However, it should be understood that this disclosure is not intended to limit itself to any particular one or more disclosed forms, but rather to encompass all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure, as defined in the appended claims.

[0248] In the context describing the disclosed embodiments (particularly in the context of the following claims), the terms “a,” “an,” and “the,” and similar demonstrative pronouns, should be interpreted as encompassing both singular and plural, and not as definitions of terms, unless otherwise stated herein or clearly refuted by the context. The terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to,”) unless otherwise stated. The term “connected,” when unmodified and referring to a physical connection, should be interpreted as being partially or completely contained, attached, or joined to one another, even if there is something intervening. The detailing of ranges of values ​​herein is merely intended to serve as a concise way of individually referring to each distinct value that falls within a range, unless otherwise stated herein and unless each distinct value is incorporated into the specification as if it were individually detailed herein. The use of the terms “set” (for example, “set of items”) or “subset” should be interpreted as a non-empty set comprising one or more members, unless otherwise stated or denied by the context. Furthermore, unless otherwise stated or denied by the context, the term “subset” of a corresponding set does not necessarily refer to a strict subset of the corresponding set, and a subset and a corresponding set can be equivalent.

[0249] Combinations such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in contexts generally used to indicate that an item, term, etc., can be either A, B, or C, or any non-empty subset of the set of A, B, and C, unless otherwise specifically stated or explicitly denied by the context. For example, in a descriptive example of a set having three members, the combinations “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such combinations do not imply as a whole that some embodiments require the presence of each of A, B, and C. Furthermore, unless otherwise stated or negated by the context, the term "plurality" refers to a state of being multiple (for example, "a plurality of items" refers to multiple items). Plurality refers to at least two items, but can be more when explicitly or contextually indicated as such. Moreover, unless otherwise stated or otherwise evident from the context, the phrase "based on" means "at least partially based on," and does not mean "based solely on."

[0250] The operation of the processes described herein may be carried out in any preferred order unless otherwise stated herein or explicitly refuted by the context. In at least one embodiment, a process such as the process described herein (or its variations and / or combinations thereof) is carried out under the control of one or more computer systems consisting of executable instructions and is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or by a combination thereof. In at least one embodiment, the code is stored in a computer-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-temporary computer-readable storage medium that excludes temporary signals (e.g., transient electrical or electromagnetic transmissions that propagate) but includes non-temporary data storage circuit elements (e.g., buffers, caches, and queues) in transceivers for temporary signals. In at least one embodiment, code (e.g., executable code or source code) is stored in a set of one or more non-temporary computer-readable storage media which, when executed by one or more processors of a computer system (i.e., as a result of execution), causes the computer system to perform the operations described herein (or has other memory for storing executable instructions). In at least one embodiment, the set of non-temporary computer-readable storage media comprises a plurality of non-temporary computer-readable storage media which one or more of the individual non-temporary storage media of the plurality of non-temporary computer-readable storage media do not contain all of the code, but the plurality of non-temporary computer-readable storage media collectively contain all of the code.In at least one embodiment, executable instructions are executed such that different instructions are executed by different processors, for example, a non-temporary computer-readable storage medium stores the instructions, the main central processing unit ("CPU") executes some of the instructions, and the graphics processing unit ("GPU") executes others. In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0251] Accordingly, in at least one embodiment, the computer system is configured to implement one or more services that individually or collectively perform the operations of the processes described herein, and such a computer system consists of applicable hardware and / or software that enables the performance of the operations. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, a distributed computer system comprising multiple devices operating in different ways so that the distributed computer system performs the operations described herein and the single device does not perform all of the operations.

[0252] Any use of any examples or illustrative language provided herein (e.g., "such as") is intended solely to further illustrate the embodiments of this disclosure and, unless otherwise asserted, does not limit the scope of this disclosure. Nothing in this specification should be construed as indicating any unclaimed element as essential to the practice of this disclosure.

[0253] All references cited herein, including publications, patent applications, and patents, are incorporated herein by reference to the same extent as if they were included herein in their entirety, provided that each reference is clearly indicated by individual reference.

[0254] In the specification and claims, the terms “joined” and “connected” may be used together with their derivatives. It should be understood that these terms may not be intended to be synonymous with one another. Rather, in certain instances, “connected” or “joined” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with one another. “Joined” may also mean that two or more elements are not in direct contact with one another, but still interact or communicate with one another.

[0255] Unless otherwise specifically stated, throughout this specification, terms such as “processing,” “computing,” “calculating,” or “determining” should be understood to refer to actions and / or processes of a computer or computing system, or similar electronic computing device, that manipulate and / or transform data, represented as electronic or other physical quantities, in the registers and / or memory of the computing system, into other data, similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display device of the computing system.

[0256] Similarly, the term “processor” may refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. In non-limiting examples, “processor” may be a CPU, GPU, DPU, or hardware accelerator. A “computing platform” may comprise one or more processors. As used herein, “software” processes may include software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Each process may also refer to multiple processes for executing instructions serially or in parallel, continuously or intermittently. The terms “system” and “method” are used interchangeably herein only if one or more methods can be embodied by a system, and a method can be considered a system.

[0257] This specification may refer to obtaining, acquiring, receiving, or inputting analog data or digital data into a subsystem, computer system, or computer-implemented machine. Obtaining, acquiring, receiving, or inputting analog data and digital data can be implemented in various ways, such as by receiving data as parameters of a function call or a call to an application programming interface. In some implementations, the process of obtaining, acquiring, receiving, or inputting analog data or digital data can be implemented by transferring the data via a serial or parallel interface. In another implementation, the process of obtaining, acquiring, receiving, or inputting analog data or digital data can be implemented by transferring the data over a computer network from a providing entity to an acquiring entity. Reference may also be made to providing, outputting, transmitting, sending, or presenting analog data or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog data or digital data can be implemented by transferring the data as input or output parameters of a function call, parameters of an application programming interface or an inter-process communication mechanism.

[0258] While the foregoing description sets forth exemplary implementations of the described techniques, it is contemplated that other architectures may be used to implement the described functionality and are within the scope of the present disclosure. Furthermore, although specific distributions of responsibilities have been defined above for purposes of description, various functions and responsibilities may be distributed and divided in different ways depending on circumstances.

[0259] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claims.

[0260] Several portions of the preceding detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the ways used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, considered to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Typically, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0261] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure may refer to actions and processes of a computer system, or similar electronic computing device, that manipulate data represented as physical (electronic) quantities within the computer system's registers and memories and transform it into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage systems.

[0262] This disclosure also relates to an apparatus for carrying out the operations described herein. This apparatus may be constructed specifically for the intended purpose, or it may include a general-purpose computer that is selectively activated or reconfigured by a computer program stored in the computer. Such computer programs may be stored in computer-readable storage media, including, but are not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, and magneto-optical disks, each coupled to a computer system bus, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions.

[0263] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general-purpose systems may be used with the programs taught herein, or it may be advantageous to construct more specialized devices for carrying out the methods. Structures for various such systems will appear as described below. Furthermore, this disclosure does not describe any particular programming language. It will be understood that various programming languages ​​may be used to implement the teachings of this disclosure described herein.

[0264] This disclosure may be provided as a computer program product or software that may include a machine-readable medium storing instructions, which may be used to program a computer system (or other electronic device) to perform a process in accordance with this disclosure. The machine-readable medium includes a mechanism for storing information in a machine-readable format. In some embodiments, the machine-readable (e.g., non-temporary computer-readable) medium includes machine-readable storage media such as read-only memory ("ROM"), random-access memory ("RAM"), magnetic disk storage media, optical storage media, and flash memory components.

[0265] In the above specifications, the embodiments of the Disclosure have been described in relation to those specific exemplary embodiments. It will be apparent that various modifications can be made to them without departing from the broader spirit and scope of the embodiments of the Disclosure described in the following claims. Accordingly, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. Receive image data including infrared values ​​and visible light values. Based on the comparison between the infrared value and the visible light value, a scene detection value for the image data is calculated. Based on the scene detection values, several quantities, including the infrared subtraction factor and the white balance correction factor, are determined. The image data is transformed by controlling the amount of infrared correction to be applied to the image data based on the infrared subtraction factor, and by controlling the amount of white balance correction to be applied to the infrared-corrected image data based on the white balance correction factor. One or more processing units A processor equipped with the following features.

2. The processor according to claim 1, wherein the image data includes raw image data of an image sensor including a pixel sensor for red, blue, green, and infrared (RGB-IR), the infrared value includes an infrared pixel value, and the visible light value includes a red pixel value, a blue pixel value, and a green pixel value.

3. The processor according to claim 1, wherein the scene detection value controls the amount of infrared correction and the amount of white balance correction to be applied to the image data based on the relationship between the infrared value and the visible light value present in the image data, and the amount of infrared correction and the amount of white balance correction have an inverse relationship with the scene detection value.

4. The processor according to claim 1, wherein the scene detection value includes the ratio between the intensity of the infrared value and the intensity of the visible light value.

5. The processor according to claim 4, wherein the intensity of the visible light value is determined at least in part on the intensity of the green pixel value.

6. The one or more processing units described above, A first value representing the intensity of the infrared value is calculated, which includes the average of multiple infrared pixel values ​​in the image data. A second value representing the intensity of the visible light value is calculated, which includes the average of the values ​​of multiple green pixels in the image data. The scene detection value is generated based on the first and second values. The processor according to claim 1, which calculates the scene detection value by doing so.

7. The one or more processing units described above, Based on the scene detection values, the multiple quantities, including the color correction factor and the color saturation factor, are determined. The image data is transformed by controlling the amount of color correction to be applied to the image data after white balance correction based on the color correction factor, and by controlling the amount of color saturation to be applied to the image data after color correction based on the color saturation factor. The processor according to claim 1.

8. The processor according to claim 1, wherein one or more processing units determine the plurality of quantities by using the scene detection value as an index for selecting the plurality of quantities from a set of quantities.

9. The one or more processing units, based on the scene detection value, In response to calculating a first scene detection value based on a first ratio of the infrared value to the visible light value, a first infrared subtraction factor is determined to subtract a first amount caused by infrared from the visible light value. In response to calculating a second scene detection value based on a second ratio of the infrared value to the visible light value that is higher than the first ratio of the infrared value to the visible light value, a second infrared subtraction factor is determined, which subtracts a second amount attributable to infrared radiation, less than the first amount, from the visible light value. The processor according to claim 1, wherein the infrared subtraction factor is determined by the above.

10. The one or more processing units, based on the scene detection value, In response to calculating the first scene detection value, a first white balance correction factor is determined that takes into account the first remaining visible light value caused by infrared radiation. In response to calculating the second scene detection value, a second white balance correction factor is determined that takes into account the second remaining visible light value attributable to infrared radiation. The processor according to claim 9, which determines the white balance correction factor by doing so.

11. The processor according to claim 1, wherein one or more processing units determine the plurality of quantities by performing a lookup using a lookup table that relates the scene detection values ​​to the infrared subtraction factor and the white balance correction factor, based on the scene detection values.

12. The processor according to claim 1, wherein the image data is in an RGB space including a red component, a green component, and a blue component, and the converted image data is in a YUV space including a luminance component and a chrominance component.

13. The aforementioned one or more processing units further, The transformed image data is processed using a machine learning model trained to output at least one of prediction, estimation, or classification based on the input image data. Based on the output generated by the machine learning model, one or more actions are performed. The processor according to claim 1.

14. One or more image sensors configured to generate image data including infrared values ​​and visible light values, Upon receiving the aforementioned image data, Based on the comparison between the infrared value and the visible light value, a scene detection value for the image data is calculated. Based on the scene detection values, several quantities, including the infrared subtraction factor and the white balance correction factor, are determined. The image data is transformed by controlling the amount of infrared correction to be applied to the image data based on the infrared subtraction factor, and by controlling the amount of white balance correction to be applied to the infrared-corrected image data based on the white balance correction factor. One or more processing units and A system equipped with these features.

15. Display device for displaying the converted image data. The system according to claim 14, further comprising:

16. The system according to claim 15, wherein one or more image sensors are configured to generate the image data inside the vehicle, and the display device is disposed on the instrument panel of the vehicle.

17. The system according to claim 14, wherein the scene detection value controls the amount of infrared correction and the amount of white balance correction to be applied to the image data based on the relationship between the infrared value and the visible light value present in the image data, and the amount of infrared correction and the amount of white balance correction have an inverse relationship with the scene detection value.

18. The system according to claim 14, wherein the scene detection value includes the ratio between the intensity of the infrared value and the intensity of the visible light value.

19. The one or more processing units, based on the scene detection value, In response to calculating a first scene detection value based on a first ratio of the infrared value to the visible light value, a first infrared subtraction factor is determined to subtract a first amount caused by infrared from the visible light value. In response to calculating a second scene detection value based on a second ratio of the infrared value to the visible light value that is higher than the first ratio of the infrared value to the visible light value, a second infrared subtraction factor is determined, which subtracts a second amount attributable to infrared radiation, less than the first amount, from the visible light value. The system according to claim 14, wherein the infrared subtraction factor is determined by the above.

20. The aforementioned system Control systems for autonomous or semi-autonomous machines, Perceptual systems for autonomous or semi-autonomous machines, Driver monitoring systems for autonomous or semi-autonomous machines, Crew monitoring systems for autonomous or semi-autonomous machines, A system for performing simulation operations. A system for implementing digital twin operation. A system for performing deep learning operations. Systems implemented using edge devices, Systems implemented using robots, A system incorporating one or more virtual machines (VMs), A system that will be implemented, at least partially, in a data center. A system including a collaborative creation platform for three-dimensional (3D) content, or A system implemented using cloud computing resources, at least partially. The system according to claim 14, comprising at least one of the following.

21. A method that is executed by a processor, The steps include receiving image data that includes infrared values ​​and visible light values, A step of calculating a scene detection value for the image data based on a comparison between the infrared value and the visible light value, The steps include determining a plurality of quantities, including an infrared subtraction factor and a white balance correction factor, based on the scene detection value, The steps include: converting the image data by controlling the amount of infrared correction to be applied to the image data based on the infrared subtraction factor, and controlling the amount of white balance correction to be applied to the infrared-corrected image data based on the white balance correction factor; Methods that include...

Images