Glare mitigation using image contrast analysis for autonomous systems and applications

Abstract

The present disclosure relates to glare mitigation using image contrast analysis for autonomous systems and applications. In various examples, a contrast value corresponding to pixels of one or more images generated using one or more sensors of a vehicle can be computed to detect and identify objects that trigger glare mitigation operations. A pixel luminance value is determined and used to compute a contrast value based on comparing the pixel luminance value to a reference luminance value based on a set of pixels and corresponding luminance values. A contrast threshold can be applied to the computed contrast value to identify glare in the image data, thereby triggering a glare mitigation operation so that the vehicle can modify a configuration of one or more illumination sources to reduce the glare experienced by occupants and / or sensors of the vehicle.

Description

Background Technology

[0001] Autonomous and semi-autonomous driving systems, as well as advanced driver assistance systems (ADAS), can use sensors (such as cameras) to form an understanding of the vehicle's surroundings in real-time or near real-time. This understanding can include information about the location of objects, obstacles, road signs, road surfaces, and / or other markings. Road signs and surfaces are typically designed to be easily visible to the driver and other occupants of the vehicle in low-light conditions by using reflectors, reflective paints, and / or reflective coatings. However, this can sometimes result in road signs, indicators, and / or markings producing high levels of reflected light when illuminated by the vehicle's headlights. In some cases, this reflected light can cause glare in the field of vision of vehicle occupants or in the field of vision of one or more cameras and / or sensors in the vehicle. This problem can be exacerbated when the headlights emit particularly strong light, such as when illuminating distant objects and / or when light reflects from close range. Strong glare can cause visual discomfort and reduce the ability of vehicle sensors to capture and analyze sensor data. Attempts to mitigate glare include disabling the vehicle's high beams based on glare detection, or, in the case of matrix high beams, eliminating the illuminated area by disabling the portion of the matrix high beams that is directed to the location to be eliminated.

[0002] Traditionally, glare detection in vehicles relies on measuring the luminous energy reflected from a surface using absolute luminance metrology. Because absolute luminance metrology directly measures the reflected light energy from a specific object, the size of the object reflecting the light is required. Determining the object's size for the purposes of absolute luminance measurement requires sensing and computational resources to estimate the object's size (e.g., surface area), which is beyond the scope of conventional glare detection solutions. For example, the size of an object depicted in a camera image can be determined by converting the object's position to a three-dimensional (3D) position (e.g., estimating surface area). This requires additional computational operations due to various possible camera orientations, optical distortions, and / or real-time distance measurements. Furthermore, accurately performing absolute luminance measurements using this method requires additional calibration operations and associated hardware to store calibration parameters, potentially requiring routine camera calibration against known luminance targets to maintain accurate luminance metrology over time as sensor component wear and / or degradation occur. Summary of the Invention

[0003] Embodiments of this disclosure relate to glare mitigation using image contrast analysis for autonomous systems and applications. Systems and methods are disclosed that calculate contrast based on sensor data from one or more sensors from a vehicle or other machine or system to detect areas with significant levels of glare, thereby enabling mitigation operations.

[0004] Compared to conventional methods such as those described above, this disclosure provides the use of relative luminance metrology to detect glare. Using the disclosed method, contrast values ​​can be calculated for pixels of an image, and these contrast values ​​can be analyzed to detect glare (e.g., light reflected from signs and other objects). In one or more embodiments, the system can receive image data depicting one or more objects illuminated by the headlights of a vehicle. Pixels represented in the image data can be used to calculate contrast values ​​relative to other pixels in the image data. In at least one embodiment, one or more local contrast values ​​can be calculated for a pixel (e.g., by calculating the contrast relative to the brightness of all or a portion of the image).

[0005] In one or more embodiments, an average luminance value for a set of pixels in an image can be calculated to determine a reference luminance. A local contrast value can be determined by calculating the ratio of a pixel's luminance to the calculated reference luminance. A luminance threshold can be applied to the local contrast values ​​associated with a set of pixels to determine the number of pixels with contrast values ​​that satisfy the luminance threshold. A size threshold can be applied to determine whether glare mitigation operations should be triggered based on the size and / or number of pixels that satisfy the luminance threshold. Attached Figure Description

[0006] The present system and method for glare reduction using image contrast analysis for autonomous systems and applications are described in detail below with reference to the accompanying drawings, wherein:

[0007] Figure 1 This is an example system diagram of a contrast analysis system for detecting glare using sensor data, according to some embodiments of the present disclosure;

[0008] Figure 2A Example images from a vehicle's perspective are shown, according to some embodiments of this disclosure;

[0009] Figure 2B Example images of glare reduction from the vehicle's perspective are shown, according to some embodiments of the present disclosure.

[0010] Figure 3 This is a visualization of example images used to determine the local contrast values ​​of pixels associated with a detected object, according to some embodiments of this disclosure;

[0011] Figures 4A-4C This is an example of applying a contrast analyzer to pixels corresponding to detected objects, according to some embodiments of this disclosure;

[0012] Figure 5 This is a flowchart illustrating an example method for image contrast analysis for glare reduction according to some embodiments of the present disclosure;

[0013] Figure 6 This is a flowchart illustrating an example method for image contrast analysis for object detection-based glare reduction according to some embodiments of the present disclosure;

[0014] Figure 7A These are illustrations of example autonomous vehicles according to some embodiments of the present disclosure;

[0015] Figure 7B According to some embodiments of this disclosure Figure 7A An example of the camera position and field of view of an exemplary autonomous vehicle;

[0016] Figure 7C According to some embodiments of this disclosure Figure 7A A block diagram of an example system architecture for an example autonomous vehicle;

[0017] Figure 7D This is a cloud-based server according to some embodiments of the present disclosure. Figure 7A A system diagram illustrating communication between autonomous vehicles;

[0018] Figure 8 This is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0019] Figure 9 This is a block diagram of an example data center applicable to implementing some embodiments of this disclosure. Detailed Implementation

[0020] Systems and methods relating to the use of image contrast analysis to mitigate glare for autonomous systems and applications are disclosed. Although this disclosure is directed to an example autonomous vehicle 700 (also referred to herein as “vehicle 700”), examples are referenced thereto. Figures 7A-7DThe description herein is intended to be limiting. For example, the systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, aircraft, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles, drones, and / or other vehicle types. Furthermore, while this disclosure may be described with respect to glare mitigation for vehicles, this is not intended to be limiting, and the systems and methods described herein can be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technological space that may require the detection or estimation of glare, for example, for mitigation or reduction of glare. The disclosed methods can be implemented in one or more of the following: control systems of autonomous or semi-autonomous machines; perception systems of autonomous or semi-autonomous machines; systems for performing simulated operations; systems for performing deep learning operations; systems implemented using edge devices; systems implemented using robots; systems combining one or more virtual machines (VMs); systems implemented at least partially in a data center; or systems implemented at least partially using cloud computing resources.

[0021] Compared to conventional methods such as those described above, this disclosure provides the use of relative luminance metrology to detect glare. Using the disclosed method, contrast values ​​can be calculated for pixels of an image, and these contrast values ​​can be further analyzed to detect glare (e.g., light reflected to the driver and / or vehicle sensors). In one or more embodiments, the system can receive image data depicting one or more objects illuminated by the vehicle's headlights. Pixels represented in the image data can be used to calculate contrast values ​​relative to other pixels in the image data. In at least one embodiment, one or more global contrast values ​​can be calculated for a pixel (e.g., by calculating the contrast relative to the brightness of all or substantially all of the image). In at least one embodiment, one or more regional contrast values ​​can be calculated for a pixel (e.g., by calculating the contrast relative to the brightness of one or more regions of an image comprising one or more pixels). In at least one embodiment, one or more local contrast values ​​can be calculated for a pixel (e.g., by calculating the contrast relative to the brightness of one or more regions of an image based on proximity or distance to the pixel). By using relative luminance, glare can be detected without performing sensor calibration or having to determine the physical dimensions of objects in 3D space, because only relative contrast is calculated.

[0022] In some examples, the system can receive image data generated using one or more cameras, which can be positioned at various locations relative to the vehicle (e.g., front center, dashboard, etc.). When using multiple cameras, each camera can share similar or different fields of view (e.g., wide-angle, telephoto, etc.). In various examples, the image data can represent one or more images captured by the cameras at the same resolution, and / or can be downsampled or upsampled to different resolutions. In at least one embodiment, the image data can include RAW, RAW-like, and / or other sensor data, where the image pixel signals are linearly represented (e.g., preserving the raw pixel levels read from the imaging sensor) and proportional to the intensity of light. Image data can be generated using one or more camera lenses that do not introduce significant vignetting or "lens shadow" (corner darkening) phenomena, or pixels exhibiting vignetting can be corrected (e.g., using lens shadow correction on RAW in an image signal processor (ISP).

[0023] In at least one embodiment, a pixel's contrast value can be calculated from image data, at least based on comparing the pixel's brightness with the brightness of one or more other pixels. For example, one or more statistical values ​​can be calculated from the brightness values ​​of any number of pixels in the image to establish a reference or relative brightness for calculating one or more contrast values. By way of example and not limitation, this can include calculating a mean (average) image brightness level. Suitable techniques for calculating the reference brightness include calculating a histogram or mean of pixel contrast values, performing a global pixel averaging, and / or calculating a trimmed mean (e.g., producing a more stable mean in high-contrast scenes by removing lower and higher percentiles in consideration).

[0024] The reference brightness pixels used to calculate global contrast values ​​can correspond to all or substantially all images. In at least one embodiment, one or more pixels can be discarded or excluded from the calculated reference brightness, such as outliers, pixels corresponding to image or visual artifacts, pixels outside the region of interest (e.g., those that will not be applied to CNNs or other MLMs), etc. The reference brightness pixels used to calculate regional contrast values ​​can correspond to one or more regions of the image, including one or more pixels (and / or at least a portion of the object corresponding to the pixel). For example, a region can include 50% or different percentages of pixels in the image, pixels in the central region of the image, pixels in the quadrant of the image, etc. In at least one embodiment, the reference brightness of a region can be used to calculate the contrast value of a pixel based at least on pixels included in the region (where the region is defined regardless of the region's location). Pixels used to calculate local contrast values ​​can be selected or determined based at least on proximity and / or distance to one or more pixels (and / or at least a portion of the object corresponding to the pixel).

[0025] In at least one embodiment, the system can generate an image in which each pixel records a corresponding contrast value. The contrast value of a pixel can be calculated at least based on the ratio between the brightness of the pixel value and a reference brightness (e.g., the mean). Depending on the size (e.g., radius) of the region chosen to calculate the reference brightness, the calculated contrast value may represent different phenomena. A small radius (e.g., <0.1% of the image size) can represent image sharpness, a small to medium radius (e.g., <3-4% of the image size) can represent the contrast of object details, a medium radius can represent the contrast of objects, and a large radius that encompasses or substantially encompasses the entire image can represent global contrast.

[0026] In at least one embodiment, an object detector, such as a convolutional neural network (CNN) and / or other machine learning model (MLM), can be used to detect one or more objects (e.g., road signs, reflectors, road markings, etc.) in image data. For example, the MLM can predict the location and / or region (e.g., one or more boundary shapes) of one or more objects in one or more images represented by the image data. Detected objects can be associated with one or more pixels in the image data using one or more object locations and / or regions (e.g., an object can be associated with one or more pixels within a boundary shape). The system can calculate the contrast value of one or more pixels in the input image based at least on one or more of the detected locations and / or boundary shapes. For example, the location or boundary shape can be used to calculate the local, global, and / or regional contrast values ​​of one or more corresponding pixels.

[0027] Alternatively or concurrently, other methods may be used to detect or select one or more objects and / or regions in an image, and these methods do not necessarily include machine learning. For example, pixels in an image may be grouped using any suitable clustering algorithm, which may cluster pixels into one or more groups based at least on image features such as brightness, color values, contrast values ​​(e.g., by first determining global contrast values ​​for grouping and then using the groups to determine local contrast values), relative luminance, relative chroma, relative hue, etc. In one or more embodiments, contrast values ​​may be calculated for the groups (e.g., applied to all pixels in the group) and / or for one or more pixels in the group (e.g., each pixel).

[0028] After calculating the contrast value, the system can analyze the contrast value to detect glare and / or other lighting phenomena. In at least one embodiment, analyzing the contrast value may include comparing the contrast value to one or more thresholds. The thresholds may be predetermined and / or dynamically determined or calculated. By way of example and not limitation, the threshold for the contrast value may be calculated or selected based at least on one or more reference luminance values ​​used to calculate the contrast value. In at least one embodiment, the threshold may be a multiple of the reference luminance, such as, but not limited to, 5 times, 10 times, or 100 times the reference luminance. In at least one embodiment, the threshold may be applied to the contrast image to generate a thresholded contrast image. In at least one embodiment, the thresholded contrast image may be generated directly (without an initial contrast image) based at least on applying one or more thresholds to the contrast value when calculating the contrast value. For example, any pixel with a contrast value higher than the threshold (e.g., a high-contrast pixel) may be retained or set to a specific value (e.g., 1), while pixels with a contrast value lower than the threshold (e.g., a low-contrast pixel) may be deleted or set to a specific value (e.g., 0).

[0029] In one or more embodiments, based on the number of pixels corresponding to a group of pixels with contrast values ​​that meet a threshold (e.g., associated with a detected object), the system can transmit data to cause the vehicle to change the intensity (e.g., increase luminance, decrease luminance, deactivate, etc.) or configuration (e.g., position, orientation, lens focal length, etc.) of one or more headlights and / or other lights of the vehicle. For example, the intensity can be modified or changed based at least on one or more locations of pixel groups in an image and / or their corresponding locations in the real world.

[0030] In at least one embodiment, the number of pixels exceeding a threshold can be counted or compared to the number of pixels in a specific region. For example, the number of pixels in a bounding box or shape corresponding to an object with a contrast value exceeding the threshold can be compared to the number of pixels in the bounding box or shape to determine the ratio of high-contrast pixels to low-contrast pixels.

[0031] In at least one embodiment, the system may apply a size threshold to the number of pixels having a corresponding contrast value that satisfies the threshold. For example, the size threshold may indicate the minimum number of pixels required to trigger glare reduction or other lighting operations. In some examples, the size threshold may be applied based on the number of pixels, the size of the region occupied by the pixels (e.g., the bounding box or sub-region occupied by pixels that satisfy the threshold), one or more dimensions of one or more images, and / or any suitable method for determining the size associated with a pixel. For example, the ratio of pixels within a bounding box having a contrast value higher than the threshold may be compared to one or more dimensions of the bounding box.

[0032] As described herein, the system can transmit data that causes a vehicle or machine to perform operations attempting to mitigate glare. For example, based on pixel contrast values, data can be transmitted that causes the vehicle to modify or change one or more lighting sources (e.g., headlights, fog lights, etc.) to mitigate detected glare. In some examples, glare mitigation may involve modifying the configuration of one or more elements of the headlights. For example, based on detected glare, matrix beam headlights—having multiple independently configurable lighting elements that illuminate a portion of the vehicle's environment—can deactivate and / or reduce the intensity of one or more lighting elements associated with the physical location of high-contrast pixels corresponding to image data.

[0033] refer to Figure 1 , Figure 1 This is an example system diagram of a contrast analysis system 100 (also referred to as "System 100") for detecting glare using sensor data, according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groups, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can use... Figures 7A-7D Exemplary autonomous vehicles 700, Figure 8 Example computing devices 800 and / or Figure 9 Example data center 900 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the operation.

[0034] The contrast analysis system 100 may include sensor data 102, a lighting analyzer 104, an object detector 120, a glare manager 130, a position calculator 124, one or more control components 140, and one or more headlights 150 (or more generally, lighting elements), etc. The contrast analysis system 100 can perform operations for generating and / or receiving sensor data 102 from one or more sensors. As a non-limiting example, the sensor data 102 can be obtained from a vehicle (e.g., Figures 7A-7D The sensor data 102 may be received by one or more sensors of the vehicle 700 described herein. The sensor data 102 may include, but is not limited to, one or more portions of sensor data from any combination of sensors of the vehicle 700, including, for example, and referenced from… Figures 7A-7DThe sensor includes a Global Navigation Satellite System (GNSS) sensor 758 (e.g., a Global Positioning System sensor), a Radar (RADAR) sensor 760, an Ultrasonic sensor 762, a LiDAR (LiDAR) sensor 764, an Inertial Measurement Unit (IMU) sensor 766 (e.g., an accelerometer, gyroscope, magnetic compass, magnetometer, etc.), a microphone 796, a stereo camera 768, a wide-angle camera 770 (e.g., a fisheye camera), an infrared camera 772, a surround camera 774 (e.g., a 360-degree camera), a long-range and / or medium-range camera 798, a speed sensor 744 (e.g., for measuring the speed of vehicle 700), and / or other sensor types.

[0035] Alternatively, sensor data 102 may include virtual (e.g., simulated or augmented) sensor data generated from any number of sensors in a virtual vehicle or other virtual object in a virtual (e.g., test) environment. In such an example, the virtual sensors may correspond to virtual vehicles or other virtual objects in a simulated environment (e.g., for testing, training, and / or validating neural network performance), and the virtual sensor data may represent sensor data captured by virtual sensors in a simulated or virtual environment. Thus, by using virtual sensor data, the machine learning model 122 described herein can be tested, trained, and / or validated using simulated data and / or augmented real-world data from a simulated environment. This allows for testing more extreme scenarios outside of real-world environments, where such testing might be less safe.

[0036] In some embodiments, sensor data 102 may include image data representing an image, image data representing video (e.g., a snapshot of video), and / or sensor data representing a representation of the sensor's sensory field (e.g., a depth map of a LiDAR sensor, a value map of an ultrasonic sensor, etc.). Where sensor data 102 includes image data, any type of image data format may be used, such as, but not limited to, compressed images, such as compressed images in Joint Image Experts Group (JPEG) or Luminosity / Chromatography (YUV) formats, compressed images derived from frames of compressed video formats such as H.264 / Advanced Video Coding (AVC) or H.265 / High-Efficiency Video Coding (HEVC), raw images (e.g., derived from Red-to-Blue Transparent (RCCB), Red-to-Blue Transparent (RCCC), Red-to-Green-to-Blue (RGGB), or other types of imaging sensors), and / or other formats. Furthermore, in some examples, sensor data 102 can be used within system 100 without any preprocessing (e.g., in raw or captured format), while in other examples, sensor data 102 may undergo preprocessing (e.g., noise balancing, depigmentation, scaling, cropping, enhancement, white balance, tone curve adjustment, etc., for example, using a sensor data preprocessor (not shown)). As used herein, sensor data 102 may reference unprocessed sensor data, preprocessed sensor data, or a combination thereof.

[0037] Referring to illumination analyzer 104, illumination analyzer 104 may, among other things, include one or more components for measuring and / or calculating contrast represented by image data. For example, illumination analyzer 104 may be used to provide contrast values ​​for image data such as sensor data 102. In some embodiments, illumination analyzer 104 may include luminance analyzer 106, contrast analyzer 108, and / or contrast thresholder 110. Although in Figure 1 Only a few components and / or features of the lighting analyzer 104 are shown in this document, and this is not limiting. For example, the lighting analyzer 104 may include additional or alternative components, such as those described below. Figures 7A-7D The exemplary autonomous vehicles 700 described above.

[0038] Brightness analyzer 106 may include one or more components for performing brightness value measurement and / or extraction from one or more pixels of image data. For example, brightness values ​​may be determined for one or more pixels included in image data representing sensor data 102. In at least one embodiment, brightness analyzer 106 may determine the brightness value for each pixel of an image. For example, an image represented by sensor data 102 may have brightness values ​​associated with each pixel in that particular image. In at least one embodiment, fewer than the number of pixels per pixel in the image may be selected, and the corresponding brightness values ​​may be determined by brightness analyzer 106. For example, brightness values ​​102 may be determined based on a portion of sensor data indicated by object detector 120 and / or MLM 122. For example, object detector 120 may provide information associated with detected objects (e.g., object location, object orientation, classification, label, features, and / or other relevant information about the detected objects) to illumination analyzer 104, which may use the provided information to determine the selection of pixels for which brightness and / or contrast values ​​are to be determined (e.g., pixels representing objects, pixels near objects, etc.).

[0039] Contrast analyzer 108 can use pixel luminance values ​​determined by luminance analyzer 106 to calculate contrast values ​​corresponding to image pixels (e.g., global, regional, and / or local). For example, contrast analyzer 108 can evaluate the luminance value of a particular pixel based on the average luminance value of a set of pixels (or other statistical values ​​described herein), such as the average luminance value of each pixel in an image. In some embodiments, contrast analyzer 108 can calculate an average luminance value of a set of pixels as a reference luminance. Although an average pixel luminance value is described herein, it is not intended to be limiting, and any suitable techniques for calculating the reference luminance are considered herein, including calculating a histogram average, performing a global pixel average, and / or calculating a trimmed average. For example, contrast analyzer 108 can determine the reference luminance by calculating a histogram or average of the luminance values ​​of a set of pixels corresponding to a detected object and a region within a specific proximity and / or distance radius to the detected object. The reference luminance can be used to calculate the contrast value of one or more pixels in an image. Contrast analyzer 108 can calculate the contrast value of a pixel, which is expressed as a ratio or other relationship between the luminance value of a particular pixel and the reference luminance.

[0040] Contrast thresholder 110 may include one or more components for applying a contrast threshold to determine a set of pixels having corresponding contrast values ​​that satisfy the contrast threshold. In at least one embodiment, the contrast threshold is predetermined. For example, the contrast threshold may be determined as a value that glare might become uncomfortable for an observer (e.g., 5 times, 10 times, 100 times, etc. relative to a reference luminance) and / or any value that glare might impair the sensor's ability to capture sensor data. The contrast threshold may be predetermined or dynamically determined based on pixel luminance values ​​determined by luminance analyzer 106. Contrast thresholder 110 may receive contrast values ​​associated with a set of pixels from contrast analyzer 108. For example, contrast thresholder 110 may count or determine the number of pixels having relative contrast values ​​higher than a threshold.

[0041] Referring now to object detector 120, among other things, object detector 120 may include one or more components for detecting one or more objects (e.g., road signs, reflectors, road markings, etc.) in image data. In some embodiments, object detector 120 may use convolutional neural networks (CNNs) and / or other machine learning models (MLMs) to detect one or more objects (e.g., road signs, reflectors, road markings, etc.) in image data. For example, object detector 120 may use one or more MLMs 122 to detect objects in sensor data 102 (e.g., in an image). In at least one embodiment, object detector 120 may use methods that do not necessarily rely on MLMs 122 to detect one or more objects. For example, object detector 120 may detect objects by clustering and analyzing features such as brightness, color values, contrast values, relative luminance, relative chroma, relative hue, etc., corresponding to pixels of the image represented by sensor data 102.

[0042] Where MLM 122 can be used to detect one or more objects in an image, sensor data 102 and / or data derived therefrom can be applied to MLM 122 to predict the location and / or region of one or more objects in one or more images represented by sensor data 102 as containing objects. Detected objects can be associated with one or more pixels in the image data using one or more object locations and / or regions (e.g., objects can be associated with one or more pixels within a boundary shape or box). Contrast analyzer 108 can calculate the contrast value of one or more pixels in the input image based at least on one or more of the detected locations and / or boundary shapes. For example, locations or boundary shapes can be used to calculate local, global, and / or regional contrast values ​​for one or more corresponding pixels.

[0043] Referring now to location calculator 124, the location calculator may include one or more components configured to determine one or more 2D and / or 3D locations of detected objects. For example, location calculator 124 may receive indications of one or more objects detected by object detector 120 and may determine the location of one or more detected objects and / or the distance to one or more detected objects. In one or more embodiments, this may include post-processing of prediction data from MLM 122. In some embodiments, location calculator 124 may determine the location of one or more detected objects relative to an autonomous machine (such as...). Figures 7A-7D The location of the autonomous vehicle 700 is calculated. For example, the location calculator 124 can calculate the distance from the vehicle to a detected object (e.g., a sign) and generate information associated with the calculated position of the sign relative to the vehicle (e.g., distance, angle relative to the vehicle's direction of travel, height of the object, size of the object, etc.). In some embodiments, the location calculator 124 can provide location information associated with one or more detected objects to the deglare manager 130.

[0044] The deglare manager 130 may include one or more components for determining the glare mitigation operation to be performed, at least based on relative contrast values ​​corresponding to pixels in the image corresponding to sensor data 102. For example, based on contrast values ​​calculated by illumination analyzer 104, the deglare manager 130 may identify one or more objects represented in sensor data 102 as candidates for glare mitigation operations, thereby triggering the glare mitigation operation. In some embodiments, the deglare manager 130 may determine objects represented in sensor data 102 as candidates for glare mitigation operations based on applying a size threshold to pixels associated with the object. For example, the deglare manager 130 may determine that glare mitigation operations should be performed in association with the detected object based on the object's size and / or the number of pixels exceeding a contrast threshold (e.g., as indicated by contrast thresholder 110). For example, in one or more embodiments, glare mitigation operations may be performed only if a minimum number of high-contrast pixels associated with the object is met. In some examples, if the size threshold is not met, glare mitigation operations will not be triggered even if the contrast values ​​of all pixels associated with the detected object are higher than the contrast threshold.

[0045] In at least one embodiment, the deglare manager 130 may receive location information associated with one or more detected objects. In at least one embodiment, the location information from the location calculator 124 may be used to determine whether the detected objects meet a size threshold. For example, the size of the object may be estimated using the distance to the object determined by the location calculator 124 and based on sensor data 102. In at least one embodiment, the deglare manager 130 may receive information from the location calculator 124 corresponding to the calculated 3D position of the detected objects. The deglare manager 130 may use the location information from the location calculator 124 to determine the glare mitigation operation to be performed. For example, if the location information from the location calculator 124 indicates, based on the location of the detected objects, vehicle occupants, vehicle sensors, and / or one or more headlights 150, the possibility that light emitted by the headlights 150 may be reflected from a reflecting object in the direction of the vehicle occupants or sensors, a glare mitigation operation (e.g., reducing the illumination power of one or more elements of the headlights) may be triggered. As an example, the deglare manager 130 can estimate one or more distances and / or angles between the detected sign and the headlight 150 and / or the location of a specific part of the vehicle, such as the location of a sensor or potential driver / occupant.

[0046] In some embodiments, based on the glare mitigation operation that should be performed in association with a detected object, the glare manager 130 may provide information associated with the detected object to one or more control components 140, which may cause modifications (e.g., control changes) to one or more headlights 150. For example, the glare manager 130 may provide an indication to one or more control components 140 of the location of a glare-inducing object to cause an adjustment to at least one illumination element of the headlight 150.

[0047] Headlights 150 may include one or more connected to autonomous machines (e.g., Figures 7A-7DThe headlights 150 are illumination sources associated with the autonomous vehicle 700. While forward-facing headlights are described herein, this is not intended to be limiting, and any other light source or orientation of light source is contemplated herein. For example, headlights 150 may include low beam headlights, high beam headlights, fog lights, daytime running lights, hazard lights, signal lights, and / or any other illumination source (e.g., an illumination source capable of reflecting light back to passengers and / or sensors of vehicle 700 or otherwise causing or contributing to glare). In some embodiments, headlights 150 may emit light having wavelengths outside the visible spectrum (e.g., visible to the human eye). For example, headlights 150 may emit invisible radiation or invisible light (e.g., to the human eye), such as infrared (IR) light. Headlights 150 may include one or more independently configurable illumination elements. For example, headlights 150 may include multiple illumination elements arranged in a matrix arrangement (e.g., a grid). In such an example, each illumination element may be configured individually or in association with one or more other elements. For example, a single lighting element can be configured by activating / deactivating the emitted light, increasing or decreasing the illumination power of the element, adjusting the direction or focus of the projected light, or any combination thereof. By modifying the configuration of one or more lighting elements of the headlight 150, glare reflected from objects can be mitigated. For example, if glare is detected from a sign, the lighting element illuminating the sign can be deactivated or projected away from the sign (e.g., while maintaining illumination around the sign or other objects).

[0048] As a non-restrictive example, and regarding Figure 2A-2B , Figure 2A-2B Examples of glare reduction used to illustrate at least some embodiments of this disclosure. For example, Figure 2A An example image 200A is illustrated from the perspective of a vehicle such as vehicle 700 (e.g., the view from its camera), and... Figure 2B Example image 200B is shown, viewed from the vehicle 700 after glare reduction, according to some embodiments of the present disclosure.

[0049] As an example, vehicle 700 may be on road 204, which includes objects such as pavement markings, vehicles, pedestrians, obstacles, visual indicators, and / or signs such as road sign 206. Road 204 may be illuminated by one or more light sources (e.g., corresponding to...). Figure 1 One or more headlights 150 (or headlights 240) provide complete or partial illumination. The illumination sources can be divided and / or grouped into lighting elements that can be controlled individually and / or collectively. For example, headlights 240 may include any one of a plurality of lighting elements, such as lighting elements 242A-242H. Although headlights 240 are depicted as having eight elements, this is not intended to limit and any number and / or positioning arrangement of lighting elements is contemplated herein.

[0050] A lighting source can project or shadow onto a road 204 in association with an illuminated coverage area 210, such that the road 204 can be at least partially illuminated. A portion of the illuminated coverage area 210 may correspond to at least one light-emitting element of the lighting source. For example, each beam segment of beam segment 212 may correspond to a light element such as light element 242A-242H. In some examples, reducing power and / or disabling specific light elements may result in a reduced illumination effect on the road 204 for the associated beam segment. For example, as... Figure 2B As shown and reflected by image 200B, light elements 242A and 242B have been deactivated, causing a portion of beam segment 212 to stop illuminating road sign 206. In this example, since light elements 242A and 242B associated with that portion of beam segment 212 corresponding to road sign 206 have been deactivated, glare caused by headlight 240 projected onto road sign 206 can be reduced or eliminated for the sensors and / or occupants of vehicle 700.

[0051] Now for reference Figure 3 , Figure 3 The illustration shows a visualization of an example image 300 for determining local contrast values ​​of pixels associated with a detected object, according to some embodiments of the present disclosure. Image 300 includes [data / methods / information] that can be used to [determine local contrast values ​​of pixels associated with a detected object]. Figure 1 The object detector 120 detects a detected object 310 (e.g., a sign). As described herein, the object detector 120 may indicate one or more of the size and / or location of a boundary shape (e.g., a bounding box 330) associated with the detected object 310. The bounding box 330 may define a region of image 300 that includes one or more pixels associated with image 300. Image 300 may also include a local contrast region 320 that may be located or defined in association with the bounding box 330. The local contrast region 320 may include one or more pixels of image 300 that can be used to calculate a reference brightness for determining a local contrast value (as described herein) for one or more pixels (e.g., each pixel) within the bounding box 330.

[0052] One or more dimensions of the bounding box 330 can be compared with one or more size thresholds to determine whether glare mitigation or other lighting actions are triggered. For example, Figure 1The deglare manager 130 can apply size thresholds to the dimensions of the bounding box 330. Size thresholds (e.g., indicating a minimum size or dimension) can be applied to the horizontal and / or vertical components of the bounding box 330. For example, a horizontal size threshold 350A can be applied to a horizontal object size 340A and / or a vertical size threshold 350B can be applied to a vertical object size 340B. In some embodiments, glare mitigation operations can be triggered if the horizontal object size 340A exceeds the horizontal size threshold 350A and if the vertical object size 340B exceeds the vertical size threshold 350B. In some embodiments, glare mitigation operations can be triggered if the horizontal object size 340A exceeds the horizontal size threshold 350A or the vertical object size 340B exceeds the vertical size threshold 350B. In some embodiments, the vertical object size 340B and the horizontal object size 340A can be used to determine areas that can be compared to a threshold area (e.g., the minimum area required to trigger glare mitigation operations). For example, the vertical object size 340B and the horizontal object size 340A can be multiplied to determine the object area that can be compared with the area calculated by multiplying the vertical size threshold 350B and the horizontal size threshold 350A.

[0053] Now for reference Figures 4A-4C , Figures 4A-4C The illustration shows the application of pixels corresponding to the detected object 310 according to at least some embodiments of the present disclosure. Figure 1 Example of a lighting analyzer 104. Figure 4A The diagram shows... Figure 2A Image 200A can correspond to Figure 1 The sensor data 102 includes a region 410. For example, region 410 may include a portion of pixels represented by sensor data 102. In some embodiments, region 410 includes one or more pixels associated with an object 310 detected in sensor data 102. For example, object detector 120 may provide the size, location, and / or other indications of the pixels associated with the detected object. For example, object detector 120 may generate a boundary shape corresponding to the detected object (e.g., a road sign, vehicle, pedestrian, etc.) and determine a set of pixels within the boundary shape. This set of pixels may be provided to lighting analyzer 104. Using luminance analyzer 106, luminance values ​​may be determined (e.g., individually for each pixel). For example, Figure 4A Region 410 in the diagram can correspond to the bounding box 330 of a detected marker, which comprises a set of pixels, each pixel having a corresponding luminance value, as indicated by luminance value 420A (e.g., 10 K candela (cd / m²)). 2Using the luminance value 420A of region 410, the contrast analyzer 108 can calculate each contrast value by comparing the corresponding luminance value with a reference luminance.

[0054] Figure 4B An example of a contrast value 420B that can be calculated using a luminance value 420A is illustrated. In at least one embodiment, the contrast value of a particular pixel's contrast value 420B can be determined by calculating the ratio between the luminance value of that particular pixel and a reference luminance (e.g., mean) of a determined set of pixels, as described herein. For example, determining the reference luminance can be based on selecting a region of image 200A. A region of image 200A can be selected to capture details of the relative contrast associated with one or more objects within image 200A. For example, radii of various sizes (e.g., small radius, small to medium radius, medium radius, large radius, etc.) can be used to determine the set of pixels used to determine the reference luminance by averaging and / or calculating a central tendency. For example, Figure 3 The local contrast region 320 can be used to determine reference brightness based on the pixels within the corresponding region. For example... Figure 4B As shown, a contrast value can be determined for each pixel in region 410. For example, in... Figure 4B In the example shown, in this non-limiting example, each pixel in the pixel set is associated with a corresponding luminance value of 420A and a reference luminance of 75 cd / m². 2 The contrast value is calculated based on the ratio.

[0055] Figure 4C An example of a threshold of 420V is shown, which can be obtained by applying a thresholding operation to a contrast value of 420C. The thresholding operation can be performed by... Figure 1 The contrast thresholder 110 is executed. Figure 4C In the non-limiting example shown, threshold 100 is applied to each contrast value of contrast value 420C. In one or more embodiments, only pixels in region 410 with contrast values ​​that satisfy threshold 100 will be retained or otherwise identified as satisfying the contrast threshold. As an example, a value of 1 can be assigned to pixels with contrast values ​​420C that satisfy the threshold, while a value of 0 can be assigned to pixels with contrast values ​​420C that do not satisfy the threshold, or otherwise negated or indicated. The number of pixels in region 410 that satisfy the threshold can be determined and can be used for glare mitigation operations, for example by... Figure 1 The glare manager 130, control component 140, and / or headlight 150 are used for glare reduction operation.

[0056] Now for reference Figure 5Each block of the method 500 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by standalone applications, services, or managed services (standalone or in combination with another managed service) or plug-ins to another product, to name a few. Furthermore, by way of example, regarding... Figure 1 The contrast analysis system describes method 500. However, these methods may be performed additionally or alternatively by any system or any combination of systems, including but not limited to those described herein.

[0057] Figure 5 This is a flowchart illustrating a method 500 for image contrast analysis to reduce glare according to some embodiments of the present disclosure. In block B502, method 500 includes receiving image data representing one or more images depicting one or more objects. For example, sensor data 102 may be received by an illumination analyzer 104 and an object detector 120.

[0058] In box B504, method 500 includes using image data to calculate one or more contrast values ​​for one or more pixels corresponding to one or more objects. For example, contrast analyzer 108 can determine the contrast value of a pixel represented by sensor data 102, which corresponds to one or more objects that can be detected by object detector 120.

[0059] In block B506, method 500 includes transmitting data to enable the machine to perform one or more operations based at least on an analysis of a set of contrast values. For example, based on the contrast values ​​determined by the lighting analyzer 104, one or more control components 140 may perform operations such as adjusting, modifying, or controlling the headlights 150.

[0060] Now for reference Figure 6 Each block of the method 600 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by standalone applications, services, or managed services (standalone or in combination with another managed service) or plug-ins to another product, to name a few. Furthermore, as an example, relative to... Figure 1 The contrast analysis system describes method 600. However, these methods may be performed additionally or alternatively by any system or any combination of systems, including but not limited to those described herein.

[0061] Figure 6 This is a flowchart illustrating a method 600 for glare reduction based on object detection in image contrast analysis according to some embodiments of the present disclosure. In block B602, method 600 includes receiving image data representing one or more images. For example, illumination analyzer 104 may receive sensor data 102.

[0062] In box B604, method 600 includes detecting one or more objects. For example, using sensor data 102, object detector 120 can detect one or more objects represented in sensor data 102.

[0063] In box B606, method 600 includes determining a region of interest (ROI) in one or more images that corresponds to one or more objects. For example, object detector 120 may determine a ROI that indicates pixels corresponding to one or more objects detected in sensor data 102.

[0064] In block B608, method 600 includes calculating one or more contrast values ​​for one or more pixels in one or more images using image data. For example, contrast analyzer 108 of illumination analyzer 104 can determine the contrast value of a pixel represented by sensor data 102. In one or more embodiments, the calculation of contrast values ​​may be limited to or otherwise based at least on a region of interest or a region of a detected object of interest (e.g., a sign). For example, calculating local contrast values ​​may require a large amount of computation (because each pixel may depend on all surrounding pixels), and limiting the pixels analyzed can save computational resources.

[0065] In box B610, method 600 includes analyzing a set of contrast values ​​from one or more contrast values ​​corresponding to a region of interest. For example, contrast thresholder 110 may analyze one or more contrast values ​​corresponding to one or more pixels of a region of interest determined by object detector 120.

[0066] In block B612, method 600 includes transmitting data to enable the machine to perform one or more operations based at least on the analysis of a set of contrast values. For example, based on the contrast values ​​analyzed from lighting analyzer 104, control component 140 may perform operations such as configuring one or more elements of headlight 150.

[0067] Example autonomous vehicles

[0068] Figure 7AThis is an illustration of an example autonomous vehicle 700 according to some embodiments of the present disclosure. The autonomous vehicle 700 (or, alternatively, referred to herein as “vehicle 700”) may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police vehicles, ambulances, boats, construction vehicles, underwater vessels, drones, trailer-mounted vehicles, and / or other types of vehicles (e.g., driverless and / or vehicles accommodating one or more passengers). Autonomous vehicles are generally described according to the level of automation defined by a branch of the U.S. Department of Transportation—the National Highway Traffic Safety Administration (NHTSA)—and the Society of Automotive Engineers (SAE) in its “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of that standard). Vehicle 700 may be able to perform one or more functions that meet Level 3 to Level 5 of autonomous driving. Vehicle 700 may be able to function according to one or more Level 1 to Level 5 of autonomous driving. For example, depending on the embodiment, vehicle 700 may be able to provide driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). As used herein, the term “autonomy” may include any and / or all types of autonomy of vehicle 700 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, providing assisted autonomy, being semi-autonomous, primarily autonomous, or other specified.

[0069] Vehicle 700 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 700 may include a propulsion system 750, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 750 may be connected to the drivetrain of vehicle 700, which may include a transmission, to enable propulsion of vehicle 700. Propulsion system 750 may be controlled in response to receiving a signal from throttle / accelerator 752.

[0070] A steering system 754, which may include a steering wheel, can be used to steer the vehicle 700 (e.g., along a desired path or route) when the propulsion system 750 is operating (e.g., when the vehicle is in motion). The steering system 754 may receive signals from the steering actuator 756. For fully automatic (level 5) functionality, the steering wheel may be optional.

[0071] The brake sensor system 746 can be used to operate the vehicle brakes in response to receiving signals from the brake actuator 748 and / or the brake sensor.

[0072] It can include one or more System-on-a-Chip (SoC) 704 ( Figure 7C One or more controllers 736, including and / or one or more GPUs, can provide signals (e.g., signals representing commands) to one or more components and / or systems of vehicle 700. For example, one or more controllers can send signals to operate vehicle brakes via one or more brake actuators 748, to operate steering system 754 via one or more steering actuators 756, and to operate propulsion system 750 via one or more throttles / accelerators 752. One or more controllers 736 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving vehicle 700. One or more controllers 736 may include a first controller 736 for autonomous driving functions, a second controller 736 for functional safety functions, a third controller 736 for artificial intelligence functions (e.g., computer vision), a fourth controller 736 for infotainment functions, a fifth controller 736 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 736 can handle two or more of the functions described above, and two or more controllers 736 can handle a single function, and / or any combination thereof.

[0073] One or more controllers 736 may provide signals for controlling one or more components and / or systems of vehicle 700 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, a global navigation satellite system sensor 758 (e.g., a Global Positioning System sensor), a RADAR sensor 760, an ultrasonic sensor 762, a LIDAR sensor 764, an inertial measurement unit (IMU) sensor 766 (e.g., an accelerometer, gyroscope, magnetic compass, magnetometer, etc.), a microphone 796, a stereo camera 768, a wide-angle camera 770 (e.g., a fisheye camera), an infrared camera 772, a surround camera 774 (e.g., a 360-degree camera), a long-range and / or medium-range camera 798, a speed sensor 744 (e.g., for measuring the rate of vehicle 700), a vibration sensor 742, a steering sensor 740, a braking sensor (e.g., as part of a braking sensor system 746), and / or other sensor types.

[0074] One or more of the controllers 736 may receive inputs (e.g., represented by input data) from the instrument cluster 732 of the vehicle 700 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 734, an auditory signaling device, a speaker, and / or via other components of the vehicle 700. These outputs may include information such as vehicle speed, rate, time, map data (e.g., [missing information]). Figure 7C Information such as the HD map 722, location data (e.g., the location of vehicle 700 on the map), direction, and the location of other vehicles (e.g., occupying a grid), as well as information about objects and their states perceived by the controller 736, etc. For example, the HMI display 734 may display information about the existence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0075] The vehicle 700 also includes a network interface 724, which can communicate via one or more networks using one or more wireless antennas 726 and / or a modem. For example, the network interface 724 may be able to communicate via LTE, WCDMA, UMTS, GSM, CDMA2000, etc. One or more wireless antennas 726 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc., and / or one or more low-power wide area networks (LPWANs) such as LoRaWAN, SigFox, etc.

[0076] Figure 7BFor use in accordance with some embodiments of this disclosure Figure 7A This is an example of the camera position and field of view of an example autonomous vehicle 700. The camera and its respective field of view are an example embodiment and are not intended to be limiting. For example, additional and / or replaceable cameras may be included, and / or these cameras may be located at different positions on the vehicle 700.

[0077] The camera type used for the camera may include, but is not limited to, a digital camera suitable for use with components and / or systems of vehicle 700. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The camera may be able to use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a sharp-pixel camera, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to improve light sensitivity.

[0078] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0079] One or more of the cameras can be mounted in mounting components such as custom-designed (3-D printed) parts to cut off stray light and reflections from inside the vehicle (e.g., reflections from the dashboard in the windshield mirror) that may interfere with the camera's image data capture capabilities. Regarding the wing mirror mounting components, the wing mirror components can be custom-3-D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0080] A camera with a field of view that includes the environment in front of the vehicle 700 (e.g., a front-facing camera) can be used for surround view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 736 and / or control SoCs, to provide information crucial for generating an occupancy grid and / or determining a preferred vehicle path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including Lane Departure Warning (“LDW”), Autonomous Cruise Control (“ACC”), and / or other functions such as traffic sign recognition.

[0081] A variety of cameras can be used in front-facing configurations, including, for example, monocular camera platforms including CMOS (Complementary Metal-Oxide-Semiconductor) color imagers. Another example could be a wide-angle camera 770, which can be used to perceive objects entering the field of view from the periphery (such as pedestrians, traffic at intersections, or bicycles). Although Figure 7B The image shows only one wide-angle camera, but any number of wide-angle cameras 770 can be present on the vehicle 700. Furthermore, a remote camera 798 (e.g., a pair of long-view stereo cameras) can be used for depth-based object detection, especially for objects for which a neural network has not yet been trained. The remote camera 798 can also be used for object detection and classification, as well as basic object tracking.

[0082] One or more stereo cameras 768 may also be included in a front-mounted configuration. The stereo camera 768 may include an integrated control unit comprising a scalable processing unit that can provide a multi-core microprocessor and programmable logic (FPGA) with an integrated CAN or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 768 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip capable of measuring the distance from the vehicle to a target object and using the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 768 may be used in addition to those described herein, or alternatively.

[0083] Cameras with a field of view including the side portion of the vehicle 700 (e.g., side-view cameras) can be used for surround view, providing information for creating and updating occupancy grids and generating side-impact collision warnings. For example, surround camera 774 (e.g., ... Figure 7BThe four surround cameras 774 shown can be mounted on the vehicle 700. The surround cameras 774 can include a wide-angle camera 770, a fisheye camera, a 360-degree camera, and / or the like. Four examples are provided; the four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 774 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround-view camera.

[0084] A camera with a field of view that includes the environment behind the vehicle 700 (e.g., a rear-view camera) can be used for parking assistance, surround view, rear collision warning, and creating and updating occupancy grids. A wide variety of cameras can be used, including but not limited to those also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range camera 798, stereo camera 768, infrared camera 772, etc.).

[0085] Figure 7C For use in accordance with some embodiments of this disclosure Figure 7A The example autonomous vehicle 700 is illustrated in the block diagram of an example system architecture. It should be understood that this arrangement, and other arrangements described herein, are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities, which may be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by these entities can be implemented via hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in memory.

[0086] Figure 7C Each component, feature, and system in vehicle 700 is illustrated as being connected via bus 702. Bus 702 may include a Controller Area Network (CAN) data interface (or, alternatively, referred to herein as the "CAN bus"). CAN may be a network within vehicle 700 used to assist in the control of various features and functions of vehicle 700, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0087] Although bus 702 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to a CAN bus. Furthermore, although bus 702 is represented by a single line, this is not intended to be limiting. For example, any number of buses 702 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 702 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 702 may be used for collision avoidance functions, and a second bus 702 may be used for drive control. In any example, each bus 702 may communicate with any component of vehicle 700, and two or more buses 702 may communicate with the same component. In some examples, each SoC 704, each controller 736, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors of vehicle 700) and may be connected to a common bus such as a CAN bus.

[0088] Vehicle 700 may include one or more controllers 736, such as those described herein. Figure 7A The controllers described herein. Controller 736 can be used for a wide variety of functions. Controller 736 can be coupled to any other different components and systems of vehicle 700 and can be used for the control of vehicle 700, artificial intelligence of vehicle 700, infotainment and / or the like for vehicle 700.

[0089] Vehicle 700 may include one or more System-on-Chip (SoC) 704s. SoC 704 may include a CPU 706, GPU 708, processor 710, cache 712, accelerator 714, data storage 716, and / or other components and features not shown. SoC 704 can be used to control vehicle 700 across a wide variety of platforms and systems. For example, one or more SoCs 704s may be combined with an HD map 722 in a system (e.g., the system of vehicle 700), the HD map being transmitted via a network interface 724 from one or more servers (e.g., [server name missing]). Figure 7D One or more servers (778) receive map refresh and / or updates.

[0090] The CPU 706 may include CPU clusters or CPU complexes (or, alternatively, referred to herein as "CCPLEX"). The CPU 706 may include multiple cores and / or L2 cache. For example, in some embodiments, the CPU 706 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 706 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 2MB L2 cache). The CPU 706 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of clusters of the CPU 706 can be active at any given time.

[0091] The CPU 706 can implement power management capabilities including one or more of the following features: automatic clock gating of hardware blocks when idle to conserve dynamic power; clock gating of each core when the core is not actively executing instructions due to the execution of WFI / WFE instructions; independent power gating of each core; independent clock gating of each core cluster when all cores are clock-gated or power-gated; and / or independent power gating of each core cluster when all cores are power-gated. The CPU 706 can further implement enhanced algorithms for managing power states, where allowed power states and desired wake-up times are specified, and the hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core can support simplified power state entry sequences in software, with this work offloaded to the microcode.

[0092] The GPU 708 may include an integrated GPU (or, alternatively, referred to herein as an "iGPU"). The GPU 708 may be programmable and efficient for parallel workloads. In some examples, the GPU 708 may use an enhanced tensor instruction set. The GPU 708 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, the GPU 708 may include at least eight streaming microprocessors. The GPU 708 may use a computation application programming interface (API). Furthermore, the GPU 708 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0093] In automotive and embedded applications, the GPU 708 can be power-optimized for optimal performance. For example, the GPU 708 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 708 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can combine several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, dispatch units, and / or a 64KB register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to leverage the mixture of computation and addressing computations for efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can include a combination of L1 data cache and shared memory units to improve performance while simplifying programming.

[0094] The GPU 708 may include, in some examples, a High Bandwidth Memory (HBM) and / or a 16GB HBM2 memory subsystem providing a peak memory bandwidth of approximately 900GB / s. In some examples, in addition to HBM memory or alternatively, Synchronous Graphics Random Access Memory (SGRAM), such as Generation 5 Graphics Double Data Rate Synchronous Random Access Memory (GDDR5), may be used.

[0095] The GPU 708 may include unified memory technology, which includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support can be used to allow the GPU 708 to directly access the CPU 706 page tables. In such examples, when the GPU 708 Memory Management Unit (MMU) experiences a miss, the address translation request can be transferred to the CPU 706. In response, the CPU 706 can look up the virtual-physical mapping for the address in its page tables and transfer the translation back to the GPU 708. Thus, unified memory technology can allow a single unified virtual address space for the memory of both the CPU 706 and GPU 708, simplifying GPU 708 programming and porting applications to the GPU 708.

[0096] In addition, the GPU 708 may include access counters that track how frequently the GPU 708 accesses the memory of other processors. These access counters help ensure that memory pages are moved to the physical memory of the processor that accesses those pages most frequently.

[0097] SoC 704 may include any number of caches 712, including those described herein. For example, cache 712 may include an L3 cache available to both CPU 706 and GPU 708 (e.g., it is connected to both CPU 706 and GPU 708). Cache 712 may include a write-back cache, which can track the state of rows, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4MB or more, but a smaller cache size may also be used.

[0098] SoC 704 may include an arithmetic logic unit (ALU) that can be utilized in processing any of the various tasks or operations performed on the vehicle 700, such as processing a DNN. Furthermore, SoC 704 may include a floating-point unit (FPU) (or other mathematical coprocessor or digital coprocessor type) for performing mathematical operations within the system. For example, SoC 704 may include one or more FPUs integrated as execution units within CPU 706 and / or GPU 708.

[0099] SoC 704 may include one or more accelerators 714 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 704 may include a hardware accelerator cluster, which may include optimized hardware accelerators and / or large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to complement GPU 708 and offload some tasks from GPU 708 (e.g., freeing up more cycles of GPU 708 to perform other tasks). As an example, accelerator 714 can be used for targeted workloads (e.g., perceptrons, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. When used herein, the term "CNN" can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0100] Accelerator 714 (e.g., a hardware accelerator cluster) may include a Deep Learning Accelerator (DLA). A DLA may include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. TPUs may be accelerators configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for performing image processing functions. DLAs may be further optimized for a specific set of neural network types and floating-point operations as well as inference. DLAs are designed to provide higher performance per millimeter than general-purpose GPUs and significantly outperform CPUs. TPUs can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0101] DLA can execute neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for any function across a wide variety of applications, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition using data from microphones; CNNs for face recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for safety and / or safety-related events.

[0102] The DLA can perform any function of the GPU 708, and by using inference accelerators, for example, a designer can make either the DLA or the GPU 708 target any function. For example, a designer can focus the CNN processing and floating-point operations on the DLA and leave other functions to the GPU 708 and / or other accelerators 714.

[0103] Accelerator 714 (e.g., a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0104] RISC cores can interact with image sensors (such as the image sensor of any camera described herein), image signal processors, and / or the like. Each of these RISC cores may include any amount of memory. Depending on the embodiment, the RISC core may use any of several protocols. In some examples, the RISC core may execute a real-time operating system (RTOS). RISC cores may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0105] DMA enables PVA components to access system memory independently of the CPU 706. DMA can support any number of features to provide optimizations to the PVA, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0106] A vector processor can be a programmable processor designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as, for example, a Single Instruction Multiple Data (SIMD) or Very Long Instruction Word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0107] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. As a result, in some examples, each of the vector processors may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a cluster of hardware accelerators, and any number of vector processors may be included in each of these PVAs. Furthermore, the PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0108] Accelerator 714 (e.g., a hardware accelerator cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for accelerator 714. In some examples, on-chip memory may include at least 4MB of SRAM consisting of, for example, but not limited to, eight field-configurable memory blocks, accessible by both PVA and DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. PVA and DLA may access memory via a backbone that provides high-speed memory access to PVA and DLA. The backbone may include (e.g., using an APB) an on-chip computer vision network that interconnects PVA and DLA to memory.

[0109] On-chip computer vision networks can include interfaces that ensure both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such interfaces can provide separate phases and channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0110] In some examples, the SoC 704 may include, for example, a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. This real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison with LiDAR data for localization and / or other functional purposes, and / or other uses. In some embodiments, one or more Tree Traversal Units (TTUs) may be used to perform one or more ray tracing-related operations.

[0111] Accelerators 714 (e.g., hardware accelerator clusters) have broad applications in autonomous driving. PVAs can be programmable vision accelerators used in critical processing stages of ADAS and autonomous vehicles. The capabilities of PVAs are a good match for algorithmic domains requiring predictable processing, low power, and low latency. In other words, PVAs perform well in semi-dense or dense rule computation, even on small datasets requiring predictable runtimes with low latency and low power. Therefore, in the context of platforms for autonomous vehicles, PVAs are designed to run classical computer vision algorithms because they are efficient in object detection and integer mathematical operations.

[0112] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. In some examples, semi-global matching-based algorithms may be used, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., from moving structures, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0113] In some examples, PVA can be used to perform intensive optical flow, providing processed RADAR data from the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, PVA is used for time-of-flight depth processing, which, for example, involves processing raw time-of-flight data to provide processed time-of-flight data.

[0114] DLA can be used to run any type of network to enhance control and driving safety, including, for example, neural networks that output a confidence metric for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative “weight” for each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a threshold for the confidence and only consider detections exceeding the threshold as true positives. In an Automatic Emergency Braking (AEB) system, false positives can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network to regress the confidence value. This neural network can take at least a subset of parameters as input, such as bounding box dimensions, ground plane estimates (e.g., from another subsystem), inertial measurement unit (IMU) sensor 766 outputs related to vehicle orientation and distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 764 or RADAR sensor 760), etc.

[0115] SoC 704 may include one or more data storage units 716 (e.g., memory). The data storage unit 716 may be on-chip memory of SoC 704, which may store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and security, the data storage unit 716 may be large enough to store multiple instances of the neural network. Data storage unit 712 may include L2 or L3 cache 712. References to the data storage unit 716 may include references to memory associated with PVA, DLA, and / or other accelerators 714 as described herein.

[0116] SoC 704 may include one or more processors 710 (e.g., embedded processors). Processor 710 may include a startup and power management processor, which may be a dedicated processor and subsystem for handling startup power and management functions, as well as safety implementation. The startup and power management processor may be part of the SoC 704 startup sequence and may provide runtime power management services. The startup power and management processor may provide clock and voltage programming, auxiliary system low-power state transitions, SoC 704 thermal and temperature sensor management, and / or SoC 704 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 704 may use the ring oscillator to detect the temperature of CPU 706, GPU 708, and / or accelerator 714. If it is determined that the temperature exceeds a threshold, the startup and power management processor may enter a temperature fault routine and place SoC 704 into a lower power state and / or place vehicle 700 into a driver-safe parking mode (e.g., safely stop vehicle 700).

[0117] The processor 710 may also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio via multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0118] The processor 710 may also include an always-on-processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. This always-on-processor engine may include a processor core, tightly coupled RAM, support for peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0119] The processor 710 may also include a security cluster engine, which comprises a dedicated processor subsystem for handling security management for automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic that detects any differences between their operations.

[0120] The processor 710 may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0121] The processor 710 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0122] Processor 710 may include a video image compositer, which may be (e.g., implemented on a microprocessor) a processing block, implementing video post-processing functions required by the video playback application to generate the final image for the player window. The video image compositer may perform lens distortion correction on the wide-angle camera 770, the surround camera 774, and / or the in-cabin monitoring camera sensor. The in-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of an advanced SoC, configured to recognize in-cabin events and respond accordingly. The in-cabin system may perform lip reading to activate mobile phone services and make calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. Some functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other situations.

[0123] Video image compositers can include enhanced temporal denoising for both spatial and temporal noise reduction. For example, in the case of motion in the video, denoising appropriately weights spatial information, reducing the weight of information provided by neighboring frames. In cases where the image or part of the image does not contain motion, the temporal denoising performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0124] The video image compositer can also be configured to perform stereo correction on input stereo lens frames. When the operating system desktop is in use and the GPU 708 does not need to continuously render new surfaces, the video image compositer can be further used for user interface components. Even when the GPU 708 is powered on and activated, performing 3D rendering, the video image compositer can be used to offload the GPU 708 to improve performance and responsiveness.

[0125] The SoC 704 may also include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and pixel-related input functions for receiving video and input from a camera. The SoC 704 may also include an input / output controller that can be software-controlled and can be used to receive I / O signals not assigned to a specific role.

[0126] SoC 704 may also include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management and / or other devices. SoC 704 can be used to process data from cameras and sensors (e.g., LIDAR sensor 764, RADAR sensor 760, etc., which can be connected via Gigabit Multimedia Serial Link and Ethernet), data from bus 702 (e.g., vehicle 700 speed, steering wheel position, etc.), and data from GNSS sensor 758 (connected via Ethernet or CAN bus). SoC 704 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and can be used to free up CPU 706 from routine data management tasks.

[0127] The SoC 704 can be an end-to-end platform with a flexible architecture spanning Automation Levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS technologies for diversity and redundancy, along with deep learning tools to deliver a flexible and reliable driving software stack. The SoC 704 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, when combined with the CPU 706, GPU 708, and data storage 716, the accelerator 714 can provide a fast and efficient platform for Level 3-5 autonomous vehicles.

[0128] Therefore, this technology offers capabilities and functionalities that cannot be achieved through conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as C to execute a wide variety of processing algorithms across a diverse range of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for automotive ADAS applications and practical Level 3-5 autonomous vehicles.

[0129] In contrast to conventional systems, the techniques described in this paper, by providing CPU complexes, GPU complexes, and hardware accelerator clusters, allow multiple neural networks to be executed simultaneously and / or sequentially, and the results combined to achieve Level 3–5 autonomous driving capabilities. For example, a CNN executed on a DLA or dGPU (e.g., GPU 720) could include text and word recognition, allowing a supercomputer to read and understand traffic signs, including those for which neural networks have not yet been specifically trained. The DLA could also include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs, and passing that semantic understanding to a path planning module running on the CPU complex.

[0130] As another example, multiple neural networks can operate simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Caution: Flashing lights indicate icy conditions," along with a light, can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a deployed first neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a deployed second neural network, which informs the vehicle's path planning software (preferably executing on a CPU complex) that icy conditions exist when the flashing lights are detected. The flashing lights can be identified by a deployed third neural network operating across multiple frames, which informs the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can operate simultaneously, for example, within a DLA and / or on a GPU 708.

[0131] In some examples, the CNN used for facial recognition and owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 700. A processing engine always on the sensors can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in safe mode, to disable the vehicle when the owner leaves. In this way, SoC 704 provides security against theft and / or carjacking.

[0132] In another example, the CNN used for emergency vehicle detection and identification can use data from microphone 796 to detect and identify emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect siren and manually extract features, SoC 704 uses a CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to recognize the relative shut-off rate of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the localized area in which the vehicle operates, as identified by GNSS sensor 758. Thus, for example, when operating in Europe, the CNN will seek to detect European siren, and when operating in the United States, the CNN will seek to identify siren only in North America. Once an emergency vehicle is detected, with the assistance of ultrasonic sensor 762, the control program can be used to execute emergency vehicle safety routines, causing the vehicle to slow down, pull over to the side of the road, stop, and / or idle until the emergency vehicle passes.

[0133] The vehicle may include a CPU 718 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 704 via a high-speed interconnect (e.g., PCIe). The CPU 718 may include, for example, an x86 processor. The CPU 718 can be used to perform any of a wide variety of functions, including, for example, arbitrating the results of potential inconsistencies between ADAS sensors and the SoC 704, and / or monitoring the status and health of the controller 736 and / or the infotainment SoC 730.

[0134] Vehicle 700 may include a GPU 720 (e.g., a discrete GPU or dGPU) that can be coupled to SoC 704 via a high-speed interconnect (e.g., NVIDIA's NVLINK). GPU 720 may provide additional artificial intelligence capabilities, for example by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on inputs from sensors of vehicle 700 (e.g., sensor data).

[0135] Vehicle 700 may also include a network interface 724, which may include one or more wireless antennas 726 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). Network interface 724 can be used to enable wireless connectivity via the Internet to the cloud (e.g., with server 778 and / or other network devices), with other vehicles, and / or with computing devices (e.g., passenger client devices). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across networks and via the Internet). A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 700 with information about vehicles approaching vehicle 700 (e.g., vehicles in front, to the side, and / or behind vehicle 700). This functionality can be part of vehicle 700's cooperative adaptive cruise control function.

[0136] Network interface 724 may include a SoC that provides modulation and demodulation functions and enables controller 736 to communicate via a wireless network. Network interface 724 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be performed by known processes and / or using a superheterodyne process. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functions for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0137] Vehicle 700 may also include data storage 728, which may include off-chip (e.g., off-chip SoC 704) storage devices. Data storage 728 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0138] Vehicle 700 may also include a GNSS sensor 758. The GNSS sensor 758 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used for auxiliary mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 758 can be used, including, for example, but not limited to, GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0139] Vehicle 700 may also include a RADAR sensor 760. The RADAR sensor 760 can be used by vehicle 700 for remote vehicle detection even in dark and / or inclement weather conditions. The RADAR functional safety level may be ASIL B. The RADAR sensor 760 can use CAN and / or bus 702 (e.g., to transmit data generated by the RADAR sensor 760) for control and access to object tracking data, and in some examples, Ethernet access for accessing raw data. A wide variety of RADAR sensor types can be used. For example, and without limitation, the RADAR sensor 760 can be adapted for front, rear, and side RADAR use. In some examples, a pulse Doppler RADAR sensor is used.

[0140] The RADAR sensor 760 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, the long-range RADAR can be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view (e.g., within 250m) achieved through two or more independent scans. The RADAR sensor 760 can help distinguish between stationary and moving objects and can be used by ADAS systems for emergency braking assist and forward collision warning. The long-range RADAR sensor can include a single-site multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the vehicle 700's surroundings at higher rates with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, enabling rapid detection of vehicles entering or leaving the vehicle 700's lane.

[0141] As an example, a mid-range RADAR system can include a range of up to 760m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 750 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor blind spots behind and beside the vehicle.

[0142] Short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0143] Vehicle 700 may also include ultrasonic sensors 762. Ultrasonic sensors 762, which may be positioned at the front, rear, and / or sides of vehicle 700, can be used for parking assistance and / or creating and updating occupancy grids. A wide variety of ultrasonic sensors 762 can be used, and different ultrasonic sensors 762 can be used for different detection ranges (e.g., 2.5m, 4m). Ultrasonic sensors 762 can operate at functional safety level ASIL B.

[0144] Vehicle 700 may include a LIDAR sensor 764. The LIDAR sensor 764 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 764 may be of functional safety level ASIL B. In some examples, vehicle 700 may include multiple LIDAR sensors 764 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0145] In some examples, the LiDAR sensor 764 may be able to provide a list of objects and their distances within a 360-degree field of view. A commercially available LiDAR sensor 764 may have an advertising range of, for example, approximately 700m, with an accuracy of 2cm-3cm, and support for 700Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 764 may be used. In such examples, the LiDAR sensor 764 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of a vehicle 700. In such examples, the LiDAR sensor 764 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, even for low-reflectivity objects, with a range of 200m. A front-mounted LiDAR sensor 764 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0146] In some examples, LiDAR technologies such as 3D flash LiDAR can also be used. 3D flash LiDAR uses a flash of laser light as the emission source to illuminate the vehicle's surroundings up to approximately 200 meters. A flash LiDAR unit includes a receiver that records the laser pulse propagation time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment using each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the vehicle. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-scanning LiDAR devices) without moving parts other than a fan. Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using a flash LiDAR, and because a flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 764 is less susceptible to motion blur, vibration, and / or shock.

[0147] The vehicle may also include an IMU sensor 766. In some examples, the IMU sensor 766 may be located at the center of the rear axle of the vehicle 700. The IMU sensor 766 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 766 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 766 may include an accelerometer, a gyroscope, and a magnetometer.

[0148] In some embodiments, the IMU sensor 766 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 766 can enable the vehicle 700 to estimate heading by directly observing and correlating velocity changes from GPS to the IMU sensor 766 without input from a magnetic sensor. In some examples, the IMU sensor 766 and the GNSS sensor 758 can be combined into a single integrated unit.

[0149] The vehicle may include a microphone 796 placed in and / or around the vehicle 700. Among other things, the microphone 796 may be used for emergency vehicle detection and identification.

[0150] The vehicle may also include any number of camera types, including stereo camera 768, wide-angle camera 770, infrared camera 772, surround camera 774, long-range and / or mid-range camera 798, and / or other camera types. These cameras can be used to capture image data around the entire perimeter of the vehicle 700. The camera types used depend on the embodiment and the requirements of the vehicle 700, and any combination of camera types can be used to provide the necessary coverage around the vehicle 700. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described herein with respect to... Figure 7A and Figure 7B It was described in more detail.

[0151] Vehicle 700 may also include vibration sensor 742. Vibration sensor 742 can measure vibrations of vehicle components such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 742 are used, differences between vibrations can be used to determine friction or slippage on the road surface (e.g., when there is a vibration difference between a power drive shaft and a free-rotating shaft).

[0152] Vehicle 700 may include ADAS system 738. In some examples, ADAS system 738 may include SoC. ADAS system 738 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC) and / or other features and functions.

[0153] The ACC system can use a RADAR sensor 760, a LIDAR sensor 764, and / or a camera. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to vehicles immediately in front of vehicle 700 and automatically adjusts the vehicle speed to maintain a safe distance. Lateral ACC performs distance holding and, if necessary, advises vehicle 700 to change lanes. Lateral ACC is associated with other ADAS applications such as LCA and CWS.

[0154] CACC uses information from other vehicles, which can be received indirectly from other vehicles via a wireless link or network connection (e.g., via the Internet) through network interface 724 and / or wireless antenna 726. Direct links can be provided by vehicle-to-vehicle (V2V) communication links, while indirect links can be infrastructure-to-vehicle (I2V) communication links. Typically, the V2V communication concept provides information about vehicles immediately ahead (e.g., vehicles immediately in front of vehicle 700 and in the same lane), while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of these I2V and V2V information sources. Given information about vehicles ahead of vehicle 700, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0155] The Forward-Looking Warning (FCW) system is designed to alert the driver to hazards, enabling the driver to take corrective action. The FCW system uses a front-facing camera and / or RADAR sensor 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components. The FCW system can provide warnings in the form of, for example, audible, visual, haptic, and / or rapid braking pulses.

[0156] An AEB (Autonomous Emergency Braking) system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system can use a front-facing camera and / or RADAR sensor 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes to attempt to prevent or at least mitigate the effects of the predicted collision. The AEB system may include technologies such as dynamic brake support and / or collision approach braking.

[0157] The Lane Departure Warning (LDW) system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle crosses lane markings. When the driver indicates intentional lane departure, the LDW system is deactivated by activating a turn signal. The LDW system can utilize a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0158] The LKA system is a variation of the LDW system. If vehicle 700 begins to leave the lane, the LKA system provides corrective steering input or braking to vehicle 700.

[0159] The BSW system detects and warns the driver of vehicles in the vehicle's blind spot. The BSW system can provide visual, auditory, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses turn signals. The BSW system can utilize a rear-facing camera and / or RADAR sensor 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0160] The RCTW system can provide visual, auditory, and / or tactile notifications when an object is detected outside the range of the rear camera while the vehicle is reversing. Some RCTW systems include AEB to ensure the application of the vehicle's brakes to avoid a collision. The RCTW system may use one or more rear-mounted RADAR sensors 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0161] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for the driver, but typically not catastrophic, as ADAS systems alert the driver and allow them to determine whether a safe condition truly exists and take appropriate action. However, in an autonomous vehicle 700, in the event of conflicting results, the vehicle 700 itself must decide whether to heed the results from the main computer or auxiliary computer (e.g., the first controller 736 or the second controller 736). For example, in some embodiments, the ADAS system 738 may be a backup and / or auxiliary computer for providing perception information to a backup computer rationality module. The backup computer rationality monitor may run redundant and diverse software on hardware components to detect faults in perception and dynamic driving tasks. Outputs from the ADAS system 738 may be provided to a supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0162] In some examples, the master computer can be configured to provide a confidence score to the supervisory MCU, indicating the master computer's confidence level in the selected result. If the confidence score exceeds a threshold, the supervisory MCU can follow the master computer's direction regardless of whether the auxiliary computer provides conflicting or inconsistent results. If the confidence score does not meet the threshold and the master and auxiliary computers indicate different results (e.g., conflict), the supervisory MCU can arbitrate between these computers to determine the appropriate result.

[0163] The supervisory MCU can be configured to run a neural network trained and configured to determine, at least in part, the conditions under which the auxiliary computer provides a false alarm, based on outputs from both the host and auxiliary computers. Thus, the neural network in the supervisory MCU can learn when the output of the auxiliary computer can be trusted and when it cannot. For example, when the auxiliary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying a metallic object that is not actually dangerous, such as a drain grid or manhole cover that triggers an alarm. Similarly, when the auxiliary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running the neural network using associated memory. In a preferred embodiment, the supervisory MCU may include components of and / or be included as components of the SoC 704.

[0164] In other examples, ADAS system 738 may include an auxiliary computer that performs ADAS functions using conventional computer vision rules. This allows the auxiliary computer to use classic computer vision rules (if-then), and the presence of neural networks in the supervising MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the entire system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functionality. For instance, if a software vulnerability or bug exists in the software running on the host computer and non-identical software code running on the auxiliary computer provides the same overall result, the supervising MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the host computer does not cause a substantial error.

[0165] In some examples, the output of ADAS system 738 can be fed to the perception block and / or the dynamic driving task block of the main computer. For example, if ADAS system 738 issues a forward collision warning because an object is immediately in front, the perception block can use this information when identifying the object. In other examples, the assistance computer can have its own neural network, which is trained and thus reduces the risk of false positives as described herein.

[0166] Vehicle 700 may also include an infotainment SoC 730 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 730 may include a combination of hardware and software that can be used to provide vehicle 700 with audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.) and / or information services (e.g., navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total coverage distance, brake fuel level, fuel level, door opening / closing, air filter information, etc.). For example, the infotainment SoC 730 may include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, WiFi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 734, telematics device, control panel (e.g., for controlling and / or interacting with various components, features, and / or systems) and / or other components. The infotainment SoC 730 may further be used to provide information (e.g., visual and / or auditory) to the vehicle's users, such as information from the ADAS system 738, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0167] The infotainment SoC 730 may include GPU functionality. The infotainment SoC 730 can communicate with other devices, systems, and / or components of the vehicle 700 via bus 702 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 730 may be coupled to a supervisory MCU, allowing the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 736 (e.g., the main computer and / or backup computer of the vehicle 700). In such an example, the infotainment SoC 730 may place the vehicle 700 into a driver-safe parking mode as described herein.

[0168] Vehicle 700 may also include instrument cluster 732 (e.g., digital instrument panel, electronic instrument cluster, digital instrument panel, etc.). Instrument cluster 732 may include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). Instrument cluster 732 may include a set of instruments such as speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between infotainment SoC 730 and instrument cluster 732. In other words, instrument cluster 732 may be included as part of infotainment SoC 730, or vice versa.

[0169] Figure 7D For cloud-based servers and according to some embodiments of this disclosure Figure 7A This is a system diagram illustrating communication between example autonomous vehicles 700. System 776 may include server 778, network 790, and vehicles including vehicle 700. Server 778 may include multiple GPUs 784(A)-784(H) (collectively referred to herein as GPU 784), PCIe switches 782(A)-782(H) (collectively referred to herein as PCIe switch 782), and / or CPUs 780(A)-780(B) (collectively referred to herein as CPU 780). GPUs 784, CPUs 780, and PCIe switches may be interconnected with high-speed interconnects and / or PCIe connections 786, such as, but not limited to, NVLink interface 788 developed by NVIDIA. In some examples, GPUs 784 are connected via NVLink and / or NVSwitch SoCs, and GPUs 784 and PCIe switches 782 are connected via PCIe interconnects. Although eight GPUs 784, two CPUs 780, and two PCIe switches are shown in the diagram, this is not intended to be limiting. Depending on the embodiment, each of the servers 778 may include any number of GPUs 784, CPUs 780, and / or PCIe switches. For example, each of the servers 778 may include eight, sixteen, thirty-two, and / or more GPUs 784.

[0170] Server 778 can receive image data from vehicles via network 790, representing images of unexpected or changed road conditions such as recently commenced roadworks. Server 778 can also transmit neural network 792, updated neural network 792, and / or map information 794, including information about traffic and road conditions, to vehicles via network 790. Updates to map information 794 may include updates to HD map 722, such as information about construction sites, potholes, bends, floods, or other obstacles. In some examples, neural network 792, updated neural network 792, and / or map information 794 may have been generated from new training and / or data received from any number of vehicles in the environment, and / or based on experience gained from training performed at a data center (e.g., using server 778 and / or other servers).

[0171] Server 778 can be used to train machine learning models (e.g., neural networks) based on training data. Training data can be generated by the vehicle and / or generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., where the neural network does not require supervised learning). Training can be performed according to any one or more classes of machine learning techniques, including but not limited to: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by the vehicle (e.g., transmitted to the vehicle via network 790), and / or the machine learning model can be used by server 778 to remotely monitor the vehicle.

[0172] In some examples, server 778 can receive data from vehicles and apply that data to state-of-the-art real-time neural networks for real-time intelligent inference. Server 778 may include a deep learning supercomputer powered by GPU 784 and / or a dedicated AI computer, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 778 may include a deep learning infrastructure in a data center that uses only CPU power.

[0173] The deep learning infrastructure of server 778 may be capable of rapid, real-time inference and can be used to assess and verify the health status of the processor, software, and / or associated hardware in vehicle 700. For example, the deep learning infrastructure may receive periodic updates from vehicle 700, such as image sequences and / or objects located in those image sequences by vehicle 700 (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them with objects identified by vehicle 700. If the results do not match and the infrastructure concludes that the AI ​​in vehicle 700 has malfunctioned, then server 778 may transmit a signal to vehicle 700 instructing its fail-safe computer to take control, notify passengers, and complete a safe stopping operation.

[0174] For inference, server 778 may include GPU 784 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of a GPU-powered server and inference acceleration enables real-time response. In other examples, such as where performance is less critical, CPU, FPGA, and other processor-powered servers can be used for inference.

[0175] Example computing device

[0176] Figure 8 This is a block diagram of an example computing device 800 suitable for implementing some embodiments of the present disclosure. The computing device 800 may include an interconnect system 802 directly or indirectly coupled to: a memory 804, one or more central processing units (CPUs) 806, one or more graphics processing units (GPUs) 808, a communication interface 810, input / output (I / O) ports 812, input / output components 814, a power supply 816, one or more presentation components 818 (e.g., one or more displays), and one or more logic units 820. In at least one embodiment, the computing device 800 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 808 may include one or more vGPUs, one or more CPUs 806 may include one or more vCPUs, and / or one or more logic units 820 may include one or more virtual logic units. Thus, (one or more) computing devices 800 may include discrete components (e.g., a full GPU dedicated to computing device 800), virtual components (e.g., a portion of the GPU dedicated to computing device 800), or a combination thereof.

[0177] although Figure 8 The various blocks are shown as connected via interconnect system 802 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 818 (such as a display device) may be considered I / O component 814 (e.g., if the display is a touchscreen). As another example, CPU 806 and / or GPU 808 may include memory (e.g., memory 804 may represent a storage device other than the memory of GPU 808, CPU 806, and / or other components). In other words, Figure 8 The computing devices described are for illustrative purposes only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of… Figure 8 Within the scope of computing devices.

[0178] Interconnect system 802 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 802 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 806 may be directly connected to memory 804. Further, CPU 806 may be directly connected to GPU 808. In cases where there is a direct or point-to-point connection between components, interconnect system 802 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 800.

[0179] The memory 804 may include any computer-readable medium from a variety of computer-readable media. A computer-readable medium may be any available medium accessible by the computing device 800. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0180] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented with any method or technology for storing information such as computer-readable instructions, data structures, program modules and / or other data types. For example, memory 804 may store computer-readable instructions (e.g., representing (one or more) programs and / or (one or more) program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 800. As used herein, computer storage media does not include the signal itself.

[0181] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0182] CPU 806 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 800 to perform one or more of the methods and / or processes described herein. Each CPU 806 may contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 806 may contain any type of processor and may contain different types of processors depending on the type of computing device 800 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 800, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 800 may also include one or more CPUs 806.

[0183] In addition to or in lieu of one or more CPUs 806, one or more GPUs 808 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 800 to perform one or more of the methods and / or processes described herein. One or more GPUs 808 may be integrated GPUs (e.g., having one or more CPUs 806) and / or one or more GPUs 808 may be discrete GPUs. In embodiments, one or more GPUs 808 may be coprocessors of one or more CPUs 806. GPUs 808 may be used by computing device 800 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 808 may be used for general-purpose computing on a GPU (GPGPU). GPUs 808 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPUs 808 may produce pixel data of an output image in response to rendering commands (e.g., rendering commands received from CPU 806 via a host interface). GPU 808 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 804. GPU 808 may include two or more GPUs operating in parallel (e.g., via links). The links may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 808 may produce pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0184] In addition to or in lieu of CPU 806 and / or GPU 808, logic unit 820 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 800 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 806, one or more GPUs 808, and / or one or more logic units 820 may perform any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 820 may be a portion of one or more CPUs 806 and / or GPUs 808 and / or integrated into one or more CPUs 806 and / or GPUs 808, and / or one or more logic units 820 may be discrete components or otherwise external to CPUs 806 and / or GPUs 808. In an embodiment, one or more of the logic units 820 may be coprocessors of one or more of the CPU 806 and / or one or more of the GPU 808.

[0185] Examples of logic unit 820 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or fast peripheral component interconnect (PCIe) element, etc.

[0186] The communication interface 810 may include one or more receivers, transmitters, and / or transceivers enabling the computing device 800 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). The communication interface 810 may include components and functions for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, one or more logic units 820 and / or the communication interface 810 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 802 to one or more GPUs 808 (e.g., memory of one or more GPUs 808).

[0187] I / O port 812 enables computing device 800 to be logically coupled to other devices including I / O component 814, (one or more) presentation component 818, and / or other components, some of which may be built into (e.g., integrated into) computing device 800. Illustrative I / O component 814 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 814 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 800. Computing device 800 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 800 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 800 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0188] Power supply 816 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 816 may provide power to computing device 800 so that the components of computing device 800 can operate.

[0189] The presentation component 818 may include a display (e.g., a monitor, touch screen, television screen, head-up display (HUD), other display types or combinations thereof), speakers and / or other presentation components.

[0190] The presentation component 818 may receive data from other components (e.g., GPU 808, CPU 806, DPU, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0191] Example Data Center

[0192] Figure 9 An example data center 900 that may be used in at least one embodiment of this disclosure is shown. The data center 900 may include a data center infrastructure layer 910, a framework layer 920, a software layer 930, and / or an application layer 940.

[0193] like Figure 9As shown, the data center infrastructure layer 910 may include a resource coordinator 912, grouped computing resources 914, and node computing resources (“nodes CRs”) 916(1)-916(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CRs 916(1)-916(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules and / or cooling modules, etc. In some embodiments, one or more node CRs from nodes CRs 916(1)-916(N) may correspond to servers having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CRs916(1)-9161(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CRs 916(1)-916(N) may correspond to virtual machines (VMs).

[0194] In at least one embodiment, the grouped computing resources 914 may include individual groups of node CRs 916 housed within one or more racks (not shown), or multiple racks housed within a data center in different geographical locations (also not shown). Individual groups of node CRs 916 within the grouped computing resources 914 may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs 916, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0195] Resource coordinator 912 may be configured or otherwise control one or more nodes CRs916(1)-916(N) and / or grouped computing resources 914. In at least one embodiment, resource coordinator 912 may include a Software Design Infrastructure (“SDI”) management entity for data center 900. Resource coordinator 912 may include hardware, software, or some combination thereof.

[0196] In at least one embodiment, such as Figure 9As shown, framework layer 920 may include a job scheduler 932, a configuration manager 934, a resource manager 936, and / or a distributed file system 938. Framework layer 920 may include a framework of software 932 supporting software layer 930 and / or one or more applications 942 of application layer 940. Software 932 or application 942 may respectively contain web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 920 may be, but is not limited to, free and open-source software web application frameworks (such as Apache Spark) that can utilize distributed file system 938 for large-scale data processing (e.g., "big data"). TM (Hereinafter referred to as "Spark") is a type of resource. In at least one embodiment, the job scheduler 932 may include Spark drivers to facilitate the scheduling of workloads supported by different layers of data center 900. The configuration manager 934 may be able to configure different layers, such as software layer 930 and framework layer 920 (which includes Spark and distributed file system 938 for supporting large-scale data processing). The resource manager 936 may be able to manage clusters or groups of computing resources mapped to distributed file system 938 and job scheduler 932 or allocated to support clusters or groups of distributed file system 938 and job scheduler 933. In at least one embodiment, clusters or groups of computing resources may include grouped computing resources 914 in data center infrastructure layer 910. The resource manager 936 may coordinate with resource coordinator 912 to manage these mapped or allocated computing resources.

[0197] In at least one embodiment, the software 932 included in the software layer 930 may include software used in at least a portion of the nodes CRs 916(1)-916(N), the grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of software may include, but are not limited to, internet webpage search software, email virus scanning software, database software, and streaming video content software.

[0198] In at least one embodiment, the application 942 included in the application layer 940 may include one or more types of applications used at least in part by nodes CRs 916(1)-916(N), grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in combination with one or more embodiments.

[0199] In at least one embodiment, any of the configuration manager 934, resource manager 936, and resource coordinator 912 can implement any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. Self-modification actions can free data center operators of data center 900 from making potentially poor configuration decisions and may prevent underutilization and / or poor performance of the data center.

[0200] According to one or more embodiments described herein, data center 900 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, one or more machine learning models can be trained by using the software and / or computing resources described above with respect to data center 900 to compute weight parameters according to a neural network architecture. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks can be used to infer or predict information using the resources described above with respect to data center 900 by using weight parameters computed through one or more training techniques (such as, but not limited to, those described herein).

[0201] In at least one embodiment, the data center 900 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform services that infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0202] Example network environment

[0203] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 8 This is implemented on one or more instances of computing device 800—for example, each device may include similar components, features, and / or functions of computing device 800. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of data center 900, examples of which are described in this document. Figure 9 To describe in more detail.

[0204] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0205] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0206] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0207] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). The core server may assign at least a portion of the functionality to the edge server if the connection to the user (e.g., a client device) is relatively close to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0208] (One or more) client devices may include the information described in this article. Figure 8 At least some of the components, features, and functions of the described (one or more) example computing device 800. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0209] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, which are executed by a computer or other machine such as a personal digital assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a wide variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a communication network.

[0210] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0211] This document describes in detail the subject matter of this disclosure to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be construed as suggesting any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. A system for reducing glare, comprising: One or more processing units for performing operations, the operations including: Receive image data generated using one or more cameras, the image data representing one or more images depicting one or more objects illuminated by one or more headlights of a vehicle; Based at least on determining a first region in one or more images that corresponds to one or more objects, calculate one or more contrast values ​​corresponding to the first region in one or more images; Using the one or more contrast values, and for multiple pixels in the first region, determine that the ratio between the brightness of the multiple pixels and the brightness corresponding to the second region in the one or more images is greater than a threshold; and At least based on the ratio being greater than the threshold, data is sent to result in an adjustment of one or more parameters of the one or more headlights.

2. The system of claim 1, wherein the headlights of the one or more headlights comprise a plurality of illumination elements, and wherein transmitting the data is based at least on the illumination elements being associated with the first region of the one or more images, resulting in an adjustment of at least one of the plurality of illumination elements.

3. The system of claim 1, wherein the brightness corresponding to the second region includes a global reference brightness value.

4. The system of claim 1, wherein the second region comprises a set of pixels selected based on proximity to the one or more objects.

5. The system of claim 1, wherein the one or more contrast values ​​are stored in one or more channels of at least one image, and determining that the ratio is greater than the threshold comprises analyzing the at least one image.

6. The system of claim 1, further comprising determining the number of the plurality of pixels having a contrast value that satisfies the threshold, wherein the adjustment is based at least on the number of the plurality of pixels.

7. The system of claim 1, wherein the threshold corresponds to a reference luminance value used to calculate the one or more contrast values.

8. The system of claim 1, further comprising comparing the first region with a minimum size threshold, wherein sending the data is further based at least on the first region exceeding the minimum size threshold.

9. The system according to claim 1, wherein the second region is larger than the first region.

10. A method for reducing glare, comprising: Receive image data generated using one or more cameras, the image data representing one or more images depicting one or more objects illuminated by one or more headlights of a vehicle; Based at least on determining a first region in one or more images that corresponds to one or more objects, one or more contrast values ​​corresponding to the first region in one or more images are calculated using the image data; Using the one or more contrast values, and for multiple pixels in the first region, determine that the ratio between the brightness of the multiple pixels and the brightness corresponding to the second region is greater than a threshold. as well as At least based on the ratio being greater than the threshold, data is sent to result in an adjustment of one or more parameters of the one or more headlights.

11. The method of claim 10, wherein the headlights of the one or more headlights comprise a plurality of lighting elements, and wherein transmitting the data results in an adjustment of at least one of the plurality of lighting elements.

12. The method of claim 10, wherein the one or more contrast values ​​are calculated relative to a global reference luminance value.

13. The method of claim 10, wherein the one or more contrast values ​​are calculated relative to the second region, at least based on the proximity of the second region to the plurality of pixels in the first region.

14. The method of claim 10, wherein the one or more contrast values ​​are stored in one or more channels of at least one image.

15. At least one processor, comprising: One or more circuits are configured to receive image data generated using one or more cameras of a machine, calculate a contrast value corresponding to the first region based at least on determining that a first region of pixels in one or more images represented by the image data corresponds to one or more objects, and transmit data to cause one or more operations associated with the machine based at least on an analysis of a set of said contrast values ​​greater than a threshold, wherein the contrast values ​​indicate a ratio between the brightness associated with the first region and the brightness associated with a second region of pixels in the one or more images.

16. The at least one processor according to claim 15, wherein, The first region includes one or more boundary shapes of the one or more objects detected in the one or more images.

17. The at least one processor of claim 15, wherein the one or more operations include, at least based on a subset of lighting elements of a plurality of lighting elements of one or more headlights of the machine, associating the subset with the first region, modifying the subset.

18. The at least one processor of claim 15, wherein the analysis includes determining the number of pixels in the first region having a ratio greater than the threshold, and the one or more operations are based at least on the number of pixels.

19. The at least one processor of claim 15, wherein the second region is selected based at least on the proximity of one or more pixels in the second region to the first region.

20. The at least one processor according to claim 15, wherein, The processor includes at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; A system used to perform deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that uses cloud computing resources at least in part.

Images