INTERNAL SENSOR CALIABILITY FOR AUTONOMOUS SYSTEMS AND APPLICATIONS

Abstract

This document describes the calibration of internal sensors for autonomous systems and applications in several examples. Systems and methods are disclosed for recalibrating vehicle sensors, such as position sensors, located inside the vehicle, using one or more techniques. For example, if a sensor is attached to a component inside the vehicle, an additional sensor associated with the component can output data indicating the component's location and / or orientation within the vehicle. The specified position and / or orientation of the component can then be used to recalibrate the sensor relative to a reference coordinate system of the vehicle. In a second example, the sensor can output data representing at least one feature located inside the vehicle.The sensor can then be recalibrated based on at least one section of the data that represents the feature.

Description

GENERAL STATE OF THE ART

[0001] Modern vehicles often include sensors that can be used to determine information about the vehicle's internal chambers. For example, a vehicle may include a camera positioned within the cabin and essentially directed at the vehicle's powertrain. The vehicle may then use the images captured by the camera to determine information associated with the powertrain and / or one or more components of the vehicle, such as the location of the driver's head, the driver's line of sight, the position of the seatbelt, etc. For example, a vehicle may include a RADAR sensor located within the cabin that emits RADAR signals that are essentially directed at the vehicle's occupants.The vehicle can then use the radar data generated by the radar sensor to determine information associated with the passengers, such as the presence and locations of the passengers within the vehicle (e.g., which seats are occupied).

[0002] To ensure that vehicles can reliably use the data generated by the sensors inside the vehicle, it is important that the sensors inside the vehicle are calibrated, for example, with reference to the vehicle's reference coordinate systems. For example, in-cabin sensors are typically calibrated during manufacturing, before the vehicles are made available for use (e.g., sale, rental, etc.). However, under certain circumstances, such as after use, problems can arise if the position and / or orientation of a sensor inside the vehicle changes, rendering the calibration for the sensor in the vehicle inaccurate—or at least less accurate.For example, if an interior camera is mounted on a vehicle's steering column, the values ​​for the interior camera's calibration parameters may become inaccurate (or have reduced accuracy) if a vehicle driver changes the position of the steering wheel. Another example: If an interior camera is mounted on a vehicle component, such as a vehicle's dashboard, the values ​​for the interior camera's calibration parameters may lose accuracy and / or precision over time during normal use, e.g., due to misalignment due to accumulated vibration, temperature-related deformation, displacement due to accidental impact, etc., over a period of time. SUMMARY

[0003] Embodiments of the present disclosure relate to the internal calibration of sensors for autonomous systems and applications. Systems and methods are disclosed that recalibrate sensors of a vehicle, such as position sensors located inside the vehicle, using one or more techniques. According to one or more embodiments, for example, implementations including a sensor attached to a component within the vehicle's interior, such as a component with automatic position adjustment enabled, an additional sensor connected to the component may output data indicating the position and / or orientation of the component within the vehicle. The indicated position and / or orientation of the component may then be used to determine updated values ​​for the calibration parameters associated with the sensor.In one or more alternative embodiments, the position of features as represented by a sensor representation (e.g., an image) may be used to determine updated values ​​for calibration parameters associated with the sensor.

[0004] Unlike conventional systems such as those described above, current systems in some embodiments are capable of dynamically updating the values ​​for the calibration parameters of an interior vehicle sensor, such as by using one or more of the techniques described herein. By updating the values ​​for the sensor's calibration parameters, even if the position and / or orientation of the sensor changes based on a position of an adjusted, displaced, misaligned, and / or deforming component, the sensor can still be calibrated with respect to a reference coordinate system, such as a coordinate system associated with the interior of the vehicle.As such, the vehicle may further use sensor data generated with the sensor to accurately determine information associated with the interior of the vehicle and / or perform one or more operations using that information. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The present systems and methods for calibrating internal sensors for autonomous systems and applications are described in detail below with reference to the attached figures, where: Fig. 1 illustrates an exemplary flowchart for a process for calibrating a sensor located inside a vehicle, according to some embodiments of the present disclosure; Fig. 2 illustrates an example of a sensor located inside a vehicle according to some embodiments of the present disclosure; Fig. 3 illustrates an example of determining a position of a component within a vehicle according to some embodiments of the present disclosure; Fig. 4 illustrates an example of determining a portion of a sensor representation that represents a position located inside a vehicle, according to some embodiments of the present disclosure; Fig. 5A and Fig. 5B illustrate examples of tables associating information from sensors with values ​​of one or more calibration parameters, according to some embodiments of the present disclosure; Fig. 6 illustrates an example of associating portions of a sensor representation with values ​​of one or more calibration parameters according to some embodiments of the present disclosure; Fig. 7 and Fig. 8 are flowcharts illustrating methods for calibrating sensors located inside a vehicle according to some embodiments of the present disclosure; Fig. 9A is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure; Fig. 9B is an example of camera positions and fields of view of the example autonomous vehicle from Fig. 9A according to some embodiments of the present disclosure; Fig. 9C is a block diagram of an example system architecture for the example autonomous vehicle of Fig. 9A according to some embodiments of the present disclosure; Fig. 9D is a system diagram for communication between (a) cloud-based server(s) and the example autonomous vehicle of Fig. 9A according to some embodiments of the present disclosure; Fig. 10 is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure; and Fig. 11 is a block diagram of an example data center suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0006] Systems and methods are disclosed relating to the calibration of internal sensors for autonomous systems and applications. Although the present disclosure is described with respect to an example of an autonomous or semi-autonomous vehicle 900 (alternatively referred to herein as "vehicle 900" or "ego machine 900", an example of which is described with respect to Fig. 9A-9D), this is not intended to be limiting. The systems and methods described herein may be used, without limitation, by, for example, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), guided and unguided robots or robot platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, airships, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other types of vehicles.Furthermore, although the present disclosure may be described with respect to sensor calibration, this is not intended to be limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technology spaces where sensor data may be used.

[0007] For example, one or more systems may use one or more techniques to determine one or more values ​​for one or more calibration parameters associated with one or more sensors located within an interior of a vehicle. As described herein, in some examples, the calibration parameter(s) may include, but are not limited to, one or more translation parameters (e.g., a position with x-coordinates, a position with y-coordinates, a position with z-coordinates, etc.), one or more rotation parameters (e.g., a pitch, a yaw, a roll, etc.), and / or any other calibration parameter. Furthermore, in some examples, a sensor may include, but is not limited to, an image sensor (e.g., a red-green-blue (RGB) camera, an infrared camera, etc.), a radar sensor, a lidar sensor, and / or any other type of sensor.Furthermore, in some examples, the interior of the vehicle may include, but is not limited to, a cabin, a passenger compartment, a cockpit, and / or other interior space within the vehicle.

[0008] Specifically, the system(s) may determine one or more initial values ​​(e.g., nominal values) for the calibration parameter(s) of a sensor, where the initial value(s) are used to calibrate a coordinate system associated with the sensor (e.g., a two-dimensional (2D) coordinate system, a three-dimensional (3D) coordinate system, etc.) with a reference coordinate system associated with the vehicle (e.g., a 2D coordinate system, a 3D coordinate system, etc.). In some examples, the reference coordinate system is associated with a feature of the vehicle, such as an axle, instrument panel, seat, window, decal, and / or other component. In some examples, the system(s) determine the initial value(s) for the calibration parameter(s) based on one or more factory calibration methods (e.g.,prior to use of the vehicle) and / or using one or more of the methods described herein. The system(s) may then determine one or more updated values ​​for the calibration parameter(s) based on the occurrence of one or more events. As described herein, an event may include, but is not limited to, a lapse of time (e.g., every second, minute, day, etc.), a detection that a position and / or orientation of the sensor has changed, an input that a position and / or orientation of a component with which the sensor is associated has adjusted, a user input, and / or any other event.

[0009] For example, a location (position) and / or orientation assigned to the sensor may change within the vehicle. In some examples, the change in the position and / or orientation of the sensor may occur at least based on an adjustment of the position and / or orientation of a component to which the sensor is attached. For example, if the sensor is attached to a steering column of the vehicle, a user (e.g., a driver) of the vehicle may adjust the position of the steering wheel, which may further cause a change in the position and / or orientation of the sensor. In some examples, the change in the position and / or orientation of the sensor may result based on a deformation of a component to which the sensor is attached, which may occur during normal use of the vehicle over a (e.g., long) period of time.Any system may no longer be precisely or accurately calibrated with respect to the reference coordinate system associated with the vehicle based on a change in the position and / or orientation of the sensor. As such, the system(s) may use one or more techniques to determine the updated value(s) for the calibration parameter(s), with the updated value(s) recalibrating the sensor with respect to the reference coordinate system associated with the vehicle.

[0010] One or more embodiments of the present disclosure may include a first example technique (e.g., a direct observation technique), such as when the sensor is associated (e.g., attached) with a component that allows automatic adjustment (e.g., an electric steering column). In embodiments implementing this first example technique, the system(s) may receive (e.g., using a Controller Area Network (CAN) signal) data generated using one or more additional sensors connected to the component, where the data indicates a new location and / or orientation of the component. The system(s) may then use the new location and / or orientation of the component to determine the updated value(s) for the sensor's calibration parameter(s).In some examples, the system(s) determines the updated value(s) using a lookup table, such as a lookup table that maps different positions and / or orientations of the component to different values ​​for the calibration parameter(s). In some examples, the system determines the updated value(s) using one or more other techniques, such as one or more equations that determine the updated value(s) using the initial value(s) for the calibration parameter(s) and the new location and / or orientation of the component.

[0011] One or more embodiments may include a second example technique (e.g., an indirect observation technique), such as when the sensor is connected (e.g., attached) to a component that allows for manual adjustment (e.g., when the vehicle does not include a sensor that locates the position and / or orientation of the component). In embodiments that include implementations of this second example technique, the system(s) may receive sensor data generated with the sensor, where the sensor data represents a sensor representation associated with the interior of the vehicle. The system(s) may then process the sensor data and, based on the processing, determine that a portion of the sensor representation is associated with a feature located within the interior of the vehicle.For example, if the sensor data includes image data representing an image, then the system may determine that one or more pixels (e.g., a portion) of the image represents the feature. The system(s) may then use the portion of the sensor representation to determine the updated value(s) for the sensor's calibration parameter(s). In some examples, the system determines the updated value(s) using a lookup table, such as a lookup table that maps different portions of sensor representations to different values ​​for the calibration parameter(s).In some examples, the system determines the updated value(s) using one or more other techniques, such as one or more equations that determine the updated value(s) using the initial value(s) for the calibration parameter(s) and the portion of the sensor representation.

[0012] In some examples, the system(s) may use a combination of the two example techniques to determine the updated value(s) for the calibration parameter(s). For example, the system(s) may determine one or more first updated values ​​for the calibration parameter(s) using the first example technique and one or more second updated values ​​for the calibration parameter(s) using the second example technique.The system(s) may then use the first updated value(s) to verify the second updated value(s), use the second updated value(s) to verify the first updated value(s), and / or determine one or more third updated values ​​for the calibration parameter(s) using the first updated value(s) and the second updated value(s) (e.g., averaging the first updated value(s) and the second updated value(s)). Additionally or alternatively, in some examples, the system may select one of the example techniques for determining the updated value(s) if both example techniques are available. For example, the system(s) may select the example technique that is more accurate, can be used more quickly to determine the updated value(s), and / or provides some other advantage.

[0013] In some examples, after determining the updated value(s) for the calibration parameter(s), the system may use the updated value(s) to perform one or more operations. For example, the system(s) may use the updated value(s) to determine information associated with one or more users and / or one or more components of the vehicle, such as a location of a user, a position of a user, a direction of gaze of a user, a location of a component (e.g., the position of a seatbelt), and / or any other information. In some examples, the system(s) may then perform one or more additional operations based at least on the information. For a first example, the system(s) may...the systems may use at least the gaze direction of a user to perform an additional operation, such as activating a feature (e.g., a radio) when the user looks at the feature and says a command. In a second example, the system(s) may use at least the position of a driver to perform an additional operation, such as determining a condition of the driver. While these are just a few example operations that the system(s) may perform using the updated value(s) for the calibration parameter(s), in other examples, the system(s) may perform additional and / or alternative operations using the updated value(s).

[0014] The systems and methods described herein may be used for a variety of purposes, including, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twin operations, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, generative AI, (large-scale) language models, and / or any other suitable applications.

[0015] Disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot, antenna systems, media systems, boat systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twinning operations, systems implemented using an edge device, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems,that implement one or more large-scale language models (LLMs), systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets, systems implemented at least partially using cloud computing resources, systems for performing generative AI operations, systems for implementing - or performing operations using - a large-scale language model (LLM), and / or other types of systems.

[0016] With reference to Fig. 1 illustrates Fig. 1 is an exemplary flowchart for a process 100 for calibrating a sensor located inside a vehicle, according to some embodiments of the present disclosure. It should be understood that these and other arrangements described herein are provided as examples only. Other arrangements and elements (e.g., engines, interfaces, functions, instructions, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Further, many of the elements described herein are functional units that may be implemented as discrete or distributed components, or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by units may be performed by hardware, firmware, and / or software.For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein may be implemented using similar components, features, and / or functions as those of the exemplary autonomous vehicle 900 of FIG. Fig. 9A-9D, the exemplary computing device 1000 of Fig. 10 and / or the exemplary data center 1100 from Fig. 11 are executed.

[0017] The process 100 may include a calibration component 102 that receives first sensor data 104 from a first sensor 106 and / or second sensor data 108 from a second sensor 110. In the example of Fig. 1, the first sensor 106 may include, but is not limited to, an image sensor (e.g., an RGB camera, an infrared camera, etc.), a radar sensor, a lidar sensor, and / or any other type of sensor. As such, the first sensor data 104 may include image data, radar data, lidar data, and / or any other type of data. Furthermore, the second sensor 110 may include a position sensor, a position sensor, a motion sensor, and / or any other type of sensor configured to generate the second sensor data 108 indicating the location and / or orientation of a component 112 located inside a vehicle.

[0018] For example, in some examples, the first sensor 106 may be located on the component 112 within the vehicle, with one or more users of the vehicle being able to adjust the position and / or orientation of the component 112. In some examples, the user(s) may adjust the component 112 either manually and / or automatically using one or more controllers 114 associated with the component 112. The second sensor 110 may then be connected to the component 112 and output the second sensor data 108 indicating the current location and / or orientation of the component 112 and / or the change in the location and / or orientation of the component 112, such as when the component is adjusted.

[0019] Fig. For example, Figure 2 illustrates an example of a sensor 202 (which may represent and / or include the first sensor 106) located in an interior 204 of a vehicle according to some embodiments of the present disclosure. In the example of Fig. 2, the sensor 202 may include a camera that is substantially directed toward a driver 206 of the vehicle. For example, the sensor data generated with the sensor 202 (e.g., the first sensor data 104) may be used to determine information associated with the driver 206, such as a viewing direction of the driver 206. As shown, the sensor 202 may be attached to a component (e.g., component 112) located within the interior 204 of the vehicle, such as a steering column 208 of the vehicle. In the example of Fig. 2, a position and / or orientation of the steering column 208 may be adjusted by the driver 206, such as to increase the comfort of the driver 206 when driving the vehicle.

[0020] In some examples, the vehicle may include one or more inputs (e.g., controller(s) 114) configured to automatically adjust the position and / or orientation of a steering wheel 210 based on inputs from the driver 206. In some examples, the vehicle may include one or more additional components that the driver 206 may use to manually adjust the position and / or orientation of the steering wheel 210. In both examples, an adjustment to the position and / or orientation of the steering column 208 may further occur based on the driver 206's adjustment of the position and / or orientation of the steering wheel 210. Because the sensor 202 is attached to the steering column 208, a change in the location and / or orientation of the sensor 202 may result with the adjustment of the steering column 208.As described herein, a change in the location and / or orientation of the sensor 202 may cause the sensor 202 to no longer be calibrated with reference to a vehicle reference coordinate system.

[0021] In some examples, such as when automatically adjusting the steering wheel 210 using the one or more inputs, the steering column 208 may include and / or be connected to another sensor (e.g., the second sensor 110) configured to determine the position and / or orientation of the steering column 208. As described in more detail herein, sensor data (e.g., the second sensor data 108) generated with the additional sensor may be used to recalibrate the sensor 202 with respect to the reference coordinate system. However, in other examples, such as when manually adjusting the steering wheel 210 using the other component(s), the steering column 208 may not include and / or be associated with such an additional sensor. In this case, the sensor data generated with the sensor 202 (e.g.,the first sensor data 104) can still be used to recalibrate the sensor 202 with respect to the reference coordinate system.

[0022] For example, and going back to the example of Fig. 1, the process 100 may include the calibration component 102, which uses the first sensor data 104 generated with the first sensor 106 and / or the second sensor data 108 generated with the second sensor 110 to recalibrate the first sensor 106. As shown, the calibration component 102 may store initial sensor data 116 representing one or more initial (e.g., nominal) values ​​for one or more calibration parameters that calibrate the first sensor 106 with respect to a reference coordinate system of the vehicle. As described herein, the calibration parameter(s) may include, but are not limited to, one or more translational parameters (e.g., a position with x-coordinates, a position with y-coordinates, a position with z-coordinates, etc.), one or more rotational parameters (e.g., a pitch, yaw, roll, etc.), and / or any other calibration parameter.Thus, the initial calibration data 116 may represent a first initial value associated with the x-coordinate position, a second initial value associated with the y-coordinate position, a third initial value associated with the z-coordinate position, a fourth initial value associated with pitch, a fifth initial value associated with yaw, and / or a sixth initial value associated with roll.

[0023] The process 100 may include the calibration component 102 using a processing component 118 to determine one or more updated values ​​for the calibration parameter(s) using the first sensor data 104 and / or the second sensor data 108. For example, the processing component 118 may first use the first sensor data 104 and / or the second sensor data 108 to determine information associated with performing the calibration. For a first example, the processing component 118 may process the second sensor data 108 and, based on the process, determine the current position and / or orientation associated with the component 112 with which the first sensor 106 is associated.For a second example, the processing component 118 may process the first sensor data 104 and, based on at least the processing, determine a portion of a sensor representation represented by the first sensor data 104, wherein the portion of the sensor representation represents a feature that is also located inside the vehicle. As described herein, a feature may include a component of the vehicle (e.g., a handle, a window, a panel (e.g., a window trim, etc.), a headrest, a knob, a button, etc.), a sticker (e.g., an airbag sticker, etc.), and / or any other feature located inside the vehicle.

[0024] For further details, Fig. 3 illustrates an example of determining the position of a component 302 (which may represent and / or include component 112 and / or steering column 208) in a vehicle according to some embodiments of the present disclosure. In the example of Fig. 3, component 302 may initially be located at a current position indicated by the white rectangle, with portions 304(1)-(2) of component 302 located at positions 306(1)-(2), respectively, and portions 308(1)-(2) of component 302 located at positions 310(1)-(2), respectively, when component 302 is at the current position. Component 302 may then be adjusted by a distance 312(1) in a first direction such that portion 304(1) reaches a position 314(1), or by a distance 312(2) in a second direction such that portion 304(2) reaches a position 314(2). Additionally, component 302 may be adjusted by a distance 316(1) in a third direction such that portion 308(1) reaches a position 318(1), or by a distance 316(2) in a fourth direction such that portion 308(2) reaches a position 318(2).

[0025] For example, if component 302 includes a steering column, the first direction and second direction may be associated with reach (e.g., outward or inward) and the third direction and fourth direction may be associated with pitch (e.g., down or up). As such, processing component 118 may process sensor data (e.g., second sensor data 108) generated with a sensor (e.g., second sensor 110) associated with component 302 to locate the position of component 302. As described herein, a portion 304(1)-(2) and / or a portion 308(1)-(2) may include, but is not limited to, a side, a component, a closure, and / or any other portion of component 302.

[0026] Fig. 4 illustrates an example of determining a portion 402 of a sensor representation 404 that represents a feature 406 located in an interior 408 of a vehicle, according to some embodiments of the present disclosure. In the example of Fig. 4, the sensor representation 404 may include an image captured using an image sensor, such as an image sensor substantially directed toward a driver 410 of the vehicle. However, in other examples, the sensor representation 404 may be generated using any other sensor. As shown, the processing component 118 may process the sensor representation 404 to determine that the portion 402 of the sensor representation 404 depicts the feature 406. For example, and again when the sensor representation 404 includes an image, the portion 402 of the image may include one or more pixels depicting the feature 406. As such, the processing component 118 may process the image to determine one or more locations associated with the pixel(s).

[0027] Going back to the example in Fig. 1, the processing component 118 may then use one or more techniques to calibrate the first sensor 106 relative to the vehicle's reference coordinate system. For example, the processing component 118 may analyze the information determined with the first sensor data 104 and / or the second sensor data 108 relative to a data structure (e.g., a table represented by table data 120), where the data structure associates information with different values ​​for the calibration parameter(s). For a first example, the data structure may associate different positions and / or orientations of the adjustable component 112 associated with the second sensor data 108 with the different values ​​for the calibration parameter(s). For example, the data structure may associate different portions of sensor data with the different values ​​for the calibration component(s).At least based on the analysis, the processing component 118 may compare the information with corresponding information from the data structure and use the comparison to determine the value(s) for the calibration parameter(s).

[0028] Fig. 5A and Fig. 5B illustrate examples of tables that associate information from sensors with values ​​of one or more calibration parameters according to some embodiments of the present disclosure. In the example of Fig. 5A, a first table 502 may, for example, associate values ​​506(1)-(X) (also individually referred to as "position 504" or plurally as "positions 504") for the calibration parameter(s) with locations 504(1)-(X) (also individually referred to as "value(s) 506" or plurally as "values ​​506"). For example, the first position 504(1) at which a component is located may be associated with the first value(s) 506(1) for the calibration parameter(s), the second position 504(2) at which the component is located may be associated with the second value(s) 506(2) for the calibration parameter(s), and / or so on. In some examples and as described herein, a position 504 may be associated with one or more positions in one or more directions.

[0029] For example, and to refer to the example of Fig. 3, the first position 504(1) may be associated with the component 302 at the current location (e.g., as indicated by the white rectangle), the second position 504(2) may be associated with the component 302 located at position 306(1) and position 318(1), the third position 504(3) may be associated with the component 302 located at position 306(2) and position 318(2), and / or so on.

[0030] As such, the processing component 118 may use the first table 502 to determine the value(s) associated with the calibration parameter(s) based on at least a current position 504 of the component. For example, the processing component 118 may receive sensor data (e.g., the second sensor data 108) generated with a sensor (e.g., the second sensor 110) associated with the component. The processing component 118 may then analyze the sensor data to determine the current position 504 associated with the component. Based on at least the current position 504 and the first table 502, the processing component 118 may then determine the value(s) for the calibration parameter(s) for another sensor (e.g., the first sensor 106).For example, if the processing component 118 determines, based at least on the analysis of the sensor data, that the component is located at the third position 504(3), then the processing component 118 may determine the third value(s) 506(3) for the calibration parameter(s).

[0031] Additionally and in the example of Fig. 5B, a second table 508 may associate sections 510(1)-(X) (also individually referred to as a "section 510" or plurally as "sections 510") with values ​​512(1)-(Y) (also individually referred to as "value(s) 512" or plurally as "values ​​512") for the calibration parameter(s). For example, the first section 510(1) of a sensor representation may be associated with the first value(s) 512(1) for the calibration parameter(s), the second section 510(2) of a sensor representation may be associated with the second value(s) 512(2) for the calibration parameter(s), and / or so on. In some examples and as described herein, a section 510 of a sensor representation may correspond to one or more pixel locations of an image. However, in other examples, a portion 510 of a sensor representation may also correspond to other characteristics associated with a different type of sensor representation.For example, when a sensor representation is associated with a frame of LIDAR data, a portion 402 of the frame may correspond to one or more points.

[0032] For example, Fig. 6 shows an example of the association of portions of a sensor representation 602 with the values ​​of one or more calibration parameters according to some embodiments of the present disclosure. As shown, the sensor representation 602 may be associated with a detection area 604 for which a feature located inside the vehicle may be represented. For example, a sensor used to generate the sensor representation 602 may be attached to a component whose location and / or orientation a user can adjust (e.g., manually and / or automatically), thereby changing the position and / or orientation of the sensor itself. Thus, the sensor representation 602 may be associated with four extreme adjustments, such as those of component 302 in the example of Fig. 3 associated hard stops, represent the feature either at a first section 606(1), a second section 606(2), a third section 606(3), or a fourth section 606(4). These four sections 606(1)-(4) can then represent the corners of the detection area 604 associated with the feature.

[0033] As the example in Fig. 6 further shows, the sensor representation 602 may further represent the feature in other portions within the detection area 604, such as at least a fifth portion 608(1) and a sixth portion 608(2), based on the position and / or orientation of the sensor. As such, and referring to the example of Fig. 5B, the first portion 606(1) may include the first portion 510(1) and be associated with the first value(s) 512(1), the second portion 606(2) may include the second portion 510(2) and be associated with the second value(s) 512(2), the third portion 606(3) may include the third portion 510(3) and be associated with the third value(s) 512(3), the fourth portion 606(4) may include the fourth portion 510(4) and be associated with the fourth value(s) 512(5), the fifth portion 608(1) may include the last portion 510(Y) and be associated with the last value(s) 512(Y), and / or so on.

[0034] In the example of Fig. 6, the sensor representation 602 may include an image generated using one or more image sensors. In such an example, the portions 606(1)-(4) and / or the portions 608(1)-(2) may correspond to pixel locations associated with the image. For example, the first portion 606(1) may correspond to one or more pixel locations within the image, where a pixel location may be specified using at least a first distance in a first direction (e.g., a number of pixels in the x-direction) and a second distance in a second direction (e.g., a number of pixels in the y-direction). Furthermore, the fifth portion 608(1) may correspond to one or more additional pixel locations within the image, where a pixel location may in turn be specified by at least a first distance in a first direction (e.g., a number of pixels in the x-direction) and a second distance in a second direction (e.g.,a number of pixels in the y-direction).

[0035] As such, the processing component 118 may use the second table 508 to determine the value(s) to be assigned to the calibration parameter(s) based on at least a portion 510 of the sensor representation. For example, the processing component 118 may receive sensor data (e.g., the first sensor data 104) generated using a sensor (e.g., the first sensor 106). The processing component 118 may then analyze the sensor data to determine a portion 510 of a sensor representation that represents a feature located inside the vehicle. Based on at least the portion 510 and the second table 508, the processing component 118 may then determine the value(s) 512 for the calibration parameter(s) of the sensor.For example, if the processing component 118 determines, based on the analysis of the sensor data, that the third portion 510(3) of the sensor representation depicts the feature, the processing component 118 may determine the third value(s) 506(3) for the calibration parameter(s).

[0036] To refer to the example of Fig. Returning to Figure 1, in addition to or alternatively to using the tables to calibrate the first sensor 106 with respect to the vehicle's reference coordinate system, the processing component 118 may, in some examples, use one or more additional techniques. For example, the processing component 118 may use the initial calibration data 116, the information determined using the first sensor data 104 and / or the second sensor data 108 (e.g., the position and / or orientation of the component 112, the portion of the sensor representation, etc.), and / or one or more equations to calibrate the first sensor 106 with respect to the reference coordinate system.For a first example, the one or more equations may be configured to determine the updated value(s) for the calibration parameter(s) based on at least the initial value(s) for the calibration parameter(s), an initial position and / or orientation of the component 112, and / or a new position and / or orientation of the component 112. For a second example, the one or more equations may be configured to determine the updated value(s) for the calibration parameter(s) based on at least the initial value(s) for the calibration parameter(s), an initial portion of an initial sensor representation representing the feature, and / or a new portion of a new sensor representation representing the feature.

[0037] In some examples, when calibrating the first sensor 106 with respect to the reference coordinate system, the processing component 118 may use additional data associated with the vehicle, such as model data 122 representing a model associated with the vehicle (e.g., a computer-aided design model). For a first example, the processing component 118 may use the model in addition to the first sensor data 104 to determine the portion of the sensor representation that depicts the feature. For a second example, the processing component 118 may use the model in addition to the second sensor data 108 to determine the position and / or orientation of the component 112 associated with the first sensor 106. In either example, the processing component 118 may then perform one or more of the processes described herein (e.g., using the table, using the one or more equations, etc.).) to process the information (e.g., the position and / or orientation of the component 112, the section of the sensor representation, etc.) to calibrate the first sensor 106 with respect to the reference coordinate system.

[0038] Additionally or alternatively, in some examples, because the model may include information associated with the vehicle, such as the initial value(s) for the calibration parameter(s) associated with the initial position and / or orientation of the first sensor 106 and / or the initial position and / or orientation of the component 112, the processing component 118 may use the model to directly determine the updated value(s) for the calibration parameter(s). For a first example, the processing component 118 may determine the updated value(s) for the calibration parameter(s) using the model and a new position and / or orientation of the component 112, such as by using one or more equations.For a second example, processing component 118 may determine the updated value(s) for the calibration parameter(s) using the model and a new portion of a new sensor representation representing the feature, such as using one or more equations. While these are just a few example techniques of how processing component 118 may calibrate first sensor 106 relative to the vehicle's reference coordinate system, in other examples, processing component 118 may use additional and / or alternative techniques.

[0039] In some examples, the process 100 may include using a processing component 124 to perform one or more verifications associated with calibrating the first sensor 106. For example, if the processing component 124 determines one or more first values ​​for the calibration parameter(s) using a first technique, such as the directly observed technique using the second sensor data 108 (e.g., component motion), and one or more second values ​​for the calibration parameter(s) using a second technique, such as the indirectly observed technique using the first sensor data 104 (e.g., imaging data), the verification component 124 may use the first value(s) and the second value(s) to calibrate the first sensor 106.For a first example, the verification component 124 may use the first value(s) determined using the first technique to verify the second value(s) determined using the second technique. For a second example, the verification component 124 may use the second value(s) determined using the second technique to verify the first value(s) determined using the first technique. In both examples, a value may be verified against another, corresponding value based on a threshold that is within the corresponding value.

[0040] For a first example, if the values ​​are associated with a translation parameter in a particular direction (e.g., the x-direction), the checking component 124 may check a first value for the translation parameter based on a second value for the translation parameter that meets a threshold distance from the first value (e.g., is less than or equal to). Furthermore, the checking component 124 may not check the first value based on the second value that does not meet the threshold distance from the first value (e.g., is greater than). For a second example, if the values ​​are associated with a rotation parameter, the checking component 124 may verify a first value for the rotation parameter based on at least one second value for the rotation parameter that meets a threshold angle from the first value (e.g., is less than or equal to).Furthermore, the checking component 124 may not check the first value based on the second value not meeting (e.g., being greater than) the threshold angle to the first value.

[0041] In some examples, the verification component 124 may verify the value(s) for the calibration parameter(s) by determining one or more third values ​​for the calibration parameter(s) using the first value(s) determined using the first technique and the second value(s) determined using the second technique. For a first example, the verification component 124 may determine the third value(s) based on at least an average of the first value(s) and the second value(s). For a second example, the verification component 124 may determine the third value(s) using any other type of equation and based on at least the first value(s) and the second value(s).

[0042] The process 100 may include the calibration component 102 generating and / or outputting updated calibration data 126. As described herein, the updated calibration data 126 may represent the updated value(s) for the calibration parameter(s) that calibrate the first sensor 106 relative to the vehicle's reference coordinate system. For example, the updated calibration data 126 may represent a first updated value associated with the x-coordinate position, a second updated value associated with the y-coordinate position, a third updated value associated with the z-coordinate position, a fourth updated value associated with pitch, a fifth updated value associated with yaw, and / or a sixth updated value associated with roll.

[0043] In some examples, process 100 may include calibration component 102 continuing calibration of first sensor 106 with respect to the reference coordinate system using one or more of the processes described herein, based on at least one or more events occurring. As described herein, an event may include, but is not limited to, a lapse of time (e.g., every second, minute, day, etc.), detection of a change in a position and / or orientation of first sensor 106, detection of an adjustment in a position and / or orientation of component 112 with which first sensor 106 is associated, user input, and / or any other event.

[0044] In some examples, the vehicle (e.g., one or more systems of the vehicle) may then use the updated calibration data 126 to perform one or more operations associated with the vehicle. For a first example, if the first sensor 106 includes a forward-facing camera that generates image data representing one or more images depicting a driver of the vehicle, then the vehicle may process the image data to determine information associated with the driver, such as a gaze direction associated with the driver. To accurately determine the gaze direction, the vehicle may use the updated calibration data 126 to determine the driver's gaze direction relative to the vehicle's reference coordinate system.In this way, even if the locations and / or orientations of the cameras change within the vehicle, the vehicle is able to accurately use the image data generated by the camera to determine the drivers' viewing directions with respect to the reference coordinate system.

[0045] For a second example, if the first sensor 106 includes a distance sensor (e.g., a RADAR sensor, a LIDAR sensor, etc.) that generates sensor data 104 representing distances to the occupants of the vehicle, then the vehicle may process the sensor data 104 to determine at least a distance to one of the occupants. Furthermore, the vehicle may use the distance along with the updated calibration data 126 to determine at least the position of the occupant within the vehicle, such as the seat the occupant is occupying. In this way, even if the locations and / or orientations of the first sensor 106 change within the vehicle, the vehicle may accurately use the sensor data 104 generated by the first sensor 106 to determine the positions of the occupants within the vehicle.While these are only two example operations that the vehicle may perform using the updated calibration data 126, in other examples, the vehicle may also perform additional and / or alternative operations.

[0046] Now referring to Fig. 7 and Fig. 8, each block of methods 700 and 800 described herein comprises a computational process that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. Methods 700 and 800 may also be embodied as computer-usable instructions stored on computer storage media. Methods 700 and 800 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in for another product, to name a few. Furthermore, methods 700 and 800 are described, by way of example, with reference to Fig. 1. However, these methods 700 and 800 may additionally or alternatively be performed by any system or combination of systems, including, but not limited to, those described herein.

[0047] Fig. 7 is a flowchart illustrating a method 700 for calibrating a sensor located inside a vehicle, according to some embodiments of the present disclosure. The method 700, at block B702, may include determining a position associated with a feature located within an interior of the vehicle based on first sensor data generated with one or more first sensors or a second sensor of a vehicle. For example, in some examples, the calibration component 102 (e.g., the processing component 118) may receive the sensor data 108 generated with the sensor(s) 110, where the sensor(s) 110 is / are associated with the feature (e.g., position 112) located within the vehicle. The calibration component 102 may then analyze the sensor data 108 to determine at least the position associated with the feature within the vehicle.Additionally or alternatively, in some examples, the calibration component 102 may receive the sensor data 104 generated by the sensor 106. The calibration component 102 may then determine the position of the feature as localized by a sensor representation represented by the sensor data 104.

[0048] The method 700 may include, at block B704, determining one or more values ​​for one or more calibration parameters of the second sensor based on the position associated with the feature. For example, the calibration component 102 (e.g., the processing component 118) may determine the value(s) for the calibration parameter(s) of the sensor 106 based on at least the position (and / or orientation) of the feature. As described herein, in some examples, the calibration component 102 may determine the value(s) using a table that maps different positions of the feature to different values ​​for the calibration parameter(s). In some examples, the calibration component 102 may determine the value(s) using one or more additional and / or alternative techniques, such as using one or more equations and / or models associated with the vehicle.

[0049] The method 700 may include receiving second sensor data generated with the second sensor at block B706. Thus, the vehicle (e.g., one or more systems of the vehicle) may receive the sensor data 104 generated with the sensor 106. As described herein, the sensor data 104 may include, among other things, image data generated with one or more image sensors, radar data generated with one or more radar sensors, LiDAR data generated with one or more LiDAR sensors, and / or any other type of sensor data.

[0050] Method 700 may, at block B708, cause the vehicle to perform one or more operations associated with the vehicle based on the second sensor data and the one or more values ​​for the one or more calibration parameters. For example, the vehicle may use the sensor data 104 and the value(s) for the calibration parameter(s) to determine information associated with the interior of the vehicle, such as information associated with a user and / or a component of the vehicle. The vehicle may then use the information to perform the one or more operations described herein.

[0051] Fig. 8 is a flowchart illustrating a method 800 for calibrating a sensor located inside a vehicle, according to some embodiments of the present disclosure. The method 800 may include, at block B802, determining one or more first values ​​for one or more calibration parameters of a first sensor located at an interior location of a vehicle. For example, the calibration component 102 may use the initial calibration data 116 to determine the first value(s) for the calibration parameter(s).As described herein, the first value(s) for the calibration parameter(s) may include a first initial value associated with the x-coordinate position, a second initial value associated with the y-coordinate position, a third initial value associated with the z-coordinate position, a fourth initial value associated with pitch, a fifth initial value associated with yaw, and / or a sixth initial value associated with roll.

[0052] The method 800 may include, at block B804, receiving sensor data generated with at least one of the first sensors or a second sensor located inside the vehicle. For example, the calibration component 102 may receive the sensor data 104 generated with the sensor 106 and / or the sensor data 108 generated with the sensor 110. As described herein, the sensor data 104 may represent a sensor representation. For example, if the sensor 106 includes an image sensor, the sensor data 104 may include image data representing an image. Additionally, the sensor data 108 may represent position information indicating at least one position and / or orientation of the component 112 located inside the vehicle.

[0053] The method 800 may include, at block B806, determining a position associated with a feature located inside the vehicle based on the sensor data. For example, the calibration component 102 (e.g., the processing component 118) may use the sensor data 104 and / or the sensor data 108 to determine the position associated with the feature. For a first example, and using the sensor data 104, the calibration component 102 may determine that a portion of the sensor representation depicts the feature. The calibration component 102 may then determine the position associated with the feature as a portion of the sensor representation. For a second example, using the sensor data 108, the calibration component 102 may determine the position to be associated with the feature based on at least the sensor data 108 representing the position.

[0054] The method 800 may include, at block B808, determining one or more second values ​​for the one or more calibration parameters of the first sensor based at least on the position associated with the feature. For example, the calibration component 102 (e.g., the processing component 118) may determine the second value(s) for the calibration parameter(s) based at least on the position associated with the feature. As described herein, in some examples, the calibration component 102 may determine the second value(s) using a data structure (e.g., a table, list, tree structure, or other suitable data structure for storing and organizing information) that maps different positions of the feature to different values ​​for the calibration parameter(s).In some examples, the calibration component 102 may determine the second value(s) using one or more additional and / or alternative techniques, such as using one or more equations and / or models associated with the vehicle. As described herein, the second value(s) for the calibration parameter(s) may include a first updated value associated with the x-coordinate position, a second updated value associated with the y-coordinate position, a third updated value associated with the z-coordinate position, a fourth updated value associated with pitch, a fifth updated value associated with yaw, and / or a sixth updated value associated with roll. EXAMPLE OF AN AUTONOMOUS VEHICLE

[0055] Fig. 9A is an illustration of an exemplary autonomous vehicle 900 according to some embodiments of the present disclosure. The autonomous vehicle 900 (alternatively referred to herein as "vehicle 900") may include, without limitation, a passenger vehicle, such as a car, a truck, a bus, an emergency vehicle, a shuttle, an electric or motorized bicycle, a motorcycle, a fire engine, a police vehicle, an ambulance, a boat, a construction vehicle, an underwater vehicle, a robotic vehicle, a drone, an aircraft, a vehicle coupled to a trailer (e.g., a semi-truck used to haul cargo), and / or another type of vehicle (e.g., that is unmanned and / or accommodates one or more passengers).Autonomous vehicles are generally described in terms of automation levels defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (e.g., Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and prior and future versions of this standard). The vehicle 900 may be capable of operating in accordance with one or more of Levels 3-5 of the autonomous driving levels. The vehicle 900 may be capable of operating in accordance with one or more of Levels 1-5 of the autonomous driving levels.For example, depending on the embodiment, the vehicle 900 may be capable of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term "autonomous" as used herein may include any and / or all types of autonomy for the vehicle 900 or other machine, such as being fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing assistive autonomy, semi-autonomous, primarily autonomous, or other designations.

[0056] The vehicle 900 may include components such as the chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. The vehicle 900 may include a propulsion system 950, such as an internal combustion engine, a hybrid-electric prime mover, an all-electric motor, and / or another type of propulsion system. The propulsion system 950 may be connected to a drivetrain of the vehicle 900, which may include a transmission, to enable propulsion of the vehicle 900. The propulsion system 950 may be controlled in response to receiving signals from the accelerator pedal / throttle 952.

[0057] A steering system 954, which may include a steering wheel, may be used to steer the vehicle 900 (e.g., along a desired path or route) when the propulsion system 950 is operating (e.g., when the vehicle is moving). The steering system 954 may receive signals from a steering actuator 956. For full automation functionality (Level 5), the steering wheel may be optional.

[0058] The brake sensor system 946 may be used to actuate the vehicle brakes when signals are received from the brake actuators 948 and / or brake sensors.

[0059] Controller 936, which includes one or more System-on-Chips (SoCs) 904 ( Fig. 9C) and / or GPUs may send signals (e.g., representative of commands) to one or more components and / or systems of the vehicle 900. For example, the controller may send signals to apply the vehicle brake via one or more brake actuators 948, to operate the steering system 954 via one or more steering actuators 956, to operate the propulsion system 950 via one or more accelerator pedals / throttle levers 952. The controller 936 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and issue operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 900.Controller 936 may include a first controller 936 for autonomous driving functions, a second controller 936 for functional safety functions, a third controller 936 for artificial intelligence functionality (e.g., computer vision), a fourth controller 936 for infotainment functionality, a fifth controller 936 for emergency redundancy, and / or other controllers. In some examples, a single controller 936 may process two or more of the above functionalities, two or more controllers 936 may process a single functionality, and / or any combination thereof.

[0060] The controller(s) 936 may provide the signals to control one or more components and / or systems of the vehicle 900 in response to sensor data received from one or more sensors (e.g., sensor inputs). The sensor data may include, for example and without limitation, global navigation satellite system ("GNSS") sensors 958 (e.g., Global Positioning System sensor(s)), RADAR sensor(s) 960, ultrasonic sensor(s) 962, LIDAR sensor(s) 964, inertial measurement unit (IMU) sensor(s) 966 (e.g., accelerometer, gyroscope(s), magnetic compass(es), magnetometer, etc.), microphone(s) 996, stereo cameras 968, wide-angle cameras 970 (e.g., fisheye cameras), infrared cameras 972, environmental cameras 974 (e.g., 360-degree cameras), long-range and / or medium-range cameras 998, speed sensor(s) 944 (e.g., for measuring vehicle speed 900), vibration sensor(s) 942, Steering sensor(s) 940, brake sensor(s) (e.g.as part of the brake sensor system 946) and / or other sensor types.

[0061] One or more of the controllers 936 may receive inputs (e.g., represented by input data) from an instrument cluster 932 of the vehicle 900 and receive outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 934, an audible chime, a speaker, and / or via other components of the vehicle 900. The outputs may include information such as vehicle speed, RPM, time, mapping data (e.g., the high-resolution ("HD") map 922 of Fig. 9C), location data (e.g., the location of the vehicle 900, such as on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as perceived by the controller(s) 936, etc. For example, the HMI display 934 may display information about the presence of one or more objects (e.g., a road sign, a warning sign, a changing traffic light, etc.) and / or information about maneuvers the vehicle has performed, is performing, or will perform (e.g., that it is changing lanes now, will take exit 34B in two miles, etc.).

[0062] The vehicle 900 further includes a network interface 924 that may utilize one or more wireless antennas 926 and / or modems to communicate over one or more networks. For example, the network interface 924 may be capable of communicating over Long Term Evolution ("LTE"), Wideband Code Division Multiple Access ("WCDMA"), Universal Mobile Telecommunications System ("UMTS"), Global System for Mobile Communication ("GSM"), IMT-CDMA Multi-Carrier ("CDMA2000"), etc. The wireless antenna(s) 926 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area network(s) such as Bluetooth, Bluetooth Low Energy ("LE"), Z-Wave, ZigBee, etc. and / or low power wide-area networks (LPWANs) such as LoRaWAN, SigFox, etc.

[0063] Fig. 9B is an example of camera positions and fields of view for the example autonomous vehicle 900 of Fig. 9A, according to some embodiments of the present disclosure. The cameras and corresponding fields of view represent an example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or the cameras may be located at various positions within the vehicle 900.

[0064] The camera types for the cameras may include, but are not limited to, digital cameras that can be adapted for use with the components and / or systems of the vehicle 900. The camera(s) may be operated at automotive safety integrity level (ASIL) B and / or another ASIL. The camera types may be capable of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The cameras may be capable of using rolling shutters, global shutters, another type of shutter, or a combination thereof.In some examples, the color filter array may include a red clear clear (RCCC) color filter array, a red clear clear blue (RCCB) color filter array, a red blue green clear (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras with an RCCC, an RCCB, and / or an RBGC color filter array, may be used in an effort to increase light sensitivity.

[0065] In some examples, one or more of the cameras may be used to perform advanced driver assistance systems (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multifunction mono camera may be installed, providing functions such as lane departure warning, traffic sign assist, and intelligent headlight control. One or more of the cameras (e.g., all cameras) may simultaneously capture and provide image data (e.g., a video).

[0066] One or more of the cameras may be mounted in a mounting assembly, such as a custom-designed (three-dimensionally ("3D") printed) assembly, to eliminate stray light and reflections from inside the car (e.g., reflections from the dashboard reflected in the windshield mirrors) that can impair the camera's image data collection capabilities. With reference to side-view mirror mounting assemblies, the side-view mirror assemblies may be custom 3D printed so that the camera mounting plate conforms to the shape of the side-view mirror. In some examples, the camera(s) may be integrated into the side-view mirror. For side-view cameras, the camera(s) may also be integrated into the four pillars at each corner of the cab.

[0067] Cameras with a field of view that includes portions of the environment in front of the vehicle 900 (e.g., forward-facing cameras) can be used for surround vision to help identify forward paths and obstacles, and to help provide, with the aid of one or more controllers 936 and / or control SoCs, important information for generating an occupancy grid and / or determining preferred vehicle paths. Forward-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras can also be used for ADAS functions and systems, including lane departure warnings (LDW), autonomous cruise control (ACC), and / or other functions such as traffic sign recognition.

[0068] A variety of cameras may be used in a forward-facing configuration, including, for example, a monocular camera platform incorporating a CMOS (complementary metal oxide semiconductor) color imager. Another example is a wide-angle camera(s) 970, which may be used to perceive objects coming into view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although Fig. 9B illustrates only one wide-angle camera, there may be any number (including zero) of wide-angle cameras 970 on the vehicle 900. Additionally, any number of long-range camera(s) 998 (e.g., a long-range stereo camera pair) may be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. The long-range camera(s) 998 may also be used for object detection and classification, as well as simple object tracking.

[0069] Any number of stereo cameras 968 may also be included in a forward-facing configuration. In at least one embodiment, one or more of the stereo camera(s) 968 may include an integrated control unit comprising a scalable processing unit that may provide a programmable logic ("FPGA") and a multi-core microprocessor with an integrated Controller Area Network ("CAN") or Ethernet interface on a single chip. Such a unit may be used to generate a 3D map of the vehicle's surroundings that includes a distance estimate for all points in the image. Alternative stereo camera(s) 968 may include one or more compact stereo vision sensors that may include two camera lenses (one each on the left and right) and an image processing chip that can measure the distance from the vehicle to the target object and process the generated information (e.g.,metadata) to activate the autonomous emergency braking and lane departure warning functions. Other stereo camera types 968 may be used in addition to or alternatively to those described here.

[0070] Cameras with a field of view that includes parts of the environment on the side of the vehicle 900 (e.g., side view cameras) may be used for the surround view and provide information to generate and update the occupancy grid and generate side collision warnings. For example, surround camera(s) 974 (e.g., four surround cameras 974, as shown in Fig. 9B) may be positioned on the vehicle 900. The surround camera(s) 974 may include wide-angle cameras 970, fisheye cameras, 360-degree cameras, and / or similar cameras. For example, four fisheye cameras may be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle may utilize three surround-view cameras 974 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround-view camera.

[0071] Cameras with a field of view that includes portions of the environment at the rear of the vehicle 900 (e.g., rearview cameras) can be used for parking assistance, surround view, rear collision warning, and occupancy grid creation and updating. A variety of cameras may be used, including, but not limited to, cameras that are also suitable as front-facing cameras (e.g., long-range and / or medium-range cameras 998, stereo cameras 968, infrared cameras 972, etc.), as described herein.

[0072] Fig. 9C is a block diagram of an example system architecture for the example autonomous vehicle 900 of Fig. 9A, according to some embodiments of the present disclosure. It should be understood that these and other arrangements described herein are provided as examples only. Other arrangements and elements (e.g., engines, interfaces, functions, instructions, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Further, many of the elements described herein are functional units that may be implemented as discrete or distributed components, or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by units may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory.

[0073] All components, features and systems of the vehicle 900 in Fig. 9C are shown connected via bus 902. Bus 902 may include a Controller Area Network (CAN) data interface (alternatively referred to herein as a "CAN bus"). A CAN may be a network within vehicle 900 used to support the control of various features and functionalities of vehicle 900, such as brake application, acceleration, braking, steering, windshield wipers, etc. A CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus may be read to determine steering wheel angle, ground speed, engine revolutions per minute (RPMs), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0074] Although bus 902 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to the CAN bus. Again, although a single line is used to represent bus 902, this is not intended to be limiting. For example, there may be any number of buses 902, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other bus types using a different protocol. In some examples, two or more buses 902 may be used to perform different functions and / or to provide redundancy. For example, a first bus 902 may be used for collision avoidance and a second bus 902 may be used for control.In each example, each bus 902 may communicate with any component of the vehicle 900, and two or more buses 902 may communicate with the same components. In some examples, each SoC 904, controller 936, and / or computer in the vehicle may access the same input data (e.g., inputs from sensors of the vehicle 900) and connect to a common bus, such as the CAN bus.

[0075] The vehicle 900 may include one or more controllers 936, as described herein with respect to Fig. 9A. The controller 936 may be used for a variety of functions. The controller 936 may be coupled to any of the various other components and systems of the vehicle 900 and may be used to control the vehicle 900, the artificial intelligence of the vehicle 900, the infotainment for the vehicle 900, and / or the like.

[0076] The vehicle 900 may include a system-on-a-chip (SoC) 904. The SoC 904 may include CPUs 906, GPUs 908, processor(s) 910, caches 912, accelerators 914, data storage 916, and / or other components and functions not shown. The SoC 904 may be used to control the vehicle 900 in a variety of platforms and systems. For example, the SoC 904 may be combined in a system (e.g., the system of the vehicle 900) with an HD map 922 that may receive map updates and / or updates via a network interface 924 from one or more servers (e.g., server(s) 978 from Fig. 9D).

[0077] The CPUs 906 may include a CPU cluster or a CPU complex (alternatively referred to herein as a "CCPLEX"). The CPUs 906 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPUs 906 may include eight cores in a coherent multi-processor configuration. In some embodiments, the CPUs 906 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). The CPUs 906 (e.g., CCPLEX) may be configured to support concurrent cluster operation, so that any combination of the CPUs 906 cluster(s) may be active at any given time.

[0078] The CPUs 906 may implement power management features that include one or more of the following: individual hardware blocks may be automatically clocked when idle to conserve dynamic power; each core clock may be controlled when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core may be independently power controlled; each core cluster may be independently clocked when all cores are clock controlled or power controlled; and / or each core cluster may be independently power controlled when all cores are power controlled. The CPUs 906 may implement an advanced power state management algorithm in which allowable power states and expected wake-up times are specified, and the hardware / microcode determines the best power state for the core, cluster, and CCPLEX.The processing cores can support simplified performance status entry sequences in software, offloading the work to microcode.

[0079] The GPUs 908 may include an integrated GPU (alternatively referred to herein as a "GPU"). The GPUs 908 may be programmable and efficient for parallel workloads. In some examples, the GPUs 908 may use an extended Tensor instruction set. The GPUs 908 may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96 KB of memory capacity) and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache with a memory capacity of 512 KB). In some embodiments, the GPUs 908 may include at least eight streaming microprocessors. The GPUs 908 may use application programming interfaces (AP1). In addition, the GPUs 908 may support one or more parallel computing platforms and / or programming models (e.g.,NVIDIA CUDA).

[0080] The GPUs 908 may be performance-optimized for best performance in automotive and embedded use cases. For example, the GPUs 908 may be fabricated on a fin field-effect transistor (FinFET). However, this is not intended to be limiting, and the GPUs 908 may be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor may include a number of mixed-precision processing cores partitioned into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores could be partitioned into four processing blocks. In such an example, each processing block could be assigned 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two NVIDIA TENSOR COREs with mixed precision for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64 KB register file.Additionally, streaming microprocessors can include independent parallel integer and floating-point datapaths to enable efficient execution of workloads with a mix of computation and addressing calculations. Streaming microprocessors can include independent thread scheduling to enable finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0081] The GPUs 908 may include high-bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem to provide, in some examples, a peak memory bandwidth of approximately 900 GB / s. In some examples, in addition to or alternatively to the HBM memory, a synchronous graphics random-access memory (SGRAM) may be used, such as a graphics double data rate type five (GDDR5) synchronous random-access memory.

[0082] The GPU(s) 908 may include unified memory technology, including access processors, to enable more accurate migration of memory pages to the processor that accesses them most frequently, thereby improving efficiency for memory regions shared between processors. In some examples, Address Translation Services (ATS) support may be used to allow the GPU(s) 908 to directly access the CPU(s) 906 page tables. In such examples, an address translation request may be sent to the CPU(s) 906 when the memory management unit (MMU) of the GPU(s) 908 experiences a failure. In response, the CPU(s) 906 may look up the address in their virtual-to-physical mapping page tables and transmit the translation back to the GPU(s) 908.Therefore, unified memory technology can enable a single, unified virtual address space for the memory of the CPUs 906 and the GPUs 908, simplifying GPUs 908 programming and porting applications to the GPUs 908.

[0083] Additionally, the GPU(s) 908 may include an access counter that can track the frequency of GPU(s) 908 accessing the memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that accesses the pages most frequently.

[0084] The SoC 904 may include any number of caches 912, including those described herein. For example, the cache 912 may include an L3 cache available to both the CPU(s) 906 and the GPU(s) 908 (e.g., connected to both the CPU(s) 906 and the GPU(s) 908). The cache(s) 912 may include a write-back cache that can track the status of lines, e.g., by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4 MB or more, depending on the embodiment, although smaller cache sizes may also be used.

[0085] The SoCs 904 may include arithmetic logic units (ALUs) that may be utilized in processing related to any of the various tasks or operations of the vehicle 900—e.g., in processing DNNs. Additionally, the SoCs 904 may include floating-point units (FPUs)—or other types of mathematical coprocessors or numerical coprocessors—for performing mathematical operations within the system. For example, the SoCs 104 may include one or more FPUs integrated as execution units within a CPU 906 and / or GPU 908.

[0086] The SoC 904 may include one or more accelerators 914 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 904 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. The large on-chip memory (e.g., 4 MB SRAM) may enable a hardware acceleration cluster for accelerating neural networks and other computations. The hardware acceleration cluster may be used to supplement the GPUs 908 and offload some of the tasks of the GPUs 908 (e.g., to free up more cycles of the GPUs 908 to perform other tasks). As an example, the accelerator(s) 914 may be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are robust enough to be suitable for acceleration.The term “CNN” as used herein can include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0087] The accelerator(s) 914 (e.g., the hardware acceleration cluster) may include one or more deep learning accelerators (DLAs). The DLA(s) may include one or more tensor processing units (TPUs), which may be configured to provide an additional tens of trillion operations per second for deep learning applications and inferencing. The TPUs may be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). The DLA(s) may be further optimized for a specific set of types of neural networks and floating-point operations, as well as for inference. The design of the DLA(s) may provide more performance per millimeter than a typical general-purpose GPU, far exceeding the performance of a CPU. The TPU(s) may perform multiple functions, including a single-instance convolution function, for example,INT8, INT16 and FP16 data types are supported for both features and weights, as well as post-processor functions.

[0088] The DLA(s) can quickly and efficiently execute neural networks, in particular CNNs, on processed or unprocessed data for a variety of functions, including, for example and without limitation: a CNN for object identification and recognition using data from camera sensors; a CNN for distance estimation using data from camera sensors; a CNN for emergency vehicle detection and identification and recognition using data from microphones; a CNN for facial recognition and vehicle owner identification using data from camera sensors; and / or a CNN for safety and / or security events.

[0089] The DLA(s) can perform any function of the GPUs 908, and by using an inference accelerator, a developer can, for example, target either the DLA(s) or the GPUs 908 for each function. For example, the designer can focus the processing of CNNs and floating-point operations on the DLA(s) and leave other functions to the GPU(s) 908 and / or other accelerators 914.

[0090] The accelerator(s) 914 (e.g., the hardware acceleration cluster) may include one or more programmable vision accelerators (PVAs), which may alternatively be referred to as image processing accelerators in this document. The PVA(s) may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, augmented reality (AR), and / or virtual reality (VR) applications. The PVA(s) may provide a balance between performance and flexibility. For example, and without limitation, all PVA(s) may include any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0091] RISC cores may interact with image sensors (e.g., the image sensors of any of the cameras described herein), image signal processor(s), and / or the like. Each of the RISC cores may include any amount of memory. The RISC cores may use any of a number of protocols, depending on the embodiment. In some examples, the RISC cores may execute a real-time operating system (RTOS). The RISC cores may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores may include an instruction cache and / or tightly coupled RAM.

[0092] The DMA may enable components of the PVA(s) to access system memory independent of the CPU(s) 906. The DMA may support any number of features used to provide optimization of the PVA, including, but not limited to, support for multi-dimensional addressing and / or circular addressing. In some examples, the DMA may support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block gradation, vertical block gradation, and / or depth gradation.

[0093] The vector processors may be programmable processors that can be configured to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, DMA engine(s) (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). A VPU core may include a digital signal processor, such as a single-instruction-multiple-data (SIMD) very-long-instruction-word (VLIW) digital signal processor. The combination of SIMD and VLIW may increase throughput and speed.

[0094] Each of the vector processors may include an instruction cache and be coupled to dedicated memory. Therefore, in some examples, each of the vector processors may be configured to execute independently of the other vector processors. In other examples, vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, the plurality of vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA may simultaneously execute different computer vision algorithms on the same image or even execute different algorithms on sequential images or portions of an image.Among other things, any number of PVAs may be included in the hardware acceleration cluster, and any number of vector processors may be included in each of the PVAs. Additionally, the PVA(s) may include additional error correction code (ECC) memory to increase overall system security.

[0095] The accelerator(s) 914 (e.g., the hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM to the accelerator(s) 914. In some examples, the on-chip memory may include at least 4 MB of SRAM, consisting of, for example, and without limitation, eight field-configurable memory blocks accessible by both the PVA and the DLA. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA may access the memory via a backbone that provides high-speed access to the memory to the PVA and DLA.The backbone may include an on-chip machine vision network that interconnects the PVA and DLA with the memory (e.g., using the APB).

[0096] The on-chip computer vision network may include an interface that determines that both a PVA and a DLA are providing ready and valid signals before transmitting any control signal / address / data. Such an interface may provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transmission. This type of interface may conform to ISO 26262 or IEC 61508, although other standards and protocols may also be used.

[0097] In some examples, the SoC 904 may include a real-time hardware accelerator for ray tracing monitoring, as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the positions and extents of objects (e.g., within a world model) to generate real-time visualization simulations, for radar signal interpretation, for sound propagation synthesis and / or analysis, for simulation of sonar systems, for general wave propagation simulation, for comparison with lidar data for localization, and / or for other functions, and / or for other applications. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0098] The accelerator(s) 914 (e.g., the hardware accelerator cluster) have a variety of applications for autonomous driving. The PVA can be a programmable vision accelerator that can be used for critical processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good complement for algorithmic domains that require predictable processing with low power and low latency. In other words, the PVA demonstrates good performance for semi-dense or dense regular computations, even on small datasets that require predictable runtimes with low latency and low power. Consequently, in the context of autonomous vehicle platforms, the PVAs are designed to execute classical computer vision algorithms because they are efficient at object detection and operate with integer mathematics.

[0099] For example, according to one embodiment of the technology, the PVA is used to perform machine stereo vision. An algorithm based on semi-global matching may be used, although this is not a limitation. Many Level 3-5 autonomous driving applications require motion estimation / stereo matching on the fly (e.g., structure from motion, pedestrian detection, lane detection, etc.). The PVA can perform a machine stereo vision function on inputs from two monocular cameras.

[0100] In some examples, the PVA can be used to perform dense optical flow, such as processing raw radar data (e.g., using a 4D Fast Fourier Transform) to provide processed radar. In other examples, the PVA is used for runtime deep processing, such as processing raw runtime data to provide processed runtime data.

[0101] The DLA can be used to run any type of network to improve control and driving safety, including, for example, a neural network that outputs a confidence level for each object detection. Such a confidence level can be interpreted as a probability or as providing a relative "weight" to each detection compared to other detections. The confidence level allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only detections that exceed the threshold as true positives.In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform emergency braking, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. The DLA can run a neural network to regress the confidence score. The neural network can take as input at least a subset of parameters, such as bounding box dimensions, ground plane estimation (e.g., from another subsystem), Inertial Measurement Unit (IMU) sensor 966 output related to the orientation of the vehicle 900, distance, 3D position estimates of the object obtained from the neural network and / or other sensors (e.g., LIDAR sensor(s) 964 or RADAR sensor(s) 960), among others.

[0102] The SoC 904 may include data storage 916 (e.g., memory). The data storage(s) 916 may be on-chip memory of the SoC 904, which may store neural networks executing on the GPU and / or the DLA. In some examples, the data storage 916 may be large enough to store multiple neural network instances for redundancy and security. The data storage 912 may include L2 or L3 caches 912. The reference to the data storage 916 may include the reference to the memory of the PVA, DLA, and / or other accelerators 914, as described herein.

[0103] The SoC 904 may include one or more processors 910 (e.g., embedded processors). The processors 910 may include a boot and power management processor, which may be a dedicated processor and subsystem to handle boot power and management functions and associated security enforcement. The boot and power management processor may be part of the SoC 904 boot sequence and may provide runtime power management services. The boot power and management processor may provide clock frequency and voltage programming, support for system power-saving state transitions, management of SoC 904 thermals and temperature sensors, and / or management of SoC 904 power states.Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 904 can use the ring oscillators to detect temperatures of CPUs 906, GPUs 908, and / or accelerators 914. If temperatures exceed a threshold, the boot and power management processor can enter a temperature fault routine and place the SoC 904 into a lower power state and / or place the vehicle 900 into a safe stop mode (e.g., bring the vehicle 900 to a safe stop).

[0104] Processors 910 may also include a set of integrated processors that can serve as an audio processor. The audio processing engine may be an audio subsystem that enables full hardware support for multi-channel audio across multiple interfaces, as well as a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

[0105] The processor(s) 910 may also be equipped with an "always-on" processor that provides the necessary hardware features to support low-power sensor management and power-on use cases. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0106] The processor(s) 910 may also include a safety cluster engine, which includes a dedicated processor subsystem for safety management for automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, an interrupt controller, etc.), and / or routing logic. In a safety mode, the two or more cores may operate in a lockstep mode, functioning as a single core with comparison logic to detect any differences between their operations.

[0107] The processor(s) 910 may also include a real-time camera engine, which may include a dedicated processor subsystem for real-time camera management.

[0108] The processor(s) 910 may also include a high dynamic range signal processor including an image signal processor that is part of the camera processing pipeline as a hardware engine.

[0109] The processor(s) 910 may include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate the final image for the player window. The video image compositor may perform lens distortion correction on wide-angle cameras 970, surround cameras 974, and / or on in-cabin surveillance camera sensors. The in-cabin surveillance camera sensor is preferably monitored by a neural network running on another instance of the Advanced SoC and configured to detect and respond to in-cabin events.An in-cab system can perform lip reading to activate cellular service and place a call, dictate emails, change a vehicle destination, activate or change a vehicle's infotainment system and its settings, or provide voice-activated web browsing. Certain functions are available to the driver only when the vehicle is operating in autonomous mode and are disabled otherwise.

[0110] The video compositor can incorporate advanced temporal noise reduction for both spatial and temporal noise reduction. For example, if motion is present in a video, the noise reduction appropriately weights the spatial information by reducing the weight of the information provided by neighboring frames. If an image or a portion of an image contains no motion, the temporal noise reduction performed by the video compositor can use information from the previous image to suppress noise in a current image.

[0111] The video image compositor can also be configured to perform stereo rectification on stereo input lens frames. The video image compositor can also be used for user interface creation when the operating system desktop is in use and the GPU 908 is not required to continuously render new surfaces. Even when the GPU 908 is turned on and actively performing 3D rendering, the video image compositor can be used to offload the GPU 908 to improve performance and responsiveness.

[0112] The SoC 904 may also include a mobile industry processor interface (MIPI) serial interface for receiving video and input from cameras, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions. The SoC 904 may also include a software-controlled input / output controller that can be used to receive I / O signals that are not assigned to a specific role.

[0113] The SoC 904 may further include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management, and / or other devices. The SoC 904 may be used to process data from cameras (e.g., connected via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LIDAR sensor(s) 964, RADAR sensor(s) 960, etc., which may be connected via Ethernet), data from bus 902 (e.g., speed of vehicle 900, steering wheel position, etc.), and data from GNSS sensor(s) 958 (e.g., connected via Ethernet or CAN bus). The SoC(s) 904 may also include dedicated high-performance mass storage controllers, which may include their own DMA engines and which may be used to free the CPU(s) 906 from routine data management tasks.

[0114] The SoC 904 can be an end-to-end platform with a flexible architecture spanning automation levels 3 through 5, thus providing a comprehensive functional safety architecture that efficiently leverages computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack, along with deep learning tools. The SoC 904 can be faster, more reliable, and even more power- and space-efficient than conventional systems. Thus, the accelerator 914, in combination with the CPU 906, the GPU 908, and the data memory 916, can provide a fast, efficient platform for Level 3-5 autonomous vehicles.

[0115] The technology thus offers capabilities and functionality that cannot be achieved with conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using a high-level programming language, such as the C programming language, to perform a wide variety of processing algorithms on a wide variety of visual data. However, these CPUs are often unable to meet the performance requirements of many computer vision applications, such as execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real time, which are required in in-vehicle ADAS applications and practical Level 3-5 autonomous vehicles.

[0116] In contrast to conventional systems, the technology described here, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, enables the simultaneous and / or sequential execution of multiple neural networks and the combination of the results to enable Level 3-5 functionality for autonomous driving. For example, a CNN running on the DLA or dGPU (e.g., the GPU 920) may include text and word recognition, allowing the supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing semantic understanding of signs, and passing this semantic understanding to the path planning modules running on the CPU complex.

[0117] As another example, multiple neural networks can be executed simultaneously, as required for driving at Level 3, 4, or 5. For example, a warning sign reading "Caution: Flashing lights indicate icing," along with an electric light, can be interpreted independently or jointly by multiple neural networks. The sign itself can be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing lights indicate refrain" can be interpreted by a second deployed neural network, which informs the vehicle's path-planning software (preferably running on the CPU complex) that if flashing lights are detected, icing is present.The flashing light can be identified by running a third deployed neural network over multiple frames, informing the vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks can be executed simultaneously, e.g., within the DLA and / or on the GPU 908.

[0118] In some examples, a CNN for facial recognition and vehicle identification may use data from camera sensors to identify the presence of an authorized driver and / or vehicle owner 900. The always-on sensor processing engine may be used to unlock the vehicle when the owner approaches the driver's door and turns on lights, and to disable the vehicle in secure mode when the owner exits the vehicle. In this way, the SoCs 904 provide security against pilferage and / or theft.

[0119] In another example, a CNN for emergency vehicle detection and identification may use data from microphones 996 to detect and identify emergency vehicle sirens. Unlike conventional systems that use general classifiers to detect sirens and manually extract features, the SoCs 904 use CNNs to classify environmental and urban sounds, as well as to classify visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative approach speed of the emergency vehicle (e.g., by using the Doppler effect). The CNN may also be trained to identify emergency vehicles specific to the vehicle's local area of ​​operation, as identified by GNSS sensor(s) 958.Consequently, for example, when operating in Europe, the CNN attempts to detect European sirens, and in the United States, the CNN attempts to identify only North American sirens. Once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine, slow the vehicle, pull over to the side of the road, park the vehicle, and / or idle the vehicle using ultrasonic sensors 962 until the emergency vehicle passes.

[0120] The vehicle may include a CPU 918 (e.g., discrete CPU or dCPU) that may be coupled to the SoC 904 via a high-speed connection (e.g., PCIe). The CPU 918 may include, for example, an x86 processor. The CPU 918 may be used to perform a variety of functions, including, for example, assessing potentially inconsistent results between ADAS sensors and the SoC 904 and / or monitoring the status and health of the controllers 936 and / or the infotainment SoC 930.

[0121] The vehicle 900 may include a GPU 920 (e.g., discrete GPU or dGPU) that may be coupled to the SoC 904 via a high-speed interconnect (e.g., NVIDIA's NVLINK). The GPU 920 may provide additional artificial intelligence capabilities, e.g., by executing redundant and / or different neural networks, and may be used to train and / or update neural networks based on inputs (e.g., sensor data) from sensors of the vehicle 900.

[0122] Vehicle 900 may also include network interface 924, which may include one or more radio antennas 926 (e.g., one or more radio antennas for various communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). Network interface 924 may be used to enable wireless connections over the internet to the cloud (e.g., to server 978 and / or other network devices), to other vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other vehicles, a direct link may be established between the two vehicles and / or an indirect link (e.g., via networks and over the internet) may be established. Direct links may be established using a vehicle-to-vehicle communication link.The vehicle-to-vehicle communication link may provide information to the vehicle 900 about vehicles in the vicinity of the vehicle 900 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 900). This function may be part of a cooperative adaptive cruise control function of the vehicle 900.

[0123] The network interface 924 may include an SoC that provides modulation and demodulation and enables the controller 936 to communicate over wireless networks. The network interface 924 may include a radio frequency front-end for upconversion from baseband to radio frequency and downconversion from radio frequency to baseband. The frequency conversions may be performed by well-known processes and / or using overlay techniques. In some examples, the radio frequency front-end functionality may be provided by a separate chip. The network interface may include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0124] Vehicle 900 may also include data storage 928, which may also include off-chip memory (e.g., off-the-SoC 904). Data storage 928 may include one or more memory elements, including RAM, SRAM, DRAM, VRAM, flash, hard drives, and / or other components and / or devices capable of storing at least one bit of data.

[0125] The vehicle 900 may also include one or more GNSS sensors 958. The GNSS sensors 958 (e.g., GPS, assisted GPS sensors, differential GPS (DGPS), etc.) assist in mapping, sensing, occupancy grid generation, and / or path planning. Any number of GNSS sensors 958 may be used, including, for example, a GPS device using a USB port with an Ethernet-to-serial (RS-232) bridge.

[0126] The vehicle 900 may also include one or more RADAR sensors 960. The RADAR sensor 960 may be used by the vehicle 900 for remote vehicle detection, even in darkness and / or severe weather conditions. The functional safety levels of the RADAR may be ASIL B. The RADAR sensor 960 may use the CAN and / or bus 902 (e.g., to transmit the data generated by the RADAR sensor 960) for control and to access object tracking data, with Ethernet being accessible in some examples to access raw data. A wide variety of RADAR sensor types may be used. For example, the RADAR sensors 960 may be used without restriction as front, rear, and side RADAR. In some examples, pulse-Doppler RADAR sensors are used.

[0127] The RADAR sensors 960 can include various configurations, such as long range with a narrow field of view, short range with a wide field of view, short side coverage, etc. In some examples, a long-range RADAR can be used for adaptive cruise control. The long-range RADAR systems can provide a wide field of view realized by two or more independent scans, e.g., within a range of 250 m. The RADAR sensors 960 can assist in distinguishing between static and moving objects and can be used by ADAS systems for emergency braking assistance and collision warning. Long-range RADAR sensors can include a monostatic multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and a high-speed CAN and FlexRay interface.In a six-antenna example, the central four antennas can create a focused beam pattern designed to record the surroundings of vehicle 900 at higher speeds with minimal interference from traffic in adjacent lanes. The other two antennas can expand the field of view, making it possible to quickly detect vehicles entering or exiting the path of vehicle 900.

[0128] Mid-range radar systems, for example, can have a range of up to 960 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 950 degrees (rear). Short-range radar systems can, without limitation, include radar sensors designed for installation at both ends of the rear bumper. When installed at both ends of the rear bumper, such a radar sensor system can generate two beams that constantly monitor the blind spot behind and to the side of the vehicle.

[0129] Short-range radar systems can be used in an ADAS system for blind spot detection and / or lane change assistance.

[0130] The vehicle 900 may also include ultrasonic sensors 962. The ultrasonic sensors 962, which may be located at the front, rear, and / or sides of the vehicle 900, may be used for parking and / or for creating and updating an occupancy grid. A plurality of ultrasonic sensors 962 and different ultrasonic sensors 962 for different detection ranges (e.g., 2.5 m, 4 m) may be used. The ultrasonic sensor(s) 962 may operate at functional safety levels of ASIL B.

[0131] The vehicle 900 may include LIDAR sensors 964. The LIDAR sensors 964 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensors 964 may be ASIL B functional safety level. In some examples, the vehicle 900 may include multiple LIDAR sensors 964 (e.g., two, four, six, etc.) that may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0132] In some examples, the LIDAR sensors 964 may provide a list of objects and their distances for a 360-degree field of view. Commercially available LIDAR sensors 964 may have an advertised range of approximately 900 m, an accuracy of 2 cm to 3 cm, and support for, for example, a 900 Mbps Ethernet connection. In some examples, one or more non-prominent LIDAR sensors 964 may be used. In such examples, the LIDAR sensors 964 may be implemented as a small device that may be embedded in the front, rear, sides, and / or corners of the vehicle 900. In such examples, the LIDAR sensor(s) 964 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees with a range of 200 m even for low reflectivity objects.Front-mounted LIDAR sensors 964 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0133] In some examples, LIDAR technologies, such as 3D flash LIDAR, may also be used. 3D flash LIDAR uses a laser flash as the transmitting source to illuminate the vehicle's surroundings up to a distance of approximately 200 m. A flash LIDAR unit includes a receptor that records the laser pulse time of flight and the reflected light at each pixel, which in turn corresponds to the range from the vehicle to the objects. Flash LIDAR can enable highly accurate and distortion-free images of the surroundings to be generated with each laser flash. In some examples, four LIDAR flash sensors may be deployed, one on each side of the vehicle. Available 3D flash LIDAR systems include a solid-state 3D staring array LIDAR camera with no moving parts other than a fan (e.g., a non-scanning LIDAR device).The flash lidar device can use a 5-nanosecond Class I (eye-safe) laser pulse per image and can capture the reflected laser light as 3D range point clouds and co-registered intensity data. By using flash lidar and being a solid-state device with no moving parts, the 964 lidar sensor may be less susceptible to motion blur, vibration, and / or shock.

[0134] The vehicle may also include IMU sensors 966. In some examples, the IMU sensors 966 may be located at the center of the rear axle of the vehicle 900. For example, the IMU sensors 966 may include one or more accelerometers, magnetometers, gyroscopes, magnetic compasses, and / or other sensor types. In some examples, such as in six-axis applications, the IMU sensors 966 may include accelerometers and gyroscopes, while in nine-axis applications, the IMU sensors 966 may include accelerometers, gyroscopes, and magnetometers.

[0135] In some embodiments, the IMU sensors 966 can be implemented as a miniature GPS-Aided Inertial Navigation System (GPS / INS) that combines micro-electro-mechanical systems (MEMS) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filter algorithms to provide estimates of position, velocity, and inclination. In some examples, the IMU sensors 966 can enable the vehicle 900 to estimate heading without magnetic sensor input by observing and correlating velocity changes from the GPS directly on the IMU sensors 966. In some examples, the IMU sensors 966 and GNSS sensors 958 can be combined into a single integrated unit.

[0136] The vehicle may be equipped with microphones 996 placed in and / or around the vehicle 900. The microphones 996 may be used, among other things, for detecting and identifying emergency vehicles.

[0137] The vehicle may also include any number of camera types, including stereo cameras 968, wide-angle cameras 970, infrared cameras 972, surround cameras 974, long-range and medium-range cameras 998, and / or other camera types. The cameras may capture image data around an entire periphery of the vehicle 900. The camera types used depend on the embodiments and requirements for the vehicle 900, and any combination of camera types may be used to provide the required coverage around the vehicle 900. Additionally, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may support, for example and without limitation, Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet.Each camera is described in this document in terms of . Fig. 9A and Fig. 9B.

[0138] The vehicle 900 may also include vibration sensors 942. The vibration sensors 942 may measure vibrations from components of the vehicle, such as the axles. For example, changes in vibrations may indicate a change in the road surface. In another example, when two or more vibration sensors 942 are used, the differences between the vibrations may be used to determine the friction or slippage of the road surface (e.g., if the vibration difference is between an electrically driven and a freely rotating axle).

[0139] The vehicle 900 may include an ADAS system 938. The ADAS system 938 may include an SoC in some examples. The ADAS system 938 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning (CWS), lane centering (LC), and / or other features and functionality.

[0140] The ACC systems may use RADAR sensors 960, LIDAR sensors 964, and / or cameras. The ACC systems may include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of the vehicle 900 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicles ahead. Lateral ACC performs the distance measurement and advises the vehicle 900 to change lanes if necessary. Lateral ACC is connected to other ADAS applications, such as LCA and CWS.

[0141] CACC utilizes information from other vehicles, which may be received via the network interface 924 and / or the wireless antenna(s) 926 from other vehicles over a wireless connection or indirectly via a network connection (e.g., over the Internet). Direct links may be provided by a vehicle-to-vehicle (V2V) communication link, while indirect links may be provided by an infrastructure-to-vehicle (I2V) communication link. The V2V communication concept typically provides information about the immediately preceding vehicles (e.g., vehicles immediately in front of and in the same lane as vehicle 900), while the I2V communication concept provides information about additional traffic ahead. CACC systems may incorporate either or both I2V and V2V information sources.Given the information about the vehicles ahead of vehicle 900, CACC could be more reliable and has the potential to improve traffic flows and reduce congestion on the road.

[0142] FCW systems are designed to alert the driver to a hazard so the driver can take corrective action. FCW systems utilize a forward-facing camera and / or radar sensors 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback, such as a display, speaker, and / or vibrating component. FCW systems can provide a warning, such as a sound, a visual warning, a vibration, and / or a rapid braking pulse.

[0143] AEB systems detect an impending collision with another vehicle or object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. AEB systems can use forward-facing cameras and / or radar sensors 960 coupled with a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision. If the driver does not take corrective action, the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the effects of the predicted collision. AEB systems can incorporate techniques such as dynamic brake support and / or impending collision braking.

[0144] LDW systems provide visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle 900 crosses lane markings. An LDW system will not activate if the driver indicates an intentional lane departure by activating the turn signal. LDW systems may utilize forward-facing cameras coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker, and / or vibrating component.

[0145] LKA systems are a variant of LDW systems. LKA systems provide steering inputs or braking to correct the vehicle 900 when the vehicle 900 leaves the roadway.

[0146] BSW systems detect and warn the driver of vehicles in an automobile's blind spot. BSW systems may provide a visual, audible, and / or tactile alert to indicate that merging or changing lanes is unsafe. The system may provide an additional warning when the driver activates a turn signal. BSW systems may utilize rear- and side-facing camera(s) and / or radar sensor(s) 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, i.e., electrically coupled to a driver feedback device, such as a display, speaker, and / or vibrating component.

[0147] RCTW systems can provide visual, audible, and / or tactile notifications when an object is detected outside the range of the rearview camera when the vehicle 900 is reversing. Some RCTW systems include the AEB system to ensure the vehicle brakes are applied to avoid a collision. RCTW systems can utilize one or more rear-facing RADAR sensors 960 coupled with a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibrating component.

[0148] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for the driver, but are typically not catastrophic because the ADAS systems alert the driver and allow the driver to decide whether a safety condition actually exists and act accordingly. However, in an autonomous vehicle 900, when conflicting results occur, the vehicle 900 itself must decide whether to consider the result of a primary or secondary computer (e.g., a first controller 936 or a second controller 936). In some embodiments, for example, the ADAS system 938 may be a backup and / or secondary computer to provide perception information to a rationality module of the backup computer.The backup computer's rationality monitor can run redundant, diverse software on hardware components to detect errors in perception and dynamic driving tasks. Outputs from the ADAS system 938 can be provided to a monitoring MCU. If the outputs of the primary and secondary computers conflict, the monitoring MCU must determine how to resolve the conflict to ensure safe operation.

[0149] In some examples, the primary computer may be configured to provide the monitoring MCU with a confidence score indicating the primary computer's confidence in the selected outcome. If the confidence score exceeds a threshold, that monitoring MCU may follow the primary computer's lead, regardless of whether that secondary computer provides a contradictory or inconsistent result. If the confidence score does not meet the threshold and the primary and secondary computers indicate different results (e.g., a contradiction), the monitoring MCU may mediate between the computers to determine the appropriate outcome.

[0150] The monitoring MCU may be configured to run neural network(s) trained and configured to determine, based at least on outputs from the primary computer and the secondary computer, the conditions under which the secondary computer provides false alarms. Consequently, the neural network(s) in the monitoring MCU can learn when the output of a secondary computer can and cannot be trusted. For example, if the secondary computer is a radar-based FCW system, neural network(s) in the monitoring MCU can learn when the FCW system identifies metallic objects that are not actually hazards, such as a drain grate or manhole cover, which triggers an alarm.Similarly, if the secondary computer is a camera-based LDW system, a neural network in the monitoring MCU can learn to override the LDW when cyclists or pedestrians are present and lane departure is actually the safest maneuver. In embodiments including one or more neural networks running on the control MCU, the control MCU can include at least one DLA or GPU suitable for executing the neural network(s) with associated memory. In preferred embodiments, the monitoring MCU can be included and / or incorporated as part of the SoC 904.

[0151] In other examples, the ADAS system 938 may include a secondary computer that executes the ADAS functionality using conventional computer vision rules. Thus, the secondary computer may use classic computer vision rules (if-then), and the presence of a neural network(s) in the monitoring MCU may improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity makes the overall system more fault-tolerant, particularly against errors caused by software (or software-hardware interface) functionality.For example, if there is a software bug or error in the software running on the primary computer and the non-identical software code running on the secondary computer provides the same overall result, the monitoring MCU can have greater confidence that the overall result is correct and the bug in the software or hardware on the primary computer is not causing a significant failure.

[0152] In some examples, the output of the ADAS system 938 may be fed into the perception block of the primary computer and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 938 displays a collision warning due to an object immediately ahead, the perception block may use this information in identifying objects. In other examples, the secondary computer may have its own neural network trained, thus reducing the risk of false positives, as described herein.

[0153] The vehicle 900 may also include the infotainment SoC 930 (e.g., an in-vehicle infotainment (IVI) system). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 930 may include a combination of hardware and software that can be used to provide the vehicle 900 with audio (e.g., music, a personal digital assistant, navigation directions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation systems, rear parking assist, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fluid level, oil level, door open / closed, air filter information, etc.).For example, the infotainment SoC 930 may include radios, CD players, navigation systems, video players, USB and Bluetooth connectivity, car computers, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, a heads-up display (HUD), an HMI display 934, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, functions, and / or systems), and / or other components. The infotainment SoC 930 may also be used to provide information (e.g., visual and / or audible) to users of the vehicle, such as information from the ADAS system 938, autonomous driving information such as planned vehicle maneuvers, trajectories, environmental data (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0154] The infotainment SoC 930 may include GPU functionality. The infotainment SoC 930 may communicate with other devices, systems, and / or components of the vehicle 900 via the bus 902 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 930 may be coupled to a higher-level MCU so that the infotainment system's GPU can perform some self-driving functions if the primary controller 936 (e.g., the primary and / or backup computer of the vehicle 900) fails. In such an example, the infotainment SoC 930 may place the vehicle 900 into safe stop mode, as described herein.

[0155] The vehicle 900 may also include an instrument cluster 932 (e.g., a digital instrument panel, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 932 may include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 932 may include a set of instruments such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn signals, shift position indicator, seat belt warning light(s), parking brake warning light(s), engine malfunction light(s), airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 930 and the instrument cluster 932. In other words, the instrument cluster 932 may be included as part of the infotainment SoC 930 or vice versa.

[0156] Fig. 9D is a system diagram for communication between cloud-based servers and the example autonomous vehicle 900 of Fig. 9A, according to some embodiments of the present disclosure. System 976 may include servers 978, networks 990, and vehicles including vehicle 900. Servers 978 may include a plurality of GPUs 984(A)-984(H) (collectively, GPUs 984), PCIe switches 982(A)-982(H) (collectively, PCIe switches 982), and / or CPUs 980(A)-980(B) (collectively, CPUs 980). GPUs 984, CPUs 980, and PCIe switches may be connected using high-speed interconnects such as NVLink interfaces 988 and / or PCIe connections 986 developed by NVIDIA. In some examples, the 984 GPUs are connected via NVLink and / or NVSwitch SoC, and the 984 GPUs and the 982 PCIe switches are connected via PCIe connections. Although eight 984 GPUs, two 980 CPUs, and two PCIe switches are shown, this is not intended to be limiting.Depending on the embodiment, each server 978 may include any number of GPUs 984, CPUs 980, and / or PCIe switches. For example, the servers 978 may each include eight, sixteen, thirty-two, and / or more GPUs 984.

[0157] The server(s) 978 may receive, via the network(s) 990 and from the vehicles, image data representative of images depicting unexpected or changing road conditions, such as recently commenced roadwork. The server(s) 978 may transmit, via the network(s) 990 and to the vehicles, neural networks 992, updated neural networks 992, and / or map information 994, including information about traffic and road conditions. The updates to the map information 994 may include updates to the HD map 922, such as information about construction, potholes, detours, flooding, and / or other obstacles.In some examples, the neural networks 992, the updated neural networks 992, and / or the map information 994 may result from new training and / or experience represented in data from any number of vehicles in the environment, and / or may be based on training performed in a data center (e.g., using the server(s) 978 and / or other servers).

[0158] The servers 978 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated by vehicles and / or can be generated in a simulation (e.g., using a game engine). In some examples, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not tagged and / or preprocessed (e.g., if the neural network does not require supervised learning).Training may be performed using one or more classes of machine learning techniques, including, but not limited to, supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including replacement dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. After training the machine learning models, the machine learning models may be used by the vehicles (e.g., transmitted to the vehicles via the network 990) and / or the machine learning models may be used by the servers 978 to remotely monitor the vehicles.

[0159] In some examples, the server(s) 978 may receive data from the vehicles and apply the data to real-time neural networks to enable intelligent real-time inference. The servers 978 may include deep learning supercomputers and / or dedicated AI computers with GPUs 984, such as DGX and DGX Station machines developed by NVIDIA. However, in some examples, the servers 978 may include a deep learning infrastructure using only CPU-powered data centers.

[0160] The deep learning infrastructure of servers 978 can enable fast, real-time inference and use this capability to evaluate and verify the health of the processors, software, and / or associated hardware in vehicle 900. For example, the deep learning infrastructure can receive regular updates from vehicle 900, such as a sequence of images and / or objects that vehicle 900 has located within that sequence of images (e.g., through computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify the objects and compare them to the objects identified by vehicle 900.If the results do not match and the infrastructure concludes that the AI ​​in the vehicle 900 is defective, the server 978 can send a signal to the vehicle 900 instructing a fail-safe computer of the vehicle 900 to take control, notify the passengers, and perform a safe parking maneuver.

[0161] For inference, servers 978 may include GPUs 984 and one or more programmable inference accelerators (e.g., NVIDIA TensorRT). The combination of GPU-powered servers and inference acceleration can enable real-time responsiveness. In other examples, such as when performance is less critical, servers powered by CPUs, FPGAs, and other processors can be used for inference. EXAMPLE CALCULATION DEVICE

[0162] Fig. 10 is a block diagram of exemplary computing device(s) 1000 suitable for use in implementing some embodiments of the present disclosure. Computing device 1000 may include an interconnect system 1002 that directly or indirectly couples the following devices: memory 1004, one or more central processing units (CPUs) 1006, one or more graphics processing units (GPUs) 1008, a communications interface 1010, input / output (I / O) ports 1012, input / output components 1014, a power supply 1016, and one or more presentation components 1018 (e.g., displays), and one or more logic units 1020. In at least one embodiment, computing device(s) 1000 may include one or more virtual machines (VMs) and / or each of its components may include virtual components (e.g., virtual hardware components).By way of non-limiting examples, one or more GPUs 1008 may include one or more vGPUs, one or more CPUs 1006 may include one or more vCPUs, and / or one or more logic units 1020 may include one or more virtual logic units. Thus, the computing device(s) 1000 may include discrete components (e.g., a full GPU associated with the computing device 1000), virtual components (e.g., a portion of a GPU associated with the computing device 1000), or a combination thereof.

[0163] Although the different blocks of Fig. 10 as being connected to wires via interconnect system 1002, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 1018, such as a display device, may be considered an I / O component 1014 (e.g., if the display is a touchscreen). As another example, the CPUs 1006 and / or GPUs 1008 may include memory (e.g., memory 1004 may be representative of a storage device in addition to the memory of the GPUs 1008, the CPUs 1006, and / or other components). In other words, the computing device of Fig. 10 is merely illustrative. No distinction is made between categories such as "workstation", "server", "laptop", "desktop", "tablet", "client device", "mobile device", "handheld device", "game console", "electronic control unit (ECU)", "virtual reality system" and / or other device or system types, since all components of the computing device Fig. 10 lie.

[0164] The interconnect system 1002 may represent one or more connections or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 1002 may include one or more bus or connection types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another bus or connection type. In some embodiments, there are direct connections between components. As an example, the CPU 1006 may be directly connected to the memory 1004. Additionally, the CPU 1006 may be directly connected to the GPU 1008. For direct or point-to-point connections between components, the interconnect system 1002 may include a PCIe connection to perform the connection.In these examples, no PCI bus needs to be included in the computing device 1000.

[0165] Memory 1004 may include a variety of computer-readable media. The computer-readable media may be any available media accessible by computing device 1000. The computer-readable media may include both volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable media may include computer storage media and communication media.

[0166] The computer storage medium may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, computer-readable instructions (e.g., representing one or more programs and / or program elements, such as an operating system) may be stored in memory 1004. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disks (DVDs) or other optical media, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by computing device 1000.As used herein, computer storage media does not include signals per se.

[0167] The computer storage media may embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and include any information delivery media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics altered in such a way that information is encoded in the signal. For example, and not as a limitation, computer storage media may include wired media, such as a wired network or a wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of all of the foregoing are also intended to be included within the scope of computer-readable media.

[0168] The CPUs 1006 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the methods and / or processes described herein. The CPUs 1006 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a plurality of software threads concurrently. The CPUs 1006 may include any type of processor and may include different types of processors depending on the type of computing device 1000 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). Depending on the computing device 1000, it may be, for example, an Advanced RISC Machine (ARM) processor with reduced instruction set computing (RISC) or an x86 processor with complex instruction set computing (CISC).The computing device 1000 may include one or more CPUs 1006 in addition to one or more microprocessors or additional coprocessors, such as math coprocessors.

[0169] The GPUs 1008 may furthermore or alternatively be configured by the CPUs 1006 to execute at least some computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the methods and / or processes described herein. One or more GPUs 1008 may be an integrated GPU (e.g., with one or more CPUs 1006) and / or one or more GPUs 1008 may be a separate GPU. In embodiments, one or more GPUs 1008 may be a coprocessor of one or more CPUs 1006. The GPUs 1008 may be used by the computing device 1000 to render graphics (e.g., 3D graphics) or perform general-purpose computations. For example, the GPUs 1008 may be used for general-purpose computing on GPUs (GPGPU). The 1008 GPUs can contain hundreds or thousands of cores that can process hundreds or thousands of software threads simultaneously.The GPUs 1008 may generate pixel data for output images in response to rendering commands (e.g., rendering commands from the CPUs 1006 received via a host interface). The GPUs 1008 may include graphics memory, such as display memory, for storing pixel data or other suitable data, such as GPGPU data. The display memory may be included as part of the memory 1004. The GPUs 1008 may include two or more GPUs operating in parallel (e.g., via an interconnect). The interconnect may connect the GPUs directly (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined together, each GPU 1008 may generate pixel data or GPGPU data for different parts of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image).Each GPU can contain its own memory or can share memory with other GPUs.

[0170] The logic units 1020 may furthermore or alternatively be configured by the CPUs 1006 and / or the GPUs 1008 to execute at least some computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the methods and / or processes described herein. In embodiments, the CPUs 1006, the GPUs 1008, and / or the logic units 1020 may discretely or jointly perform any combination of the methods, processes, and / or portions thereof. One or more of the logic units 1020 may be part of one or more of the CPUs 1006 and / or GPUs 1008, and / or one or more of the logic units 1020 may be discrete components or otherwise external to the CPUs 1006 and / or the GPUs 1008. In embodiments, one or more of the logic units 1020 may be a coprocessor of one or more of the CPUs 1006 and / or one or more of the GPUs 1008.

[0171] Examples of the logic unit(s) 1020 include one or more processing cores and / or components thereof, such as data processing units (DPUs), tensor cores (TC), tensor processing units (TPU), pixel visual cores (PVC), vision processing units (VPU), graphics processing clusters (GPC), texture processing clusters (TPC), streaming multiprocessors (SM), tree traversal units (TTU), artificial intelligence accelerators (AIA), deep learning accelerators (DLA), arithmetic logic units (ALU), application-specific integrated circuits (ASIC), floating-point units (FPU), input / output (I / O) elements,Elements for interconnecting peripheral components (PCI) or express interconnecting peripheral components (peripheral component interconnect express - PCIe) and / or the like.

[0172] The communication interface 1010 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1000 to communicate with other computing devices over an electronic communication network, including wired and / or wireless communication. The communication interface 1010 may include components and functions that enable communication over a variety of networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, Sigfox, etc.), and / or the Internet.In one or more embodiments, the logic unit(s) 1020 and / or the communication interface 1010 may include one or more data processing units (DPUs) to transfer data received over a network and / or through the interconnect system 1002 directly to one or more GPU(s) 1008 (e.g., a memory thereof).

[0173] The I / O ports 1012 may enable the computing device 1000 to be logically coupled to other devices, including the I / O components 1014, the presentation component(s) 1018, and / or other components, some of which may be built into (e.g., integrated) the computing device 1000. Illustrative I / O components 1014 include a microphone, a mouse, a keyboard, a joystick, a gamepad, a game controller, a satellite dish, a scanner, a printer, a wireless device, etc. The I / O components 1014 may provide a natural user interface (NUI) that processes air gestures, voice input, or other physiological inputs generated by a user. In some cases, inputs may be transmitted to a suitable network element for further processing.A NUI may implement any combination of speech recognition, tactile recognition, facial recognition, biometric recognition, both on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch detection (as further described below) connected to a display of the computing device 1000. The computing device 1000 may include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. In addition, the computing device 1000 may include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable motion detection. In some examples, the output of the accelerometers or gyroscopes from the computing device 1000 may be used to present immersive augmented reality or virtual reality.

[0174] The power supply 1016 may comprise a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1016 may supply power to the computing device 1000 so that the components of the computing device 1000 can function.

[0175] The presentation components 1018 may include a display (e.g., a monitor, a touchscreen, a television screen, a heads-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation components 1018 may receive data from other components (e.g., GPUs 1008, CPUs 1006, DPUs, etc.) and output the data (e.g., as an image, video, audio, etc.). EXEMPLARY DATA CENTER

[0176] Fig. 11 illustrates an example data center 1100 that may be used in at least one embodiment of the present disclosure. Data center 1100 may include a data center infrastructure layer 1110, a framework layer 1120, a software layer 1130, and / or an application layer 1140.

[0177] As in Fig. 11, the data center infrastructure layer 1110 may include a resource orchestrator 1112, clustered compute resources 1114, and node compute resources ("Node RRs") 1116(1)-1116(N), where "N" represents any positive integer. In at least one embodiment, the Node RRs 1116(1)-1116(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), data storage devices (e.g., solid-state or hard disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules, etc.In some embodiments, one or more Node RRs among Node RRs 1116(1)-1116(N) may correspond to a server having one or more of the aforementioned computing resources. Furthermore, in some embodiments, Node RRs 1116(1)-1116(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or the like, and / or one or more Node RRs 1116(1)-1116(N) may correspond to a virtual machine (VM).

[0178] In at least one embodiment, the grouped computing resources 1114 may include separate groupings of node RRs 1116 housed within one or more racks (not shown) or many racks housed in data centers in different geographical locations (also not shown). Separate groupings of node RRs 1116 within the grouped computing resources 1114 may include grouped computing, networking, memory, or storage resources that may be configured or assigned to support one or more workloads. In at least one embodiment, multiple node RRs 1116, including CPUs, GPUs, DPUs, and / or processors, may be grouped in one or more racks to provide computing resources to support one or more workloads.The one or more racks may also contain any number of power modules, cooling modules, and / or network switches in any combination.

[0179] Resource orchestrator 1112 may configure or otherwise control one or more node RRs 1116(1)-1116(N) and / or clustered computing resources 1114. In at least one embodiment, resource orchestrator 1112 may include a software design infrastructure (SDI) management entity for data center 1100. Resource orchestrator 1112 may be comprised of hardware, software, or a combination thereof.

[0180] In at least one embodiment, as in Fig. 11, the framework layer 1120 may include a task scheduler 1133, a configuration manager 1134, a resource manager 1136, and / or a distributed file system 1138. The framework layer 1120 may include a framework for supporting software 1132 of a software layer 1130 and / or one or more applications 1142 of an application layer 1140. The software 1132 or the application(s) 1142 may each include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1120 may be some type of free and open-source software web application framework, such as, but not limited to, Apache Spark™ (hereinafter "Spark"), which may utilize the distributed file system 1138 for large-scale data processing (e.g., "Big Data").In at least one embodiment, task scheduler 1133 may include a Spark driver to facilitate scheduling workloads supported by different layers of data center 1100. Configuration manager 1134 may be capable of configuring different layers, such as software layer 1130 and framework layer 1120, which includes Spark and distributed file system 1138 to support large-scale data processing. Resource manager 1136 may be capable of managing clustered or grouped compute resources allocated or assigned to support distributed file system 1138 and task scheduler 1133. In at least one embodiment, clustered or grouped compute resources may include clustered compute resource 1114 on data center infrastructure layer 1110.The resource manager 1136 and the resource orchestrator 1112 may coordinate to manage these allocated or assigned computing resources.

[0181] In at least one embodiment, the software 1132 included in software layer 1130 may include software used by at least portions of node RRs 1116(1)-1116(N), clustered computing resources 1114, and / or distributed file system 1138 of framework layer 1120. One or more types of software may include, but are not limited to, software for browsing Internet web pages, software for scanning emails for viruses, database software, and software for streaming video content.

[0182] In at least one embodiment, the application(s) 1142 included in the application layer 1140 may include one or more types of applications used by at least portions of the node RRs 1116(1)-1116(N), the clustered compute resources 1114, and / or the distributed file system 1138 of the framework layer 1120. One or more types of applications may include, but are not limited to, any number of a genomics application, a cognitive computing application, a machine learning application that includes training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in connection with one or more embodiments.

[0183] In at least one embodiment, any of the configuration manager 1134, the resource manager 1136, and the resource orchestrator 1112 may implement any number and type of self-modifying actions based on any amount and type of data collected in any technically feasible manner. Self-modifying actions may relieve a data center operator of the data center 1100 of potentially making poor configuration decisions and potentially avoiding underutilized and / or poorly performing portions of a data center.

[0184] Data center 1100 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model(s) may be trained by calculating weighting parameters according to a neural network architecture using software and / or computational resources described above with respect to data center 1100.In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1100 by using weighting parameters calculated by one or more training techniques, such as, but not limited to, those described herein.

[0185] In at least one embodiment, data center 1100 may utilize CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inferencing with the resources described above. Furthermore, one or more of the software and / or hardware resources described above may be configured as a service to enable users to train or perform inferencing of information, such as image recognition, speech recognition, or other artificial intelligence services. EXEMPLARY NETWORK ENVIRONMENTS

[0186] Network environments suitable for use in implementing embodiments of the disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may reside on one or more instances of the computing device 1000 in Fig. 10 - each device may, for example, include similar components, features, and / or functions of the computing device(s) 1000. In addition, if backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may also be part of a data center 1100, an example of which is shown in Fig. 11 is described in more detail.

[0187] Components of a network environment can communicate with each other over one or more networks, which can be wired, wireless, or both. The network can include multiple networks or a network of networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and / or a public switched telephone network (PSTN), and / or one or more private networks. If the network includes a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) can provide wireless connectivity.

[0188] Compatible network environments may include one or more peer-to-peer network environments—in which case, a server may not be included in a network environment—and one or more client-server network environments—in which case, one or more servers may be included in a network environment. In peer-to-peer network environments, the functionality described herein with respect to one or more servers may be implemented on any number of client devices.

[0189] In at least one embodiment, a network environment may include one or more cloud-based network environments, a distributed computing environment, a combination thereof, etc. A cloud-based network environment may include a framework layer, a task scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. A framework layer may include a framework for supporting software of a software layer and / or one or more applications of an application layer. The software or application(s) may each include web-based service software or applications. In embodiments, one or more of the client devices may use the web-based service software or applications (e.g.,by accessing the service software and / or applications through one or more application programming interfaces (APIs). The framework layer may be, but is not limited to, some type of free and open-source software web application framework, which may use, for example, a distributed file system for processing large amounts of data (e.g., "big data").

[0190] A cloud-based network environment may provide cloud computing and / or cloud storage performing any combination of computing and / or data storage functions (or one or more portions thereof) described herein. Each of these various functions may be distributed across multiple locations of central or core servers (e.g., one or more data centers that may be distributed across a state, region, country, globe, etc.). When a connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers may allocate at least some functionality to the edge server(s). A cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0191] The client devices may implement at least some of the components, features, and functionalities of the exemplary computing device(s) 1000 described herein with respect to Fig.10. By way of example and not limitation, a client device may be embodied as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning (GPS) system or device, video player, video camera, surveillance device or system, vehicle, boat, aircraft, virtual machine, drone, robot, portable communications device, device in a hospital, gaming device or system, entertainment system, vehicle computing system, embedded system controller, remote control, household appliance, consumer electronics device, workstation, edge device, any combination of these outlined devices, or any other suitable device.

[0192] The disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules, executed by a computer or another computer, such as a personal data assistant or other handheld device. In general, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs specific tasks or implements specific abstract data types. The disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network.

[0193] As used herein, the use of "and / or" with reference to two or more elements should be construed as meaning only one element or a combination of elements. For example, "Element A, Element B, and / or Element C" may include only Element A, only Element B, only Element C, Element A and Element B, Element A and Element C, Element B and Element C, or Elements A, B, and C. Furthermore, "at least one of Element A or Element B" may include at least one of Element A, at least one of Element B, or at least one of Element A and at least one of Element B. Further, "at least one of Element A and Element B" may include at least one of Element A, at least one of Element B, or at least one of Element A and at least one of Element B.

[0194] The subject matter of the present disclosure is described herein in detail to satisfy legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors contemplated that the claimed subject matter could be embodied in other ways to include other steps or combinations of steps similar to those described herein in connection with other present or future technologies. Although the terms "step" and / or "block" may be used in this document to refer to various elements of the methods employed, the terms should not be interpreted as implying any particular order among or between the various steps recited herein unless the order of each step is expressly described. EXAMPLE CLAUSES

[0195] A: A method comprising: determining, based on at least first sensor data generated using one or more first sensors of a vehicle, a position associated with a component located in an interior of the vehicle; determining, based on at least the position associated with the component, one or more values ​​for one or more calibration parameters of a second sensor located in the interior of the vehicle; and causing, based on at least second sensor data generated using the second sensor and the one or more values ​​of the one or more calibration parameters, one or more operations associated with the vehicle to be performed.

[0196] B: The method of paragraph A, further comprising: obtaining data representing a table associating at least the position associated with the component with the one or more values ​​for the one or more calibration parameters, wherein determining the one or more values ​​for the one or more calibration parameters of the second sensor is based at least on the data representing the table.

[0197] C: The method of either paragraph A or paragraph B, further comprising: determining one or more second values ​​for the one or more calibration parameters of the second sensor, wherein determining the one or more values ​​for the one or more calibration parameters of the second sensor is further based at least on the one or more second values ​​for the one or more calibration parameters.

[0198] D: The method of any one of paragraphs AC, further comprising: determining, using the one or more first sensors of the vehicle, a second position associated with the component located inside the vehicle; and determining, based at least on the second position associated with the component, one or more second values ​​for the one or more calibration parameters of the second sensor located inside the vehicle.

[0199] E: The method of any one of paragraphs AD, further comprising: receiving third sensor data generated using the second sensor, wherein determining the one or more values ​​for the one or more calibration parameters is further based at least on the third sensor data.

[0200] F: The method of any one of paragraphs AE, further comprising: receiving third sensor data generated using the second sensor, wherein the third sensor data represents a sensor representation; and determining that a portion of the sensor representation represents a feature located within the vehicle, wherein determining the one or more values ​​for the one or more calibration parameters is further based at least on determining the portion of the sensor representation representing the feature.

[0201] G: The method of any one of paragraphs AF, further comprising: determining, based at least on the data of the second sensor and the one or more values ​​for the one or more calibration parameters, a position of an object inside the vehicle, wherein causing the performance of the one or more operations associated with the vehicle is localized at least to the position of the object inside the vehicle.

[0202] H: The method of any one of paragraphs AG, further comprising: receiving one or more inputs associated with adjusting a position of the component inside the vehicle, wherein the second sensor is attached to the component; and receiving the first sensor data generated using the one or more first sensors after adjusting the position of the component.

[0203] I: The method of any one of paragraphs AH, wherein the one or more values ​​for the one or more calibration parameters comprise at least one of the following: one or more values ​​associated with one or more translation parameters to calibrate the second sensor with respect to a coordinate system associated with the vehicle; or one or more values ​​associated with one or more rotation parameters to calibrate the second sensor with respect to the coordinate system associated with the vehicle.

[0204] J: System comprising: one or more processing units to: determine one or more first values ​​for one or more calibration parameters of a first sensor located in an interior of a vehicle; receive sensor data generated using the first sensor and / or a second sensor located in the interior of the vehicle; determine a position associated with a feature located in the interior of the vehicle based on at least the sensor data; and determine one or more second values ​​for the one or more calibration parameters of the first sensor based on at least the position associated with the feature.

[0205] K: The system of paragraph J, wherein the one or more processing units are further to: obtain data representing a table associating at least the position associated with the feature with the one or more second values ​​for the one or more calibration parameters, wherein the one or more second values ​​for the one or more calibration parameters of the first sensor are further determined based at least on the data representing the table.

[0206] L: The system of either paragraph J or paragraph K, wherein: the sensor data includes image data representing an image; the one or more processing units are to determine the position associated with the feature located inside the vehicle by determining the position of a pixel within the image representing the feature based at least on the image data; and the one or more processing units are to locate the one or more second values ​​for the one or more calibration parameters of the first sensor based at least on the position of the pixel.

[0207] M: The system of any one of paragraphs JL, wherein: the sensor data is representative of a sensor representation; the one or more processing units are to determine the position associated with the feature located inside the vehicle by determining, at least based on the sensor data, that a portion of the sensor representation represents the feature, wherein the portion of the sensor representation corresponds to the position associated with the feature; and the one or more processing units are to determine the one or more second values ​​for the one or more calibration parameters of the first sensor based at least on the portion of the sensor representation.

[0208] N: The system of any one of paragraphs JM, wherein: the sensor data is generated using the second sensor; the one or more processing units determine the position associated with the feature located inside the vehicle by determining a position of a component inside the vehicle based at least on the sensor data, the position of the component corresponding to the position of the feature; and the one or more processing units determine the one or more second values ​​for the one or more calibration parameters of the first sensor based at least on the position of the component.

[0209] O: The system of any one of paragraphs JN, wherein the one or more processing units are to determine the one or more second values ​​for the one or more calibration parameters of the first sensor based on at least the one or more first values ​​for the one or more calibration parameters of the first sensor.

[0210] P: The system of any one of paragraphs JO, wherein the one or more processing units are further operable to: determine a location of an object within the vehicle based on at least the second sensor data generated using the first sensor and the one or more second values ​​for the one or more calibration parameters; and cause one or more vehicle-associated operations to be performed based on at least the position of the object.

[0211] Q: A system according to any one of paragraphs JP, wherein the system comprises at least one of the following systems: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system implementing one or more large language models (LLMs); a system for performing conversational AI operations; a system for generating synthetic data; a system including one or more virtual machines (VMs);a system that is implemented at least in part in a data center; or a system that is implemented at least in part using cloud computing resources.

[0212] R: Processor comprising: one or more processing units to determine one or more values ​​for one or more calibration parameters of a sensor located in an interior of a vehicle, wherein the one or more values ​​for the one or more calibration parameters are determined based at least on sensor data representing a position of a component in the interior of the vehicle.

[0213] S: The processor of paragraph R, wherein the one or more processing units further determine one or more second values ​​for the one or more calibration parameters of the sensor located inside the vehicle, wherein the one or more second values ​​for the one or more calibration parameters are determined based on at least second sensor data representing a second position of the component inside the vehicle.

[0214] T: The processor of either paragraph R or paragraph S, wherein the processor comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system implementing one or more large language models (LLMs); a system for performing conversational AI operations; a system for generating synthetic data; a system including one or more virtual machines (VMs);a system that is implemented at least in part in a data center; or a system that is implemented at least in part using cloud computing resources. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 16 / 101,232

[0097] Cited non-patent literature

[0000] Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (e.g., Standard No. J3016-201806, published on June 15, 2018, Standard No. J3016-201609, published on September 30, 2016

[0055]

Claims

[1] Method comprising: Determining, based on at least first sensor data generated using one or more first sensors of a vehicle, a position associated with a component located in an interior of the vehicle; Determining, based at least on the position associated with the component, one or more values for one or more calibration parameters of a second sensor located inside the vehicle; and Causing, at least based on second sensor data generated using the second sensor and the one or more values of the one or more calibration parameters, the performance of one or more operations associated with the vehicle. [2] The method of claim 1, further comprising: Obtaining data representing a table that associates at least the position associated with the component with the one or more values for the one or more calibration parameters, wherein determining the one or more values for the one or more calibration parameters of the second sensor is based at least on the data representing the table. [3] The method of claim 1 or 2, further comprising: Determining one or more second values for the one or more calibration parameters of the second sensor, wherein determining the one or more values for the one or more calibration parameters of the second sensor is further based at least on the one or more second values for the one or more calibration parameters. [4] Method according to one of the preceding claims, further comprising: Determining, using the one or more first sensors of the vehicle, a second position associated with the component located inside the vehicle; and Determining, based at least on the second position associated with the component, one or more second values for the one or more calibration parameters of the second sensor located inside the vehicle. [5] Method according to one of the preceding claims, further comprising: Receiving third sensor data generated using the second sensor, wherein determining the one or more values for the one or more calibration parameters is further based on at least the third sensor data. [6] Method according to one of the preceding claims, further comprising: Receiving third sensor data generated using the second sensor, wherein the third sensor data represents a sensor representation; and Determining that a portion of the sensor representation depicts a feature located within the vehicle, wherein determining the one or more values for the one or more calibration parameters is further performed at least based on determining the portion of the sensor representation representing the feature. [7] Method according to one of the preceding claims, further comprising: Determine, based on at least the second sensor data and the one or the plurality of values for the one or more calibration parameters, a position of an object within the interior of the vehicle, wherein causing the one or more operations associated with the vehicle to be performed is based at least on the position of the object within the vehicle. [8] Method according to one of the preceding claims, further comprising: Receiving one or more inputs associated with adjusting a position of the component within the interior of the vehicle, wherein the second sensor is attached to the component; and Receiving, after adjusting the position of the component, the first sensor data generated using the one or more first sensors. [9] A method according to any one of the preceding claims, wherein the one or more values for the one or more calibration parameters comprise at least one of the following: one or more values associated with one or more translation parameters to calibrate the second sensor with respect to a coordinate system associated with the vehicle; or one or more values associated with one or more rotation parameters to calibrate the second sensor with respect to the coordinate system associated with the vehicle. [10] System comprising: one or more processing units for the following: determine one or more first values for one or more calibration parameters of a first sensor located inside a vehicle; Receiving sensor data generated by at least one of the first sensors or a second sensor located inside the vehicle; Determining, based on at least the sensor data, a position associated with a feature located inside the vehicle; and Determining, based on at least the position associated with the feature, one or more second values for the one or more calibration parameters of the first sensor. [11] The system of claim 10, wherein the one or more processing units are further configured to: Obtaining data representing a table that associates at least the location associated with the feature with the one or more second values for the one or more calibration parameters, wherein the one or more second values for the one or more calibration parameters of the first sensor are further determined based on at least the data representing the table. [12] System according to claim 10 or 11, wherein: the sensor data includes image data representing an image; the one or more processing units determine the position associated with the feature located inside the vehicle by determining, based on at least the image data, the position of a pixel within the image representing the feature; and the one or more processing units are to determine the one or more second values for the one or more calibration parameters of the first sensor based at least on the location of the pixel. [13] System according to one of claims 10 to 12, wherein: the sensor data represent a sensor representation; the one or more processing units are to locate the position associated with the feature located inside the vehicle by determining, based on at least the sensor data, that a portion of the sensor representation represents the feature, wherein the portion of the sensor representation corresponds to the position associated with the feature; and the one or more processing units determine the one or more second values for the one or more calibration parameters of the first sensor based on at least the portion of the sensor representation. [14] System according to one of claims 10 to 13, wherein: the sensor data is generated using the second sensor; the one or more processing units determine the position associated with the feature located inside the vehicle by determining a position of a component inside the vehicle based on at least the sensor data, the position of the component corresponding to the position of the feature; and the one or more processing units are to determine the one or more second values for the one or more calibration parameters of the first sensor based at least on the position of the component. [15] The system of any one of claims 10 to 14, wherein the one or more processing units are to determine the one or more second values for the one or more calibration parameters of the first sensor based on at least the one or more first values for the one or more calibration parameters of the first sensor. [16] The system of any one of claims 10 to 15, wherein the one or more processing units further serve to: Determining, based on at least second sensor data generated using the first sensor and the one or more second values for the one or more calibration parameters, a position of an object within the interior of the vehicle; and Cause, based at least on the position of the object, the performance of one or more operations associated with the vehicle. [17] System according to any one of claims 10 to 16, wherein the system is comprised in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system that implements one or more large language models (LLMs); a system for performing conversational AI operations; a system for generating synthetic data; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources. [18] Processor comprising: one or more processing units to determine one or more values for one or more calibration parameters of a sensor located inside a vehicle, wherein the one or more values for the one or more calibration parameters are determined based at least on sensor data representing a position of a component inside the vehicle. [19] The processor of claim 18, wherein the one or more processing units further determine one or more second values for the one or more calibration parameters of the sensor located inside the vehicle, wherein the one or more second values for the one or more calibration parameters are determined based on at least second sensor data representing a second position of the component inside the vehicle. [20] A processor according to claim 18 or 19, wherein the processor is comprised in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system that implements one or more large language models (LLMs); a system for performing conversational AI operations; a system for generating synthetic data; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources.

Images