Image-based three-dimensional occupant assessment for in-cabin monitoring systems and applications

Abstract

In various examples, image-based three-dimensional occupant evaluation for in-cabin monitoring systems and applications is provided. The evaluation function may determine a 3D representation of an occupant of the machine by evaluating sensor data containing image frames from the optical image sensor. The 3D representation may include at least one characteristic (e.g., 3D pose and / or 3D shape) representing the size of the occupant, which may be used to derive other characteristics, such as, but not limited to, weight, height, and / or age. The first processing path may generate a representation of one or more features corresponding to at least a portion of the occupant based on the optical image data, and the second processing path may determine a depth corresponding to the one or more features based on depth data derived from the optical image data and truth-valued depth data corresponding to the machine interior.

Description

[0001] Cross-references to related applications

[0002] This application relates to U.S. Patent Application No. 17 / 935,473, filed September 26, 2022, entitled "Multimodal Sensor Calibration for In-Car Monitoring Systems and Applications," and U.S. Patent Application No. 17 / 935,465, filed September 26, 2022, entitled "Sensor Calibration Using Reference Markers for In-Car Monitoring Systems and Applications," the entire contents of which are incorporated herein by reference.

[0003] This application relates to U.S. Patent Application No. 18 / 349,842, filed July 10, 2023 (Attorney File No. 22-SC-1598US01 / 396522), entitled "Three-Dimensional Pose Estimation Using Two-Dimensional Images"; U.S. Patent Application No. 18 / 349,827, filed July 10, 2023 (Attorney File No. 23-SC-0156US01 / 396005), entitled "Occupant Assessment Using Multimodal Sensor Fusion for In-Car Monitoring Systems and Applications"; and U.S. Patent Application No. 18 / 219,969, filed July 10, 2023 (Attorney File No. 23-SC-0158US01 / 396006), entitled "Child Presence Detection for In-Car Monitoring Systems and Applications," the entire contents of which are incorporated herein by reference. Background Technology

[0004] Occupant monitoring systems (OMS) can be used within a vehicle cabin to assess the presence, gaze, alertness, or other conditions of the driver and occupants in real time. For example, OMS—which uses data generated or acquired by sensors in the vehicle or machine—may be used to track the driver's eye gaze, head posture, or blinking (e.g., to detect drowsiness, fatigue, and / or distraction), for hand position and / or gesture detection, for the presence of children and / or pets, and / or in conjunction with functions such as, but not limited to, seatbelt reminders, seat heating, and / or airbag deployment. For instance, optical sensors may be used to detect movement occurring within the vehicle interior. Optical image sensor data may also be processed to extract image features, thereby identifying and classifying motion sources. Depth-sensing sensors may use, for example, radio waves, lasers, and / or sound waves to detect the presence or movement of living beings (e.g., humans or pets) within the vehicle interior. Such detection may be used to prevent vehicle theft and / or to prevent children or pets from being unintentionally left alone in the vehicle. Summary of the Invention

[0005] Embodiments of this disclosure relate to image-based three-dimensional occupant assessment for in-vehicle monitoring systems and applications. Systems and methods for occupant evaluation functions of OMS (Optical Mode System) for three-dimensional (3D) attitude estimation and / or 3D shape estimation that may predict vehicle occupants are disclosed.

[0006] Compared to existing OMS technologies, among other things, the occupant evaluation function proposed in this disclosure provides an occupant monitoring system that may use optical image data captured by an OMS optical image sensor (e.g., a fisheye camera or wide-angle camera on the overhead console inside the vehicle compartment) to determine a 3D representation of vehicle occupants (such as their 3D size, 3D shape, and / or 3D pose). The occupant evaluation function may generate 3D pose estimates and / or 3D shape estimates of the vehicle occupants, which may be used as input to other systems and functions that further predict other characteristics related to the vehicle occupants, such as their size, weight, and / or age. The occupant evaluation function may determine a 3D representation of vehicle occupants (such as 3D shape and / or 3D pose) by evaluating optical image sensor data comprising image frames captured by the OMS optical image sensor. The image frame may contain a representation of the vehicle occupants from a viewpoint at least partially based on the position and orientation of the OMS optical image sensor within the vehicle compartment. In some embodiments, the occupant evaluation function executes a first processing path that operates to generate a representation of one or more features corresponding to at least a portion of an occupant based on optical image data (e.g., image frames), and a second processing path that operates to determine the depth corresponding to one or more features based on depth data derived from the optical image data and ground-value depth data corresponding to the vehicle interior (such as a ground-value cabin depth image). Typically, the depth information derived from the second processing path may be used by the occupant evaluation function to provide an absolute sense of scale for generating a true-scale 3D representation of the vehicle occupant from the representation of the vehicle occupant generated by the first processing path of the occupant evaluation function. The ground-value cabin depth image corresponding to the vehicle interior may be generated in various ways. For example, in some embodiments, vehicle geometry specifications (e.g., CAD model information) regarding the empty vehicle geometry, combined with precise calibration of the vehicle geometry specifications from the OMS optical image sensor, may be used to generate the ground-value cabin depth image.

[0007] OMS and / or other vehicle systems may use 3D representations for various purposes, such as estimating other characteristics representing occupant dimensions (e.g., estimating occupant height and / or limb length). In some embodiments, the 3D representation of an occupant may be used to control at least one operation of the vehicle based on estimated occupant characteristics. For example, characteristics representing occupant dimensions may be used in conjunction with a child presence detection system to estimate an occupant's age and / or to control an alarm system (e.g., an alarm and / or notification system) based on determining that an occupant below an estimated age threshold may have been unintentionally left alone in the vehicle. In some embodiments, characteristics representing occupant dimensions may be used in conjunction with other vehicle safety features. For example, in some embodiments, airbag deployment, driver monitoring systems, HMI applications, and / or other vehicle or machine functions may be controlled at least based on 3D posture and / or size estimates of vehicle or machine occupants. Attached Figure Description

[0008] The system and method for image-based three-dimensional occupant assessment used in in-car monitoring systems and applications will be described in detail below with reference to the accompanying drawings, wherein:

[0009] Figure 1 This is an example data flow diagram of a system for image-based three-dimensional occupant assessment according to some embodiments of this disclosure;

[0010] Figure 2 This is an example data flow diagram of a system for an example implementation of a 3D pose-based three-dimensional occupant assessment according to some embodiments of the present disclosure;

[0011] Figure 3 This is a diagram illustrating ground truth depth data generated for use with image-based 3D occupant assessment, according to some embodiments of this disclosure;

[0012] Figure 4 This is a diagram illustrating the determination of the absolute depth of one or more kinematic elements based on ray tracing according to some embodiments of the present disclosure;

[0013] Figure 5 This is an example data flow diagram of a system for an example implementation of a 3D shape-based three-dimensional occupant assessment according to some embodiments of this disclosure;

[0014] Figure 6 This is a diagram illustrating the use of attitude detection combined with 3D shape-based three-dimensional occupant assessment according to some embodiments of the present disclosure;

[0015] Figure 7 This is a diagram illustrating an example method for image-based three-dimensional occupant assessment according to some embodiments of the present disclosure;

[0016] Figure 8AThese are illustrations of example autonomous vehicles according to some embodiments of the present disclosure;

[0017] Figure 8B According to some embodiments of this disclosure Figure 8A Examples of camera positions and fields of view for autonomous vehicles;

[0018] Figure 8C According to some embodiments of this disclosure Figure 8A A block diagram of an example system architecture for an example autonomous vehicle;

[0019] Figure 8D This is based on some embodiments of the present disclosure for use in cloud-based servers and Figure 8A A system diagram illustrating communication between autonomous vehicles;

[0020] Figure 9 This is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0021] Figure 10 This is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0022] Systems and methods related to image-based three-dimensional occupant assessment for use in-cabin monitoring systems and applications are disclosed. While this disclosure may be directed to examples of autonomous or semi-autonomous vehicles or machines 800 (which may alternatively be referred to herein as "vehicle 800" or "self-machine 800"), examples are provided. Figures 8A-8D The description is intended to be limited in scope, but is not intended to be restrictive. For example, the systems and methods described herein may be used with respect to non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more advanced driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, engineering vehicles, trains, underwater vehicles, teleoperated vehicles (e.g., drones), and / or other vehicle types (but not limited thereto). Furthermore, while this disclosure may be described with respect to image-based assessment of vehicle occupant characteristics (e.g., estimated occupant size and / or posture for vehicle occupant monitoring systems), this is not intended to be restrictive, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technological space where physical characteristics of humans may be used.

[0023] This disclosure relates to vehicle occupant detection and monitoring techniques. More specifically, the systems and methods proposed in this disclosure provide techniques for occupant monitoring systems (OMS) that may predict three-dimensional (3D) pose estimates and / or 3D shape estimates of vehicle occupants from image frames captured by an optical image sensor (e.g., a camera). The size of the vehicle occupant may be derived based on this 3D pose estimate and / or 3D shape estimate. Among the tasks that may be performed by the OMS, child presence detection (CPD) specifically involves detecting when a child is a vehicle occupant, among other things. For example, a child protection system may attempt to assess when an occupant in an idle vehicle is under a threshold age and alert the owner if a child may have been unintentionally left inside the vehicle. Size estimation may be an upstream task related to age estimation, as human size is closely related to age and is one of the most easily observed characteristics of a person, which can be used to assess the age of a vehicle occupant.

[0024] Especially in terms of detection characteristics, such as the 3D shape and / or 3D body pose of vehicle occupants, conventional systems have used in-cabin depth sensors, such as RADAR sensors. These sensors may directly generate 3D data corresponding to objects detected within the vehicle cabin, and may penetrate structural elements inside the vehicle (e.g., car seats) to detect occluded objects not within the line of sight of optical image sensors. RADAR sensors may generate sensor data that can be used to derive 3D size estimates that typically represent occupant dimensions (e.g., to distinguish between child and adult occupants). However, deploying in-cabin sensing RADAR sensors in production vehicles can be expensive. Furthermore, sensor data from RADAR sensors is resolution-limited, restricting their ability to generate accurate 3D body pose estimates that capture limb positions (e.g., for child presence detection and / or occupant age prediction), and they rely primarily on sensed motion to sense objects. Other depth-sensing sensors, such as depth-sensing cameras, also known as range cameras, generate two-dimensional (2D) range images, where pixel values ​​may correspond to distances from the sensor to the sensed image within the sensor's field of view. However, deploying depth-sensing cameras in production vehicles is also expensive. Furthermore, the accuracy of occupant 3D depth data derived from 2D distance images generated by depth-sensing cameras can be limited by factors such as relatively low resolution, short sensing distance, and sensitivity to occlusion and / or optical interference.

[0025] Optical image sensor data from a single-eye optical image sensor, such as a camera capturing standard RGB, IR, and / or RGB-IR image frames, may be obtained using relatively inexpensive equipment that may already be deployed in the vehicle cabin for other purposes (e.g., driver gaze detection). That said, a single-eye optical image sensor itself does not generate data that conveys a 3D sense of the position of objects in the captured scene, making it difficult to train machine learning models (such as deep neural networks (DNNs)) to generate accurate 3D shape, 3D size, and / or 3D body pose estimates for people captured in 2D images. Furthermore, the further an occupant deviates from an upright posture, the more likely the 2D image-based estimation is to produce inaccurate estimates, such as when an occupant is sitting in a slouching or hunched posture and / or turned to one side. This is often because 2D-to-3D imaging mapping is an ill-posed problem with fuzzy solutions; for example, the same 2D projection may be derived from multiple 3D poses.

[0026] Compared to existing OMS technologies, the systems and methods proposed in this disclosure provide an occupant monitoring system that may use one or more image frames captured by at least one OMS optical image sensor (e.g., an overhead fisheye camera or a wide-angle camera inside the vehicle compartment) to determine the 3D representation of vehicle occupants (such as their 3D size, 3D shape, and / or 3D pose).

[0027] In some embodiments, the occupant assessment function disclosed herein may generate a 3D pose estimate and / or 3D shape estimate of a vehicle occupant. This estimate may be used as input to other systems and functions that further predict other characteristics of the vehicle occupant, such as the occupant's size, weight, and / or age. Therefore, the occupant assessment function may be used in conjunction with a child presence detection system designed to prevent children and / or pets from being accidentally left alone in a vehicle. In some embodiments, the occupant monitoring system may be used in conjunction with other vehicle safety features. For example, in some embodiments, airbag deployment may be controlled based on a 3D pose estimate of the vehicle occupant. Based on the 3D pose estimate, the airbag deployment system may determine that a vehicle occupant is out of position and therefore the airbag should not deploy. In some embodiments, the airbag deployment system may control the airbag deployment pressure based on the 3D pose estimate of the vehicle occupant and / or at least partially derived from the estimated size or age of the vehicle occupant from the 3D pose estimate. Furthermore, in some embodiments, the 3D shape and / or pose estimates generated by OMS as described herein may also be used for occupant pose classification, activity recognition, gaze detection, and human-machine interface (HMI) applications. Advantageously, the occupant monitoring system can generate 3D shape estimates and / or 3D attitude estimates that are invariant to the viewpoint, providing greater leeway in determining the placement and / or orientation of the OMS optical image sensor that captures the image frame. Therefore, the OMS may not need to rely on precise calibration of the OMS optical image sensor's orientation to the cabin coordinate system to predict occupant size or attitude.

[0028] More specifically, in some embodiments, the occupant evaluation function described herein may determine a 3D representation of a vehicle occupant (such as 3D shape and / or 3D pose) by evaluating optical image sensor data comprising image frames captured by an OMS optical image sensor. The image frames may comprise a representation of the vehicle occupant from a perspective at least partially based on the position and orientation of the OMS optical image sensor within the vehicle cabin. In some embodiments, the OMS optical image sensor may comprise a monocular optical image sensor, such as a camera that captures standard RGB, IR, and / or RGB-IR image frames of the vehicle interior. In some embodiments, the OMS optical image sensor may comprise a depth-sensing camera that generates a 2D distance image of the vehicle interior. This 2D distance image may comprise an image frame of pixels, where the pixel value corresponds to the distance from the depth-sensing camera to an element in the depth-sensing camera's field of view. In some embodiments, as described in more detail below, the OMS includes an occupant evaluation function that may predict a scale-normalized 3D attitude estimate corresponding to a vehicle occupant, and then scale the 3D attitude estimate to the full-scale dimension (e.g., inches or meters) of a linear measurement to determine at least one characteristic representing the occupant size. In some embodiments, the OMS includes an occupant evaluation function that may predict a scale-normalized 3D shape corresponding to a vehicle occupant, and then scale the 3D shape to the full-scale dimension (e.g., inches or meters) of a linear measurement to determine at least one characteristic representing the occupant size.

[0029] In some embodiments, the occupant evaluation function executes a first processing path that operates to generate representations of one or more features corresponding to at least a portion of an occupant based on optical image data (e.g., image frames). A second processing path operates to determine the depth corresponding to the one or more features based on depth data derived from the optical image data and ground-value depth data corresponding to the vehicle interior (such as ground-value cabin depth images). Typically, the depth information derived from the second processing path may be used by the occupant evaluation function to provide an absolute sense of scale for generating a true-to-scale 3D representation of the vehicle occupant from the representation of the vehicle occupant generated by the first processing path of the occupant evaluation function.

[0030] As discussed in more detail below, a ground-value cabin depth image corresponding to the vehicle interior can be generated in various ways. For example, in some embodiments, vehicle geometry specifications (e.g., CAD model information) regarding the empty vehicle geometry, combined with precise calibration of the vehicle geometry specifications to the OMS optical image sensor, may be used to generate the ground-value cabin depth image.

[0031] Occupant evaluation functions may generate a 3D representation of an occupant, including at least one characteristic representing occupant size, such as 3D posture and / or 3D shape. OMS and / or other vehicle systems may use this 3D representation for various purposes, such as estimating other characteristics representing occupant size (e.g., estimating occupant height and / or body limb length). In some embodiments, the occupant evaluation function may generate one or more outputs containing the 3D representation of the occupant, which are used to control at least one operation of the vehicle based on the estimated occupant characteristics. For example, characteristics representing occupant size may be used in conjunction with a child presence detection system to estimate the occupant's age and / or to control an alarm system (e.g., an alarm and / or notification system) based on determining that an occupant below an estimated age threshold may have been unintentionally left alone in the vehicle. In some embodiments, characteristics representing occupant size may be used in conjunction with other vehicle safety features. For example, in some embodiments, airbag deployment, driver monitoring systems, HMI applications, and / or other vehicle functions may be controlled based at least on 3D posture estimation and / or 3D size estimation of vehicle occupants.

[0032] In some embodiments, the occupant evaluation function processes input optical image frames from an OMS optical image sensor to derive a 3D pose estimate of a vehicle occupant. In such embodiments, the representation of one or more features corresponding to at least a portion of the occupant may include a scale-normalized 3D pose estimate. For example, in some embodiments, the occupant evaluation function may execute a first processing path including a human detection model and a 3D pose detection model (e.g., both models may be implemented using one or more of a convolutional neural network (CNN), a deep neural network (DNN), or other machine learning model architectures). The image frames may be processed by the human detection model, which identifies occupant features and crops the images to generate a cropped image (e.g., an image defined by the occupant's contour). Based on this cropped image, the 3D pose detection model may generate a scale-normalized 3D pose of the occupant. This scale-normalized 3D pose may include 3D representations of kinematic elements (e.g., body limbs and / or joints) indicating the 3D coordinates of the kinematic elements. In other words, the 3D pose detection model receives cropped images of the occupants from the human detection model and generates 3D pose estimates of the vehicle occupants from the captured image frames. This 3D pose detection model may be trained based on simultaneous multi-view images of the training object, thus using scale-normalized coordinates to generate the 3D pose estimates. That is, the 3D coordinates are scale-normalized because they may indicate the size and / or relative position of kinematic elements with respect to each other, rather than in absolute numerical terms (e.g., linear units of measurement).

[0033] To map the occupant's scale-normalized 3D pose to an absolute 3D pose, the second processing path of the occupant evaluation function may determine the absolute 3D depth corresponding to at least one joint from the scale-normalized 3D pose based on depth data obtained from at least one image frame and further using a ground-value cabin depth image corresponding to the vehicle interior. Using this at least one joint to anchor the scale-normalized 3D pose to the absolute scale, the occupant evaluation function may determine a set of absolute coordinates for other kinematic elements of the scale-normalized 3D pose to derive the absolute 3D pose.

[0034] In some embodiments, the second processing path of the occupant assessment function includes an optical depth estimation model. This optical depth estimation model may be implemented using, for example, a neural network architecture (such as a convolutional neural network (CNN), a deep neural network (DNN), or other (one or more) machine learning models). The optical depth estimation model may output a distance image based on inputs including image frames from an optical image sensor used to generate scale-normalized 3D poses. The pixel values ​​of this distance image may correspond to distances from the optical image sensor to sensed features appearing in the sensor's field of view in the image frame. Therefore, the distance image output from the optical depth estimation model provides the occupant assessment function with depth data corresponding to pixels in the image frame, representing features at least partially corresponding to the occupant, including pixels representing kinematic elements (such as at least one body joint) of the occupant captured by scale-normalized 3D poses. As discussed further below, to recover the actual depth scale of the kinematic elements in absolute numerical terms, the optical depth estimation model may further use a ground-value cabin depth image corresponding to the vehicle interior.

[0035] The occupant assessment function has established coordinates for the occupant's joint positions in three dimensions. This joint may be used as an anchoring joint to anchor the scale-normalized 3D pose to an absolute scale. The occupant assessment function may derive the absolute 3D pose from a set of absolute coordinates of other kinematic elements that determine this scale-normalized 3D pose. Given the absolute 3D pose of the vehicle occupant, the occupant assessment function may calculate the occupant's dimensions based on limb lengths that can be calculated, for example, as a function of distances between body joints. Since the 3D coordinates of kinematic elements in the absolute 3D pose can be directly mapped to absolute distances, the occupant's body limb lengths may be calculated directly from the 3D coordinates. For example, the width of the occupant's torso may be estimated by calculating the distance between the 3D coordinates of the occupant's left and right shoulder joints. For example, the occupant assessment function may estimate the overall dimensions of the occupant based on the summation of various body limb lengths. In some embodiments, a single limb length may be calculated as the distance between two consecutively connected joints. In some embodiments, composite limb lengths may be calculated as a function of a combination of kinematic elements. For example, an occupant's wingspan (extending from the occupant's left wrist to the occupant's right wrist when their arms are outstretched) might define a kinematic element that includes the combination of individual limbs and the calculated composite limb length of the wingspan. In some embodiments, pre-programmed statistics on body proportions might be used to estimate body dimensions that cannot be directly derived from absolute 3D pose. For example, statistically, a person's height is typically about 1.1 times their wingspan. Therefore, if the absolute 3D pose lacks kinematic information about the occupant's lower torso (e.g., if that part of their body is occluded in the image frame), the occupant evaluation function 110 might refer to statistics on body proportions, for example, estimating their height based on their derived wingspan length multiplied by 1.1.

[0036] In some embodiments, the occupant assessment function processes input image frames from an OMS optical image sensor to derive a 3D shape estimate of a vehicle occupant, rather than a 3D pose estimate, or something other than a 3D pose estimate. For example, in some embodiments, the occupant assessment function may execute a first processing path that includes a segmentation model that segments the input (one or more) image frames based on inferring pixels representing occupants from pixels that do not represent occupants. Based on this inference, the segmentation model may generate an occupant mask (e.g., defining pixel contours corresponding to occupants) and a vehicle interior mask (e.g., defining pixel contours corresponding to structural elements within the vehicle interior). A second processing path for the occupant assessment function may include a depth estimation model that may be implemented using, for example, a neural network architecture (such as CNN, DNN, or other (one or more) machine learning model architectures). The optical depth estimation model may output a distance image based on input including image frames from the optical image sensor, where the pixel values ​​of the distance image may correspond to distances from the optical image sensor to structural features within the vehicle interior that appear in the sensor's field of view in the optical image frame. The distance image output from the optical depth estimation model may provide occupant assessment functions with depth data corresponding to pixels in an image frame representing structural features. This depth data for structural features may be associated with pixels representing kinematic elements (such as at least one body joint) of the occupant captured within the boundaries of the occupant mask.

[0037] In some embodiments, to recover the actual dimensions of the depth of at least one body joint in absolute numerical terms, the optical depth estimation model may further use a ground-value cabin depth image corresponding to the vehicle interior, combined with a vehicle interior mask, to affine scale the kinematic elements captured within the boundaries of the occupant mask. As described above, the ground-value cabin depth image may include frames of distance images representing the field of view of the vehicle interior calibrated according to the field of view of the OMS optical image sensor. The optical depth estimation model may learn from the vehicle interior mask which pixels in the optical image data are structural components of the vehicle interior (such as those not representing occupants and / or (one or more) other objects located within the vehicle interior), and based on those pixels identified as vehicle interior components, refer to the ground-value cabin depth image to determine the distances between various structural interior elements and the OMS optical image sensor. For example, the optical depth estimation model may include a structural element depth estimator model that outputs structural element depth data based on an image frame containing optical image data and a ground-value cabin depth image as input. This structural element depth data may contain images with pixel values ​​corresponding to the distances from the optical image sensor to structural features within the vehicle interior. For example, structural element depth data generated by a structural element depth estimator model might indicate a given distance d1 between structural components within the vehicle compartment (e.g., the location of features on the compartment roof) and an OMS optical image sensor (e.g., in linear units of measurement). Occupant feature scaling functionality might be used as a 3D shape estimator to perform affine scaling, thereby scaling the pixels of an image frame using the structural element depth data to generate a 3D shape estimate of the occupants, where these pixels represent occupant features captured within the boundaries of an occupant mask.

[0038] To determine the 3D shape of an occupant, the occupant evaluation function may include a 3D shape estimator that takes into input an occupant mask generated by a segmentation model and structure element depth data generated by an optical depth estimation model. For example, a neural network architecture (such as a CNN, DNN, or one or more other machine learning model architectures) may be used to implement the segmentation model and / or the optical depth estimation model. Using the occupant mask and structure element depth data, the 3D shape estimator may infer that one or more pixels representing a first location corresponding to the occupant's body surface are closer to the OMS optical image sensor by a relative distance d2 than pixels corresponding to structural components inside the vehicle, and calculates the absolute distance between the occupant's body surface and the OMS optical image sensor based on a function of distances d1 and d2. The 3D shape estimator may infer the relative positions of one or more other surface locations on the occupant's body to this first location, and having established coordinates for the first location of the occupant's body, may determine the absolute distance from the first location to one or more other locations, thus estimating the overall 3D shape of the occupant.

[0039] In some embodiments, the OMS performing occupant assessment may be used for the vehicle interior space other than the passenger compartment. For example, the vehicle interior space of the embodiments described herein may use optical image data of occupants in the trunk, cargo compartment, or other spaces to determine 3D pose estimation and / or shape estimation. While the embodiments presented in this disclosure may be implemented in the context of occupant monitoring systems for vehicles (including driver monitoring systems), such as, but not limited to, non-autonomous vehicles, semi-autonomous vehicles, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, spacecraft, ships, shuttles, emergency response vehicles, construction vehicles, underwater vehicles, drones, and / or other vehicle types, other embodiments may include determining external calibration parameters for sensors used to capture image frames of other interior spaces, such as rooms, warehouses, stadiums, containers, airport terminals, mines, factories, construction zones, and / or studio sets, as suitable examples and not limitations, for monitoring occupants in these spaces.

[0040] The various sensor data processing, attitude estimation, size estimation, and / or other models, functions, and algorithms disclosed herein may be executed at least in part on one or more processing units, such as one or more graphics processing units, which may operate in conjunction with software executing on a central processing unit coupled to memory. The graphics processing unit may be programmed to execute a kernel to implement one or more aspects of the occupant monitoring functions described herein. In some embodiments, the execution of some algorithms may be distributed and performed by a combination of processors and cloud computing resources.

[0041] refer to Figure 1 , Figure 1 This is an example data flow diagram of a system 100 for image-based three-dimensional occupant assessment according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groups of functions, etc.) may be used as supplements or alternatives to the illustrated arrangements and elements, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and may be implemented in any suitable combination and location. The various functions performed by the entities described herein may be implemented by hardware, firmware, and / or software. For example, various functions may be implemented by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein may use... Figures 8A-8D Example of autonomous vehicles 800 Figure 9 Example computing device 900 and / or Figure 10 The example data center 1000 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the same task.

[0042] like Figure 1 As shown, the image-based 3D occupant assessment system 100 may include an occupant evaluation function 110, which includes at least one occupant feature detection model 120, at least one optical depth estimation model 122, and an occupant feature scaling function 130. The occupant feature detection model 120 and / or the optical depth estimation model 122 may be implemented using machine learning models, such as using convolutional neural networks (CNNs), deep neural networks (DNNs), and / or other neural network architectures.

[0043] like Figure 1 As shown, the occupant evaluation function 110 may execute a first processing path including the occupant feature detection model 120 and generate representations of one or more features of the vehicle occupants based on optical image data 107, which includes representations of the vehicle occupants (e.g., optical image frames) captured by an optical image sensor 106 that may be located within the vehicle interior 105 (e.g., the interior of vehicle 800). The optical image sensor 106 may include, for example, a camera or other optical sensor that captures RGB, IR, and / or RGB-IR image frames. In some embodiments, the optical image sensor 106 may include an OMS sensor, such as a reference sensor. Figure 8A The described (one or more) OMS sensor 801.

[0044] Occupant evaluation function 110 may execute a second processing path including optical depth estimation model 122 and generate depth data (e.g., depth from optical image sensor 106 to the corresponding element) corresponding to elements appearing within vehicle interior 105 based on optical image data 107 and ground truth depth data 108. In some embodiments, ground truth depth data 108 may include at least one ground truth cabin depth image 109, which may be used as a ground truth reference for distances between optical image sensor 106 and various structural elements of vehicle interior 105 appearing in optical image data 107. In some embodiments, optical depth estimation model 122 may be further input with representations of vehicle occupants generated by occupant feature detection model 120 to identify structural elements of vehicle interior 105 from one or more features of the vehicle occupants, thereby better estimating the depth of one or more features of the vehicle occupants. Occupant evaluation function 110 may input representations of one or more features of a vehicle occupant from occupant feature detection model 120 and depth data from optical depth estimation model 122 into occupant feature scaling function 130 to determine the absolute (e.g., real-scale) depth corresponding to one or more occupant features and / or generate a 3D representation of the occupant as output as 3D occupant representation data 140. This 3D occupant representation data 140 may include at least one feature representing the occupant's dimensions (e.g., occupant height and / or body limb length). The features included in the 3D occupant representation data 140 may include representations such as 3D pose estimation, 3D size estimation, and / or 3D shape estimation of the vehicle occupant. At least one operation of the vehicle may then be controlled based on this feature.

[0045] For example, based at least in part on 3D occupant representation data 140, an interior monitoring system 150 (which may implement one or more components of the OMS) may generate one or more outputs 154. These outputs 154 may be generated using one or more machine learning models and / or deep neural networks (DNNs) 152. For example, the interior monitoring system 150 may use the 3D occupant representation data 140 (alone or in combination with other data such as optical image data 107) to predict the presence and / or location of occupants—such as objects, people, and / or animals—within the interior space of the vehicle 800. Other systems of the vehicle 800 may determine one or more actions to be taken and / or may control other tasks or operations based on this prediction. For example, based on the outputs 154, alarms or warnings may be generated, door locks and / or windows may be operated, various functions may be opened / closed, data for digital assistants, chatbots, digital avatars, etc., may be generated, and / or air conditioning or air recirculation functions may be operated. In some embodiments, the characteristics representing occupant size from the 3D occupant representation data 140 may be used in conjunction with a child presence detection system to estimate the occupant's age and / or to control an alarm system (e.g., an alarm and / or notification system) based on determining that an occupant below an estimated age threshold may have been unintentionally left alone in the vehicle. In some embodiments, the characteristics representing occupant size may be used in conjunction with other vehicle safety features. For example, in some embodiments, airbag deployment, driver monitoring systems, HMI applications, and / or other vehicle functions may be controlled at least based on estimates of the vehicle occupant's 3D posture, 3D shape, and / or 3D size provided by the 3D occupant representation data 140.

[0046] Reference Figure 2 , Figure 2 This is an example data flow diagram 200 of a system 200 for image-based three-dimensional occupant assessment based on an example 3D pose implementation, according to some embodiments of the present disclosure, such as... Figure 1 As shown.

[0047] exist Figure 2 In the illustrated embodiment, the occupant evaluation function 110 processes optical image data 107 from the optical image sensor 106 to derive 3D occupant representation data 140 containing 3D pose estimates of the vehicle occupants. For example, in Figure 2In some embodiments, the occupant feature detection model 120 may include a human detection model 220 and a 3D pose detection model 222, which uses optical image data 107 to generate a scale-normalized 3D pose estimate 224 of the vehicle occupant. The scale-normalized 3D pose estimate 224 may contain representations of one or more features corresponding to at least a portion of the occupant. The human detection model 220 and / or the 3D pose detection model 222 may be implemented using one or more of, for example, convolutional neural networks (CNNs), deep neural networks (DNNs), or other machine learning model architectures.

[0048] Optical image data 107 may be processed by human detection model 220 to detect features of vehicle occupants, for example, to generate cropped images of the vehicle occupants (e.g., images defined by the outlines of the vehicle occupants). In some embodiments, when human detection model 220 detects more than one occupant, separate cropped images of the vehicle occupants may be generated and processed individually for each occupant as described herein.

[0049] Based on the detected features of the vehicle occupants generated by the human detection model 220 (e.g., cropped images), the 3D pose detection model 222 may generate a scale-normalized 3D pose 224 of the occupants. The scale-normalized 3D pose 224 may contain 3D representations of the kinematic elements of the vehicle occupants (e.g., one or more body limbs and / or joints) and may indicate the relative positions of the 3D coordinates using these kinematic elements. In other words, the 3D pose detection model 222 may receive a cropped image of the occupants from the human detection model 220 and predict the scale-normalized 3D pose 224 of the vehicle occupants based on the captured optical image data 107. As further discussed herein, the 3D pose detection model 222 may include a machine learning model trained on synchronized multi-view images of the training object to generate the scale-normalized 3D pose 224. That is, the 3D coordinates are scale-normalized because they may indicate the size and / or relative position of the kinematic elements with respect to each other, rather than in absolute numerical terms (e.g., linear units of measurement).

[0050] In some embodiments, the scale-normalized 3D pose 224 of a vehicle occupant may be temporally derived from optical image data 107 comprising a sequence of image frames from optical image sensor 106. In such a temporal embodiment, 3D pose detection model 222 may receive a cropped image sequence from human detection model 220 and predict a corresponding sequence of scale-normalized 3D poses 224 using the corresponding confidence scores. Occupant evaluation function 110 may implement voting functions, for example, to select the scale-normalized 3D pose with the highest confidence score from the sequence, and / or to select a composite scale-normalized 3D pose based on a subset of sequences with the most consistent (e.g., within an alignment threshold) pose estimates to achieve the scale-normalized 3D pose 224. In some embodiments, 3D pose detection model 222 may compute scale-normalized 3D pose estimates at different periods, such as computed scale-normalized 3D poses based on single frames that may be generated rapidly, and scale-normalized 3D poses derived from optimized image frame sequences and / or 3D pose estimates over predefined time windows. It is possible to evaluate the sequence of image frames to better identify moving objects in the scene, so that the human detection model 220 and / or the 3D pose detection model 222 can focus on processing dynamic segments of the optical image data 107, rather than static regions.

[0051] In some embodiments, such as Figure 2 In one embodiment, the second processing path of the occupant evaluation function 110 may include an occupant feature depth estimation model 230 that determines the absolute 3D depth of at least one anchoring joint appearing in the optical image data 107. The scaling function 130 may use this absolute 3D depth to determine a set of absolute coordinates for other kinematic elements of the scale-normalized 3D pose 224 to derive an absolute 3D pose estimate 240 of the vehicle occupant, thereby generating 3D occupant representation data 140. For example, the occupant feature depth estimation model 230 may be implemented using a neural network architecture such as a convolutional neural network (CNN), a deep neural network (DNN), or other (one or more) machine learning models.

[0052] To map the scale-normalized 3D pose 224 of the occupant to the absolute 3D pose estimate 240, the occupant feature depth estimation model 230 may take the scale-normalized 3D pose 224 as input and determine absolute joint depth data 232 corresponding to at least one joint from the scale-normalized 3D pose 224. Using the joint depth data 232 to correlate the scale-normalized 3D pose with the absolute scale, the occupant feature scaling function 130 may determine a set of absolute coordinates of other kinematic elements of the scale-normalized 3D pose 224, as indicated by the scale-normalized 3D pose 224, based on their relative distances and / or positions to designated (one or more) anchored joints.

[0053] In some embodiments, the occupant feature depth estimation model 230 may output joint depth data 232 in the form of a distance image. The pixel values ​​of the distance image may correspond to the distance from the optical image sensor 106 to the sensed image appearing in the sensor's field of view in the image frame. Therefore, the occupant feature scaling function 130 may correlate the distance image with a scale-normalized 3D pose 224 to determine the depth of the occupant features, including the depth of at least one kinematic element of the anchored joint.

[0054] To generate joint depth data 232, the occupant feature depth estimation model 230 may further input a ground-valued cabin depth image 109, which represents the field of view inside the vehicle calibrated according to the field of view of the OMS optical image sensor 106. For example, the occupant feature depth estimation model 230 may know from the ground-valued cabin depth image 109 that structural components (e.g., the position of features on the cabin roof) within the vehicle interior 105 are at a given distance d1 from the OMS optical image sensor 106 (e.g., in linear units of measurement). The occupant feature depth estimation model 230, based on the optical image data 107 and the scale-normalized 3D pose 224, may infer that one or more kinematic elements represented by the scale-normalized 3D pose 224 are closer to the optical image sensor 106 by a relative distance d2 than the pixels corresponding to the structural components, which are at a known absolute distance d1 from the optical image sensor 106, and calculate the absolute distance of the kinematic elements from the OMS optical image sensor based on a function of distances d1 and d2. The occupant feature scaling function 130 based on joint depth data 232 may use the true-scale absolute coordinates of the anchored joint position to anchor the scale-normalized 3D pose 224 to an absolute scale. The occupant feature scaling function 130 may determine a set of absolute coordinates of other kinematic elements of the scale-normalized 3D pose from this absolute scale to obtain an absolute 3D pose estimate 240.

[0055] As discussed above and also by Figure 3As shown at 300, ground truth depth data 108 may provide a reference to depth estimator model 230, based on a known vehicle geometry specification 310, from which the distances between the reference optical image sensor 106 and structural elements of the vehicle interior 105 are known. In some embodiments, the ground truth depth data 108, which may be represented as a ground truth cabin depth image 109, may be generated in a variety of ways. For example, in some embodiments, depth rendering function 320 may take vehicle geometry specification 310 (e.g., a CAD model of the vehicle interior 105) as input, which specifies the geometry (e.g., 3D coordinates) of structural elements of the vehicle interior 105, and convert it to the viewpoint of the optical image sensor 106 using sensor calibration parameters 305 to generate a ground truth cabin depth image. For example, in some embodiments, vehicle geometry specification 310 may define the mounting location and orientation of the optical image sensor 106 within the vehicle interior 105. When projected onto the 2D coordinate space of the optical image sensor 106, parameters affecting how the 3D volume of the vehicle interior 105 is presented include external and internal parameters. External parameters may refer to factors describing the physical orientation of the sensor, such as rotation and translation (also known as roll and tilt) and / or other parameters. Internal parameters may refer to factors describing the optical characteristics of the sensor device, such as optical center (also known as principal point), focal length, skew coefficient, field of view, and / or other parameters. While the sensor's internal parameters can be established during manufacturing and are expected to remain stable, the external parameters of rotation and translation depend on how the sensor is mounted and oriented within the vehicle cabin space. Both external and internal parameters can play a role in how scene features within the 3D coordinate space of the vehicle cabin are mapped to image frames captured by the sensor. In some embodiments, sensor calibration parameters 305 include a rotation-translation (RT) transformation describing the external relationship between the coordinate system of vehicle geometry 310 and the optical image sensor 106. That is, sensor calibration parameters 305 may include an RT transformation that takes into account the rotation and translation of the optical image sensor 106 during mounting, thereby potentially mapping the 3D cabin coordinate system of the vehicle interior 105 as described by vehicle geometry 310 to the 2D coordinate space of the field of view of the optical image sensor 106. Since the vehicle geometry specification 310 includes data indicating the 3D coordinates of structural elements within the vehicle interior 105, these 3D coordinates can be mapped to the coordinate space of the optical image sensor 106 using a Real-Time Transform (RT) transformation. As an example, the calibration process (e.g., a factory calibration procedure) for generating ground truth depth data 108 may involve inputting the vehicle geometry specification 310 and sensor calibration parameters 305 into the depth rendering function 320.Because it is observed by the optical image sensor 106, the depth rendering function 320 may determine the field of view of the vehicle compartment, which may be based at least on a combination of sensor internal parameters, external parameters, and / or mounting locations within the vehicle, and synthesized ground truth depth data 108 (such as a ground truth compartment depth image 109) specifying the depth data of structural elements of the vehicle interior 105 appearing in the sensor's field of view. The ground truth compartment depth image 109 may include a distance image, where the pixel values ​​of the distance image correspond to the distances from the optical image sensor 106 to those structural elements of the vehicle compartment as predicted by the depth rendering function 320 and appearing in the field of view of the optical image sensor 106. The representations of the ground truth depth data 108 and / or the ground truth compartment depth image 109 may then be programmed into the memory of the vehicle 800 and used by the optical depth estimation model 122 to scale the relative dimensions of structural elements and / or occupant kinematic elements to absolute dimensions, as described herein.

[0056] In some embodiments, the occupant feature scaling function 130 may use a ray tracing algorithm to transform the scale-normalized 3D pose 224 into an absolute 3D pose estimate 240 to map pixels identified in the joint depth data 232 in the distance image back to pixels in the image frame from the optical image data 107, and determine the positions where pixels corresponding to the kinematic elements of the scale-normalized 3D pose 224 interact with pixels in the distance image. For example, the occupant feature scaling function 130 may select preferred kinematic elements (e.g., body joints) observable from the optical image data 107 to define as anchor joints. The occupant feature scaling function 130 may contain a predetermined priority list of body joints and may select observable body joints from the optical image data 107 to define as anchor joints based on this priority list (e.g., joints indicated as having a preferred priority). For example, neck, hip, or other trunk body joints often perform well as preferred anchor joints because these joints are typically limited in their range of motion and position for a seated vehicle occupant (e.g., compared to joints such as the elbow and / or wrist). Therefore, 3D pose detection models 222 are generally more likely to reliably identify these kinematic elements than limbs. For example... Figure 4As shown at 400, the occupant feature scaling function 130 may apply a ray tracing algorithm to calculate a ray 410 that extends from the center 415 of the optical image sensor 106 through the 2D pixel coordinates (u, v) of the image frame 425 corresponding to the selected anchor joint 420 (e.g., from optical image data 107). The ray tracing algorithm may determine pixels and / or pixel clusters 430 from the joint depth data 232 closest to the ray 410 and assign depth values ​​435 to the selected anchor joint 420 based on the depth indicated by the depth of one or more pixels closest to the ray 410. Based on the absolute depth of the selected anchor joint 420 and the relative positions of other kinematic elements indicated by the scale-normalized 3D pose 224, the occupant feature scaling function 130 may scale one or more other kinematic elements of the scale-normalized 3D pose 224 as a function of their relative positions and distances to the anchor joint.

[0057] In some embodiments, the depth of multiple different detected body joints observable from the optical image data 107 may be identified by extending multiple rays 420 using a ray tracing algorithm, thereby selecting and evaluating multiple anchored joints 420. The occupant feature scaling function 130 may apply an optimization algorithm to optimally correlate the distances determined by the ray tracing algorithm of the multiple rays and assign depths to different body joints. In some embodiments, joint depth data 232 may be generated by accumulating optical image sensor data 107 over a predefined duration to obtain a more stable representation of the vehicle occupant.

[0058] Refer again Figure 2Occupant evaluation function 110 may continue to calculate 3D representation data 140, which includes at least one characteristic representing occupant size. Because the 3D coordinates of kinematic elements in absolute 3D pose 240 can be directly used to determine absolute distances, the occupant's body limb lengths may be calculated directly from these 3D coordinates. For example, the width of the occupant's torso, as indicated by absolute 3D pose estimation 240, may be estimated by calculating the distance between the 3D coordinates of the occupant's left and right shoulder joints. Therefore, by summing various body limb lengths, occupant evaluation function 110 may estimate the overall 3D dimensions of the occupant that may be included in the 3D occupant representation data 140. In some embodiments, individual limb lengths may be calculated as the distance between two consecutively connected joints. In some embodiments, composite limb lengths may be calculated as a function of a combination of kinematic elements. For example, the occupant's wingspan, extending from the occupant's left wrist to the occupant's right wrist when the occupant's arms are extended, may define a kinematic element that includes a combination of individual limbs and a composite limb length for calculating the wingspan. In some embodiments, the occupant evaluation function 110 uses pre-programmed statistics on human proportions to estimate body dimensions that cannot be directly derived from the absolute 3D pose. For example, statistically, a person's height is approximately 1.1 times their wingspan. Therefore, if the absolute 3D pose 240 lacks kinematic information about the occupant below the torso (e.g., if parts of their body are occluded in the image frame), the occupant evaluation function 110 may refer to statistics on human proportions, such as estimating the occupant height based on their derived wingspan length multiplied by 1.1.

[0059] In some embodiments, a 3D pose detection model 222 may be trained to generate scale-normalized 3D poses 224 based on synchronized multi-view images of the training object. Restricting the 3D pose detection model 222 to the task of predicting 3D poses at relative scales allows it to output more accurate predictions by preventing it from learning to predict ill-posed estimates of depth information directly from a single 2D image. In some embodiments, the 3D pose detection model 222 may be trained on multi-view training based on real-world data obtained by capturing synchronized sets of 2D images (e.g., as described in U.S. Patent Application No. 18 / 349,842, (Attorney General's File No. 22-SC-1598 / 396522), filed concurrently with this document, entitled "Training a Pose Detection Model for Predicting 3D Pose Estimation Using Two-Dimensional Image Data," the entire contents of which are incorporated herein by reference) to generate a 3D perception model from 2D image data. In some embodiments, the 3D pose detection model 222 may further, or alternatively, be trained using synthetic ground truth 3D pose data. The synthetic 3D pose ground truth data allows the 3D pose detection model 222 to be trained from data capturing a wide variety of demographic data and simulated training objects in various vehicles. Furthermore, the synthetic data essentially provides complete 3D pose ground truth, which facilitates the ability to directly supervise the training process of the 3D pose detection model 222.

[0060] Reference Figure 5 , Figure 5 This is an example data flow diagram 500 of a system 500 for an image-based 3D occupant assessment according to some embodiments of the present disclosure, for example, a system based on 3D shape. Figure 1 As shown. In Figure 5 In the illustrated embodiment, the occupant evaluation function 110 processes optical image data 107 from the optical image sensor 106 to derive 3D occupant representation data 140 containing 3D shape estimates of the vehicle occupants. Here, the occupant feature detection model 120 may include a segmentation model 520 that uses the optical image data 107 to generate an occupant mask 522 and a vehicle interior mask 524. For example, the segmentation model 520 may segment the optical image data 107 based on inferring that pixels in image frames representing the vehicle occupants are distinguished from pixels in image frames not representing the vehicle occupants. Based on this inference, the segmentation model 520 may generate an occupant mask 520 (e.g., defining the outlines of pixels corresponding to occupants) and a vehicle interior mask 524 (e.g., defining the outlines of pixels corresponding to structural elements of the vehicle interior 105).

[0061] In such an embodiment, the optical depth estimation model 122 may include a structure element depth estimator model 530 that outputs structure element depth data 542. For example, the segmentation model 520 and / or the structure element depth estimator model 530 may be implemented using a neural network architecture (such as CNN, DNN, or other machine learning model architectures).

[0062] The structural element depth estimator model 530 may output structural element depth data 542 in the form of a distance image based on input including image frames from optical image data 107. The pixel values ​​of this distance image may correspond to the distances from optical image sensor 106 to structural features of the vehicle interior 105 sensed from the field of view of a sensor such as those appearing in optical image data 107. In these embodiments, the occupant feature scaling function 130 functions as a 3D shape estimator to generate a 3D shape estimate 550 of the occupant based on pixels of the image frame scaled using the structural element depth data 542, where each pixel represents a feature of the occupant captured within the boundaries of the occupant mask 522.

[0063] In some embodiments, the structural element depth estimation model 530 may learn from the vehicle interior mask 524 which pixels in the optical image data 107 are components of the vehicle interior 105 (as opposed to pixels representing occupants and / or other (one or more) objects located within the vehicle interior). Based on those pixels identified as vehicle interior components, the structural element depth estimation model 530 may infer distances based on ground truth depth data 108 to determine the distances between various structural interior elements appearing in the optical image data 107. In such embodiments, the structural element depth data 542 output from the structural element depth estimation model 530 may include a distance image, where pixel values ​​of the distance image indicate the distance from the optical image sensor 106 to the surface of a structural element of the vehicle interior 105 appearing in the optical image data 107. For example, the structural element depth data 542 generated by the structural element depth estimation model 530 may indicate a given distance d1 between a structural component within the vehicle compartment (e.g., the location of a feature on the vehicle compartment roof) and the OMS optical image sensor (e.g., in linear units of measurement).

[0064] To determine the 3D shape of an occupant, the occupant evaluation function, occupant feature scaling function 130, may take into input an occupant mask 524 generated by segmentation model 520 and structuring element depth data 542 generated by structuring element depth estimation model 530. Using the occupant mask 524 and structuring element depth data 542, the occupant feature scaling function 130 performs 3D shape estimation to infer that one or more pixels representing a first position corresponding to the surface of the occupant's body are closer to the optical image sensor 106 by a relative distance d2 than pixels corresponding to one or more structural components of the vehicle interior 105, and calculates the absolute distance of the occupant's body surface from the optical image sensor 106 based on a function of distances d1 and d2. The occupant feature scaling function 130 may further infer the relative positions of one or more other positions on the occupant's body to the first position from the occupant mask 522, and since an absolute 3D position has been established for the first position of the occupant's body, the absolute distance from the first position to one or more other positions is determined, thus estimating the overall 3D shape of the occupant. Therefore, based on the 3D shape estimation 550, the occupant evaluation function 110 may estimate the overall 3D dimensions of the occupants that may be included in the 3D occupant representation data 140.

[0065] Figure 6 Figure 600 illustrates the use of attitude detection combined with 3D occupant assessment based on 3D dimensions to compute 3D occupant representation data 140 according to some embodiments of the present disclosure. In some embodiments, the 3D shape estimation performed by the occupant feature scaling function 130 may be further input to an occupant attitude estimate (e.g., 2D attitude estimate or 3D attitude estimate) generated by further applying optical image data 107 to the attitude detection model 610 to generate a 3D shape estimate 550 of the vehicle occupant. For example, the occupant feature scaling function 130 may be input to a scale-normalized 3D attitude estimate, such as that discussed above. In such embodiments, the occupant feature scaling function 130 may use kinematic elements (e.g., body limbs and / or joints) represented in the attitude estimate, and these kinematic elements appear in the occupant mask 522 as the position of the occupant body, from which the relative position used to estimate the overall 3D shape may be inferred. The occupant feature scaling function 130 may determine that a first position corresponding to a first kinematic element is closer to the optical image sensor 106 by a relative distance d2 than a pixel corresponding to a structural component of the vehicle interior 105 (e.g., based on structural element depth data 542), and calculate the absolute distance between the kinematic element and the optical image sensor 106 based on a function of distances d1 and d2. The occupant feature scaling function 130 may infer the relative positions of one or more other kinematic elements to the first kinematic element, determine the absolute distances between the kinematic elements, and thus estimate the overall 3D shape of the occupant.

[0066] Now for reference Figure 7 , Figure 7 This is a flowchart illustrating a method 700 for an image-based three-dimensional occupant assessment system according to some embodiments of the present disclosure. It should be understood that the details herein refer to… Figure 7 The features and elements described in method 700 may be used in combination with, combined with, or substituted for elements of any of the other embodiments discussed herein, and vice versa. Furthermore, it should be understood that the features and elements used in... Figure 7 The functionality, structure, and other descriptions of the elements of the embodiments described herein may be applied to similar or similar named or described elements across any figures and / or embodiments described herein, and vice versa.

[0067] Each block of method 700 described herein contains a computational process that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. These methods may also be embodied as computer-usable instructions stored on a computer storage medium. These methods may be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or plug-in to another product, to name a few. Furthermore, method 700 is described by way of example with respect to an image-based three-dimensional occupant assessment system in any of the accompanying drawings herein. However, these methods may be additionally or alternatively performed by any system or combination of systems, including but not limited to the systems described herein.

[0068] Figure 7 An example method 700 is shown for generating a three-dimensional representation of the dimensions of a vehicle occupant based on at least one or more features, the features representing at least a portion of the occupant derived from two-dimensional optical image data and the depth of one or more features generated from the two-dimensional optical imaging data, as well as a ground truth cabin depth image corresponding to the vehicle interior.

[0069] At box B702, method 700 includes detecting one or more features representing at least a portion of a vehicle occupant based at least on optical image data. In some embodiments, detecting one or more features may include performing at least one three-dimensional pose detection model to generate a scale-normalized three-dimensional pose estimate of the occupant based at least on one or more features. For example, an occupant evaluation function may apply optical image data to at least one person detection model to define a portion of an image frame representing the occupant and generate a scale-normalized three-dimensional pose estimate based at least on the portion of the image frame representing the occupant. In some embodiments, detecting one or more features may include generating a scale-normalized three-dimensional pose estimate of the occupant based at least on one or more features, wherein the scale-normalized three-dimensional pose estimate represents a set of kinematic elements of the occupant. In some embodiments, detecting one or more features may include performing a segmentation model, such as segmentation model 520. In such embodiments, detecting (one or more) features may include generating one or both of a vehicle interior mask and / or an occupant mask based at least on segmentation of optical image data. The vehicle interior mask may include a first boundary that delineates pixels corresponding to structural elements within the vehicle interior. The occupant mask may include a second boundary that delineates pixels corresponding to one or more features of the occupant.

[0070] At box B704, method 700 includes determining a depth corresponding to one or more features based at least on optical image data and a ground truth cabin depth image corresponding to the vehicle interior. For example, the method may generate a distance image based at least on the optical image data and the ground truth cabin depth image, wherein the depth corresponding to one or more features is derived at least from the distance image. In some embodiments, determining the depth may include particle transport simulation (e.g., ray tracing, for example...). Figure 4 As shown in the diagram, at least a first pixel of the ground truth vehicle depth image is associated with one or more features to determine the depth corresponding to one or more features. In some embodiments, a vehicle interior mask may be used at least in part to generate the distance image. In some embodiments, an occupant's three-dimensional shape estimate may be determined at least based on an occupant's pose estimate generated from optical image data.

[0071] At box B706, method 700 includes generating a three-dimensional representation of an occupant based on at least one or more features and the depth of one or more features, wherein the three-dimensional representation includes at least one characteristic representing the occupant's size. The at least one characteristic may include a three-dimensional shape estimate and / or a three-dimensional attitude estimate. In some embodiments, the at least one characteristic representing the occupant's size is determined based at least on a three-dimensional attitude estimate at a linear measurement scale. In some embodiments, the three-dimensional representation may include a three-dimensional size estimate derived from the three-dimensional shape estimate and / or the three-dimensional attitude estimate. In some embodiments, the characteristic representing the occupant's size may be further used to derive other characteristics of the occupant, such as, but not limited to, weight, height, and / or age.

[0072] At box B708, method 700 includes controlling at least one operation of the vehicle based on at least one characteristic. For example, based at least in part on at least one characteristic representing occupant size, interior monitoring system 150 may generate one or more outputs 154. For example, interior monitoring system 150 may use 3D occupant representation data 140 (alone or in combination with other data such as optical image data 107) to predict the presence and / or location of occupants—such as objects, people, and / or animals—within the interior space of vehicle 800. Other systems of vehicle 800 may determine one or more actions to be taken based on other tasks or operations included in the predicted and / or controlled outputs 154. For example, based on outputs 154, alarms or warnings may be generated, door locks and / or windows may be operated, various functions may be opened / closed, data on digital assistants, chatbots, digital avatars, and / or the like may be generated, and / or air conditioning or air recirculation functions may be operated. In some embodiments, characteristics representing occupant size from the 3D occupant representation data 140 may be used in conjunction with a child presence detection system to estimate the occupant's age, and / or to control an alarm system (e.g., an alarm and / or notification system) based on the determination that an occupant below an estimated age threshold may have been unintentionally left alone in the vehicle. In some embodiments, characteristics representing occupant size may be used in conjunction with other vehicle safety features. For example, in some embodiments, airbag deployment, driver monitoring systems, HMI applications, and / or other vehicle functions may be controlled based at least on estimates of the vehicle occupant's 3D posture, 3D shape, and / or 3D size provided by the 3D occupant representation data 140.

[0073] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, airships, boats, space shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, trains, underwater vehicles, remotely controlled vehicles (e.g., drones), and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, including, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and monitoring, simulation, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twins, data center processing, conversational AI (artificial intelligence), generative AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0074] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems containing one or more virtual machines (VMs), systems for performing generative AI operations using language models, systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0075] Example autonomous vehicles

[0076] Figure 8AThis is an illustration of an example autonomous vehicle 800 according to some embodiments of the present disclosure. The autonomous vehicle 800 (or, alternatively, referred to herein as “vehicle 800”) may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police vehicles, ambulances, boats, construction vehicles, underwater vessels, robotic vehicles, drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer trucks for hauling goods) and / or other types of vehicles (e.g., driverless and / or vehicles accommodating one or more occupants). Autonomous vehicles are typically described according to the levels of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) in its "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 800 may be able to perform one or more functions corresponding to Levels 3 through 5 of the autonomous driving level. Vehicle 800 may be able to perform one or more functions according to Levels 1 through 5 of the autonomous driving level. For example, depending on the embodiment, vehicle 800 may be able to perform driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). As used herein, the term “autonomy” can include any and / or all types of autonomy for the vehicle 800 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, providing auxiliary autonomy, semi-autonomy, primary autonomy, or other names.

[0077] Vehicle 800 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 800 may include a propulsion system 850, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 850 may be connected to the drivetrain of vehicle 800, which may include a transmission, to enable propulsion of vehicle 800. Propulsion system 850 may be controlled in response to receiving a signal from throttle / accelerator 852.

[0078] A steering system 854, which may include a steering wheel, can be used to steer the vehicle 800 (e.g., along a desired path or route) when the propulsion system 850 is operating (e.g., when the vehicle is in motion). The steering system 854 may receive signals from the steering actuator 856. For fully automatic (level 5) functionality, the steering wheel may be optional.

[0079] The brake sensor system 846 can be used to operate the vehicle brakes in response to receiving signals from the brake actuator 848 and / or the brake sensor.

[0080] It can include one or more System-on-a-Chip (SoC) 804 ( Figure 8C One or more controllers 836, including one or more GPUs, may provide signals (e.g., indicating commands) to one or more components and / or systems of vehicle 800. In some embodiments, one or more operations performed by controllers 836 may be performed in response to one or more outputs 154 generated by internal monitoring system 150. For example, one or more controllers may send signals to operate vehicle brakes via one or more brake actuators 848, to operate steering system 854 via one or more steering actuators 856, and to operate propulsion system 850 via one or more throttles / accelerators 852. One or more controllers 836 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals indicating commands) to enable autonomous driving and / or assist a human driver in driving vehicle 800. One or more controllers 836 may include a first controller 836 for autonomous driving functions, a second controller 836 for functional safety functions, a third controller 836 for artificial intelligence functions (e.g., computer vision), a fourth controller 836 for infotainment functions, a fifth controller 836 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 836 may handle two or more of the above functions, two or more controllers 836 may handle a single function, and / or any combination thereof.

[0081] One or more controllers 836 may provide signals for controlling one or more components and / or systems of vehicle 800 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, Global Navigation Satellite System (“GNSS”) sensor 858 (e.g., Global Positioning System sensor), RADAR sensor 860, ultrasonic sensor 862, LIDAR sensor 864, Inertial Measurement Unit (IMU) sensor 866 (e.g., accelerometer, gyroscope, magnetic compass, magnetometer, etc.), microphone 896, stereo camera 868, wide-angle camera 870 (e.g., fisheye camera), infrared camera 872, surround camera 874 (e.g., 360-degree camera), long-range and / or medium-range camera 898, speed sensor 844 (e.g., for measuring the rate of vehicle 800), vibration sensor 842, steering sensor 840, braking sensor (e.g., as part of braking sensor system 846), one or more Occupant Monitoring System (OMS) sensor 801 (e.g., one or more interior cameras) and / or other sensor types.

[0082] One or more controllers 836 may receive inputs (e.g., represented by input data) from the instrument cluster 832 of the vehicle 800 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 834, an auditory signaling device, a speaker, and / or via other components of the vehicle 800. These outputs may include information such as vehicle speed, rate, time, map data (e.g., [missing information]). Figure 8C Information such as high-definition (“HD”) maps 822, location data (e.g., the location of vehicle 800 on the map), orientation, and the location of other vehicles (e.g., occupying grids), as well as information about objects and their states perceived by controller 836. For example, HMI display 834 may display information about the existence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0083] The vehicle 800 also includes a network interface 824, which can communicate via one or more networks using one or more wireless antennas 826 and / or a modem. For example, the network interface 824 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), etc. One or more wireless antennas 826 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc., and / or one or more low-power wide area networks (“LPWAN”) such as LoRaWAN, SigFox, etc.

[0084] Figure 8B For use in accordance with some embodiments of this disclosure Figure 8A This is an example of the camera position and field of view of an example autonomous vehicle 800. The camera and its respective field of view are an example embodiment and are not intended to be limiting. For example, additional and / or replaceable cameras may be included, and / or these cameras may be located at different positions on the vehicle 800.

[0085] The camera type used for the camera may include, but is not limited to, a digital camera suitable for use with components and / or systems of vehicle 800. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The camera may be able to use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a sharp-pixel camera, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to improve light sensitivity.

[0086] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0087] One or more of the cameras can be mounted in mounting components such as custom-designed (3D-printed) components to cut off stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capabilities. Regarding the wing mirror mounting components, the wing mirror components can be custom-3D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0088] A camera with a field of view that includes the environment in front of the vehicle 800 (e.g., a front-facing camera) can be used for surround view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 836 and / or control SoCs, to provide information crucial for generating an occupancy grid and / or determining a preferred vehicle path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including Lane Departure Warning (“LDW”), Autonomous Cruise Control (“ACC”), and / or other functions such as traffic sign recognition.

[0089] A variety of cameras can be used in front-facing configurations, including, for example, monocular camera platforms including complementary metal-oxide-semiconductor (“CMOS”) color imagers. Another example could be a wide-angle camera 870, which can be used to perceive objects entering the field of view from the periphery (such as pedestrians, traffic at intersections, or bicycles). Although Figure 8B The middle image shows only one wide-angle camera, but any number (including zero) of wide-angle cameras 870 can be present on vehicle 800. Furthermore, any number of remote cameras 898 (e.g., long-view stereo camera pairs) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. Remote cameras 898 can also be used for object detection and classification, as well as basic object tracking.

[0090] Any number of stereo cameras 868 may also be included in the front-mounted configuration. In at least one embodiment, one or more stereo cameras 868 may include an integrated control unit comprising a scalable processing unit that can provide a multi-core microprocessor and programmable logic (“FPGA”) with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 868 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip capable of measuring the distance from the vehicle to a target object and using the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 868 may be used in addition to those described herein or alternatively.

[0091] Cameras with a field of view including the side portion of the vehicle 800 (e.g., side-view cameras) can be used for surround view, providing information for creating and updating occupancy grids and generating side-impact collision warnings. For example, surround camera 874 (e.g., ... Figure 8B The four surround cameras 874 shown can be mounted on the vehicle 800. The surround cameras 874 can include wide-angle cameras 870, fisheye cameras, 360-degree cameras, and / or the like. Four examples are provided; the four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 874 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround-view camera.

[0092] A camera with a field of view that includes the environment behind the vehicle 800 (e.g., a rear-view camera) can be used for assisted parking, surround view, rear collision warning, and creating and updating occupancy grids. A wide variety of cameras can be used, including but not limited to those also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range camera 898, stereo camera 868, infrared camera 872, etc.).

[0093] Cameras (e.g., one or more OMS sensors 801) that encompass a portion of the interior environment of the vehicle 800's passenger compartment can be used as part of an Occupant Monitoring System (OMS), such as, but not limited to, a Driver Monitoring System (DMS). For example, the OMS sensor (e.g., OMS sensor 801) can (e.g., by controller 836) track the gaze direction, head posture, and / or blinking of an occupant and / or driver. This gaze information can be used to determine the occupant's or driver's level of attention (e.g., detecting drowsiness, fatigue, and / or distraction), and / or to take responsive actions to prevent harm to the occupant or operator. In some embodiments, data from the OMS sensor can be used to implement gaze control operations triggered by the driver and / or non-driver occupants, such as, but not limited to, adjusting the passenger compartment temperature and / or airflow, opening and closing windows, controlling passenger compartment lighting, controlling the entertainment system, adjusting rearview mirrors, adjusting seat position, and / or other operations. In some embodiments, the OMS can be used for applications such as determining when an object and / or occupant remains in the passenger compartment (e.g., by detecting the presence of an occupant after the driver has left the vehicle).

[0094] Figure 8C For use in accordance with some embodiments of this disclosure Figure 8A The example autonomous vehicle 800 is illustrated in the block diagram of an example system architecture. It should be understood that this arrangement, and other arrangements described herein, are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities, which may be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by these entities can be implemented via hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in memory.

[0095] Figure 8C Each component, feature, and system in vehicle 800 is illustrated as being connected via bus 802. Bus 802 may include a Controller Area Network (CAN) data interface (or, alternatively, referred to herein as the "CAN bus"). CAN may be a network within vehicle 800 used to assist in the control of various features and functions of vehicle 800, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0096] Although bus 802 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to a CAN bus. Furthermore, although bus 802 is represented by a single line, this is not intended to be limiting. For example, any number of buses 802 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 802 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 802 may be used for collision avoidance functions, and a second bus 802 may be used for drive control. In any example, each bus 802 may communicate with any component of vehicle 800, and two or more buses 802 may communicate with the same component. In some examples, each SoC 804, each controller 836, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors of vehicle 800) and may be connected to a common bus such as a CAN bus.

[0097] Vehicle 800 may include one or more controllers 836, such as those described herein. Figure 8A The controllers described herein. Controller 836 can be used for a wide variety of functions. Controller 836 can be coupled to any other different components and systems of vehicle 800 and can be used for the control of vehicle 800, artificial intelligence of vehicle 800, infotainment and / or the like for vehicle 800.

[0098] Vehicle 800 may include one or more System-on-Chip (SoC) 804s. SoC 804 may include a CPU 806, GPU 808, processor 810, cache 812, accelerator 814, data storage 816, and / or other components and features not shown. SoC 804 can be used to control vehicle 800 across a wide variety of platforms and systems. For example, one or more SoCs 804s may be combined with an HD map 822 in a system (e.g., the system of vehicle 800), the HD map being transmitted via a network interface 824 from one or more servers (e.g., [server name missing]). Figure 8D One or more servers (878) receive map refresh and / or updates.

[0099] The CPU 806 may include CPU clusters or CPU complexes (or, alternatively, referred to herein as "CCPLEX"). The CPU 806 may include multiple cores and / or L2 cache. For example, in some embodiments, the CPU 806 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 806 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 2MB L2 cache). The CPU 806 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of clusters of the CPU 806 can be active at any given time.

[0100] The CPU 806 can implement power management capabilities including one or more of the following features: automatic clock gating of hardware blocks when idle to conserve dynamic power; clock gating of each core when the core is not actively executing instructions due to the execution of WFI / WFE instructions; independent power gating of each core; independent clock gating of each core cluster when all cores are clock-gated or power-gated; and / or independent power gating of each core cluster when all cores are power-gated. The CPU 806 can further implement enhanced algorithms for managing power states, where allowed power states and desired wake-up times are specified, and the hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core can support simplified power state entry sequences in software, with this work offloaded to the microcode.

[0101] The GPU 808 may include an integrated GPU (or, alternatively, referred to herein as an "iGPU"). The GPU 808 may be programmable and efficient for parallel workloads. In some examples, the GPU 808 may use an enhanced tensor instruction set. The GPU 808 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, the GPU 808 may include at least eight streaming microprocessors. The GPU 808 may use a computation application programming interface (API). Furthermore, the GPU 808 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0102] In automotive and embedded applications, the GPU 808 can be power-optimized for optimal performance. For example, the GPU 808 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 808 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can combine several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, dispatch units, and / or a 64KB register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to leverage the mixture of computation and addressing computations for efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can include a combination of L1 data cache and shared memory units to improve performance while simplifying programming.

[0103] The GPU 808 may include, in some examples, a High Bandwidth Memory (HBM) and / or a 16GB HBM2 memory subsystem providing a peak memory bandwidth of approximately 900GB / s. In some examples, in addition to HBM memory or alternatively, Synchronous Graphics Random Access Memory (SGRAM), such as Generation 5 Graphics Double Data Rate Synchronous Random Access Memory (GDDR5), may be used.

[0104] The GPU 808 may include unified memory technology, which includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support can be used to allow the GPU 808 to directly access the CPU 806 page tables. In such examples, when the GPU 808 Memory Management Unit (MMU) experiences a miss, the address translation request can be transferred to the CPU 806. In response, the CPU 806 can look up the virtual-physical mapping for the address in its page tables and transfer the translation back to the GPU 808. Thus, unified memory technology can allow a single unified virtual address space for the memory of both the CPU 806 and GPU 808, simplifying GPU 808 programming and porting applications to the GPU 808.

[0105] In addition, the GPU 808 may include access counters that track how frequently the GPU 808 accesses the memory of other processors. These access counters help ensure that memory pages are moved to the physical memory of the processor that accesses those pages most frequently.

[0106] SoC 804 may include any number of caches 812, including those described herein. For example, cache 812 may include an L3 cache available to both CPU 806 and GPU 808 (e.g., it is connected to both CPU 806 and GPU 808). Cache 812 may include a write-back cache, which can track the state of rows, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4 MB or more, but a smaller cache size may also be used.

[0107] SoC 804 may include an arithmetic logic unit (ALU) that can be utilized in processing of any of the various tasks or operations performed with respect to vehicle 800, such as processing a DNN. Furthermore, SoC 804 may include a floating-point unit (FPU) (or other mathematical coprocessor or digital coprocessor type) for performing mathematical operations within the system. For example, SoC 804 may include one or more FPUs integrated as execution units within CPU 806 and / or GPU 808.

[0108] SoC 804 may include one or more accelerators 814 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 804 may include a hardware accelerator cluster, which may include optimized hardware accelerators and / or large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to complement GPU 808 and offload some tasks from GPU 808 (e.g., freeing up more cycles of GPU 808 to perform other tasks). As an example, accelerator 814 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. When used herein, the term "CNN" can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0109] Accelerator 814 (e.g., a hardware accelerator cluster) may include a Deep Learning Accelerator (DLA). A DLA may include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional trillion operations per second for deep learning applications and inference. TPUs may be accelerators configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for performing image processing functions. DLAs may be further optimized for a specific set of neural network types and floating-point operations as well as inference. DLAs are designed to provide higher performance per millimeter than general-purpose GPUs and significantly outperform CPUs. TPUs can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0110] DLA can execute neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for any function across a wide variety of applications, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition using data from microphones; CNNs for face recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for safety and / or safety-related events.

[0111] The DLA can perform any function of the GPU 808, and by using an inference accelerator, for example, the designer can make the DLA or GPU 808 targeted at any function. For example, the designer can focus the CNN processing and floating-point operations on the DLA and leave other functions to the GPU 808 and / or other accelerators 814. In some embodiments, one or more models of the occupant evaluation function 110 disclosed above may be performed by the SoC 804, (one or more) CPU 806, (one or more) GPU 808 and / or accelerator 814.

[0112] Accelerator 814 (e.g., a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. A PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. A PVA can provide a balance between performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0113] RISC cores can interact with image sensors (such as the image sensor of any camera described herein), image signal processors, and / or the like. Each of these RISC cores may include any amount of memory. Depending on the embodiment, the RISC core may use any of several protocols. In some examples, the RISC core may execute a real-time operating system (RTOS). RISC cores may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0114] DMA enables PVA components to access system memory independently of the CPU 806. DMA can support any number of features to provide optimizations to the PVA, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0115] A vector processor can be a programmable processor designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as, for example, a Single Instruction Multiple Data (SIMD) or Very Long Instruction Word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0116] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. As a result, in some examples, each of the vector processors may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a cluster of hardware accelerators, and any number of vector processors may be included in each of these PVAs. Furthermore, the PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0117] Accelerator 814 (e.g., a hardware accelerator cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for accelerator 814. In some examples, on-chip memory may include at least 4MB of SRAM consisting of, for example, but not limited to, eight field-configurable memory blocks, accessible by both the PVA and DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA may access memory via a backbone that provides high-speed memory access to the PVA and DLA. The backbone may include (e.g., using an APB) an on-chip computer vision network that interconnects the PVA and DLA to memory.

[0118] On-chip computer vision networks can include interfaces that ensure both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such interfaces can provide separate phases and channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0119] In some examples, SoC 804 may include, for example, a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. This real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison with LiDAR data for localization and / or other functional purposes, and / or other uses. In some embodiments, one or more Tree Traversal Units (TTUs) may be used to perform one or more ray tracing-related operations.

[0120] Accelerators 814 (e.g., hardware accelerator clusters) have broad applications in autonomous driving. PVAs can be programmable vision accelerators used in critical processing stages of ADAS and autonomous vehicles. The capabilities of PVAs are a good match for algorithmic domains requiring predictable processing, low power, and low latency. In other words, PVAs perform well in semi-dense or dense rule computation, even on small datasets requiring predictable runtimes with low latency and low power. Therefore, in the context of platforms for autonomous vehicles, PVAs are designed to run classical computer vision algorithms because they are efficient in object detection and integer mathematical operations.

[0121] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. In some examples, semi-global matching-based algorithms may be used, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., from moving structures, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0122] In some examples, PVA can be used to perform intensive optical flow, providing processed RADAR data from the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, PVA is used for time-of-flight depth processing, which, for example, involves processing raw time-of-flight data to provide processed time-of-flight data.

[0123] DLA can be used to run any type of network to enhance control and driving safety, including, for example, neural networks that output a confidence metric for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative “weight” for each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a threshold for the confidence and only consider detections exceeding the threshold as true positives. In an Automatic Emergency Braking (AEB) system, false positives can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network to regress the confidence value. This neural network can take at least a subset of parameters as input, such as bounding box dimensions, ground plane estimates (e.g., from another subsystem), inertial measurement unit (IMU) sensor 866 outputs related to vehicle orientation and distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 864 or RADAR sensor 860), etc.

[0124] The SoC 804 may include one or more data storage units 816 (e.g., memory). The data storage unit 816 may be on-chip memory of the SoC 804, which may store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and security, the data storage unit 816 may be large enough to store multiple instances of the neural network. The data storage unit 816 may include L2 or L3 cache 812. References to the data storage unit 816 may include references to memory associated with the PVA, DLA, and / or other accelerators 814 as described herein.

[0125] SoC 804 may include one or more processors 810 (e.g., embedded processors). Processor 810 may include a startup and power management processor, which may be a dedicated processor and subsystem for handling startup power and management functions, as well as safety implementation. The startup and power management processor may be part of the SoC 804 startup sequence and may provide runtime power management services. The startup power and management processor may provide clock and voltage programming, auxiliary system low-power state transitions, SoC 804 thermal and temperature sensor management, and / or SoC 804 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 804 may use the ring oscillator to detect the temperature of CPU 806, GPU 808, and / or accelerator 814. If it is determined that the temperature exceeds a threshold, the startup and power management processor may enter a temperature fault routine and place SoC 804 into a lower power state and / or place vehicle 800 into a driver-safe parking mode (e.g., safely stop vehicle 800).

[0126] The processor 810 may also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio via multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0127] The processor 810 may also include an always-on-processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. This always-on-processor engine may include a processor core, tightly coupled RAM, support for peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0128] The processor 810 may also include a security cluster engine, which comprises a dedicated processor subsystem for handling security management for automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic that detects any differences between their operations.

[0129] The processor 810 may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0130] The processor 810 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0131] Processor 810 may include a video image compositer, which may be (e.g., implemented on a microprocessor) a processing block, implementing video post-processing functions required by the video playback application to generate the final image for the player window. The video image compositer may perform lens distortion correction on the wide-angle camera 870, the surround camera 874, and / or on in-cabin monitoring camera sensors (such as OMS sensor 801). The in-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of an advanced SoC, configured to recognize in-cabin events and respond accordingly. The in-cabin system may perform lip reading to activate mobile phone services and make calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. Some functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other situations.

[0132] Video image compositers can include enhanced temporal denoising for both spatial and temporal noise reduction. For example, in the case of motion in the video, denoising appropriately weights spatial information, reducing the weight of information provided by neighboring frames. In cases where the image or part of the image does not contain motion, the temporal denoising performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0133] The video image compositer can also be configured to perform stereo correction on input stereo camera frames. When the operating system desktop is in use and the GPU 808 does not need to continuously render new surfaces, the video image compositer can be further used for user interface components. Even when the GPU 808 is powered on and active, performing 3D rendering, the video image compositer can be used to offload the GPU 808 to improve performance and responsiveness.

[0134] The SoC 804 may also include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and pixel-related input functions for receiving video and input from a camera. The SoC 804 may also include an input / output controller that can be software-controlled and can be used to receive I / O signals not assigned to a specific role.

[0135] SoC 804 may also include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management and / or other devices. SoC 804 can be used to process data from cameras and sensors (e.g., LIDAR sensor 864, RADAR sensor 860, etc., which can be connected via Gigabit Multimedia Serial Link and Ethernet), data from bus 802 (e.g., vehicle 800 speed, steering wheel position, etc.), and data from GNSS sensor 858 (connected via Ethernet or CAN bus). SoC 804 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and can be used to free up CPU 806 from routine data management tasks.

[0136] The SoC 804 can be an end-to-end platform with a flexible architecture spanning Automation Levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS technologies for diversity and redundancy, along with deep learning tools to deliver a flexible and reliable driving software stack. The SoC 804 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, when combined with the CPU 806, GPU 808, and data storage 816, the accelerator 814 can provide a fast and efficient platform for Level 3-5 autonomous vehicles.

[0137] Therefore, this technology offers capabilities and functionalities that cannot be achieved through conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as C to execute a wide variety of processing algorithms across a diverse range of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for automotive ADAS applications and practical Level 3-5 autonomous vehicles.

[0138] In contrast to conventional systems, the techniques described in this paper, by providing CPU complexes, GPU complexes, and hardware accelerator clusters, allow multiple neural networks to be executed simultaneously and / or sequentially, and the results combined to achieve Level 3–5 autonomous driving capabilities. For example, a CNN executed on a DLA or dGPU (e.g., GPU 820) could include text and word recognition, allowing a supercomputer to read and understand traffic signs, including those for which neural networks have not yet been specifically trained. The DLA could also include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs, and passing that semantic understanding to a path planning module running on the CPU complex.

[0139] As another example, multiple neural networks can operate simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Caution: Flashing lights indicate icy conditions," along with a light, can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a deployed first neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a deployed second neural network that informs the vehicle's path planning software (preferably executing on a CPU complex) that icy conditions exist when the flashing lights are detected. The flashing lights can be identified by a deployed third neural network operating across multiple frames, informing the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can operate simultaneously, for example, within a DLA and / or on a GPU 808.

[0140] In some examples, the CNN used for facial recognition and owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 800. A processing engine always on the sensors can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in safe mode, to disable the vehicle when the owner leaves. In this way, SoC 804 provides security against theft and / or carjacking.

[0141] In another example, the CNN used for emergency vehicle detection and identification can use data from microphone 896 to detect and identify emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect siren and manually extract features, SoC 804 uses a CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative shut-off rate of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the localized area in which the vehicle operates, as identified by GNSS sensor 858. Thus, for example, when operating in Europe, the CNN will seek to detect European siren, and when operating in the United States, the CNN will seek to identify siren only in North America. Once an emergency vehicle is detected, with the assistance of ultrasonic sensor 862, a control program can be used to execute emergency vehicle safety routines, causing the vehicle to slow down, pull over to the side of the road, stop, and / or idle until the emergency vehicle passes.

[0142] The vehicle may include a CPU 818 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 804 via a high-speed interconnect (e.g., PCIe). The CPU 818 may include, for example, an x86 processor. The CPU 818 can be used to perform any of a wide variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 804, and / or monitoring the status and health of the controller 836 and / or the infotainment SoC 830.

[0143] Vehicle 800 may include a GPU 820 (e.g., a discrete GPU or dGPU) that can be coupled to SoC 804 via a high-speed interconnect (e.g., NVIDIA's NVLINK). GPU 820 may provide additional artificial intelligence capabilities, for example by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on inputs (e.g., sensor data) from sensors of vehicle 800.

[0144] Vehicle 800 may also include a network interface 824, which may include one or more wireless antennas 826 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). Network interface 824 can be used to enable wireless connectivity via the Internet to the cloud (e.g., with server 878 and / or other network devices), with other vehicles, and / or with computing devices (e.g., occupant client devices). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across networks and via the Internet). A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 800 with information about vehicles approaching vehicle 800 (e.g., vehicles in front, to the side, and / or behind vehicle 800). This functionality can be part of vehicle 800's cooperative adaptive cruise control function.

[0145] Network interface 824 may include a SoC that provides modulation and demodulation functions and enables controller 836 to communicate over a wireless network. Network interface 824 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be performed using known processes and / or using a superheterodyne process. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functions for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0146] Vehicle 800 may also include data storage 828, which may include off-chip (e.g., off-chip SoC 804) storage devices. Data storage 828 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0147] The vehicle 800 may also include a GNSS sensor 858. The GNSS sensor 858 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used for auxiliary mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 858 can be used, including, for example, but not limited to, GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0148] Vehicle 800 may also include a RADAR sensor 860. The RADAR sensor 860 can be used by vehicle 800 for remote vehicle detection even in dark and / or inclement weather conditions. The RADAR functional safety level may be ASIL B. The RADAR sensor 860 can use CAN and / or bus 802 (e.g., to transmit data generated by the RADAR sensor 860) for control and access to object tracking data, and in some examples, Ethernet access for accessing raw data. A wide variety of RADAR sensor types can be used. For example, and without limitation, the RADAR sensor 860 can be adapted for front, rear, and side RADAR use. In some examples, a pulse Doppler RADAR sensor is used.

[0149] The RADAR sensor 860 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, the long-range RADAR can be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view (e.g., within 250m) achieved through two or more independent scans. The RADAR sensor 860 can help distinguish between stationary and moving objects and can be used by ADAS systems for emergency braking assist and forward collision warning. The long-range RADAR sensor can include a single-site multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the vehicle 800's surroundings at higher rates with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, enabling rapid detection of vehicles entering or leaving the vehicle 800's lane.

[0150] As an example, a mid-range RADAR system can include a range of up to 860m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 850 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor blind spots behind and beside the vehicle.

[0151] Short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0152] Vehicle 800 may also include ultrasonic sensors 862. Ultrasonic sensors 862, which may be positioned at the front, rear, and / or sides of vehicle 800, can be used for parking assistance and / or creating and updating occupancy grids. A wide variety of ultrasonic sensors 862 can be used, and different ultrasonic sensors 862 can be used for different detection ranges (e.g., 2.5m, 4m). Ultrasonic sensors 862 can operate at functional safety level ASIL B.

[0153] Vehicle 800 may include a LIDAR sensor 864. The LIDAR sensor 864 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 864 may be of functional safety level ASIL B. In some examples, vehicle 800 may include multiple LIDAR sensors 864 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0154] In some examples, the LiDAR sensor 864 may be able to provide a list of objects and their distances within a 360-degree field of view. Commercially available LiDAR sensors 864 may have an advertising range of, for example, approximately 800m, with an accuracy of 2cm-3cm, and support for 800Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 864 may be used. In such examples, the LiDAR sensor 864 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of a vehicle 800. In such examples, the LiDAR sensor 864 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, even for low-reflectivity objects, with a range of 200m. Front-mounted LiDAR sensors 864 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0155] In some examples, LiDAR technologies such as 3D flash LiDAR can also be used. 3D flash LiDAR uses a flash of laser light as the emission source to illuminate the vehicle's surroundings up to approximately 200 meters. A flash LiDAR unit includes a receiver that records the laser pulse propagation time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment using each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the vehicle. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-scanning LiDAR devices) without moving parts other than a fan. Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using a flash LiDAR, and because a flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 864 is less susceptible to motion blur, vibration, and / or shock.

[0156] The vehicle may also include an IMU sensor 866. In some examples, the IMU sensor 866 may be located at the center of the rear axle of the vehicle 800. The IMU sensor 866 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 866 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 866 may include an accelerometer, a gyroscope, and a magnetometer.

[0157] In some embodiments, the IMU sensor 866 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 866 can enable the vehicle 800 to estimate heading by directly observing and correlating velocity changes from GPS to the IMU sensor 866 without input from a magnetic sensor. In some examples, the IMU sensor 866 and the GNSS sensor 858 can be combined into a single integrated unit.

[0158] The vehicle may include a microphone 896 placed in and / or around the vehicle 800. Among other things, the microphone 896 may be used for emergency vehicle detection and identification.

[0159] The vehicle may also include any number of camera types, including stereo camera 868, wide-angle camera 870, infrared camera 872, surround camera 874, long-range and / or mid-range camera 898, and / or other camera types. These cameras can be used to capture image data around the entire perimeter of the vehicle 800. The types of cameras used depend on the embodiment and the requirements of the vehicle 800, and any combination of camera types can be used to provide the necessary coverage around the vehicle 800. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described herein with respect to... Figure 8A and Figure 8B It was described in more detail.

[0160] Vehicle 800 may also include vibration sensor 842. Vibration sensor 842 can measure vibrations of vehicle components such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 842 are used, differences between vibrations can be used to determine friction or slippage on the road surface (e.g., when there is a vibration difference between a power drive shaft and a free-rotating shaft).

[0161] Vehicle 800 may include ADAS system 838. In some examples, ADAS system 838 may include SoC. ADAS system 838 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC) and / or other features and functions.

[0162] The ACC system can use a RADAR sensor 860, a LIDAR sensor 864, and / or a camera. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to vehicles immediately in front of vehicle 800 and automatically adjusts the vehicle speed to maintain a safe distance. Lateral ACC performs distance holding and, if necessary, advises vehicle 800 to change lanes. Lateral ACC is associated with other ADAS applications such as LCA and CWS.

[0163] CACC uses information from other vehicles, which can be received indirectly from other vehicles via a wireless link or network connection (e.g., via the Internet) through network interface 824 and / or wireless antenna 826. Direct links can be provided by vehicle-to-vehicle (V2V) communication links, while indirect links can be infrastructure-to-vehicle (I2V) communication links. Typically, the V2V communication concept provides information about vehicles immediately ahead (e.g., vehicles immediately in front of vehicle 800 and in the same lane), while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of these I2V and V2V information sources. Given information about vehicles ahead of vehicle 800, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0164] The Forward-Looking Warning (FCW) system is designed to alert the driver to hazards, enabling the driver to take corrective action. The FCW system uses a front-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components. The FCW system can provide warnings in the form of, for example, audible, visual, haptic, and / or rapid braking pulses.

[0165] An AEB (Autonomous Emergency Braking) system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system can use a front-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes to attempt to prevent or at least mitigate the effects of the predicted collision. The AEB system may include technologies such as dynamic brake support and / or collision proximity braking.

[0166] The Lane Departure Warning (LDW) system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle crosses lane markings. When the driver indicates intentional lane departure, the LDW system is deactivated by activating a turn signal. The LDW system can utilize a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0167] The LKA system is a variation of the LDW system. If vehicle 800 begins to leave the lane, the LKA system provides corrective steering input or braking to vehicle 800.

[0168] The BSW system detects and warns the driver of vehicles in the vehicle's blind spot. The BSW system can provide visual, auditory, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses turn signals. The BSW system can utilize a rear-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0169] RCTW systems can provide visual, auditory, and / or tactile notifications when an object is detected outside the range of a rear-view camera while the vehicle is reversing. Some RCTW systems include AEB (Autonomous Emergency Braking) to ensure the application of the vehicle's brakes to avoid a collision. RCTW systems may use one or more rear-view RADAR sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0170] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for the driver, but typically not catastrophic, as they alert the driver and allow them to determine whether a safe condition truly exists and take appropriate action. However, in an autonomous vehicle 800, in the event of conflicting results, the vehicle 800 itself must decide whether to heed the results from the main computer or auxiliary computer (e.g., the first controller 836 or the second controller 836). For example, in some embodiments, the ADAS system 838 may be a backup and / or auxiliary computer used to provide perception information to a backup computer rationality module. The backup computer rationality monitor may run redundant and varied software on hardware components to detect faults in perception and dynamic driving tasks. Outputs from the ADAS system 838 may be provided to a supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0171] In some examples, the master computer can be configured to provide a confidence score to the supervisory MCU, indicating the master computer's confidence level in the selected result. If the confidence score exceeds a threshold, the supervisory MCU can follow the master computer's direction regardless of whether the auxiliary computer provides conflicting or inconsistent results. If the confidence score does not meet the threshold and the master and auxiliary computers indicate different results (e.g., conflict), the supervisory MCU can arbitrate between these computers to determine the appropriate result.

[0172] The supervisory MCU can be configured to run a neural network trained and configured to determine, at least in part, the conditions under which the auxiliary computer provides a false alarm, based on outputs from both the host and auxiliary computers. Thus, the neural network in the supervisory MCU can learn when the output of the auxiliary computer can be trusted and when it cannot. For example, when the auxiliary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying a metallic object that is not actually dangerous, such as a drain grid or manhole cover that triggers an alarm. Similarly, when the auxiliary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running the neural network using associated memory. In a preferred embodiment, the supervisory MCU may include components of and / or be included as components of the SoC 804.

[0173] In other examples, the ADAS system 838 may include an auxiliary computer that performs ADAS functions using conventional computer vision rules. This allows the auxiliary computer to use classic computer vision rules (if-then), and the presence of neural networks in the supervising MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the entire system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functionality. For instance, if a software vulnerability or bug exists in the software running on the host computer and non-identical software code running on the auxiliary computer provides the same overall result, the supervising MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the host computer does not cause a substantial error.

[0174] In some examples, the output of ADAS system 838 can be fed to the perception block and / or the dynamic driving task block of the main computer. For example, if ADAS system 838 issues a forward collision warning because an object is immediately in front, the perception block can use this information when identifying the object. In other examples, the assistance computer can have its own neural network, which is trained and thus reduces the risk of false positives as described herein.

[0175] Vehicle 800 may also include an infotainment SoC 830 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 830 may include a combination of hardware and software that can be used to provide vehicle 800 with audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.) and / or information services (e.g., navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total coverage distance, brake fuel level, fuel level, door opening / closing, air filter information, etc.). For example, the infotainment SoC 830 may include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, WiFi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 834, telematics device, control panel (e.g., for controlling and / or interacting with various components, features, and / or systems) and / or other components. The infotainment SoC 830 may further be used to provide information (e.g., visual and / or auditory) to the vehicle's users, such as information from the ADAS system 838, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0176] The infotainment SoC 830 may include GPU functionality. The infotainment SoC 830 can communicate with other devices, systems, and / or components of the vehicle 800 via bus 802 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 830 may be coupled to a supervisory MCU, allowing the GPU of the infotainment system to perform autonomous driving functions in the event of a failure of the main controller 836 (e.g., the primary and / or backup computer of the vehicle 800). In such an example, the infotainment SoC 830 may place the vehicle 800 into a driver-safe parking mode as described herein.

[0177] Vehicle 800 may also include an instrument cluster 832 (e.g., a digital instrument panel, electronic instrument cluster, digital instrument panel, etc.). The instrument cluster 832 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 832 may include a set of instruments such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 830 and the instrument cluster 832. In other words, the instrument cluster 832 may be included as part of the infotainment SoC 830, or vice versa.

[0178] Figure 8D For cloud-based servers and according to some embodiments of this disclosure Figure 8A This is a system diagram illustrating communication between example autonomous vehicles 800. System 876 may include server 878, network 890, and vehicles including vehicle 800. Server 878 may include multiple GPUs 884(A)-884(H) (collectively referred to herein as GPU 884), PCIe switches 882(A)-882(H) (collectively referred to herein as PCIe switch 882), and / or CPUs 880(A)-880(B) (collectively referred to herein as CPU 880). GPUs 884, CPUs 880, and PCIe switches may be interconnected with high-speed interconnects and / or PCIe connections 886, such as, but not limited to, NVLink interface 888 developed by NVIDIA. In some examples, GPUs 884 are connected via NVLink and / or NVSwitch SoCs, and GPUs 884 and PCIe switches 882 are connected via PCIe interconnects. Although eight GPUs 884, two CPUs 880, and two PCIe switches are shown in the diagram, this is not intended to be limiting. Depending on the embodiment, each of the servers 878 may include any number of GPUs 884, CPUs 880, and / or PCIe switches. For example, each of the servers 878 may include eight, sixteen, thirty-two, and / or more GPUs 884.

[0179] Server 878 can receive image data from vehicles via network 890, representing images of unexpected or changed road conditions such as recently commenced roadworks. Server 878 can also transmit neural network 892, updated neural network 892, and / or map information 894, including information about traffic and road conditions, to vehicles via network 890. Updates to map information 894 may include updates to HD map 822, such as information about construction sites, potholes, bends, floods, or other obstacles. In some examples, neural network 892, updated neural network 892, and / or map information 894 may have been generated from new training and / or data received from any number of vehicles in the environment, and / or based on experience gained from training performed at a data center (e.g., using server 878 and / or other servers).

[0180] Server 878 can be used to train machine learning models (e.g., neural networks) based on training data. Training data can be generated by the vehicle and / or generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., where the neural network does not require supervised learning). Training can be performed according to any one or more classes of machine learning techniques, including but not limited to: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by the vehicle (e.g., transmitted to the vehicle via network 890), and / or the machine learning model can be used by server 878 to remotely monitor the vehicle.

[0181] In some examples, server 878 can receive data from vehicles and apply that data to state-of-the-art real-time neural networks for real-time intelligent inference. Server 878 may include a deep learning supercomputer powered by GPU 884 and / or a dedicated AI computer, such as the DGX and DGX station machines developed by NVIDIA. However, in some examples, server 878 may include a deep learning infrastructure in a data center that uses only CPU power.

[0182] The deep learning infrastructure of server 878 may be capable of rapid, real-time inference and can be used to assess and verify the health status of the processor, software, and / or associated hardware in vehicle 800. For example, the deep learning infrastructure may receive periodic updates from vehicle 800, such as image sequences and / or objects located in those image sequences by vehicle 800 (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them with objects identified by vehicle 800. If the results do not match and the infrastructure concludes that the AI ​​in vehicle 800 has malfunctioned, then server 878 may transmit a signal to vehicle 800 instructing the vehicle's fail-safe computer to take control, notify the occupants, and complete a safe stopping operation.

[0183] For inference, server 878 may include GPU 884 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of a GPU-powered server and inference acceleration enables real-time response. In other examples, such as where performance is less critical, CPU, FPGA, and other processor-powered servers can be used for inference.

[0184] Example computing device

[0185] Figure 9This is a block diagram of an example computing device 900 suitable for implementing some embodiments of the present disclosure. The computing device 900 may include an interconnect system 902 directly or indirectly coupled to: a memory 904, one or more central processing units (CPUs) 906, one or more graphics processing units (GPUs) 908, a communication interface 910, input / output (I / O) ports 912, input / output components 914, a power supply 916, one or more presentation components 918 (e.g., one or more displays), and one or more logic units 920. In at least one embodiment, the computing device 900 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 908 may include one or more vGPUs, one or more CPUs 906 may include one or more vCPUs, and / or one or more logic units 920 may include one or more virtual logic units. Thus, (one or more) computing devices 900 may include discrete components (e.g., a full GPU dedicated to computing device 900), virtual components (e.g., a portion of the GPU dedicated to computing device 900), or combinations thereof. In some embodiments, one or more aspects of the occupant evaluation function 110 may be implemented using code that executes on CPU 906 and / or GPU 908.

[0186] although Figure 9 The various blocks are shown as connected via interconnect system 902 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 918 (such as a display device) may be considered I / O component 914 (e.g., if the display is a touchscreen). As another example, CPU 906 and / or GPU 908 may include memory (e.g., memory 904 may represent a storage device other than the memory of GPU 908, CPU 906, and / or other components). In other words, Figure 9 The computing devices described are for illustrative purposes only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of… Figure 9 Within the scope of computing devices.

[0187] Interconnect system 902 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 902 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 906 may be directly connected to memory 904. Further, CPU 906 may be directly connected to GPU 908. In cases where there is a direct or point-to-point connection between components, interconnect system 902 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 900.

[0188] The memory 904 may include any computer-readable medium from a variety of computer-readable media. A computer-readable medium may be any available medium accessible by the computing device 900. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0189] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented with any method or technology for storing information such as computer-readable instructions, data structures, program modules and / or other data types. For example, memory 904 may store computer-readable instructions (e.g., representing (one or more) programs and / or (one or more) program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 900. As used herein, computer storage media does not include signals themselves.

[0190] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0191] CPU 906 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. Each CPU 906 may contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 906 may contain any type of processor and may contain different types of processors depending on the type of computing device 900 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 900, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 900 may also include one or more CPUs 906.

[0192] In addition to or in lieu of one or more CPUs 906, one or more GPUs 908 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. One or more GPUs 908 may be integrated GPUs (e.g., having one or more CPUs 906) and / or one or more GPUs 908 may be discrete GPUs. In embodiments, one or more GPUs 908 may be coprocessors of one or more CPUs 906. GPUs 908 may be used by computing device 900 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 908 may be used for general-purpose computing on a GPU (GPGPU). GPUs 908 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPUs 908 may generate pixel data of an output image in response to rendering commands (e.g., rendering commands received from CPU 906 via a host interface). GPU 908 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 904. GPU 908 may include two or more GPUs operating in parallel (e.g., via links). The links may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 908 may produce pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0193] In addition to or in lieu of CPU 906 and / or GPU 908, logic unit 920 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 906, one or more GPUs 908, and / or one or more logic units 920 may perform any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 920 may be a portion of one or more CPUs 906 and / or GPUs 908 and / or integrated into one or more CPUs 906 and / or GPUs 908, and / or one or more logic units 920 may be discrete components or otherwise external to CPUs 906 and / or GPUs 908. In an embodiment, one or more of the logic units 920 may be coprocessors of one or more of the CPU 906 and / or one or more of the GPU 908.

[0194] Examples of logic unit 920 include one or more processing cores and / or components thereof, such as data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or fast peripheral component interconnect (PCIe) element, etc.

[0195] The communication interface 910 may include one or more receivers, transmitters, and / or transceivers enabling the computing device 900 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). The communication interface 910 may include components and functions for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, one or more logic units 920 and / or the communication interface 910 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 902 to one or more GPUs 908 (e.g., memory).

[0196] I / O port 912 enables computing device 900 to be logically coupled to other devices including I / O component 914, (one or more) presentation component 918, and / or other components, some of which may be built into (e.g., integrated into) computing device 900. Illustrative I / O component 914 includes microphone, mouse, keyboard, joystick, gamepad, game controller, dish satellite dish, scanner, printer, wireless device, etc. I / O component 914 can provide a natural user interface (NUI) for processing air gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 900. Computing device 900 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 900 may include an accelerometer or gyroscope that enables motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the computing device 900 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0197] Power supply 916 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 916 may provide power to computing device 900 so that the components of computing device 900 can operate.

[0198] The presentation component 918 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 918 may receive data from other components (e.g., GPU 908, CPU 906, DPU, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0199] Example Data Center

[0200] Figure 10 An example data center 1000 that may be used in at least one embodiment of this disclosure is shown. The data center 1000 may include a data center infrastructure layer 1010, a framework layer 1020, a software layer 1030, and / or an application layer 1040.

[0201] like Figure 10As shown, the data center infrastructure layer 1010 may include a resource coordinator 1012, grouped computing resources 1014, and node computing resources (“nodes CR”) 1016(1)-1016(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CR 1016(1)-1016(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and / or cooling modules, etc. In some embodiments, one or more node CRs from nodes CR 1016(1)-1016(N) may correspond to servers having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR1016(1)-10161(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more of nodes CR1016(1)-1016(N) may correspond to virtual machines (VMs).

[0202] In at least one embodiment, the grouped computing resources 1014 may include individual groups of nodes CR1016 housed within one or more racks (not shown), or a plurality of racks housed within a data center in different geographical locations (also not shown). Individual groups of nodes CR1016 within the grouped computing resources 1014 may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of nodes CR1016, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0203] Resource coordinator 1012 may be configured or otherwise control one or more nodes CRs 1016(1)-1016(N) and / or grouped computing resources 1014. In at least one embodiment, resource coordinator 1012 may include a Software Design Infrastructure (SDI) management entity for data center 1000. Resource coordinator 1012 may include hardware, software, or some combination thereof.

[0204] In at least one embodiment, such as Figure 10As shown, framework layer 1020 may include a job scheduler 1033, a configuration manager 1034, a resource manager 1036, and / or a distributed file system 1038. Framework layer 1020 may include a framework for software 1032 supporting software layer 1030 and / or one or more applications 1042 supporting application layer 1040. Software 1032 or application 1042 may respectively contain web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1020 may be, but is not limited to, free and open-source software web application frameworks (such as Apache Spark) that can utilize distributed file system 1038 for large-scale data processing (e.g., "big data"). TM (Hereinafter referred to as "Spark") is a type of application. In at least one embodiment, the job scheduler 1033 may include Spark drivers to facilitate the scheduling of workloads supported by different layers of data center 1000. The configuration manager 1034 may be able to configure different layers, such as the software layer 1030 and the framework layer 1020 (which includes Spark and a distributed file system 1038 for supporting large-scale data processing). The resource manager 1036 may be able to manage computing resources mapped to or allocated to clusters of distributed file system 1038 and job scheduler 1033 or to support clusters of distributed file system 1038 and job scheduler 1033. In at least one embodiment, clusters or groups of computing resources may include groups of computing resources 1014 in the data center infrastructure layer 1010. The resource manager 1036 may coordinate with the resource coordinator 1012 to manage these mapped or allocated computing resources. In some embodiments, one or more aspects of the occupancy evaluation function 110 may be implemented by software 1032 or (one or more) applications 1042.

[0205] In at least one embodiment, the software 1032 included in the software layer 1030 may include software used in at least a portion of the nodes CRs 1016(1)-1016(N), the grouped computing resources 1014, and / or the distributed file system 1038 of the framework layer 1020. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0206] In at least one embodiment, the application 1042 included in the application layer 1040 may include one or more types of applications used at least in part by nodes CRs 1016(1)-1016(N), grouped computing resources 1014, and / or the distributed file system 1038 of the framework layer 1020. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in combination with one or more embodiments.

[0207] In at least one embodiment, any of the configuration manager 1034, resource manager 1036, and resource coordinator 1012 can implement any number and type of self-modification actions based on any amount and type of data obtained in any technically feasible manner. Self-modification actions can free the data center operator of data center 1000 from making potentially poor configuration decisions and may prevent underutilization and / or poor performance of the data center.

[0208] According to one or more embodiments described herein, data center 1000 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, one or more machine learning models can be trained by using the software and / or computing resources described above with respect to data center 1000 to compute weight parameters according to a neural network architecture. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks can be used to infer or predict information using the resources described above with respect to data center 1000 by using weight parameters computed through one or more training techniques (such as, but not limited to, those described herein).

[0209] In at least one embodiment, the data center 1000 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform services that infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0210] Example network environment

[0211] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 9 This is implemented on one or more instances of computing device 900—for example, each device may include similar components, features, and / or functions of computing device 900. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of data center 1000, examples of which are described herein. Figure 10 To describe in more detail.

[0212] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0213] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0214] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0215] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). The core server may assign at least a portion of the functionality to the edge server if the connection to the user (e.g., a client device) is relatively close to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0216] (One or more) client devices may include the information described in this article. Figure 9 At least some of the components, features, and functions of the described (one or more) example computing device 900. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0217] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, which are executed by a computer or other machine such as a personal digital assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a wide variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a communication network.

[0218] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0219] The subject matter of this disclosure is described in detail herein to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms “step” and / or “block” may be used herein to imply different elements of the method employed, these terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. A system comprising: One or more processing units, for: Detect at least one or more features representing at least a portion of the occupants of the machine, based on optical image data; The depth corresponding to one or more features is determined based at least on the optical image data and the true depth image of the carriage corresponding to the interior of the machine; A three-dimensional representation of the occupant is generated based at least on one or more features and the depth of the one or more features, wherein, The three-dimensional representation includes: at least one characteristic representing the size of the occupant; and At least one operation of the machine is controlled based on at least one of the aforementioned characteristics.

2. The system as claimed in claim 1, wherein, The at least one feature includes: three-dimensional shape estimation or three-dimensional pose estimation.

3. The system as described in claim 1, wherein, The one or more processing units are further configured to: Execute at least one three-dimensional pose detection model to generate a scale-normalized three-dimensional pose estimate of the occupant based on at least one or more features; as well as The scale-normalized 3D pose estimate is scaled to a linear measurement scale based at least on the depth corresponding to one or more of the features.

4. The system as described in claim 3, wherein, The one or more processing units are further configured to: At least one characteristic representing the occupant's dimensions is generated based on a three-dimensional attitude estimation at least on the linear measurement scale.

5. The system as described in claim 3, wherein, The one or more processing units are further configured to: The scale-normalized three-dimensional pose estimate is generated based at least on a set of kinematic elements of the occupant detected by the at least one three-dimensional pose detection model using one or more features.

6. The system of claim 3, wherein, The one or more processing units are further configured to: The optical image data is applied to at least one person detection model to define a portion of the image frame representing the occupant; as well as The scale-normalized 3D pose estimate is generated based at least on the portion of the image frame representing the occupant.

7. The system as claimed in claim 1, wherein, The one or more processing units are further configured to: Ray tracing is performed to associate at least a first pixel of the ground truth car depth image with the one or more features to determine the depth corresponding to the one or more features.

8. The system as claimed in claim 1, wherein, The one or more processing units are further configured to: A scale-normalized three-dimensional pose estimate of the occupant is generated based on at least one or more of the features, wherein the scale-normalized three-dimensional pose estimate represents a set of kinematic elements of the occupant.

9. The system of claim 1, wherein, The one or more processing units are further configured to: A distance image is generated based at least on the optical image data and the ground truth carriage depth image, wherein the depth corresponding to one or more features is derived at least from the distance image.

10. The system of claim 9, wherein, The one or more processing units are further configured to: A vehicle interior mask is generated based at least on the segmentation of the optical image data, wherein the vehicle interior mask comprises: delineating the boundaries of pixels corresponding to structural elements within the interior; and The distance image is further generated based on the vehicle interior mask.

11. The system of claim 1, wherein, The one or more processing units are further configured to: A vehicle interior mask is generated based at least on the segmentation of the optical image data, wherein the vehicle interior mask includes: a first boundary delineating pixels corresponding to structural elements of the interior; A occupant mask is generated based at least on the segmentation of the optical image data, wherein the occupant mask comprises: delineating a second boundary of pixels corresponding to the one or more features; and A three-dimensional shape estimate of the occupant is generated based at least on the occupant mask and at least on the distance image derived from the optical image data, the ground truth car depth image, and the vehicle interior mask.

12. The system of claim 11, wherein, The one or more processing units are further configured to: Further, based at least on the occupant's pose estimation generated from the optical image data, the occupant's three-dimensional shape estimation is generated.

13. The system of claim 1, wherein, The system is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation for 3D assets; A system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system used to perform deep learning operations; Systems used for performing real-time streaming; Systems implemented using edge devices; Systems implemented using robots; Systems used to perform conversational AI operations; A system for generating synthetic data; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; Systems used to perform generative AI operations; A system that uses at least a language model in its implementation; or A system that utilizes cloud computing resources at least in part.

14. A processor, comprising: One or more processing units, for: A first representation of at least a portion of the occupants of the machine is generated based on at least two-dimensional optical image data; Based at least on the two-dimensional optical image data and the true-value car interior depth image corresponding to the interior of the machine, the depth corresponding to one or more surface positions of the occupant is determined; Based at least on the depth corresponding to the one or more surface positions of the occupant, the first representation of at least said portion of the occupant is scaled to a second representation of at least said portion of the occupant, wherein the second representation is a three-dimensional representation; At least one characteristic representing the size of the occupant is determined based on the second representation; and At least one operation of the machine is controlled based on at least one characteristic of the occupant's dimensions.

15. The processor of claim 14, wherein, The at least one feature includes: three-dimensional shape estimation or three-dimensional pose estimation.

16. The processor of claim 14, wherein, The one or more processing units are further configured to: Execute at least one three-dimensional pose detection model to generate a scale-normalized three-dimensional pose estimate of the occupant based on at least one or more features, said one or more features representing at least said portion of the occupant of the machine based at least on said two-dimensional optical image data; as well as The scale-normalized 3D pose estimate is scaled to a linear measurement scale based at least on the depth corresponding to one or more of the features.

17. The processor of claim 14, wherein, The one or more processing units are further configured to: A distance image is generated based at least on the two-dimensional optical image data and the true depth image of the carriage, wherein the depth corresponding to the one or more surface locations is derived at least from the distance image.

18. The processor of claim 14, wherein, The one or more processing units are further configured to: A vehicle interior mask is generated based at least on the segmentation of the two-dimensional optical image data, wherein the vehicle interior mask includes: a first boundary delineating pixels corresponding to the structural elements of the interior; A occupant mask is generated based at least on the segmentation of the two-dimensional optical image data, wherein the occupant mask comprises: delineating a second boundary of pixels corresponding to the one or more surface locations; and A three-dimensional shape estimate of the occupant is generated based at least on the occupant mask and at least on the distance image derived from the two-dimensional optical image data, the ground truth car depth image, and the vehicle interior mask.

19. The processor of claim 14, wherein, The processor is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation for 3D assets; A system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system used to perform deep learning operations; Systems used for performing real-time streaming; Systems implemented using edge devices; Systems implemented using robots; Systems used to perform conversational AI operations; A system for generating synthetic data; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; Systems used to perform generative AI operations; A system that uses at least a language model in its implementation; or A system that utilizes cloud computing resources at least in part.

20. A method comprising: A three-dimensional representation of the occupant's dimensions is generated based at least on one or more features representing at least a portion of the occupant derived from two-dimensional optical image data and the depth of said one or more features generated from said two-dimensional optical image data and a ground truth cabin depth image corresponding to the interior of the machine.

Images