Secure Decomposition for Routing in Autonomous Systems

Abstract

To provide an architecture to generate a lane graph or path determination.SOLUTION: A system fuses data from a plurality of sources useful for determining localization with each other. To ensure safety compliance, the fused data is compared with data from a reliable system for safety, and as long as at least two comparators are matched for the fused localization data, the fused localization data is used and verified for compliance with safety regulations. The system also fuses pieces of information useful and available for lane recognition with each other. The fused data is compared with data from a reliable system for safety, and as long as at least two comparators for these safety compliance systems are matched for fused lane graph data, the fused lane graph data is provided for navigation, and is verified for compliance with the regulations.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] This application claims priority to U.S. Provisional Application No. 63 / 111,250, entitled "Safety Decomposition for Autonomous Vehicles Path Detection," filed November 9, 2020, which is incorporated herein in its entirety for all purposes. [Background technology]

[0002] As various industries move increasingly toward increased automation, whether for vehicles, robots, or other automated systems, there is a corresponding need to improve the safety of operations performed by this automation. This may involve improving the accuracy of movement or navigation, such as to determine an optimal path for an automated machine under the current conditions of the environment in which the machine is located. In many automated systems, there may be a variety of sensors and sources that provide data useful in determining such a path. To ensure that such path-determination techniques meet specific safety requirements or regulations, individual decisions are made to comply with these requirements or regulations. While such techniques may provide safe operation in most cases, they may also provide a less-than-desirable operating experience, such as in the case of passengers in a car that provides a safe but not particularly smooth or continually enjoyable ride. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (eg, Standard No. J3016-201806, published on June 15, 2018, Standard No. J3016-201609, published on September 30, 2016, and previous and future versions of this standard) Summary of the Invention [Means for solving the problem]

[0004] Various embodiments according to the present disclosure will now be described with reference to the drawings. [Brief explanation of the drawings]

[0005] [Figure 1A] FIG. 1 illustrates an example of components of an autonomous vehicle, according to at least one embodiment. [Figure 1B] FIG. 1 illustrates an example of components of an autonomous vehicle, according to at least one embodiment. [Figure 2A] FIG. 1 illustrates a view of a lane graph generation architecture that may be utilized, according to at least one embodiment. [Figure 2B] FIG. 1 illustrates a view of a lane graph generation architecture that may be utilized, according to at least one embodiment. [Figure 2C] FIG. 1 illustrates a view of a lane graph generation architecture that may be utilized, according to at least one embodiment. [Figure 3] FIG. 1 illustrates a combined lane graph decision architecture that may be utilized, according to at least one embodiment. [Figure 4] FIG. 1 illustrates an example process for determining a navigation path, according to at least one embodiment. [Figure 5] FIG. 1 illustrates an example process for utilizing fusion propagation for route determination, according to at least one embodiment. [Figure 6] FIG. 1 illustrates components of a system for determining a navigation path, according to at least one embodiment. [Figure 7A] FIG. 1 illustrates inference and / or training logic, according to at least one embodiment. [Figure 7B] FIG. 1 illustrates inference and / or training logic, according to at least one embodiment. [Figure 8] FIG. 1 illustrates an exemplary data center system, according to at least one embodiment. [Figure 9] FIG. 1 illustrates a computer system according to at least one embodiment. [Figure 10] FIG. 1 illustrates a computer system according to at least one embodiment. [Figure 11] FIG. 1 illustrates at least a portion of a graphics processor according to one or more embodiments. [Figure 12] FIG. 1 illustrates at least a portion of a graphics processor according to one or more embodiments. [Figure 13] FIG. 1 illustrates an exemplary data flow diagram for an advanced computing pipeline, according to at least one embodiment. [Figure 14] FIG. 1 is a system diagram for an exemplary system for training, adapting, instantiating, and deploying machine learning models in an advanced computing pipeline, according to at least one embodiment. [Figure 15A] FIG. 1 is a data flow diagram for a process for training a machine learning model, according to at least one embodiment. [Figure 15B] FIG. 1 illustrates a client-server architecture for extending an annotation tool with pre-trained annotation models, according to at least one embodiment. [Figure 16A] FIG. 1 illustrates components that may be used with a vehicle system, according to at least one embodiment. [Figure 16B] FIG. 1 illustrates components that may be used with a vehicle system, according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0006] Techniques according to various embodiments can provide for the generation of navigation routes, graphs, or trajectories for automated or semi-automated devices, such as vehicles or robots. In particular, various embodiments fuse or otherwise consider information from multiple sources to determine optimal localization and lane graph determinations that also comply with corresponding requirements or regulations, such as those related to safety of operation. In the case of an autonomous (or semi-autonomous) vehicle or robot, this can involve, for example, determining an optimal lane graph for use in directing navigation or movement based on a combination of map data and real-time perception data. In at least one embodiment, a system (e.g., an electronic control unit (ECU)) can fuse sensor data that can help determine localization, such as data from a radar or camera system. To provide safety compliance, this fused data can be compared to data from individual systems that are trusted for safety, such as from a safety-compliant camera or radar system. As long as at least two comparators or "voters" for these safety-compliant systems agree on the fused localization data, the fused localization data can be used and verified to comply with safety regulations. This fused localization data can be provided as input to a lane perception system.

[0007] In a similar manner, such a system (e.g., an ECU) can fuse or otherwise consider available and reliable information useful for lane perception and graph generation, such as HD graphs and live lane perception data, as may be determined using camera or LIDAR systems, and fused localization data. To provide safety compliance, the fused lane graph data can also be compared to data from safety-trusted systems, such as those generated using data from safety-compliant camera or LIDAR systems. As long as at least two comparators for these safety-compliant systems agree on the fused lane graph data, the fused lane graph data can be provided for navigation and verified as compliant with safety regulations. If the fused lane graph data does not receive at least two votes from these comparators or is otherwise determined to not meet one or more other safety criteria, another of these lane graph or navigation decisions can be used. Such a system can provide switching between navigation decisions to ensure the vehicle always operates safely, even if one or more of the systems on the vehicle become unavailable or unreliable.

[0008] Such systems may also utilize at least some amount of dead reckoning, where historical data may be utilized for a limited period of time if live perception data is unavailable or at least unreliable. Furthermore, the selection of inputs to be fused together, either localization data or live perception data, may be switched over time depending on reliability and availability. One or more arbiters may analyze the fused data and, if unavailable or unreliable, may substitute one of the safety-compliant signals to ensure the vehicle can continue to operate safely. In some embodiments, at least two ECUs may also be used, such as one for vehicle data and one for satellite data, and live switching may be implemented to operate on lane graphs from either or both ECUs to ensure the vehicle can continue to operate safely if one signal is unavailable or unreliable.

[0009] Consider autonomous vehicle 100 shown in FIG. 1A. In at least one embodiment, vehicle 100 may be a passenger vehicle, such as, but not limited to, a car, truck, bus, and / or another type of vehicle that accommodates one or more passengers. In at least one embodiment, vehicle 100 may be a semi-tractor-trailer truck used to transport cargo. In at least one embodiment, vehicle 100 may be an aircraft, robotic vehicle, or other type of vehicle that may or may not be intended to transport passengers or cargo, but may be intended to perform one or more tasks that require at least some amount of vehicle movement or displacement.

[0010] Autonomous vehicles may be described in terms of levels of automation as defined by the National Highway Traffic Safety Administration ("NHTSA"), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers ("SAE"), in their respective publications. In one or more embodiments, vehicle 100 may be capable of functionality according to one or more of levels 1 through 5 of autonomous driving. For example, in at least one embodiment, vehicle 100 may be capable of conditional automation (Level 3), highly automated (Level 4), and / or fully automated (Level 5), depending on the embodiment.

[0011] In at least one embodiment, vehicle 100 may include components such as, but not limited to, a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. In at least one embodiment, vehicle 100 may include a propulsion system 150 such as, but not limited to, an internal combustion engine, a hybrid power plant, a fully electric engine, and / or another propulsion system type. In at least one embodiment, propulsion system 150 may be connected to a drive train of vehicle 100, which may include, but is not limited to, a transmission to enable propulsion of vehicle 100. In at least one embodiment, propulsion system 150 may be controlled in response to receiving a signal from throttle / accelerator(s) 152.

[0012] In at least one embodiment, a steering system 154, which may include, but is not limited to, a steering wheel, is used to steer the vehicle 100 (e.g., along a desired path or route) when the propulsion system 150 is operating (e.g., when the vehicle is moving). In at least one embodiment, the steering system 154 may receive signals from steering actuator(s) 156. The steering wheel may be optional for fully automated (Level 5) functionality. In at least one embodiment, the brake sensor system 146 may be used to operate the vehicle brakes in response to receiving signals from brake actuator(s) 148 and / or brake sensors.

[0013] In at least one embodiment, controller(s) 136, which may include, but are not limited to, one or more system-on-chip (“SoC”) (not shown in FIG. 1A ) and / or graphics processing unit(s) (“GPU”)(s), provide signals (e.g., representing commands) to one or more components and / or systems of vehicle 100. For example, in at least one embodiment, controller(s) 136 may send signals to operate vehicle brakes via brake actuator(s) 148, to operate steering system 154 via steering actuator(s) 156, and / or to operate propulsion system 150 via throttle / accelerator(s) 152. The controller(s) 136 may include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving the vehicle 100. In at least one embodiment, the controller(s) 136 may include a first controller 136 for autonomous driving functions, a second controller 136 for functional safety functions, a third controller 136 for artificial intelligence functionality (e.g., computer vision), a fourth controller 136 for infotainment functionality, a fifth controller 136 for redundancy in emergency situations, and / or other controllers. In at least one embodiment, a single controller 136 may handle two or more of the above functionalities, two or more controllers 136 may handle a single functionality, and / or any combination thereof.

[0014] In at least one embodiment, controller(s) 136 provide signals to control one or more components and / or systems of vehicle 100 in response to sensor data (e.g., sensor inputs) received from one or more sensors. In at least one embodiment, the sensor data may be received from, for example, but not limited to, global navigation satellite system ("GNSS") sensor(s) 158 (e.g., global positioning system sensor(s)), RADAR sensor(s) 160, ultrasonic sensor(s) 162, LIDAR sensor(s) 164, inertial measurement unit(s) ("IMU") sensor(s), or other sensors. The vehicle 100 vehicle information may be received from sensors 166 (e.g., accelerometer(s), gyroscope(s), magnetic compass(s), magnetometer(s), etc.), microphone(s) 196, stereo camera(s), wide-angle camera(s) 170 (e.g., fisheye cameras), infrared camera(s) 172, surrounding camera(s) 174 (e.g., 360-degree cameras), long-range camera(s) (not shown in FIG. 1A ), mid-range camera(s) (not shown in FIG. 1A ), speed sensor(s) 144 (e.g., for measuring the speed of the vehicle 100), vibration sensor(s) 142, steering sensor(s) 140, brake sensor(s) (e.g., as part of brake sensor system 146), and / or other sensor types.

[0015] In at least one embodiment, one or more of the controller(s) 136 may receive input (e.g., represented by input data) from the instrument cluster 132 of the vehicle 100 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (“HMI”) display 134, an audible annunciator, a loudspeaker, and / or via other components of the vehicle 100. In at least one embodiment, the output may include information such as vehicle speed, speed, time, map data (e.g., a high-definition map (not shown in FIG. 1A )), location data (e.g., the location of vehicle 100 on a map, etc.), direction, the location of other vehicles (e.g., an occupancy grid), information about objects and object statuses perceived by controller(s) 136, etc. For example, in at least one embodiment, HMI display 134 may display information about the presence of one or more objects (e.g., road signs, warning signs, traffic light changes, etc.) and / or information about a driving maneuver that the vehicle has performed, is performing, or will perform (e.g., currently changing lanes, taking exit 34B in 2 miles, etc.).

[0016] In at least one embodiment, vehicle 100 further includes network interface 124, which may use wireless antenna(s) 126 and / or modem(s) to communicate over one or more networks. For example, in at least one embodiment, network interface 124 may be capable of communicating via Long-Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile communications (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”), etc. Additionally, in at least one embodiment, the wireless antenna(s) 126 may enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area network(s) such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc., and / or low power wide-area network(s) (“LPWAN”) such as LoRaWAN, SigFox, etc.

[0017] 1B illustrates an example of camera locations and fields of view for autonomous vehicle 100 of FIG. 1A, according to at least one embodiment. In at least one embodiment, the cameras and their respective fields of view are an illustrative example and are not limiting. For example, in at least one embodiment, additional and / or alternative cameras may be included and / or cameras may be located at different locations on vehicle 100.

[0018] In at least one embodiment, the camera type for the camera may include, but is not limited to, a digital camera that may be adapted for use with components and / or systems of vehicle 100. In at least one embodiment, one or more of the camera(s) may operate at Automotive Safety Integrity Level (“ASIL”) B and / or another ASIL. In at least one embodiment, the camera type may be capable of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. In at least one embodiment, the camera may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In at least one embodiment, the color filter array may include a red, clear, clear, clear ("RCCC") color filter array, a red, clear, clear, blue ("RCCB") color filter array, a red, blue, green, clear ("RBGC") color filter array, a Foveon X3 color filter array, a Bayer sensor ("RGGB") color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In at least one embodiment, a clear pixel camera may be used, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays, to increase light sensitivity.

[0019] In at least one embodiment, one or more of the camera(s) may be used to implement advanced driver assistance system ("ADAS") functions (e.g., as part of a redundant or fail-safe design). For example, in at least one embodiment, a multi-function mono camera may be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlight control. In at least one embodiment, one or more of the camera(s) (e.g., all of the cameras) may simultaneously record and provide image data (e.g., video).

[0020] In at least one embodiment, one or more of the cameras may be mounted in a mounting assembly, such as a custom-designed (e.g., three-dimensionally (“3D”) printed) assembly, to eliminate stray light and reflections from the interior of the vehicle (e.g., reflections reflected from the dashboard onto the windshield) that may interfere with the camera's image data capture ability. Referring to a door mirror mounting assembly, in at least one embodiment, the door mirror assembly may be custom 3D printed so that the camera mounting plate matches the shape of the door mirror. In at least one embodiment, the camera(s) may be integrated into the door mirror. For side view cameras, the camera(s) may also be integrated into the four pillars at each corner in at least one embodiment.

[0021] In at least one embodiment, a camera with a field of view that includes a portion of the environment ahead of vehicle 100 (e.g., a front-facing camera) may be used for a surround view to help identify the path and obstacles ahead and, with the aid of controller 136 and / or one or more of the control SoCs, to provide information essential for generating an occupancy grid and / or determining a preferred vehicle path. In at least one embodiment, the front-facing camera may be used to perform many of the same ADAS functions as LIDAR, including, but not limited to, emergency braking, pedestrian detection, and collision avoidance. In at least one embodiment, the front-facing camera may also be used for ADAS functions and systems, including, but not limited to, other functions such as Lane Departure Warning ("LDW"), Autonomous Cruise Control ("ACC"), and / or traffic sign recognition.

[0022] In at least one embodiment, various cameras may be used in a front-facing configuration, including, for example, a monocular camera platform including a CMOS (“complementary metal oxide semiconductor”) color imager. In at least one embodiment, a wide-angle camera 170 may be used to recognize objects (e.g., pedestrians, crossing traffic, or bicyclists) coming into view from the periphery. While only one wide-angle camera 170 is shown in FIG. 1B , in other embodiments, there may be any number (including zero) of wide-angle cameras 170 on vehicle 100. In at least one embodiment, any number of long-range camera(s) 198 (e.g., long-view stereo camera pairs) may be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. In at least one embodiment, the long-range camera(s) 198 may also be used for object detection and classification, as well as basic object tracking.

[0023] In at least one embodiment, any number of stereo cameras 168 may also be included in the front-facing configuration. In at least one embodiment, one or more of the stereo camera(s) 168 may include an integrated control unit with a scalable processing unit, which may provide a programmable logic on a chip ("FPGA") and a multi-core microprocessor with an integrated controller area network ("CAN") or Ethernet interface. In at least one embodiment, such a unit may be used to generate a 3D map of the vehicle's 100 environment, including distance estimates for every point in the image. In at least one embodiment, one or more of the stereo camera(s) 168 may include, but are not limited to, compact stereo vision sensor(s) that may include, but are not limited to, two camera lenses (one on the left and one on the right) and an image processing chip that may measure the distance of a target object from the vehicle 100 and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning features. In at least one embodiment, other types of stereo camera(s) 168 may be used in addition to or instead of those described herein.

[0024] In at least one embodiment, a camera with a field of view that includes portions of the environment to the sides of vehicle 100 (e.g., a side-view camera) may be used for the surroundings view, providing information used to create and update the occupancy grid and generate side collision warnings. For example, in at least one embodiment, surroundings camera(s) 174 (e.g., four surroundings cameras 174 shown in FIG. 1B ) may be positioned on vehicle 100. In at least one embodiment, surroundings camera(s) 174 may include, without limitation, any number and combination of wide-angle camera(s) 170, fisheye camera(s), 360-degree camera(s), etc. For example, in at least one embodiment, four fisheye cameras may be positioned in front, behind, and on the sides of vehicle 100. In at least one embodiment, vehicle 100 may use three surroundings cameras 174 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., a front camera) as a fourth surroundings view camera.

[0025] In at least one embodiment, a camera with a field of view that includes a portion of the environment behind vehicle 100 (e.g., a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and creating and updating an occupancy grid. In at least one embodiment, a wide variety of cameras may be used, including, but not limited to, a camera that is also suitable as a front-facing camera(s) (e.g., long-range camera 198, and / or mid-range camera(s) 176, stereo camera(s) 168, infrared camera(s) 172, etc.), as described herein.

[0026] For such a vehicle to perform a task, such as navigating to a destination, a control system can determine one or more actions to take at a given time corresponding to the task. In the case of an autonomous vehicle, this may include making one or more adjustments to a steering or braking system, etc., to cause the vehicle to operate in a determined manner along a determined path, such as a path that causes the vehicle to navigate along the current lane of the road, within lane markers, while avoiding collisions and operating at an appropriate speed. To determine the appropriate action to take, in at least some systems, it may be beneficial to first determine a path or trajectory the vehicle should follow. Determining a navigation path may include determining information about the environment in which the vehicle is currently operating, such as a high-definition (HD) map, model, or representation of the road and nearby objects, which may be obtained from a source that previously modeled the environment or may be generated in real time using sensors or other mechanisms for perceiving information about the environment. In at least some embodiments, it may also be beneficial to determine relevant localization data, or data indicating where the vehicle is located in its environment with sufficient accuracy to meet safety or operational requirements.

[0027] However, as noted above, in order to provide an enjoyable experience for passengers, it may be desirable to determine a route or sequence of maneuvers that will be at least somewhat optimized for smoothness or comfort in addition to safety. In many instances, this will involve a trade-off between safety and smooth operation, as safe driving is extremely important, but within the bounds of safe operation, it may be desirable to determine a route that provides the most enjoyable or consistent ride, such as one that does not have rapid adjustments in direction, acceleration, or deceleration.

[0028] Further complicating the issue for at least some types of automation, the operation of an automated or semi-automated device or system, such as an autonomous vehicle, often must comply with very specific or stringent safety requirements or regulations. While it may be desirable to combine data from various sources to determine an optimal route, it may be difficult to prove that such a decision meets those requirements or regulations. Furthermore, information from various sources is often kept separate for safety reasons, so that a failure of a given sensor or subsystem does not affect the overall operation of the vehicle, potentially resulting in a suboptimal route decision. Thus, from a safety standpoint, it may be desirable to keep information from different sensors, subsystems, or sources in separate, parallel, or redundant data streams. However, from an optimal operation standpoint, it may be preferable to combine the many pieces of information available to make the best possible decision at a given time.

[0029] As an example, an autonomous vehicle may utilize an automated driving system (ADS) to perform a complex dynamic driving task (DDT). For example, Society of Automotive Engineers (SAE) automation levels 3-5 currently require crash avoidance capabilities so that a human driver is not required to take over the driving task during a failure event. Analysis of dangerous events and their risks may be performed using hazard analysis and risk assessment (HARA) according to the ISO 26262 standard. One of the key functionalities of an exemplary ADS system, tightly coupled to the safe operation of the ADS system, involves path detection. The path detection module can identify drivable paths and provide information about road geometry and topology. This may include data about the center path and / or boundaries of road lanes, as may be determined using data from sources such as images or video captured by an on-vehicle camera, LIDAR data obtained from a trusted source, GPS data, IMU data, or HD map data. A failure in the path detection module could result in a dangerous event, such as leaving the roadway or entering the path of oncoming traffic. Hazard Analysis and Risk Assessment (HARA) analysis suggests that such dangerous events could be fatal and therefore need to be protected against, for example, at the highest Automotive Safety Integrity Level (ASIL), ASIL D. To ensure that the system can protect against these and other such dangerous events at a sufficiently high safety integrity level, any system, service, or process that includes or relies on any of the above-mentioned functional components needs to be verified and validated for the most stringent ASIL D compliance, which may require significant development effort and cost.

[0030] Thus, various embodiment approaches can attempt to concurrently optimize for both safety and performance while ensuring safety compliance of the resulting results. In at least one embodiment, the path detection module of an ADS can provide improved flexibility and robustness in design while providing safety compliance, such as ASIL D compliance, which can significantly reduce the amount of effort required for development, validation, and verification. Such systems can provide fusion of path detection sources and safety (e.g., ASIL D) monitoring of the fused path produced by the system. Such systems can provide run-time switching between different combinations of localization and live path perception while providing high availability. Such systems can also provide propagated fusion of fused localization and fused path, based at least in part on processes such as dead reckoning. If a vehicle cannot detect lane markers at the current time but has previously detected and stored information about those markers, the vehicle can use information about its speed and heading, along with its historical data, to infer perception data for at least a period of time, such as several seconds, depending on factors such as speed and environment.

[0031] FIG. 2A illustrates an overview 200 of an exemplary path discovery functional safety architecture that may be used with an automated or semi-automated system, device, or process, according to various embodiments. Such an architecture may enable automated safety systems to achieve SAE Level 3 ADS or higher. As shown, components of such an architecture may provide ASIL D (the highest degree of automotive hazard avoidance), ASIL B(D) (lesser), and QM(D) compliance, where quality management (QM) compliance represents a risk that does not specify safety requirements. As discussed above, it may be important for such systems to avoid sending information about incorrect or unsafe paths to the vehicle's controller, such as paths that cause the vehicle to run off the road, which could lead to a serious accident or injury. Therefore, such path determination systems should ensure that the path information sent provides safety integrity, such as path data that is ASIL D compliant. To avoid single points of failure that directly lead to incorrect paths in ASIL D systems, system-level redundancy may be employed so that the failure of a single component should not lead to unsafe behavior or incorrect operation.

[0032] An architecture such as that shown in FIG. 2A can utilize independently and diversely generated lane graphs to be monitored to ensure that each final lane graph used for driving is safe, e.g., has ASIL D certification. In at least one embodiment, the lane graphs can be independently generated based on various modules, such as perception data sources 212, which provide or utilize data from an HD map or live lane perception sources, such as one or more cameras or lidar sources on the vehicle. In at least one embodiment, the ASIL D requirement for an ADS can be achieved by ensuring that the lane graph generated using the combined data, or the “combined” or “fused” lane graph, matches both the live lane perception and the corresponding map data, at least within an acceptable tolerance or variation. This may be referred to as an ASIL decomposition scheme between pairs of independent components, where, for example, ASIL D is split and assigned ASIL B(D) according to ISO 26262 for the perception lane graph comparator and the map lane graph comparator of the set of lane graph comparators 216.

[0033] Such a system can switch at runtime between different diverse modules to ensure that the final output has integrity, such as ASIL D integrity, depending at least in part on the availability of those modules. Because such a system may not be able to ensure continued safe operation unless at least two diverse modules are available to verify that the final output meets safety and other related requirements, the ability to switch at runtime provides benefits, including increased system availability. It is not always sufficient for safety that a routing system be able to detect whether a combined path is correct, for example, with ASIL D integrity. In some cases where a driver cannot be used as a backup, such a system should ensure that the system can continue to generate a correct path until the system reaches a safe state, such as when the vehicle is stopped at a safe location. In at least one embodiment, one or more software components that are relatively simple and lightweight may be used, which has benefits over more complex software components that may provide lower safety integrity and higher latency.

[0034] The high-level architecture 200 in FIG. 2A is separated into two parts: a part related to localization (top of the figure) and a part related to lane graph generation (bottom of the figure). These parts are shown in more detail in FIGS. 2B and 2C, respectively. Localization can involve determining at least the current location of an object (such as a vehicle or robot) in an environment, such as on a road, based on available and reliable information. In this example, there can be multiple localization data sources 202, such as cameras (e.g., front-view cameras) and radar sensors, among other options. Data from each of these sources can be used to generate respective localization decisions, such as a camera-based localization decision and a radar-based localization decision, each of which can involve analyzing captured images or sensor data for purposes of making the respective decision. In this instance, these separate decisions may be passed to the localization fusion component 204, which may generate a “combined” or “fused” localization decision based at least in part on the available individual localization decisions, at least those generated with at least a minimum degree of reliability. Various algorithms or techniques may be used to combine these decisions, such as weighted or spatial averaging, neural networks for inferring optimal values ​​based on available inputs, or complex fusion functions, among other options. If weighting is used, it may be based at least in part on the relative reliability values ​​of the individual decisions. As mentioned above, because there may be disturbances or other incidents that cause at least a portion of a camera or sensor's view to become unreliable, it may be beneficial to combine or fuse data from different subsystems or sources, which may be reliably determined or filled in from another source while retaining reliable data obtained from that camera or sensor.Using only that single system can be problematic, as reliable decisions may not be made with sufficient confidence if part of the view is obstructed.

[0035] One or more localization comparators 206 may be used to compare the individual localization decisions to the fused decision. As described in more detail later herein, these comparators may also utilize additional inputs, such as localization decisions from other systems, such as satellite-based systems. The use of a satellite system may be beneficial in situations such as when a problem occurs on a vehicle and the vehicle needs to safely pull over to the side of the road and stop, and a route may be generated based on satellite data to accomplish this task. Satellite systems may not be affected by problems with individual vehicles and may have additional sensors and processing available to provide additional reliable inputs. The results produced by the localization comparators may be provided as input to a localization confidence monitor 208, which may continuously determine 210 a localization confidence value for at least the fused localization data. This may involve comparing the fused localization decision to the individual localization decisions at each point in time to determine whether there is sufficient agreement. If the fused localization is consistent with respect to the individual localization determinations for ASIL D, the fused localization data may also be used to verify compliance with ASIL D (or other such requirements).

[0036] As mentioned above, there may be times when individual sensors or sources are not capable of providing accurate localization decisions, such as when the view is obscured, the sensor is saturated, or an event affects the reliability of the data. In such cases, the reliability of the localization data may be low, or at least lower than that required by the relevant safety standard or requirement. In architecture 200 such as that shown in FIG. 2A, multiple sources may be dynamically compared, combined, or selected in any of several different combinations so that sources that are out of agreement can be ignored or discarded, at least for the time being. In at least one embodiment, it is sufficient for at least two of these ASIL D-compliant sources to agree on each localization decision to be used from one of those sources. Furthermore, if at least two of those sources agree on the fused localization decision, the fused localization decision may be utilized. As mentioned above, one or more comparators 206 may be used to perform each comparison, and the results of these comparisons may be analyzed by monitor 208 to determine which localization decision is reliable and should be used. In at least one embodiment, these comparators 206 may be considered voters because they vote on whether two or more inputs match, for example, within a tolerance or threshold. Each monitor can then analyze the results of these voters, and if at least two voters agree that the fused localization decision is reliable, the fused localization decision may be utilized. In this example, a reliability or health decision 210 may be provided for consideration, such as to ensure that the system is operating properly. The health decision 210 and the fused (or individual) localization decision may also be provided as inputs to one or more cognitive data sources, such as an HD mapping source, for purposes of determining an appropriate lane graph based at least in part on the localization decision.

[0037] The lane graph portion of this architecture 200 operates somewhat similarly to the localization portion. One or more perception data sources 212 are utilized, which may include one or more HD lane graph maps and one or more perception sources, such as cameras or LIDAR, that can generate data that can be analyzed to attempt to make a live or real-time determination of the lane graph based on this live perception data or what a subsystem on the vehicle (or related to the vehicle) perceives to be an appropriate lane graph based on available information, such as imagery or sensor data. These proposed lane graphs can be provided to a lane graph fusion module 214, which can generate a fused or combined lane graph determination based at least in part on fusing or combining these individual lane graph determinations (or predictions). A set of comparators 216 can also compare these individual and fused lane graphs to determine whether they sufficiently match. The results of these comparisons may be provided to lane graph reliability monitor 218, which may determine whether the fused lane graph decision agrees for two or more of these individual decisions with sufficient confidence and may output a decision 220 of confidence for at least the fused lane graph. If the confidence for the fused decision is sufficiently high, such as meeting at least a minimum confidence threshold or value, fused lane graph 222 may be provided to, for example, a control system for use in navigating the vehicle. If the confidence for the fused lane graph is not sufficiently high, lane graphs from one of the individual or other fused sources may be utilized to provide continued safe operation, such as by selecting the individual lane graph with the highest reliability or health score. As noted above, operating using the individual lane graphs may provide safe operation, but may not be optimal from a performance perspective.

[0038] 2B and 2C show more detailed views of the localization determination portion 230 and lane graph determination portion 260, respectively, of this exemplary architecture 200. In FIG. 2B, localization data sources, such as a corner radar 232 and a front wide camera 236, are shown to provide data to respective localization modules 234, 238, which can produce individual localization decisions that can be provided to a localization fusion module 240 to generate a fused localization decision. The fused localization data, along with each individual localization decision, can be provided to separate camera localization component 242 and radar localization component 244 for comparison, which can be passed to a localization reliability (or health) monitor 242. The reliability monitors can include a primary localization monitor 244 and a secondary localization monitor 246 that can generate respective reliability decisions, which can be analyzed by the localization monitors to generate a single localization reliability value. In at least some embodiments, these reliability decisions can be binary, indicating whether the fusion decision is reliable or not. The reliability can, in at least some embodiments, be determined using a reliability score against a threshold. This health information can be passed to a propagation localization module, which can be used by the localization fusion module for subsequent fusion decisions. The propagation fusion information and fused localization data can also be passed to a localization arbiter 246.

[0039] In FIG. 2C , the portion of the architecture for lane graph determination 260 is shown receiving the fused (or individual) localization decisions and confidence values, or health decisions, to the map lane graph 262, which can use this information, for example, along with HD map data, to generate a lane graph for the current location and environmental conditions or states. One or more components, such as a front-view camera, can provide image data that can be utilized by the lane perception graph component 266 to generate a lane graph based at least in part on recognizing objects, such as lane markers and vehicles, in the captured image or video data. These individual lane graph decisions can be provided to the lane graph fusion module 274, which can generate a fused lane graph decision based at least in part on these or other provided individual lane graph decisions. The fused lane graph decision and the individual lane graph decisions may be provided to comparators 268, 270 for comparing the information and providing the results to a lane graph health monitor, which may include a primary monitor 278 and a secondary monitor 280 for analyzing the comparison data and providing a health or reliability decision to a fuse lane graph monitor module or component 284, which may determine and output health or reliability data for the fused lane graph, thereby enabling the system to ensure that the fused lane graph meets all safety requirements, etc. This health information is also provided to the propagation fused lane graph component, which provides data to the lane graph fusion module 274 for subsequent fused lane graph decisions and to the localization arbiter 272 for determining whether fused lane graph data is available and reliable; if fused lane graph data is not available and reliable, one or more of the individual lane graph decisions should be utilized.

[0040] As shown, to improve the accuracy of the lane graph function, the lane graph fusion module 274 can utilize different sources of lane graph data, such as HD map-based lane graph data and perception-based lane graph data, which can be fused together. This function, in at least one embodiment of this architecture, can be graded QM(D), and any suitable fusion function can be utilized to achieve and maintain the most accurate lane graph output possible. The localization fusion module 240 can operate in a similar manner, providing a QM(D) fusion of multiple localization sources, such as radar-based and camera-based localization sources, to achieve the best accuracy for the localization function. To achieve the required ASIL D completeness for the fused lane graph and fused localization, the fused lane graph and fused localization can be compared to two independent and diverse ASIL B(D) sources, such as HD map-based lane graph and perception-based lane graph, along with camera localization and radar localization, and the results are sent to an ASIL D monitor, which checks the status of the two comparisons. Thus, an exemplary decomposition scheme can be given by: ASIL D → ASIL B(D) + ASIL B(D) Fusion Localization Camera Localization Comparator Radar Localization Comparator ASIL D → ASIL B(D) + ASIL B(D) Fusion Lane Graph HD Map-Based Lane Graph Perception-Based Lane Graph The ASIL D monitor 242, 276 provides a simple check on the lane graph comparator input or the localization comparator input to ensure that both comparators are valid simultaneously so that an ASIL D compliant lane graph or localization decision can be ensured.

[0041] One of the most important aspects of an ADS system performing complex dynamic driving tasks is system availability in the event of a failure. For ADS systems rated at SAE automation levels 3-5, the system should be designed so that a failure can be detected and the system can continue to operate with other available components and signals still functioning correctly. The propagation fused lane graph component 282 can generate a representation of lane lines on the roadway, for example, based on the last good value of the fused lane graph that passed the checks of both the perception lane graph comparator and the map lane graph comparator. This module can take the last good value of the fused lane graph and estimate new lane lines around the ego vehicle, for example, using relative ego motion (e.g., vehicle direction, speed, and acceleration / deceleration). Such an approach may be referred to as dead reckoning in some embodiments because it relies at least in part on historical data when current data may not be sufficiently reliable. In this embodiment, the propagation fused lane graph module 282 is ASIL D compliant and can independently generate lane lines. In at least some embodiments, this propagation-fused lane graph is accurate only for a limited period of time, at least in part due to the ego-motion nature of the error accumulation. Similarly, the propagation-fused localization module 248 can generate a precise six degrees-of-freedom (DOF) position and orientation of the ego-vehicle based, at least in part, on the last good value of the fused localization that passed the checks of both the camera localization comparator and the radar localization comparator. The module takes the last good value of the fused localization and uses the relative ego-motion to estimate a new 6DOF position and orientation of the ego-vehicle. This module is also ASIL D compliant and can independently generate 6DOF localization values. This propagation localization can also be accurate only for a limited period of time, at least in part due to the ego-motion nature of the error accumulation.Therefore, dead reckoning can be used to transform an ASIL D assured lane graph and maintain ASIL D integrity without an available monitor for a limited period of time.

[0042] Propagation fusion for both localization fusion and lane graph fusion can be used in combination with run-time switching between different variants to further improve system availability. For example, if a system is running with radar-based localization and camera-based path perception and the camera-based path perception is lost for a few seconds, the system can continue with this variant on the propagation-fused lane graph for a while and then switch to the camera-based and radar-based localization variant.

[0043] In this example, the localization arbiter 246 can determine when fused localization data is unavailable or not sufficiently reliable and provide this data to the comparator and reliability monitor, allowing the system to decide to operate using the individual localization decisions rather than the fused localization decision, at least for a time instance or period. The architecture allows the fusion of the localization source and the lane graph source to occur in QM(D). This can result in the fusion function not being available for operation when the signals are not accurate enough to pass the comparator test. The arbiter can pass the ASIL B(D) localization and lane graph signals when the fused signal is unavailable. With such an approach, the ASIL D output still holds because the comparator independently checks the incoming inputs and the ASIL D monitor still checks two independent ASIL B(D) values.

[0044] In at least one embodiment, it may be further beneficial to be able to dynamically switch between different combinations of localization and live path awareness. FIG. 3 shows an example architecture 300 that can provide such dynamic switching. This may include, for example, runtime switching between two localization decisions instead of localization and live path awareness. In this example architecture, the map-based lane graph serves as the ASIL D component, and the same ASIL D decomposition arguments described for the previous architecture may be used for localization. This runtime switching between live awareness / localization lane graphs and localization-only lane graphs can provide a significant amount of flexibility and significantly improve availability. Other types of dynamic switching are also possible, such as for map-based lane graphs based on two independent sources of localization, such as camera localization and radar localization, and for live awareness-based lane graphs based on two independent sources of live awareness, such as camera lane awareness and LIDAR lane awareness. Other dynamic switching may be implemented for map-based lane graphs and live awareness-based lane graphs.

[0045] The architecture 300 of Figure 3 also enables dynamic switching between at least two electronic control units (ECUs), such as a main ECU in the vehicle, represented in the upper portion of the architecture in Figure 3, and a satellite-based ECU, represented in the lower portion. As shown, each ECU can include similar components for taking its respective localization data 302, 320, generating a fused localization using fusion components 304, 322, comparing the fused localization to individual localizations using comparators 306, 324, and providing that information to a localization confidence monitor 308. In this example, at least a portion of the localization data from the satellite ECU can be provided as an input to the comparator 306 of the main ECU, which can help determine agreement between additional localization decisions. As shown, each ECU can take a localization determination and use it with a map lane graph module 312, 328, and can use data from one or more perception data sources 310, 330 to determine a fused lane graph using a respective fusion module 316, 332. The fused lane graph can be compared to the individual lane graphs by respective comparators 314, 334 to make a reliability determination by a respective lane graph reliability monitor 318, 336. The reliability values ​​for these lane graphs can be used to determine which lane graph to utilize. In at least some embodiments, the lane graph from the main ECU is utilized unless it provides sufficient reliability, in which case the lane graph determination from the satellite ECU can be utilized. Distributing functional components across two ECUs can further improve availability during normal operation by routing localization and path detection signals across the ECUs, which can compensate for degraded operation or loss of one of the ECUs.

[0046] In some situations, it may be desirable to perform at least some amount of switching, depending, at least in part, on location in the environment. For example, if a vehicle is driving on a highway with very clear lane markers, live lane perception may produce a very accurate lane graph. However, the vehicle may be at a large intersection where there are no clear markers as to how to proceed. In such situations, live lane perception may not be able to provide a reliable lane graph, but a map-based lane graph may be sufficiently reliable. In that case, it may be possible to switch between live lane perception mode and localization mode multiple times during a trip to continue driving safely based on the condition, state, or type of road alone.

[0047] For autonomous vehicles, where decisions need to be made extremely quickly to ensure safety, many of these components and much of this processing may occur on the vehicle itself to avoid latency and connectivity issues. Thus, the vehicle may have one or more processors (e.g., CPUs or GPUs) to perform at least some of this processing. Portions of this functionality that may not be time-critical, such as updating map data, may be performed by a remote server or in the cloud, among other options. For autonomous vehicles, route decisions may need to be made on the order of milliseconds, and therefore it is desirable to have at least independent decision modules produce results in a quick, simplified manner that can be relied upon if the fusion process is delayed or unable to produce results on at least the same time scale for a period of time.

[0048] 4 illustrates an example process 400 for determining a navigation path, such as a lane graph, that may be utilized, according to various embodiments. For this and other processes described herein, unless otherwise specified, it should be understood that there may be additional, fewer, or alternative steps performed in a similar or alternative order, or at least partially in parallel, within the scope of various embodiments. Furthermore, while this example is described with respect to an autonomous vehicle operating within a given lane, it should be understood that other types of safety-compliant paths or graphs may be determined for other autonomous or semi-autonomous vehicles, devices, robots, processes, or systems, according to various embodiments.

[0049] In this example, localization data is obtained from at least two independent, trusted sources at 402. This may include, for example, capturing image or sensor data using various cameras or sensors and making independent localization decisions based on each type of data. Trusted data, in this example, may refer to data obtained from systems or processes determined to comply with one or more regulations or requirements, such as those related to safety, and determined with at least a minimum level of reliability. At 404, a fused localization decision may be generated based at least in part on the independent localization decisions from these independent, trusted sources. As described above, the fusion decision may be made based on any combination or subset of these individual decisions, such as may be based at least in part on factors such as reliability and availability. At 406, the fused localization decision and the individual localization decisions may be provided to one or more comparators to determine a match between at least the independent decisions and the fused localization decision. In at least one embodiment, each comparator compares a respective independent localization decision to the fused decision. At 408, the results of these comparators may be utilized to determine the reliability or soundness of the fused localization decision, such as a binary decision as to whether the fused decision should be used or not, based on the reliability of the fused localization decision meeting at least a minimum reliability threshold, and if the reliability of the fused localization decision does not meet at least a minimum reliability threshold, a different localization decision should be selected for use.

[0050] At 410, the fused localization decision, or other localization decision if the fusion decision is not selected according to the soundness of the reliability information, may be provided along with live perception data from at least two independent, trusted sources to generate an independent lane graph. At 412, a fused lane graph may be generated based at least in part on these individual lane graphs from these independent sources. At 414, the fused lane graph decision and the individual lane graph decisions may be provided to one or more comparators for comparison between the fused lane graph and the individual lane graphs. At 416, a reliability for the fused lane graph may be determined based at least in part on the number of matches from the comparator(s). If at 418, it is determined that the reliability or soundness of the fused lane graph meets at least a minimum reliability threshold or criterion, then at 420, the fused lane graph may be provided to an operational component, such as a drive unit, or other device, system, service, or process for navigation or operation of the respective vehicle. If the confidence in the fused line graph does not meet at least a minimum confidence threshold or criterion, then at 422 an alternative lane graph may be selected to provide navigation or operation, which may be, among other options, one of the individual lane graphs or a fused lane graph from another system or control unit.

[0051] FIG. 5 illustrates another exemplary process 500 that can utilize dead reckoning in localization or lane graph determination, according to various embodiments. In this example, live perception data is obtained from at least two independent, reliable sources at 502. At 504, the live perception data received from each of these independent sources can be analyzed to determine whether the data meets at least a minimum reliability standard, such as may be necessary to ensure compliance with one or more safety regulations or requirements. If perception data is not received from one of these independent sources for a period of time, that source can be treated similarly to one of these sources with insufficient reliability. At 506, if it is determined that perception data from at least one of these independent sources does not meet the corresponding standard, a dead reckoning process can be initiated. This can involve accessing recent fused and / or independent localization data at 580 and determining the vehicle's current movement (e.g., ego-motion) or state at 510. At 512, localization data may be inferred for at least a period of time based at least in part on recent localization data and vehicle movement or state, and the inference may be determined to be reliable. This may involve inferring one or more individual localization decisions, or in various embodiments, inferring a fused localization. Alternatively, if all localization data is determined to meet all relevant criteria, then at 514, individual localization decisions may be determined using live perception data. If there are three or more localization decisions available, then at 516, a fused localization decision may be generated based at least in part on the available localization decisions, and this fused localization information may be analyzed for safety compliance using a process such as that described with respect to FIG. 4.If only an inferred localization determination is available, it can be used for route determination to at least allow the vehicle to operate safely or move to a safe location until reliable data is again available.

[0052] The various techniques presented herein are lightweight enough to execute in real time on various types of devices or systems, such as smart vehicles, robots, servers, personal computers, smart vehicles, or autonomous equipment. Such processing may be performed using data captured or generated on the device or system or received from an external source, such as sensor or map data received over at least one network. The source may be any suitable source, such as a separate client device, a device management system, or a third-party data provider, among other options. In some cases, the processing and / or use of this data may be performed by one of these other devices, systems, or entities and then provided to the vehicle (or another such recipient) for use.

[0053] 6 illustrates an exemplary network configuration 600 that may be used to provide, generate, modify, encode, and / or transmit data. In at least one embodiment, a vehicle 602, or other automated or semi-automated device or system, may generate or receive data using components of a condition monitoring application 604 on the vehicle 602 and data stored locally on the vehicle. In at least one embodiment, a vehicle monitoring application 624 running on a data or content server 620 (e.g., a cloud server or an edge server) may initiate a session associated with at least the vehicle 602 that may utilize a session manager and user data stored in a vehicle database 634, causing content related to route, localization, or operation to be determined or managed by a data manager 626. The route generation module 628 may attempt to utilize data from various sources to provide map, route, or localization data to the vehicle 602, using information that may be stored locally in a map database 632, received from a third-party service 660, or received from a satellite 670, among other options. Compliance of this data with one or more safety requirements may be determined using the safety module 630. In at least some embodiments, this data may be intended to be at least redundant with respect to data determined on-board the vehicle 602. At least a portion of this data may be transmitted to the vehicle 602 using a suitable transmission manager 622 for transmission by download, streaming, or another such transmission channel. An encoder may be used to encode and / or compress this data before transmission to the vehicle 602.In at least one embodiment, a vehicle 602 receiving this data can provide this data to a corresponding condition monitor 604, which may also (or alternatively) include a route generator 608 and a safety compliance module 6104 to analyze the data received by or captured by the vehicle 602 and generate or select appropriate operational instructions or decisions to provide to a control system 606 for the vehicle, such as to cause the vehicle to navigate along a determined route. A decoder may also be used to appropriately decode data received over network(s) 640. In at least one embodiment, at least a portion of this data (e.g., map data or historical perception data) may already be stored on, generated on-board, or accessible to the vehicle 602 such that transmission over network 640 is not required for at least that portion of the data, such as if the data was previously downloaded or stored locally on a hard drive or optical disk. In at least one embodiment, a transmission mechanism such as data streaming may be used to transfer this data from server 620. In at least one embodiment, at least a portion of this data may be obtained, determined, or streamed from another source, such as a separate client device 650 or third-party service 660, which may also include functionality for making the associated decisions. In at least one embodiment, some of this functionality may be implemented using multiple computing devices or multiple processors within one or more computing devices, such as may include a combination of a CPU and a GPU.

[0054] In this example, the devices may include any suitable device with computing or sensing capabilities, such as a vehicle, a robot, an autonomous system, a desktop computer, a notebook computer, a set-top box, a streaming device, a gaming console, a smartphone, a tablet computer, a smart vehicle, a robotic-assisted machine, a virtual reality headset, an augmented reality goggles, a wearable computer, or a smart television. Each device may be capable of sending and receiving requests over at least one wired or wireless network, such as the Internet, an Ethernet, a local area network (LAN), or a cellular network, among other options. In this example, these requests may be submitted to or received from an address associated with a cloud provider, which may operate or control one or more electronic resources in the cloud provider environment, such as a data center or server farm. In at least one embodiment, the requests may be received or processed by at least one edge server located on the network edge and outside at least one security layer associated with the cloud provider environment. In this way, latency may be reduced by allowing client devices to interact with servers that are closer to them, while also improving the security of resources in the cloud provider environment.

[0055] In at least one embodiment, such a system may be used for automated operations. In other embodiments, such a system may be used for other purposes, such as to provide image or video content for testing or validating autonomous machine applications or to perform deep learning operations. In at least one embodiment, such a system may be implemented using edge devices or may incorporate one or more virtual machines (VMs). In at least one embodiment, such a system may be implemented at least in part in a data center or at least in part using cloud computing resources.

[0056] Inference and Training Logic 7A illustrates inference and / or training logic 715 used to perform inference and / or training operations associated with one or more embodiments. More details regarding inference and / or training logic 715 are provided below in conjunction with FIG. 7A and / or FIG. 7B.

[0057] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, code and / or data storage 701 for storing forward and / or output weights and / or input / output data and / or other parameters for configuring neurons or layers of a neural network that are trained and / or used to infer in one or more aspects of the embodiment. In at least one embodiment, the training logic 715 may include or be coupled to code and / or data storage 701 for storing graph code or other software for controlling timing and / or ordering, and may include, in the code and / or data storage 701, integer and / or floating point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code such as graph code loads weights or other parameter information into a processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, code and / or data storage 701 stores weight parameters and / or input / output data for each layer of a neural network that is trained or used in conjunction with one or more embodiments during forward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, any portion of code and / or data storage 701 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory.

[0058] In at least one embodiment, any portion of code and / or data storage 701 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or code and / or data storage 701 may be cache memory, dynamic randomly addressable memory (“DRAM”), static randomly addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the selection of whether code and / or code and / or data storage 701 is internal or external to the processor, or whether it consists of DRAM, SRAM, flash, or some other storage type, may depend on available storage, on-chip versus off-chip, latency requirements of the training and / or inference functions being performed, batch size of data used in inferencing and / or training of the neural network, or some combination of these factors.

[0059] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, code and / or data storage 705 for storing back and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used to infer in accordance with aspects of one or more embodiments. In at least one embodiment, the code and / or data storage 705 stores weight parameters and / or input / output data for each layer of a neural network trained or used in conjunction with one or more embodiments during backpropagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, training logic 715 may include or be coupled to code and / or data storage 705 for storing graph code or other software for controlling timing and / or sequence, and in code and / or data storage 705, logic including integer and / or floating point units (collectively, arithmetic logic units (ALUs) into which weights and / or other parameter information should be loaded. In at least one embodiment, code such as graph code may load weights or other parameter information into processor ALUs based on the architecture of the neural network to which the code corresponds. In at least one embodiment, any portion of code and / or data storage 705 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of code and / or data storage 705 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 705 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage.In at least one embodiment, the selection of whether code and / or data storage 705 is internal or external to the processor, for example, or whether it consists of DRAM, SRAM, flash, or some other storage type, may depend on available storage, on-chip versus off-chip, latency requirements of the training and / or inference functions being performed, batch sizes of data used in inferring and / or training of the neural network, or some combination of these factors.

[0060] In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be separate storage structures. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be the same storage structure. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be partially the same storage structure and partially separate storage structures. In at least one embodiment, code and / or data storage 701 and any portion of code and / or data storage 705 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory.

[0061] In at least one embodiment, the inference and / or training logic 715 may include one or more arithmetic logic units (“ALUs”) 710, including, but not limited to, integer and / or floating point units, for performing logical and / or mathematical operations based at least in part on or indicated by the training and / or inference code (e.g., graph code), the results of which may produce activations (e.g., output values ​​from layers or neurons in a neural network) that are stored in activation storage 720, where these activations are a function of input / output and / or weight parameter data stored in code and / or data storage 701 and / or code and / or data storage 705. In at least one embodiment, the activations stored in activation storage 720 are generated according to linear algebra and / or matrix-based mathematics performed by ALU(s) 710 in response to executing instructions or other code, and the weight values ​​stored in code and / or data storage 705 and / or code and / or data storage 701 are used as operands along with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in code and / or data storage 705 or code and / or data storage 701, or in another storage, on-chip or off-chip.

[0062] In at least one embodiment, ALU(s) 710 are contained within one or more processors or other hardware logic devices or circuits, while in other embodiments, ALU(s) 710 may be external to the processor or other hardware logic device or circuit (e.g., a coprocessor) that uses them. In at least one embodiment, ALU 710 may be contained within an execution unit of a processor, or otherwise within a bank of ALUs accessible by execution units of a processor, either within the same processor or distributed among different processors of different types (e.g., central processing units, graphics processing units, fixed function units, etc.). In at least one embodiment, code and / or data storage 701, code and / or data storage 705, and activation storage 720 may be on the same processor or other hardware logic device or circuit, while in other embodiments, they may be in different processors or other hardware logic devices or circuits, or in some combination of the same processor or other hardware logic device or circuit and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation storage 720 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Additionally, inference and / or training code may be stored with other code accessible to the processor or other hardware logic or circuitry, and may be fetched and / or processed using the processor's fetch, decode, schedule, execute, retirement, and / or other logic circuitry.

[0063] In at least one embodiment, activation storage 720 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, activation storage 720 may be completely or partially within or external to one or more processors or other logic circuits. In at least one embodiment, the selection of whether activation storage 720 is internal or external to a processor, or whether it comprises DRAM, SRAM, flash, or some other storage type, may depend on available storage, on-chip versus off-chip, latency requirements of the training and / or inference functions being performed, batch sizes of data used in inference and / or training of neural networks, or some combination of these factors. In at least one embodiment, the inference and / or training logic 715 shown in FIG. 7a may be used in conjunction with an application-specific integrated circuit ("ASIC"), such as a Tensorflow® processing unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., "Lake Crest") processor from Intel Corp. In at least one embodiment, the inference and / or training logic 715 shown in FIG. 7a may be used in conjunction with other hardware, such as central processing unit ("CPU") hardware, graphics processing unit ("GPU") hardware, or a field programmable gate array ("FPGA").

[0064] FIG. 7B illustrates inference and / or training logic 715, according to at least one or more embodiments. In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, hardware logic in which computational resources are dedicated or otherwise used only in conjunction with weight values ​​or other information corresponding to one or more layers of neurons in a neural network. In at least one embodiment, the inference and / or training logic 715 illustrated in FIG. 7B may be used in conjunction with an application-specific integrated circuit (ASIC), such as a Tensorflow® processing unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, the inference and / or training logic 715 illustrated in FIG. 7B may be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware, such as a field-programmable gate array (FPGA). In at least one embodiment, inference and / or training logic 715 includes, but is not limited to, code and / or data storage 701 and code and / or data storage 705, which may be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. In at least one embodiment shown in FIG. 7B , code and / or data storage 701 and code and / or data storage 705 are each associated with dedicated computational resources, such as computation hardware 702 and computation hardware 706, respectively. In at least one embodiment, computation hardware 702 and computation hardware 706 each include one or more ALUs that perform mathematical functions, such as linear algebra functions, solely on the information stored in code and / or data storage 701 and code and / or data storage 705, respectively, with the results stored in activation storage 720.

[0065] In at least one embodiment, each of the code and / or data storage 701 and 705 and corresponding computation hardware 702 and 706 corresponds to a different layer of a neural network, whereby activations resulting from one “storage / computation pair 701 / 702” of code and / or data storage 701 and computation hardware 702 are provided as input to a “storage / computation pair 705 / 706” of code and / or data storage 705 and computation hardware 706 to mirror the conceptual organization of the neural network. In at least one embodiment, each of the storage / computation pairs 701 / 702 and 705 / 706 may correspond to two or more neural network layers. In at least one embodiment, additional storage / computation pairs (not shown) may be included in the inference and / or training logic 715 after or in parallel with the storage / computation pairs 701 / 702 and 705 / 706.

[0066] Data Center 8 illustrates an exemplary data center 800 in which at least one embodiment may be used. In at least one embodiment, data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

[0067] 8, data center infrastructure layer 810 may include a resource orchestrator 812, grouped computing resources 814, and node computing resources (“node CRs”) 816(1) through 816(N), where “N” represents any positive integer. In at least one embodiment, node CRs 816(1) through 816(N) may include, without limitation, any number of central processing units (“CPUs”) or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules, cooling modules, etc. In at least one embodiment, one or more of the nodes CR from among the nodes CR 816(1) through 816(N) may be a server having one or more of the computing resources described above.

[0068] In at least one embodiment, the grouped computing resources 814 may include a distinct grouping of node CRs housed within one or more racks (not shown), or many racks housed in a data center at various geographic locations (also not shown). The distinct groupings of node CRs within the grouped computing resources 814 may include grouped compute resources, network resources, memory resources, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs, including CPUs or processors, may be grouped within one or more racks to provide compute resources to support one or more workloads. In at least one embodiment, the one or more racks may also include any number of power modules, cooling modules, and network switches, in any combination.

[0069] In at least one embodiment, resource orchestrator 812 may configure or otherwise control one or more nodes CR 816(1)-816(N) and / or grouped computing resources 814. In at least one embodiment, resource orchestrator 812 may include a software design infrastructure (“SDI”) management entity for data center 800. In at least one embodiment, resource orchestrator may include hardware, software, or some combination thereof.

[0070] 8 , framework layer 820 includes a job scheduler 822, a configuration manager 824, a resource manager 826, and a distributed file system 828. In at least one embodiment, framework layer 820 may include a framework for supporting software 832 in software layer 830 and / or one or more applications 842 in application layer 840. In at least one embodiment, software 832 or application(s) 842 may include web-based service software or applications, such as those offered by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. In at least one embodiment, framework layer 820 may be a type of free and open-source software web application framework, such as, but not limited to, Apache Spark™ (hereinafter “Spark”), which may utilize distributed file system 828 for large-scale data processing (e.g., “big data”). In at least one embodiment, job scheduler 822 may include a Spark driver to facilitate scheduling of workloads supported by various tiers of data center 800. In at least one embodiment, configuration manager 824 may be capable of configuring different tiers, such as software tier 830, as well as framework tier 820 including Spark and distributed file system 828 to support large-scale data processing. In at least one embodiment, resource manager 826 may be capable of managing clustered or grouped computing resources mapped or allocated to support distributed file system 828 and job scheduler 822. In at least one embodiment, the clustered or grouped computing resources may include grouped computing resources 814 in data center infrastructure tier 810.In at least one embodiment, resource manager 826 may manage these mapped or allocated computing resources in coordination with resource orchestrator 812.

[0071] In at least one embodiment, software 832 included in software layer 830 may include software used by nodes CR 816(1)-816(N), grouped computing resources 814, and / or at least a portion of distributed file system 828 of framework layer 820. The one or more types of software may include, but are not limited to, internet web page searching software, email virus scanning software, database software, and streaming video content software.

[0072] In at least one embodiment, the application(s) 842 included in the application layer 840 may include one or more types of applications used by the nodes CR 816(1)-816(N), the grouped computing resources 814, and / or at least a portion of the distributed file system 828 of the framework layer 820. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive compute, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0073] In at least one embodiment, any of configuration manager 824, resource manager 826, and resource orchestrator 812 may implement any number and types of self-correcting actions based on any amount and type of data obtained in any technically feasible manner. In at least one embodiment, the self-correcting actions may relieve data center operators of data center 800 from determining potentially faulty configurations and potentially avoiding underutilized and / or underperforming portions of the data center.

[0074] In at least one embodiment, data center 800 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, machine learning models may be trained by calculating weight parameters according to a neural network architecture using the software and computing resources described above with respect to data center 800. In at least one embodiment, the trained machine learning models corresponding to the one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 800 by using the weight parameters calculated through one or more training techniques described herein.

[0075] In at least one embodiment, the data center may use a CPU, application specific integrated circuit (ASIC), GPU, FPGA, or other hardware to perform training and / or inference using the resources described above. Additionally, one or more of the software and / or hardware resources described above may be configured as a service to enable a user to train or perform inference on information, such as image recognition, speech recognition, or other artificial intelligence services.

[0076] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. More details regarding inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B. In at least one embodiment, inference and / or training logic 715 may be used in the system of Figure 8 for inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functionality and / or architecture, or neural network use cases described herein.

[0077] Such components may be used to combine the constituent images into a single representation using parameters determined from one or more quality assessment values.

[0078] Computer Systems 9 is a block diagram illustrating an exemplary computer system, which may be a system 900 having interconnected devices and components, a system-on-a-chip (SOC), or some combination thereof, formed with a processor that may include an execution unit for executing instructions, according to at least one embodiment. In at least one embodiment, computer system 900 may include components such as processor 902 to employ an execution unit that includes logic for implementing algorithms to process data according to the present disclosure, such as, but not limited to, embodiments described herein. In at least one embodiment, computer system 900 may include a processor such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™, and / or StrongARM™, Intel® Core™, or Intel® Nervana™ microprocessors available from Intel Corporation of Santa Clara, California, although other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, etc.) may also be used. In at least one embodiment, computer system 900 may run a version of the WINDOWS® operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (e.g., UNIX® and Linux®), embedded software, and / or graphical user interfaces may also be used.

[0079] Embodiments may be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, an embedded application may include a microcontroller, a digital signal processor ("DSP"), a system-on-chip, a network computer ("NetPC"), a set-top box, a network hub, a wide area network ("WAN") switch, or any other system capable of implementing one or more instructions according to at least one embodiment.

[0080] In at least one embodiment, computer system 900 may include, but is not limited to, a processor 902, which may include one or more execution units 908 for performing machine learning model training and / or inference according to the techniques described herein. In at least one embodiment, computer system 900 is a single-processor desktop or server system, while in other embodiments, computer system 900 may be a multi-processor system. In at least one embodiment, processor 902 may include, but is not limited to, a complex instruction set computer (“CISC”) microprocessor, a reduced instruction set computing (“RISC”) microprocessor, a very long instruction word (“VLIW”) microprocessor, a processor implementing a combination of instruction sets, or any other processor device, such as, for example, a digital signal processor. In at least one embodiment, the processor 902 may be coupled to a processor bus 910 that may transmit data signals between the processor 902 and other components in the computer system 900.

[0081] In at least one embodiment, processor 902 may include, but is not limited to, level 1 ("L1") internal cache memory ("cache") 904. In at least one embodiment, processor 902 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, cache memory may reside external to processor 902. Other embodiments may include a combination of both internal and external cache, depending on the particular implementation and needs. In at least one embodiment, register file 906 may store different types of data in various registers, including, but not limited to, integer registers, floating-point registers, status registers, and instruction pointer registers.

[0082] In at least one embodiment, an execution unit 908 including logic for performing integer and floating-point operations may also be present in the processor 902. In at least one embodiment, the processor 902 may also include a microcode (“u-code”) read-only memory (“ROM”) that stores microcode for some macroinstructions. In at least one embodiment, the execution unit 908 may include logic for dealing with a packed instruction set 909. In at least one embodiment, by including the packed instruction set 909, along with associated circuitry for executing the instructions, in the instruction set of the general-purpose processor 902, operations used by many multimedia applications may be performed using packed data in the general-purpose processor 902. In one or more embodiments, many multimedia applications may be accelerated and run more efficiently by using the full width of the processor's data bus to perform operations on packed data, which may eliminate the need to transfer smaller units of data across the processor's data bus to perform one or more operations one data element at a time.

[0083] In at least one embodiment, the execution unit 908 may also be used in a microcontroller, an embedded processor, a graphics device, a DSP, and other types of logic circuitry. In at least one embodiment, the computer system 900 may include, but is not limited to, a memory 920. In at least one embodiment, the memory 920 may be implemented as a dynamic random access memory ("DRAM") device, a static random access memory ("SRAM") device, a flash memory device, or other memory device. In at least one embodiment, the memory 920 may store instruction(s) 919 and / or data 921 represented by data signals that may be executed by the processor 902.

[0084] In at least one embodiment, a system logic chip may be coupled to the processor bus 910 and the memory 920. In at least one embodiment, the system logic chip may include, but is not limited to, a memory controller hub (“MCH”) 916, and the processor 902 may communicate with the MCH 916 via the processor bus 910. In at least one embodiment, the MCH 916 may provide a high-bandwidth memory path 918 to the memory 920 for instruction and data storage, and for storage of graphics commands, data, and textures. In at least one embodiment, the MCH 916 may direct data signals between the processor 902, the memory 920, and other components in the computer system 900, and may bridge data signals between the processor bus 910, the memory 920, and the system I / O 922. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH 916 may be coupled to memory 920 through a high-bandwidth memory path 918, and the graphics / video card 912 may be coupled to the MCH 916 through an Accelerated Graphics Port (“AGP”) interconnect 914.

[0085] In at least one embodiment, computer system 900 may use system I / O 922, a proprietary hub interface bus, to couple MCH 916 to I / O controller hub (“ICH”) 930. In at least one embodiment, ICH 930 may provide direct connectivity to several I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, but is not limited to, a high-speed I / O bus for connecting peripherals to memory 920, a chipset, and processor 902. Examples may include, but are not limited to, an audio controller 929, a firmware hub ("flash BIOS") 928, a wireless transceiver 926, data storage 924, a legacy I / O controller 923 including a user input and keyboard interface 925, a serial expansion port 927 such as a Universal Serial Bus ("USB"), and a network controller 934. Data storage 924 may comprise a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0086] In at least one embodiment, Figure 9 illustrates a system including interconnected hardware devices or "chips," although in other embodiments, Figure 9 may illustrate an exemplary system-on-a-chip ("SoC"). In at least one embodiment, the devices may be interconnected with proprietary interconnects, standard interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of computer system 900 are interconnected using a compute express link (CXL) interconnect.

[0087] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. More details regarding inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B. In at least one embodiment, inference and / or training logic 715 may be used in the system of Figure 9 for inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functionality and / or architecture, or neural network use cases described herein.

[0088] Such components may be used to combine the constituent images into a single representation using parameters determined from one or more quality assessment values.

[0089] 10 is a block diagram illustrating an electronic device 1000 for utilizing a processor 1010, according to at least one embodiment. In at least one embodiment, the electronic device 1000 may be, for example, but not limited to, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop, a tablet, a mobile device, a phone, an embedded computer, or any other suitable electronic device.

[0090] In at least one embodiment, system 1000 may include a processor 1010 communicatively coupled to any suitable number or type of components, peripherals, modules, or devices, including, but not limited to, a 1010 bus, a System Management Bus (“SMBus”), a Low Pin Count (“LPC”) bus, a Serial Peripheral Interface (“SPI”), a High Definition Audio (“HDA”) bus, a Serial Advance Technology Attachment (“SATA”) bus, a Universal Serial Bus (“USB”) (versions 1, 2, and 3), or a Universal Asynchronous Receiver / Transmitter (“UART”) bus. In at least one embodiment, FIG. 10 illustrates a system including interconnected hardware devices or "chips," although in other embodiments, FIG. 10 may illustrate an exemplary system-on-a-chip ("SoC"). In at least one embodiment, the devices illustrated in FIG. 10 may be interconnected with a proprietary interconnect, a standard interconnect (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of FIG. 10 are interconnected using a Compute Express Link (CXL) interconnect.

[0091] In at least one embodiment, FIG. 10 includes a display 1024, a touch screen 1025, a touch pad 1030, a Near Field Communication ("NFC") unit 1045, a sensor hub 1040, a thermal sensor 1046, an Express Chipset ("EC") 1035, a Trusted Platform Module ("TPM") 1038, a BIOS / firmware / flash memory ("BIOS,FW flash") 1022, a DSP 1060, a drive 1020, such as a solid state disk ("SSD") or hard disk drive ("HDD"), a wireless local area network ("WLAN") unit 1050, a Bluetooth unit 1052, a wireless wide area network ("WWAN") unit 1054, a Bluetooth module 1056, a Bluetooth-enabled device 1058 ... Network) 1056, Global Positioning System (GPS) 1055, a camera such as a USB 3.0 camera (“USB 3.0 Camera”) 1054, and / or a Low Power Double Data Rate (“LPDDR”) memory unit (“LPDDR3”) implemented, for example, in the LPDDR3 standard. Each of these components may be implemented in any suitable manner.

[0092] In at least one embodiment, other components may be communicatively coupled to the processor 1010 through the components described above. In at least one embodiment, an accelerometer 1041, an ambient light sensor (“ALS”) 1042, a compass 1043, and a gyroscope 1044 may be communicatively coupled to the sensor hub 1040. In at least one embodiment, a thermal sensor 1039, a fan 1037, a keyboard 1046, and a touchpad 1030 may be communicatively coupled to the EC 1035. In at least one embodiment, a speaker 1063, headphones 1064, and a microphone (“mic”) 1065 may be communicatively coupled to an audio unit (“audio codec and class D amplifier”) 1062, which may be communicatively coupled to the DSP 1060. In at least one embodiment, the audio unit 1064 may include, for example, without limitation, an audio coder / decoder ("codec") and a Class D amplifier. In at least one embodiment, a SIM card ("SIM") 1057 may be communicatively coupled to the WWAN unit 1056. In at least one embodiment, components such as the WLAN unit 1050 and Bluetooth unit 1052, and the WWAN unit 1056 may be implemented in a Next Generation Form Factor ("NGFF").

[0093] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. More details regarding inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B. In at least one embodiment, inference and / or training logic 715 may be used in the system of Figure 10 for inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functionality and / or architecture, or neural network use cases described herein.

[0094] Such components may be used to combine the constituent images into a single representation using parameters determined from one or more quality assessment values.

[0095] 11 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, system 1100 includes one or more processors 1102 and one or more graphics processors 1108 and may be a single-processor desktop system, a multiprocessor workstation system, or a server system with multiple processors 1102 or processor cores 1107. In at least one embodiment, system 1100 is a processing platform integrated into a system-on-chip (SoC) integrated circuit for use in a mobile, handheld, or embedded device.

[0096] In at least one embodiment, system 1100 may include or be incorporated within a server-based gaming platform, a game console including a game and media console, a mobile gaming console, a handheld game console, or an online game console. In at least one embodiment, system 1100 is a mobile phone, a smart phone, a tablet computing device, or a mobile internet device. In at least one embodiment, processing system 1100 may also include, be coupled to, or be incorporated within a wearable device, such as a smart watch wearable device, a smart eyewear device, an augmented reality device, or a virtual reality device. In at least one embodiment, processing system 1100 is a television or set-top box device having one or more processors 1102 and a graphical interface generated by one or more graphics processors 1108.

[0097] In at least one embodiment, the one or more processors 1102 each include one or more processor cores 1107 for processing instructions that, when executed, perform operations for system and user software. In at least one embodiment, each of the one or more processor cores 1107 is configured to process a particular instruction set 1109. In at least one embodiment, the instruction set 1109 may facilitate Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computing via Very Long Instruction Word (VLIW). In at least one embodiment, the processor cores 1107 may each process a different instruction set 1109, and the instruction set 1109 may include instructions to facilitate emulation of other instruction sets. In at least one embodiment, the processor cores 1107 may also include other processing devices, such as a digital signal processor (DSP).

[0098] In at least one embodiment, the processor 1102 includes cache memory 1104. In at least one embodiment, the processor 1102 can have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory is shared among various components of the processor 1102. In at least one embodiment, the processor 1102 also uses an external cache (e.g., a level 3 (L3) cache or a last level cache (LLC)) (not shown), which can be shared among the processor cores 1107 using known cache coherency techniques. In at least one embodiment, a register file 1106 is additionally included in the processor 1102, which can include different types of registers (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers) for storing different types of data. In at least one embodiment, the register file 1106 can include general-purpose registers or other registers.

[0099] In at least one embodiment, one or more processors 1102 are coupled to one or more interface buses 1110 for transmitting communication signals, such as address, data, or control signals, between the processors 1102 and other components in the system 1100. In at least one embodiment, the interface bus 1110 in one embodiment may be a processor bus, such as a version of a Direct Media Interface (DMI) bus. In at least one embodiment, the interface 1110 is not limited to a DMI bus, but may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), a memory bus, or other types of interface buses. In at least one embodiment, the processor(s) 1102 include an integrated memory controller 1116 and a platform controller hub 1130. In at least one embodiment, a memory controller 1116 facilitates communication between memory devices and other components of the system 1100, and a platform controller hub (PCH) 1130 provides connectivity to I / O devices via a local I / O bus.

[0100] In at least one embodiment, memory device 1120 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or any other memory device with performance suitable for serving as process memory. In at least one embodiment, memory device 1120 may operate as system memory for system 1100 to store data 1122 and instructions 1121 for use when one or more processors 1102 execute applications or processes. In at least one embodiment, memory controller 1116 also couples to an optional external graphics processor 1112, which may communicate with one or more graphics processors 1108 in processor 1102 to perform graphics and media operations. In at least one embodiment, display device 1111 may be connected to processor(s) 1102. In at least one embodiment, display device 1111 may include one or more of an internal display device, such as in a mobile electronic device or laptop device, or an external display device attached via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, display device 1111 may include a head mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

[0101] In at least one embodiment, the platform controller hub 1130 allows peripherals to connect to the memory device 1120 and the processor 1102 via a high-speed I / O bus. In at least one embodiment, the I / O peripherals include, but are not limited to, an audio controller 1146, a network controller 1134, a firmware interface 1128, a wireless transceiver 1126, a touch sensor 1125, and a data storage device 1124 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1124 can be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensor 1125 can include a touch screen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, wireless transceiver 1126 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at least one embodiment, firmware interface 1128 enables communication with system firmware and may be, for example, a unified extensible firmware interface (UEFI). In at least one embodiment, network controller 1134 may enable network connectivity to a wired network. In at least one embodiment, a high-performance network controller (not shown) couples to interface bus 1110. In at least one embodiment, audio controller 1146 is a multi-channel high-definition audio controller.In at least one embodiment, system 1100 includes an optional legacy I / O controller 1140 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system. In at least one embodiment, platform controller hub 1130 can also connect to one or more universal serial bus (USB) controller 1142-connected input devices, such as a keyboard and mouse 1143 combination, a camera 1144, or other USB input devices.

[0102] In at least one embodiment, instances of memory controller 1116 and platform controller hub 1130 may be embedded in a discreet external graphics processor, such as external graphics processor 1112. In at least one embodiment, platform controller hub 1130 and / or memory controller 1116 may be external to one or more processors 1102. For example, in at least one embodiment, system 1100 may include external memory controller 1116 and platform controller hub 1130, which may be configured as a memory controller hub and a peripheral controller hub within a system chipset in communication with processor(s) 1102.

[0103] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. More details regarding inference and / or training logic 715 are provided below in conjunction with FIG. 7A and / or FIG. 7B . In at least one embodiment, portions or all of inference and / or training logic 715 may be incorporated into graphics processor 1500. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more of the ALUs embodied in the graphics processor. Moreover, in at least one embodiment, the inference and / or training operations described herein may be performed using logic other than that shown in FIG. 7A or FIG. 7B . In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure the ALUs of the graphics processor for implementing one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0104] Such components may be used to combine the constituent images into a single representation using parameters determined from one or more quality assessment values.

[0105] 12 is a block diagram of a processor 1200 having one or more processor cores 1202A-1202N, an integrated memory controller 1214, and an integrated graphics processor 1208, according to at least one embodiment. In at least one embodiment, the processor 1200 may include additional cores, up to additional core 1202N, represented by dashed boxes. In at least one embodiment, each of the processor cores 1202A-1202N includes one or more internal cache units 1204A-1204N. In at least one embodiment, each processor core also has access to one or more shared cached units 1206.

[0106] In at least one embodiment, the internal cache units 1204A-1204N and the shared cache unit 1206 represent a cache memory hierarchy within the processor 1200. In at least one embodiment, the cache memory units 1204A-1204N may include at least one level of instruction and data cache within each processor core, and one or more levels of shared mid-level cache, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of cache, where the highest level of cache before external memory is classified as LLC. In at least one embodiment, cache coherency logic maintains coherency between the various cache units 1206 and 1204A-1204N.

[0107] In at least one embodiment, processor 1200 may also include a set of one or more bus controller units 1216 and a system agent core 1210. In at least one embodiment, one or more bus controller units 1216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, system agent core 1210 provides management functionality for various processor components. In at least one embodiment, system agent core 1210 includes one or more integrated memory controllers 1214 for managing access to various external memory devices (not shown).

[0108] In at least one embodiment, one or more of the processor cores 1202A-1202N include support for simultaneous multithreading. In at least one embodiment, the system agent core 1210 includes components for coordinating and operating the cores 1202A-1202N during multithreaded processing. In at least one embodiment, the system agent core 1210 may additionally include a power control unit (PCU), which includes logic and components for adjusting the power state of one or more of the processor cores 1202A-1202N and the graphics processor 1208.

[0109] In at least one embodiment, processor 1200 additionally includes a graphics processor 1208 for performing graphics processing operations. In at least one embodiment, graphics processor 1208 couples to a system agent core 1210 that includes a shared cache unit 1206 and one or more integrated memory controllers 1214. In at least one embodiment, system agent core 1210 also includes a display controller 1211 for driving graphics processor output to one or more coupled displays. In at least one embodiment, display controller 1211 may also be a separate module coupled to graphics processor 1208 via at least one interconnect or may be incorporated within graphics processor 1208.

[0110] In at least one embodiment, a ring-based interconnect unit 1212 is used to couple the internal components of processor 1200. In at least one embodiment, alternative interconnect units, such as a point-to-point interconnect, a switched interconnect, or other techniques, may be used. In at least one embodiment, graphics processor 1208 couples to ring interconnect 1212 via I / O link 1213.

[0111] In at least one embodiment, I / O link 1213 represents at least one of several types of I / O interconnect, including an on-package I / O interconnect that facilitates communication between various processor components and a high-performance embedded memory module 1218, such as an eDRAM module. In at least one embodiment, each of processor cores 1202A-1202N and graphics processor 1208 use embedded memory module 1218 as a shared last-level cache.

[0112] In at least one embodiment, processor cores 1202A-1202N are homogeneous cores that execute a common instruction set architecture. In at least one embodiment, processor cores 1202A-1202N are heterogeneous in terms of instruction set architecture (ISA), where one or more of processor cores 1202A-1202N execute a common instruction set and one or more other of processor cores 1202A-1202N execute a subset of the common instruction set or a different instruction set. In at least one embodiment, processor cores 1202A-1202N are heterogeneous in terms of microarchitecture, where one or more cores with relatively high power consumption are combined with one or more cores with lower power consumption. In at least one embodiment, processor 1200 may be implemented on one or more chips or as an SoC integrated circuit.

[0113] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. More details regarding inference and / or training logic 715 are provided below in conjunction with FIG. 7A and / or FIG. 7B. In at least one embodiment, portions or all of inference and / or training logic 715 may be incorporated into processor 1200. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more of the ALUs embodied in graphics processor 1512, graphics core(s) 1202A-1202N, or other components in FIG. 12. Moreover, in at least one embodiment, the inference and / or training operations described herein may be performed using logic other than that shown in FIG. 7A or FIG. 7B. In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that comprise the ALU of graphics processor 1200 for implementing one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0114] Such components may be used to combine the constituent images into a single representation using parameters determined from one or more quality assessment values.

[0115] Virtualized Computing Platform 13 is an exemplary data flow diagram for a process 1300 for generating and deploying an image processing and inference pipeline, according to at least one embodiment. In at least one embodiment, the process 1300 may be deployed for use with imaging devices, processing devices, and / or other device types at one or more facilities 1302. The process 1300 may be executed within a training system 1304 and / or a deployment system 1306. In at least one embodiment, the training system 1304 may be used to train, deploy, and implement machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use in the deployment system 1306. In at least one embodiment, the deployment system 1306 may be configured to offload processing and compute resources between distributed computing environments to reduce infrastructure requirements at the facility 1302. In at least one embodiment, one or more applications in the pipeline may use or call services (e.g., inference, visualization, compute, AI, etc.) of the deployment system 1306 during execution of the application.

[0116] In at least one embodiment, some of the applications used in the advanced processing and inference pipeline may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning models may be trained at facility 1302 using data (e.g., imaging data) 1308 generated at facility 1302 (and stored in one or more picture archiving and communication system (PACS) servers at facility 1302), may be trained using imaging or sequencing data 1308 from another facility(ies), or a combination thereof. In at least one embodiment, training system 1304 may be used to provide applications, services, and / or other resources for generating practical, deployable machine learning models for deployment system 1306.

[0117] In at least one embodiment, model registry 1324 may be backed by object storage that may support versioning and object metadata. In at least one embodiment, object storage may be accessible, for example, from within a cloud platform, through a cloud storage (e.g., cloud 1426 of FIG. 14 ) compatible application programming interface (API). In at least one embodiment, machine learning models in model registry 1324 may be uploaded, listed, modified, or deleted by a system developer or partner interacting with the API. In at least one embodiment, the API may provide access to methods that allow a user with appropriate credentials to associate a model with an application, so that the model can be executed as part of the execution of a containerized instantiation of the application.

[0118] In at least one embodiment, the training pipeline 1404 ( FIG. 14 ) may include a scenario in which the facility 1302 is training its own machine learning models or has existing machine learning models that need to be optimized or updated. In at least one embodiment, imaging data 1308 generated by one or more imaging devices, sequencing devices, and / or other device types may be received. In at least one embodiment, once the imaging data 1308 is received, AI-assisted annotation 1310 may be used to assist in generating annotations corresponding to the imaging data 1308 to be used as ground truth data for the machine learning models. In at least one embodiment, the AI-assisted annotation 1310 may include one or more machine learning models (e.g., convolutional neural networks (CNNs)), which may be trained to generate annotations corresponding to several types of imaging data 1308 (e.g., from several devices). In at least one embodiment, the AI-assisted annotations 1310 may then be used directly or may be adjusted or fine-tuned using an annotation tool to generate ground truth data. In at least one embodiment, the AI-assisted annotations 1310, the labeled clinical data 1312, or a combination thereof may be used as ground truth data for training a machine learning model. In at least one embodiment, the trained machine learning model may be referred to as an output model 1316 and may be used by the deployment system 1306 as described herein.

[0119] In at least one embodiment, the training pipeline 1404 ( FIG. 14 ) may include a scenario in which the facility 1302 needs a machine learning model for use in performing one or more processing tasks for one or more applications in the installation system 1306, but the facility 1302 may not currently have such a machine learning model (or may not have an optimized, efficient, or effective model for such purpose). In at least one embodiment, an existing machine learning model may be selected from the model registry 1324. In at least one embodiment, the model registry 1324 may include machine learning models trained to perform a variety of different inference tasks on imaging data. In at least one embodiment, the machine learning models in the model registry 1324 may have been trained on imaging data from a facility different from the facility 1302 (e.g., a remote facility). In at least one embodiment, the machine learning models may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when training on imaging data from a particular location, the training may occur at that location, or at least in a manner that protects the confidentiality of the imaging data or limits its transfer off-premise. In at least one embodiment, once a model is trained or partially trained at one location, the machine learning model may be added to model registry 1324. In at least one embodiment, the machine learning model may then be retrained or updated at any number of other facilities, and the retrained or updated model may be made available in model registry 1324. In at least one embodiment, a machine learning model may then be selected from model registry 1324, sometimes referred to as output model 1316, and used in installation system 1306 to perform one or more processing tasks for one or more applications of the installation system.

[0120] In at least one embodiment, training pipeline 1404 (FIG. 14), a scenario may include facility 1302 needing a machine learning model for use in performing one or more processing tasks for one or more applications in deployment system 1306, but facility 1302 may not currently have such a machine learning model (or may not have a model optimized, efficient, or effective for such purpose). In at least one embodiment, the machine learning model selected from model registry 1324 may not be fine-tuned or optimized for the imaging data 1308 generated at facility 1302 due to differences in the population, robustness of the training data used to train the machine learning model, variety of anomalies in the training data, and / or other issues with the training data. In at least one embodiment, AI-assisted annotation 1310 may be used to assist in generating annotations corresponding to imaging data 1308 to be used as ground truth data for retraining or updating the machine learning model. In at least one embodiment, labeled data 1312 may be used as ground truth data for training the machine learning model. In at least one embodiment, retraining or updating the machine learning model may be referred to as model training 1314. In at least one embodiment, model training 1314, e.g., AI-assisted annotations 1310, labeled clinic data 1312, or a combination thereof, may be used as ground truth data for retraining or updating the machine learning model. In at least one embodiment, the trained machine learning model may be referred to as output model 1316 and may be used by deployment system 1306 as described herein.

[0121] In at least one embodiment, deployment system 1306 may include software 1318, services 1320, hardware 1322, and / or other components, features, and functionality. In at least one embodiment, deployment system 1306 may include a software “stack” whereby software 1318 may build on and use services 1320 to perform some or all of the processing tasks, and services 1320 and software 1318 may build on and use hardware 1322 to perform the processing, storage, and / or other compute tasks of deployment system 1306. In at least one embodiment, software 1318 may include any number of different containers, each of which may perform an instantiation of an application. In at least one embodiment, each application may perform one or more processing tasks (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.) in an advanced processing and inference pipeline. In at least one embodiment, an advanced processing and inference pipeline may be defined based on a selection of different containers desired or needed to process imaging data 1308, in addition to the containers that receive and configure the imaging data for use by each container and / or by facility 1302 after processing through the pipeline (e.g., to convert the output to a usable data type). In at least one embodiment, the combination of containers in software 1318 (e.g., making up a pipeline) may be referred to as a virtual device (as described in more detail herein), which may utilize services 1320 and hardware 1322 to perform some or all processing tasks of applications instantiated in the containers.

[0122] In at least one embodiment, the data processing pipeline may receive input data (e.g., imaging data 1308) in a particular format in response to an inference request (e.g., a request from a user of deployment system 1306). In at least one embodiment, the input data may represent one or more images, videos, and / or other data representations generated by one or more imaging devices. In at least one embodiment, the data may undergo pre-processing as part of the data processing pipeline to prepare the data for processing by one or more applications. In at least one embodiment, post-processing may be performed on the output of one or more inference tasks or other processing tasks of the pipeline to prepare output data for a subsequent application and / or to prepare output data for transmission and / or use by a user (e.g., in response to an inference request). In at least one embodiment, the inference tasks may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include output model 1316 of training system 1304.

[0123] In at least one embodiment, tasks in a data processing pipeline may be encapsulated in container(s), with each container(s) representing a separate, fully functional instantiation of an application and a virtualized computing environment capable of referencing machine learning models. In at least one embodiment, containers or applications may be published to a private (e.g., restricted access) area of ​​a container registry (described in more detail herein), and trained or deployed models may be stored in a model registry 1324 and associated with one or more applications. In at least one embodiment, images of applications (e.g., container images) may be available in the container registry, and when selected from the container registry by a user for deployment in the pipeline, the images may be used to generate a container for instantiation of the application for use by the user's system.

[0124] In at least one embodiment, a developer (e.g., a software developer, clinician, physician, etc.) may develop, publish, and store (e.g., as a container) an application for performing image processing and / or inference on the provided data. In at least one embodiment, the development, publishing, and / or storage may be performed using a software development kit (SDK) associated with the system (e.g., to ensure that the developed application and / or container conforms to or is compatible with the system). In at least one embodiment, the developed application may be tested locally (e.g., at a first facility against data from the first facility) using an SDK that may support at least some of the services 1320 as a system (e.g., system 1400 of FIG. 14 ). In at least one embodiment, because a DICOM object may contain anywhere from one to hundreds of images or other data types, and because of data variability, the developer may be responsible for managing the extraction and preparation of the incoming data (e.g., setting up constructs for the application, incorporating pre-processing into the application, etc.). In at least one embodiment, once verified (e.g., for accuracy) by system 1400, the application may be made available in a container registry for selection and / or implementation by a user to perform one or more processing tasks on the data at the user's facility (e.g., a second facility).

[0125] In at least one embodiment, the developer may then share the application or container over a network for access and use by users of the system (e.g., system 1400 of FIG. 14 ). In at least one embodiment, the completed and validated application or container may be stored in a container registry, and the associated machine learning model may be stored in a model registry 1324. In at least one embodiment, a requesting entity providing an inference or image processing request may browse the container registry and / or the model registry 1324 for applications, containers, datasets, machine learning models, etc., select a desired combination of elements for inclusion in a data processing pipeline, and submit the image processing request. In at least one embodiment, the request may include input data (and, in some instances, associated patient data) required to perform the request and / or may include a selection of application(s) and / or machine learning model(s) to be executed in processing the request. In at least one embodiment, the request may then be passed to one or more components of the deployment system 1306 (e.g., the cloud) to perform the processing of the data processing pipeline. In at least one embodiment, processing by the deployment system 1306 may include referencing selected elements (e.g., applications, containers, models, etc.) from the container registry and / or the model registry 1324. In at least one embodiment, once results are produced by the pipeline, the results may be returned to the user for viewing (e.g., for viewing in a viewing application suite running on a local, on-premise workstation or terminal).

[0126] In at least one embodiment, services 1320 may be utilized to assist in the processing or execution of applications or containers in a pipeline. In at least one embodiment, services 1320 may include compute services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, services 1320 may provide functionality that is common to one or more applications in software 1318, and thus the functionality may be abstracted into services that can be called or utilized by the applications. In at least one embodiment, the functionality provided by services 1320 may operate dynamically and more efficiently and may scale well by allowing applications to process data in parallel (e.g., using parallel computing platform 1430 (FIG. 14)). In at least one embodiment, services 1320 may be shared among and between various applications, rather than each application that shares the same functionality provided by service 1320 being required to have its own instance of service 1320. In at least one embodiment, the services may include, by way of non-limiting example, an inference server or engine that may be used to perform detection or segmentation tasks. In at least one embodiment, a model training service may be included that may provide machine learning model training and / or retraining capabilities. In at least one embodiment, a data augmentation service may further be included that may provide GPU-accelerated data (e.g., DICOM, RIS, CIS, REST-compliant, RPC, raw, etc.) extraction, resizing, scaling, and / or other enhancements. In at least one embodiment, a visualization service may be used that may add image rendering effects such as ray tracing, rasterization, denoising, sharpening, etc. to add realism to two-dimensional (2D) and / or three-dimensional (3D) models.In at least one embodiment, virtual device services may be included that provide beamforming, segmentation, inference, imaging, and / or support for other applications in the virtual device pipeline.

[0127] In at least one embodiment, if services 1320 include an AI service (e.g., an inference service), the one or more machine learning models may be executed by calling (e.g., as an API call) an inference service (e.g., an inference server) to execute the machine learning model(s), or processing thereof, as part of the application execution. In at least one embodiment, if another application includes one or more machine learning models for a segmentation task, the application may call the inference service to execute the machine learning models to perform one or more of the processing operations associated with the segmentation task. In at least one embodiment, software 1318 implementing an advanced processing and inference pipeline, including a segmentation application and an anomaly detection application, may be streamlined because each application may call the same inference service to perform one or more inference tasks.

[0128] In at least one embodiment, hardware 1322 may include a GPU, a CPU, a graphics card, an AI / deep learning system (e.g., an AI supercomputer such as NVIDIA's DGX), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 1322 may be used to provide efficient and dedicated support for software 1318 and services 1320 in deployment system 1306. In at least one embodiment, the use of GPU processing for processing locally (e.g., at facility 1302), within the AI / deep learning system, in a cloud system, and / or in other processing components of deployment system 1306 may be implemented to improve the efficiency, accuracy, and effectiveness of image processing and generation. In at least one embodiment, software 1318 and / or services 1320 may be optimized for GPU processing for, by way of non-limiting example, deep learning, machine learning, and / or high-performance computing. In at least one embodiment, at least a portion of the computing environment of deployment system 1306 and / or training system 1304 may be executed on one or more supercomputers or high-performance computing systems in a data center using GPU-optimized software (e.g., a hardware and software combination of an NVIDIA DGX system). In at least one embodiment, hardware 1322 may include any number of GPUs, which may be called upon to perform data parallel processing as described herein. In at least one embodiment, the cloud platform may further include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computing tasks. In at least one embodiment, the cloud platform (e.g., NVIDIA NGC) may be executed using one or more AI / deep learning supercomputers and / or GPU-optimized software (e.g., provided on an NVIDIA DGX system) as a hardware abstraction and scaling platform.In at least one embodiment, the cloud platform may integrate an application container clustering system or orchestration system (e.g., Kubernetes) across multiple GPUs to enable seamless scaling and load balancing.

[0129] 14 is a system diagram for an exemplary system 1400 for generating and deploying an imaging deployment pipeline, according to at least one embodiment. In at least one embodiment, system 1400 may be used to implement process 1300 of FIG. 13 and / or other processes, including advanced processing and inference pipelines. In at least one embodiment, system 1400 may include a training system 1304 and a deployment system 1306. In at least one embodiment, training system 1304 and deployment system 1306 may be implemented using software 1318, services 1320, and / or hardware 1322, as described herein.

[0130] In at least one embodiment, system 1400 (e.g., training system 1304 and / or deployment system 1306) may be implemented in a cloud computing environment (e.g., using cloud 1426). In at least one embodiment, system 1400 may be implemented locally with respect to a healthcare service facility or as a combination of both cloud and local computing resources. In at least one embodiment, access to APIs in cloud 1426 may be limited to authorized users through enacted security measures or protocols. In at least one embodiment, the security protocol may include web tokens, which may be signed by an authentication (e.g., AuthN, AuthZ, Gluecon, etc.) service and may have appropriate permissions. In at least one embodiment, the APIs of a virtual appliance (as described herein) or other instantiation of system 1400 may be limited to a set of public IPs that are verified or authorized for interaction.

[0131] In at least one embodiment, the various components of system 1400 may communicate among themselves via wired and / or wireless communication protocols using any of a variety of different network types, including, but not limited to, a local area network (LAN) and / or a wide area network (WAN). In at least one embodiment, communications between the facility and the components of system 1400 (e.g., for sending inference requests, receiving results of inference requests, etc.) may be communicated via data bus(es), wireless data protocol (Wi-Fi), wired data protocol (e.g., Ethernet), etc.

[0132] In at least one embodiment, the training system 1304 may execute a training pipeline 1404 similar to that described herein with respect to FIG. 13 . In at least one embodiment, if one or more machine learning models are to be used by the deployment system 1306 in the deployment pipeline 1410, the training pipeline 1404 may be used to train or retrain one or more (e.g., pre-trained) models and / or implement one or more of the pre-trained models 1406 (e.g., without the need for retraining or updating). In at least one embodiment, the training pipeline 1404 may result in the generation of output model(s) 1316. In at least one embodiment, the training pipeline 1404 may include any number of processing steps, such as, but not limited to, imaging data (or other input data) conversion or adaptation. In at least one embodiment, different training pipelines 1404 may be used for different machine learning models used by the deployment system 1306. In at least one embodiment, a training pipeline 1404 similar to the first example described with respect to FIG. 13 may be used for the first machine learning model, a training pipeline 1404 similar to the second example described with respect to FIG. 13 may be used for the second machine learning model, and a training pipeline 1404 similar to the third example described with respect to FIG. 13 may be used for the third machine learning model. In at least one embodiment, any combination of tasks in training system 1304 may be used depending on what is required for each respective machine learning model. In at least one embodiment, one or more of the machine learning models may already be trained and ready for deployment; thus, the machine learning models may not undergo any processing by training system 1304 and may be implemented by deployment system 1306.

[0133] In at least one embodiment, the output model(s) 1316 and / or the pre-trained model(s) 1406 may include any type of machine learning model, depending on the implementation or embodiment. In at least one embodiment, and without limitation, the machine learning models used by system 1400 may include machine learning model(s) using linear regression, logistic regression, decision trees, support vector machines (SVMs), naive Bayes, k-nearest neighbor (Knn), k-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutional, recurrent, perceptron, long / short-term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid state machines, etc.), and / or other types of machine learning models.

[0134] In at least one embodiment, the training pipeline 1404 may include AI-assisted annotation, as described in more detail herein with respect to at least FIG. 15B. In at least one embodiment, the labeled data 1312 (e.g., traditional annotations) may be generated by any number of techniques. In at least one embodiment, the labels or other annotations may be generated within a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a labeling program, another type of program suitable for generating annotations or labels for ground truth, and / or may be drawn by hand in some instances. In at least one embodiment, the ground truth data can be synthetically generated (e.g., generated from a computer model or rendering), realistically generated (e.g., designed and generated from real-world data), machine-automated (e.g., using feature analysis and learning to extract features from data and then generate labels), human-annotated (e.g., a labeler or annotation expert defines label locations), and / or a combination thereof. In at least one embodiment, for each instance of imaging data 1308 (or other data type used by a machine learning model), there can be corresponding ground truth data generated by training system 1304. In at least one embodiment, AI-assisted annotation can be performed as part of the deployment pipeline 1410, either in addition to or instead of AI-assisted annotation included in the training pipeline 1404. In at least one embodiment, system 1400 may include a multi-tier platform that may include a diagnostic application (or other application type) software layer (e.g., software 1318) that may perform one or more medical imaging and diagnostic functions. In at least one embodiment, system 1400 may be communicatively coupled (e.g., via an encrypted link) to one or more institutional PACS server networks.In at least one embodiment, system 1400 may be configured to access and reference data from a PACS server to perform operations such as training machine learning models, deploying machine learning models, image processing, inference, and / or other operations.

[0135] In at least one embodiment, the software layer may be implemented as a secure, encrypted, and / or authenticated API through which applications or containers may be invoked (e.g., called) from external environment(s) (e.g., facility 1302). In at least one embodiment, the applications may then call or execute one or more services 1320 to perform compute, AI, or visualization tasks associated with the respective application, and software 1318 and / or services 1320 may leverage hardware 1322 to perform the processing tasks in an effective and efficient manner.

[0136] In at least one embodiment, the deployment system 1306 may execute an deployment pipeline 1410. In at least one embodiment, the deployment pipeline 1410 may include any number of applications, which may be applied sequentially, non-sequentially, or otherwise to imaging data (and / or other data types) generated by an imaging device, a sequencing device, a genomics device, etc., including AI-assisted annotation, as described above. In at least one embodiment, as described herein, the deployment pipeline 1410 for an individual device may be referred to as a virtual instrument for the device (e.g., a virtual ultrasound instrument, a virtual CT scan instrument, a virtual sequencing instrument, etc.). In at least one embodiment, there may be two or more deployment pipelines 1410 for a single device, depending on the information desired from the data generated by the device. In at least one embodiment, there may be a first deployment pipeline 1410 if anomaly detection is desired from an MRI machine, and a second deployment pipeline 1410 if image enhancement is desired from the output of the MRI machine.

[0137] In at least one embodiment, the image generation application may include processing tasks that involve the use of machine learning models. In at least one embodiment, a user may desire to use their own machine learning model or select a machine learning model from model registry 1324. In at least one embodiment, a user may implement their own machine learning model or select a machine learning model for inclusion in an application to perform a processing task. In at least one embodiment, applications may be selectable and customizable, and by defining the architecture of an application, the deployment and implementation of the application for a particular user may be presented as a more seamless user experience. In at least one embodiment, by leveraging other features of system 1400, such as services 1320 and hardware 1322, deployment pipeline 1410 may be even more user-friendly, provide easier integration, and produce more accurate, efficient, and timely results.

[0138] In at least one embodiment, the deployment system 1306 may include a user interface 1414 (e.g., a graphical user interface, a web interface, etc.) that may be used to select applications for inclusion in the deployment pipeline(s) 1410, deploy applications, modify or change applications or their parameters or configuration, use and interact with the deployment pipeline(s) 1410 during setup and / or deployment, and / or otherwise interact with the deployment system 1306. In at least one embodiment, although not shown with respect to the training system 1304, the user interface 1414 (or a different user interface) may be used to select models for use in the deployment system 1306, to select models for training or retraining in the training system 1304, and / or to otherwise interact with the training system 1304.

[0139] In at least one embodiment, a pipeline manager 1412 may be used in addition to an application orchestration system 1428 to manage interactions between applications or containers of deployment pipeline(s) 1410 and services 1320 and / or hardware 1322. In at least one embodiment, the pipeline manager 1412 may be configured to facilitate application-to-application interactions, application-to-service interactions, and / or application or service-to-hardware interactions. While shown in at least one embodiment as being included in software 1318, this is not intended to be limiting, and in some instances (e.g., shown in FIG. 12C), the pipeline manager 1412 may be included in services 1320. In at least one embodiment, the application orchestration system 1428 (e.g., Kubernetes, DOCKER, etc.) may include a container orchestration system that may group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications (e.g., reconstruction applications, segmentation applications, etc.) from the deployment pipeline(s) 1410 with individual containers, each application may run in a self-contained environment (e.g., at the kernel level) for improved speed and efficiency.

[0140] In at least one embodiment, each application and / or container (or image thereof) may be developed, modified, and deployed individually (e.g., a first user or developer may develop, modify, and deploy a first application, and a second user or developer may develop, modify, and deploy a second application separately from the first user or developer), allowing for focused attention to be paid to the tasks of a single application and / or container(s) without being distracted by the tasks of another application(s) or container(s). In at least one embodiment, communication and coordination between different containers or applications may be assisted by pipeline manager 1412 and application orchestration system 1428. In at least one embodiment, application orchestration system 1428 and / or pipeline manager 1412 may facilitate communication between, and sharing of resources between, each of the applications or containers, as long as the expected inputs and / or outputs of each container or application are known by the system (e.g., based on the construction of the application or container). In at least one embodiment, one or more of the applications or containers in the deployment pipeline(s) 1410 may share the same services and resources, and therefore, the application orchestration system 1428 may orchestrate, load balance, and determine sharing of services or resources among the various applications or containers. In at least one embodiment, a scheduler may be used to track the resource requirements of the applications or containers, the current or planned usage of those resources, and the availability of the resources. Thus, in at least one embodiment, the scheduler may allocate resources to different applications and distribute the resources among the applications taking into account the requirements and availability of the system.In some instances, the scheduler (and / or other components of the application orchestration system 1428) may determine resource availability and distribution based on constraints imposed on the system (e.g., user constraints), such as quality of service (QoS), the urgency of needing data output (e.g., to determine whether real-time or delayed processing should be performed), etc.

[0141] In at least one embodiment, services 1320 utilized and shared by applications or containers in deployment system 1306 may include compute services 1416, AI services 1418, visualization services 1420, and / or other service types. In at least one embodiment, an application may call (e.g., execute) one or more of services 1320 to perform processing operations for the application. In at least one embodiment, compute services 1416 may be utilized by an application to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, compute service(s) 1416 may be utilized to perform parallel processing (e.g., using parallel computing platform 1430) to process data through one or more of the applications and / or one or more tasks of a single application substantially simultaneously. In at least one embodiment, a parallel computing platform 1430 (e.g., NVIDIA's CUDA) may enable general purpose computing on GPUs (e.g., GPU 1422). In at least one embodiment, a software layer of the parallel computing platform 1430 may provide a virtual instruction set and access to the parallel computational elements of the GPU for execution of compute kernels. In at least one embodiment, the parallel computing platform 1430 may include memory, which in some embodiments may be shared among multiple containers and / or among different processing tasks within a single container.In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and / or multiple processes within a container to use the same data from a shared segment of memory in the parallel computing platform 1430 (e.g., when multiple different stages of an application or multiple applications are processing the same information). In at least one embodiment, rather than making copies of the data and moving the data to different locations in memory (e.g., read / write operations), the same data in the same location in memory may be used for any number of processing tasks (e.g., at the same time, different times, etc.). In at least one embodiment, as data is used and new data is generated as a result of processing, this information of the data's new location may be stored and shared among various applications. In at least one embodiment, the location of the data and the location of updated or modified data may be part of the definition of how the payload is understood within the container.

[0142] In at least one embodiment, AI service 1418 may be utilized to perform inference services to execute machine learning model(s) associated with an application (e.g., tasked with performing one or more processing tasks of the application). In at least one embodiment, AI service 1418 may utilize AI system 1424 to execute machine learning model(s) (e.g., neural networks such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, applications in deployment pipeline(s) 1410 may perform inference on imaging data using output model 1316 from training system 1304 and / or one or more of the application's other models. In at least one embodiment, two or more instances of inference using application orchestration system 1428 (e.g., a scheduler) may be available. In at least one embodiment, the first category may include a high priority / low latency path that may achieve a higher service level agreement, such as for performing inference on urgent requests during an emergency or for radiologists during a diagnosis. In at least one embodiment, the second category may include a standard priority path that may be used for requests that may not be urgent or where analysis can be performed at a later time. In at least one embodiment, the application orchestration system 1428 may distribute resources (e.g., services 1320 and / or hardware 1322) based on the priority path for different inference tasks of the AI ​​service 1418.

[0143] In at least one embodiment, shared storage may be attached to the AI ​​service 1418 within the system 1400. In at least one embodiment, the shared storage may act as a cache (or other storage device type) and may be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, the request may be received by a set of API instances of the installation system 1306, and one or more instances may be selected (e.g., for best fit, for load balancing, etc.) to process the request. In at least one embodiment, to process the request, the request may be entered into a database, the machine learning model may be located from the model registry 1324 if not already in the cache, and a validation step may ensure that the appropriate machine learning model is loaded into the cache (e.g., shared storage) and / or a copy of the model may be saved to the cache. In at least one embodiment, if the application is not already running or if there are not enough instances of the application, a scheduler (e.g., of the pipeline manager 1412) may be used to launch the application referenced in the request. In at least one embodiment, an inference server for executing a model may be started if it has not already been started. Any number of inference servers may be started per model. In at least one embodiment, in a pull model where inference servers are clustered, models may be cached whenever load balancing is advantageous. In at least one embodiment, inference servers may be statically loaded onto corresponding distributed servers.

[0144] In at least one embodiment, inference may be performed using an inference server running in a container. In at least one embodiment, an instance of an inference server may be associated with a model (optionally, multiple versions of the model). In at least one embodiment, when a request to perform inference on a model is received, if an instance of the inference server does not exist, a new instance may be loaded. In at least one embodiment, when the inference server is started, the model may be passed to the inference server, such that the same container may be used to serve different models as long as the inference servers are running as different instances.

[0145] In at least one embodiment, during application execution, an inference request for a given application may be received, a container (e.g., hosting an instance of an inference server) may be loaded (if not already loaded), and a start procedure may be called. In at least one embodiment, preprocessing logic in the container may load, decode, and / or perform any additional preprocessing on the incoming data (e.g., using CPU(s) and / or GPU(s)). In at least one embodiment, once the data is prepared for inference, the container may perform inference on the data as needed. In at least one embodiment, this may involve a single inference call for one image (e.g., a hand X-ray) or may require inference on hundreds of images (e.g., a chest CT). In at least one embodiment, the application may summarize results before completion, which may include, but is not limited to, generating a single confidence score, pixel-level segmentation, voxel-level segmentation, visualization, or generating text to summarize findings. In at least one embodiment, different models or applications may be assigned different priorities. For example, some models may have real-time (TAT<1 minute) priority, while other models may have low priority (e.g., TAT<10 minutes). In at least one embodiment, model execution time may be measured from the requesting institution or entity and may include partner network traversal time, as well as execution against the inference service.

[0146] In at least one embodiment, the transfer of requests between the service 1320 and the inference application may be hidden behind a software development kit (SDK), and robust transport may be provided through queues. In at least one embodiment, requests are queued via an API for each application / tenant ID combination, and the SDK pulls the requests from the queue and provides them to the application. In at least one embodiment, the name of the queue may be provided in the environment where the SDK picks up the request. In at least one embodiment, asynchronous communication through queues may be useful because it may allow any instance of the application to pick up work when it becomes available. Results may be sent back through queues to ensure that data is not lost. In at least one embodiment, queues may also provide the ability to segment work, as the highest priority work may go to a queue with most instances of the application attached to it, while the lowest priority work may go to a queue with a single instance attached to it, processing tasks in the order they were received. In at least one embodiment, the application may run on a GPU-accelerated instance created in the cloud 1426, and the inference service may perform inference on the GPU.

[0147] In at least one embodiment, a visualization service 1420 may be utilized to generate visualizations for viewing the output of the application and / or the deployment pipeline(s) 1410. In at least one embodiment, a GPU 1422 may be utilized by the visualization service 1420 to generate the visualization. In at least one embodiment, rendering effects such as ray tracing may be implemented by the visualization service 1420 to generate higher quality visualizations. In at least one embodiment, the visualization may include, but is not limited to, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomography slices, virtual reality displays, augmented reality displays, etc. In at least one embodiment, a virtualized environment may be used to generate a virtual interactive display or environment (e.g., a virtual environment) for interaction by a user of the system (e.g., a doctor, nurse, radiologist, etc.). In at least one embodiment, visualization services 1420 may include internal visualizer, cinematics, and / or other rendering or image processing capabilities or functionality (e.g., ray tracing, rasterization, internal optics, etc.).

[0148] In at least one embodiment, hardware 1322 may include GPU 1422, AI system 1424, cloud 1426, and / or any other hardware used to run training system 1304 and / or deployment system 1306. In at least one embodiment, GPU 1422 (e.g., NVIDIA TESLA and / or QUADRO GPUs) may include any number of GPUs that may be used to perform processing tasks of compute services 1416, AI services 1418, visualization services 1420, other services, and / or any of the features or functionality of software 1318. For example, with respect to AI services 1418, GPU 1422 may be used to perform pre-processing on imaging data (or other data types used by machine learning models), post-processing on the output of machine learning models, and / or to perform inference (e.g., to run machine learning models). In at least one embodiment, cloud 1426, AI system 1424, and / or other components of system 1400 may use GPUs 1422. In at least one embodiment, cloud 1426 may include a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI system 1424 may use GPUs, and cloud 1426, or at least a portion tasked with deep learning or inference, may be executed using one or more AI systems 1424. Thus, while hardware 1322 is shown as separate components, this is not intended to be limiting, and any component of hardware 1322 may be combined with and utilized by any other component of hardware 1322.

[0149] In at least one embodiment, AI system 1424 may include a specialized computing system (e.g., a supercomputer or HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, AI system 1424 (e.g., NVIDIA's DGX) may include GPU optimization software (e.g., a software stack) that may be executed using multiple GPUs 1422 in addition to CPUs, RAM, storage, and / or other components, features, or functionality. In at least one embodiment, one or more AI systems 1424 may be implemented in cloud 1426 (e.g., in a data center) to perform some or all of the AI-based processing tasks of system 1400.

[0150] In at least one embodiment, cloud 1426 may include a GPU-accelerated infrastructure (e.g., NVIDIA's NGC), which may provide a GPU-optimized platform for executing processing tasks of system 1400. In at least one embodiment, cloud 1426 may include AI system(s) 1424 (e.g., as a hardware abstraction and scaling platform) for performing one or more of the AI-based tasks of system 1400. In at least one embodiment, cloud 1426 may leverage multiple GPUs and integrate with application orchestration system 1428 to enable seamless scaling and load balancing between applications and services 1320. In at least one embodiment, cloud 1426 may be tasked with running at least some of the services 1320 of system 1400, including compute services 1416, AI services 1418, and / or visualization services 1420, as described herein. In at least one embodiment, cloud 1426 may perform large or small batch inference (e.g., running NVIDIA's TENSOR RT), provide an accelerated parallel computing API and platform 1430 (e.g., NVIDIA's CUDA), run an application orchestration system 1428 (e.g., KUBERNETES), provide a graphics rendering API and platform (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher quality cinematics), and / or provide other functionality for system 1400.

[0151] 15A shows a data flow diagram for a process 1500 for training, retraining, or updating a machine learning model, according to at least one embodiment. In at least one embodiment, process 1500 may be performed using system 1400 of FIG. 14 as a non-limiting example. In at least one embodiment, process 1500 may leverage services 1320 and / or hardware 1322 of system 1400, as described herein. In at least one embodiment, improved model 1512 generated by process 1500 may be executed by deployment system 1306 for one or more containerized applications in deployment pipeline 1410.

[0152] In at least one embodiment, model training 1314 may include retraining or updating the initial model 1504 (e.g., a pre-trained model) using new training data (e.g., new input data, such as the customer dataset 1506 and / or new ground truth data associated with the input data). In at least one embodiment, to retrain or update the initial model 1504, output or loss layer(s) of the initial model 1504 may be reset or removed and / or replaced with updated or new output or loss layer(s). In at least one embodiment, the initial model 1504 may have previously fine-tuned parameters (e.g., weights and / or biases) remaining from previous training; therefore, training or retraining 1314 may not take as long or require as much processing as training a model from scratch. In at least one embodiment, by having the output or loss layer(s) of the initial model 1504 reset or replaced during model training 1314, parameters can be updated or retuned for the new data set based on a loss calculation associated with the accuracy of the output or loss layer(s) in generating predictions for the new customer data set 1506 (e.g., image data 1308 in FIG. 13 ).

[0153] In at least one embodiment, the pre-trained model 1406 may be stored in a data store or registry (e.g., model registry 1324 of FIG. 13 ). In at least one embodiment, the pre-trained model 1406 may have been trained, at least in part, at one or more facilities other than the facility performing process 1500. In at least one embodiment, to protect the privacy and rights of patients, subjects, or customers of different facilities, the pre-trained model 1406 may have been trained on-premise using customer or patient data generated on-premise. In at least one embodiment, the pre-trained model 1406 may be trained using the cloud 1426 and / or other hardware 1322, but privacy-protected sensitive patient data may not be transferred to, used by, or accessible to any components of the cloud 1426 (or other off-premise hardware). In at least one embodiment, if pre-trained model 1406 is trained using patient data from more than one facility, pre-trained model 1406 may be trained individually for each facility and then trained on patient or customer data from another facility. In at least one embodiment, customer or patient data from any number of facilities may be used to train pre-trained model 1406 on-premise and / or off-premise, such as in a data center or other cloud computing infrastructure, if the customer or patient data is free from privacy concerns (e.g., via a waiver, for experimental use, etc.) or if the customer or patient data is included in a public data set.

[0154] In at least one embodiment, when selecting an application for use in the deployment pipeline 1410, a user may also select a machine learning model to be used for the particular application. In at least one embodiment, a user may not have a model for use, and therefore the user may select a pre-trained model 1406 for use with the application. In at least one embodiment, the pre-trained model 1406 may not be optimized to produce accurate results for the user's facility's customer dataset 1506 (e.g., based on patient diversity, demographics, type of medical imaging device used, etc.). In at least one embodiment, before deploying the pre-trained model 1406 in the deployment pipeline 1410 for use with application(s), the pre-trained model 1406 may be updated, retrained, and / or fine-tuned for use at the respective facility.

[0155] In at least one embodiment, a user may select a pre-trained model 1406 to be updated, retrained, and / or fine-tuned; the pre-trained model 1406 may be referred to as an initial model 1504 for the training system 1304 in process 1500. In at least one embodiment, a customer dataset 1506 (e.g., imaging data, genomics data, sequencing data, or other data types generated by devices at the facility) may be used to perform model training 1314 (which may include, but is not limited to, transfer learning) on ​​the initial model 1504 to generate an improved model 1512. In at least one embodiment, ground truth data corresponding to the customer dataset 1506 may be generated by the training system 1304. In at least one embodiment, the ground truth data may be generated, at least in part, by clinicians, scientists, physicians, or practitioners at the facility (e.g., as labeled clinic data 1312 of FIG. 13 ).

[0156] In at least one embodiment, AI-assisted annotation 1310 may be used in some instances to generate ground truth data. In at least one embodiment, AI-assisted annotation 1310 (e.g., implemented using an AI-assisted annotation SDK) may utilize machine learning models (e.g., neural networks) to generate suggested or predicted ground truth data for a customer dataset. In at least one embodiment, user 1510 may use annotation tools within a user interface (graphical user interface (GUI)) on computing device 1508.

[0157] In at least one embodiment, a user 1510 may interact with a GUI via a computing device 1508 to edit or fine-tune the (automatic) annotation. In at least one embodiment, a polygon editing feature may be used to move vertices of a polygon to more precise or fine-tuned locations.

[0158] In at least one embodiment, once the customer dataset 1506 has associated ground truth data, the ground truth data (e.g., from AI-assisted annotation, manual labeling, etc.) may be used during model training 1314 to generate an improved model 1512. In at least one embodiment, the customer dataset 1506 may be applied to the initial model 1504 any number of times, and the ground truth data may be used to update parameters of the initial model 1504 until an acceptable level of accuracy is achieved for the improved model 1512. In at least one embodiment, once the improved model 1512 is generated, the improved model 1512 may be deployed in one or more deployment pipelines 1410 at a facility to perform one or more processing tasks on the medical imaging data.

[0159] In at least one embodiment, the improved model 1512 may be uploaded to the pre-trained models 1406 in the model registry 1324 to be selected by another facility. In at least one embodiment, this process may be completed at any number of facilities, such that the improved model 1512 may be further improved any number of times on new datasets to produce a more generalized model.

[0160] FIG. 15B is an example diagram of a client-server architecture 1532 for extending an annotation tool with a pre-trained annotation model, according to at least one embodiment. In at least one embodiment, an AI-assisted annotation tool 1536 may be instantiated based on the client-server architecture 1532. In at least one embodiment, the annotation tool 1536 in an imaging application may assist a radiologist, for example, in identifying organs and abnormalities. In at least one embodiment, the imaging application may include, by way of non-limiting example, a software tool that helps a user 1510 identify a few extrema points on a particular organ of interest in a raw image 1534 (e.g., in a 3D MRI or CT scan) and receive automatically annotated results for all 2D slices of the particular organ. In at least one embodiment, the results may be stored in a data store as training data 1538 and used (for example, without limitation) as ground truth data for training. In at least one embodiment, when computing device 1508 sends extremum points for AI-assisted annotation 1310, for example, a deep learning model may receive this data as input and return inferences of segmented organs or anomalies. In at least one embodiment, a pre-instantiated annotation tool, such as AI-assisted annotation tool 1536B in FIG. 15B , may be extended by making API calls (e.g., API call 1544) to a server, such as annotation-assisted server 1540, which may include a set of pre-trained models 1542 stored in an annotation model registry. In at least one embodiment, annotation model registry may store pre-trained models 1542 (e.g., machine learning models such as deep learning models) that have been pre-trained to perform AI-assisted annotation for specific organs or anomalies. These models may be further updated using training pipeline 1404.In at least one embodiment, the pre-installed annotation tools may be improved over time as new labeled clinic data 1312 is added.

[0161] Such components may be used to combine the constituent images into a single representation using parameters determined from one or more quality assessment values.

[0162] Automated Technology FIG. 16A is a block diagram illustrating an example system architecture for the autonomous vehicle 1600 of FIG. 16A , according to at least one embodiment. In at least one embodiment, each of the components, features, and systems of the vehicle 1600 in FIG. 16A is shown as connected via a bus 1602. In at least one embodiment, the bus 1602 may include, but is not limited to, a CAN data interface (alternatively referred to herein as a “CAN bus”). In at least one embodiment, the CAN bus may be a network internal to the vehicle 1600 used to help control various features and functionality of the vehicle 1600, such as brake application, acceleration, brake control, steering, windshield wipers, etc. In at least one embodiment, the bus 1602 may be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). In at least one embodiment, bus 1602 can be read to determine steering angle, ground speed, engine revolutions per minute ("RPM"), button position, and / or other vehicle status indicators. In at least one embodiment, bus 1602 can be an ASIL B compliant CAN bus.

[0163] In at least one embodiment, FlexRay and / or Ethernet may be used in addition to or as an alternative to CAN. In at least one embodiment, there may be any number of buses 1602, including, but not limited to, zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and / or zero or more other types of buses using different protocols. In at least one embodiment, two or more buses 1602 may be used to perform different functions and / or for redundancy. For example, a first bus 1602 may be used for collision avoidance functionality, and a second bus 1602 may be used for actuation control. In at least one embodiment, each bus 1602 may communicate with any of the components of the vehicle 1600, and two or more buses 1602 may communicate with the same component. In at least one embodiment, each of any number of system-on-chip (“SoC”) 1604, each of the controller(s) 1636, and / or each computer in the vehicle may have access to the same input data (e.g., input from sensors in the vehicle 1600) and may be connected to a common bus, such as a CAN bus.

[0164] In at least one embodiment, vehicle 1600 may include one or more controllers 1636, such as those described herein with respect to FIG. 1A. Controller(s) 1636 may be used for a variety of functions. In at least one embodiment, controller(s) 1636 may be coupled to any of various other components and systems of vehicle 1600 and may be used for control of vehicle 1600, artificial intelligence of vehicle 1600, infotainment for vehicle 1600, etc.

[0165] In at least one embodiment, vehicle 1600 may include any number of SoCs 1604. Each of SoCs 1604 may include, without limitation, a central processing unit ("CPU") 1606, a graphics processing unit ("GPU") 1608, processor(s) 1610, cache(s) 1612, accelerator(s) 1614, data store(s) 1616, and / or other components and features not shown. In at least one embodiment, SoC(s) 1604 may be used to control vehicle 1600 in a variety of platforms and systems. For example, in at least one embodiment, the SoC(s) 1604 may be combined in a system (e.g., a system in a vehicle 1600) with a high-definition (“HD”) map 1622 that may obtain map refreshes and / or updates via a network interface 1624 from one or more servers (not shown in FIG. 16A).

[0166] In at least one embodiment, the CPU(s) 1606 may include a CPU cluster or CPU complex (alternatively referred to herein as a “CCPLEX”). In at least one embodiment, the CPU(s) 1606 may include multiple cores and / or level 2 (“L2”) caches. For example, in at least one embodiment, the CPU(s) 1606 may include eight cores in a coherent multiprocessor configuration. In at least one embodiment, the CPU(s) 1606 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., 2 MB of L2 cache). In at least one embodiment, the CPU(s) 1606 (e.g., a CCPLEX) may be configured to support simultaneous cluster operation, allowing any combination of clusters of the CPU(s) 1606 to be active at any given time.

[0167] In at least one embodiment, one or more of the CPU(s) 1606 may implement power management capabilities including, but not limited to, one or more of the following features: individual hardware blocks may be automatically clock gated when idle to conserve dynamic power; each core clock may be gated when the core is not actively executing instructions by executing a Wait for Interrupt ("WFI") / Wait for Event ("WFE") instruction; each core may be independently power gated; each core cluster may be independently clock gated when all cores are clock gated or power gated; and / or each core cluster may be independently power gated when all cores are power gated. In at least one embodiment, the CPU(s) 1606 may further implement an advanced algorithm for managing power states, where allowed power states and expected wake-up times are specified and the hardware / microcode determines the best power state to enter for the core, cluster, and CCPLEX. In at least one embodiment, the processing core may support a simple power state entry sequence in software with work offloaded to microcode.

[0168] In at least one embodiment, the GPU(s) 1608 may include an integrated GPU (alternatively referred to herein as an “iGPU”). In at least one embodiment, the GPU(s) 1608 may be programmable and efficient for parallel workloads. In at least one embodiment, the GPU(s) 1608 may use an extended tensor instruction set. In at least one embodiment, the GPU(s) 1608 may include one or more streaming microprocessors, each of which may include a level 1 (“L1”) cache (e.g., an L1 cache with at least 96 KB of storage capacity), and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512 KB of storage capacity). In at least one embodiment, the GPU(s) 1608 may include at least eight streaming microprocessors. In at least one embodiment, the GPU(s) 1608 may use one or more compute application programming interfaces (APIs). In at least one embodiment, the GPU(s) 1608 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0169] In at least one embodiment, one or more of the GPU(s) 1608 may be power-optimized for best performance in automotive and embedded use cases. For example, in one embodiment, the GPU(s) 1608 may be fabricated on Fin field-effect transistors ("FinFETs"). In at least one embodiment, each streaming microprocessor may incorporate several mixed-precision processing cores partitioned into multiple blocks. For example, without limitation, 64 PF32 cores and 32 PF64 cores may be partitioned into four processing blocks. In at least one embodiment, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for deep learning matrix arithmetic, a level 0 ("L0") instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In at least one embodiment, the streaming microprocessor may include independent parallel integer and floating-point data paths for efficient execution of workloads with a mix of computation and addressing calculations. In at least one embodiment, the streaming microprocessor may include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation among parallel threads. In at least one embodiment, the streaming microprocessor may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0170] In at least one embodiment, one or more of the GPU(s) 1608 may include high bandwidth memory (“HBM”) and / or a 16 GB HBM2 memory subsystem to provide, in some instances, approximately 900 GB / s of peak memory bandwidth. In at least one embodiment, synchronous graphics random-access memory (“SGRAM”), such as graphics double data rate type five synchronous random-access memory (“GDDR5”), may be used in addition to or as an alternative to the HBM memory.

[0171] In at least one embodiment, the GPU(s) 1608 may include unified memory technology. In at least one embodiment, address translation service ("ATS") support may be used to allow the GPU(s) 1608 to directly access the page tables of the CPU(s) 1606. In at least one embodiment, when the memory management unit ("MMU") of the GPU(s) 1608 encounters a miss, an address translation request may be sent to the CPU(s) 1606. In at least one embodiment, in response, the CPU(s) 1606 may look up a virtual-to-physical mapping for the address in its page table and send the translation back to the GPU(s) 1608. In at least one embodiment, the unified memory technology enables a single unified virtual address space for memory of both the CPU(s) 1606 and the GPU(s) 1608, which may simplify programming the GPU(s) 1608 and porting applications to the GPU(s) 1608.

[0172] In at least one embodiment, the GPU(s) 1608 may include any number of access counters that may track the frequency of the GPU(s) 1608's accesses to the memory of other processors. In at least one embodiment, the access counter(s) may help ensure that memory pages are moved to the physical memory of the processor that is accessing the pages most frequently, thereby improving the efficiency of memory ranges shared between processors.

[0173] In at least one embodiment, one or more of the SoC(s) 1604 may include any number of caches 1612, including those described herein. For example, in at least one embodiment, the cache(s) 1612 may include a level 3 (“L3”) cache that is available to both the CPU(s) 1606 and the GPU(s) 1608 (e.g., connected to both the CPU(s) 1606 and the GPU(s) 1608). In at least one embodiment, the cache(s) 1612 may include a write-back cache that may track line states, such as by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). In at least one embodiment, the L3 cache may include 4 MB or more, depending on the embodiment, although smaller cache sizes may be used.

[0174] In at least one embodiment, one or more of the SoC(s) 1604 may include one or more accelerators 1614 (e.g., hardware accelerators, software accelerators, or a combination thereof). In at least one embodiment, the SoC(s) 1604 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. In at least one embodiment, large on-chip memory (e.g., 4 MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other calculations. In at least one embodiment, the hardware acceleration cluster may complement the GPU(s) 1608 and be used to offload some of the GPU(s) 1608's tasks (e.g., to free up more cycles of the GPU(s) 1608 to perform other tasks). In at least one embodiment, accelerator(s) 1614 may be used for workloads of interest that are stable enough to accommodate acceleration (e.g., cognition, convolutional neural networks (“CNNs”), recurrent neural networks (“RNNs”), etc.). In at least one embodiment, the CNNs may include region-based, i.e., regional convolutional neural networks (“RCNNs”), and Fast RCNNs (e.g., as used for object detection), or other types of CNNs.

[0175] In at least one embodiment, the accelerator(s) 1614 (e.g., a hardware acceleration cluster) may include deep learning accelerator(s) (“DLA”). The DLA(s) may include, without limitation, one or more tensor processing units (“TPU”), which may be configured to provide an additional tens of trillion operations per second for deep learning applications and inference. In at least one embodiment, the TPU(s) may be accelerators configured and optimized for performing image processing functions (e.g., for CNN, RCNN, etc.). The DLA(s) may be further optimized for a specific set of neural network types and floating-point operations, as well as for inference. In at least one embodiment, the design of the DLA(s) may provide more performance per millimeter than a typical general-purpose GPU, generally far exceeding the performance of a CPU. In at least one embodiment, the TPU(s) may perform several functions, including, for example, single-instance convolution functions, supporting INT8, INT16, and FP16 data types for both features and weights, and post-processor functions. In at least one embodiment, the DLA(s) may quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including, for example, but not limited to, CNNs for object identification and detection using data from a camera sensor, CNNs for distance estimation using data from a camera sensor, CNNs for emergency vehicle detection and identification using data from microphone 1696, CNNs for face recognition and vehicle owner identification using data from a camera sensor, and / or CNNs for security and / or safety-related events.

[0176] In at least one embodiment, the DLA(s) may perform any function of the GPU(s) 1608; for example, by using an inference accelerator, a designer may target either the DLA(s) or the GPU(s) 1608 for any function. For example, in at least one embodiment, a designer may centralize the processing of CNNs and floating-point operations in the DLA(s) and offload other functions to the GPU(s) 1608 and / or other accelerator(s) 1614.

[0177] In at least one embodiment, the accelerator(s) 1614 (e.g., a hardware acceleration cluster) may include programmable vision accelerator(s), which may alternatively be referred to herein as computer vision accelerators. In at least one embodiment, the PVA(s) may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems ("ADAS") 1638, autonomous driving, augmented reality ("AR") applications, and / or virtual reality ("VR") applications. The PVA(s) may provide a balance between performance and flexibility. For example, in at least one embodiment, each PVA(s) may include, for example, without limitation, any number of reduced instruction set computer ("RISC") cores, direct memory access ("DMA") processors, and / or any number of vector processors.

[0178] In at least one embodiment, the RISC core may interact with an image sensor (e.g., an image sensor of any of the cameras described herein), image signal processor(s), etc. In at least one embodiment, each of the RISC cores may include any amount of memory. In at least one embodiment, the RISC cores may use any of several protocols, depending on the embodiment. In at least one embodiment, the RISC cores may execute a real-time operating system (“RTOS”). In at least one embodiment, the RISC cores may be implemented using one or more integrated circuit devices, application specific integrated circuits (“ASICs”), and / or memory devices. For example, in at least one embodiment, the RISC cores may include an instruction cache and / or tightly coupled RAM.

[0179] In at least one embodiment, the DMA may enable components of the PVA(s) to access system memory independently of the CPU(s) 1606. In at least one embodiment, the DMA may support any number of features used to provide optimizations to the PVA, including, but not limited to, supporting multi-dimensional addressing and / or circular addressing. In at least one embodiment, the DMA may support up to six or more addressing dimensions, which may include, but are not limited to, block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0180] In at least one embodiment, the vector processor is a programmable processor that can be designed to efficiently and flexibly execute programming for computer vision algorithms and can provide signal processing capabilities. In at least one embodiment, the PVA can include a PVA core and two vector processing subsystem partitions. In at least one embodiment, the PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. In at least one embodiment, the vector processing subsystem can act as the PVA's primary processing engine and can include a vector processing unit ("VPU"), an instruction cache, and / or vector memory (e.g., "VMEM"). In at least one embodiment, the VPU can include a digital signal processor, such as a single instruction, multiple data ("SIMD"), very long instruction word ("VLIW") digital signal processor. In at least one embodiment, the combination of SIMD and VLIW can improve throughput and speed.

[0181] In at least one embodiment, each of the vector processors may include an instruction cache and may be coupled to dedicated memory. As a result, in at least one embodiment, each of the vector processors may be configured to execute independently of the other vector processors. In at least one embodiment, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in at least one embodiment, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In at least one embodiment, the vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on consecutive images or portions of an image. In at least one embodiment, among other things, any number of PVAs may be included in a hardware-accelerated cluster, and any number of vector processors may be included within each PVA. In at least one embodiment, the PVA(s) may include additional error correction code ("ECC") memory to improve the overall security of the system.

[0182] In at least one embodiment, the accelerator(s) 1614 (e.g., a hardware acceleration cluster) may include a computer vision network-on-chip and static random access memory (“SRAM”) to provide high-bandwidth, low-latency SRAM for the accelerator(s) 1614. In at least one embodiment, the on-chip memory may include, for example, but not limited to, at least 4 MB of SRAM consisting of eight field-configurable memory blocks, which may be accessible by both the PVA and DLA. In at least one embodiment, each pair of memory blocks may include an advanced peripheral bus (“APB”) interface, configuration circuitry, a controller, and a multiplexer. In at least one embodiment, any type of memory may be used. In at least one embodiment, the PVA and DLA may access memory through a backbone that provides the PVA and DLA with high-speed access to memory. In at least one embodiment, the backbone may include a computer vision network-on-chip that interconnects the PVA and DLA to memory (e.g., using APBs).

[0183] In at least one embodiment, the computer vision network-on-chip may include an interface that determines whether both the PVA and DLA provide ready and enable signals before transmitting any control signals / addresses / data. In at least one embodiment, the interface may provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. In at least one embodiment, the interface may conform to International Organization for Standardization (“ISO”) 26262 or International Electrotechnical Commission (“IEC”) 61508 standards, although other standards and protocols may be used.

[0184] In at least one embodiment, one or more of the SoC1604(s) may include a real-time ray tracing hardware accelerator that may be used to quickly and efficiently determine the location and range of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general waveform propagation simulation, for comparison with LIDAR data for localization and / or other functions, and / or other uses.

[0185] In at least one embodiment, accelerator(s) 1614 (e.g., a hardware accelerator cluster) have diverse uses for autonomous driving. In at least one embodiment, the PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. In at least one embodiment, the capabilities of the PVA are well-matched for algorithm domains that require predictable processing with low power and low latency. In other words, the PVA performs well on small data sets for semi-dense or dense regular calculations that require predictable runtime with low latency and low power. In at least one embodiment, an autonomous vehicle, such as vehicle 1600, the PVA is designed to run traditional computer vision algorithms because they are efficient at object detection and integer arithmetic.

[0186] For example, according to at least one embodiment of the technology, the PVA is used to perform computer stereo vision. In at least one embodiment, semi-global matching-based algorithms may be used in some instances, but this is not intended to be limiting. In at least one embodiment, applications for Level 3-5 autonomous driving use motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.) on the fly. In at least one embodiment, the PVA may perform computer stereo vision functions on input from two monocular cameras.

[0187] In at least one embodiment, the PVA may be used to perform dense optical flow. For example, in at least one embodiment, the PVA may process raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR data. In at least one embodiment, the PVA may be used for time-of-flight depth processing, for example, by processing the raw time-of-flight data to provide processed time-of-flight data.

[0188] In at least one embodiment, DLA may be used to run any type of network for improving control and driving safety, including, for example, but not limited to, a neural network that outputs a measure of confidence for each object detection. In at least one embodiment, confidence may be expressed or interpreted as the probability of each detection compared to other detections or as providing its relative “weight.” In at least one embodiment, confidence allows the system to make further decisions regarding which detections should be considered true positives rather than false positives. For example, in at least one embodiment, the system may set a confidence threshold and consider only detections above the threshold to be true positives. In embodiments where an automatic emergency braking (“AEB”) system is used, a false positive detection may cause the vehicle to automatically apply emergency braking, which is clearly undesirable. In at least one embodiment, a highly confident detection may be considered a trigger for AEB. In at least one embodiment, DLA may run a neural network to regress a confidence value. In at least one embodiment, the neural network may take as its input at least some subset of parameters, such as bounding box dimensions, a ground plane estimate obtained (e.g., from another subsystem), an output from IMU sensor(s) 1666 that correlates with the orientation of the vehicle 1600, distance, and a 3D location estimate of the object obtained from the neural network and / or other sensors (e.g., LIDAR sensor(s) 1664 or RADAR sensor(s) 1660), among others.

[0189] In at least one embodiment, one or more of the SoC(s) 1604 may include data store(s) 1616 (e.g., memory). In at least one embodiment, the data store(s) 1616 may be on-chip memory of the SoC(s) 1604, which may store neural networks to be executed on the GPU(s) 1608 and / or DLA. In at least one embodiment, the data store(s) 1616 may be large enough in capacity to store multiple instances of a neural network for redundancy and safety. In at least one embodiment, the data store(s) 1616 may comprise L2 or L3 cache(s).

[0190] In at least one embodiment, one or more of the SoC(s) 1604 may include any number of processors 1610 (e.g., embedded processors). In at least one embodiment, the processor(s) 1610 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions and related security enforcement. In at least one embodiment, the boot and power management processor may be part of the boot sequence of the SoC(s) 1604 and may provide runtime power management services. In at least one embodiment, the boot power and management processor may provide clock and voltage programming, assist with system low power state transitions, manage thermal and temperature sensors of the SoC(s) 1604, and / or manage the power state of the SoC(s) 1604. In at least one embodiment, each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC(s) 1604 may use the ring oscillator to detect the temperature of the CPU(s) 1606, the GPU(s) 1608, and / or the accelerator(s) 1614. In at least one embodiment, if the temperature is determined to exceed a threshold, the boot and power management processor may enter a temperature fault routine, place the SoC(s) 1604 in a low power state, and / or place the vehicle 1600 in a chauffeur to safe stop mode (e.g., bring the vehicle 1600 to a safe stop).

[0191] In at least one embodiment, processor(s) 1610 may further include a set of embedded processors that may act as an audio processing engine. In at least one embodiment, the audio processing engine may be an audio subsystem that enables full hardware support for multi-channel audio over multiple interfaces and a wide variety of flexible audio I / O interfaces. In at least one embodiment, the audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

[0192] In at least one embodiment, the processor(s) 1610 may further include an always-on processor engine that may provide the hardware features necessary to support low-power sensor management and wake-up use cases. In at least one embodiment, the always-on processor engine may include, but is not limited to, a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0193] In at least one embodiment, the processor(s) 1610 may further include a safety cluster engine, which may include, but is not limited to, a dedicated processor subsystem for handling safety management for automotive applications. In at least one embodiment, the safety cluster engine may include, but is not limited to, two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores, in at least one embodiment, may operate in lockstep mode and function as a single core with comparison logic to detect any differences between their operations. In at least one embodiment, the processor(s) 1610 may further include a real-time camera engine, which may include, but is not limited to, a dedicated processor subsystem for handling real-time camera management. In at least one embodiment, the processor(s) 1610 may further include a high dynamic range signal processor, which may include, but is not limited to, an image signal processor, which is a hardware engine that is part of a camera processing pipeline.

[0194] In at least one embodiment, the processor(s) 1610 may include a video image composer, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to produce a final image for a player window. In at least one embodiment, the video image composer may perform lens distortion correction for the wide-angle camera(s) 1670, the surrounding camera(s) 1674, and / or the in-cabin surveillance camera sensor(s). In at least one embodiment, the in-cabin surveillance camera sensor(s) is / are preferably monitored by a neural network running on a separate instance of the SoC(s) 1604 configured to identify events within the cabin and respond accordingly. In at least one embodiment, the in-cabin system may perform lip reading to, but not limited to, activate cellular service, make phone calls, write emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. In at least one embodiment, some functions are available to the driver when the vehicle is operating in autonomous mode and are disabled at other times.

[0195] In at least one embodiment, the video image combiner may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, in at least one embodiment, when motion occurs in the video, noise reduction appropriately weights spatial information and reduces the weight of information provided by adjacent frames. In at least one embodiment, when an image or portion of an image does not contain motion, the temporal noise reduction performed by the video image combiner may use information from previous images to reduce noise in the current image.

[0196] In at least one embodiment, the video image composer may also be configured to perform stereo rectification on the input stereo lens frames. In at least one embodiment, the video image composer may further be used for user interface compositing when the operating system desktop is in use, so that the GPU(s) 1608 are not required to continually render new surfaces. In at least one embodiment, when the GPU(s) 1608 are powered on, active, and performing 3D rendering, the video image composer may be used to offload the GPU(s) 1608 to improve performance and responsiveness.

[0197] In at least one embodiment, one or more of the SoC1604(s) may further include a mobile industry processor interface ("MIPI") camera serial interface, a high-speed interface for receiving video and input from a camera, and / or a video input block that may be used for camera and related pixel input functions. In at least one embodiment, one or more of the SoC1604(s) may further include input / output controller(s), which may be controlled by software and may be used to receive I / O signals that are not committed to a specific role.

[0198] In at least one embodiment, one or more of the SoC(s) 1604 may further include peripherals, audio encoders / decoders ("codecs"), power management, and / or a wide range of peripheral interfaces to enable communication with other devices. The SoC(s) 1604 may be used to process data from cameras (e.g., connected via gigabit multimedia serial links and Ethernet), data from sensors (e.g., LIDAR sensor(s) 1664, RADAR sensor(s) 1660, etc., which may be connected via Ethernet), data from bus 1602 (e.g., vehicle 1600 speed, steering wheel position, etc.), data from GNSS sensor(s) 1658 (e.g., connected via Ethernet or CAN bus), etc. In at least one embodiment, one or more of the SoC(s) 1604 may further include dedicated high performance mass storage controllers, which may include their own DMA engines and may be used to offload the CPU(s) 1606 from routine data management tasks.

[0199] In at least one embodiment, the SoC(s) 1604 may be an end-to-end platform with a flexible architecture spanning levels 3-5 of automation, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and providing a platform for a flexible and reliable driving software stack, along with deep learning tools. In at least one embodiment, the SoC(s) 1604 may be faster, more reliable, and more energy- and space-efficient than conventional systems. For example, in at least one embodiment, the accelerator(s) 1614, when combined with the CPU(s) 1606, GPU(s) 1608, and data store(s) 1616, may provide a fast and efficient platform for a level 3-5 autonomous vehicle.

[0200] In at least one embodiment, computer vision algorithms may be executed on a CPU, which may be configured using a high-level programming language, such as the C programming language, to perform a wide variety of processing algorithms across a wide variety of visual data. However, in at least one embodiment, CPUs often cannot meet the performance requirements of many computer vision applications, including requirements related to execution time and power consumption. In at least one embodiment, many CPUs are unable to execute complex object detection algorithms in real time, as used in in-vehicle ADAS applications and practical Level 3-5 autonomous vehicles.

[0201] The embodiments described herein allow multiple neural networks to be implemented simultaneously and / or sequentially, with the results being combined together to enable Levels 3-5 autonomous driving functionality. For example, in at least one embodiment, a CNN running on the DLA or a separate GPU (e.g., GPU(s) 1620) can include text and word recognition, enabling the supercomputer to read and understand traffic signs, including signs for which the neural network was not specifically trained. In at least one embodiment, the DLA can further include a neural network that can identify and interpret signs and provide a semantic understanding of the signs, which can then be passed to a route planning module running on the CPU complex.

[0202] In at least one embodiment, multiple neural networks may be run simultaneously for Level 3, 4, or 5 driving. For example, in at least one embodiment, a warning sign consisting of "Caution: Flashing lights indicate icy conditions" along with an electric light may be interpreted independently or collectively by several neural networks. In at least one embodiment, the sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" may be interpreted by a second deployed neural network, which, when the flashing light is detected, informs the vehicle's route planning software (preferably running on a CPU complex) that an icy condition exists. In at least one embodiment, the flashing light may be identified by running a third deployed neural network over multiple frames, and the third deployed neural network informs the vehicle's route planning software of the presence (or absence) of the flashing light. In at least one embodiment, all three neural networks may run simultaneously, such as within the DLA and / or on the GPU(s) 1608.

[0203] In at least one embodiment, a CNN for facial recognition and vehicle owner identification may use data from the camera sensor to identify the presence of an authorized driver and / or owner of the vehicle 1600. In at least one embodiment, an always-on sensor processing engine may be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and in security mode, disable the vehicle when the owner leaves the vehicle. In this manner, the SoC(s) 1604 provide security against theft and / or carjacking.

[0204] In at least one embodiment, a CNN for emergency vehicle detection and identification may detect and identify emergency vehicle sirens using data from microphone 1696. In at least one embodiment, SoC(s) 1604 uses a CNN to classify environmental and urban sounds as well as visual data. In at least one embodiment, a CNN running on the DLA is trained to identify the relative speed at which an emergency vehicle is approaching (e.g., by using the Doppler effect). In at least one embodiment, the CNN may also be trained to identify emergency vehicles specific to the region in which the vehicle is operating, as identified by GNSS sensor(s) 1658. In at least one embodiment, when operating in Europe, the CNN attempts to detect European sirens, and when in the United States, the CNN attempts to identify only North American sirens. In at least one embodiment, when an emergency vehicle is detected, the control program may be used to execute an emergency vehicle safety routine, slow the vehicle, pull over, stop the vehicle, and / or idle the vehicle in conjunction with the ultrasonic sensor(s) 1662 until the emergency vehicle(s) have passed.

[0205] In at least one embodiment, vehicle 1600 may include CPU(s) 1618 (e.g., discrete CPU(s) or dCPU(s)), which may be coupled to SoC(s) 1604 via a high-speed interconnect (e.g., PCIe). In at least one embodiment, CPU(s) 1618 may include, for example, an X86 processor. CPU(s) 1618 may be used to perform any of a variety of functions, including, for example, reconciling potentially inconsistent results between ADAS sensors and SoC(s) 1604 and / or monitoring the status and health of controller(s) 1636 and / or infotainment system on a chip (“infotainment SoC”) 1630.

[0206] In at least one embodiment, vehicle 1600 may include GPU(s) 1620 (e.g., discrete GPU(s) or dGPU(s)), which may be coupled to SoC(s) 1604 via a high-speed interconnect (e.g., NVIDIA's NVLINK). In at least one embodiment, GPU(s) 1620 may provide additional artificial intelligence functionality, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based at least in part on input (e.g., sensor data) from sensors of vehicle 1600.

[0207] In at least one embodiment, vehicle 1600 may further include a network interface 1624, which may include, but is not limited to, wireless antenna(s) 1626 (e.g., one or more wireless antenna(s) 1626 for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). In at least one embodiment, network interface 1624 may be used to enable wireless connectivity over the Internet with the cloud (e.g., with server(s) and / or other network devices), with other vehicles, and / or with computing devices (e.g., passenger client devices). In at least one embodiment, to communicate with other vehicles, a direct link may be established between vehicle 160 and the other vehicles and / or an indirect link may be established (e.g., across a network and via the Internet). In at least one embodiment, the direct link may be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link may provide vehicle 1600 with information about vehicles in its vicinity (e.g., vehicles in front of, to the sides of, and / or behind vehicle 1600). In at least one embodiment, the aforementioned functionality may be part of the cooperative adaptive cruise control functionality of vehicle 1600.

[0208] In at least one embodiment, the network interface 1624 may include an SoC that provides modulation and demodulation functionality and enables the controller(s) 1636 to communicate over a wireless network. In at least one embodiment, the network interface 1624 may include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. In at least one embodiment, the frequency conversion may be performed in any technically feasible manner. For example, the frequency conversion may be performed through well-known processes and / or using a super-heterodyne process. In at least one embodiment, the radio frequency front end functionality may be provided by a separate chip. In at least one embodiment, the network interface may include wireless functionality for communicating via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0209] In at least one embodiment, vehicle 1600 may further include data store(s) 1628, which may include, but are not limited to, off-chip (e.g., not on SoC(s) 1604) storage. In at least one embodiment, data store(s) 1628 may include one or more storage elements, including, but not limited to, RAM, SRAM, dynamic random access memory (“DRAM”), video random-access memory (“VRAM”), flash, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0210] In at least one embodiment, vehicle 1600 may further include GNSS sensor(s) 1658 (e.g., GPS and / or assisted GPS sensors) to assist in mapping, perception, occupancy grid generation, and / or route planning functions. In at least one embodiment, any number of GNSS sensors 1658 may be used, including, for example, but not limited to, a GPS using a USB connector with an Ethernet-to-serial (e.g., RS-232) bridge.

[0211] In at least one embodiment, vehicle 1600 may further include RADAR sensor(s) 1660. RADAR sensor(s) 1660 may be used by vehicle 1600 for long-range vehicle detection, even in darkness and / or severe weather conditions. In at least one embodiment, the RADAR functional safety level may be ASIL B. RADAR sensor(s) 1660 may use CAN and / or bus 1602 for control (e.g., to transmit data generated by RADAR sensor(s) 1660) and to access object tracking data, in some instances, along with access to Ethernet for accessing raw data. In at least one embodiment, a wide variety of RADAR sensor types may be used. For example, without limitation, RADAR sensor(s) 1660 may be suitable for forward, rear, and side RADAR use. In at least one embodiment, one or more of the RADAR sensor(s) 1660 are pulse-Doppler RADAR sensor(s).

[0212] In at least one embodiment, the RADAR sensor(s) 1660 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, and short-range lateral coverage. In at least one embodiment, the long-range RADAR may be used for adaptive cruise control functionality. In at least one embodiment, the long-range RADAR system may provide a wide field of view achieved by two or more independent scans, such as within a 250 meter range. In at least one embodiment, the RADAR sensor(s) 1660 may help distinguish between static and moving objects and may be used by the ADAS system 1638 for emergency braking assistance and forward collision warning. The sensor(s) 1660 included in a long-range RADAR system may include, but are not limited to, multiple (e.g., six or more) fixed RADAR antennas, as well as monostatic and multimodal RADARs with high-speed CAN and FlexRay interfaces. In at least one embodiment, if there are six antennas, the center four antennas may create a focused beam pattern designed to record the surroundings of vehicle 1600 at higher speeds with minimal interference from traffic in adjacent lanes. In at least one embodiment, the other two antennas may increase the field of view, which may allow for quick detection of vehicles entering or exiting vehicle 1600's lane.

[0213] In at least one embodiment, the medium-range RADAR system may include, by way of example, a range of up to 160 meters (forward) or 80 meters (rearward) and a field of view of up to 42 degrees (forward) or 150 degrees (rearward). In at least one embodiment, the short-range RADAR system may include any number of RADAR sensors 1660 designed to be mounted on either end of a rear bumper, without limitation. When mounted on either end of a rear bumper, in at least one embodiment, the RADAR sensor system may create two beams that constantly monitor blind spots behind and adjacent to the vehicle. In at least one embodiment, the short-range RADAR system may be used in an ADAS system 1638 for blind spot detection and / or lane change assistance.

[0214] In at least one embodiment, the vehicle 1600 may further include ultrasonic sensor(s) 1662. The ultrasonic sensor(s) 1662, which may be positioned at the front, rear, and / or sides of the vehicle 1600, may be used for parking assistance and / or to create and update an occupancy grid. In at least one embodiment, a variety of ultrasonic sensor(s) 1662 may be used, and different ultrasonic sensor(s) 1662 may be used for different detection ranges (e.g., 2.5 m, 4 m). In at least one embodiment, the ultrasonic sensor(s) 1662 may operate at a functional safety level of ASIL B.

[0215] In at least one embodiment, the vehicle 1600 may include one or more LIDAR sensors 1664. The LIDAR sensor(s) 1664 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. In at least one embodiment, the LIDAR sensor(s) 1664 may be functional safety level ASIL B. In at least one embodiment, the vehicle 1600 may include multiple LIDAR sensors 1664 (e.g., two, four, six, etc.), and the LIDAR sensors 1664 may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0216] In at least one embodiment, the LIDAR sensor(s) 1664 may be capable of providing a list of objects and their distances for a 360-degree field of view. In at least one embodiment, commercially available LIDAR sensor(s) 1664 may have an advertised range of approximately 100 meters, with an accuracy of 2 cm to 3 cm, and support for a 100 Mbps Ethernet connection, for example. In at least one embodiment, one or more non-protruding LIDAR sensors 1664 may be used. In such an embodiment, the LIDAR sensor(s) 1664 may be implemented as small devices that may be integrated into the front, rear, sides, and / or corners of the vehicle 1600. In at least one embodiment, the LIDAR sensor(s) 1664 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, with a range of 200 meters even for low-reflectivity objects. In at least one embodiment, the forward mounted LIDAR sensor(s) 1664 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0217] In at least one embodiment, LIDAR technology such as 3D flash LIDAR may also be used. 3D flash LIDAR uses a laser flash as a transmitting source to illuminate the surroundings of the vehicle 1600 up to approximately 200 meters. In at least one embodiment, the flash LIDAR unit may include, but is not limited to, a receptor that records the transit time of the laser pulse and the reflected light on each pixel, which corresponds to the range from the vehicle 1600 to the object. In at least one embodiment, flash LIDAR may enable a highly accurate, distortion-free image of the surroundings to be generated with each laser flash. In at least one embodiment, four flash LIDAR sensors may be deployed, one on each side of the vehicle 1600. In at least one embodiment, the 3D flash LIDAR system may include, but is not limited to, a solid-state 3D staring array LIDAR camera (e.g., a non-scanning LIDAR device) with no moving parts other than a fan. In at least one embodiment, the flash LIDAR device(s) may use 5 nanosecond Class I (eye-safe) laser pulses per frame and may capture reflected laser light in the form of a 3D range point cloud and co-registered intensity data.

[0218] In at least one embodiment, the vehicle may further include IMU sensor(s) 1666. In at least one embodiment, the IMU sensor(s) 1666 may be located at the center of the rear axle of the vehicle 1600. In at least one embodiment, the IMU sensor(s) 1666 may include, for example, but not limited to, accelerometer(s), magnetometer(s), gyroscope(s), magnetic compass(es), and / or other sensor types. In at least one embodiment, such as in a 6-axis application, the IMU sensor(s) 1666 may include, but are not limited to, an accelerometer and a gyroscope. In at least one embodiment, such as in a 9-axis application, the IMU sensor(s) 1666 may include, but are not limited to, an accelerometer, a gyroscope, and a magnetometer.

[0219] In at least one embodiment, the IMU sensor(s) 1666 may be implemented as a compact, high-performance GPS-Aided Inertial Navigation System ("GPS / INS") that combines micro-electro-mechanical systems ("MEMS") inertial sensors, a highly sensitive GPS receiver, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. In at least one embodiment, the IMU sensor(s) 1666 may enable the vehicle 1600 to estimate orientation without requiring input from a magnetic sensor by directly observing changes in velocity and correlating that from GPS to the IMU sensor(s) 1666. In at least one embodiment, the IMU sensor(s) 1666 and the GNSS sensor(s) 1658 may be combined in a single, integrated unit.

[0220] In at least one embodiment, vehicle 1600 may include microphone(s) 1696 positioned in and / or around vehicle 1600. In at least one embodiment, microphone(s) 1696 may be used for, among other things, emergency vehicle detection and identification.

[0221] In at least one embodiment, vehicle 1600 may further include any number of camera types, including stereo camera(s) 1668, wide-angle camera(s) 1670, infrared camera(s) 1672, surrounding camera(s) 1674, long-range camera(s) 1698, mid-range camera(s) 1676, and / or other camera types. In at least one embodiment, cameras may be used to capture image data around the entire perimeter of vehicle 1600. In at least one embodiment, the types of cameras used depend on vehicle 1600. In at least one embodiment, any combination of camera types may be used to provide the required coverage around vehicle 1600. In at least one embodiment, the number of cameras may vary depending on the embodiment. For example, in at least one embodiment, vehicle 1600 may include six cameras, seven cameras, ten cameras, twelve cameras, or another number of cameras. The cameras may support, by way of example and not limitation, Gigabit Multimedia Serial Link ("GMSL") and / or Gigabit Ethernet. In at least one embodiment, each of the camera(s) is described in more detail herein above with respect to Figures 16A and 16B.

[0222] In at least one embodiment, vehicle 1600 may further include vibration sensor(s) 1642. In at least one embodiment, vibration sensor(s) 1642 may measure vibration of a component of vehicle 1600, such as an axle(s). For example, in at least one embodiment, a change in vibration may indicate a change in the road surface. In at least one embodiment, when two or more vibration sensors 1642 are used, the difference in vibration may be used to determine the amount of friction or slippage of the road surface (e.g., when the difference in vibration is between a powered axle and a free-spinning axle).

[0223] In at least one embodiment, vehicle 1600 may include an ADAS system 1638. ADAS system 1638 may include, but is not limited to, an SoC in some instances. In at least one embodiment, the ADAS system 1638 may include, without limitation, any number and combination of autonomous / adaptive / automatic cruise control ("ACC") systems, cooperative adaptive cruise control ("CACC") systems, forward crash warning ("FCW") systems, automatic emergency braking ("AEB") systems, lane departure warning ("LDW") systems, lane keep assist ("LKA") systems, blind spot warning ("BSW") systems, rear cross-traffic warning ("RCTW") systems, collision warning ("CW") systems, lane centering ("LC") systems, and / or other systems, features, and / or functionality.

[0224] In at least one embodiment, the ACC system may use RADAR sensor(s) 1660, LIDAR sensor(s) 1664, and / or any number of cameras. In at least one embodiment, the ACC system may include a longitudinal ACC system and / or a lateral ACC system. In at least one embodiment, the longitudinal ACC system monitors and controls the distance to the vehicle immediately preceding the vehicle 1600 and automatically adjusts the speed of the vehicle 1600 to maintain a safe distance from the vehicle ahead. In at least one embodiment, the lateral ACC system enforces distance maintenance and advises the vehicle 1600 to change lanes when necessary. In at least one embodiment, the lateral ACC is related to other ADAS applications, such as LC and CW.

[0225] In at least one embodiment, the CACC system uses information from other vehicles, which may be received via a wireless link or indirectly, via a network connection (e.g., via the Internet), from other vehicles via network interface 1624 and / or wireless antenna(s) 1626. In at least one embodiment, the direct link may be provided by a vehicle-to-vehicle ("V2V") communication link, and the indirect link may be provided by an infrastructure-to-vehicle ("I2V") communication link. Generally, V2V communication concepts provide information about an immediate preceding vehicle (e.g., a vehicle immediately preceding vehicle 1600 and in the same lane), while I2V communication concepts provide information about traffic further ahead. In at least one embodiment, the CACC system may include either or both I2V and V2V information sources. In at least one embodiment, with information about vehicles in front of vehicle 1600, the CACC system can be more reliable, which has the potential to improve traffic flow and reduce congestion on roads.

[0226] In at least one embodiment, the FCW system is designed to alert the driver of hazards so that the driver can take corrective action. In at least one embodiment, the FCW system uses a front-facing camera and / or RADAR sensor(s) 1660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibration components. In at least one embodiment, the FCW system may provide warnings in the form of an audible, visual warning, vibration, and / or quick brake pulse.

[0227] In at least one embodiment, the AEB system may detect an imminent forward collision with another vehicle or other object and automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. In at least one embodiment, the AEB system may use a front-facing camera(s) and / or a RADAR sensor(s) 1660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. In at least one embodiment, when the AEB system detects a hazard, the AEB system typically first alerts the driver to take corrective action to avoid the collision; if the driver does not take corrective action, the AEB system may automatically apply the brakes to prevent or at least mitigate the impact of the predicted collision. In at least one embodiment, the AEB system may include techniques such as dynamic brake support and / or pre-crash braking.

[0228] In at least one embodiment, the LDW system provides visual, audible, and / or tactile warnings, such as vibration of the steering wheel or seat, to alert the driver when the vehicle 1600 crosses a lane marker. In at least one embodiment, the LDW system does not activate when the driver indicates an intentional lane departure by activating a turn signal. In at least one embodiment, the LDW system may use a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibration components. In at least one embodiment, the LKA system is a variation of the LDW system. The LKA system provides steering inputs or brake control to correct the vehicle 1600 if the vehicle 1600 begins to drift out of its lane.

[0229] In at least one embodiment, the BSW system detects vehicles in the vehicle's blind spot and warns the driver about the vehicles. In at least one embodiment, the BSW system may provide visual, audible, and / or tactile alerts to indicate that it is unsafe to merge or change lanes. In at least one embodiment, the BSW system may provide an additional warning when the driver uses a turn signal. In at least one embodiment, the BSW system may use rear-facing camera(s) and / or RADAR sensor(s) 1660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibration components.

[0230] In at least one embodiment, the RCTW system may provide visual, audible, and / or tactile notifications when an object is detected outside the range of the rear camera when the vehicle 1600 is reversing. In at least one embodiment, the RCTW system includes an AEB system to ensure vehicle brakes are applied to avoid a crash. In at least one embodiment, the RCTW system may use one or more rear RADAR sensors 1660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibration components.

[0231] In at least one embodiment, conventional ADAS systems may be prone to false positive results, which can be annoying and distracting to the driver, but this is usually not a major issue because conventional ADAS systems alert the driver, allowing the driver to determine whether a safety condition truly exists and act accordingly. In at least one embodiment, the vehicle 1600 itself determines whether to follow the result from the primary computer (e.g., the first controller 1636) or the result from the secondary computer (e.g., the second controller 1636) in the case of conflicting results. For example, in at least one embodiment, the ADAS system 1638 may be a backup and / or secondary computer for providing cognitive information to a backup computer rationality module. In at least one embodiment, the backup computer rationality monitor may run redundant and diverse software on hardware components to detect impairments in cognitive and dynamic driving tasks. In at least one embodiment, output from the ADAS system 1638 may be provided to a supervisory MCU. In at least one embodiment, if the output from the primary computer and the output from the secondary computer conflict, the supervising MCU determines how to reconcile the conflict to ensure safe operation.

[0232] In at least one embodiment, the primary computer may be configured to provide the overseer MCU with a reliability score indicating the primary computer's reliability in a selected outcome. In at least one embodiment, if the reliability score exceeds a threshold, the overseer MCU may follow the primary computer's instructions regardless of whether the secondary computers provide conflicting or inconsistent results. In at least one embodiment, if the reliability score does not meet the threshold, and if the primary and secondary computers exhibit different (e.g., conflicting) results, the overseer MCU may arbitrate between the computers to determine an appropriate outcome.

[0233] In at least one embodiment, the supervisory MCU can be configured to run neural network(s) trained and configured to determine conditions under which the secondary computer will provide a false alarm based at least in part on outputs from the primary and secondary computers. In at least one embodiment, the neural network(s) in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot be trusted. For example, in at least one embodiment, when the secondary computer is a RADAR-based FCW system, the neural network(s) in the supervisory MCU can learn when the FCW system identifies a metal object that is not actually a hazard, such as a drain grate or manhole cover, which triggers an alarm. In at least one embodiment, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disable LDW when a bicyclist or pedestrian is present and lane departure is actually the safest maneuver. In at least one embodiment, the supervising MCU may include at least one of a DLA or a GPU suitable for running neural network(s) along with associated memory. In at least one embodiment, the supervising MCU may comprise and / or be included as a component of SoC1604(s).

[0234] In at least one embodiment, the ADAS system 1638 may include a secondary computer that implements ADAS functionality using traditional rules of computer vision. In at least one embodiment, the secondary computer may use traditional computer vision rules (if-then), and the presence of neural network(s) in the supervisory MCU may improve reliability, safety, and performance. For example, in at least one embodiment, diverse implementations and intentional non-identity make the overall system more fault-tolerant, particularly to failures caused by software (or software-hardware interface) functionality. For example, in at least one embodiment, if there is a software bug or error in software running on the primary computer and non-identical software code running on the secondary computer provides the same overall result, the supervisory MCU may have higher confidence that the overall result is correct and that a bug in the software or hardware on the primary computer did not cause a critical error.

[0235] In at least one embodiment, the output of the ADAS system 1638 can be fed to the perception block of the primary computer and / or the dynamic driving task block of the primary computer. For example, in at least one embodiment, if the ADAS system 1638 indicates a frontal crash warning due to an upcoming object, the perception block can use this information when identifying the object. In at least one embodiment, the secondary computer can have its own neural network trained as described herein, thus reducing the risk of false positives.

[0236] In at least one embodiment, vehicle 1600 may further include an infotainment SoC 1630 (e.g., an in-vehicle infotainment system (IVI)). Although shown and described as an SoC, infotainment system 1630, in at least one embodiment, may not be an SoC and may include, without limitation, two or more separate components. In at least one embodiment, infotainment SoC 1630 may include, but is not limited to, a combination of hardware and software that may be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation systems, rear parking assist, wireless data systems, vehicle-related information such as fuel level, total mileage, brake fuel level, oil level, door opening / closing, air filter information, etc.) to vehicle 1600. For example, infotainment SoC 1630 may include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, a car computer, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, a heads-up display (“HUD”), an HMI display 1634, telematics devices, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. In at least one embodiment, infotainment SoC 1630 may further be used to provide information (e.g., visual and / or auditory) to a user(s) of the vehicle, such as information from an ADAS system 1638, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0237] In at least one embodiment, infotainment SoC 1630 may include any amount and type of GPU functionality. In at least one embodiment, infotainment SoC 1630 may communicate with other devices, systems, and / or components of vehicle 1600 via bus 1602 (e.g., CAN bus, Ethernet, etc.). In at least one embodiment, infotainment SoC 1630 may be coupled to a supervisory MCU such that the infotainment system's GPU may perform some self-driving functions if primary controller(s) 1636 (e.g., vehicle 1600's primary and / or backup computer) fails. In at least one embodiment, infotainment SoC 1630 may place vehicle 1600 in a driver-safety shutdown mode, as described herein.

[0238] In at least one embodiment, vehicle 1600 may further include an instrument cluster 1632 (e.g., a digital dashboard, an electronic instrument cluster, a digital instrument panel, etc.). In at least one embodiment, instrument cluster 1632 may include, but is not limited to, a controller and / or a supercomputer (e.g., a separate controller or supercomputer). In at least one embodiment, instrument cluster 1632 may include any number and combination of instrumentation sets, such as, but not limited to, a speedometer, fuel level, oil pressure, a tachometer, an odometer, direction indicators, a shift lever position indicator, seat belt warning light(s), parking brake warning light(s), engine malfunction light(s), supplemental restraint system (e.g., airbag) information, light control, safety system control, navigation information, etc. In some instances, information may be displayed and / or shared between infotainment SoC 1630 and instrument cluster 1632. In at least one embodiment, the instrument cluster 1632 may be included as part of the infotainment SoC 1630, or vice versa.

[0239] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. More details regarding inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B. In at least one embodiment, inference and / or training logic 715 may be used in the system of Figure 16A for inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functionality and / or architecture, or neural network use cases described herein.

[0240] 16B is a diagram of a system 1676 for communication between cloud-based server(s) and the autonomous vehicle 1600 of FIG. 16A , according to at least one embodiment. In at least one embodiment, the system 1676 may include, but is not limited to, server(s) 1678, network(s) 1690, and any number and type of vehicles, including vehicle 1600. In at least one embodiment, the server(s) 1678 may include, but is not limited to, multiple GPUs 1684(A)-1684(H) (collectively referred to herein as GPUs 1684), PCIe switches 1682(A)-1682(D) (collectively referred to herein as PCIe switch 1682), and / or CPUs 1680(A)-1680(B) (collectively referred to herein as CPUs 1680). The GPUs 1684, CPUs 1680, and PCIe switches 1682 may be interconnected with a high-speed interconnect, such as, but not limited to, an NVLink interface 1688 and / or PCIe connections 1686 developed by NVIDIA. In at least one embodiment, the GPUs 1684 are connected via an NVLink and / or NVSwitch SoC, and the GPUs 1684 and PCIe switches 1682 are connected via a PCIe interconnect. In at least one embodiment, eight GPUs 1684, two CPUs 1680, and four PCIe switches 1682 are shown, but this is not intended to be limiting. In at least one embodiment, each of the server(s) 1678 may include any number of GPUs 1684, CPUs 1680, and / or PCIe switches 1682 in any combination, including, but not limited to, For example, in at least one embodiment, server(s) 1678 may each include 8, 16, 32, and / or more GPUs 1684.

[0241] In at least one embodiment, server(s) 1678 may receive image data from the vehicle over network(s) 1690 representing images showing unexpected or changed road conditions, such as recently begun road construction. In at least one embodiment, server(s) 1678 may transmit neural network 1692, updated neural network 1692, and / or map information 1694, including, but not limited to, information about traffic and road conditions, to the vehicle over network(s) 1690. In at least one embodiment, updates to map information 1694 may include updates to HD map 1622, such as, but not limited to, information about construction sites, potholes, detours, flooding, and / or other obstacles. In at least one embodiment, neural network 1692, updated neural network 1692, and / or map information 1694 may result from new training and / or experience represented in data received from any number of vehicles in the environment and / or based at least in part on training performed at a data center (e.g., using server(s) 1678 and / or other servers).

[0242] In at least one embodiment, server(s) 1678 may be used to train a machine learning model (e.g., a neural network) based at least in part on the training data. In at least one embodiment, the training data may be generated by the vehicle and / or generated in a simulation (e.g., using a game engine). In at least one embodiment, any amount of the training data may be tagged and / or subjected to other preprocessing (e.g., if the associated neural network benefits from supervised learning). In at least one embodiment, any amount of the training data may not be tagged and / or preprocessed (e.g., if the associated neural network does not require supervised learning). In at least one embodiment, once the machine learning model is trained, it may be used by the vehicle (e.g., transmitted to the vehicle via network(s) 1690) and / or used by server(s) 1678 to remotely monitor the vehicle.

[0243] In at least one embodiment, server(s) 1678 may receive data from vehicles and apply the data to state-of-the-art real-time neural networks for real-time intelligent inference. In at least one embodiment, server(s) 1678 may include deep learning supercomputers and / or special-purpose AI computers powered by GPU(s) 1684, such as the DGX and DGX Station machines developed by NVIDIA. However, in at least one embodiment, server(s) 1678 may include a deep learning infrastructure using CPU-powered data centers.

[0244] In at least one embodiment, the deep learning infrastructure of server(s) 1678 may be capable of fast real-time inference and may use that capability to assess and verify the health of processors, software, and / or associated hardware in vehicle 1600. For example, in at least one embodiment, the deep learning infrastructure may receive periodic updates from vehicle 1600, such as a series of images and / or objects that vehicle 1600 has located in the series of images (e.g., via computer vision and / or other machine learning object classification techniques). In at least one embodiment, the deep learning infrastructure may run its own neural network to identify objects and compare them to the objects identified by vehicle 1600; if the results do not match and the deep learning infrastructure concludes that the AI ​​in vehicle 1600 is malfunctioning, server(s) 1678 may send a signal to vehicle 1600 instructing a fail-safe computer in vehicle 1600 to assume control, notify passengers, and complete a safe parking maneuver.

[0245] In at least one embodiment, server(s) 1678 may include GPU(s) 1684 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT3). In at least one embodiment, the combination of GPU-powered servers and inference acceleration may enable real-time response. In at least one embodiment, servers powered by CPUs, FPGAs, and other processors may be used for inference, such as when performance is less critical. In at least one embodiment, inference and / or training logic 715 is used to implement one or more embodiments. More details regarding the inference and / or training logic 715 are provided below in conjunction with FIG. 7A and / or FIG. 7B.

[0246] Other variations are within the scope of the present disclosure. Thus, while the disclosed techniques are susceptible to various modifications and alternative constructions, several illustrative examples of which have been shown in the drawings and described above in detail. There is no intention, however, to limit the disclosure to the particular disclosed form or forms, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the disclosure, as defined in the appended claims.

[0247] The use of the terms "a," "an," and "the," and similar referents in the context of describing the disclosed embodiments (particularly in the context of the claims that follow) should be construed to cover both the singular and the plural and not as defining terms, unless otherwise stated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" should be construed as open-ended terms (meaning "including, but not limited to"), unless otherwise indicated. The term "connected," when unmodified and referring to a physical connection, should be construed as partially or completely contained within, attached to, or joined to one another, even if there is something intervening. Reciting ranges of values ​​herein is merely intended to serve as a shorthand method of individually referring to each separate value falling within the range, unless otherwise stated herein and unless each separate value is incorporated into the specification as if it were individually recited herein. Use of the term "set" (e.g., "set of items") or "subset" should be construed as a non-empty collection comprising one or more members, unless otherwise stated or contradicted by context. Furthermore, unless otherwise stated or contradicted by context, the term "subset" of a corresponding set does not necessarily refer to a strict subset of the corresponding set; a subset and a corresponding set may be equivalent.

[0248] Connective language, such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C," is understood in the context in which it is generally used to indicate that an item, term, etc. may be either A or B or C, or any non-empty subset of the set A, B, and C, unless specifically stated otherwise or clearly contradicted by context. For example, in the illustrative example of a set having three members, the connective phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such connective language does not generally imply that some embodiments require that at least one of A, at least one of B, and at least one of C each be present. Further, unless stated otherwise or negated by context, the term "plurality" refers to a plurality (e.g., "a plurality of items" refers to multiple items). A plurality is at least two items, but may be more when so indicated either explicitly or by context. Further, unless stated otherwise or otherwise clear from context, the phrase "based on" means "based at least in part on," and not "based only on."

[0249] The operations of processes described herein may be performed in any suitable order unless otherwise stated herein or otherwise clearly contradicted by context. In at least one embodiment, processes such as those described herein (or variations and / or combinations thereof) are performed under the control of one or more computer systems configured of executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, e.g., in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transitory signals (e.g., propagating transitory electrical or electromagnetic transmissions), but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within a transceiver of a transitory signal. In at least one embodiment, the code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media that store executable instructions (or have other memory for storing executable instructions) that, when executed by (i.e., as a result of being executed by) one or more processors of a computer system, cause the computer system to perform the operations described herein. The set of non-transitory computer-readable storage media, in at least one embodiment, comprises a plurality of non-transitory computer-readable storage media, wherein one or more individual non-transitory storage media of the plurality of non-transitory computer-readable storage media do not contain all of the code, but the plurality of non-transitory computer-readable storage media collectively store all of the code.In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors, e.g., a non-transitory computer-readable storage medium stores the instructions, a main central processing unit ("CPU") executes some of the instructions, and a graphics processing unit ("GPU") executes other instructions. In at least one embodiment, different components of a computer system have separate processors, with different processors executing different subsets of the instructions.

[0250] Thus, in at least one embodiment, a computer system is configured to implement one or more services that, singly or collectively, perform the operations of the processes described herein, and such a computer system is configured with applicable hardware and / or software that enables the performance of the operations. Further, a computer system that implements at least one embodiment of the present disclosure is a single device, and in another embodiment, is a distributed computer system comprising multiple devices that operate in different ways, such that a distributed computer system performs the operations described herein, and such that no single device performs all of the operations.

[0251] The use of any illustrative or exemplary language (e.g., "such as") provided herein is intended only to further clarify embodiments of the disclosure and does not limit the scope of the disclosure unless otherwise stated. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0252] All references cited herein, including publications, patent applications, and patents, are hereby incorporated by reference to the same extent as if each reference was individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0253] In the specification and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms may not be intended as synonyms for each other. Rather, in particular instances, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but yet still cooperate or interact with each other.

[0254] Unless specifically stated otherwise, throughout the specification, terms such as "processing," "computing," "calculating," or "determining" refer to the actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or transform data represented as physical quantities, such as electronic quantities, in the computing system's registers and / or memory to other data similarly represented as physical quantities in the computing system's memory, registers, or other such information storage, transmission, or display device.

[0255] Similarly, the term "processor" may refer to any device, or portion of a device, that processes electronic data from registers and / or memory and converts the electronic data into other electronic data that may be stored in registers and / or memory. As a non-limiting example, a "processor" may be a CPU or GPU. A "computing platform" may comprise one or more processors. As used herein, a "software" process may include software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Also, each process may refer to multiple processes for executing instructions serially or in parallel, continuously or intermittently. The terms "system" and "method" are used interchangeably herein, provided that a system may embody one or more methods and a method may be considered a system.

[0256] As used herein, references may be made to acquiring, obtaining, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. Acquiring, obtaining, receiving, or inputting analog or digital data may be accomplished in various ways, such as by receiving data as parameters of a function call or a call to an application programming interface. In some implementations, the process of acquiring, obtaining, receiving, or inputting analog or digital data may be accomplished by transferring data over a serial or parallel interface. In other implementations, the process of acquiring, obtaining, receiving, or inputting analog or digital data may be accomplished by transferring data over a computer network from a providing entity to an acquiring entity. References may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various instances, the process of providing, outputting, transmitting, sending, or presenting analog or digital data may be accomplished by transferring data as input or output parameters of a function call, an application programming interface, or an inter-process communication mechanism.

[0257] While the above description describes example implementations of the described techniques, other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. Additionally, while a specific distribution of responsibilities has been defined above for purposes of explanation, various functions and responsibilities may be distributed and divided in different ways depending on the circumstances.

[0258] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claims.

Claims

1. generating a localization decision using at least two sources of localization data; comparing the localization decision to at least two trusted localization values ​​to ascertain confidence in the localization decision; generating a navigation path using at least two sources of knowledge about the environment and the localization determination; comparing the navigation route to at least two trusted sources of navigation route data to verify trustworthiness of the navigation route; providing the navigation path to a control system for navigating an object within the environment; A method comprising:

2. 2. The method of claim 1, wherein the object is a vehicle and the navigation route comprises a lane graph generated using at least one source of map data and at least one source perceiving the information about the environment corresponding to the vehicle.

3. The method of claim 1 , wherein the source of localization data includes at least one of a camera, a global positioning system (GPS), or a radar system.

4. The method of claim 1 , wherein the source of knowledge of the environment comprises at least one of a camera, a radar system, or a LIDAR system, an ultrasound system, or high-definition (HD) map data.

5. determining the reliability of the localization determination before performing the step of comparing with the at least two trusted localization values; if the reliability is below a reliability threshold, replacing the localization decision with a most reliable localization value from the at least two sources of localization data; The method of claim 1 further comprising:

6. determining the reliability of the navigation route prior to performing the step of comparing with the at least two trusted sources of navigation route data; if the reliability is below a reliability threshold, replacing the navigation path with navigation data from the at least two sources of localization data corresponding to a highest reliability level; The method of claim 1 further comprising:

7. determining that at least one source of localization data is temporarily unavailable or unreliable; utilizing historical localization data in place of said at least one source of localization data; The method of claim 1 further comprising:

8. dynamically adjusting a selection of said at least two sources of localization data or said at least two sources of knowledge of said information about said environment based on an availability or reliability determination. The method of claim 1 further comprising:

9. generating a second navigation path; dynamically determining whether to provide the navigation path or the second navigation path for navigation of the object; The method of claim 1 further comprising:

10. a localization fusion module for generating a fused localization decision based at least in part on the independent localization decisions received from the at least two independent sources; a localization confidence monitor for determining confidence in the fused localization decision based at least in part on a match between the fused localization decision and the independent localization decisions; a lane graph fusion module for generating a fused or combined lane graph based at least in part on the independent lane graphs received from at least two independent sources and further based at least in part on the fused localization determination; a lane graph reliability monitor for determining reliability for the fused or combined lane graph based at least in part on a match between the fused or combined lane graph and the independent lane graphs; a control system for causing a vehicle to navigate according to the fused or combined lane graph in response to the fused or combined lane graph having at least a minimum level of reliability; and A vehicle control system comprising:

11. 11. The vehicle control system of claim 10, wherein the control system, in response to the fused or combined lane graph having less than a minimum level of reliability, causes the vehicle to navigate according to an alternate lane graph.

12. 11. The vehicle control system of claim 10, further comprising a plurality of sensors of two or more sensor types configured to capture data for use in determining at least one of the independent localization determinations or the independent lane graphs, the sensor types including at least one of camera, radar, ultrasonic, LIDAR, GPS, or IMU data.

13. a plurality of localization comparators for comparing the independent localization decisions with the fused localization decision; a plurality of lane graph comparators for comparing the independent lane graphs with the fused or combined lane graph, wherein the localization confidence monitor determines the confidence for the fused localization decision based at least in part on results of the localization comparators, and the lane graph confidence monitor determines the confidence for the fused or combined lane graph based at least in part on results of the lane graph comparators; The vehicle control system of claim 10 further comprising:

14. 11. The vehicle control system of claim 10, further comprising an arbiter for determining when at least one source of localization data is temporarily unavailable or unreliable, wherein historical localization data is allowed to be utilized in place of the at least one source of localization data.

15. The vehicle control system of claim 10 , wherein the vehicle control system is enabled to dynamically adjust the selection of the at least two independent sources based on availability or reliability determinations.

16. one or more processing units; A memory containing instructions and wherein the instructions, when executed by one or more processors, cause the system to: generating a localization decision using at least two sources of localization data; comparing the localization decision to at least two trusted localization values ​​to ascertain confidence in the localization decision; generating a navigation path using at least two sources of knowledge about the environment and the localization determination; comparing the navigation route with at least two trusted sources of navigation route data to verify trustworthiness of the navigation route; and providing the navigation path to a control system for navigating an object within the environment.

17. The one or more processing units: determining a reliability of the localization determination before performing the comparison with the at least two trusted localization values; replacing the localization decision with a most reliable localization value from the at least two sources of localization data if the reliability is below a reliability threshold; determining the reliability of the navigation route prior to performing the comparison with the at least two trusted sources of navigation route data; if the reliability is below a reliability threshold, replacing the navigation path with the most reliable navigation data from the at least two sources of localization data; The system of claim 16 , further comprising:

18. The one or more processing units determining that at least one source of localization data is temporarily unavailable or unreliable; utilizing historical localization data in place of said at least one source of localization data; and The system of claim 16 , further comprising:

19. One or more processing units generating a second navigation path; dynamically determining whether to provide the navigation path or the second navigation path for navigation of the object; The system of claim 16 , further comprising:

20. The system comprises: a system for performing simulation operations; A system for performing simulation operations to test or validate an autonomous machine application; a system for rendering graphical output; A system for performing deep learning operations, a system implemented using edge devices; a system incorporating one or more virtual machines (VMs); a system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources 17. The system of claim 16, comprising at least one of:

Images