System for achieving algorithmic security in heterogeneous computing platforms

Abstract

Methods, systems, non-transitory media, and devices for supporting security-compliant computing in a heterogeneous computing system, such as a vehicle heterogeneous computing system, are disclosed. Various aspects include methods of enabling vehicles, such as autonomous vehicles, semi-autonomous vehicles, and the like, to achieve algorithm security of various algorithms on heterogeneous computing platforms having various security levels.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 949,426, filed December 17, 2019, entitled “System To Achieve Algorithm Safety In Heterogeneous Compute Platform,” the entirety of which is incorporated by reference and for all purposes.

[0003] BACKGROUND

[0004] As the industry moves toward deployment of autonomous and semi-autonomous vehicles, cars and trucks are becoming more intelligent. Autonomous and semi-autonomous vehicles are capable of detecting information about their location and surrounding environment (e.g., using radar, lidar, GPS, odometer, accelerometers, cameras, and other sensors) and include control systems that interpret the sensory information to identify hazards and determine a navigation path to follow. Autonomous and semi-autonomous vehicles include control systems to operate with limited or no control from an occupant or other operator of the vehicle.

[0005] SUMMARY

[0006] Various aspects include methods of enabling vehicles, such as autonomous vehicles, semi-autonomous vehicles, and the like, to achieve algorithm safety of various algorithms on heterogeneous compute platforms having various levels of safety.

[0007] Various aspects include methods for supporting safety-compliant computing in a heterogeneous computing system, such as a vehicle heterogeneous computing system, that can include receiving an indication that an algorithm requiring safety compliance, such as a vehicle algorithm requiring safety compliance, is to be run in the heterogeneous computing system, determining whether a non-safety-compliant computing unit of the heterogeneous computing system is preferred for running the algorithm, and in response to determining that the non-safety-compliant computing unit of the heterogeneous computing system is preferred for running the algorithm, modifying execution of the algorithm to execute a portion of the algorithm using the non-safety-compliant computing unit of the heterogeneous computing system and to execute another portion of the algorithm using a safety-compliant computing unit of the heterogeneous computing system. In some aspects, modifying execution of the algorithm to execute a portion of the algorithm using the non-safety-compliant computing unit of the heterogeneous computing system and to execute another portion of the algorithm using the safety-compliant computing unit of the heterogeneous computing system can include modifying execution of the algorithm to create a light version of the algorithm for running on the safety-compliant computing unit of the heterogeneous computing system.

[0008] Some aspects can further include running the algorithm on the non-security-compliant computing unit of the heterogeneous computing system to generate a more refined output for the data set, running a light version of the algorithm on the security-compliant computing unit of the heterogeneous computing system to generate a coarse output for the data set, determining whether the more refined output and the coarse output match, and generating an alert in response to determining that the more refined output and the coarse output do not match.

[0009] Some aspects can further include running the algorithm on the non-security-compliant computing unit of the heterogeneous computing system to generate a more refined output for the data set, running a light version of the algorithm on the security-compliant computing unit of the heterogeneous computing system to generate a coarse output for the data set, determining whether the more refined output and the coarse output match, and generating an alert in response to determining that the more refined output and the coarse output do not match.

[0010] In some aspects, modifying execution of the algorithm to execute a portion of the algorithm using the non-security-compliant computing unit of the heterogeneous computing system and to execute another portion of the algorithm using the security-compliant computing unit of the heterogeneous computing system can include identifying a critical portion of the data set, running the algorithm on the security-compliant computing unit of the heterogeneous computing system to generate a more refined output for the identified critical portion of the data set, and running the algorithm on the non-security-compliant computing unit of the heterogeneous computing system to generate a more refined output for all other portions of the data set.

[0011] Various aspects include a method for supporting security-compliant computing in a heterogeneous computing system, such as a vehicle heterogeneous computing system, which can include receiving an indication that an algorithm requiring security compliance, such as a vehicle algorithm requiring security compliance, is to be run in the heterogeneous computing system, determining whether a non-security-compliant computing unit of the heterogeneous computing system is preferred for running the algorithm, modifying execution of the algorithm to create a light version of the algorithm in response to determining that the non-security-compliant computing unit of the heterogeneous computing system is preferred for running the algorithm, identifying an important portion of a data set, running the algorithm on a security-compliant computing unit of the heterogeneous computing system to generate a more refined output for the identified important portion of the data set, and running the light version of the algorithm on the security-compliant computing unit of the heterogeneous computing system to generate a coarse output for all other portions of the data set.

[0012] In various aspects, heterogeneous computing systems (such as vehicle heterogeneous computing systems) can be system-on-a-chip. In various aspects, safety-compliant computing units can be central processing units or digital signal processing units, while non-safety-compliant computing units are graphics processing units. In various aspects, heterogeneous computing systems can be vehicle heterogeneous computing systems, and algorithms requiring safety compliance can be vehicle algorithms requiring safety compliance. In various embodiments, vehicle algorithms requiring safety compliance can be vehicle algorithms requiring Automotive Safety Integrity Level B (ASIL B) compliance.

[0013] A further aspect includes a vehicle containing a processor configured with processor-executable instructions for performing operations of any of the methods outlined above. A further aspect includes a non-transient processor-readable storage medium having processor-executable software instructions stored thereon, the processor-executable software instructions being configured to cause the processor to perform operations of any of the methods outlined above. A further aspect includes a processing device used in a vehicle and configured to perform operations of any of the methods outlined above. Brief description of the attached diagram

[0015] The accompanying drawings, which are incorporated herein and form part of this specification, illustrate various exemplary embodiments and, together with the general description given above and the detailed description given below, serve to explain the features of the various embodiments.

[0016] Figure 1A and 1B This is a block diagram illustrating the components of a vehicle suitable for implementing various embodiments.

[0017] Figure 1C This is a component block diagram illustrating the components of a vehicle suitable for implementing various embodiments.

[0018] Figure 2A This is a component block diagram illustrating the components of an example transportation management system according to various embodiments.

[0019] Figure 2B This is a component block diagram illustrating the components of another example transportation management system according to various embodiments.

[0020] Figure 3 This is a block diagram illustrating components of an example on-chip system for use in a vehicle according to various embodiments, the example on-chip system being configured to broadcast, receive and / or otherwise use intent and / or motion planning.

[0021] Figure 4 A component block diagram of an example system configured to support security-compliant computing in heterogeneous computing systems, such as vehicle heterogeneous computing systems, is shown.

[0022] Figure 5 This is a process flow diagram illustrating methods for supporting security-compliant computing in heterogeneous computing systems, such as heterogeneous computing systems for vehicles.

[0023] Figure 6 This is a process flow diagram illustrating methods for supporting security-compliant computing in heterogeneous computing systems, such as heterogeneous computing systems for vehicles.

[0024] Figure 7 This is a process flow diagram illustrating methods for supporting security-compliant computing in heterogeneous computing systems, such as heterogeneous computing systems for vehicles.

[0025] Figure 8 This is a process flow diagram illustrating methods for supporting security-compliant computing in heterogeneous computing systems, such as heterogeneous computing systems for vehicles.

[0026] Figure 9 This is a process flow diagram illustrating methods for supporting security-compliant computing in heterogeneous computing systems, such as heterogeneous computing systems for vehicles.

[0027] Figure 10 This is a process flow diagram illustrating methods for supporting security-compliant computing in heterogeneous computing systems, such as heterogeneous computing systems for vehicles.

[0028] Detailed description

[0029] Various aspects will be described in detail with reference to the accompanying drawings. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. References to specific examples and embodiments are for illustrative purposes and are not intended to limit the scope of the aspects or claims.

[0030] Various embodiments enable vehicles (such as autonomous vehicles, semi-autonomous vehicles, etc.) to achieve algorithmic security for various algorithms on heterogeneous computing platforms with varying security levels. Various embodiments enable non-security-compliant computing units to be used at least partially for performing security-critical functions. Since some processes can be performed more efficiently on non-security-compliant computing units than on security-compliant computing units, various embodiments can improve the processing efficiency of heterogeneous computing platforms by using non-security-compliant computing units to perform security-critical functions more efficiently, while achieving the same algorithmic security that would otherwise be achieved by exclusively using security-compliant computing units.

[0031] The ground transportation industry is increasingly looking to leverage the growing capabilities of cellular and wireless communication technologies by adopting Intelligent Transportation Systems (ITS) to improve interoperability and safety between driver-operated and autonomous vehicles. The Cellular Vehicle-to-Everything (C-V2X) protocol, defined by the 3GPP project, supports ITS technology and serves as the foundation for direct communication between vehicles and surrounding communication equipment.

[0032] C-V2X defines two transmission modes that together provide 360° non-line-of-sight awareness and a higher level of predictability to enhance road safety and autonomous driving. The first transmission mode includes direct C-V2X, encompassing vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), and vehicle-to-pedestrian (V2P) communication, providing enhanced range and reliability in a dedicated ITS 5.9 GHz spectrum independent of cellular networks. The second transmission mode includes vehicle-to-network (V2N) communication in mobile broadband systems and technologies such as third-generation (3G) wireless mobile communication technologies (e.g., GSM Evolution (EDGE) systems, Code Division Multiple Access (CDMA) 2000 systems, etc.), fourth-generation (4G) wireless mobile communication technologies (e.g., LTE systems, LTE Advanced systems, Mobile Global System for Mobile Communications Interoperability (Mobile WiMAX) systems, etc.), fifth-generation (5G) wireless mobile communication technologies (e.g., 5G New Radio (5GNR) systems, etc.), and so on.

[0033] The term "System-on-a-Chip" (SOC) is used herein to refer to a set of interconnected electronic circuits, typically but not exclusively including one or more processors, memory, and communication interfaces. An SOC can include various types of processors and processor cores, such as general-purpose processors, central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), accelerated processing units (APUs), subsystem processors, auxiliary processors, single-core processors, and multi-core processors. An SOC may further embody other hardware and hardware combinations, such as field-programmable gate arrays (FPGAs), configuration and status registers (CSRs), application-specific integrated circuits (ASICs), other programmable logic devices, discrete gate logic, transistor logic, registers, performance monitoring hardware, watchdog hardware, counters, and time references. An SOC can be an integrated circuit (IC) configured such that the components of the IC are located on the same substrate, such as a monolithic semiconductor material (e.g., silicon).

[0034] The term "safety compliance calculation unit" is used herein to refer to a calculation unit that complies with the vehicle safety integrity level. A safety compliance calculation unit can be a calculation unit that conforms to the Vehicle Safety Integrity Level (ASIL) for automobiles as defined in the International Organization for Standardization (ISO) standard ISO 26262, which defines ASILs for automobiles, such as ASIL A, ASIL B, ASIL C, ASIL D, etc. The term "non-safety compliance calculation unit" is used herein to refer to a calculation unit that does not comply with the vehicle safety integrity level, has not passed ASIL A or ASIL B certification, or has a safety level lower than ASIL B.

[0035] Various embodiments include methods, vehicles, vehicle management systems, and processing devices configured to implement methods for achieving algorithmic security for various algorithms on heterogeneous computing platforms having various security levels for various vehicles (such as autonomous vehicles, semi-autonomous vehicles, driver-operated vehicles, etc.). Various embodiments include methods, vehicles, vehicle management systems, and processing devices configured to implement methods for achieving algorithmic security for various algorithms on heterogeneous computing platforms having various security levels for a vehicle (such as autonomous vehicles, semi-autonomous vehicles, driver-operated vehicles, etc.).

[0036] Autonomous and semi-autonomous vehicles (such as cars, trucks, and tour buses) are becoming a reality on city streets. These vehicles typically include multiple sensors (including cameras, radar, and lidar) that collect information about their surroundings. For example, this information can enable vehicles to identify roads, mark objects to avoid, and track the movement and future positions of other vehicles, allowing for partial or full autonomous navigation.

[0037] Automotive System-on-Chip (SoC) can be used in safety-critical applications such as advanced driver assistance systems (ADAS) or autonomous driving systems, and typically includes multiple computing units such as multi-core central processing units (CPUs), graphics processing units (GPUs), digital signal processing units (DSPs), and neural processing units (NPUs). Some of these components are designed to meet safety standards, such as those defined in ISO 26262, the International Organization for Standardization (ISO) standard that defines Automotive Safety Integrity Levels (ASILs) for automobiles (such as ASIL A, ASIL B, ASIL C, ASIL D, etc.), while other components in the same heterogeneous computing system may not meet the safety standards (or not meet the same level of safety standards) due to cost, engineering, or other constraints. Various embodiments provide systems, methods, and apparatus for achieving algorithmic security for various algorithms on heterogeneous computing platforms with various safety levels.

[0038] Various embodiments may provide a method for supporting safety-compliant computing in heterogeneous computing systems, such as vehicle heterogeneous computing systems. A heterogeneous computing system may be a computing system comprising one or more computing units that may be safety-compliant (e.g., computing units conforming to vehicle safety integrity levels (e.g., Vehicle Safety Integrity Level B (ASIL B), ASIL C, etc.)) and one or more computing units that may not be safety-compliant (e.g., computing units that do not conform to vehicle safety integrity levels or have safety levels lower than ASIL B, etc.). In various embodiments, the heterogeneous computing system may be a System-on-a-Chip (SOC), such as a SOC in a vehicle. Various embodiments may include receiving an instruction to run an algorithm (such as a vehicle algorithm) requiring safety compliance (e.g., ASIL B compliance, ASIL C compliance, etc.) on the heterogeneous computing system (such as a vehicle heterogeneous computing system). Various embodiments may include determining whether a non-safety-compliant computing unit in the heterogeneous computing system (such as a vehicle heterogeneous computing system) is preferably used to run the algorithm (such as a vehicle algorithm). Determining that a non-safety-compliant computing unit is preferably used to run the algorithm (such as a vehicle algorithm) may be based on the nature of the algorithm. For example, an algorithm requiring a series of parallel operations may be more preferably run on a GPU. As another example, highly vectorized algorithms may be better suited to run on a DSP. The determination of computational units can be controlled by settings associated with the algorithm, and / or can be determined based on the state of computational units in the system (e.g., estimated latency), the properties of the dataset to be run on, or any other considerations during algorithm runtime.

[0039] Various embodiments may include modifying the execution of an algorithm (such as a transportation algorithm) in response to determining that a non-security-compliant computing unit of a heterogeneous computing system (such as a transportation heterogeneous computing system) is preferably used to run the algorithm (such as a transportation algorithm) to at least partially utilize the security-compliant computing unit of the heterogeneous computing system (such as a transportation heterogeneous computing system) to run the algorithm (such as a transportation algorithm). In some embodiments, modifying the execution of an algorithm (such as a transportation algorithm) in response to determining that a non-security-compliant computing unit of a heterogeneous computing system (such as a transportation heterogeneous computing system) is preferably used to run the algorithm (such as a transportation algorithm) to at least partially utilize the security-compliant computing unit of the transportation heterogeneous computing system to run the algorithm (such as a transportation algorithm) may include: modifying the execution of the algorithm (such as a transportation algorithm) in response to determining that a non-security-compliant computing unit of a heterogeneous computing system (such as a transportation heterogeneous computing system) is preferably used to run the algorithm (such as a transportation algorithm) to create a lightweight version of the algorithm (such as a lightweight version of a transportation algorithm) for at least partially running on the security-compliant computing unit of the heterogeneous computing system (such as a transportation heterogeneous computing system). A lightweight version of an algorithm (such as a lightweight version of a transportation algorithm) can be a version of the algorithm that requires fewer computational resources to execute compared to the full version of the algorithm (such as a transportation algorithm). For example, the full version of the algorithm (such as a full version of a transportation algorithm) can use a mesh or filter setting with fine granularity (or producing higher resolution), while a lightweight version of the algorithm (such as a lightweight version of a transportation algorithm) can use a mesh or filter setting with coarser granularity (or producing lower resolution).

[0040] In some embodiments, modifying the execution of an algorithm (such as a transportation algorithm) in response to determining that a non-safety compliance computing unit of a heterogeneous computing system (such as a transportation heterogeneous computing system) is preferably used to run the algorithm (such as a transportation algorithm) to at least partially utilize the safety compliance computing unit of the transportation heterogeneous computing system to run the algorithm (such as a transportation algorithm) may include: identifying key portions of the dataset. In various embodiments, key portions of the dataset may be portions of the dataset that are likely associated with safety, such as grid segments including pedestrians, data related to accident avoidance, etc. Various embodiments may include: running the algorithm (such as a transportation algorithm) on the safety compliance computing unit of the heterogeneous computing system (such as a transportation heterogeneous computing system) to generate more refined output for the identified key portions of the dataset, and running the algorithm (such as a transportation algorithm) on the non-safety compliance computing unit of the heterogeneous computing system (such as a transportation heterogeneous computing system) to generate more refined output for all other portions of the dataset.

[0041] Various embodiments may include: running the algorithm (such as the vehicle algorithm) on a non-security-compliant computing unit of a heterogeneous computing system (such as a vehicle heterogeneous computing system) to generate finer outputs for a dataset; running a lightweight version of the algorithm (such as a lightweight version of the vehicle algorithm) on a security-compliant computing unit of the heterogeneous computing system (such as a vehicle heterogeneous computing system) to generate coarse outputs for the dataset; determining whether the finer outputs and coarse outputs match; and generating an alert in response to determining that the finer outputs and coarse outputs do not match. In various embodiments, determining whether the finer outputs and coarse outputs match may include various operations for determining whether the parts match, such as comparing the parts, calculating the hash of the parts, etc.

[0042] Various embodiments may include: running a lightweight version of the algorithm (such as a lightweight version of the vehicle algorithm) on a security compliance computing unit of a heterogeneous computing system (such as a vehicle heterogeneous computing system) to generate a coarse output for a dataset; identifying portions of the coarse output as urgent parts; running the algorithm (such as the vehicle algorithm) at least twice on a non-security compliance computing unit for a portion of the dataset corresponding to the identified urgent parts to generate at least a first and a second finer output; determining whether the first and second finer outputs match; generating an alarm in response to determining that the first and second finer outputs do not match; and replacing the identified urgent parts of the coarse output with one of the first or second finer outputs in response to determining that the first and second finer outputs do match. In various embodiments, determining whether the first and second finer outputs match may include various operations for determining that the outputs match, such as comparing the outputs, calculating the hash of the outputs, etc.

[0043] Various embodiments may include: running an algorithm (such as a transportation algorithm) on a non-security-compliant computing unit of a heterogeneous computing system (such as a transportation heterogeneous computing system) to generate finer outputs for a dataset; running a lightweight version of the algorithm (such as a lightweight version of the transportation algorithm) on a security-compliant computing unit for a random sampled portion of the dataset to generate coarse outputs for that random sampled portion; determining whether the coarse outputs and finer outputs for the random sampled portion match; and generating an alert in response to determining that the coarse outputs and finer outputs for the random sampled portion do not match. In various embodiments, determining whether the coarse outputs and finer outputs for the random sampled portion match may include various operations for determining that the outputs match, such as comparing the outputs, calculating the hash of each output, etc.

[0044] Various embodiments may utilize redundant computational processes, where some or all of the algorithm may be computed two or more times by the same computational unit or by different computational units to determine whether the output matches. Matching outputs can verify these outputs, while non-matching outputs can indicate that an error may have occurred.

[0045] Sensor fusion meshes are an example automotive algorithm suitable for various embodiments. Sensor fusion is an automotive algorithm that estimates various attributes of each cell of a spatial mesh surrounding an autonomous vehicle or ego vehicle. These estimated attributes can be occupancy, drivability, visibility, semantic category, etc. This can be achieved by combining information from various sensors. An example algorithm for estimating visibility attributes in a sensor fusion mesh can illustrate various aspects of various embodiments. For example, a visibility mesh is a mesh describing whether a given location on a high-resolution map is visible to any sensors available on the car. The mesh cells are defined by coordinates on the high-resolution map. An example algorithm for calculating a visibility mesh can be explained by the following operations, but different variations of the operation of the algorithm can be used in various aspects. In a first operation, a mesh is generated using points within a given radius, which can be from a high-resolution map or an online-generated map. In a second operation, a dynamic list of objects is obtained from the sensor fusion pipeline. In a third operation, these points are then filtered based on whether they are within the sensor's field of view. In a fourth operation, a ray is then traced from that point to the sensor currently being considered. In the fifth operation, it is then checked whether the ray intersects with any dynamic objects on the road. In the sixth operation, if the ray does not intersect with any dynamic object, or if the point is within a dynamic object, then the point is considered visible to the sensor and therefore visible to the sensor fusion pipeline. In the seventh operation, if the ray intersects with a dynamic object, then the point is considered occluded within the sensor's field of view and therefore invisible to the sensor fusion pipeline. The algorithm can also include tracking different dynamic objects in the scene, such as pedestrians, cars, cyclists, motorcycles, trucks, animals crossing the road, objects flying out of cars, etc.

[0046] As an example, the occlusion mesh algorithm can be an automotive algorithm suitable for use with various embodiments of methods for supporting safety-compliant computing in heterogeneous computing systems for vehicles, and various operations of various embodiments can be explained. However, the algorithmic operations discussed with reference to the occlusion mesh algorithm and sensor fusion are merely provided as examples, and various embodiments can be adapted for use with other algorithms, such as mesh fusion algorithms, motion planning algorithms, Monte Carlo sampling algorithms, etc. Turning to the example of the occlusion mesh algorithm, since the ray drawing operation is likely critical to the algorithm, which is a very GPU-friendly operation, running the occlusion algorithm on a GPU can be the most efficient. However, if the GPU does not meet the Automotive Safety Integrity Level (ASIL) requirements (e.g., the GPU does not meet ASIL-B safety requirements), the occlusion mesh algorithm may not be deployed on a GPU, as the occlusion mesh algorithm may require deployment on hardware that meets ASIL requirements (e.g., meets ASIL B safety requirements). To circumvent this problem, various embodiments of methods for achieving algorithm safety can be used during the runtime of the occlusion mesh algorithm. In the following examples, the CPU or DSP can be a security-compliant computing unit (e.g., an ASIL B compliant computing unit), while the GPU can be a non-security-compliant computing unit (e.g., a computing unit that is not ASIL B compliant).

[0047] As an example, coarse and fine mesh methods can be used in various embodiments. Since the CPU or DSP can be an ASIL-B compliant component, a coarse mesh (e.g., a mesh with 5m by 5m grid segments) can be run on the CPU or DSP, while a fine mesh (e.g., a mesh with 0.5m by 0.5m grid segments) can be run on the GPU. The coarse mesh can be computationally cheaper and can indicate whether larger cells have any occlusion (invisibility). In cases where occlusion is observed in the region of interest, the system can obtain finer results from the GPU. If the finer and coarser results are inconsistent (e.g., neither observes occlusion, etc.), the system can issue a fault alarm to notify higher layers to take safety actions (e.g., warn the driver, perform evasive maneuvers, etc.).

[0048] As another example, a selective meshing method can be used in various embodiments. A coarse mesh (e.g., a mesh with 5m by 5m grid segments) can be run on a CPU or DSP. The system can identify mesh tiles that urgently need to be understood with finer detail (e.g., mesh tiles containing pedestrians or small objects). The system can acquire the important tiles and run them twice on a GPU using a fine mesh (e.g., a mesh with 0.5m by 0.5m grid segments). The system can check if these two runs on the GPU match. If the two runs on the GPU match, the fine mesh output for these tiles can be used in subsequent steps. If the two runs on the GPU do not match, an alert can be issued.

[0049] As another example, random sampling methods can be used in various embodiments. In random sampling, the full algorithm can be deployed on the GPU at a finer scale to compute visibility values ​​for all units. Some of these units can be randomly sampled to recompute their properties on a CPU or DSP. The random sampled output computed on the ASIL-compliant safety computing unit (such as a CPU or DSP) can be verified against the values ​​generated from the GPU. If these values ​​do not match, the system can issue a fault alarm to notify higher layers to take safety actions (warn the driver, take evasive maneuvers, etc.).

[0050] As another example, preference-based computation methods can be used in various embodiments. Mesh cells identified as more important can be computed on ASIL-B compliant computational units, while less important / critical mesh cells can be computed on non-ASIL-B computational units. As a specific example, during lane change planning, the target lane mesh cell is more important and can be computed on an ASIL-B compliant computational unit (e.g., a CPU). Other mesh cells can be computed on a non-ASIL-B compliant computational unit (e.g., a GPU).

[0051] As another example, adaptive grid cell sizes can be used in various embodiments. For instance, as the longitudinal distance between the grid cells and the autonomous vehicle increases, the grid cell size can be increased based on that distance to reduce the load on the ASIL-B computing cells.

[0052] This document discusses various examples with reference to vehicles, heterogeneous computing systems for vehicles, and vehicle algorithms to better illustrate various aspects of the embodiments. However, the discussion of vehicles, heterogeneous computing systems for vehicles, and vehicle algorithms is merely illustrative and is not intended to limit the scope of this disclosure or the claims. In the various examples, other devices, other heterogeneous computing systems, and / or other algorithms may replace vehicles, heterogeneous computing systems for vehicles, and vehicle algorithms.

[0053] Various embodiments can be implemented in various means of transportation, with example vehicle 100 in... Figure 1A and 1B Chinese explanation. Reference. Figure 1A and 1B The vehicle 100 may include a control unit 140 and multiple sensors 102-138, including a satellite geolocation system receiver 108, occupancy sensors 112, 116, 118, 126, 128, tire pressure sensors 114, 120, cameras 122, 136, microphones 124, 134, an impact sensor 130, radar 132, and lidar 138. The multiple sensors 102-138 arranged in or on the vehicle can be used for various purposes, such as autonomous and semi-autonomous navigation and control, collision avoidance, location determination, etc., and to provide sensor data about objects and people in or on the vehicle 100. Sensors 102-138 may include one or more of a wide variety of sensors capable of detecting various information useful for navigation and collision avoidance. Each of the sensors 102-138 may be in wired or wireless communication with the control unit 140 and with each other. Specifically, the sensors may include one or more cameras 122, 136, or other optical or photoelectric sensors. The sensors may further include other types of object detection and ranging sensors, such as radar 132, lidar 138, IR sensors, and ultrasonic sensors. The sensors may further include tire pressure sensors 114 and 120, humidity sensors, temperature sensors, satellite geolocation sensors 108, accelerometers, vibration sensors, gyroscopes, gravimeters, impact sensors 130, force gauges, pressure gauges, strain sensors, fluid sensors, chemical sensors, gas content analyzers, pH sensors, radiation sensors, Geiger counters, neutron detectors, biomaterial sensors, microphones 124 and 134, occupancy sensors 112, 116, 118, 126, and 128, proximity sensors, and other sensors.

[0054] The vehicle control unit 140 may be configured with processor-executable instructions to perform various embodiments using information received from various sensors, particularly cameras 122 and 136. In some embodiments, the control unit 140 may supplement the processing of camera images with distance and relative position (e.g., relative bearing angle) obtainable from radar 132 and / or lidar 138 sensors. The control unit 140 may be further configured to control the steering, braking, and speed of the vehicle 100 using information about other vehicles determined using various embodiments when operating in autonomous or semi-autonomous mode.

[0055] Figure 1C This is a component block diagram illustrating system 150, which is suitable for implementing various embodiments and supporting systems. (Refer to...) Figure 1A ,1B And 1C, the vehicle 100 may include a control unit 140, which may include various circuits and devices for controlling the operation of the vehicle 100. In Figure 1C In the illustrated example, control unit 140 includes processor 164, memory 166, input module 168, output module 170, and radio module 172. Control unit 140 may be coupled to and configured to control driving control component 154, navigation component 156, and one or more sensors 158 of vehicle 100.

[0056] As used herein, the terms “component,” “system,” “unit,” “module,” and similar terms include computer-related entities such as, but not limited to, hardware, firmware, combinations of hardware and software, software, or software in execution configured to perform a particular operation or function. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. By extension, both an application running on a communication device and the communication device itself can be referred to as a component. One or more components may reside within a process and / or a thread of execution, and components may be localized on a single processor or core and / or distributed across two or more processors or cores. Furthermore, these components may execute from various non-transitory computer-readable media on which various instructions and / or data structures are stored. Components may communicate via local and / or remote processes, function or procedure calls, electronic signals, data packets, memory read / write, and other known computer, processor, and / or process-related communication methods.

[0057] Control unit 140 may include processor 164, which may be configured with processor-executable instructions to control the maneuvering, navigation, and / or other operations of vehicle 100, including operations according to various embodiments. Processor 164 may be coupled to memory 166. Control unit 162 may include input module 168, output module 170, and radio module 172.

[0058] Radio module 172 can be configured for wireless communication. Radio module 172 can exchange signals 182 (e.g., command signals for controlling maneuvering, signals from navigation facilities, etc.) with network transceiver 180, and can provide signals 182 to processor 164 and / or navigation unit 156. In some embodiments, radio module 172 can enable vehicle 100 to communicate with wireless communication device 190 via wireless communication link 192. Wireless communication link 192 can be a bidirectional or unidirectional communication link, and can use one or more communication protocols.

[0059] The input module 168 can receive sensor data from one or more vehicle sensors 158 and electronic signals from other components, including the driving control component 154 and the navigation component 156. The output module 170 can be used to communicate with or activate various components of the vehicle 100, including the driving control component 154, the navigation component 156, and the sensors 158.

[0060] Control unit 140 may be coupled to driving control component 154 to control physical elements of vehicle 100 related to the maneuvering and navigation of the vehicle, such as engine, motor, throttle, steering elements, flight control elements, braking or deceleration elements, etc. Driving control component 154 may also include components for controlling other devices of the vehicle, including environmental controls (e.g., air conditioning and heating), external and / or interior lighting, internal and / or external information displays (which may include displays or other devices for displaying information), safety devices (e.g., haptic devices, auditory alarms, etc.), and other similar devices.

[0061] Control unit 140 may be coupled to navigation component 156 and may receive data from navigation component 156 and be configured to use such data to determine the current position and orientation of vehicle 100, and the appropriate route toward its destination. In various embodiments, navigation component 156 may include or be coupled to a Global Navigation Satellite System (GNSS) receiver system (e.g., one or more Global Positioning System (GPS) receivers) enabling vehicle 100 to use GNSS signals to determine its current position. Alternatively or additionally, navigation component 156 may include a radio navigation receiver for receiving navigation beacons or other signals from radio nodes (such as Wi-Fi access points, cellular network sites, radio stations, telecomputing devices, other vehicles, etc.). Through the control of driving control element 154, processor 164 may control vehicle 100 for navigation and maneuvering. The processor 164 and / or navigation component 156 may be configured to communicate with a server 184 on a network 186 (e.g., the Internet) via a wireless connection 182 to the cellular data network 180 to receive commands for controlling maneuvers, receive data useful in navigation, provide real-time position reports, and evaluate other data.

[0062] The control unit 162 may be coupled to one or more sensors 158. The sensors 158 may include sensors 102-138 as described, and may be configured to provide various data to the processor 164.

[0063] Although control unit 140 is described as comprising separate components, in some embodiments, some or all of the components (e.g., processor 164, memory 166, input module 168, output module 170, and radio module 172) may be integrated into a single device or module, such as a system-on-a-chip (SOC) processing device. Such an SOC processing device may be configured for use in a vehicle and configured (e.g., configured with processor-executable instructions that execute in processor 164) to perform operations according to various embodiments when installed in a vehicle.

[0064] Figure 2A Examples of subsystems, computing elements, computing devices, or units within the transportation management system 200 that can be used within the transportation vehicle 100 are explained. (See reference...) Figures 1A-2A In some embodiments, various computing elements, computing devices, or units within the vehicle management system 200 may be implemented within a system of interconnected computing devices (i.e., subsystems) that communicate data and commands to each other (e.g., by...). Figure 2A (As indicated by the arrow in the diagram). In other embodiments, various computing elements, computing devices, or units within the transportation management system 200 can be implemented within a single computing device, such as a separate thread, process, algorithm, or computing element. Therefore, Figure 2A Each subsystem / computing element described herein is also generally referred to as a "layer" within the computational "stack" constituting the vehicle management system 200. However, the use of the terms "layer" and "stack" in describing various embodiments is not intended to imply or require that the corresponding functionality be implemented within a single autonomous (or semi-autonomous) vehicle management system computing device, although this is a potential implementation embodiment. Rather, the use of the term "layer" is intended to encompass subsystems with independent processors, computational elements (e.g., threads, algorithms, subroutines, etc.) running in one or more computing devices, and combinations of subsystems and computational elements.

[0065] In various embodiments, the transportation management system stack 200 may include a radar perception layer 202, a camera perception layer 204, a localization engine layer 206, a map fusion and arbitration layer 208, a route planning layer 210, a sensor fusion and road world model (RWM) management layer 212, a motion planning and control layer 214, and a behavior planning and prediction layer 216. Layers 202-216 are merely examples of some layers in one example configuration of the transportation management system stack 200. In other configurations consistent with the various embodiments, additional layers may be included, such as additional layers for other perception sensors (e.g., a lidar perception layer, etc.), additional layers for planning and / or control, additional layers for modeling, etc., and / or some layers of layers 202-216 may be excluded from the transportation management system stack 200. Figure 2AAs indicated by the arrows, each of layers 202-216 can exchange data, calculation results, and commands. Furthermore, the vehicle management system stack 200 can receive and process data from sensors (e.g., radar, lidar, cameras, inertial measurement units (IMUs), etc.), navigation systems (e.g., GPS receivers, IMUs, etc.), vehicle networks (e.g., Controller Area Network (CAN) buses), and databases in memory (e.g., digital map data). The vehicle management system stack 200 can output vehicle control commands or signals to the drive-by-wire (DBW) system / control unit 220, which is a system, subsystem, or computing device that directly interfaces with the vehicle's steering, throttle, and braking controls. Figure 2A The configuration of the vehicle management system stack 200 and DBW system / control unit 220 described herein is merely an example configuration, and other configurations of the vehicle management system and other vehicle components can be used in various embodiments. As an example, Figure 2A The configurations of the vehicle management system stack 200 and DBW system / control unit 220 described herein can be used in vehicles configured for autonomous or semi-autonomous operation, while different configurations can be used in non-autonomous vehicles.

[0066] The radar perception layer 202 may receive data from one or more detection and ranging sensors (such as radar (e.g., 132) and / or lidar (e.g., 138)) and process that data to identify and determine the location of other vehicles and objects in the vicinity of vehicle 100. The radar perception layer 202 may include the use of neural network processing and artificial intelligence methods to identify objects and vehicles and pass such information to the sensor fusion and RWM management layer 212.

[0067] The camera perception layer 204 may receive data from one or more cameras (such as cameras (e.g., 122, 136)) and process that data to identify and determine the location of other vehicles and objects in the vicinity of vehicle 100. The camera perception layer 204 may include the use of neural network processing and artificial intelligence methods to identify objects and vehicles and pass such information to the sensor fusion and RWM management layer 212.

[0068] The positioning engine layer 206 can receive and process data from various sensors to determine the location of the vehicle 100. These various sensors may include, but are not limited to, GPS sensors, IMUs, and / or other sensors connected via a CAN bus. The positioning engine layer 206 can also utilize input from one or more cameras (such as cameras (e.g., 122, 136)) and / or any other available sensors (such as radar, lidar, etc.).

[0069] Map fusion and arbitration layer 208 can access data within a high-definition (HD) map database and receive output from positioning engine layer 206, processing the data to further determine the location of vehicle 100 within the map, such as its location within a traffic lane, its location within a street map, etc. The HD map database can be stored in memory (e.g., memory 166). For example, map fusion and arbitration layer 208 can convert latitude and longitude information from GPS into a location within a ground road map contained in the HD map database. GPS location locking includes errors, so map fusion and arbitration layer 208 can act as an arbitrator between GPS coordinates and HD map data to determine the best guessed location of the vehicle within the road. For example, while GPS coordinates might place the vehicle near the middle of a two-lane road in the HD map, map fusion and arbitration layer 208 can determine, based on the direction of travel, that the vehicle is most likely aligned with the lane in the same direction of travel. Map fusion and arbitration layer 208 can then pass the map-based location information to sensor fusion and RWM management layer 212.

[0070] Route planning layer 210 can utilize HD maps and input from operators or dispatchers to plan routes for vehicle 100 to a specific destination. Route planning layer 210 can pass map-based location information to sensor fusion and RWM management layer 212. However, the use of prior maps is not required for other layers (such as sensor fusion and RWM management layer 212). For example, other stacks can operate and / or control vehicles based solely on perception data without providing maps, constructing concepts of lanes, boundaries, and local maps as perception data is received.

[0071] The sensor fusion and RWM management layer 212 can receive data and outputs generated by the radar perception layer 202, camera perception layer 204, map fusion and arbitration layer 208, and route planning layer 210, and use some or all of these inputs to estimate or refine the position and status of the vehicle 100 relative to the road, other vehicles on the road, and other objects near the vehicle 100. For example, the sensor fusion and RWM management layer 212 can combine image data from the camera perception layer 204 with arbitration map position information from the map fusion and arbitration layer 208 to refine the determined position of the vehicle within a traffic lane. As another example, the sensor fusion and RWM management layer 212 can combine object recognition and image data from the camera perception layer 204 with object detection and ranging data from the radar perception layer 202 to determine and refine the relative positions of other vehicles and objects near the vehicle. As another example, the sensor fusion and RWM management layer 212 can receive information about the position and direction of travel of other vehicles from vehicle-to-vehicle (V2V) communication (such as via a CAN bus) and combine this information with information from the radar perception layer 202 and the camera perception layer 204 to refine the position and motion of other vehicles. The sensor fusion and RWM management layer 212 can output the refined position and status information of vehicle 100, as well as the refined position and status information of other vehicles and objects in the vicinity of vehicle 100, to the motion planning and control layer 214 and / or the behavior planning and prediction layer 216.

[0072] As a further example, the sensor fusion and RWM management layer 212 can use dynamic traffic control commands to instruct vehicle 100 to change speed, lane, direction of travel, or other navigation elements, and combine this information with other received information to determine refined position and status information. The sensor fusion and RWM management layer 212 can output the refined position and status information of vehicle 100, as well as the refined position and status information of other vehicles and objects near vehicle 100, via wireless communication (such as via C-V2X connection, other wireless connections, etc.) to the motion planning and control layer 214, the behavior planning and prediction layer 216, and / or devices remote from vehicle 100, such as data servers, other vehicles, etc.

[0073] As a further example, the sensor fusion and RWM management layer 212 can monitor perception data from various sensors (such as perception data from radar perception layer 202, camera perception layer 204, other perception layers, etc.) and / or data from one or more sensors themselves to analyze the conditions in the vehicle sensor data. The sensor fusion and RWM management layer 212 can be configured to detect conditions in the sensor data, such as sensor measurements being at, above, or below thresholds, the occurrence of certain types of sensor measurements, etc., and can output the sensor data as part of refined location and status information of the vehicle 100 provided to the behavior planning and prediction layer 216 and / or devices remotely located from the vehicle 100 (such as data servers, other vehicles, etc.) via wireless communication (such as via C-V2X connections, other wireless connections, etc.).

[0074] The refined location and status information may include vehicle descriptors associated with the vehicle and its owner and / or operator, such as: vehicle specifications (e.g., size, weight, color, onboard sensor type, etc.); vehicle location, speed, acceleration, direction of travel, attitude, orientation, destination, fuel / power level, and other status information; vehicle emergency status (e.g., whether the vehicle is an emergency vehicle or a private vehicle); vehicle restrictions (e.g., heavy / wide load, turning restrictions, high-occupancy vehicle (HOV) authorization, etc.); vehicle capabilities (e.g., all-wheel drive, four-wheel drive, snow tires, chains, supported connection types, onboard sensor operating status, onboard sensor resolution level, etc.); equipment issues (e.g., low tire pressure, weak braking, sensor malfunction, etc.); owner / operator driving preferences (e.g., preferred lanes, roads, routes, and / or destinations, preference for avoiding toll booths or highways, preference for the fastest route, etc.); permission to provide sensor data to a data broker server (e.g., 184); and / or owner / operator identification information.

[0075] The behavior planning and prediction layer 216 of the autonomous vehicle system stack 200 can use refined position and state information of vehicle 100, as well as position and state information of other vehicles and objects output from sensor fusion and RWM management layer 212, to predict the future behavior of other vehicles and / or objects. For example, the behavior planning and prediction layer 216 can use such information to predict the future relative position of other vehicles based on its own vehicle position and speed, as well as the positions and speeds of other vehicles in the vicinity. Such predictions can take into account information from high-definition maps and route planning to predict changes in the relative vehicle position as the primary vehicle and other vehicles travel along the road. The behavior planning and prediction layer 216 can output the behavior and position predictions of other vehicles and objects to the motion planning and control layer 214. Furthermore, the behavior planning and prediction layer 216 can use object behavior combined with position predictions to plan and generate control signals for controlling the motion of vehicle 100. For example, based on route planning information, refined location data in road information, and the relative positions and movements of other vehicles, the behavior planning and prediction layer 216 can determine that vehicle 100 needs to change lanes and accelerate, such as to maintain or achieve a minimum distance from other vehicles, and / or prepare for a turn or exit. As a result, the behavior planning and prediction layer 216 can calculate or otherwise determine the wheel steering angle and throttle settings to be commanded to the motion planning and control layer 214 and the DBW system / control unit 220, along with various parameters required to implement such lane changes and accelerations. One such parameter could be the calculated steering wheel command angle.

[0076] The motion planning and control layer 214 can receive data and information output from the sensor fusion and RWM management layer 212, as well as behavior and position predictions of other vehicles and objects from the behavior planning and prediction layer 216. It uses this information to plan and generate control signals for controlling the motion of the vehicle 100, and to verify that such control signals meet the safety requirements of the vehicle 100. For example, based on route planning information, refined positions in road information, and the relative positions and movements of other vehicles, the motion planning and control layer 214 can verify various control commands or instructions and transmit them to the DBW system / control unit 220.

[0077] The DBW system / control unit 220 can receive commands or instructions from the motion planning and control layer 214 and translate such information into mechanical control signals for controlling the wheel angles, brakes, and throttle of the vehicle 100. For example, the DBW system / control unit 220 can respond to a calculated steering wheel command angle by sending corresponding control signals to the steering wheel controller.

[0078] In various embodiments, the vehicle management system stack 200 may include functionality to perform safety checks or oversight of various commands, plans, or other decisions at various layers that may affect the safety of vehicles and occupants. Such safety check or oversight functionality may be implemented within a dedicated layer or distributed across layers and included as part of that functionality. In some embodiments, various safety parameters may be stored in memory, and the safety check or oversight functionality may compare determined values ​​(e.g., relative distance to nearby vehicles, distance to the road centerline, etc.) with the corresponding safety parameters and issue warnings or commands if the safety parameters are violated or will be violated. For example, safety or supervisory functions in the behavior planning and prediction layer 216 (or a separate layer) can determine the current or future separation distance between this vehicle and another vehicle (e.g., based on a world model refined by the sensor fusion and RWM management layer 212), compare this separation distance with a safe separation distance parameter stored in memory, and issue instructions to the motion planning and control layer 214 to accelerate, decelerate, or turn if the current or predicted separation distance violates the safe separation distance parameter. As another example, safety or supervisory functions in the motion planning and control layer 214 (or a separate layer) can compare the determined or commanded steering wheel command angle with a safe wheel angle limit or parameter, and issue an override command and / or alarm in response to the commanded angle exceeding the safe wheel angle limit.

[0079] Some safety parameters stored in memory can be static (i.e., do not change over time), such as the maximum vehicle speed. Other safety parameters stored in memory can be dynamic, as these parameters are continuously or periodically determined or updated based on vehicle state information and / or environmental conditions. Non-limiting examples of safety parameters include maximum safe speed, maximum braking pressure, maximum acceleration, and safe wheel angle limits, all of which can be functions of road and weather conditions.

[0080] Figure 2B Examples of subsystems, computing elements, computing devices, or units within the transportation management system 250 that can be used within the transportation vehicle 100 are explained. (Reference) Figures 1A-2B In some embodiments, layers 202, 204, 206, 208, 210, 212, and 216 of the vehicle management system stack 200 can be similar to those in reference 1. Figure 2AThe vehicle management system stack 250 described herein can operate similarly to the vehicle management system stack 200, except that the vehicle management system stack 250 can pass various data or instructions to the vehicle safety and collision avoidance system 252 instead of the DBW system / control unit 220. For example, Figure 2B The configuration of the vehicle management system stack 250 and the vehicle safety and collision avoidance system 252 described herein can be used in non-autonomous vehicles.

[0081] In various embodiments, the behavior planning and prediction layer 216 and / or the sensor fusion and RWM management layer 212 can output data to the vehicle safety and collision avoidance system 252. For example, the sensor fusion and RWM management layer 212 can output sensor data as part of refined position and state information of the vehicle 100 provided to the vehicle safety and collision avoidance system 252. The vehicle safety and collision avoidance system 252 can use the refined position and state information of the vehicle 100 to make safety determinations regarding the vehicle 100 and / or its occupants. As another example, the behavior planning and prediction layer 216 can output behavior models and / or predictions related to the motion of other vehicles to the vehicle safety and collision avoidance system 252. The vehicle safety and collision avoidance system 252 can use behavior models and / or predictions related to the motion of other vehicles to make safety determinations regarding the vehicle 100 and / or its occupants.

[0082] In various embodiments, the vehicle safety and collision avoidance system 252 may include the functionality to perform safety checks or supervision on various commands, planning, or other decisions, as well as human driver actions, at various layers that may affect the safety of the vehicle and its occupants. In some embodiments, various safety parameters may be stored in memory, and the vehicle safety and collision avoidance system 252 may compare determined values ​​(e.g., relative distance to nearby vehicles, distance to the road centerline, etc.) with the corresponding safety parameters and issue warnings or commands if the safety parameters are violated or will be violated. For example, the vehicle safety and collision avoidance system 252 may determine the current or future separation distance between itself and another vehicle (e.g., based on a world model refined by sensor fusion and RWM management layer 212) (e.g., based on a world model refined by sensor fusion and RWM management layer 212), compare this separation distance with a safe separation distance parameter stored in memory, and issue instructions to the driver to accelerate, decelerate, or turn if the current or predicted separation distance violates the safe separation distance parameter. As another example, the vehicle safety and collision avoidance system 252 can compare changes in the steering wheel angle by a human driver with safe wheel angle limits or parameters, and issue an overdrive command and / or alarm in response to the steering wheel angle exceeding the safe wheel angle limits.

[0083] Figure 3 An example system-on-a-chip (SoC) architecture suitable for implementing various embodiments of a processing device SOC 300 in a vehicle is described. (Refer to...) Figures 1A-3 The processing device SOC 300 may include several heterogeneous processors, such as a digital signal processor (DSP) 303, a modem processor 304, an image and object recognition processor 306, a mobile display processor 307, an application processor 308, and a resource and power management (RPM) processor 317. The processing device SOC 300 may also include one or more coprocessors 310 (e.g., vector coprocessors) connected to one or more of the heterogeneous processors 303, 304, 306, 307, 308, and 317. Each processor may include one or more cores and an independent / internal clock. Each processor / core may perform operations independently of the other processors / cores. For example, the processing device SOC 300 may include a processor running a first type of operating system (e.g., FreeBSD, LINUX, OS X, etc.) and a processor running a second type of operating system (e.g., Microsoft Windows). In some embodiments, the application processor 308 may be the main processor of the SOC 300, a central processing unit (CPU), a microprocessor unit (MPU), a neural processing unit (NPU), an arithmetic logic unit (ALU), etc. The graphics processor 306 may be a graphics processing unit (GPU).

[0084] The processing device SOC 300 may include analog circuitry and custom circuitry 314 for managing sensor data, analog-to-digital conversion, wireless data transmission, and for performing other specialized operations, such as processing encoded audio and video signals for rendering in a web browser. The processing device SOC 300 may further include system components and resources 316, such as voltage regulators, oscillators, phase-locked loops, peripheral bridges, data controllers, memory controllers, system controllers, access ports, timers, and other similar components for supporting processors and software clients (e.g., web browsers) running on computing devices.

[0085] The processing device SOC 300 also includes a dedicated circuitry for camera actuation and management (CAM) 305, which includes, provides, controls, and / or manages the operation of one or more cameras 122, 136 (e.g., a main camera, webcam, 3D camera, etc.), video display data from camera firmware, image processing, video preprocessing, video front-end (VFE), embedded JPEG, high-definition video codecs, etc. CAM 305 may be a separate processing unit and / or include a separate or internal clock.

[0086] In some embodiments, the image and object recognition processor 306 may be configured with processor-executable instructions and / or specialized hardware to perform image processing and object recognition analysis as described in various embodiments. For example, the image and object recognition processor 306 may be configured to process images received from cameras (e.g., 122, 136) via CAM 305 to identify and / or characterize the operation of other vehicles, and otherwise perform the functions of the camera perception layer 204 as described. In some embodiments, the processor 306 may be configured to process radar or lidar data and perform the functions of the radar perception layer 202 as described.

[0087] System components and resources 316, analog and custom circuit systems 314, and / or CAM 305 may include circuit systems that interface with peripheral devices such as cameras 122, 136, radar 132, lidar 138, electronic displays, wireless communication devices, external memory chips, etc. Processors 303, 304, 306, 307, and 308 may be interconnected to one or more memory elements 312, system components and resources 316, analog and custom circuit systems 314, CAM 305, and RPM processor 317 via interconnect / bus module 324. Interconnect / bus module 324 may include reconfigurable gate arrays and / or implement bus architectures (e.g., CoreConnect, AMBA, etc.). Communication may be provided by advanced interconnects such as high-performance on-chip networks (NoC).

[0088] The processing device SOC 300 may further include input / output modules (not described) for communicating with external resources such as clock 318 and voltage regulator 320. External resources (e.g., clock 318, voltage regulator 320) may be shared by two or more internal SOC processors / cores (e.g., DSP 303, modem processor 304, graphics processor 306, application processor 308, etc.).

[0089] In some embodiments, the processing device SOC 300 may be included in a control unit (e.g., 140) for use in a vehicle (e.g., 100). The control unit may include communication links for communicating with a telephone network (e.g., 180), the Internet, and / or a web server (e.g., 184), as described.

[0090] The processing device SOC 300 may also include additional hardware and / or software components suitable for collecting sensor data from sensors, including motion sensors (e.g., accelerometers and gyroscopes of an IMU), user interface elements (e.g., input buttons, touchscreen displays, etc.), microphone arrays, sensors for monitoring physical conditions (e.g., position, orientation, motion, orientation, vibration, pressure, etc.), cameras, compasses, GPS receivers, and communication circuitry systems (e.g., ...). WLAN, WiFi, and other well-known components of modern electronic devices.

[0091] Figure 4 A component block diagram of a system 400 configured to support safety-compliant computing in a heterogeneous computing system for vehicles is shown, according to various embodiments. In some embodiments, system 400 may include one or more vehicle computing devices 402 and / or one or more remote platforms 404. (Refer to...) Figures 1A-4 The vehicle computing device 402 may include a processor (e.g., 164), a processing device (e.g., 300), and / or a control unit (e.g., 104) of the vehicle (e.g., 100) (referred to as a "processor"). The remote platform 404 may include a processor (e.g., 164), a processing device (e.g., 300), a control unit (e.g., 104) of the vehicle (e.g., 100) (referred to as a "processor"), a server (e.g., 184), and / or a wireless communication device (e.g., 190) that the vehicle computing device 402 can be connected to via a network 405 (e.g., 186) or other communication link.

[0092] The vehicle computing device 402 can be configured by machine-executable instructions 406. Machine-executable instructions 406 may include one or more instruction modules. Instruction modules may include computer program modules. Instruction modules may include one or more of the following: instruction receiving module 408, computing unit determining module 410, execution modification module 412, vehicle algorithm running module 414, version running module 416, output determining module 418, alarm generating module 420, partial identification module 422, output substitution module 424, and / or other instruction modules.

[0093] The instruction receiving module 408 can be configured to receive an instruction to run a vehicle algorithm requiring safety compliance in a heterogeneous computing system for vehicles. Safety compliance can be a requirement for the algorithm to run on a safety-compliant device, such as an ASIL B compliant device.

[0094] The computing unit determination module 410 can be configured to determine whether a non-safety compliant computing unit of a heterogeneous computing system for vehicles is preferably used to run vehicle algorithms.

[0095] The execution modification module 412 can be configured to modify the execution of the transportation algorithm in response to determining that a non-safety compliance computing unit of the heterogeneous transportation computing system is preferably used to run the transportation algorithm, to execute a portion of the transportation algorithm using the non-safety compliance computing unit of the heterogeneous transportation computing system, and to execute another portion of the transportation algorithm using a safety compliance computing unit of the heterogeneous transportation computing system. The safety compliance computing unit can be a central processing unit or a digital signal processing unit, while the non-safety compliance computing unit is a graphics processing unit. The execution modification module 412 can be configured to modify the execution of the transportation algorithm to create a lightweight version of the transportation algorithm for running on the safety compliance computing unit of the heterogeneous transportation computing system. The execution modification module 412 can be configured to modify the execution of the transportation algorithm to create a lightweight version of the transportation algorithm in response to determining that a non-safety compliance computing unit of the heterogeneous transportation computing system is preferably used to run the transportation algorithm.

[0096] The vehicle algorithm execution module 414 can be configured to run the vehicle algorithm on a non-safety compliance computing unit of a heterogeneous vehicle computing system to generate more refined output for the dataset. The vehicle algorithm execution module 414 can be configured to run the vehicle algorithm at least twice on the non-safety compliance computing unit for a portion of the dataset corresponding to the identified emergency section to generate at least a first and a second more refined output. The vehicle algorithm execution module 414 can be configured to run the vehicle algorithm on a non-safety compliance computing unit of a heterogeneous vehicle computing system to generate more refined output for the dataset. The vehicle algorithm execution module 414 can be configured to run the vehicle algorithm on a safety compliance computing unit of a heterogeneous vehicle computing system to generate more refined output for the identified critical sections of the dataset. The vehicle algorithm execution module 414 can be configured to run the vehicle algorithm on a non-safety compliance computing unit of a heterogeneous vehicle computing system to generate more refined output for all other sections of the dataset. The vehicle algorithm execution module 414 can be configured to run the vehicle algorithm on a safety compliance computing unit of a heterogeneous vehicle computing system to generate more refined output for the identified important sections of the dataset.

[0097] Version execution module 416 can be configured to run a lightweight version of the vehicle algorithm on the safety compliance computing unit of a vehicle heterogeneous computing system to generate coarse output for the dataset. Version execution module 416 can be configured to run a lightweight version of the vehicle algorithm on the safety compliance computing unit of a vehicle heterogeneous computing system to generate coarse output for the dataset. Version execution module 416 can be configured to run a lightweight version of the vehicle algorithm on a randomly sampled portion of the dataset on the safety compliance computing unit to generate coarse output for that randomly sampled portion of the dataset. Version execution module 416 can be configured to run a lightweight version of the vehicle algorithm on the safety compliance computing unit of a vehicle heterogeneous computing system to generate coarse output for all other portions of the dataset.

[0098] The output determination module 418 can be configured to determine whether a finer output and a coarse output match. The output determination module 418 can be configured to determine whether a first finer output and a second finer output match. The output determination module 418 can be configured to determine whether a coarse output and a finer output match for a random sampling portion.

[0099] Alarm generation module 420 can be configured to generate an alarm in response to determining a mismatch between finer and coarse outputs. Alarm generation module 420 can also be configured to generate an alarm in response to determining a mismatch between a first finer output and a second finer output. Furthermore, alarm generation module 420 can be configured to generate an alarm in response to determining a mismatch between coarse and finer outputs for a random sampling portion.

[0100] Partial identification module 422 can be configured to identify portions of the coarse output that are considered urgent. Partial identification module 422 can be configured to identify critical portions of the dataset. Partial identification module 422 can be configured to identify important portions of the dataset.

[0101] The output replacement module 424 can be configured to replace the identified urgent portion of the coarse output with one of the first finer output or the second finer output in response to determining that the first finer output and the second finer output do indeed match.

[0102] In some embodiments, the heterogeneous computing system of a vehicle may be a System-on-a-Chip (SOC). In some embodiments, the required security compliance may be ASIL B compliance.

[0103] Figure 5 A method 500 for supporting safety-compliant computing in a heterogeneous computing system for transportation vehicles, according to various embodiments, is described. (See also...) Figures 1A-5 Method 500 may be implemented in a processor (e.g., 164), a processing device (e.g., 300), and / or a control unit (e.g., 104) of a vehicle (e.g., 100) (referred to differently as a “processor”) or a vehicle computing device 402. In some embodiments, method 500 may be executed by one or more layers within a vehicle management system stack (such as vehicle management stack 200, vehicle management stack 250, etc.). In other embodiments, method 500 may be executed by a processor independently of but in conjunction with a vehicle control system stack (such as vehicle management stack 200, vehicle management stack 250, etc.). For example, method 500 may be implemented as a stand-alone software module or within dedicated hardware that monitors data and commands from / within a vehicle management system stack (e.g., vehicle management stacks 200, 250, etc.) and is configured to take actions and store data as described.

[0104] In block 502, the processor may perform operations including receiving an instruction to run a security-compliant vehicle algorithm on a vehicle heterogeneous computing system. In various embodiments, the instruction to run a security-compliant vehicle algorithm on a vehicle heterogeneous computing system may be a notification received from a scheduler, which includes an identifier of the security-compliant vehicle algorithm.

[0105] In box 504, the processor may perform operations including determining whether a non-safety-compliant computing unit in a heterogeneous computing system for vehicles is preferably used to run a vehicle algorithm. Determining whether a non-safety-compliant computing unit is preferred for running an algorithm (such as a vehicle algorithm) may be based on the nature of the algorithm. For example, an algorithm requiring a series of parallel operations may be more preferably run on a GPU. As another example, a highly vectorized algorithm may be more preferably run on a DSP. The determination of the computing unit may be controlled by settings associated with the algorithm and / or may be determined based on the state of the computing units in the system (e.g., estimated latency, etc.), the properties of the dataset to be run with the algorithm, or any other considerations during algorithm runtime.

[0106] In block 506, the processor can perform operations including modifying the execution of a vehicle algorithm in response to determining that a non-safety compliance computing unit of the vehicle heterogeneous computing system is preferably used to run the vehicle algorithm, to execute a portion of the vehicle algorithm using the non-safety compliance computing unit of the vehicle heterogeneous computing system and another portion of the vehicle algorithm using the safety compliance computing unit of the vehicle heterogeneous computing system. In some embodiments, modifying the execution of the algorithm (such as a vehicle algorithm) may include creating a lightweight version of the algorithm for at least partially running on the safety compliance computing unit of the heterogeneous computing system. The lightweight version of the algorithm (such as a lightweight version of a vehicle algorithm) may be a version that requires fewer computational resources to execute compared to the full version of the algorithm. For example, the full version of the vehicle algorithm may use a grid or filter setting with fine granularity (or producing higher resolution), while the lightweight version of the vehicle algorithm may use a grid or filter setting with coarser granularity (or producing lower resolution). In some embodiments, modifying the execution of the algorithm may include identifying key portions of the dataset. In various embodiments, key portions of the dataset may be portions of the dataset that are likely associated with safety, such as grid segments including pedestrians, data related to accident avoidance, etc. In some embodiments, the execution of a modified algorithm (such as a vehicle algorithm) may include: generating more refined output for identified key portions of the dataset, and running the algorithm on a non-security-compliant computing unit of a heterogeneous computing system (such as a vehicle heterogeneous computing system) to generate more refined output for all other portions of the dataset.

[0107] Figure 6 A method 500 for supporting safety-compliant computing in a heterogeneous computing system for transportation vehicles, according to various embodiments, is described. (See also...) Figures 1A-6Method 600 may be implemented in a processor (e.g., 164), a processing device (e.g., 300), and / or a control unit (e.g., 104) of a vehicle (e.g., 100) (referred to differently as a "processor") or a vehicle computing device 402. In some embodiments, method 600 may be executed by one or more layers within a vehicle management system stack (such as vehicle management stack 200, vehicle management stack 250, etc.). In other embodiments, method 600 may be executed by a processor independently of but in conjunction with a vehicle control system stack (such as vehicle management stack 200, vehicle management stack 250, etc.). For example, method 600 may be implemented as a stand-alone software module or implemented in dedicated hardware that monitors data and commands from / within a vehicle management system stack (e.g., vehicle management stacks 200, 250, etc.) and is configured to take action and store data as described. In various embodiments, operation of method 600 may be combined with method 500 ( Figure 5 The operation of method 600 is performed in response to or as part of the operation in block 506 that modifies the vehicle algorithm.

[0108] In box 508, the processor can perform operations including modifying the execution of the vehicle algorithm to create a lightweight version of the vehicle algorithm for operation on a security-compliant computing unit of a heterogeneous vehicle computing system. The lightweight version of the algorithm can be a version that requires fewer computational resources to execute compared to the full version of the algorithm. For example, the full version of the vehicle algorithm can use a mesh or filter setting with fine granularity (or producing higher resolution), while the lightweight version of the vehicle algorithm can use a mesh or filter setting with coarser granularity (or producing lower resolution).

[0109] In box 510, the processor can perform operations including running a vehicle algorithm on a non-safety-compliant computing unit of a vehicle heterogeneous computing system to generate a more refined output for a dataset. For example, running a vehicle algorithm on a non-safety-compliant computing unit of a vehicle heterogeneous computing system to generate a more refined output for a dataset may include running the vehicle algorithm using a grid or filter setting with fine granularity (or producing a higher resolution).

[0110] In box 512, the processor can perform operations including running a lightweight version of a vehicle algorithm on a safety compliance computing unit of a vehicle heterogeneous computing system to generate coarse output for a dataset. For example, running a lightweight version of a vehicle algorithm on a safety compliance computing unit of a vehicle heterogeneous computing system to generate coarse output for a dataset may include running the lightweight version of the vehicle algorithm using a grid or filter setting with coarse granularity (or producing lower resolution).

[0111] In block 514, the processor can perform operations including determining whether a finer-grained output and a coarse-grained output match. In various embodiments, determining whether a finer-grained output and a coarse-grained output match may include various operations for determining the matching portions, such as comparing the portions, calculating the hash of the portions, etc.

[0112] In block 516, the processor can perform operations including generating an alarm in response to determining a mismatch between fine and coarse outputs. For example, generating an alarm in response to determining a mismatch between fine and coarse outputs may include sending a fault alarm to an upper layer to notify the upper layer to take safety actions (e.g., warn the driver, perform evasive maneuvers, etc.).

[0113] Figure 7 A method 700 for supporting safety-compliant computing in a heterogeneous computing system for transportation vehicles, according to various embodiments, is described. (See also...) Figures 1A-7 Method 700 may be implemented in a processor (e.g., 164), a processing device (e.g., 300), and / or a control unit (e.g., 104) of a vehicle (e.g., 100) (referred to differently as a “processor”) or a vehicle computing device 402. In some embodiments, method 700 may be executed by one or more layers within a vehicle management system stack (such as vehicle management stack 200, vehicle management stack 250, etc.). In other embodiments, method 700 may be executed by a processor independently of but in conjunction with a vehicle control system stack (such as vehicle management stack 200, vehicle management stack 250, etc.). For example, method 700 may be implemented as a stand-alone software module or within dedicated hardware that monitors data and commands from / within a vehicle management system stack (e.g., vehicle management stacks 200, 250, etc.) and is configured to take action and store data as described. In various embodiments, operation of method 700 may be combined with method 500 ( Figure 5 The operation of method 700 is performed in response to or as part of the operation in block 506 that modifies the vehicle algorithm.

[0114] In block 508, the processor can perform operations including modifying the execution of the transportation algorithm to create a lightweight version of the transportation algorithm for operation on a security-compliant computing unit of a heterogeneous transportation computing system, as described in method 600. Figure 6 The operation discussed is as follows.

[0115] In box 518, the processor can perform operations including running a lightweight version of a vehicle algorithm on a safety compliance computing unit of a vehicle heterogeneous computing system to generate coarse output for a dataset. For example, running a lightweight version of a vehicle algorithm on a safety compliance computing unit of a vehicle heterogeneous computing system to generate coarse output for a dataset may include running the lightweight version of the vehicle algorithm using a grid or filter setting with coarse granularity (or producing lower resolution).

[0116] In block 520, the processor can perform operations including identifying portions of the coarse output as urgent. In some embodiments, identifying portions of the coarse output as urgent may include determining whether any portion of the coarse output is associated with an object requiring further monitoring, an object marked as having stringent security requirements, or an object otherwise indicated as important, with an importance setting, or a security setting. As a specific example, urgent grid tiles may be identified by determining that a grid tile is associated with a pedestrian or a small object.

[0117] In box 522, the processor can perform operations including running a vehicle algorithm at least twice on a non-safety compliance computing unit for a portion of the dataset corresponding to the identified emergency portion, to generate at least a first finer output and a second finer output. For example, the vehicle algorithm can be run separately at least twice for the emergency portion with a coarse output using a grid or filter setting with fine granularity (or producing a higher resolution), to generate at least a first finer output and a second finer output.

[0118] In block 524, the processor may perform operations including determining whether a first finer output and a second finer output match. In various embodiments, determining whether the first finer output and the second finer output match may include various operations for determining that the outputs match, such as comparing the outputs, calculating the hash of the outputs, etc.

[0119] In block 526, the processor may perform operations including generating an alarm in response to determining that the first finer output and the second finer output do not match. For example, generating an alarm in response to determining that the first finer output and the second finer output do not match may include sending a fault alarm to an upper layer to notify the upper layer to take safety action (e.g., warn the driver, perform evasive maneuvers, etc.).

[0120] In block 528, the processor may perform operations including replacing an identified urgent portion of a coarse output with one of the first finer output or the second finer output in response to determining that the first finer output and the second finer output do indeed match. For example, replacing an identified urgent portion of a coarse output with one of the first finer output or the second finer output in response to determining that the first finer output and the second finer output do indeed match may include replacing the coarse output data in the identified urgent portion with the first finer output data for the urgent portion or the second finer output data for the urgent portion.

[0121] Figure 8 A method 800 for supporting safety-compliant computing in a heterogeneous computing system for transportation vehicles, according to various embodiments, is described. (See also...) Figures 1A-8 Method 800 may be implemented in a processor (e.g., 164), a processing device (e.g., 300), and / or a control unit (e.g., 104) of a vehicle (e.g., 100) (referred to differently as a "processor") or a vehicle computing device 402. In some embodiments, method 800 may be executed by one or more layers within a vehicle management system stack (such as vehicle management stack 200, vehicle management stack 250, etc.). In other embodiments, method 800 may be executed by a processor independently of but in conjunction with a vehicle control system stack (such as vehicle management stack 200, vehicle management stack 250, etc.). For example, method 800 may be implemented as a stand-alone software module or implemented in dedicated hardware that monitors data and commands from / within a vehicle management system stack (e.g., vehicle management stacks 200, 250, etc.) and is configured to take action and store data as described. In various embodiments, operation of method 800 may be combined with method 500 ( Figure 5 The operation of method 800 is performed in response to or as part of the operation in block 506 that modifies the vehicle algorithm.

[0122] In block 508, the processor can perform operations including modifying the execution of the transportation algorithm to create a lightweight version of the transportation algorithm for operation on a security-compliant computing unit of a heterogeneous transportation computing system, as described in method 600. Figure 6 The operation discussed is as follows.

[0123] In box 530, the processor can perform operations including running a vehicle algorithm on a non-safety-compliant computing unit of a vehicle heterogeneous computing system to generate a more refined output for a dataset. For example, running a vehicle algorithm on a non-safety-compliant computing unit of a vehicle heterogeneous computing system to generate a more refined output for a dataset may include running the vehicle algorithm using a grid or filter setting with fine granularity (or producing a higher resolution).

[0124] In box 532, the processor may perform operations including running a lightweight version of the vehicle algorithm on a randomly sampled portion of the dataset on the security compliance computing unit to generate a coarse output for that randomly sampled portion of the dataset. For example, running a lightweight version of the vehicle algorithm on a randomly sampled portion of the dataset on the security compliance computing unit to generate a coarse output for that randomly sampled portion of the dataset may include running the lightweight version of the vehicle algorithm on a randomly selected subset of the dataset with a grid or filter setting that has coarse granularity (or produces a lower resolution).

[0125] In block 534, the processor may perform operations including determining whether coarse and finer outputs for a random sample portion match. In various embodiments, determining whether coarse and finer outputs for a random sample portion match may include various operations for determining that the outputs match, such as comparing the outputs, calculating the hash of the outputs, etc.

[0126] In block 536, the processor may perform operations including generating an alarm in response to determining a mismatch between coarse and fine output for a random sample portion. For example, generating an alarm in response to determining a mismatch between coarse and fine output for a random sample portion may include sending a fault alarm to an upper layer to notify the upper layer to take safety actions (e.g., warn the driver, perform evasive maneuvers, etc.).

[0127] Figure 9 A method 900 for supporting safety-compliant computing in a heterogeneous computing system for transportation vehicles, according to various embodiments, is described. (See also...) Figures 1A-9Method 900 may be implemented in a processor (e.g., 164), a processing device (e.g., 300), and / or a control unit (e.g., 104) of a vehicle (e.g., 100) (referred to differently as a “processor”) or a vehicle computing device 402. In some embodiments, method 900 may be executed by one or more layers within a vehicle management system stack (such as vehicle management stack 200, vehicle management stack 250, etc.). In other embodiments, method 900 may be executed by a processor independently of but in conjunction with a vehicle control system stack (such as vehicle management stack 200, vehicle management stack 250, etc.). For example, method 900 may be implemented as a stand-alone software module or within dedicated hardware that monitors data and commands from / within a vehicle management system stack (e.g., vehicle management stacks 200, 250, etc.) and is configured to take action and store data as described. In various embodiments, operation of method 900 may be combined with method 500 ( Figure 5 The operation of method 900 is performed in response to or as part of the operation in block 506 that modifies the vehicle algorithm.

[0128] In box 538, the processor can perform operations including identifying key portions of the dataset. In some embodiments, key portions of the dataset may be portions of the dataset that are likely associated with safety, such as grid segments including pedestrians, data related to accident avoidance, etc.

[0129] In box 540, the processor can perform operations including running a vehicle algorithm on the security compliance computing unit of the vehicle heterogeneous computing system to generate finer-grained output for identified critical portions of the dataset. For example, the vehicle algorithm can run on the security compliance computing unit of the vehicle heterogeneous computing system with fine-grained (or higher-resolution) grids or filters for identified critical portions of the dataset to generate finer-grained output for identified critical portions of the dataset.

[0130] In box 542, the processor can perform operations including running a vehicle algorithm on a non-safety-compliant computing unit of a vehicle heterogeneous computing system to generate finer-grained outputs for all other parts of the dataset. For example, the vehicle algorithm can be run on a non-safety-compliant computing unit of a vehicle heterogeneous computing system with fine-grained (or higher-resolution) grids or filters for all non-critical parts of the dataset to generate finer-grained outputs for all other parts of the dataset.

[0131] Figure 10 A method 1000 for supporting safety-compliant computing in a heterogeneous computing system for transportation vehicles, according to various embodiments, is described. (Refer to...)Figures 1A-10 Method 1000 may be implemented in a processor (e.g., 164), a processing device (e.g., 300), and / or a control unit (e.g., 104) of a vehicle (e.g., 100) (referred to differently as a "processor") or a vehicle computing device 402. In some embodiments, method 1000 may be executed by one or more layers within a vehicle management system stack (such as vehicle management stack 200, vehicle management stack 250, etc.). In other embodiments, method 1000 may be executed by a processor independently of but in conjunction with a vehicle control system stack (such as vehicle management stack 200, vehicle management stack 250, etc.). For example, method 1000 may be implemented as a stand-alone software module or within dedicated hardware that monitors data and commands from / within a vehicle management system stack (e.g., vehicle management stacks 200, 250, etc.) and is configured to take actions and store data as described.

[0132] In block 502, the processor can perform operations including receiving instructions to run a vehicle algorithm requiring safety compliance in a vehicle heterogeneous computing system, as in reference method 500. Figure 5 The operation discussed is as follows.

[0133] In block 504, the processor may perform operations including determining whether a non-safety-compliant computing unit of a heterogeneous computing system for vehicles is preferably used to run vehicle algorithms, as in reference method 500. Figure 5 The operation discussed is as follows.

[0134] In box 548, the processor can perform operations including modifying the execution of the vehicle algorithm to create a lightweight version of the vehicle algorithm in response to determining that a non-security-compliant computing unit of the heterogeneous computing system of the vehicle is preferably used to run the vehicle algorithm. The lightweight version of the algorithm (such as a lightweight version of the vehicle algorithm) can be a version of the algorithm that requires fewer computational resources to execute compared to the full version of the algorithm. For example, the full version of the vehicle algorithm can use a mesh or filter setting with fine granularity (or producing higher resolution), while the lightweight version of the algorithm (such as a lightweight version of the vehicle algorithm) can use a mesh or filter setting with coarser granularity (or producing lower resolution).

[0135] In box 550, the processor can perform operations including identifying significant portions of the dataset. In some embodiments, significant portions of the dataset may be portions of the dataset associated with importance settings, such as importance thresholds, object types identified as important, etc.

[0136] In box 552, the processor can perform operations including running a vehicle algorithm on the safety compliance computing unit of the vehicle heterogeneous computing system to generate finer-grained output for identified important portions of the dataset. For example, the vehicle algorithm can run on the safety compliance computing unit of the vehicle heterogeneous computing system with fine-grained (or higher-resolution) grids or filters for identified important portions of the dataset to generate finer-grained output for identified important portions of the dataset.

[0137] In box 554, the processor can perform operations including running a lightweight version of a vehicle algorithm on a safety compliance computing unit of a vehicle heterogeneous computing system to generate coarse output for all other parts of the dataset. For example, the lightweight vehicle algorithm can be run on a non-safety compliance computing unit of the vehicle heterogeneous computing system with a grid or filter set with coarse granularity (or producing lower resolution) for all non-critical parts of the dataset to generate coarse output for all other parts of the dataset.

[0138] The various embodiments illustrated and described are provided merely as examples illustrating the various features of the claims. However, the features shown and described with respect to any given embodiment are not necessarily limited to the associated embodiment and may be used in conjunction with or combined with other illustrated and described embodiments. Furthermore, the claims are not intended to be limited to any single example embodiment.

[0139] The above method descriptions and process flow diagrams are provided as illustrative examples only and are not intended to require or imply that the blocks of the various embodiments must be performed in the given order. As those skilled in the art will appreciate, the order of the blocks in the foregoing embodiments can be performed in any order. Terms such as “then,” “following,” “next,” etc., are not intended to limit the order of the blocks; these terms are simply used to guide the reader through the descriptions of the methods. Furthermore, any reference to singular claim elements (e.g., references using the articles “a,” “some,” or “the”) should not be construed as limiting that element to the singular.

[0140] The various illustrative logic blocks, modules, circuits, and algorithm blocks described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this hardware-software interchangeability, the various illustrative components, blocks, modules, circuits, and chunks are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of the various embodiments.

[0141] The hardware used to implement the various illustrative logics, logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed by a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of communication devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration. Alternatively, some blocks or methods may be performed by a circuit system dedicated to a given function.

[0142] In various embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored as one or more instructions or code on a non-transient computer-readable medium or a non-transient processor-readable medium. The operation of the methods or algorithms disclosed herein may be implemented in a processor-executable software module that may reside on a non-transient computer-readable or processor-readable storage medium. A non-transient computer-readable or processor-readable storage medium may be any storage medium accessible to a computer or processor. By way of example and not limitation, such non-transient computer-readable or processor-readable media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible to a computer. As used herein, disks and discs include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically reproduce data magnetically and discs optically using lasers. The above combinations are also included within the scope of non-transient computer-readable and processor-readable media. Additionally, the operation of a method or algorithm may reside as a single line of code and / or instruction, or any combination or set of code and / or instructions, on a non-transient processor-readable and / or computer-readable medium that may be incorporated into a computer program product.

[0143] The prior description of the disclosed embodiments is intended to enable any person skilled in the art to make or use the embodiments of the invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the embodiments. Thus, the various embodiments are not intended to be limited to those shown herein, but are to be given the widest scope consistent with the following claims and the principles and novel features disclosed herein.

Claims

1. A method for supporting security-compliant computing in heterogeneous computing systems, comprising: Receive an instruction to run an algorithm requiring security compliance in the heterogeneous computing system; Determine whether the non-security compliant computing unit of the heterogeneous computing system is preferably used to run the algorithm; as well as In response to determining that the non-security-compliant computing unit of the heterogeneous computing system is preferably used to run the algorithm, the execution of the algorithm is modified to execute a portion of the algorithm using the non-security-compliant computing unit of the heterogeneous computing system and execute another portion of the algorithm using the security-compliant computing unit of the heterogeneous computing system. Modifying the execution of the algorithm to execute a portion of the algorithm using the non-security-compliant computing unit of the heterogeneous computing system and to execute another portion of the algorithm using the security-compliant computing unit of the heterogeneous computing system includes: The execution of the algorithm is modified to create a lightweight version of the algorithm for operation on the security-compliant computing unit of the heterogeneous computing system.

2. The method of claim 1, further comprising: The algorithm is run on the non-security-compliant computing unit of the heterogeneous computing system to generate more refined output for the dataset; The lightweight version of the algorithm is run on the security-compliant computing unit of the heterogeneous computing system to generate a coarse output for the dataset; Determine whether the finer output and the coarser output match; as well as An alarm is generated in response to the determination that the finer output and the coarse output do not match.

3. The method of claim 1, further comprising: The algorithm is run on the non-security-compliant computing unit of the heterogeneous computing system to generate more refined output for the dataset; The lightweight version of the algorithm is run on a randomly sampled portion of the dataset on the security compliance computing unit to generate a coarse output for the randomly sampled portion of the dataset; Determine whether the coarse output and the finer output for the random sampling portion match; as well as An alarm is generated in response to the determination that the coarse output and the finer output for the random sampling portion do not match.

4. The method of claim 1, further comprising: Identify important parts of the dataset; The algorithm is run on the security and compliance computing unit of the heterogeneous computing system to generate more refined output for the identified important parts of the dataset; as well as The lightweight version of the algorithm is run on the security and compliance computing unit of the heterogeneous computing system to generate coarse output for all other parts of the dataset.

5. The method of claim 1, wherein modifying the execution of the algorithm to execute a portion of the algorithm using the non-security-compliant computing unit of the heterogeneous computing system and executing another portion of the algorithm using the security-compliant computing unit of the heterogeneous computing system comprises: Identify key parts of the dataset; The algorithm is run on the security and compliance computing unit of the heterogeneous computing system to generate more refined output for the identified key portions of the dataset; as well as The algorithm is run on the non-security-compliant computing unit of the heterogeneous computing system to generate more refined outputs for all other parts of the dataset.

6. The method of claim 1, wherein the heterogeneous computing system is a system-on-a-chip.

7. The method of claim 1, wherein the security compliance computing unit is a central processing unit or a digital signal processing unit, and the non-security compliance computing unit is a graphics processing unit.

8. The method of claim 1, wherein the heterogeneous computing system is a vehicle heterogeneous computing system, and the algorithm requiring safety compliance is a vehicle algorithm requiring safety compliance.

9. The method of claim 8, wherein the vehicle algorithm requiring safety compliance is a vehicle algorithm requiring Automotive Safety Integrity Level B (ASIL B) compliance.

10. A vehicle computing device, comprising: A processor configured with processor-executable instructions for the following operations: Receive instructions to run algorithms that require security compliance on heterogeneous computing systems; Determine whether the non-security compliant computing unit of the heterogeneous computing system is preferably used to run the algorithm; as well as In response to determining that the non-security-compliant computing unit of the heterogeneous computing system is preferably used to run the algorithm, the execution of the algorithm is modified to execute a portion of the algorithm using the non-security-compliant computing unit of the heterogeneous computing system and execute another portion of the algorithm using the security-compliant computing unit of the heterogeneous computing system. The processor is further configured with processor-executable instructions to modify the execution of the algorithm by performing the following operations: executing a portion of the algorithm using the non-security-compliant computing unit of the heterogeneous computing system and executing another portion of the algorithm using the security-compliant computing unit of the heterogeneous computing system. The execution of the algorithm is modified to create a lightweight version of the algorithm for operation on the security-compliant computing unit of the heterogeneous computing system.

11. The vehicle computing device of claim 10, wherein the processor is further configured with processor-executable instructions for the following operations: The algorithm is run on the non-security-compliant computing unit of the heterogeneous computing system to generate more refined output for the dataset; The lightweight version of the algorithm is run on the security-compliant computing unit of the heterogeneous computing system to generate a coarse output for the dataset; Determine whether the finer output and the coarser output match; as well as An alarm is generated in response to the determination that the finer output and the coarse output do not match.

12. The vehicle computing device of claim 10, wherein the processor is further configured with processor-executable instructions for the following operations: The algorithm is run on the non-security-compliant computing unit of the heterogeneous computing system to generate more refined output for the dataset; The lightweight version of the algorithm is run on a randomly sampled portion of the dataset on the security compliance computing unit to generate a coarse output for the randomly sampled portion of the dataset; Determine whether the coarse output and the finer output for the random sampling portion match; as well as An alarm is generated in response to the determination that the coarse output and the finer output for the random sampling portion do not match.

13. The vehicle computing device of claim 10, wherein the processor is further configured with processor-executable instructions for the following operations: Identify important parts of the dataset; The algorithm is run on the security-compliant computing unit of the heterogeneous computing system to generate more refined output for the identified important portions of the dataset; and The lightweight version of the algorithm is run on the security and compliance computing unit of the heterogeneous computing system to generate coarse output for all other parts of the dataset.

14. The vehicle computing device of claim 10, wherein the processor is further configured with processor-executable instructions to modify the execution of the algorithm by performing the following operations: executing a portion of the algorithm using the non-security-compliant computing unit of the heterogeneous computing system and executing another portion of the algorithm using the security-compliant computing unit of the heterogeneous computing system: Identify key parts of the dataset; The algorithm is run on the security and compliance computing unit of the heterogeneous computing system to generate more refined output for the identified key portions of the dataset; as well as The algorithm is run on the non-security-compliant computing unit of the heterogeneous computing system to generate more refined outputs for all other parts of the dataset.

15. The vehicle computing device of claim 10, wherein the processor is further configured with processor-executable instructions such that the safety compliance computing unit is a central processing unit or a digital signal processing unit, and the non-safety compliance computing unit is a graphics processing unit.

16. The vehicle computing device of claim 10, wherein the processor is further configured with processor-executable instructions such that the algorithm requiring safety compliance is a vehicle algorithm requiring Automotive Safety Integrity Level B (ASIL B) compliance.

17. A non-transient processor-readable medium having processor-executable instructions stored thereon, the instructions being configured to cause a processor of a vehicle computing device to perform operations, the operations including: Receive instructions to run algorithms that require security compliance on heterogeneous computing systems; Determine whether the non-security compliant computing unit of the heterogeneous computing system is preferably used to run the algorithm; as well as In response to determining that the non-security-compliant computing unit of the heterogeneous computing system is preferably used to run the algorithm, the execution of the algorithm is modified to execute a portion of the algorithm using the non-security-compliant computing unit of the heterogeneous computing system and execute another portion of the algorithm using the security-compliant computing unit of the heterogeneous computing system. The stored processor-executable instructions are configured to cause the processor of the vehicle computing device to perform operations that modify the execution of the algorithm to execute a portion of the algorithm using the non-security-compliant computing unit of the heterogeneous computing system and to execute another portion of the algorithm using the security-compliant computing unit of the heterogeneous computing system, including: The execution of the algorithm is modified to create a lightweight version of the algorithm for operation on the security-compliant computing unit of the heterogeneous computing system.

18. The non-transient processor-readable medium of claim 17, wherein the stored processor-executable instructions are configured to cause a processor of a vehicle computing device to perform operations, said operations further comprising: The algorithm is run on the non-security-compliant computing unit of the heterogeneous computing system to generate more refined output for the dataset; The lightweight version of the algorithm is run on the security-compliant computing unit of the heterogeneous computing system to generate a coarse output for the dataset; Determine whether the finer output and the coarser output match; as well as An alarm is generated in response to the determination that the finer output and the coarse output do not match.

19. The non-transient processor-readable medium of claim 17, wherein the stored processor-executable instructions are configured to cause a processor of a vehicle computing device to perform operations, said operations further comprising: The algorithm is run on the non-security-compliant computing unit of the heterogeneous computing system to generate more refined output for the dataset; The lightweight version of the algorithm is run on a randomly sampled portion of the dataset on the security compliance computing unit to generate a coarse output for the randomly sampled portion of the dataset; Determine whether the coarse output and the finer output for the random sampling portion match; as well as An alarm is generated in response to the determination that the coarse output and the finer output for the random sampling portion do not match.

20. The non-transient processor-readable medium of claim 17, wherein the stored processor-executable instructions are configured to cause a processor of a vehicle computing device to perform operations, said operations further comprising: Identify important parts of the dataset; The algorithm is run on the security and compliance computing unit of the heterogeneous computing system to generate more refined output for the identified important parts of the dataset; as well as The lightweight version of the algorithm is run on the security and compliance computing unit of the heterogeneous computing system to generate coarse output for all other parts of the dataset.

21. The non-transient processor-readable medium of claim 17, wherein the stored processor-executable instructions are configured to cause a processor of a vehicle computing device to perform operations such that modifying the execution of the algorithm to execute a portion of the algorithm using the non-security-compliant computing unit of the heterogeneous computing system and executing another portion of the algorithm using the security-compliant computing unit of the heterogeneous computing system comprises: Identify key parts of the dataset; The algorithm is run on the security and compliance computing unit of the heterogeneous computing system to generate more refined output for the identified key portions of the dataset; as well as The algorithm is run on the non-security-compliant computing unit of the heterogeneous computing system to generate more refined outputs for all other parts of the dataset.

22. The non-transient processor-readable medium of claim 17, wherein the stored processor-executable instructions are configured to cause the processor of the vehicle computing device to perform operations such that the safety compliance computing unit is a central processing unit or a digital signal processing unit, and the non-safety compliance computing unit is a graphics processing unit.

23. The non-transient processor-readable medium of claim 17, wherein the stored processor-executable instructions are configured to cause the processor of the vehicle computing device to perform operations such that the algorithm requiring safety compliance is a vehicle algorithm requiring Automotive Safety Integrity Level B (ASIL B) compliance.

24. A vehicle computing device, comprising: A means for receiving instructions to run algorithms that require security compliance in a heterogeneous computing system; A means for determining whether a non-security compliant computing unit of the heterogeneous computing system is preferably used to run the algorithm; as well as A means for modifying the execution of the algorithm in response to determining that the non-security compliant computing unit of the heterogeneous computing system is preferably used to run the algorithm, so as to execute a portion of the algorithm using the non-security compliant computing unit of the heterogeneous computing system and execute another portion of the algorithm using the security compliant computing unit of the heterogeneous computing system. The means for modifying the execution of the algorithm to execute a portion of the algorithm using the non-security-compliant computing unit of the heterogeneous computing system and to execute another portion of the algorithm using the security-compliant computing unit of the heterogeneous computing system includes: A means for modifying the execution of the algorithm to create a lightweight version of the algorithm for operation on the security-compliant computing unit of the heterogeneous computing system.

Images