Redundant hardware system for autonomous vehicles

Abstract

To provide a technology that relates to partially redundant equipment architectures for vehicles able to operate in an autonomous driving mode.SOLUTION: Aspects of a technology according to the invention employ fallback configurations, such as two or more fallback sensor configurations that provide a minimum amount of field of view around a vehicle. For instance, different sensor arrangements are logically associated with different operating domains of the vehicle. Fallback configurations for computing resources and / or power resources are also provided. The respective fallback configurations may have different reasons for being triggered and may result in different types of fallback modes of operation. Triggering conditions may relate, e.g., to a type of failure, fault, or other reduction in component capability, a current driving mode, environmental conditions in a vicinity of a vehicle or along a planned route, or other factors. The fallback modes may involve altering a previously planned trajectory, altering vehicle speed, and / or altering a destination of the vehicle.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Patent Application No. 16 / 215,713, filed December 11, 2018, and is related to co-pending U.S. Patent Application No. 16 / 180,267, entitled “Systems for Implementing Fallback Behaviors for Autonomous Vehicles,” filed November 5, 2018 (Attorney Docket No. XSDV3.0F-2059), the entire disclosure of which is incorporated herein by reference. [Background technology]

[0002] Autonomous vehicles, such as vehicles that do not require a human driver, may be used to assist in transporting passengers or cargo from one location to another. Such vehicles may operate in a fully autonomous mode or in a partially autonomous mode, where a human may provide some driving input. To operate in an autonomous mode, the vehicle may employ sensors and perform various driving actions using received sensor information. However, if a sensor or other component of the system fails or otherwise experiences degradation in functionality, this may adversely affect the driving functionality of the vehicle. Summary of the Invention

[0003] The present technology relates to a redundant architecture for sensor, computation, and power systems of a vehicle configured to operate in a fully or partially autonomous driving mode. Although it may be possible to fully redundant all components and subsystems, this may not be feasible, especially in vehicles that are constrained by the size and placement of the sensor constellation, as well as other limiting factors such as cost. Therefore, aspects of the present technology use a fallback configuration for partial redundancy. For example, a fallback sensor configuration may provide some minimum field of view (FOV) around the vehicle, as well as minimal computing power for perception and planning processing.

[0004] According to an aspect of the present technology, a vehicle is configured to operate in an autonomous driving mode. The vehicle includes a driving system, a perception system, and a control system. The driving system includes a steering subsystem, an acceleration subsystem, and a deceleration subsystem for controlling driving of the vehicle in the autonomous driving mode. The perception system has a plurality of sensors configured to detect information about an environment surrounding the vehicle. The plurality of sensors includes a first set of sensors associated with a first operational domain and a second set of sensors associated with a second operational domain. The control system is operably coupled to the driving system and the perception system. The control system includes a first computing subsystem associated with the first operational domain and a second computing subsystem associated with the second operational domain. Each of the first and second computing subsystems has one or more processors. The first and second computing subsystems are each configured to receive sensor data from one or both of the first set of sensors and the second set of sensors in the first operating mode. In response to the sensor data received in the first operating mode, the control system is configured to control the driving system to drive the vehicle in the autonomous driving mode. In response to an error condition for one or more of the plurality of sensors, a first computing subsystem is configured to process sensor data from a first set of sensors in a first operating domain and a second computing subsystem is configured to process sensor data from a second set of sensors in a second operating domain, and in response to the error condition, only one of the first computing subsystem or the second computing subsystem is configured to control the driving system in a fallback driving mode.

[0005] In an example, each of the first and second sets of sensors includes at least one sensor selected from the group consisting of a lidar sensor, a radar sensor, a camera sensor, an auditory sensor, and a positioning sensor, where each of the first and second sets of sensors may include a respective group of lidar, radar, and camera sensors, each group providing a selected view of the environment surrounding the vehicle.

[0006] Each of the first set of sensors can have a respective field of view, and each of the second set of sensors can have a respective field of view, where the respective field of view of the second set of sensors is different from the respective field of view of the first set of sensors. One or more interior sensors may be disposed within the interior of the vehicle. The one or more interior sensors may include at least one of a camera sensor, an auditory sensor, and an infrared sensor.

[0007] In another example, the fallback driving mode includes a first fallback mode and a second fallback mode. The first fallback mode includes a first set of driving actions and the second fallback mode includes a second set of driving actions that are different from the first set of driving actions. In this example, the first computing subsystem is configured to control the driving system in the first fallback mode and the second computing subsystem is configured to control the driving system in the second fallback mode.

[0008] In yet another example, the vehicle further comprises first and second power distribution subsystems, where in the fallback operating mode, the first power distribution subsystem is associated with the first operating domain and supplies power only to devices of the first operating domain, and in the fallback operating mode, the second power distribution subsystem is associated with the second operating domain and supplies power only to devices of the second operating domain, and the first and second operating domains are electrically isolated from each other. In this case, the first power distribution subsystem may be configured to provide power to a first set of base vehicle loads in the first operating mode and to provide power to the devices of the first operating domain in the fallback operating mode, and the second power distribution subsystem may be configured to provide power to a second set of base vehicle loads different from the first set of base vehicle loads in the first operating mode and to provide power to the devices of the second operating domain in the fallback operating mode.

[0009] In a further example, the plurality of sensors of the perception system includes a first set of fallback sensors operably coupled to the first computing subsystem, a second set of fallback sensors operably coupled to the second computing subsystem, and a set of non-fallback sensors. In this scenario, the set of non-fallback sensors may be operably coupled to one or both of the first computing subsystem and the second computing subsystem. In yet another example, the plurality of sensors of the perception system includes a subset of sensors operably coupled to both the first and second computing subsystems in a fallback driving mode.

[0010] In accordance with another aspect of the present technique, a method for operating a vehicle in an autonomous driving mode is provided. The method includes detecting information regarding an environment surrounding the vehicle by a plurality of sensors of a perception system of the vehicle, the plurality of sensors including a first set of sensors associated with a first operating domain and a second set of sensors associated with a second operating domain; receiving the detected information regarding the environment surrounding the vehicle as sensor data by a control system of the vehicle, the control system including a first computing subsystem associated with the first operating domain and a second computing subsystem associated with the second operating domain; in response to receiving the sensor data in the first operating mode, the control system controls a driving system of the vehicle to drive the vehicle in an autonomous driving mode; detecting an error condition for one or more of the plurality of sensors; upon detecting the error condition, the first computing subsystem processes sensor data from only the first set of sensors in the first operating domain and the second computing subsystem processes sensor data from only the second set of sensors in the second operating domain; and in response to the error condition, only one of the first computing subsystem or the second computing subsystem controls the driving system in a fallback driving mode.

[0011] In one example, the fallback driving mode includes multiple fallback modes including a first fallback mode and a second fallback mode, where the first fallback mode includes a first set of driving actions and the second fallback mode includes a second set of driving actions different from the first set of driving actions, where controlling the driving system in the fallback driving mode can include a first computing subsystem controlling the driving system in the first fallback mode or a second computing subsystem controlling the driving system in the second fallback mode.

[0012] In another example, the method may further include supplying power by a first power distribution subsystem of the vehicle only to devices of the first operational domain and by a second power distribution subsystem of the vehicle only to devices of the second operational domain in the fallback operating mode. In this scenario, the first power distribution subsystem may provide power to a first set of base vehicle loads in the first operating mode and provide power to the devices of the first operational domain in the fallback operating mode, and the second power distribution subsystem may provide power to a second set of base vehicle loads different from the first set of base vehicle loads in the first operating mode and provide power to the devices of the second operational domain in the fallback operating mode.

[0013] During the fallback driving mode, a subset of the plurality of sensors of the perception system can provide sensor data to both the first and second computing subsystems. Controlling the driving system in the fallback driving mode includes at least one of altering a previously planned trajectory of the vehicle, altering a speed of the vehicle, or altering a destination of the vehicle. The method can further include stopping processing of sensor data from non-fallback-critical sensors during the fallback driving mode. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 shows a perspective view of an exemplary vehicle configured for use with aspects of the present technique. [Diagram 2] 2 illustrates a top view of the example vehicle of FIG. 1. [Diagram 3] FIG. 1 shows a block diagram of an exemplary vehicle, in accordance with aspects of the present technique. [Figure 4] FIG. 1 is a block diagram of an exemplary perception system, in accordance with aspects of the present technique. [Figure 5A] 1 illustrates an example of an area surrounding a vehicle according to an aspect of the present disclosure. [Figure 5B] 1 illustrates an example of an area surrounding a vehicle according to an aspect of the present disclosure. [Figure 6] 1 illustrates an exemplary sensor field of view according to an aspect of the present disclosure. [Figure 7] 1 illustrates an exemplary sensor assembly according to an aspect of the present disclosure. [Figure 8] 1 illustrates example sensor orientations according to aspects of the present disclosure. [Figure 9A] 1 illustrates an example of overlapping sensor fields of view according to an aspect of the present disclosure. [Figure 9B] 1 illustrates an example of overlapping sensor fields of view according to an aspect of the present disclosure. [Figure 10A] 1 illustrates examples of operations in different domains in accordance with aspects of the present technology. [Figure 10B] 1 illustrates examples of operations in different domains in accordance with aspects of the present technology. [Figure 11A] 13 illustrates further examples of operations in different domains in accordance with aspects of the present technology. [Figure 11B] 13 illustrates further examples of operations in different domains in accordance with aspects of the present technology. [Figure 11C] 13 illustrates further examples of operations in different domains in accordance with aspects of the present technology. [Figure 11D] 13 illustrates further examples of operations in different domains in accordance with aspects of the present technology. [Figure 12] 1 illustrates an exemplary automated driving system configuration in accordance with aspects of the present technology. [Figure 13A] 1 illustrates an exemplary power distribution configuration in accordance with aspects of the present technique. [Figure 13B] 1 illustrates an exemplary power distribution configuration in accordance with aspects of the present technique. [Figure 14] 1 illustrates a method of operation in accordance with an aspect of the present technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The partially redundant vehicle architectures discussed herein are associated with fallback configurations that use different sensor arrangements that can be logically associated with different operational domains of the vehicle. Each fallback configuration may have different reasons for being triggered and may result in different types of fallback operating modes. The triggering conditions may relate, for example, to a type of failure, fault, or other degradation in functionality of a component, the current autonomous driving mode, environmental conditions around the vehicle or along the planned route, or other factors.

[0016] Exemplary Vehicle System FIG. 1 illustrates a perspective view of a passenger vehicle 100, such as a minivan, sedan, or sports utility vehicle. FIG. 2 illustrates a top view of the passenger vehicle 100. The passenger vehicle 100 can include various sensors for obtaining information about the vehicle's external environment. For example, the rooftop housing 102 can include a Lidar sensor, as well as various cameras, radar units, infrared and / or acoustic sensors. The housing 104 located at the front end of the vehicle 100, and the housings 106a, 106b on the driver's and passenger's sides of the vehicle can each incorporate Lidar and / or other sensors. For example, the housing 106a may be located in front of the driver's side door along the vehicle's quarter panel. As illustrated, the passenger vehicle 100 also includes housings 108a, 108b for radar units, Lidar, and / or cameras that are also positioned toward the rear roof portion of the vehicle. Additional Lidar, radar units, and / or cameras (not shown) may be located elsewhere along the vehicle 100. For example, arrow 110 indicates that sensor units (112 in FIG. 2) may be positioned along the lead of the vehicle 100, such as on or adjacent to the bumper, and arrow 114 indicates a series of sensor units 116 arranged along a forward-facing direction of the vehicle. In some examples, the passenger vehicle 100 may also include various sensors for obtaining information regarding the interior space of the vehicle. The interior sensor(s) may include at least one of a camera sensor, an auditory sensor, and an infrared sensor.

[0017] Although certain aspects of the present disclosure are particularly useful in connection with certain types of vehicles, the vehicle may be any type of vehicle, including, but not limited to, cars, trucks, motorcycles, buses, recreational vehicles, and the like.

[0018] FIG. 3 illustrates a block diagram 300 with various components and systems of an exemplary vehicle configured to operate in a fully or semi-autonomous operating mode. By way of example, different degrees of autonomy may occur in a vehicle operating in a partial or fully autonomous driving mode. The US National Highway Traffic Safety Administration and the Society of Automotive Engineers have identified various levels to indicate how much, or how little, the vehicle controls the driving. For example, Level 0 is not automated, and the driver makes all driving-related decisions. Level 1, the lowest semi-autonomous mode, includes some drive assistance, such as cruise control. Level 2 has partial automation of certain driving actions, and Level 3 involves conditional automation that may allow a person in the driver's seat to take control as needed. In contrast, Level 4 is a highly automated level where the vehicle can drive without assistance in conditions of its choosing. In contrast, Level 5 is a full automation level where the vehicle can drive without assistance under all circumstances. The architectures, components, systems, and methods described herein can function in either semi-autonomous or fully autonomous modes, for example, Levels 1-5, referred to herein as "autonomous" driving modes. Thus, references to autonomous driving modes include both partial autonomy and full autonomy.

[0019] 3, the exemplary vehicle includes one or more computing devices 302, such as a computing device that includes one or more processors 304, memory 306, and other components typically present in a general-purpose computing device. The memory 306 stores information accessible by the one or more processors 304, including instructions 308 and data 310 that may be executed or otherwise used by the processor(s) 304. The computing system may control the overall operation of the vehicle when operating in an autonomous mode.

[0020] The memory 306 stores information accessible by the processor 304, including instructions 308 and data 310 that may be executed or otherwise used by the processor 304. The memory 306 may be of any type capable of storing information accessible by the processor, including a computing device readable medium. The memory is a non-transitory medium, such as a hard drive, memory card, optical disk, solid state, etc. The system may include different combinations of the foregoing, whereby different portions of the instructions and data are stored on different types of media.

[0021] The instructions 308 may be any set of instructions executed directly (such as machine code) or indirectly (such as script) by a processor. For example, the instructions may be stored as computing device code on a computing device readable medium. In that regard, the terms "instructions," "modules," and "programs" may be used interchangeably herein. The data 310 may be retrieved, stored, or modified by one or more processors 304 in accordance with the instructions 308. In one example, some or all of the memory 306 may be an event data recorder or other secure data storage system configured to store vehicle diagnostics and / or detected sensor data, and may be on-board the vehicle or at a remote location, depending on the implementation.

[0022] The processor 304 may be any conventional processor, such as a commercially available CPU. Alternatively, each processor may be a dedicated device, such as an ASIC or other hardware-based processor. Although FIG. 3 functionally illustrates the processor, memory, and other elements of the computing device 302 as being in the same block, such a device may actually include multiple processors, computing devices, or memories that may or may not be housed in the same physical housing. Similarly, the memory 306 may be a hard drive or other storage medium located in a different housing than that of the processor(s) 304. Thus, it will be understood that reference to a processor or computing device includes reference to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0023] In one example, the computing device 302 may form an autonomous driving computing system integrated into the vehicle 100. The autonomous driving computing system may be capable of communicating with various components of the vehicle. For example, the computing device 302 may be capable of communicating with various systems of the vehicle, including driving systems, including a deceleration system 312 (for controlling braking of the vehicle), an acceleration system 314 (for controlling acceleration of the vehicle), a steering system 316 (for controlling the orientation of the wheels and the direction of the vehicle), a signaling system 318 (for controlling the turn signals), a navigation system 320 (for navigating the vehicle to a location or around an object), and a positioning system 322 (for determining the position of the vehicle). The autonomous driving computing system may operate in part as a planner, for example, to determine a route from a starting point to a destination according to the navigation system 320 and the positioning system 322.

[0024] The computing device 302 is also operably coupled to a perception system 324 (for detecting objects in the vehicle's environment), a power system 326 (e.g., a battery and / or a gasoline or diesel powered engine), and a transmission system 330 to control the vehicle's movement, speed, etc., according to instructions 308 in memory 306 in an autonomous driving mode that does not request or require continuous or periodic input from a vehicle occupant. Wheels / tires 328 are coupled to the transmission system 330, and the computing device 302 may be capable of receiving information regarding tire pressure, balance, and other factors that may affect driving in the autonomous mode. The power system 326 may have multiple power distribution elements 327, each of which may supply power to selected components and other systems of the vehicle.

[0025] The computing device 302 may control the direction and speed of the vehicle by controlling various components. As an example, the computing device 302 may use map information and data from the navigation system 320 to navigate the vehicle to a destination fully autonomously. The computing device 302 may determine the location of the vehicle using the positioning system 322, and may use the perception system 324 to detect and respond to objects when necessary to safely arrive at that location. To do so, the computing device 302 may accelerate the vehicle (e.g., by increasing fuel or other energy provided to the engine by the acceleration system 314), decelerate (e.g., by reducing fuel supplied to the engine, shifting gears, and / or applying brakes by the deceleration system 312), change direction (e.g., by turning the front wheels or other wheels of the vehicle 100 by the steering system 316), and signal such change (e.g., by illuminating a turn indicator of the signaling system 318). Thus, the acceleration system 314 and the deceleration system 312 may be part of a driveline or other type of transmission system 330 that includes various components between the vehicle's engine and the vehicle's wheels. Again, by controlling these systems, the computing device 302 may also control the vehicle's transmission system 330 in order to steer the vehicle autonomously.

[0026] The navigation system 320 may be used by the computing device 302 to determine and follow a route to a location. In this regard, the navigation system 320 and / or the memory 306 may store map information, e.g., highly detailed maps that the computing device 302 may use to navigate or control a vehicle. By way of example, these maps may identify the shape and elevation of roadways, lane lines, intersections, crosswalks, speed limits, traffic signals, buildings, signs, real-time traffic information, vegetation, or other such objects and information. The lane lines may include features such as solid or dashed, double or single lane lines, solid or dashed lane lines, reflectors, etc. A given lane may be associated with left and / or right lane lines or other lane lines that define the boundaries of the lane. Thus, most lanes may be bounded by the left edge of one lane line and the right edge of another lane line.

[0027] The perception system 324 also includes sensors for detecting objects external to the vehicle. The detected objects may be other vehicles, obstacles on the road, traffic lights, signs, trees, etc. As discussed in more detail below, the perception system 324 is arranged to operate in two (or more) sensor domains, such as sensor domain A and sensor domain B as shown. Within each domain, the system may include one or both of an external sensor group and an internal sensor group. As discussed further below, the external sensor group detects objects and conditions in the vehicle's external environment using one or more sensors. The internal sensor group may detect objects and conditions within the vehicle, such as within the passenger compartment, using one or more other sensors.

[0028] 4 illustrates an example of a perception system 324. For example, as illustrated, by way of example only, each domain of the perception system 324 may include one or more light detection and ranging (Lidar) sensors 400, radar units 402, cameras 404 (e.g., optical imaging devices with or without neutral density filter (ND) filters), positioning sensors 406 (e.g., gyroscopes, accelerometers, and / or other inertial components), infrared sensors 408, acoustic sensors 410 (e.g., microphones or sonar transducers), and / or any other detection devices 412 that record data that can be processed by the computing device 302. The sensors of the perception system 324 in the external sensor group can detect objects outside the vehicle and characteristics of the objects such as location, orientation, size, shape, type (e.g., vehicle, pedestrian, bicyclist, etc.), orientation movement speed, etc. The sensors of the internal sensor group can detect objects (e.g., people, pets, luggage) and conditions (e.g., temperature, humidity, etc.) within the vehicle.

[0029] The raw data from the sensors and the aforementioned characteristics may be processed by the perception system 324 and / or transmitted periodically and continuously as data is generated by the perception system 324 to the computing device 302 for further processing. The computing device 302 may use the positioning system 322 to determine the location of the vehicle and use the perception system 324 to detect and respond to objects when necessary to safely arrive at that location. Additionally, the computing device 302 may perform calibration of individual sensors, all sensors within a particular sensor assembly, or between sensors in different sensor assemblies or other physical housings.

[0030] As shown in FIGS. 1-2, the particular sensors of the perception system 324 may be incorporated into one or more sensor assemblies or housings. In one example, these may be arranged as sensor towers incorporated into the vehicle's side view mirrors. In another example, other sensors may be part of the rooftop housing 102. The computing device 202 may communicate with sensor assemblies positioned on or otherwise distributed along the vehicle. Each assembly may have one or more types of sensors, such as those described above.

[0031] Returning to FIG. 3, the computing device 302 may include all of the components typically used in connection with a computing device, such as the processor and memory described above, as well as a user interface subsystem 334. The user interface subsystem 334 may include one or more user inputs 336 (e.g., a mouse, keyboard, touch screen, and / or microphone), as well as one or more display devices 338 (e.g., a monitor having a screen, or any other electrical device operable to display information). In this regard, an internal electronic display may be positioned within the vehicle's interior (not shown) and may be used by the computing device 302 to provide information to passengers within the vehicle. Other output devices, such as speaker(s) 340, may also be positioned within the passenger vehicle.

[0032] The passenger vehicle also includes a communications system 342. For example, the communications system 342 may also include one or more wireless network connections to facilitate communication with other computing devices, e.g., a passenger computing device in the vehicle, a computing device outside the vehicle, such as in another nearby vehicle on the road, or a remote server system. The wireless network connections may include a variety of configurations and protocols, including Bluetooth, Bluetooth Low Energy (LE), cellular connections, as well as short-range communications protocols such as the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, a private network using one or more company-proprietary communications protocols, Ethernet, WiFi, and HTTP, as well as various combinations of the foregoing.

[0033] As shown in FIG. 3, the system may include one or more buses 344 for transmitting information and / or power. The buses may provide direct or indirect connections between various components and subsystems. For example, a data communication bus may provide bidirectional communication between the cameras and other sensors of the perception system 324 and the computing device 302. The power lines may be directly or indirectly connected to the power distribution elements 327 of the power system 326 or may be connected to a separate power source such as a battery controlled by the computing device 302. Various protocols may be used for unidirectional or bidirectional data communication. As an example, a protocol using the Controller Area Network (CAN) bus architecture or an Ethernet-based technology such as 100Base-T1 (or 1000Base-T1 or 10GBase-T) Ethernet may be used. Other protocols such as FlexRay may also be used. Additionally, the Automotive Audio Bus (A2B) and / or other bus configurations may be used.

[0034] Exemplary Implementations In view of the structures and configurations described above and illustrated in the figures, various implementations are now described in accordance with aspects of the present technology.

[0035] Partial Redundancy - Sensor Fallback Mode The environment around the vehicle can be thought of as having different quadrants or regions. An example of this is shown in Figure 5A, which shows the front, rear, right and left regions as well as adjacent areas of the front right, front left, rear right and rear left areas around the vehicle. These regions are merely exemplary.

[0036] Various sensors may be positioned at different locations around the vehicle to collect data from some or all of these regions (see Figures 1-2). For example, as seen in Figure 5B, the three sensors 116 of Figure 1 may receive data primarily from the forward, front-left, and front-right regions around the vehicle.

[0037] Certain sensors may have different fields of view depending on their placement around the vehicle and the type of information they are designed to collect. For example, different Lidar sensors may be used for close-range (short-range) detection of objects adjacent to the vehicle (e.g., less than 2-10 meters) and other sensors may be used for long-range (long-range) detection of objects 100 meters (or more or less) ahead of the vehicle. Mid-range Lidar may also be used. For long-range object detection, multiple radar units may be positioned in front, behind, and / or to the sides of the vehicle. Cameras may also be positioned to provide a good view of the vehicle's surroundings. Depending on the configuration, certain types of sensors may include multiple individual sensors with overlapping fields of view. Alternatively, other sensors may provide redundant 360° fields of view.

[0038] FIG. 6 provides an example of a sensor field of view 600 associated with the sensors shown in FIG. 2. Here, where the rooftop housing 102 includes a Lidar sensor, as well as various cameras, radar units, infrared and / or acoustic sensors, each of these sensors may have a different field of view. Thus, as shown, the Lidar sensor may provide a 360° FOV 602, and the cameras located within the housing 102 may have individual FOVs 604. The sensor within the housing 104 at the front end of the vehicle has a forward facing FOV 606, and the sensor at the rear end within the housing 112 has a rear facing FOV 608. The housings 106a, 106b on the driver's and passenger's sides of the vehicle may each incorporate Lidar and / or other sensors with a respective FOV 610a or 610b. Similarly, the sensors within the housings 108a, 108b positioned toward the rear roof portion of the vehicle each have a respective FOV 612a or 612b. And, the sensor constellation units 116 positioned along the forward facing direction of the vehicle may have respective FOVs 614, 616, and 618. Each of these fields of view are merely exemplary and not to scale with respect to coverage area.

[0039] As mentioned above, multiple sensors can be arranged within a given housing or as an assembly. An example is shown in FIG. 7. This figure shows an example sensor assembly 700 according to an embodiment of the disclosure. As shown, the sensor assembly includes a housing 702 that is attached to a portion 704 of the roof of the vehicle, as shown by the dashed lines. The housing 702 may be dome-shaped, cylindrical, hemispherical, as shown, or may have a different profile. Within the housing 702 is a first sensor 706 that is located remotely or away from the roof and a second sensor 708 that is located closer to the roof. One or both of the sensors 706 and 708 may be Lidar or other types of sensors. Disposed between the first sensor 706 and the second sensor 708 is an imaging assembly 710. The camera assembly 710 includes a set of one or more cameras arranged along it. The housing 702 may be optically transparent at least along where the cameras are located. Although not shown in Figure 7, one or more processors, such as processor 304 of Figure 3, may be included as part of the sensor assembly. The processors may be configured to process raw images received from the various image sensors of the camera assembly, as well as information received from other sensors throughout the sensor assembly.

[0040] Depending on the configuration, the various sensors can be positioned to provide complementary and / or overlapping fields of view. For example, the camera assembly 710 can include a first subsystem having multiple pairs of image sensors positioned to provide an overall 360° view of the vehicle's surroundings. The camera assembly 710 can also include a second subsystem of image sensors facing generally forward of the vehicle to provide higher resolution, different exposures, different filters and / or other additional features, for example, with a forward field of view of approximately 90°, for better identifying objects on the road ahead. The field of view of this subsystem can also be greater or less than 90°, for example, approximately 60-135°. FIG. 8 provides examples 800 of various image sensor orientations of the first and second subsystems. The image sensors can be CMOS sensors, although CCD or other types of imaging elements can be used.

[0041] The height of the camera, Lidar, and / or other sensor subsystems will vary depending on the placement of the various sensors on the vehicle and the type of vehicle. For example, when the camera assembly 710 is mounted on or above the roof of a large SUV, it will typically be at a higher height than when the camera assembly is mounted on the roof of a sedan or sports car. Also, due to placement and construction limitations, the field of view may not be equal around all areas of the vehicle. By varying the diameter of the camera assembly 710 and its placement on the vehicle, a suitable 360° field of view can be obtained. For example, the diameter of the camera assembly 710 may vary, for example, from approximately 0.25 to 1.0 meters. The diameter may be selected to be larger or smaller depending on the type of vehicle on which the camera assembly is placed and the particular location on the vehicle where it is located.

[0042] As shown in FIG. 8, each image sensor pair of the first subsystem of the camera assembly includes a first image sensor 802 and a second image sensor 804. The first and second image sensors may be part of separate camera elements or may be included together in one camera module. In this scenario, the first image sensor 802 may be set to auto exposure and the second image sensor 804 may be set to fixed exposure, for example, using a dark color or ND filter. As shown, eight pairs of image sensors are shown, but more or fewer pairs may be used. The second image subsystem includes an image sensor 806 that may have a higher resolution than that of the first and second image sensors. This enhanced resolution may be particularly beneficial for a camera facing forward of the vehicle to provide the perception system 324 with as much detail as possible of the scene ahead of the vehicle. FIG. 8 shows three image sensors of the second subsystem, but more or fewer image sensors may be used. In this example, a total of 19 image sensors are incorporated into the camera assembly 710, including three from the second subsystem and eight pairs from the first subsystem. Again, more or fewer image sensors may be used in the camera assembly.

[0043] The exact field of view of each image sensor may vary depending on, for example, the characteristics of the particular sensor. By way of example, image sensors 802 and 804 may have an FOV of about 50°, e.g., 49°-51°, while image sensor 806 may each have an FOV of, for example, around 30° or slightly more, e.g., 5-10% more. This allows for overlap in the FOVs of adjacent image sensors.

[0044] The selected amount of overlap is beneficial because it is undesirable to have seams or gaps in the images or other data generated by the various sensors. In addition, the selected overlap allows the processing system to avoid stitching of images. Image stitching may be performed in traditional panoramic image processing, but can be computationally difficult to perform in real-time situations when the vehicle is operating in an autonomous driving mode. By reducing the amount of time and processing resources required, the responsiveness of the perception system when the vehicle is operating is greatly improved.

[0045] 9A-9B show two examples of image sensor overlap between adjacent sensor regions. As shown, for an image sensor covering the forward and forward right regions around a vehicle, there may be an overlap 900 of 0.5-5° (FIG. 9A), or alternatively, an overlap of 6-10° or less. This overlap 900 may apply to one type of sensor, such as image sensor 802 of FIG. 8. A different overlap may apply to another type of sensor. For example, for image sensor 804 of FIG. 8, there may be an overlap 902 of 2-8° (FIG. 9B), or alternatively, an overlap of 8-12° or less. Larger or smaller overlaps may also be used in systems depending on vehicle type, size, sensor type, etc.

[0046] Example scenario As noted above, when a sensor or other component fails or experiences a degradation in functionality, the vehicle's driving capabilities may be limited or operation may be prevented entirely. For example, one or more sensors may experience errors due to, for example, mechanical or electrical failure, degradation due to environmental conditions (such as extreme cold, snow, ice, mud, or dust blockages), or other factors. In such situations, a fallback configuration is designed to provide at least a minimum amount of sensor information so that the perception and planning system can operate according to a given mode of operation. The mode of operation may include, for example, completing the current driving activity (e.g., dropping off the passenger at a desired location) before receiving service, modifying the route and / or speed (e.g., exiting a highway and driving along a surface road, slowing to the minimum speed limit posted on the route, etc.), or stopping the vehicle as soon as it is safe to do so.

[0047] As an example, a fallback configuration may be associated with two different domains, e.g., Domain A and Domain B (see Figures 3-4). Each domain need not be a mirror image of the other domain. For example, a particular set of forward-facing sensors may be assigned to Domain A, and another set of forward-facing sensors with different capabilities may be assigned to Domain B. In one scenario, sensors of each type may be included in each domain. In another scenario, at least one (set of) forward-facing camera(s) is always available in each domain for traffic light detection. In a further scenario, at least one sensor with a 360° field of view is assigned to each domain for detection and classification of objects external to the vehicle. And in yet another scenario, one or more sensors may each be present operatively as part of both domains. Thus, there may be differences in functionality based on whether the system is using sensors from Domain A only, Domain B only, or a combination of Domains A and B.

[0048] For example, in one scenario, consider FIG. 6 illustrates a standard mode of operation in which various sensors from both (or all) domains are used by the vehicle's perception and planning system. FIG. 10A illustrates an example of Domain A operation 1000. And FIG. 10B illustrates an example of Domain B operation 1010. Here, it can be seen that different sensors or groups of sensors may be used in only one domain or in both domains. As an example, the Lidar sensor of FIG. 6 may provide a 360° FOV for both domains. However, in Domain A, it may only provide ½ the amount of vertical resolution 1002 as in standard operation. Similarly, in Domain B, the Lidar sensor provides only ½ the amount of vertical resolution 1012 as in standard operation.

[0049] And, although the cameras in the housing 102 (FIG. 2) have individual fields of view 604 that may provide an overall 360° FOV, each domain may use a different set of cameras. Here, for example, domain A may include a first image sensor 802 (e.g., auto exposure) to provide a first image sensing function 1004, while domain B may include a second image sensor 804 (e.g., set to fixed exposure) to provide a second image sensing function 1014. However, both domain A and domain B may include an image sensor 806. This is because the image sensor 806 may have a higher resolution than the resolution of image sensors 802 and 804 and may face forward of the vehicle. This enhanced resolution is particularly useful for detecting street lights, pedestrians, cyclists, etc., in front of the vehicle. Thus, as shown in both FIGS. 10A and 10B, each domain may have an image sensing function 1006 from such a high-resolution image sensor. In another example, domain B may include image sensing functionality 1006 that domain A does not include, which includes other forward image sensing functionality.

[0050] 11A-11D show how other sensors, such as radar sensors, can be used in different domains. FIG. 11A shows a combined FOV 1100 of a set of six sensors 1101-1106, each with its own FOV 1111-1116. The combined FOV 1100 may be available in a standard mode of operation. During operation in domain A, only sensors 1101, 1102, 1104, and 1106 are used, resulting in domain A FOV configuration 1110, as shown in FIG. 11B. And during operation in domain B, only sensors 1101, 1103, 1105, and 1106 are used, resulting in domain B FOV 1120, as shown in FIG. 11C. In these examples of domains A and B, both forward-facing sensors are utilized to provide overlapping fields of view.

[0051] In this scenario, either the Domain A or Domain B configurations may individually provide enough sensor data for the vehicle to operate in a first fallback mode. For example, the vehicle may continue to travel on the highway, but may drive in a slow lane, at a minimum speed limit, or below another speed threshold. Or, the vehicle may be able to select an alternative route to the destination with fewer turns or fewer expected adjacent objects (e.g., fewer cars), for example, by utilizing surface roads rather than the highway. In contrast, FIG. 11D illustrates a different scenario 1120 of the second fallback mode, where only the forward-facing sensors 1101 and 1106 are available. In this fallback mode, since only the fields of view 1111 and 1116 are providing sensor input to the vehicle, the system may make significant changes to its driving behavior. This may include, for example, selecting a nearby drop-off location different from the planned destination, turning on hazard lights with the signaling system, etc.

[0052] While the above examples consider Lidar, camera, and radar sensors for different fallback scenarios, other types of sensors, e.g., positioning, acoustic, infrared, etc., may also be allocated between different domains to provide sufficient partial redundancy to control the vehicle in a given operating mode.

[0053] Partial Redundancy - Fallback Mode for Computer Systems Other aspects of the present technology include redundancy built into the computing systems. During typical operation, a first computing system (e.g., a planner subsystem) can generate a trajectory and send the trajectory to a second computing system to control the vehicle according to the trajectory. Each of these computing systems can be part of the computing device 302 (FIG. 3). For redundancy, the two subsystems can each have one or more processors and associated memory. In this example, each subsystem is powered and operates independently but shares sensor data and resources to handle basic vehicle operations. In the event of a failure in one of the subsystems, for example due to a CPU crash, kernel error, power failure, etc., the other subsystem can control the vehicle in a designated fallback operating state.

[0054] In one arrangement, both computational subsystems can control the vehicle in standard operating and fallback modes. Each subsystem is associated with its respective domain, such as via one or more CAN or FlexRay buses. However, the sensor populations in each domain need not be identical, complementary, or completely overlapping. For example, certain "fallback" sensors may be assigned to control subsystem A (e.g., sensors 1102, 1104, and 1106 in FIG. 11B), other fallback sensors may be assigned to control subsystem B (e.g., sensors 1101, 1103, and 1105 in FIG. 11C), and other "non-fallback" sensors may be assigned only to control subsystem A (see FIG. 12).

[0055] Unlike fallback sensors, non-fallback sensors do not need to be assigned to any particular domain, nor do they require domain independence or redundancy. This is because non-fallback sensors are not relevant to providing a minimum viable fallback capability, but rather may only be used for standard operating modes. Nevertheless, if there are other reasons (e.g., vehicle integration) to place such sensors in one domain or the other, they can be accommodated without affecting fallback operation. As an example, one domain may have more power headroom than another, so that domain can more easily accommodate non-fallback sensors. As another example, it may be easier to wire to one particular domain, so non-fallback sensors can be associated with that domain.

[0056] FIG. 12 illustrates an example of an autonomous driving system configuration 1200. As illustrated, each domain includes one or more processors 1202a or 1202b, which may be processors 304 from the computing system 302. Each domain also includes a respective operational mode logic 1204a, 1204b, which may also comprise software or firmware modules for execution by the processors 1202a, 1202b. The operational mode logic 1204a, 1204b may include separate components for executing in a standard operational mode and one or more fallback operational modes. The fallback modes of the different domains may involve operating the vehicle in different manners, for example, according to the type and functionality of fallback sensors associated with the respective domain. And, as illustrated in FIG. 12, one of the domains (e.g., domain A) may include different computational resources, such as one or more graphic processing units (GPUs) 1206 or other computing accelerators (e.g., ASICs, FPGAs, etc.). Here, one set of fallback sensors is associated with domain A, another set of fallback sensors is associated with domain B, and one or more non-fallback sensors are also associated with domain A, although other configurations are possible, e.g., fallback sensors are assigned to domain B, or a subset is assigned to domain A and another subset is assigned to domain B. The two (or more) computational domains need not have the same performance, or the same types of computational elements, although in certain configurations they may have comparable performance and / or the same types of computational elements.

[0057] By way of example only, in this configuration, domain A may support a perception system (e.g., perception system 324 of FIGS. 3-4) during standard operations, and domain B may support a planner (e.g., route planning based on navigation system 320 and positioning system 322 of FIG. 3) during standard operations. GPU(s) 1206 may be configured to process sensor data received from some or all of the on-board sensors during standard operations.

[0058] In one scenario, each computing subsystem may have enough sensor inputs and enough computing capabilities (e.g., processor and memory resources to process received sensor data) to operate the vehicle in the corresponding standard and fallback operating modes. Two (or more) domains may share information in the standard mode. In this manner, the various domains and subsystems may be used to provide optical FOV coverage. According to aspects of the present technology, some computing resources may be kept in reserve to handle fallback operations. Also, in the case of fallback, some typical operations of the standard mode may cease. For example, one or more forward-facing high-resolution cameras (e.g., camera 806 in FIG. 8) may be considered non-fallback-critical sensors. As a result, processing of inputs from these cameras may cease upon entering fallback mode.

[0059] Partial Redundancy - Power Distribution Fallback Mode With respect to power distribution, aspects of the present technology provide two fault-independent power sources, each with a battery backup. By way of example, the dual power sources are operationally separated such that a primary fault is limited to only one domain. Each power source can serve a specific set of base vehicle loads. FIG. 13A illustrates an example of a redundant power distribution architecture 1300. As illustrated in this scenario, each domain has its own independent power source. Here, the power sources and loads of each domain are protected by an intervening electronic fuse. In practice, during normal operation, the electronic fuse is a closed circuit. When the electronic fuse detects a fault in the form of overcurrent, undervoltage, overheating (or other abnormal condition), it immediately reacts and opens the circuit, thereby isolating the two domains.

[0060] Each power supply has a battery backup and is configured to provide automotive standards on the order of 12 volts (e.g., in the range of 8-16 volts). Each domain has a separate power distribution block 1302a, 1302b, such as power distribution element 327 of FIG.

[0061] The DC / DC unit is a voltage converter that receives power from an upstream high-voltage battery pack and converts it to a low voltage (e.g., 12V) to charge the 12V battery. In an electronic fuse architecture, due to the presence of the electronic fuse, the DC / DC unit can charge half the system when everything is working (no faults), so two DC / DC units are not needed. In the event of a failure, the system does not necessarily need to use the DC / DC unit, as power can be provided by redundant low-voltage backup batteries in each domain.

[0062] Another example of power redundancy that does not require electronic fuses is a dual independent DC / DC system 1310 as shown in Figure 13B, where each domain is supplied with its own DC power source. In this example, each domain has a separate power distribution block 1312a, 1312b. System ground is provided so that the return current paths of the domains do not present a single point of failure.

[0063] There may be fallback critical actuators, such as braking, steering, and / or propulsion. There may also be non-fallback critical actuators, such as the horn, cabin lights, heating and air conditioning systems, etc. Actuators considered fallback critical may be redundant (e.g., two or more individual actuators) and / or such actuators may receive power from both domains. In contrast, actuators that are not fallback critical may not have redundant components and / or may receive power only from a single domain.

[0064] There may also be redundancy in other subsystems of the vehicle. For example, braking and steering subsystems may be redundant (in addition to being redundantly powered). Similarly, communications with other vehicles or remote assistance devices may use multiple cellular or other types of communications links. Two or more GPS receivers may be used. Even wipers, sprayers, or other cleaning components may be configured for redundancy and powered (as a base load) in one or both (or more) of the domains. The base vehicle load may include individual sensors, sensor groups, computing devices, actuators, and / or other components as discussed with respect to FIG. 3.

[0065] FIG. 14 illustrates a method 1400 of operating a vehicle according to aspects of the present technology. As shown in block 1402, information about the vehicle's environment is detected by the vehicle's sensors (e.g., Lidar, radar, camera, acoustic, and / or positioning sensors). In block 1404, the detected information is received by a control system of the vehicle, such as the computing system(s) 302 of FIG. 3 or the system 1200 of FIG. 12. Per block 1406, in response to receiving the sensor data in the first operating mode, a driving system of the vehicle is autonomously controlled. In block 1408, an error condition of one or more of the vehicle's sensors is detected. This may be due to, for example, a failure, a fault, or other degradation of sensor functionality.

[0066] In block 1410, upon detecting the error condition, the first computing subsystem processes sensor data from a first set of sensors in a first operational domain, and the second computing subsystem processes sensor data from a second set of sensors in a second operational domain. As a result, in block 1412, in response to the error condition, one of the first or second computing subsystems controls the vehicle's driving system in a fallback driving mode. In some examples, the first computing subsystem may operate a function (e.g., planner, perception, etc.) in a standard operating mode and the second computing subsystem may operate another function in the standard operating mode prior to detecting the error condition.

[0067] Unless otherwise stated, the foregoing alternatives are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. These and other variations and combinations of the features discussed above may be utilized without departing from the subject matter defined by the claims, and therefore the foregoing description of the embodiments should be considered as illustrative, and not limiting, of the subject matter defined by the claims. In addition, the examples described herein, as well as the provision of phrases such as "including," "including," and the like, should not be construed as limiting the subject matter of the claims to a particular example, but rather, the example is intended to illustrate only one of many possible embodiments. Furthermore, the same reference numbers in different drawings may identify the same or similar elements. Processes or other operations may be performed in different orders or simultaneously, unless otherwise specified herein.

Claims

1. 1. A vehicle configured to operate in an autonomous driving mode, the vehicle comprising: a driving system including a steering subsystem, an acceleration subsystem, and a deceleration subsystem for controlling driving of the vehicle in the autonomous driving mode; a perception system having a plurality of sensors configured to detect information regarding an environment surrounding the vehicle, the plurality of sensors including a first set of sensors associated with a first operational domain and a second set of sensors associated with a second operational domain; a control system operably coupled to the driving system and the perception system, the control system including a first computing subsystem associated with the first operational domain and a second computing subsystem associated with the second operational domain, each of the first and second computing subsystems having one or more processors; the first and second computing subsystems are each configured to receive sensor data from one or both of the first set of sensors and the second set of sensors in a first operating mode, and in response to the received sensor data in the first operating mode, the control system is configured to control the driving system to drive the vehicle in the autonomous driving mode; in response to an error condition for one or more of the plurality of sensors, the first computing subsystem is configured to process sensor data from only the first set of sensors in the first operational domain, and the second computing subsystem is configured to process sensor data from only the second set of sensors in the second operational domain; In response to the error condition, only one of the first computing subsystem or the second computing subsystem is configured to control the driving system in a fallback driving mode.

2. The vehicle of claim 1 , wherein the first and second sets of sensors each include at least one sensor selected from the group consisting of a Lidar sensor, a radar sensor, a camera sensor, an auditory sensor, and a positioning sensor.

3. 3. The vehicle of claim 2, wherein the first and second sets of sensors each include a respective group of Lidar, radar, and camera sensors, each group providing a selected view of the environment surrounding the vehicle.

4. 2. The vehicle of claim 1, wherein each of the sensors in the first set of sensors has a respective field of view and each of the sensors in the second set of sensors has a respective field of view, the respective field of view of the second set of sensors being different from the respective field of view of the first set of sensors.

5. The vehicle of claim 1 , further comprising one or more interior sensors disposed within the vehicle, the one or more interior sensors including at least one of a camera sensor, an auditory sensor, and an infrared sensor.

6. the fallback driving modes include a first fallback mode and a second fallback mode, the first fallback mode including a first set of driving behaviors and the second fallback mode including a second set of driving behaviors that are different from the first set of driving behaviors; the first computing subsystem is configured to control the driving system in the first fallback mode; The vehicle of claim 1 , wherein the second computing subsystem is configured to control the driving system in the second fallback mode.

7. further comprising first and second power distribution subsystems; in the fallback mode of operation, the first power distribution subsystem is associated with the first domain of operation to supply power only to devices in the first domain of operation; in the fallback mode of operation, the second power distribution system is associated with the second domain of operation to supply power only to devices in the second domain of operation; The vehicle of claim 1 , wherein the first and second operational domains are electrically isolated from one another.

8. the first power distribution subsystem is configured to provide power to a first set of base vehicle loads in the first operating mode and to provide power to devices of the first operating domain in the fallback operating mode; 8. The vehicle of claim 7, wherein the second power distribution subsystem is configured to provide power to a second set of base vehicle loads different from the first set of base vehicle loads in the first operational mode and to provide power to devices of the second operational domain in the fallback driving mode.

9. The sensory system, the plurality of sensors, a first set of fallback sensors operably coupled to the first computing subsystem; a second set of fallback sensors operably coupled to the second computing subsystem; and a set of non-fallback sensors.

10. The vehicle of claim 9 , wherein the set of non-fallback sensors is operatively coupled to one or both of the first computing subsystem and the second computing subsystem.

11. 2. The vehicle of claim 1, wherein the plurality of sensors of the perception system includes a subset of sensors operatively coupled to both the first and second computing subsystems in the fallback driving mode.

12. 1. A method of operating a vehicle in an autonomous driving mode, the method comprising: Detecting information about an environment surrounding the vehicle by a plurality of sensors of a perception system of the vehicle, the plurality of sensors including a first set of sensors associated with a first operational domain and a second set of sensors associated with a second operational domain; receiving, by a control system of the vehicle, the sensed information about the environment surrounding the vehicle as sensor data, the control system including a first computing subsystem associated with the first operational domain and a second computing subsystem associated with the second operational domain; In response to receiving the sensor data in a first operating mode, the control system controls a driving system of the vehicle to drive the vehicle in the autonomous driving mode; detecting an error condition for one or more of the plurality of sensors; Upon detecting the error condition, the first computing subsystem processes sensor data from only the first set of sensors in the first operational domain and the second computing subsystem processes sensor data from only the second set of sensors in the second operational domain; in response to the error condition, only one of the first computing subsystem or the second computing subsystem controlling the operating system in a fallback operating mode.

13. The method of claim 12 , wherein the fallback operating mode comprises a plurality of fallback modes including a first fallback mode and a second fallback mode.

14. 14. The method of claim 13, wherein the first fallback mode includes a first set of driving behaviors and the second fallback mode includes a second set of driving behaviors that are different from the first set of driving behaviors.

15. Controlling the operating system in the fallback operating mode includes: the first computing subsystem controlling the driving system in the first fallback mode; or The method of claim 14 , further comprising: the second computing subsystem controlling the operating system in the second fallback mode.

16. In the fallback operation mode, providing power to only devices in the first operational domain by a first power distribution subsystem of the vehicle; The method of claim 12 , further comprising: supplying power by a second power distribution subsystem of the vehicle to only devices in the second operational domain.

17. the first power distribution subsystem provides power to a first set of base vehicle loads in the first operating mode and provides power to devices of the first operating domain in the fallback operating mode; 17. The method of claim 16, wherein the second power distribution subsystem provides power to a second set of base vehicle loads different from the first set of base vehicle loads in the first operating mode and provides power to devices of the second operating domain in the fallback operating mode.

18. The method of claim 12 , wherein during the fallback driving mode, a subset of the plurality of sensors of the perception system provides sensor data to both the first and second computing subsystems.

19. 13. The method of claim 12, wherein controlling the driving system in the fallback driving mode includes at least one of altering a previously planned trajectory of the vehicle, altering a speed of the vehicle, or altering a destination of the vehicle.

20. The method of claim 12 , further comprising: ceasing processing of sensor data from non-fallback-critical sensors during the fallback operating mode.

Images