Redundant hardware systems for autonomous vehicles
A partially redundant architecture with fallback configurations for sensors and computing systems addresses the challenge of component failures in autonomous vehicles, ensuring safe navigation by maintaining minimal functionality.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- WAYMO LLC
- Filing Date
- 2024-10-03
- Publication Date
- 2026-05-26
Smart Images
Figure 0007866018000001 
Figure 0007866018000002 
Figure 0007866018000003
Abstract
Description
Technical Field
[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Patent Application No. 16 / 215,713, filed on December 11, 2018, and is related to co - pending U.S. Patent Application No. 16 / 180,267 (Attorney Docket No. XSDV3.0F - 2059), entitled "Systems for Implementing Fallback Behaviors for Autonomous Vehicles", filed on November 5, 2018, the entire disclosure of which is incorporated herein by reference.
Background Art
[0002] Autonomous vehicles, such as vehicles that do not require a human driver, can be used to assist in the transportation of passengers or cargo from one location to another. Such vehicles may operate in a fully autonomous mode or in a semi - autonomous mode where a human can provide some driving input. To operate in autonomous mode, the vehicle uses sensors and can perform various driving operations using the received sensor information. However, if a sensor or other component of the system fails or otherwise experiences a degradation in performance, this can negatively impact the vehicle's driving function. [Prior Art Documents] [Patent Document 1] U.S. Patent Application Publication No. 2017 / 0270014
Summary of the Invention
[0003] This technology relates to redundant architectures for the sensor, computing, and power systems of vehicles configured to operate in fully or partially autonomous driving modes. While it may be possible to fully redundant all components and subsystems, this may not be feasible in vehicles where there are constraints, particularly regarding the size and placement of sensor arrays, as well as other limiting factors such as cost. Therefore, aspects of this technology employ a fallback configuration for partial redundancy. For example, a fallback sensor configuration can not only provide some minimal field of view (FOV) around the vehicle, but also provide minimal computing power for perception and planning processing.
[0004] According to aspects of this technology, a vehicle is configured to operate in autonomous driving mode. The vehicle comprises a driving system, a perception system, and a control system. The driving system includes a steering subsystem, an acceleration subsystem, and a deceleration subsystem for controlling the driving of the vehicle in autonomous driving mode. The perception system has a plurality of sensors configured to detect information about the environment around the vehicle. The plurality of sensors include a first set of sensors associated with a first operating domain and a second set of sensors associated with a second operating domain. The control system is operably coupled to the driving system and the perception system. The control system includes a first computing subsystem associated with a first operating domain and a second computing subsystem associated with a second operating domain. Each of the first and second computing subsystems has one or more processors. Each of the first and second computing subsystems is configured to receive sensor data from one or both of the first set of sensors and the second set of sensors in the first operating mode. In response to the sensor data received in the first operating mode, the control system is configured to control the driving system to drive the vehicle in autonomous driving mode. Depending on the error status of one or more of the multiple sensors, a first computing subsystem is configured to process sensor data from a first set of sensors in a first operating domain, and a second computing subsystem is configured to process sensor data from a second set of sensors in a second operating domain. Then, in response to the error status, only one of the first or second computing subsystems is configured to control the operating system in fallback operating mode.
[0005] In the example, each of the first and second sets of sensors includes at least one sensor selected from the group consisting of Lidar sensors, radar sensors, camera sensors, auditory sensors, and positioning sensors. Here, each of the first and second sets of sensors may include the respective groups of Lidar, radar, and camera sensors, each group providing a selected field of view of the environment around the vehicle.
[0006] Each of the sensors in the first set may have its own field of view, and each of the sensors in the second set may have its own field of view, in which case each field of view of the sensors in the second set is different from each field of view of the sensors in the first set. One or more internal sensors may be disposed inside the vehicle. One or more internal sensors include at least one of a camera sensor, an auditory sensor, and an infrared sensor.
[0007] In another example, the fallback operating modes include a first fallback mode and a second fallback mode. The first fallback mode includes a first set of operating operations, and the second fallback mode includes a second set of operating operations that differ from the first set of operating operations. In this example, a first computing subsystem is configured to control the operating system in the first fallback mode, and a second computing subsystem is configured to control the operating system in the second fallback mode.
[0008] In yet another example, the vehicle further comprises first and second power distribution subsystems. Here, in fallback operation mode, the first power distribution subsystem is associated with a first operating domain and supplies power only to devices in the first operating domain. Also in fallback operation mode, the second power distribution subsystem is associated with a second operating domain and supplies power only to devices in the second operating domain. The first and second operating domains are electrically isolated from each other. In this case, the first power distribution subsystem may be configured to supply power to a first set of base vehicle loads in the first operating mode and to supply power to devices in the first operating domain in fallback operation mode, and the second power distribution subsystem may be configured to supply power to a second set of base vehicle loads, different from the first set of base vehicle loads, in the first operating mode and to supply power to devices in the second operating domain in fallback operation mode.
[0009] In a further example, the sensors of a perception system include a first set of fallback sensors operably connected to a first computing subsystem, a second set of fallback sensors operably connected to a second computing subsystem, and a set of non-fallback sensors. In this scenario, the set of non-fallback sensors may be operably connected to either or both of the first and second computing subsystems. In yet another example, the sensors of a perception system include a subset of sensors operably connected to both the first and second computing subsystems in fallback operation mode.
[0010] According to another aspect of this technology, a method for operating a vehicle in autonomous driving mode is provided. The method includes detecting information about the environment surrounding a vehicle using a plurality of sensors in the vehicle's perception system, wherein the plurality of sensors include a first set of sensors associated with a first operating domain and a second set of sensors associated with a second operating domain; receiving the detected information about the environment surrounding the vehicle as sensor data by the vehicle's control system, wherein the control system includes a first computing subsystem associated with a first operating domain and a second computing subsystem associated with a second operating domain; in response to receiving sensor data in a first operating mode, the control system controlling the vehicle's driving system to drive the vehicle in an autonomous driving mode; detecting an error condition for one or more of the plurality of sensors; when an error condition is detected, the first computing subsystem processing sensor data only from the first set of sensors in the first operating domain and the second computing subsystem processing sensor data only from the second set of sensors in the second operating domain; and in response to the error condition, either the first or the second computing subsystem controlling the driving system in a fallback driving mode.
[0011] In one example, a fallback operating mode may include multiple fallback modes, including a first fallback mode and a second fallback mode. In this case, the first fallback mode may include a first set of operating operations, and the second fallback mode may include a second set of operating operations that are different from the first set of operating operations. In this case, controlling the operating system in a fallback operating mode may include a first computing subsystem controlling the operating system in the first fallback mode, or a second computing subsystem controlling the operating system in the second fallback mode.
[0012] In another example, the method may further include, in fallback operation mode, powering only devices in a first operating domain by a first distribution subsystem of the vehicle, and powering only devices in a second operating domain by a second distribution subsystem of the vehicle. In this scenario, the first distribution subsystem can power a first set of base vehicle loads in the first operating mode and power devices in the first operating domain in fallback operation mode, and the second distribution subsystem can power a second set of base vehicle loads different from the first set of base vehicle loads in the first operating mode and power devices in a second operating domain in fallback operation mode.
[0013] During fallback driving mode, a subset of multiple sensors in the perception system can provide sensor data to both the first and second computing subsystems. Controlling the driving system in fallback driving mode includes at least one of the following: altering the vehicle's previously planned trajectory, altering the vehicle's speed, or altering the vehicle's destination. This method may further include stopping the processing of sensor data from non-fallback critical sensors during fallback driving mode. [Brief explanation of the drawing]
[0014] [Figure 1] An exemplary perspective view of a vehicle configured for use in an embodiment of this technology is shown. [Figure 2] Figure 1 shows a top view of an exemplary vehicle. [Figure 3] A block diagram of an exemplary vehicle according to this technology is shown. [Figure 4] This is a block diagram of an exemplary perceptual system according to an embodiment of this technology. [Figure 5A] An example of the area surrounding a vehicle according to the embodiments of this disclosure is shown. [Figure 5B] An example of the area surrounding a vehicle according to the embodiments of this disclosure is shown. [Figure 6] An exemplary sensor field of view according to an aspect of this disclosure is shown. [Figure 7] An exemplary sensor assembly according to an aspect of this disclosure is shown. [Figure 8] An example of sensor orientation according to the embodiments of this disclosure is shown. [Figure 9A] Examples of overlapping sensor fields of view according to aspects of this disclosure are shown. [Figure 9B] Examples of overlapping sensor fields of view according to aspects of this disclosure are shown. [Figure 10A] Examples of operation in different domains based on this technology are shown. [Figure 10B] Examples of operation in different domains based on this technology are shown. [Figure 11A] Further examples of how this technology operates in different domains are shown below. [Figure 11B] Further examples of how this technology operates in different domains are shown below. [Figure 11C] Further examples of how this technology operates in different domains are shown below. [Figure 11D] Further examples of how this technology operates in different domains are shown below. [Figure 12] An exemplary autonomous driving system configuration according to this technology is shown. [Figure 13A] Shows an exemplary power distribution configuration according to an aspect of the present technology. [Figure 13B] Shows an exemplary power distribution configuration according to an aspect of the present technology. [Figure 14] Shows a method of operation according to an aspect of the present technology.
Mode for Carrying Out the Invention
[0015] The partially redundant vehicle architecture considered herein is associated with a fallback configuration that uses different sensor arrangements that can be logically associated with different operating domains of the vehicle. Each fallback configuration may be triggered for different reasons and may result in different types of fallback operating modes. The trigger conditions may relate to, for example, the type of failure, an obstacle, or other degradation of a component's function, the current autonomous driving mode, environmental conditions along the vehicle's perimeter or planned route, or other factors.
[0016] Exemplary vehicle system Figure 1 shows a perspective view of a passenger vehicle 100, such as a minivan, sedan, or sports utility vehicle. Figure 2 shows a top view of the passenger vehicle 100. The passenger vehicle 100 can include various sensors for acquiring information about the vehicle's external environment. For example, the rooftop housing 102 can include a Lidar sensor, as well as various cameras, radar units, infrared and / or acoustic sensors. Housing 104 located at the front end of the vehicle 100, as well as housings 106a, 106b on the driver's and passenger's sides of the vehicle, can each incorporate Lidar and / or other sensors. For example, housing 106a may be located in front of the driver's side door along the quarter panel of the vehicle. As shown, the passenger vehicle 100 also includes housings 108a, 108b for radar units, Lidar, and / or cameras, which are also located toward the rear roof portion of the vehicle. Additional Lidar, radar units, and / or cameras (not shown) may be located elsewhere along the vehicle 100. For example, arrow 110 indicates that a sensor unit (112 in Figure 2) may be positioned along the lead of the vehicle 100, for example, on or adjacent to the bumper. Then, arrow 114 indicates a series of sensor units 116 arranged along the forward direction of the vehicle. In some examples, the passenger vehicle 100 may also include various sensors for acquiring information about the interior space of the vehicle. The internal sensors may include at least one of camera sensors, auditory sensors, and infrared sensors.
[0017] While certain aspects of this disclosure are particularly useful in relation to specific types of vehicles, the vehicles may be any type of vehicle, including but not limited to passenger cars, trucks, motorcycles, buses, and recreational vehicles.
[0018] Figure 3 shows a block diagram 300 with various components and systems of an exemplary vehicle configured to operate in fully autonomous or semi-autonomous driving modes. For example, different degrees of autonomy may occur in a vehicle operating in a partially or fully autonomous driving mode. The U.S. National Highway Traffic Safety Administration and the Society of Automotive Engineers have identified various levels to indicate how much or how little control the vehicle has over driving. For example, Level 0 is unautomated, and the driver makes all driving-related decisions. Level 1, the lowest semi-autonomous mode, includes some drive assistance, such as cruise control. Level 2 has partial automation of certain driving actions, and Level 3 has conditional automation, which may be controllable by a person in the driver's seat as needed. In contrast, Level 4 is a high level of automation where the vehicle can drive without assistance under selected conditions. In contrast, Level 5 is a fully automated level where the vehicle can drive without assistance under all circumstances. The architectures, components, systems, and methods described herein can operate in any of the semi-autonomous or fully autonomous modes, e.g., Levels 1-5, which are referred to herein as “autonomous” driving modes. Therefore, references to autonomous driving modes include both partial and full autonomy.
[0019] As shown in Figure 3, the exemplary vehicle includes one or more computing devices 302, such as a computing device, which includes one or more processors 304, memory 306, and other components typically found in general-purpose computing devices. The memory 306 stores information accessible by one or more processors 304, which includes instructions 308 and data 310 that can be executed or otherwise used by the processor(s) 304. The computing system can control the overall operation of the vehicle when operating in autonomous mode.
[0020] Memory 306 stores information accessible by the processor 304, including instructions 308 and data 310 that can be executed by the processor 304 or otherwise used. Memory 306 can be any type of computing device-readable medium that can store information accessible by the processor. Memory is a non-transient medium such as a hard drive, memory card, optical disc, or solid-state. The system can include different combinations of the above, so that different parts of instructions and data are stored in different types of medium.
[0021] Instruction 308 may be any set of instructions that are executed directly (such as machine code) or indirectly (such as a script) by the processor. For example, instructions may be stored as computing device code on a computing device-readable medium. In this regard, the terms “instruction,” “module,” and “program” may be used interchangeably herein. Data 310 may be retrieved, stored, or modified by one or more processors 304 in accordance with instruction 308. In one example, part or all of memory 306 may be an event data recorder or other secure data storage system configured to store vehicle diagnostic and / or detected sensor data, which may be mounted in the vehicle or located remotely, depending on the implementation.
[0022] The processor 304 may be any conventional processor, such as a commercially available CPU. Alternatively, each processor may be a dedicated device, such as an ASIC or other hardware-based processor. Figure 3 functionally shows that the processor, memory, and other elements of computing device 302 are in the same block, but such a device may actually include multiple processors, computing devices, or memory, which may or may not be housed in the same physical housing. Similarly, memory 306 may be a hard drive or other storage medium located in a different housing than that of processor(s) 304. Thus, references to processors or computing devices will be understood to include references to a collection of processors or computing devices or memory, which may or may not operate in parallel.
[0023] In one example, the computing device 302 may form an autonomous driving computing system integrated into the vehicle 100. The autonomous driving computing system may be capable of communicating with various components of the vehicle. For example, the computing device 302 may be capable of communicating with various systems of the vehicle, including a driving system that includes a deceleration system 312 (for controlling the braking of the vehicle), an acceleration system 314 (for controlling the acceleration of the vehicle), a steering system 316 (for controlling the direction of the wheels and the direction of the vehicle), a signaling system 318 (for controlling the turn signals), a navigation system 320 (for navigating the vehicle to a place or around an object), and a positioning system 322 (for determining the position of the vehicle). The autonomous driving computing system may partially operate as a planner, for example, for determining a route from a starting point to a destination, according to the navigation system 320 and the positioning system 322.
[0024] The computing device 302 is also operably connected to the perception system 324 (for detecting objects in the vehicle's environment), the power system 326 (e.g., battery and / or gasoline or diesel power engine), and the transmission system 330 in order to control the vehicle's movement, speed, etc., according to instructions 308 in memory 306, in an autonomous driving mode that does not require or demands continuous or periodic input from the vehicle's passengers. The wheels / tires 328 are connected to the transmission system 330, and the computing device 302 may be able to receive information about tire pressure, balance, and other factors that may affect driving in autonomous mode. The power system 326 may have multiple power distribution elements 327, each of which can supply power to selected components and other systems of the vehicle.
[0025] The computing device 302 may control the direction and speed of the vehicle by controlling various components. For example, the computing device 302 may use map information and data from the navigation system 320 to navigate the vehicle to a destination fully autonomously. The computing device 302 may use the positioning system 322 to determine the location of the vehicle and, when it is necessary to arrive safely at that location, may use the perception system 324 to detect and respond to objects. To do so, the computing device 302 may accelerate the vehicle (e.g., by increasing the fuel or other energy supplied to the engine by the acceleration system 314), decelerate it (e.g., by reducing the fuel supplied to the engine, shifting gears, and / or braking by the deceleration system 312), change its direction (e.g., by changing the direction of the front wheels or other wheels of the vehicle 100 by the steering system 316), and signal such changes (e.g., by illuminating the turn signals of the signaling system 318). Therefore, the acceleration system 314 and the deceleration system 312 may be part of a power transmission or other type of transmission system 330 that includes various components between the vehicle's engine and the vehicle's wheels. In this case as well, by controlling these systems, the computing device 302 can also control the vehicle's transmission system 330 to autonomously steer the vehicle.
[0026] The navigation system 320 may be used by the computing device 302 to determine and follow a route to a certain location. In this regard, the navigation system 320 and / or memory 306 may store map information, for example, very detailed maps that the computing device 302 can use to navigate or control a vehicle. As an example, these maps may identify the shape and elevation of roadways, lane markings, intersections, crosswalks, speed limits, traffic lights, buildings, signs, real-time traffic information, vegetation, or other such objects and information. Lane markings may include features such as solid or dashed double or single lane markings, solid or dashed lane markings, and reflectors. A given lane may be associated with left and / or right lane markings or other lane markings that define the boundary of the lane. For this reason, most lanes may be bounded by the left end of one lane marking and the right end of another lane marking.
[0027] The perception system 324 also includes sensors for detecting objects outside the vehicle. Detected objects may include other vehicles, road obstacles, traffic lights, signs, trees, etc. As will be discussed in more detail below, the perception system 324 is configured to operate in two (or more) sensor domains, such as sensor domain A and sensor domain B, as illustrated. Within each domain, the system may include either or both an external sensor group and an internal sensor group. As will be discussed further below, the external sensor group uses one or more sensors to detect objects and conditions in the vehicle's external environment. The internal sensor group may use one or more other sensors to detect objects and conditions inside the vehicle, such as inside the passenger compartment.
[0028] Figure 4 shows an example of the perception system 324. For example, as illustrated, each domain of the perception system 324 may include one or more light-detecting and ranging (Lidar) sensors 400, a radar unit 402, a camera 404 (e.g., an optical imaging device with or without a neutral density (ND) filter), a positioning sensor 406 (e.g., a gyroscope, accelerometer, and / or other inertial components), an infrared sensor 408, an acoustic sensor 410 (e.g., a microphone or sonar transducer), and / or any other detection device 412 that records data that can be processed by the computing device 302. The sensors of the perception system 324 in the external sensor group can detect objects outside the vehicle, as well as the characteristics of the objects, such as location, orientation, size, shape, type (e.g., vehicle, pedestrian, cyclist, etc.), direction, and speed. The sensors of the internal sensor group can detect objects inside the vehicle (e.g., people, pets, luggage) and conditions inside the vehicle (e.g., temperature, humidity, etc.).
[0029] Raw data from the sensors and the aforementioned characteristics are processed by the perception system 324 and / or may be periodically and continuously transmitted to the computing device 302 for further processing as data is generated by the perception system 324. The computing device 302 can use the positioning system 322 to determine the location of the vehicle and, when it needs to arrive safely at that location, use the perception system 324 to detect and respond to objects. In addition, the computing device 302 may perform calibrations between individual sensors, all sensors in a particular sensor assembly, or between sensors in different sensor assemblies or other physical housings.
[0030] As shown in Figures 1 and 2, specific sensors of the perception system 324 may be incorporated into one or more sensor assemblies or housings. In one example, these may be arranged as a sensor tower incorporated into the vehicle's side-view mirrors. In another example, other sensors may be part of the rooftop housing 102. The computing device 202 can communicate with sensor assemblies positioned on the vehicle or otherwise distributed along the vehicle. Each assembly may have one or more types of sensors, such as those described above.
[0031] Returning to Figure 3, the computing device 302 may include the processor and memory described above, as well as all components typically used in connection with a computing device, such as the user interface subsystem 334. The user interface subsystem 334 may include one or more user inputs 336 (e.g., a mouse, keyboard, touchscreen, and / or microphone), and one or more display devices 338 (e.g., a monitor with a screen, or any other electrical device capable of displaying information). In this regard, the internal electronic display may be located inside the vehicle (not shown) and may be used by the computing device 302 to provide information to passengers inside the vehicle. Other output devices, such as speakers 340, may also be located inside the passenger vehicle.
[0032] The passenger vehicle also includes a communication system 342. For example, the communication system 342 may also include one or more wireless network connections to facilitate communication with other computing devices, such as passenger computing devices within the vehicle, computing devices outside the vehicle, such as in another nearby vehicle on the road, or remote server systems. The wireless network connections may include a variety of configurations and protocols, including Bluetooth, Bluetooth Low Energy (LE), cellular connectivity, and short-range communication protocols such as the Internet, World Wide Web, intranet, virtual private network, wide area network, local network, private network using one or more proprietary corporate communication protocols, Ethernet, WiFi, and HTTP, as well as various combinations of the foregoing.
[0033] As shown in Figure 3, the system may include one or more buses 344 for transmitting information and / or power. Buses can provide direct or indirect connections between various components and subsystems. For example, a data communication bus may provide bidirectional communication between the cameras and other sensors of the perception system 324 and the computing device 302. Power lines may be directly or indirectly connected to the distribution elements 327 of the power system 326, or to a separate power source such as a battery controlled by the computing device 302. Various protocols can be used for unidirectional or bidirectional data communication. For example, protocols using the Controller Area Network (CAN) bus architecture, or Ethernet-based technologies such as 100Base-T1 (or 1000Base-T1 or 10GBase-T) Ethernet can be used. Other protocols such as FlexRay can also be used. Furthermore, Automotive Audio Bus® (A2B) and / or other bus configurations can be used.
[0034] Example Implementation Considering the structure and configuration described above and shown in the figure, various implementation forms will be described here according to the aspects of this technology.
[0035] Partial redundancy - Sensor fallback mode The environment surrounding a vehicle can be considered to have different quadrants or regions. An example of this is shown in Figure 5A, which illustrates the front, rear, right, and left regions, as well as the adjacent areas of the front right, front left, right rear, and left rear regions around the vehicle. These regions are merely illustrative.
[0036] To collect data from some or all of these areas, various sensors may be positioned at different locations around the vehicle (see Figures 1 and 2). For example, as seen in Figure 5B, the three sensors 116 in Figure 1 can primarily receive data from the front, front left, and front right areas around the vehicle.
[0037] Certain sensors may have different fields of view depending on their placement around the vehicle and the type of information they are designed to collect. For example, different Lidar sensors may be used for short-range detection of objects adjacent to the vehicle (e.g., less than 2-10 meters), while other sensors may be used for long-range detection of objects 100 meters (or more or less) in front of the vehicle. Mid-range Lidar can also be used. For long-range object detection, multiple radar units can be positioned in front of, behind, and / or to the sides of the vehicle. Cameras can also be positioned to provide a good view of the area around the vehicle. Depending on the configuration, certain types of sensors may include multiple individual sensors with overlapping fields of view. Alternatively, other sensors may provide a redundant 360° field of view.
[0038] Figure 6 provides an example of a sensor field of view 600 related to the sensor shown in Figure 2. Here, if the rooftop housing 102 includes a Lidar sensor, as well as various cameras, radar units, infrared and / or acoustic sensors, each of these sensors may have a different field of view. Thus, as shown, the Lidar sensor may provide a 360° FOV 602, and the cameras located within the housing 102 may have separate FOVs 604. The sensor in the housing 104 at the front of the vehicle has a forward-facing FOV 606, and the sensor at the rear of the vehicle in the housing 112 has a rearward-facing FOV 608. The driver-side and passenger-side housings 106a, 106b of the vehicle may each incorporate a Lidar and / or other sensor having their respective FOVs 610a or 610b. Similarly, the sensors in the housings 108a, 108b positioned toward the rear roof portion of the vehicle each have their respective FOVs 612a or 612b. The sensor group unit 116, positioned along the direction facing forward of the vehicle, may have FOVs 614, 616, and 618, respectively. Each of these fields of view is merely illustrative and not an exact scale with respect to coverage range.
[0039] As described above, multiple sensors can be arranged within a given housing or as an assembly. An example is shown in Figure 7. This figure shows an example 700 of a sensor assembly according to an aspect of the present disclosure. As shown, the sensor assembly includes a housing 702 that is mounted on a portion 704 of the roof of a vehicle, as indicated by the dashed line. The housing 702 may be dome-shaped, cylindrical, hemispherical, or have a different shape, as shown. Inside the housing 702 are a first sensor 706 positioned remotely or away from the roof and a second sensor 708 positioned near the roof. One or both of sensors 706 and 708 may be Lidar or other types of sensors. Between the first sensor 706 and the second sensor 708 is an imaging assembly 710. The camera assembly 710 includes a set of one or more cameras positioned along it. The housing 702 may be optically transparent at least along the location where the cameras are positioned. Although not shown in Figure 7, one or more processors, such as processor 304 in Figure 3, may be included as part of the sensor assembly. The processor may be configured to process raw images received from various image sensors in the camera assembly, as well as information received from other sensors throughout the sensor assembly.
[0040] Depending on the configuration, various sensors can be arranged to provide complementary and / or overlapping fields of view. For example, camera assembly 710 may include a first subsystem having multiple pairs of image sensors positioned to provide an overall 360° field of view around the vehicle. Camera assembly 710 may also include a second subsystem of image sensors generally facing forward of the vehicle to provide higher resolution, different exposures, different filters and / or other additional features, for example, with a forward field of view of about 90°, for better identification of objects on the road ahead. The field of view of this subsystem may also be greater or less than 90°, for example, about 60 to 135°. Figure 8 provides examples of various image sensor orientations for the first and second subsystems 800. Image sensors may be CMOS sensors, although CCDs or other types of imaging elements may be used.
[0041] The height of the camera, Lidar, and / or other sensor subsystems varies depending on the placement of various sensors on the vehicle and the type of vehicle. For example, if the camera assembly 710 is mounted on or above the roof of a large SUV, it will typically be taller than when the camera assembly is mounted on the roof of a sedan or sports car. Also, due to placement and structural limitations, the field of view may not be equal around all areas of the vehicle. By varying the diameter of the camera assembly 710 and its placement on the vehicle, a suitable 360° field of view can be obtained. For example, the diameter of the camera assembly 710 may vary, for example, from approximately 0.25 to 1.0 meter. The diameter may be selected to be larger or smaller depending on the type of vehicle in which the camera assembly is placed and the specific location on the vehicle.
[0042] As shown in Figure 8, each pair of image sensors in the first subsystem of the camera assembly includes a first image sensor 802 and a second image sensor 804. The first and second image sensors may be part of separate camera elements or may be included together in a single camera module. In this scenario, the first image sensor 802 may be set to auto exposure, and the second image sensor 804 may be set to fixed exposure, for example, using a dark or ND filter. As illustrated, eight pairs of image sensors are shown, but more or fewer pairs may be used. The second image subsystem includes an image sensor 806 which may have a higher resolution than the first and second image sensors. This enhanced resolution may be particularly beneficial for a camera facing forward of the vehicle to provide the perception system 324 with as much detail as possible of the scene in front of the vehicle. Figure 8 shows three image sensors in the second subsystem, but more or fewer image sensors may be used. In this example, a total of 19 image sensors, including three pairs from the second subsystem and eight pairs from the first subsystem, are incorporated into the camera assembly 710. In this case as well, the camera assembly may use more or fewer image sensors.
[0043] The exact field of view of each image sensor may differ, for example, depending on the characteristics of the particular sensor. For instance, image sensors 802 and 804 may have an FOV of approximately 50°, e.g., 49° to 51°, while image sensor 806 may each have an FOV of approximately 30° or slightly more, e.g., 5 to 10% more. This allows for overlap in the FOVs of adjacent image sensors.
[0044] Since it is undesirable for images or other data generated by various sensors to have seams or gaps, a selected amount of overlap is beneficial. In addition, the selected overlap allows the processing system to avoid image stitching. Image stitching can be performed in conventional panoramic image processing, but performing this in real-time situations when the vehicle is operating in autonomous driving mode can be computationally difficult. By reducing the amount of time and processing resources required, the responsiveness of the vehicle's perception system while driving is significantly improved.
[0045] Figures 9A and 9B show two examples of overlap between image sensors in adjacent sensor regions. As illustrated, for an image sensor covering the front and front-right regions around a vehicle, there may be an overlap of 0.5–5° (Figure 9A), or alternatively, an overlap of 6–10° or less. This overlap can be applied to one type of sensor, such as the image sensor 802 in Figure 8. Different overlaps may be applied to other types of sensors. For example, for the image sensor 804 in Figure 8, there may be an overlap of 2–8° (Figure 9B), or alternatively, an overlap of 8–12° or less. Depending on the type and size of the vehicle, the type of sensor, etc., the overlap of the system can be increased or decreased.
[0046] Exemplary Scenario As described above, if sensors or other components fail or experience a decrease in function, the vehicle's driving capabilities may be limited or even completely halted. For example, one or more sensors may encounter errors due to, for example, mechanical or electrical failure, degradation due to environmental conditions (such as extreme cold, snow, ice, mud, or dust blockage), or other factors. In such situations, a fallback configuration is designed to provide at least a minimum amount of sensor information so that the perception and planning system can operate according to a given operating mode. Operating modes may include, for example, completing the current driving activity (e.g., dropping off passengers at their desired location) before receiving service, changing the route and / or speed (e.g., leaving a highway and driving along a ground road, slowing down to the minimum speed limit posted on the route), or stopping the vehicle as soon as it is safe.
[0047] For example, a fallback configuration may be associated with two different domains, e.g., Domain A and Domain B (see Figures 3-4). Each domain does not need to be a mirror image of the other. For example, a particular set of forward-facing sensors might be assigned to Domain A, and another set of forward-facing sensors with different functions might be assigned to Domain B. In one scenario, each type of sensor can be included in each domain. In another scenario, at least one (or more) forward-facing cameras (of which there may be) are always available in each domain for traffic signal detection. In yet another scenario, at least one sensor with a 360° field of view is assigned to each domain for detecting and classifying objects outside the vehicle. And in yet another scenario, one or more sensors may each exist operationally as part of both domains. Therefore, there may be differences in functionality based on whether the system uses sensors from Domain A only, Domain B only, or a combination of Domains A and B.
[0048] For example, consider a scenario in which Figure 6 shows a typical operating mode in which various sensors from both (or all) domains are used by the vehicle's perception and planning system. Figure 10A shows an example of domain A operation, and Figure 10B shows an example of domain B operation. Here, it can be seen that different sensors or groups of sensors may be used in only one domain or in both domains. As an example, the Lidar sensor in Figure 6 can provide a 360° FOV for both domains. However, in domain A, it may only provide half the vertical resolution of typical operation. Similarly, in domain B, the amount of vertical resolution provided by the Lidar sensor is only half of that of typical operation.
[0049] The cameras within the housing 102 (Figure 2) have individual fields of view 604, which can provide a 360° FOV as a whole, but each domain can use a different set of cameras. Here, for example, domain A may include a first image sensor 802 (e.g., auto-exposure) to provide a first image detection function 1004, while domain B may include a second image sensor 804 (e.g., set to fixed exposure) to provide a second image detection function 1014. However, both domains A and B may also include an image sensor 806, because the image sensor 806 may have a higher resolution than the image sensors 802 and 804 and may face forward of the vehicle. This enhanced resolution is particularly useful when detecting streetlights, pedestrians, cyclists, etc., in front of the vehicle. Thus, as shown in both Figures 10A and 10B, each domain can have an image detection function 1006 from such a high-resolution image sensor. In another example, domain B may include an image detection function 1006 that does not include domain A, which includes other forward image detection functions.
[0050] Figures 11A to 11D illustrate how other sensors, such as radar sensors, can be used in different domains. Figure 11A shows a combined FOV 1100 of a set of six sensors 1101 to 1106, each having a separate FOV 1111 to 1116. The combined FOV 1100 may be available in standard operating mode. During operation in domain A, as shown in Figure 11B, only sensors 1101, 1102, 1104, and 1106 are used, resulting in domain A FOV configuration 1110. Then, during operation in domain B, as shown in Figure 11C, only sensors 1101, 1103, 1105, and 1106 are used, resulting in domain B FOV 1120. In these examples for domains A and B, both forward-facing sensors are utilized, providing an overlapping field of view.
[0051] In this scenario, either the configuration of Domain A or Domain B can individually provide enough sensor data for the vehicle to operate in the first fallback mode. For example, the vehicle could continue driving on the highway but could drive in a slower lane, at the minimum speed limit, or below a different speed threshold. Alternatively, the vehicle could choose an alternative route to a destination with fewer turns or fewer expected adjacent objects (e.g., fewer cars) by using a ground road instead of a highway, for example. In contrast, Figure 11D shows a different scenario 1120 of the second fallback mode, where only forward-facing sensors 1101 and 1106 are available. In this fallback mode, since only field-of-view sensors 1111 and 1116 are providing sensor input to the vehicle, the system may make significant changes to driving behavior. This may include, for example, choosing a nearby drop-off location different from the planned destination, or activating the hazard lights with the signaling system.
[0052] In the above example, we considered Lidar, camera, and radar sensors for different fallback scenarios, but other types of sensors, such as positioning, acoustic, and infrared sensors, may also be assigned across different domains to provide sufficient partial redundancy to control the vehicle in a given operating mode.
[0053] Partial redundancy - a fallback mode for computer systems Another aspect of this technology involves redundancy incorporated into the computing system. During typical operation, a first computing system (e.g., a planner subsystem) can generate a trajectory and transmit it to a second computing system to control the vehicle according to that trajectory. Each of these computing systems may be part of the computing device 302 (Figure 3). For redundancy, the two subsystems may each have one or more processors and associated memory. In this example, each subsystem is powered and operates independently, but shares sensor data and resources to handle basic vehicle operations. For example, if one subsystem fails due to a CPU crash, kernel error, power failure, etc., the other subsystem can control the vehicle in a specified fallback operating state.
[0054] In a single configuration, both computing subsystems can control the vehicle in standard operating mode and fallback mode. Each subsystem is associated with its respective domain via one or more CAN buses or FlexRay buses, etc. However, the sensor sets in each domain do not need to be identical, complementary, or completely overlapping. For example, certain “fallback” sensors could be assigned to control subsystem A (e.g., sensors 1102, 1104, and 1106 in Figure 11B), other fallback sensors to control subsystem B (e.g., sensors 1101, 1103, and 1105 in Figure 11C), and other “non-fallback” sensors could be assigned only to control subsystem A (see Figure 12).
[0055] Unlike fallback sensors, non-fallback sensors do not need to be assigned to any particular domain, nor do they require domain independence or redundancy. This is because non-fallback sensors may only be used in standard operating modes, rather than providing minimal viable fallback functionality. Nevertheless, if there are other reasons to place such sensors in one domain or the other (e.g., vehicle integration), this can be done without affecting fallback operation. For example, one domain may have more power headroom than another, making it easier to accommodate a non-fallback sensor. In another example, a non-fallback sensor may be associated with a particular domain because it is sometimes easier to wire to that domain.
[0056] Figure 12 shows an example of an autonomous driving system configuration 1200. As shown, each domain includes one or more processors 1202a or 1202b, which may be processors 304 from the computing system 302. Each domain also includes its respective operating mode logics 1204a, 1204b, which may also include software or firmware modules for execution by processors 1202a, 1202b. The operating mode logics 1204a, 1204b may include separate components for execution in a standard operating mode and one or more fallback operating modes. Fallback modes for different domains may involve operating the vehicle in different ways, for example, according to the type and function of the fallback sensors associated with each domain. And, as shown in Figure 12, one of the domains (e.g., Domain A) may include different computing resources, such as one or more graphics processing units (GPUs) 1206 or other computing accelerators (e.g., ASICs, FPGAs, etc.). Here, one set of fallback sensors is associated with Domain A, another set of fallback sensors is associated with Domain B, and one or more non-fallback sensors are also associated with Domain A; however, other configurations are possible, for example, the fallback sensors may be assigned to Domain B, or one subset may be assigned to Domain A and another subset to Domain B. The two (or more) computational domains do not have to have the same performance or the same type of computational elements, but in certain configurations they may have equivalent performance and / or the same type of computational elements.
[0057] As a simple example, in this configuration, Domain A may support a perception system (e.g., perception system 324 in Figures 3-4) during standard operation, and Domain B may support a planner (e.g., route planning based on navigation system 320 and positioning system 322 in Figure 3) during standard operation. The GPU(s) 1206 may be configured to process sensor data received from some or all of the on-board sensors during standard operation.
[0058] In one scenario, each computing subsystem may have sufficient sensor inputs and computing capabilities (e.g., processor and memory resources for processing received sensor data) to operate the vehicle in its corresponding standard and fallback operating modes. Two (or more) domains can share information in standard mode. In this way, various domains and subsystems can be used to provide optical FOV coverage. According to aspects of this technology, some computing resources may be reserved to handle fallback operation. Also, in the case of fallback, some typical operations of standard mode may cease. For example, one or more forward-facing high-resolution cameras (e.g., camera 806 in Figure 8) can be considered non-fallback critical sensors. As a result, when entering fallback mode, processing of inputs from these cameras may cease.
[0059] Partial redundancy - power distribution fallback mode With respect to power distribution, an aspect of this technology provides two fault-independent power sources, each with battery backup. For example, the dual power sources are operationally isolated so that the primary fault is limited to only one domain. Each power source can accommodate a specific set of base vehicle loads. Figure 13A shows an example of a redundant power distribution architecture 1300. As shown in this scenario, each domain has its own independent power source. Here, the power sources and loads of each domain are protected by an intervening electronic fuse. In practice, during normal operation, the electronic fuse is closed. When the electronic fuse detects a fault in the form of overcurrent, undervoltage, overheating (or other abnormal condition), it immediately reacts and opens, thereby isolating the two domains.
[0060] Each power source has battery backup and is configured to provide automotive standards of approximately 12 volts (e.g., in the range of 8 to 16 volts). Each domain has individual power distribution blocks 1302a, 1302b, such as the power distribution element 327 in Figure 3.
[0061] A DC / DC unit is a voltage converter that receives power from an upstream high-voltage battery pack and converts it to a lower voltage (e.g., 12V) to charge a 12V battery. In an electronic fuse architecture, due to the presence of electronic fuses, two DC / DC units are not necessary because one unit can charge half of the system when everything is working (no faults). In the event of a failure, the system does not necessarily need to use a DC / DC unit because power can be supplied from redundant low-voltage backup batteries in each domain.
[0062] Another example of power redundancy that does not require an electronic fuse is the dual independent DC / DC system 1310 shown in Figure 13B. Here, each domain is supplied with its own DC power supply. In this example, each domain has separate power distribution blocks 1312a, 1312b. Grounding of the system is provided so that the return current paths of the domains do not result in a single point of failure.
[0063] There may be fallback-critical actuators, such as brakes, steering, and / or thrust. There may also be non-fallback-critical actuators, such as horns, cabin lights, and heating and air conditioning systems. Actuators considered fallback-critical may be redundant (e.g., two or more separate actuators), and / or such actuators may receive power from both domains. In contrast, non-fallback-critical actuators may not have redundant components and / or may receive power from only a single domain.
[0064] Other subsystems of the vehicle may also have redundancy. For example, the braking and steering subsystems may be redundant (in addition to being redundantly powered). Similarly, communication with other vehicles or remote assistance devices may utilize multiple cellular or other types of communication links. Two or more GPS receivers may be used. Even wipers, sprayers, or other cleaning components may be configured for redundancy and powered (as a base load) in one or both (or more) domains. The base vehicle load may include individual sensors, sensor clusters, computing devices, actuators, and / or other components as considered with respect to Figure 3.
[0065] Figure 14 shows a method 1400 for operating a vehicle according to an aspect of the present technology. As shown in block 1402, information about the vehicle's environment is detected by the vehicle's sensors (e.g., Lidar, radar, cameras, auditory, and / or positioning sensors). In block 1404, the detected information is received by the vehicle's control system, such as the computing system(s) 302 in Figure 3 or the system 1200 in Figure 12. In each block 1406, the vehicle's driving system is autonomously controlled in response to the reception of sensor data in the first operating mode. In block 1408, an error condition is detected in one or more of the vehicle's sensors. This could be due to, for example, a failure, malfunction, or other degradation of the sensor function.
[0066] In block 1410, upon detecting an error condition, the first computing subsystem processes sensor data from a first set of sensors in a first operating domain, and the second computing subsystem processes sensor data from a second set of sensors in a second operating domain. As a result, in block 1412, in response to the error condition, one of the first or second computing subsystems controls the vehicle's driving system in fallback driving mode. In some examples, the first computing subsystem may operate a function (e.g., a planner, perception, etc.) in standard operating mode before detecting an error condition, and the second computing subsystem may operate another function in standard operating mode.
[0067] Unless otherwise specified, the aforementioned alternatives are not mutually exclusive but can be implemented in various combinations to achieve their own advantages. Since these and other variations and combinations of the features considered above can be utilized without departing from the subject matter defined by the claims, the foregoing description of embodiments should be considered illustrative rather than limiting the subject matter defined by the claims. Furthermore, the provision of embodiments described herein, as well as phrases such as “etc.” and “including,” should not be interpreted as limiting the subject matter of the claims to specific embodiments; rather, the embodiments are intended to illustrate only one of many possible embodiments. Additionally, the same reference numerals in different drawings may identify the same or similar elements. Processes or other operations may be performed in different orders or concurrently unless otherwise specifically stated herein.
Claims
1. A vehicle configured to operate in autonomous driving mode, wherein the vehicle is A driving system configured to perform the driving activities of the aforementioned vehicle, A perception system having a plurality of sensors configured to detect information about the environment surrounding the vehicle, wherein the plurality of sensors provide a 360° field of view that enables the classification of objects in the external environment of the vehicle, and the plurality of sensors provide sensor diversity based at least partially on overlapping sensor fields of view. A control system operably connected to the driving system and the sensing system, wherein the control system includes a first computing subsystem associated with a first fallback driving mode and a second computing subsystem associated with a second fallback driving mode different from the first fallback driving mode, and each of the first and second computing subsystems has one or more processors, The first computing subsystem is configured to receive sensor data from the perception system via a first communication link, and the second computing subsystem is configured to receive sensor data from the perception system via a second communication link that is separate from the first communication link. In response to the received sensor data, the control system is configured to control the driving system to perform the driving activity in order to drive the vehicle in the autonomous driving mode. The control system is configured to detect an error state of the vehicle, A vehicle in which, upon detecting the error condition, only one of the first or second computing subsystems is configured to process sensor data received from the plurality of sensors according to the sensor where the error condition was detected and the overlapping sensor fields of view, and to control the driving system in the corresponding fallback driving mode.
2. The vehicle according to claim 1, wherein only one of the first computing subsystem and the second computing subsystem is configured to modify the previously planned trajectory of the vehicle according to the corresponding fallback driving mode.
3. The vehicle according to claim 1, wherein the control of the driving system in the corresponding fallback driving mode includes a decision of whether to (i) continue driving along a route toward a planned destination, or (ii) stop the vehicle before the planned destination in response to a decision that it is safe to stop the vehicle.
4. The vehicle according to claim 3, wherein the decision to continue driving is made by at least one of the following: a changed vehicle speed, activation of hazard lights, or selection of a drop-off location different from the planned destination.
5. The vehicle according to claim 1, wherein the error state is a decrease in the sensor function relating to at least one sensor of the perception system.
6. The vehicle according to claim 1, wherein the error condition is related to a component failure for at least one sensor of the perception system.
7. The vehicle according to claim 6, wherein the failure of the component is a mechanical failure.
8. The vehicle according to claim 6, wherein the failure of the component is an electrical failure.
9. The vehicle according to claim 6, wherein the failure of the component is deterioration caused by one or more environmental conditions.
10. The vehicle according to claim 9, wherein the one or more environmental conditions include a temperature condition.
11. The vehicle according to claim 9, wherein the one or more environmental conditions include a type of precipitation.
12. The vehicle according to claim 9, wherein one or more of the environmental conditions include blockage of mud or dust.
13. The vehicle according to claim 1, further comprising a power distribution system, wherein the power distribution system comprises a first power subsystem configured to supply dedicated power to the first computing subsystem and not to the second computing subsystem, and a second power subsystem configured to supply dedicated power to the second computing subsystem and not to the first computing subsystem.
14. The vehicle according to claim 13, wherein the first power subsystem is operationally isolated from the second power subsystem.
15. A method for operating a vehicle in autonomous driving mode, wherein the method is A first computing subsystem of the vehicle's control system, associated with a first fallback driving mode, receives sensor data from the vehicle's perception system via a first communication link, wherein the perception system has a plurality of sensors configured to detect information about the environment surrounding the vehicle, the plurality of sensors provide a 360° field of view enabling the classification of objects in the vehicle's external environment, and the plurality of sensors provide sensor diversity based at least partially on overlapping sensor fields of view. A second computing subsystem of the vehicle's control system, associated with a second fallback driving mode different from the first fallback driving mode, receives sensor data from the vehicle's perception system via a second communication link separate from the first communication link. In response to receiving the aforementioned sensor data, the control system controls the vehicle's driving system to perform one or more driving activities in order to drive the vehicle in the autonomous driving mode. The control system detects the error state of the vehicle, A method comprising: detecting the error condition, having either the first or second computing subsystem process new sensor data received from the plurality of sensors according to the sensor in which the error condition was detected and the overlapping sensor fields of view, and controlling the operating system in a corresponding fallback operating mode.
16. The method according to claim 15, wherein only one of the first computing subsystem and the second computing subsystem is configured to modify the previously planned trajectory of the vehicle according to the corresponding fallback driving mode.
17. The method according to claim 15, wherein controlling the driving system in the fallback driving mode includes determining whether to (i) continue driving along a route toward a planned destination, or (ii) stop the vehicle before the planned destination depending on the determination that it is safe to stop the vehicle.
18. The method according to claim 17, wherein the decision to continue driving is made by at least one of changing the vehicle speed, activating the hazard lights, or selecting a drop-off location different from the planned destination.
19. The method according to claim 15, wherein the error state is a decrease in the sensor function relating to at least one sensor of the perception system.
20. The method according to claim 15, wherein the error condition is related to a component failure for at least one sensor of the perception system.