Close-range sensing camera system

Abstract

This technology relates to an external sensor system (232) for a vehicle configured to operate in an autonomous driving mode. The technology includes a proximity sensing (CIS) camera system (1404) to address blind spots around the vehicle. The CIS system is used to detect objects within a few meters of the vehicle (1808). Based on object classification, the system is able to make real-time driving decisions (1810, 1812). Classification is enhanced by using a camera in conjunction with a lidar sensor (700). The specific arrangement of multiple sensors within a single sensor housing is also important for object detection and classification. Therefore, the positioning of the sensors and support components is chosen to avoid obstruction and additionally prevent interference between the various sensor housing elements (712, 1414).

Description

[0001] Cross-reference to related applications

[0002] This application is a continuation of U.S. Patent Application No. 16 / 737,263, filed January 8, 2020, which claims the benefit of U.S. Provisional Application No. 62 / 954,930, filed December 30, 2019, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] This application relates to autonomous driving technology, and more specifically, to an external sensor system for a vehicle configured to operate in autonomous driving mode. Background Technology

[0004] Autonomous vehicles do not require a human driver in some or all of the following situations. Such vehicles can transport passengers or cargo from one location to another. They can operate in fully autonomous mode or partially autonomous mode in which a human can provide some driving input. To operate in autonomous mode, the vehicle can employ sensors to detect other vehicles and objects in its external environment and use the information received from the sensors to perform various driving maneuvers. However, objects adjacent to the vehicle and occlusions in the sensor's field of view can adversely affect driving operations. Summary of the Invention

[0005] This technology relates to an external sensor system for a vehicle configured to operate in autonomous driving (autonomous) mode. Typically, sensors are used to detect objects in the environment surrounding the vehicle. These can include lidar, radar, cameras, sonar, and / or other sensors. Different sensors have different advantages, and sensor fusion from multiple sensors can be employed to obtain a more complete understanding of the environment so that the vehicle can make driving decisions. However, depending on the size, shape, etc., of the vehicle and objects in the environment, blind spots may exist that can affect driving decisions and other autonomous operations. These include blind spots immediately adjacent to the vehicle. Such problems can be largely mitigated by carefully selecting and positioning sensor housings that can co-locate different types of sensors in an integrated unit. This can include a close-in camera system integrated with a lidar sensor, a perimeter view camera juxtaposed with radar and / or other sensors, etc.

[0006] According to one aspect, an external sensing system is provided for a vehicle configured to operate in an autonomous driving mode. The external sensing system includes a lidar sensor, an image sensor, and a control system. The lidar sensor has a field of view configured to detect objects within a given area of ​​the external environment surrounding the vehicle and within a threshold distance of the vehicle. The image sensor is disposed adjacent to the lidar sensor and arranged along the vehicle to have an overlapping field of view of the area of ​​the external environment within the threshold distance of the vehicle. The image sensor is configured to provide a selected resolution for objects within the threshold distance. The control system is operatively coupled to the image sensor and the lidar sensor. The control system includes one or more processors configured to: initiate operation of the lidar sensor to obtain lidar data in the area within the threshold distance of the vehicle; initiate image capture by the image sensor before the vehicle performs a driving action; and receive lidar data from the lidar sensor and the captured image from the image sensor. The control system is also configured to: perform processing of the lidar data to detect objects in the area within the threshold distance of the vehicle, and perform processing of the captured image to classify the detected objects. The control system is also configured to determine whether to cause one or more systems of the vehicle to perform a driving action based on the classification of detected objects. The classification of detected objects may include determining at least one of the size, proximity, or orientation of the detected objects.

[0007] The image sensor can be configured to observe a minimum threshold volume occupied by a detected object. For example, the minimum threshold volume can be at least 50% of the 3D shape surrounding the detected object. The image sensor can be positioned no more than 0.3 meters from the lidar sensor. The image sensor and lidar sensor can be housed within the same sensor housing, which is disposed on the outer surface of the vehicle. The threshold distance can be no more than 3 meters from the vehicle. The lens of the image sensor may include a hydrophobic coating.

[0008] In one embodiment, the image sensor is part of a proximity sensing camera system. Here, the proximity sensing camera system includes at least one illuminator, such as an infrared illuminator, configured to illuminate the field of view of the image sensor. The at least one illuminator may be arranged adjacent to one side of the image sensor. Alternatively, the at least one illuminator may be arranged above the image sensor. The at least one illuminator may comprise a pair of illuminators arranged on opposite sides of the image sensor. The proximity sensing camera system may further include at least one cleaning mechanism configured to clean the image sensor and / or at least one illuminator. In one example, the image sensor is vertically aligned below the LiDAR sensor. In another example, the image sensor is vertically aligned above the LiDAR sensor.

[0009] According to another aspect, the vehicle is configured to operate in an autonomous driving mode. The vehicle includes a driving system and an external sensing system. The driving system includes: a deceleration system configured to control braking of the vehicle, an acceleration system configured to control acceleration of the vehicle, and a steering system configured to control wheel orientation and the direction of the vehicle. The external sensing system includes a lidar sensor, an image sensor, and a control system. The lidar sensor has a field of view configured to detect objects in at least one area of ​​the external environment surrounding the vehicle and within a threshold distance of the vehicle. The image sensor is disposed adjacent to the lidar sensor and arranged along the vehicle to have overlapping fields of view of the external environment within the threshold distance of the vehicle. The image sensor provides a selected resolution for objects within the threshold distance. The control system is operatively coupled to the image sensor and the lidar sensor. The control system includes one or more processors configured to: initiate operation of the lidar sensor to obtain lidar data in the area within the threshold distance of the vehicle; initiate image capture by the image sensor before the driving system performs driving actions; and receive lidar data from the lidar sensor and the captured image from the image sensor. It is also configured to process lidar data to detect objects in areas within a threshold distance of the vehicle, and to process captured images to classify the detected objects. Therefore, the control system can determine whether to initiate a driving maneuver by one or more systems of the vehicle based on the classification of the detected objects.

[0010] The image sensor can be configured to observe a minimum threshold volume occupied by a detected object. The image sensor can be set to be within 0.3 meters of the lidar sensor. The image sensor and lidar sensor can be housed within the same sensor housing, which is mounted on the outer surface of the vehicle.

[0011] According to another aspect, a method includes: initiating operation of a lidar sensor of a perception system of a vehicle configured to operate in an autonomous driving mode by a control system of the vehicle to acquire lidar data within a threshold distance in an area surrounding the vehicle; initiating image capture by an image sensor of the perception system prior to the vehicle performing a driving action by the control system, the image sensor being disposed adjacent to the lidar sensor and arranged along the vehicle to have an overlapping field of view of the area surrounding the vehicle within the threshold distance, wherein the image sensor provides a selected resolution for objects within the threshold distance; receiving lidar data from the lidar sensor and the captured image from the image sensor by the control system; processing the lidar data by the control system to detect objects in an area within the threshold distance of the vehicle; processing the captured image by the control system to classify the detected objects; and determining, based on the classification of the detected objects, whether to cause one or more systems of the vehicle to perform a driving action. The classification of the detected objects may include determining at least one of the size, proximity, or orientation of the detected objects. Attached Figure Description

[0012] Figures 1A to 1B An example vehicle configured for aspects of this technology is shown.

[0013] Figure 1C Another example vehicle configured for aspects of this technology is shown.

[0014] Figures 1D to 1E An example cargo vehicle configured for aspects of this technology is shown.

[0015] Figure 2A This is a block diagram of an example vehicle system based on aspects of this technology.

[0016] Figures 2B to 2C This is a block diagram of an example cargo type vehicle system based on aspects of this technology.

[0017] Figure 3 An example of the area surrounding a vehicle according to aspects of this disclosure is shown.

[0018] Figure 4 An example sensor field of view according to an aspect of this disclosure is shown.

[0019] Figure 5 An example perimeter camera field of view according to aspects of this disclosure is shown.

[0020] Figures 6A to 6C An example arrangement of a perimeter camera and an infrared illuminator according to aspects of this disclosure is shown.

[0021] Figures 7A to 7FAn example perimeter sensor housing assembly according to aspects of this disclosure is shown.

[0022] Figure 8 An example sensor arrangement for minimizing occlusion is shown according to aspects of this disclosure.

[0023] Figure 9 An example of the field of view of a perimeter sensor according to aspects of this disclosure is shown.

[0024] Figure 10 An example occlusion scenario according to an aspect of this technology is shown.

[0025] Figures 11A to 11C An example perimeter sensor housing assembly according to aspects of this disclosure is shown.

[0026] Figure 12 An example sensor arrangement for minimizing occlusion is shown according to aspects of this disclosure.

[0027] Figure 13 An example of the field of view of a perimeter sensor according to aspects of this disclosure is shown.

[0028] Figures 14A to 14E Another example of a perimeter sensor housing assembly according to aspects of this disclosure is shown.

[0029] Figure 15 It shows Figures 14A to 14E A variation of the perimeter sensor housing assembly.

[0030] Figures 16A to 16E Another example of a perimeter sensor housing assembly according to aspects of this disclosure is shown.

[0031] Figure 17 It shows Figures 16A to 16E A variation of the perimeter sensor housing assembly.

[0032] Figure 18 The method of operation according to aspects of this disclosure is shown. Detailed Implementation

[0033] This technology relates to close-in sensing (CIS) camera systems to address blind spots around vehicles. CIS systems are used to help classify objects detected within a few meters (e.g., less than 3 meters) of a vehicle. Based on object classification, the system can distinguish between objects that are potentially "drivable" (things the vehicle can drive over) and "non-drivable." For example, drivable objects could be plants, a pile of leaves, paper, or plastic bags. Non-drivable objects could include those types of objects that must be avoided (e.g., pedestrians, cyclists, pets, etc.) or those types of objects that could damage the vehicle if driven over (e.g., high curbs, broken glass, potholes, fire hydrants, etc.). In one scenario, classification is enhanced by using cameras in conjunction with LiDAR sensors. This can be crucial when trying to determine if someone is near a vehicle. Each camera can include one or more image sensors. The image sensors can be CMOS sensors, although CCDs or other types of imaging elements can also be used.

[0034] Other aspects of this technology relate to the arrangement and configuration of multiple sensors within a single sensor housing. As discussed further below, it is advantageous to co-locate different sensor types within the same housing, for example, to facilitate sensor fusion. However, sensor positioning can be critical, for example, to prevent one sensor from being obstructed by another, to ensure more accurate calibration between sensors, and / or additionally, to prevent interference between sensors. For instance, illuminators such as infrared (IR) or optical illuminators should be arranged to prevent their light from directly illuminating the lens of a camera (e.g., a camera sensitive to IR light).

[0035] Example vehicle system

[0036] Figure 1A A perspective view of a passenger vehicle 100—such as a van, sedan, or SUV—is shown. Figure 1BA top view of a passenger vehicle 100 is shown. The passenger vehicle 100 may include various sensors for obtaining information about the vehicle's external environment. For example, the roof top housing 102 may include a lidar sensor as well as various cameras, radar units, infrared and / or acoustic sensors. Housing 104, positioned at the front of the vehicle 100, and housings 106a, 106b, positioned on the driver's side and passenger side of the vehicle, may each contain lidar and / or other sensors. For example, housing 106a may be positioned along the quarter panel of the vehicle in front of the driver's side door. As shown, the passenger vehicle 100 also includes housings 108a, 108b for radar units, lidar, and / or cameras, which are also positioned towards the rear top portion of the vehicle. Other housings 110a, 110b may be positioned along the rear quarter panel, for example, above and behind the rear wheels.

[0037] Additional lidar, radar units, and / or cameras (not shown) may be positioned at other locations along the vehicle 100. For example, arrow 112 indicates the sensor unit ( Figure 1B 112) can be positioned along the rear of the vehicle 100, such as on or adjacent to the bumper or trunk door / cover. Arrow 114 indicates a series of sensor units 116 arranged along the forward direction of the vehicle. Although shown separately, in one example, sensor units 116 may be integrated into the forward-facing portion of the roof housing 102. In some examples, the passenger vehicle 100 may also include various sensors for obtaining information about the interior space of the vehicle. Interior sensors may include at least one of camera sensors, auditory sensors, and infrared sensors.

[0038] Depending on the type and configuration of the vehicle, more or fewer sensor housings may be placed around the vehicle. For example, such as Figure 1C As shown in the example vehicle 150, similar to vehicle 100, it may have a roof sensor housing 152, a front sensor housing 154, side housings 156a and 156b along the front side panel, a side housing 158 along the rear side panel, and a rear sensor housing indicated by arrow 160. While certain aspects of this disclosure may be particularly useful for certain types of vehicles, the vehicle can be any type of vehicle, including but not limited to automobiles, trucks, motorcycles, buses, recreational vehicles, etc.

[0039] Figures 1D to 1EAn example cargo vehicle 170, such as a tractor-trailer truck, is shown. The truck may include, for example, a single trailer, a double trailer, or a triple trailer, or it may be another medium or heavy-duty truck, such as a commercial weight class 4 to 8. As shown, the truck includes a tractor unit 172 and a single cargo unit or trailer 174. Depending on the type of cargo to be transported, the trailer 174 may be fully enclosed, open (such as a flatbed), or partially open. In this example, the tractor unit 172 includes an engine and steering system (not shown) and a cab 176 for the driver and any passengers.

[0040] Trailer 174 includes a hitch point referred to as kingpin 178. Kingpin 178 is typically formed as a solid steel axle and is configured to pivotally attach to tractor unit 172. Specifically, kingpin 178 is attached to a trailer coupling 180, referred to as a fifth wheel, mounted at the rear of the cab. For two- or three-trailer trailers, the second and / or third trailers may have a simple hitch connection to the lead trailer. Alternatively, each trailer may have its own kingpin. In this case, at least the first and second trailers may include a fifth wheel configuration arranged to connect to the next trailer.

[0041] As shown, the tractor unit may have one or more sensor units 182, 184, and / or 186 disposed thereal. For example, one or more sensor units 182 may be disposed on the top or upper part of the cab 176, and one or more side sensor units 184 may be disposed on the left and / or right side of the cab 176. The sensor units may also be positioned along other areas of the cab 176, such as along the front bumper or hood area, at the rear of the cab, adjacent to the landing gear, under the chassis, etc. The trailer 174 may also have one or more sensor units 186 disposed thereal, such as along the side panels, front, rear, top, and / or undercarriage of the trailer 174.

[0042] as Figures 1A to 1B Similar to the sensor units of passenger vehicles, each sensor unit of a cargo vehicle may include one or more sensors, such as lidar, radar, cameras (e.g., optical or infrared), acoustic (e.g., microphone or sonar type sensors), inertial (e.g., accelerometers, gyroscopes, etc.) or other sensors (e.g., positioning sensors such as GPS sensors). While certain aspects of this disclosure may be particularly useful for certain types of vehicles, the vehicle can be any type of vehicle, including but not limited to automobiles, trucks, motorcycles, buses, recreational vehicles, etc.

[0043] Figure 2A Block diagram 200 is shown, which has various components and systems of exemplary vehicles (such as vehicles 100 and 150) configured to operate in fully autonomous or semi-autonomous operating modes. For example, different degrees of autonomy may exist for vehicles operating in partially or fully autonomous driving modes. The National Highway Traffic Safety Administration (NHTSA) and the Society of Automotive Engineers (SAE) have identified different levels to indicate how much control a vehicle has over driving. For example, Level 0 has no automation and the driver makes all driving-related decisions. The lowest semi-autonomous mode, Level 1, includes some driving assistance, such as cruise control. Level 2 has partial automation of some driving operations, while Level 3 involves conditional automation, which allows the person in the driver's seat to take control as needed. In contrast, Level 4 is a high level of automation, where the vehicle is able to drive without assistance under selected conditions. Level 5 is a fully autonomous mode, where the vehicle is able to drive without assistance in all situations. The architectures, components, systems, and methods described herein can operate in any of the semi-autonomous or fully autonomous modes (e.g., Levels 1–5) referred to as “autonomous” driving modes. Therefore, the autonomous driving mode can refer to both partial autonomy and full autonomy.

[0044] As shown in Figure 2, block diagram 200 includes one or more computing devices 202, such as computing devices containing one or more processors 204, memory 206, and other components typically found in general-purpose computing devices. Memory 206 stores information accessible to one or more processors 204, including instructions 208 and data 210 that can be executed or otherwise used by the processors 204. When operating in autonomous mode, the computing system can control the overall operation of the vehicle.

[0045] Memory 206 stores information accessible to processor 204, including instructions 208 and data 210 that can be executed or otherwise used by processor 204. Memory 206 can be any type capable of storing processor-accessible information, including computing device-readable media. Memory is a non-transitory medium such as a hard drive, memory card, optical disc, solid-state drive, etc. The system may include different combinations of the foregoing, whereby different portions of instructions and data are stored on different types of media.

[0046] Instructions 208 can be any set of instructions that are executed directly by a processor (such as machine code) or indirectly (such as a script). For example, instructions can be stored as computing device code on a computing device readable medium. In this regard, the terms “instruction,” “module,” and “program” are used interchangeably herein. Data 210 can be retrieved, stored, or modified by one or more processors 304 according to instructions 208. In one example, some or all of memory 206 may be an event data logger or other secure data storage system configured to store: vehicle diagnostics, detected sensor data, and / or one or more behavioral / classification models used in conjunction with object detection and classification; these can be on the vehicle or remote, depending on the implementation. For example, the model can be used to classify objects as people (e.g., pedestrians), bicycles, balls, or construction signs adjacent to the vehicle. Based on this classification, the system can predict or assign behavior for that object and use the classification / behavior when making driving-related decisions. For example, this could include warning pedestrians next to the vehicle that the vehicle has started and is planning to leave the parking spot.

[0047] Processor 204 can be any conventional processor, such as a commercially available CPU. Alternatively, each processor can be a dedicated device, such as an ASIC or other hardware-based processor. Although Figure 2 functionally shows the processor, memory, and other components of computing device 202 within the same box, such a device can actually include multiple processors, computing devices, or memories, which may or may not be housed in the same physical housing. Similarly, memory 206 can be a hard drive or other storage medium located in a housing different from that of processor 204. Therefore, references to processors or computing devices will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0048] In one example, computing device 202 can form an autonomous driving computing system incorporated into a vehicle. The autonomous driving computing system may be able to communicate with various components of the vehicle. For example, computing device 202 may communicate with various systems of the vehicle, including a driving system comprising a deceleration system 212 (for controlling the vehicle's braking), an acceleration system 214 (for controlling the vehicle's acceleration), a steering system 216 (for controlling wheel orientation and the vehicle's direction), a signaling system 218 (for controlling steering signals), a navigation system 220 (for navigating the vehicle to a location or object), and a positioning system 222 (for determining the vehicle's position). Based on the navigation system 220 and the positioning system 222, the autonomous driving computing system may partially function as a planner, for example, for determining a route from a starting point to a destination.

[0049] The computing device 202 is also operatively coupled to the sensing system 224 (for detecting objects in the vehicle environment), the power system 226 (e.g., a battery and / or gasoline or diesel-powered engine), and the transmission system 230 to control the vehicle's movement, speed, etc., according to instructions 208 from the memory 206 in an autonomous driving mode that does not require or need continuous or periodic input from the vehicle's passengers. Some or all of the wheels / tires 228 are coupled to the transmission system 230, and the computing device 202 may be able to receive information about tire pressure, balance, and other factors that may affect driving in autonomous mode. The power system 226 may have one or more power distribution elements, each of which may be able to supply power to selected components and other systems of the vehicle.

[0050] The computing device 202 can control the direction and speed of the vehicle by controlling various components. For example, the computing device 202 can autonomously navigate the vehicle to its destination using map information and data from the navigation system 220. The computing device 202 can use the positioning system 222 to determine the vehicle's position and the sensing system 224 to detect objects and respond to them as needed to safely reach the location. To this end, the computing device 202 can accelerate the vehicle (e.g., by increasing the fuel or other energy supplied to the engine by the acceleration system 214), decelerate (e.g., by reducing the fuel supplied to the engine, changing gears, and / or applying braking by the deceleration system 212), change direction (e.g., by turning the front wheels or other wheels of the vehicle by the steering system 216), and signal such changes (e.g., by illuminating the turn signal of the signaling system 218). Therefore, the acceleration system 214 and the deceleration system 212 can be part of a power transmission or other type of transmission system 230 that includes various components between the vehicle's engine and the vehicle's wheels. Furthermore, by controlling these systems, computing device 202 can also control the vehicle's transmission system 230 in order to autonomously maneuver the vehicle.

[0051] Navigation system 220 can be used by computing device 202 to determine a route and reach a location along that route. In this regard, navigation system 220 and / or memory 206 can store map information, such as highly detailed maps that computing device 202 can use to navigate or control a vehicle. For example, these maps can identify road shapes and elevations, lane markings, intersections, pedestrian crossings, speed limits, traffic lights, buildings, signs, real-time traffic information, vegetation, or other such objects and information. Lane markings may include features such as solid or dashed double or single lane lines, solid or dashed lane lines, reflectors, etc. A given lane can be associated with left and / or right lane lines or other lane markings that define lane boundaries. Therefore, most lanes can be defined by the left edge of one lane line and the right edge of another lane line.

[0052] The perception system 224 includes sensor units for detecting objects outside the vehicle. Detected objects can be other vehicles, obstacles on the road, traffic signals, signs, trees, pedestrians, cyclists, etc. As discussed further below, the external sensor suite 232 includes various housings, each housing having one or more sensors to detect objects and conditions in the vehicle's external environment. The internal sensor suite 234 may employ one or more additional sensors to detect objects and conditions inside the vehicle, such as passengers, pets, and luggage in the passenger compartment, luggage or other cargo in the trunk area, etc. For both the external sensor suite 232 and the internal sensor suite 234, housings with different sensors are arranged around the vehicle to provide object detection not only under various environmental conditions but also to enable rapid classification of detected objects. This allows the vehicle to make effective real-time driving decisions.

[0053] When the sensing system 224 generates data, the raw data from the sensors and the aforementioned characteristics can be processed by the sensing system 224 and / or periodically and continuously sent to the computing device 202 for further processing. The computing device 202 can use the positioning system 222 to determine the position of the vehicle, and use the sensing system 224 to detect objects and respond to them when necessary to safely reach the location. Additionally, the computing device 202 can perform calibrations between individual sensors, all sensors in a specific sensor assembly (housing), or sensors in different sensor assemblies or other physical housings.

[0054] In one example, the external sensor housing may be arranged as a sensor tower integrated into the side mirrors of the vehicle. In another example, other sensors may be part of the canopy housing 102 or 152, or as... Figures 1A to 1CThis is part of another housing shown. The computing device 202 can communicate with sensor assemblies located on the vehicle or otherwise distributed along the vehicle. Each assembly may have one or more types of sensors, such as those described above.

[0055] Returning to Figure 2, computing device 202 may include all components typically used in conjunction with computing devices, such as the processor and memory described above, and user interface subsystem 236. User interface subsystem 236 may include one or more user inputs 238 (e.g., mouse, keyboard, touchscreen, and / or microphone) and one or more display devices 240 (e.g., a monitor with a screen or any other electronic device operable to display information). At this point, an internal electronic display may be located in the vehicle's cab (not shown) and may be used by computing device 202 to provide information to passengers within the vehicle. Other output devices such as speaker 242 and input devices 244 such as touchscreens or buttons may also be located within the passenger vehicle.

[0056] The vehicle also includes a communication system 246. For example, the communication system 246 may also include one or more wireless network connections to facilitate communication with other computing devices (such as passenger computing devices within the vehicle) and computing devices external to the vehicle (such as another nearby vehicle on the road or a remote server system). The network connections may include short-range communication protocols (such as Bluetooth). TM ,Bluetooth TM Low power (LE), cellular connectivity) and various configurations and protocols, including the Internet, World Wide Web, intranet, virtual private network, wide area network, local area network, private network using one or more company-proprietary communication protocols, Ethernet, WiFi and HTTP, and various combinations thereof.

[0057] Figure 2B It shows a vehicle (e.g., Figures 1D to 1E A block diagram 250 shows various components and systems of a vehicle (170). For example, the vehicle may be a truck, agricultural equipment, or construction equipment configured to operate in one or more partially autonomous operating modes. As shown in block diagram 250, the vehicle includes a control system of one or more computing devices similar to those described above, such as a computing device 202' containing one or more processors 204' and a memory 206', the memory 206' storing instructions 208' and data 210', such as vehicle diagnostics, detected sensor data, and / or one or more behavioral / classification models used in conjunction with object detection and classification. In this example, the control system may constitute an electronic control unit (ECU) of the tractor unit of the cargo vehicle.

[0058] In one example, the computing device can form a driving computing system contained within vehicle 170. Similar to the above regarding... Figure 2A The driving computing system of block diagram 250, as discussed, may be able to communicate with various components of the vehicle to perform driving operations. For example, computing device 202' may communicate with various systems of the vehicle, such as the driving system including deceleration system 212', acceleration system 214', steering system 216', signal system 218', navigation system 220', and positioning system 222', each of which may be as described above regarding... Figure 2A Work as discussed.

[0059] The computing device 302 is also operatively coupled to the sensing system 224', the power system 226', and the transmission system 230'. Some or all of the wheels / tires 228' are connected to the transmission system 230', and the computing device 202' may be able to receive information about tire pressure, balance, speed, and other factors that may affect driving. Like the computing device 202, the computing device 202' can control the direction and speed of the vehicle by controlling various components. For example, the computing device 202' can use data from map information and the navigation system 220' to help navigate the vehicle to its destination.

[0060] Similar to sensing system 224, sensing system 224' also includes one or more sensors or other components, such as those described above for detecting objects outside the vehicle, objects or conditions inside the vehicle, and / or the operation of certain vehicle equipment such as wheels and deceleration system 212'. For example, as Figure 2B As shown, the sensing system 224' includes one or more sensor assemblies 252. Each sensor assembly 252 includes one or more sensors. In one example, the sensor assembly 252 may be arranged as a sensor tower integrated into the side mirror of a truck, agricultural equipment, construction equipment, etc. The sensor assembly 252 may also be located at different positions on the tractor unit 172 or trailer 174, as referenced above. Figures 1D to 1E As mentioned above, computing device 202' can communicate with sensor assemblies located on both tractor unit 172 and trailer 174. Each assembly can have one or more types of sensors as described above.

[0061] Figure 2B The diagram also shows a coupling system 254 for connecting the tractor unit and the trailer. The coupling system 254 may include one or more power and / or pneumatic connections (not shown), and a roller 256 at the tractor unit for connecting to the kingpin at the trailer. A communication system 246', equivalent to communication system 246, is also shown as part of the vehicle system 250. Similarly, a user interface 236', equivalent to user interface 236, may also be included for interaction with the vehicle's driver and any passengers.

[0062] Figure 2C It shows things like Figures 1D to 1E Example block diagram 260 of a system for a trailer 174. As shown, the system includes an ECU 262 of one or more computing devices, such as a computing device containing one or more processors 264, a memory 266, and other components typically found in general-purpose computing devices. The memory 266 stores information accessible to one or more processors 264, including instructions 268 and data 270 that can be executed or otherwise used by the processor 264. Figures 2A to 2B The description of the processor, memory, instructions, and data is applicable to Figure 2C These components.

[0063] ECU 262 is configured to receive information and control signals from the trailer unit. The onboard processor 264 of ECU 262 can communicate with various systems of the trailer, including a reduction system 272, a signal system 274, and a positioning system 276. ECU 262 can also be operatively coupled to a sensing system 278 having one or more sensors for detecting objects in the trailer environment and a power system 280 for powering local components (e.g., battery power). Some or all of the trailer's wheels / tires 282 can be coupled to the reduction system 272, and the processor 264 can be able to receive information about tire pressure, balance, wheel speed, and other factors that may affect driving in autonomous mode, and forward this information to the processing system of the towing unit. The reduction system 272, signal system 274, positioning system 276, sensing system 278, power system 280, wheels / tires 282, and sensor assembly 284 can be configured as described above. Figures 2A to 2B Operate in the manner described.

[0064] The trailer also includes a set of landing gear 286 and a coupling system 288. The landing gear provides a supporting structure for the trailer when disconnected from the tractor unit. The coupling system 288, which may be part of the coupling system 254, provides the connection between the trailer and the tractor unit. Therefore, the coupling system 288 may include a connection portion 290 (e.g., for a power and / or pneumatic link). The coupling system also includes a kingpin 292 configured for connection to the tractor unit's landing gear wheels.

[0065] Example Implementation

[0066] In view of the structures and configurations described above and shown in the accompanying drawings, various embodiments will now be described in accordance with aspects of the technology.

[0067] The environment surrounding a vehicle can be viewed as having different quadrants or regions. Figure 3Example 300 is shown, illustrating the front, rear, right, and left areas around the vehicle, as well as adjacent areas to the right front, left front, right rear, and left rear areas. These areas are merely exemplary. The vehicle's perception system may cover some or all of the areas around the vehicle to provide as much information as possible about objects in the vehicle's external environment.

[0068] For example, various sensors can be located in different places around the vehicle (see...) Figures 1A to 1C The three sensors 116 of Figure 1 can primarily receive data from the front, left front, and right front regions around the vehicle. In contrast, the canopy housing 102 may include other sensors, such as multiple cameras and / or rotating lidar or radar sensors, to provide a 360° field of view (FOV) around the vehicle.

[0069] Some sensors may have different fields of view, depending on their placement around the vehicle and the type of information they are designed to collect. For example, different lidar sensors may be used for near (short-range) detection of objects adjacent to the vehicle (e.g., less than 2-10 meters), while others may be used for far (long-range) detection of objects 100 meters (or more or less) in front of the vehicle. Mid-range lidar may also be used, for example, to detect objects between 10-100 meters from the vehicle. Multiple lidar units may be positioned toward the front, rear, and / or sides of the vehicle for short-range or long-range object detection. Cameras may be arranged to provide good visibility around the vehicle. Depending on the configuration, some sensor housings may include multiple individual sensors with overlapping fields of view. Alternatively, other sensors may provide a redundant 360° field of view.

[0070] Figure 4 Provided with Figures 1A to 1B An example 400 of the sensor field of view associated with the sensor is shown. Here, if the canopy housing 102 includes a lidar sensor as well as various cameras, radar units, infrared and / or acoustic sensors, each of these sensors can have a different field of view. Thus, as shown, the lidar sensor can provide a 360° FOV 402, while the cameras arranged within the housing 102 can have, for example, coverage such as... Figure 3A separate field of view (FOV) 404 is provided for one or more areas surrounding the vehicle. Sensors located within housing 104 at the front of the vehicle have a forward FOV 406. Housings 106a and 106b on the driver's and passenger's sides of the vehicle may each contain a lidar, radar, camera, and / or other sensors. For example, lidar units within housings 106a and 106b may have respective FOVs 406a and 406b, while radar units, cameras, and / or other sensors within housings 106a and 106b may have respective FOVs 407a and 407b. Similarly, sensors within housings 108a and 108b, positioned towards the rear top portion of the vehicle, each have a corresponding FOV. For example, lidar units within housings 108a and 108b may have corresponding FOVs 408a and 408b, while radar units, cameras, and / or other sensors within housings 108a and 108b may have corresponding FOVs 409a and 409b. Sensors in housings 110a and 110b facing the rear of the vehicle can have corresponding fields of view 410a and 410b. Sensors in housing 112 at the rear end can have a rearward FOV 412. And a series of sensor units 116 arranged along the forward direction of the vehicle can have corresponding FOVs 414, 416, and 418. Each of these fields of view is merely exemplary and is not proportional in terms of coverage. Although only one or two FOVs are shown associated with a given sensor housing, more (or fewer) fields of view may be associated with that sensor housing depending on the number of sensors and their configuration.

[0071] As discussed further below, colliding different types of sensors within the same housing can provide enhanced object detection and enable onboard systems to quickly classify detected objects. The collided sensors can have the same or substantially overlapping fields of view, or additionally provide complementary fields of view.

[0072] Example Scenario

[0073] The elevation and orientation of cameras, lidar, radar, and / or other sensor subsystems will depend on the placement of various housings on the vehicle and the type of vehicle. For example, if a sensor housing is mounted on or above the roof of a large SUV (e.g., vehicle 100), the elevation will typically be higher than if the housing is mounted on the roof of a sedan or sports car (e.g., vehicle 150). Additionally, visibility may not be uniform across all areas of the vehicle due to placement and structural limitations. By varying the placement on the vehicle, a suitable field of view can be obtained for the sensors in each housing. This is crucial for detecting objects immediately adjacent to the vehicle (e.g., within 1-2 meters or no more than 3 meters) as well as objects farther away. In various situations, it may be necessary to detect both adjacent and distant objects, such as checking the immediate vicinity before exiting a parking space to determine if an unprotected left turn is being made.

[0074] Close-range sensing camera system

[0075] In view of the above, various aspects of this technology provide a proximity sensing camera system as part of a sensor suite for objects within a threshold distance of a vehicle. This camera system is designed to prevent the vehicle from becoming stuck (when not moving) or moving awkwardly when the autonomous driving system cannot distinguish between drivable and non-drivable objects within a certain distance of the vehicle. For example, the proximity sensing camera system is configured to provide sensor information on objects within a threshold distance of the vehicle (e.g., a threshold distance not exceeding, for example, 6-10 meters). In some cases, the threshold distance may be no more than 2-3 meters from the vehicle. This information is used to help detect and classify objects such as pedestrians standing next to the vehicle, bicycles or motorcycles parked near the vehicle, and balls, construction signs, or other objects that may be in the vicinity.

[0076] LiDAR sensors can be deployed around a vehicle to minimize blind spots and detect objects. Such sensors are highly capable of detecting the presence of objects. However, the sensor data from LiDAR (e.g., LiDAR point clouds) alone may not be sufficient for an autonomous driving system to determine what type of object is present. When it is unclear what type of object is nearby, the vehicle may take conservative actions, such as waiting for a few minutes to observe its surroundings, honking the horn, flashing lights, etc., to observe how the object reacts; or reversing or slowly moving forward to gain a clearer image of its surroundings. However, this may not provide additional useful information about the object and could irritate or disturb passengers, nearby pedestrians, and other road users.

[0077] Therefore, according to one aspect of this technology, one or more cameras can be arranged together with a LiDAR sensor in a single sensor housing to enable rapid object classification, such as determining whether it is a pedestrian, bicycle, or traffic cone. The camera's field of view can encompass the LiDAR's field of view and, in some examples, can be larger than the LiDAR's field of view. This can be achieved using one or more cameras with complementary or otherwise overlapping fields of view. For example, a person may be standing or sitting next to a vehicle. This could happen, for example, when a person leaves a vehicle, appears from behind a nearby parked car, or is already in a blind spot before the vehicle starts moving or prepares to leave a parking space. Other scenarios where such a camera system is beneficial include unprotected steering, high-speed lane changes, oncoming traffic obstructed by other objects, low-mounted metering lights (such as at highway entrance ramps), identifying traffic cones and other construction items, and detecting small foreign object debris (FOD).

[0078] The classification of detected objects may include determining the size, proximity, and orientation of the detected objects. The system is configured such that the camera can see a minimum threshold volume occupied by the object of interest (e.g., at least 50% of a cuboid or other 3D shape). In one example, each camera in the proximity sensing system is co-located with a matching LiDAR sensor. For example, the camera may be no more than 1 foot or 0.3 meters away from the LiDAR sensor, for example, to avoid parallax. The cameras can be mounted to the vehicle using the same bracket or housing as the LiDAR, or they can be mounted separately. In general operation, the system's FOV should provide a 360° field of view around the vehicle up to 3 meters away.

[0079] The camera resolution should be sufficient to satisfy threshold classification based on a minimum number of pixels. For example, a classification threshold could be the ability to classify a specific object of a chosen cuboid shape using no more than 32-64 pixels when the object is within 3 meters of the vehicle. Alternatively, threshold classification could require a camera with a resolution requirement between 0.1 and 0.4 mrad / pixel. Threshold classification requirements can vary, for example, depending on the type of object and scene (e.g., whether an adult or child is standing or sitting next to the vehicle, whether a motorcyclist is approaching from more than 100 meters behind the vehicle, etc.).

[0080] like Figure 5As shown in Example 500, cameras can provide different and potentially overlapping coverage areas. In this example, up to eight (or more) cameras can be used to provide forward, lateral, and rearward FOVs. For example, FOVs 502a and 502b cover portions of the left-front and right-front areas around the vehicle. FOVs 504a and 504b overlap with FOVs 502a and 502b, providing additional coverage along the front, left-front, and right-front areas. FOV 506 provides coverage in the area in front of the vehicle. FOVs 508a and 508b provide coverage facing the rear of the vehicle, for example, along the left / right and left-rear and right-rear areas. And FOV 510 provides coverage along the area behind the vehicle.

[0081] The camera may need to operate under all ambient lighting conditions. Thus, different cameras may rely on illumination from vehicle sources (e.g., headlights, parking lights, reversing lights, driving lights) and ambient sources (e.g., other vehicles, streetlights, etc.). Alternatively or additionally, IR and / or optical illuminators may be arranged to provide illumination to the camera. For example, one or more illuminators may be arranged adjacent to the camera on the sensor housing.

[0082] For example, cameras with forward and rearward FOVs may not require separate illumination because headlights and brake lights or reverse lights can provide sufficient light in some scenarios. However, cameras with side FOVs may require supplemental illumination in low-light conditions. This supplemental lighting can be provided by a near-infrared (NIR) emitter placed near the camera. As discussed further below, in one configuration, a pair of “saddle-shaped” illuminator modules (e.g., NIR modules) can be used on either side of the camera, where each illuminator module compensates for the occlusion of the other. Alternatively, a single monolithic illuminator module can be used.

[0083] Figures 6A to 6C An exemplary camera and illuminator configuration is shown. Specifically, Figure 6AA first configuration 600 is shown, in which illuminators are positioned on the sides of the respective cameras. Here, four cameras are shown around the vehicle: a front camera 602, a rear camera 604, a left-facing camera 606a, and a right-facing camera 606b. As shown, a pair of illuminators 608a and 608b are arranged on either side of the front camera 602. Similarly, a pair of saddle-shaped illuminators 610a and 610b are arranged on either side of the rear camera 604. However, in an alternative configuration, only one illuminator may be arranged on the side of the front camera 602 and / or the rear camera 604. The side cameras 606a and 606b are shown each having a corresponding illuminator 612a, 612b positioned on its side. In these examples, the illuminator 612 is positioned behind the side camera 606. Here, the side camera 606 can receive some illumination from the front illuminators 608a and 608b, which can supplement the IR illumination from the illuminators 612a and 612b.

[0084] Figure 6B A second configuration 650 is shown, in which a single illuminator is positioned above the respective camera. (Compared to...) Figure 6A Similarly, four cameras are shown around the vehicle: a front camera 652, a rear camera 654, a left-facing camera 656a, and a right-facing camera 656b. As shown, an illuminator 658 is positioned above the front camera 652. Similarly, an illuminator 660 is positioned above the rear camera 654. The side cameras 656a and 656b are shown each having a corresponding illuminator 662a, 662b positioned above it. In an alternative configuration, the illuminators may be positioned below the respective cameras. In yet another example, the transmitter may be placed in another location around the vehicle, not co-located with the cameras, such as along the roof. For example, an illuminator placed on the roof could be used to illuminate the entire field of view of a camera on a given side of the vehicle.

[0085] In these examples, a single illuminator module can be placed on either side of the camera. However, some amount of the projected light may be obstructed. For example, with a single illuminator module on one side of the camera, there may be significant light obstruction on the other side, which can be detrimental in low-light conditions. With a single module above the camera, it may be necessary to move the module upwards and forwards away from the camera to reduce obstruction. At some locations around the vehicle (e.g., at the front and rear, but also on the sides), there may be constraints regarding the vehicle's width and potential impacts on other side sensors. And with a single module below the camera, upward illumination may be reduced due to obstruction. This could affect the sensor suite's ability to classify nearby objects, such as a person standing next to the vehicle. Therefore, it is desirable to place the illuminator module where the corresponding camera's field of view is minimized, as this reduces the likelihood of field-of-view obstruction. When the vertical field of view is smaller than the horizontal field of view, placing the illuminator above and / or below the camera may be suitable. However, this may not be possible in some cases due to other constraints related to the sensor suite, vehicle size, etc.

[0086] In view of this, Figure 6C A third configuration 680 is shown, in which paired illuminators are positioned on either side of each respective camera. (Compared to...) Figures 6A to 6B Similarly, four cameras are shown around the vehicle: a front camera 682, a rear camera 684, a left-facing camera 686a, and a right-facing camera 686b. As shown, a pair of illuminators 688a and 688b are arranged on either side of the front camera 602. Likewise, a pair of saddle-shaped IR illuminators 690a and 690b are arranged on either side of the rear camera 684. In this example, the side cameras 686a and 686b are shown each having a corresponding pair of illuminators 692a, 692b or 694a, 694b arranged on either side of them. This arrangement helps minimize occlusion that may occur when only one illuminator module is placed to the side, top, or bottom of the corresponding camera. The following discussion... Figure 7F An arrangement of a pair of side illuminators for each of cameras 682, 684 and 686 is shown.

[0087] In any of these configurations, the camera lens may have a hydrophobic coating to repel water. Furthermore, the camera can be positioned along the exterior of the vehicle to facilitate easy cleaning of the lens using an onboard cleaning system. These features will be discussed further below.

[0088] Perimeter sensor housing

[0089] According to this technology, a housing having an integrated sensor assembly including multiple different sensors (e.g., lidar, camera, radar, etc.) can be positioned at various locations along the vehicle. Figures 1A to 1C An exemplary housing placement for such an integrated sensor assembly is shown. As described above, each location provides specific coverage around the vehicle from its sensor, which has a specific field of view. The arrangement of each sensor along the housing and relative to other sensors is important because significant advantages (or disadvantages) may exist for different arrangements. One or more of the sensor housings may house proximity sensing camera assemblies as described above. Several examples are discussed below.

[0090] exist Figure 7A In the first example shown, the sensor kit is arranged in the housing along the left or right side perimeter of the vehicle, in front of the driver's or passenger's side door. Specifically, Figure 7A The following view is shown, illustrating a first housing 700a along the left front side panel and a second housing 700b along the right front side panel. Housing 700b may be a mirror image of housing 700a. Figures 7B to 7F Various views of the side perimeter housing 700 are shown. (e.g.) Figure 7B Perspective and Figure 7C As shown in the front view, the sensor suite in the side perimeter housing 700 includes a lidar unit 702, a proximity sensing camera assembly 704, a radar unit 706, a forward perimeter view camera 708, and a side perimeter view camera 710.

[0091] like Figures 7B to 7C As shown, the radar unit 706 is positioned between the forward-facing camera and the side camera on one side and the lidar and proximity sensing camera assembly on the other side. The separation of the radar unit from the aligned lidar and proximity sensing camera assembly avoids interference and potential obstruction.

[0092] A proximity sensing camera assembly 704 is positioned below the lidar unit 702, for example, to enable object classification to complement object detection by the lidar unit. Although shown aligned below the lidar unit 702, the camera of the proximity sensing camera assembly 704 can also be positioned anywhere within approximately 0.25-0.4 m of the lidar unit 702. To avoid parallax that could adversely affect image classification, the camera should be as close as possible to the lidar unit without creating occlusion between the sensors. And although shown aligned below the lidar unit 702, the camera of the proximity sensing camera assembly 704 can be positioned above the lidar unit 702. Either arrangement minimizes the possibility of occlusion and parallax. Spatial constraints of the housing unit and / or the overall dimensions of the carrier may also limit the placement of the sensors relative to each other.

[0093] Separately, such as Figure 7D and Figure 7EAs shown in the left and right views, a separating surface 712 exists between the lidar unit 702 and the proximity sensing camera assembly 704. This separating surface can be arranged at a downward-sloping angle. The outward-sloping surface allows water, snow, etc., to slide off, minimizing the possibility of the sensor being obstructed or blocked. Since the lidar sensor may have a limited field of view directly beneath it, aligning the camera assembly directly below the lidar helps in object detection of objects potentially obstructed by the lidar. For example, as shown in the side view, the proximity sensing camera assembly 704 is angled downwards to cover the immediate vicinity around the vehicle.

[0094] Figure 7F The enlarged view shows that component 704 includes a camera 714, a pair of illuminator modules 716a and 716b, and a set of cleaning mechanisms 718a, 718b, and 718c. An extension 720 extending from the housing surface may be included to ensure no light leakage into the lens of camera 714. Each module 716a and 716b may include one or more secondary lenses 722, which can be used to focus light, such as IR light, along one or more desired areas. For example, these secondary lenses 722 may increase the field of view of the illuminator modules. The cleaning mechanism 718 may include a liquid and / or compressed air spray to clean the camera and / or illuminator modules. Alternatively or additionally, one or more wipers (not shown) may be used to keep the lenses clean.

[0095] Figure 8 An example occlusion scenario 800 is illustrated. While the lidar sensor can have a wide coverage azimuth angle of, for example, 180°, as shown, it can have an occlusion region 802 immediately below the lidar sensor and adjacent to the vehicle, which is shown as a shaded triangular region. Because the proximity sensing camera assembly is configured to supplement the perceived information obtained by the lidar sensor and is configured to be angled downwards, it is able to mitigate the occlusion region 802 of the lidar sensor. For example, if there is an object adjacent to the front tire, the camera of the proximity sensing camera assembly is configured to detect it, as shown by the linear element 804 within the shaded region. While the camera may not be able to see within a few centimeters of the side of the vehicle, the camera can be positioned such that objects near the vehicle are at least 50% visible. In one example, the camera can have an azimuth field of view on the order of 170-200°.

[0096] Back Figure 7B As described above, a forward perimeter view camera 708 and a lateral perimeter view camera 710 are present in the exemplary side perimeter housing 700. These cameras are configured to provide views such as Figure 9Example 900 illustrates the minimum azimuth angle coverage of the front and side images. For example, the side camera 710 can be configured to provide a minimum FOV 902 of + / -15° to the side of the vehicle, although it can provide up to + / -30-40° or more. In one example, there might be a minimum FOV 904 of 15° towards the rear of the vehicle and a minimum FOV 906 of 25-35° towards the front of the vehicle. The forward camera 708 can have an outward azimuth FOV 908 in the order of 10-20° and an inward azimuth FOV 910 in the order of 20-40°. In some cases, the driver-side forward camera can have a wider FOV than the passenger-side forward camera, for example, to provide increased visibility for left-hand steering. Figure 7B As shown, the forward-facing camera 708 can be configured to be higher (or lower) than the side-facing camera 710. This is done to accommodate various sensor units within the housing 700.

[0097] These cameras can be used in conjunction with other sensors to improve radar detection and classification, for example, in challenging scenarios involving obstructed intersection traffic and unprotected turns. In one scenario, cameras 708 and 710 are not primarily used for close-range sensing, but rather utilize ambient light in the absence of an IR illuminator. The side perimeter view camera 710 can be positioned as far forward as possible within the housing 700 or at other locations on the vehicle to reduce the likelihood of obstruction when the vehicle is slowly entering an intersection or making a sharp turn. The forward perimeter view camera 708 can also be positioned as far forward as possible for better observation of obstructing objects in front of the vehicle.

[0098] Figure 10 An example of occlusion in a turning scenario 1000 is shown. In this scenario, vehicle 1002 is preparing to turn left, as indicated by the dashed arrow. Here, from the field of view of one of the sensors on vehicle 1002, a truck 1004 or other object may obstruct another vehicle 1006. For example, a roof-mounted sensor 1008 may have a field of view 1010, which is partially obstructed in area 1012 by the truck 1004. However, a perimeter-facing camera 1014 has a different field of view 1016 capable of seeing at least a portion of the other vehicle 1006. Perimeter-facing cameras are advantageous in a variety of other scenarios, such as, for example, maneuvering around another vehicle when there may be oncoming traffic, viewing adjacent lanes behind an autonomous vehicle when there is an obstructing vehicle behind it, and merging into high-speed traffic, such as via a highway entrance ramp.

[0099] Another example of a perimeter housing assembly is in Figures 11A to 11C shown in . Specifically, Figures 11A to 11B The rear housing assembly 1100 is shown, which is illustrated as being in Figure 11CExample location 1110 on the rear trim strip of a car or other vehicle. Although location 1110 is shown on the left rear side of the vehicle, another rear housing assembly may also be located on the right side of the vehicle. Figures 11A to 11B As shown, a first sensor 1102 and a second sensor 1104 are disposed within the housing 1100. For example, the first sensor 1102 is a radar sensor, while the second sensor is a camera. These sensors are capable of providing information about other vehicles approaching from behind, such as for high-speed lane changes to the right or left.

[0100] like Figure 12 As shown in Example 1200, a rearward perimeter view camera is configured to provide a rearward image with minimal azimuth coverage. For example, rearward camera 1100 may be configured to provide a field of view (FOV) 1202 of 30-60° behind a vehicle. For instance, the rearward camera may have an outward azimuth of 15-35° (e.g., to see approaching vehicles in adjacent lanes) and an inward azimuth of 10-25° (e.g., to see following vehicles in the same lane). Figure 13 Scene 1300 is shown, which shows a rear-facing camera able to see an approaching car in the adjacent (left) lane that would otherwise be obscured by a truck.

[0101] exist Figure 14A In another example shown, the front sensor housing 1400 is arranged along or adjacent to the front bumper, for example, for detecting and classifying objects directly in front of the vehicle. Figures 14B to 14E Various views of the front sensor housing 1400 are shown. (See attached image.) Figure 14B Perspective and Figure 14C As shown in the front view, the sensor suite in the front sensor housing 1400 includes a lidar unit 1402 and a proximity sensing camera assembly 1404.

[0102] The proximity sensing camera assembly 1404 is positioned directly above the lidar unit 1402, for example, to enable object classification to supplement object detection by the lidar unit. Although shown aligned above the lidar unit 1402, the camera of the proximity sensing camera assembly 1404 can also be positioned anywhere within approximately 0.25-0.4 m of the lidar unit 1402. To avoid parallax that could adversely affect image classification, the camera should be as close as possible to the lidar unit without creating occlusion between the sensors. Although shown above the lidar unit 1402, the camera of the proximity sensing camera assembly 1404 can be positioned below the lidar unit 1402. Either arrangement minimizes occlusion. Spatial constraints of the housing unit and / or the overall dimensions of the carrier may also limit sensor placement.

[0103] like Figure 14D As shown in the side view, a separating surface 1406 exists between the lidar unit 1402 and the proximity sensing camera assembly 1404. The separating surface may be arranged at an angle to allow, for example, water, snow, etc., to slide off, thus minimizing the possibility of the sensor being obstructed or blocked. The proximity sensing camera assembly 1404 is also shown angled downwards in the side view to cover the immediate vicinity around the vehicle. Figure 14E The enlarged view shows that component 1404 includes a camera 1408, a pair of illuminator modules 1410a and 1410b, and a set of cleaning mechanisms 1412a, 1412b, and 1412c. An extension 1414 may be included extending from the housing surface to ensure no light leakage into the lens of camera 1408. As shown, each module 1410a and 1410b may include one or more secondary lenses 1416, which can be used to focus light along one or more desired areas. The cleaning mechanism 1412 may include liquid and / or compressed air spraying to clean the camera and / or illuminator modules.

[0104] Figure 15 A variant 1500 of the front sensor housing 1400 is shown. In this variant, the housing 1500 includes a lidar unit 1402 and a proximity sensing camera 1502, omitting the IR illumination module and cleaning mechanism. In another variant without illumination, a cleaning mechanism may be included. Similarly, in yet another variant without a cleaning mechanism, illumination may be included.

[0105] exist Figure 16A In another example shown, the rear sensor housing 1600 is arranged along or adjacent to the rear bumper, for example, for detecting and classifying objects directly behind the vehicle. Figures 16B to 16E Various views of the rear sensor housing 1600 are shown. (See attached image.) Figure 16B Perspective and Figure 16C As shown in the front view, the sensor suite in the front sensor housing 1600 includes a lidar unit 1602 and a proximity sensing camera assembly 1604.

[0106] The proximity sensing camera assembly 1604 is positioned directly above the lidar unit 1602, for example, to enable object classification to supplement object detection by the lidar unit. Although shown aligned above the lidar unit 1602, the camera of the proximity sensing camera assembly 1604 can be positioned anywhere within approximately 0.25-0.4 m of the lidar unit 1602. To avoid parallax that could adversely affect image classification, the camera should be as close as possible to the lidar unit without creating occlusion between the sensors. Although shown above the lidar unit 1602, the camera of the proximity sensing camera assembly 1604 can be positioned below the lidar unit 1602. Either arrangement minimizes occlusion. Spatial constraints of the housing unit and / or the overall dimensions of the carrier may also limit sensor placement.

[0107] like Figure 16D As shown in the side view, a separating surface 1606 exists between the lidar unit 1602 and the proximity sensing camera assembly 1604. The separating surface may be arranged at an angle to allow, for example, water, snow, etc., to slide off, thus minimizing the possibility of the sensor being obstructed or blocked. The proximity sensing camera assembly 1604 is also shown angled downwards in the side view to cover the immediate vicinity around the vehicle. Figure 16E The enlarged view shows that component 1604 includes a camera 1608, a pair of illuminator modules 1610a and 1610b, and a set of cleaning mechanisms 1612a, 1612b, and 1612c. An extension 1614 may be included extending from the housing surface to ensure that no light leaks into the lens of camera 1608. As shown, each module 1610a and 1610b may include one or more secondary lenses 1616, which can be used to focus light along one or more desired areas. The cleaning mechanism 1612 may include liquid and / or compressed air spraying to clean the camera and / or illuminator modules.

[0108] Figure 17 A variant 1700 of the front sensor housing 1600 is shown. In this variant, the housing 1700 includes a lidar unit 1602 and a proximity sensing camera 1702, omitting the illumination module and cleaning mechanism. In another variant without illumination, a cleaning mechanism may be included. Similarly, in yet another variant without a cleaning mechanism, illumination may be included.

[0109] As described above, the proximity sensing cameras of various sensor housings are arranged at a downward angle. For example, they can have a downward angle on the order of 20-40° to maximize lower field-of-view coverage and cover as much of the relative FOV of the LiDAR as possible, since these cameras improve the LiDAR's detection and classification. Although different arrangements for the co-positioning of the proximity sensing camera assembly and the LiDAR unit have been shown, generally each camera is positioned relative to its LiDAR unit to minimize occlusion of the LiDAR in all cases.

[0110] Sensor cleaning is crucial for the proper and effective operation of the sensing system. When the vehicle operates in autonomous driving mode, different options exist for cleaning various sensors. For example, the cleaning system can spray cleaning fluid onto the camera (and IR transmitter), use a wiper, and / or a blower. The spraying or other cleaning unit can be in a fixed position relative to the sensor, or it can be configured to extend outwards to clean the unit as needed. The cleaning unit should be arranged to prevent any sensors in the sensor housing from being obstructed or otherwise affecting the sensor FOV. For example, the spray nozzles of cleaning mechanisms 718, 1412, and 1612 are positioned to not obstruct the camera and not reflect light back to the camera.

[0111] In addition to cleaning, sensors can be cared for, for example, by providing heat to eliminate condensation or frost on cameras or other sensors. For instance, a defrost heater could be positioned along the front window element of each perimeter view camera, such as a heating element sandwiched between the front glass and housing of the camera unit.

[0112] Shrouds or other structures can be used to limit dust and other foreign objects from covering the sensor. However, as with cleaning components, shrouds should not obstruct the sensor or otherwise affect its field of view (FOV). In the case of cameras or illumination emitters, a hydrophobic coating can be applied to the glass or plastic cover to minimize moisture buildup.

[0113] When selecting the type and placement of the cleaning unit, the sensor's location within the housing and along the vehicle should be considered. For example, cleaning a camera located in a side mirror assembly can be difficult. For instance, spraying liquid onto a specific point might be challenging, or there might be limitations on how the liquid is delivered across the vehicle to the cleaning unit (e.g., if the cleaning unit is on a door versus not). The type of cleaning mechanism can be selected based on the sensor's importance to autonomous driving. For example, cleaning the front sensor housing unit might be more critical than cleaning the rear sensor housing unit because a clear view of the vehicle ahead is more relevant for certain driving maneuvers than a view following the vehicle. Thus, redundant (e.g., spray system plus wiper) cleaning modules can be used for the more critical sensor housing. For other sensor housings, redundant cleaning mechanisms may not be available. In this case, the cleaning mechanism can be actuated only at specific speeds (e.g., below 35-45 mph) or when the vehicle is stationary, as cleaning may be less time-sensitive compared to other sensor housings.

[0114] Figure 18 A flowchart 1800 of the method according to the specific actions described above is shown. In block 1802, the control system of a vehicle configured to operate in autonomous driving mode initiates the operation of the lidar sensor of the vehicle's perception system to obtain lidar data within a threshold distance in the area surrounding the vehicle. For example, the threshold distance may be within 1-3 meters of the vehicle. In block 1804, the control system initiates image capture by the image sensor of the perception system before the vehicle performs a driving action. The image sensor is positioned adjacent to the lidar sensor and arranged along the vehicle to have an overlapping field of view of the area surrounding the vehicle within the threshold distance. The image sensor provides a selected resolution for objects within the threshold distance. In block 1806, the control system receives lidar data from the lidar sensor and the captured image from the image sensor. This can be done simultaneously or sequentially. In block 1808, the control system processes the lidar data to detect objects in the area within the threshold distance of the vehicle. In block 1810, the control system processes the captured image to classify the detected objects. And in block 1812, the control system determines whether to cause one or more systems of the vehicle to perform a driving action based on the classification of the detected objects.

[0115] Unless otherwise stated, the foregoing examples and embodiments are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the above features can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be considered illustrative rather than limiting to the subject matter defined by the claims. Furthermore, the provision of examples described herein, and the use of terms such as "such as," "comprising," etc., should not be construed as limiting the subject matter of the claims to the specific examples or embodiments. Moreover, the same reference numerals in different figures may identify the same or similar elements. These processes or other operations may be performed in different orders or simultaneously, unless expressly indicated otherwise herein.

Claims

1. An external sensing system for a vehicle configured to operate in an autonomous driving mode, the external sensing system comprising: A lidar sensor, arranged in a perimeter housing surrounding a vehicle, has a first field of view configured to detect objects in at least one region of the external environment surrounding the vehicle and within a threshold distance of the vehicle, the first field of view including an obstruction area below the lidar sensor and in immediate vicinity of the vehicle. An image sensor is arranged in the perimeter housing, adjacent to the lidar sensor and angled downwards to have a second field of view of the area of ​​the external environment within the threshold distance of the vehicle. The second field of view covers at least a portion of the occlusion area of ​​the lidar sensor in the immediate vicinity of the vehicle within the first field of view of the vehicle. The image sensor provides a selected resolution for objects within the threshold distance. as well as A control system, operably coupled to the image sensor and the lidar sensor, the control system including one or more processors configured to: The operation of the lidar sensor is initiated to obtain lidar data in the area within the threshold distance of the vehicle; Image capture by the image sensor is initiated before the vehicle performs driving actions; Receive lidar data from the lidar sensor and images captured from the image sensor; Perform processing on the lidar data to detect objects in the area within the threshold distance of the vehicle; Perform processing on the captured image to classify the detected objects; as well as The classification of the detected objects determines whether one or more systems of the vehicle should perform the driving action.

2. The external sensing system according to claim 1, wherein, Classifying the detected objects includes determining at least one of the size, proximity, or orientation of the detected objects.

3. The external sensing system according to claim 1, wherein, The image sensor is configured to observe the minimum threshold volume occupied by the detected object.

4. The external sensing system according to claim 3, wherein, The minimum threshold volume is at least 50% of the 3D shape surrounding the detected object.

5. The external sensing system according to claim 1, wherein, The image sensor is set to be no more than 0.3 meters away from the lidar sensor.

6. The external sensing system according to claim 1, wherein, The perimeter shell is arranged on the outer surface of the vehicle.

7. The external sensing system according to claim 1, wherein, The threshold distance is no more than 3 meters from the vehicle.

8. The external sensing system according to claim 1, wherein, The lens of the image sensor has a hydrophobic coating.

9. The external sensing system according to claim 1, wherein, The image sensor is part of a proximity sensing camera system, which also includes at least one illuminator configured to illuminate the field of view of the image sensor.

10. The external sensing system according to claim 9, wherein, The at least one illuminator is arranged adjacent to one side of the image sensor.

11. The external sensing system according to claim 9, wherein, The at least one illuminator is arranged above the image sensor.

12. The external sensing system according to claim 9, wherein, The at least one illuminator includes a pair of illuminators arranged on opposite sides of the image sensor.

13. The external sensing system according to claim 9, wherein, The proximity sensing camera system also includes at least one cleaning mechanism configured to clean the image sensor and / or the at least one illuminator.

14. The external sensing system according to claim 1, wherein, The image sensor is vertically aligned below the lidar sensor.

15. The external sensing system according to claim 1, wherein, The image sensor is vertically aligned above the lidar sensor.

16. The external sensing system according to claim 1, wherein, The image sensor being positioned adjacent to the lidar sensor includes placing the image sensor below the lidar sensor.

17. The external sensing system according to claim 1, wherein, The image sensor is positioned adjacent to the lidar sensor, meaning the image sensor is positioned above the lidar sensor.

18. A vehicle configured to operate in an autonomous driving mode, the vehicle comprising: The driving system includes: The deceleration system is configured to control the braking of the vehicle; The acceleration system is configured to control the acceleration of the vehicle; and The steering system is configured to control wheel orientation and vehicle direction; and External sensing system, including: A lidar sensor, arranged in a perimeter housing surrounding a vehicle, has a first field of view configured to detect objects in at least one region of the external environment surrounding the vehicle and within a threshold distance of the vehicle, the first field of view including an obstruction area below the lidar sensor and in immediate vicinity of the vehicle. An image sensor, disposed within the perimeter housing, is arranged adjacent to the lidar sensor and angled downwards to have a second field of view of the area of ​​the external environment within the threshold distance of the vehicle. The second field of view covers at least a portion of an occlusion area within the immediate vicinity of the vehicle of the lidar sensor's first field of view. The image sensor provides a selected resolution for objects within the threshold distance. A control system, operably coupled to the image sensor and the lidar sensor, the control system including one or more processors configured to: The operation of the lidar sensor is initiated to obtain lidar data in the area within the threshold distance of the vehicle; Image capture by the image sensor is initiated before the driving system performs a driving action; Receive lidar data from the lidar sensor and images captured from the image sensor; Perform processing on the lidar data to detect objects in the area within the threshold distance of the vehicle; Perform processing on the captured image to classify the detected objects; and The classification of the detected objects determines whether one or more systems of the vehicle should perform the driving action.

19. The vehicle according to claim 18, wherein, The image sensor is configured to observe the minimum threshold volume occupied by the detected object.

20. The vehicle according to claim 18, wherein, The image sensor is set to be no more than 0.3 meters away from the lidar sensor.

21. The vehicle according to claim 18, wherein, The perimeter shell is arranged on the outer surface of the vehicle.

22. The vehicle according to claim 18, wherein, The image sensor being positioned adjacent to the lidar sensor includes placing the image sensor below the lidar sensor.

23. The vehicle according to claim 18, wherein, The image sensor is positioned adjacent to the lidar sensor, meaning the image sensor is positioned above the lidar sensor.

24. A method performed by a vehicle configured to operate in an autonomous driving mode, comprising: The control system of the vehicle, configured to operate in autonomous driving mode, initiates the operation of the lidar sensor of the vehicle's perception system to obtain lidar data within a threshold distance in the area around the vehicle. The lidar sensor is arranged in a perimeter housing surrounding the vehicle and has a first field of view, which includes an obstruction area below the lidar sensor and in the immediate vicinity of the vehicle. Before the vehicle performs a driving action, the control system initiates image capture by an image sensor of the perception system. The image sensor is arranged in the perimeter housing, adjacent to the lidar sensor, and angled downwards to have a second field of view over the area surrounding the vehicle within the threshold distance. The second field of view covers at least a portion of the occluded area in the immediate vicinity of the vehicle within the first field of view of the lidar sensor. The image sensor provides a selected resolution for objects within the threshold distance. The control system receives lidar data from the lidar sensor and images captured by the image sensor. The control system processes the lidar data to detect objects in the area within the threshold distance of the vehicle; The captured images are processed by the control system to classify the detected objects; and The control system determines whether to cause one or more systems of the vehicle to perform the driving action based on the classification of the detected object.

25. The method according to claim 24, wherein, Classifying detected objects includes determining at least one of the detected objects' size, proximity, or orientation.

26. The method according to claim 24, wherein, The image sensor being positioned adjacent to the lidar sensor includes placing the image sensor below the lidar sensor.

27. The method according to claim 24, wherein, The image sensor is positioned adjacent to the lidar sensor, meaning the image sensor is positioned above the lidar sensor.

Images