Road condition deep learning model

Abstract

The present technology relates to using on-board sensor data (1002 A ), off-board information (1002 B ), and deep learning models (1008) to classify and / or perform regression analysis on road wetness based on a set of input information. Such information includes on-board (802) and / or off-board signals (804) obtained from one or more sources, including on-board perception sensors, other on-board modules, external weather measurements, external weather services, etc. Ground truth values include measurements of water film thickness and / or ice coverage on road surfaces. The ground truth values, on-board, and off-board signals are used to build models. The built models can be deployed in autonomous vehicles for classifying / regressing (1016) road wetness using on-board and / or off-board signals as inputs without reference to ground truth values. The models can be applied in various ways to enhance autonomous vehicle operation, e.g., by altering current driving actions, modifying planned routes or trajectories, activating on-board cleaning systems, etc.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of patent application serial number 16 / 893,664, filed on June 5, 2020, the entire disclosure of which is incorporated herein by reference. Background Technology

[0003] Autonomous vehicles, such as those that do not require a human driver, can be used to assist in transporting passengers or goods from one location to another. Such vehicles can operate in a fully autonomous mode or a partially autonomous mode where a human can provide some driving input. To operate in autonomous mode, vehicles can employ various onboard sensors to detect characteristics of the external environment and use the received sensor information to perform various driving maneuvers. Road conditions, including water on the road, can adversely affect vehicle operation, and considerations include how to evaluate information from the sensor system, when to use the windshield wiper system, and real-time and planned driving behavior. Summary of the Invention

[0004] This technology relates to classifying road humidity and / or performing regression analysis on road humidity based on a set of input information using onboard sensor signals, other environmental information, and deep learning (DL) models. Input information may include onboard and / or offboard signals from one or more sources such as onboard perception sensors mounted along the vehicle, onboard modules of other vehicles, external weather measurements, external weather services, etc. Ground truth includes measurements of road surface water thickness, such as water film thickness and / or ice cover. Ground truth, onboard signals from vehicle sensors or systems, and offboard signals from sources outside the specific vehicle are used to build the DL model. The constructed model can be deployed in autonomous vehicles to classify / regress road humidity using onboard and / or offboard signals as input without referencing or relying on ground truth during real-world autonomous driving. The model can be applied in various ways to enhance autonomous vehicle operation, such as by modifying current driving actions, altering planned routes or trajectories, activating onboard cleaning systems, etc.

[0005] According to one aspect, a method for generating a deep learning model of road conditions is provided. The method includes receiving, by one or more processors, sensor data of the environment along a portion of a road from one or more onboard vehicle sensors as a first set of training inputs; receiving, by one or more processors, non-onboard information associated with that portion of the road as a second set of training inputs; evaluating the received first set of training inputs and the received second set of training inputs relative to ground truth data of that portion of the road by the one or more processors to provide a classification or continuous estimate of humidity in one or more areas along that portion of the road, the ground truth data including one or more measurements of water thickness across one or more areas of that portion of the road; generating road humidity information based on the received first and second sets of training inputs and the ground truth data; generating a deep learning model of road conditions from the road humidity information; and storing the generated deep learning model of road conditions in memory.

[0006] Received sensor data may include one or more of the following: lidar return, camera still images, camera video, radar return, audio signals, or outputs from the vehicle's onboard modules. Non-vehicle information may include one or more of the following: weather station information, public weather forecasts, road map data, crowdsourced information, or observations from one or more other vehicles.

[0007] The method may also include performing signal fusion on some or all of the received sensor data and received non-vehicle information. The method may also include applying weights to different signals of the received sensor data. In one example, the method further includes performing statistical analysis to determine which received sensor data is correlated with ground truth data before constructing a deep learning model of road conditions. Here, the method may also include deemphasing any received sensor data that does not meet a correlation threshold with the ground truth data.

[0008] The method may further include masking out information about one or more dynamic objects on the road from the received sensor data before constructing the deep learning model of road conditions. The method may include limiting the received sensor data to a selected range or distance from one or more vehicle sensors before constructing the deep learning model of road conditions. The method may include smoothing benchmark ground truth measurements of benchmark ground truth data before constructing the deep learning model of road conditions.

[0009] In one example, the method further includes applying a deep learning model of road conditions to one or more road areas to identify the probability of humidity in each area or to estimate the water film depth in each area. In another example, the method includes, based on the output of the deep learning model of road conditions, causing one or more systems of the autonomous vehicle to perform modifications to the current driving action, modify the planned route or trajectory, or activate at least one of the autonomous vehicle's onboard cleaning systems.

[0010] In another example, storing the generated deep learning model of road conditions in memory includes storing the generated deep learning model of road conditions in the memory of one or more vehicles that are not equipped with a benchmark ground truth measurement sensor. The stored deep learning model of road conditions is configured to evaluate real-time road humidity based on real-time sensor data and selected non-vehicle information.

[0011] The baseline ground truth data may include human-labeled examples of road moisture. Furthermore, one or more measurements of water thickness may be the water film thickness or one or more measurements of ice cover over one or more areas spanning a section of the road.

[0012] According to another aspect, a system is configured to operate a vehicle in an autonomous driving mode. The system includes a memory storing a deep learning model of road conditions. The model involves discrete classification or continuous regression / estimation of road humidity. The system also includes one or more processors operatively coupled to the memory. The one or more processors are configured to receive sensor data from one or more sensors of the vehicle's perception system when operating in autonomous driving mode. The one or more sensors are configured to detect objects or conditions in the environment surrounding the vehicle. The one or more processors are configured to use the stored model to generate information associated with the discrete classification or continuous regression / estimation of road humidity based on the received sensor data, and to use the generated information to control the operation of the vehicle in autonomous driving mode.

[0013] In one example, a model is formed by evaluating a first set of training inputs—sensor data of the environment along that section of the road from one or more onboard sensors—relative to ground truth data about a section of the road, and a second set of training inputs—non-onboard information associated with that section of the road. The ground truth data includes one or more measurements of water thickness across one or more areas of that section of the road. The first set of training inputs—sensor data—may include one or more of the following: lidar returns, camera still images, camera video, radar returns, audio signals, or outputs from vehicle onboard modules. The second set of training inputs—non-onboard information—may include one or more of the following: weather station information, public weather forecasts, road map data, crowdsourced information, or observations from one or more other vehicles.

[0014] Using the generated information to control vehicle operation in autonomous mode may include altering the current driving action, modifying the planned route or trajectory, or activating at least one of the onboard cleaning systems.

[0015] According to another aspect, the vehicle is capable of operating in an autonomous driving mode, wherein the vehicle includes a system configured to operate the vehicle and an onboard perception system as described above. Attached Figure Description

[0016] Figure 1A-Figure 1B An example passenger-type vehicle configured for use with various aspects of this technology is illustrated.

[0017] Figures 1C-1D Example cargo type vehicles configured for use with various aspects of this technology are shown.

[0018] Figure 2 This is a block diagram of an example passenger-type vehicle system based on various aspects of this technology.

[0019] Figures 3A-3B This is a block diagram of a system for example cargo-type vehicles based on various aspects of this technology.

[0020] Figure 4 Examples of true values ​​for detection benchmarks based on various aspects of this technology are illustrated.

[0021] Figure 5 Example sensor fields of view for passenger-type vehicles according to various aspects of this disclosure are illustrated.

[0022] Figures 6A-6B Example sensor fields of view for cargo-type vehicles are illustrated according to various aspects of this disclosure.

[0023] Figure 7 Example systems based on various aspects of this technology are illustrated.

[0024] Figure 8 Examples of in-vehicle and non-vehicle training inputs according to various aspects of this technology are illustrated.

[0025] Figure 9 Examples of road humidity according to various aspects of this technology are illustrated.

[0026] Figure 10 Example deep learning models for road conditions based on various aspects of this technology are illustrated.

[0027] Figures 11A-11B Examples of driving modification scenarios based on this technology are provided.

[0028] Figures 12A-12B Example systems based on various aspects of this technology are illustrated.

[0029] Figure 13 Example procedures based on various aspects of this technology are illustrated. Detailed Implementation

[0030] As described above, various aspects of this technology utilize baseline ground truth information about road humidity, inputs from one or more sources (such as signals from other onboard sensors and / or signals from other onboard modules, as well as offboard signals) to develop a deep learning (DL) model for road humidity classification and to perform road humidity regression analysis. Certain data (e.g., onboard sensor and offboard signals) are used in the DL model, while other data (e.g., baseline ground truth information) may be used only for training. Therefore, the deployed system does not require a baseline ground truth sensor to be mounted on the vehicle. For example, training inputs are evaluated relative to baseline ground truth information for a given road segment. The output of the DL model can be used in various ways to enhance autonomous vehicle operation, such as by modifying current driving actions, modifying planned routes or trajectories, activating onboard cleaning systems, etc.

[0031] Example vehicle system

[0032] Figure 1A A perspective view of an example passenger vehicle 100, such as a minivan, SUV, or other vehicle, is shown. Figure 1B A top view of a passenger vehicle 100 is illustrated. The passenger vehicle 100 may include various sensors for obtaining information about the vehicle's external environment. For example, the roof-top housing 102 may include a lidar sensor as well as various cameras, radar units, infrared and / or acoustic sensors. Housing 104 located at the front of the vehicle 100 and housings 106a, 106b located on the driver's and passenger's sides of the vehicle may each incorporate lidar, radar, cameras, and / or other sensors. For example, housing 106a may be located along the quarter panel of the vehicle in front of the driver's side door. As shown, the passenger vehicle 100 also includes housings 108a, 108b, also located towards the rear roof portion of the vehicle, for radar units, lidar, and / or cameras. Additional lidar, radar units, and / or cameras (not shown) may be located elsewhere along the vehicle 100. For example, arrow 110 indicates a sensor unit (…). Figure 1B 112) can be positioned along the rear of vehicle 100, such as on or near the bumper. Arrow 114 indicates a series of sensor units 116 arranged along the forward-facing direction of the vehicle. In some examples, passenger vehicle 100 may also include various sensors (not shown) for obtaining information about the interior space of the vehicle.

[0033] Figures 1C-1DAn example cargo vehicle 150, such as a tractor-trailer truck, is illustrated. The truck may include, for example, a single trailer, a double trailer, or a triple trailer, or it may be another type of medium or heavy-duty truck, such as commercial weight class 4 to 8. As shown, the truck includes a tractor unit 152 and a single cargo unit or trailer 154. Depending on the type of cargo to be transported, the trailer 154 may be fully enclosed, open (such as a flatbed), or partially open. In this example, the tractor unit 152 includes an engine and steering system (not shown) and a cab 156 for the driver and any passengers. In a fully autonomous arrangement, the cab 156 may not be equipped with a seat or manual driving components, as a person may not be required.

[0034] Trailer 154 includes a bolting point referred to as a pivot bolt 158. The pivot bolt 158 ​​is typically formed as a solid steel axle configured to be pivotally attached to the towing unit 152. Specifically, the pivot bolt 158 ​​is attached to a trailer coupling 160, referred to as a half-trailer wheel, mounted behind the cab. For two or three tractor-trailers, the second and / or third trailers may have a simple bolted connection to the preceding trailer. Alternatively, each trailer may have its own pivot bolt. In this case, at least the first and second trailers may include a half-trailer wheel type structure arranged to couple to the next trailer.

[0035] As shown in the figure, the tractor unit may have one or more sensor units 162, 164 placed along it. For example, one or more sensor units 162 may be placed on the roof or top of the cab 156, and one or more side sensor units 164 may be placed on the left and / or right side of the cab 156. The sensor units may also be located in other areas along the cab 156, such as along the front bumper or hood area, at the rear of the cab, adjacent to the semi-trailer wheel, under the chassis, etc. The trailer 154 may also have one or more sensor units 166 placed along it, for example, along the side panels, front, rear, top, and / or underframe of the trailer 154.

[0036] As an example, each sensor unit may include one or more sensors, such as lidar, radar, cameras (e.g., optical or infrared), acoustic (e.g., microphone or sonar type sensors), inertial (e.g., accelerometers, gyroscopes, etc.), or other sensors (e.g., positioning sensors such as GPS sensors). While certain aspects of this disclosure may be particularly useful in combination with specific types of vehicles, vehicles can be any type of vehicle, including but not limited to cars, trucks, motorcycles, buses, recreational vehicles, etc.

[0037] Different degrees of autonomy can exist for vehicles operating in partial or full autonomous driving modes. The National Highway Traffic Safety Administration (NHTSA) and the Society of Automotive Engineers (SAE) have identified different levels to indicate the degree of vehicle control in driving. For example, Level 0 has no automation, and the driver makes all driving-related decisions. The lowest semi-autonomous mode, Level 1, includes some driver assistance, such as cruise control. Level 2 has some degree of automation in driving operations, while Level 3 involves conditional automation that allows the person in the driver's seat to take control when necessary. In contrast, Level 4 is a high level of automation, where the vehicle can drive without assistance under selected conditions. Level 5 is a fully autonomous mode, where the vehicle can drive without assistance in all situations. The architectures, components, systems, and methods described herein can operate in any semi-autonomous or fully autonomous mode, such as Levels 1 through 5, referred to herein as autonomous driving modes. Therefore, the reference to autonomous driving modes includes both partial and full autonomy.

[0038] Figure 2 A block diagram 200 illustrating various components and systems of an exemplary vehicle (such as passenger vehicle 100) operating in autonomous driving mode is shown. As illustrated, block diagram 200 includes one or more computing devices 202, such as computing devices comprising one or more processors 204, memory 206, and other components typically found in general-purpose computing devices. Memory 206 stores information accessible by one or more processors 204, including instructions 208 and data 210 that can be executed or otherwise used by the processors 204. When operating in autonomous driving mode, the computing system can control the overall operation of the vehicle.

[0039] Memory 206 stores information accessible by processor 204, including instructions 208 and data 210 that can be executed or otherwise used by processor 204. Memory 206 can be any type of memory capable of storing information accessible by a processor, including computing device readable media. Memory is a non-transitory medium, such as a hard disk drive, memory card, optical disk, solid-state drive, etc. The system may include different combinations of the foregoing, whereby different portions of instructions and data are stored on different types of media.

[0040] Instructions 208 can be any set of instructions that are executed directly by a processor (such as machine code) or indirectly (such as a script). For example, instructions can be stored as computing device code on a computing device readable medium. In this regard, the terms “instruction,” “module,” and “program” are used interchangeably herein. Instructions can be stored in an object code format for direct processing by a processor, or in any other computing device language that includes a collection of standalone source code modules that are interpreted on demand or pre-compiled, or in scripts. One or more processors 204 can retrieve, store, or modify data 210 according to instructions 208. In one example, some or all of the memory 206 can be an event data logger or other secure data storage system configured to store sensor data for vehicle diagnostics and / or detection, which, depending on the implementation, can be on-board or remote.

[0041] Processor 204 can be any conventional processor, such as a commercial CPU. Alternatively, each processor can be a dedicated device, such as an ASIC or other hardware-based processor. Although Figure 2 While the processor, memory, and other elements of computing device 202 are functionally illustrated within the same frame, such a device may actually include multiple processors, computing devices, or memories that may or may not be housed in the same physical enclosure. Similarly, memory 206 may be a hard disk drive or other storage medium located in a different enclosure than processor 204. Therefore, references to processors or computing devices will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0042] In one example, computing device 202 may form an autonomous driving computing system integrated into vehicle 100. The autonomous driving computing system may be able to communicate with various components of the vehicle. For example, computing device 202 may communicate with various systems of the vehicle, including a driving system comprising a deceleration system 212 (for controlling the vehicle's braking), an acceleration system 214 (for controlling the vehicle's acceleration), a steering system 216 (for controlling wheel orientation and vehicle direction), a signaling system 218 (for controlling turning signals), a navigation system 220 (for navigating the vehicle to a location or object), and a positioning system 222 (for determining the vehicle's location, e.g., including the vehicle's pose). Depending on the navigation system 220, the positioning system 222, and / or other components of the system, the autonomous driving computing system may employ a planner module 223, for example, for determining a route from a starting point to a destination, or for modifying various aspects of driving based on current or anticipated traction conditions.

[0043] The computing device 202 is also operatively coupled to the sensing system 224 (for detecting objects and conditions in the vehicle environment), the power system 226 (e.g., a battery and / or a gasoline or diesel-powered engine), and the transmission system 230 to control the vehicle's movement, speed, etc., according to instructions 208 from the memory 206 in an autonomous driving mode where continuous or periodic input from the vehicle's passengers is not required or necessary. Some or all of the wheels / tires 228 are coupled to the transmission system 230, and the computing device 202 may be able to receive information about tire pressure, balance, and other factors that can affect driving in autonomous mode.

[0044] The computing device 202 can control the direction and speed of the vehicle by controlling various components, such as via the planner module 223. As an example, the computing device 202 can autonomously navigate the vehicle to its destination using map information and data from the navigation system 220. The computing device 202 can use the positioning system 222 to determine the vehicle's position and the perception system 224 to detect objects and respond to them as needed to safely reach the location. To this end, the computing device 202 can accelerate the vehicle (e.g., by increasing the fuel or other energy supplied to the engine by the acceleration system 214), decelerate (e.g., by reducing the fuel supplied to the engine, shifting gears, and / or applying braking via the deceleration system 212), change direction (e.g., by turning the front wheels or other wheels of the vehicle 100 left or right via the steering system 216), and signal such changes (e.g., by illuminating the turn signals of the signal system 218). Therefore, the acceleration system 214 and the deceleration system 212 can be part of a drive system or other type of drive system 230 that includes various components between the vehicle's engine and wheels. Similarly, by controlling these systems, computing device 202 can also control the vehicle's transmission system 230 to autonomously maneuver the vehicle.

[0045] The computing device 202 can use the navigation system 220 to determine and follow a route to a location. In this regard, the navigation system 220 and / or memory 206 can store map information, such as highly detailed maps that the computing device 202 can use to navigate or control the vehicle. As examples, these maps can identify roads (e.g., including depressions, angles, etc.), lane markers, intersections, pedestrian crossings, speed limits, traffic lights, buildings, signs, real-time traffic information, the shape and height of vegetation, or other such objects and information. Lane markers can include features such as solid or dashed double-lane lines or single-lane lines, solid or dashed lane lines, reflectors, etc. A given lane can be associated with left and / or right lane lines or other lane markers that define the lane boundaries. Therefore, most lanes can be defined by the left edge of one lane line and the right edge of another lane line.

[0046] The perception system 224 includes sensors 232 for detecting objects or environmental factors outside the vehicle. Detected objects may include other vehicles, obstacles in the road, traffic signals, signs, trees, etc. Sensors 232 may also detect aspects of weather conditions, such as snow, rain, fog, puddles, ice, or other materials on the road. The selected vehicle may include enhanced sensors to provide water measurements for road sections. As an example only, a road weather information sensor from Lufft could be used.

[0047] By way of example only, the sensing system 224 may include one or more light detection and ranging (lidar) sensors and / or LED emitters, radar units, cameras (e.g., optical imaging devices with or without neutral density (ND) filters), positioning sensors (e.g., gyroscopes, accelerometers and / or other inertial components), infrared sensors, acoustic sensors (e.g., microphones or sonar transducers) and / or any other detection devices that can record data that can be processed by the computing device 202.

[0048] Such sensors in perception system 224 can detect objects outside the vehicle and their characteristics, such as position, orientation, size, shape, type (e.g., vehicle, pedestrian, cyclist, etc.), direction, and speed of movement relative to the vehicle. Environmental conditions (e.g., temperature and humidity), such as surface temperature, dew point and / or relative humidity, water film thickness, and precipitation type, as well as road conditions, can also be detected by one or more of these sensors.

[0049] The sensing system 224 may also include other sensors within the vehicle to detect objects and conditions within the vehicle, such as passengers in the passenger compartment. For example, such sensors can detect one or more people, pets, packages, etc., as well as conditions inside and / or outside the vehicle, such as temperature, humidity, etc. Another sensor 232 of the sensing system 224 can measure the rotational speed of the wheels 228, the amount or type of braking of the deceleration system 212, and other factors associated with the vehicle's own equipment.

[0050] Raw data from sensors, including road condition sensors, along with the aforementioned characteristics, can be processed by perception system 224 and / or periodically or continuously transmitted to computing device 202 for further processing as data is generated by perception system 224. Computing device 202 can use positioning system 222 to determine the vehicle's location and use perception system 224 to detect objects and road conditions and respond to them as needed to safely reach the location, for example, through adjustments made via planner module 223. Furthermore, computing device 202 can perform calibrations between individual sensors, all sensors in a specific sensor assembly, or sensors in different sensor assemblies or other physical housings.

[0051] like Figure 1A-Figure 1B As illustrated, certain sensors of the sensing system 224 may be incorporated into one or more sensor assemblies or housings. In one example, these may be integrated into the side mirrors of a vehicle. In another example, other sensors may be part of the top housing 102 or other sensor housings or units 104, 106a, 106b, 108a, 108b, 112, and / or 116. The computing device 202 may communicate with sensor assemblies located on or otherwise distributed along the vehicle. Each assembly may have one or more types of sensors, such as those described above.

[0052] Back Figure 2 The computing device 202 may include all components typically used in conjunction with a computing device, such as the processor and memory described above, and the user interface subsystem 234. The user interface subsystem 234 may include one or more user inputs 236 (e.g., a mouse, keyboard, touchscreen, and / or microphone) and one or more display devices 238 (e.g., a monitor with a screen or any other electronic device operable to display information). In this regard, an internal electronic display may be located within the vehicle's cabin (not shown) and may be used by the computing device 202 to provide information to passengers within the vehicle. Other output devices, such as speakers 240, may also be located within the passenger vehicle.

[0053] The passenger vehicle may also include a communication system 242. For example, the communication system 242 may also include one or more wireless configurations to facilitate communication with other computing devices, such as passenger computing devices within the vehicle, computing devices outside the vehicle (such as those in another nearby vehicle on the road), and / or remote server systems. Network connectivity may include short-range communication protocols such as Bluetooth. TM ,Bluetooth TMLow power (LE), cellular connectivity, and a variety of configurations and protocols, including the Internet, World Wide Web, intranet, virtual private network, wide area network, local area network, private network using communication protocols proprietary to one or more companies, Ethernet, WiFi, and HTTP, as well as various combinations thereof.

[0054] Figure 3A Examples are given for vehicles (e.g., Figure 1C A block diagram 300 shows various components and systems of a vehicle (150). As an example, the vehicle may be a truck, bus, farm equipment, construction equipment, emergency vehicle, etc., configured to operate in one or more autonomous operating modes. As shown in block diagram 300, the vehicle includes a control system for one or more computing devices, such as computing device 302, which includes one or more processors 304, memory 306, and related components. Figure 2 Other components similar to or equivalent to the components 202, 204, and 206 discussed. The control system may constitute the electronic control unit (ECU) of the tractor unit of the cargo vehicle. Like instruction 208, instruction 308 may be any set of instructions that are executed directly by a processor (such as machine code) or indirectly (such as a script). Similarly, one or more processors 304 may retrieve, store, or modify data 310 according to instruction 308.

[0055] In one example, computing device 302 can form an autonomous driving computing system integrated into vehicle 150. Similar to the above regarding... Figure 2 The arrangement discussed, the autonomous driving computing system of block diagram 300, may be able to communicate with various components of the vehicle to perform route planning and driving operations. For example, computing device 302 can communicate with various systems of the vehicle, such as the driving system, which includes deceleration system 312, acceleration system 314, steering system 316, signal system 318, navigation system 320, and positioning system 322, each of which can be used as described above regarding... Figure 2 As discussed.

[0056] The computing device 302 is also operatively coupled to the sensing system 324, the powertrain 326, and the drivetrain 330. Some or all of the wheels / tires 328 are coupled to the drivetrain 330, and the computing device 302 is capable of receiving information about tire pressure, balance, speed, and other factors that may affect driving in autonomous mode. Like the computing device 202, the computing device 302 can control the vehicle's direction and speed by controlling various components. For example, the computing device 302 can autonomously navigate the vehicle to its destination using data from map information and the navigation system 320. Similar to the above... Figure 2In this manner, computing device 302 may employ planner module 323, in conjunction with positioning system 322, sensing system 324, and other subsystems, to detect objects and respond to them as needed, in order to safely reach the location.

[0057] Similar to sensing system 224, sensing system 324 also includes one or more sensors or other components, such as those described above for detecting objects and environmental conditions outside the vehicle (including road conditions), objects or conditions inside the vehicle, and / or the operation of certain vehicle equipment (such as wheels and deceleration system 312). For example, as Figure 3A As indicated, the sensing system 324 includes one or more sensor components 332. Each sensor component 332 includes one or more sensors. In one example, the sensor component 332 may be arranged as a sensor tower integrated into the side mirror of a truck, farm equipment, construction equipment, etc. (As mentioned above regarding...) Figures 1C-1D The sensor assembly 332 can also be placed at different locations on the tractor unit 152 or the trailer 154. The computing device 302 can communicate with the sensor assemblies located on both the tractor unit 152 and the trailer 154. Each assembly can have one or more types of sensors, such as those described above.

[0058] Figure 3A The diagram also shows a coupling system 334 for connecting the tractor unit and the trailer. The coupling system 334 may include one or more power and / or pneumatic connections (not shown), and a semi-trailer wheel 336 at the tractor unit for connecting to the trailer via a central bolt. A communication system 338 (equivalent to communication system 242) is also shown as part of the vehicle system 300.

[0059] Figure 3B Examples include trailers (such as...) Figures 1C-1D Example block diagram 340 of a system (trailer 154). As shown, the system includes an ECU 342 of one or more computing devices, such as a computing device containing one or more processors 344, a memory 346, and other components typically found in general-purpose computing devices. The memory 346 stores information accessible by one or more processors 344, including instructions 348 and data 350 that can be executed or otherwise used by the processors 344. Figure 2 and Figure 3A The descriptions of processors, memory, instructions, and data in the document are applicable to Figure 3B These elements.

[0060] ECU 342 is configured to receive information and control signals from the trailer unit. The onboard processor 344 of ECU 342 can communicate with various systems of the trailer, including a reduction system 352, a signal system 354, and a positioning system 356. ECU 342 can also be operatively coupled to a sensing system 358 having one or more sensors for detecting objects and / or conditions in the trailer environment, and can be operatively coupled to a power system 360 (e.g., battery power) to provide power to local components. Some or all of the trailer's wheels / tires 362 can be coupled to the reduction system 352, and the processor 344 may be able to receive information about tire pressure, balance, wheel speed, and other factors that may affect driving in autonomous mode, and relay this information to the processing system of the towing unit. The reduction system 352, signal system 354, positioning system 356, sensing system 358, power system 360, and wheels / tires 362 can communicate with each other as described above. Figure 2 and Figure 3A Operate in the following manner.

[0061] The trailer also includes a set of landing gear 366 and a coupling system 368. The landing gear provides a support structure for the trailer when it is decoupled from the tractor unit. The coupling system 368, which provides the connection between the trailer and the tractor unit, may be part of a coupling system 334. Therefore, the coupling system 368 may include a connection portion 370 (e.g., for a power and / or pneumatic link). The coupling system also includes a pivot bolt 372 configured for connection to the semi-trailer wheels of the tractor unit.

[0062] Example Implementation

[0063] In view of the structures and configurations described above and illustrated in the accompanying drawings, various aspects will now be described in accordance with the present technology.

[0064] While models of road surfaces and other conditions can be trained on human-labeled data, such methods are subjective and potentially error-prone. Therefore, selected sensor data is used as the baseline ground truth for the model. Various model architectures can be employed, such as those using Neural Architecture Search (NAS). Different model architectures can be used depending on the data type, such as vehicle-mounted LiDAR data and road map information. Therefore, any deep learning model capable of classifying / regressing road moisture using vehicle-mounted sensor signals and other available prior information (such as road map data) can be adopted.

[0065] Various sensors can be located in different places around the vehicle (see...) Figures 1A-1DSensors are used to collect data from different parts of the external environment. Certain sensors may have different fields of view depending on their placement around the vehicle and the type of information they are designed to collect. For example, different sensors may be used for near (short-range) detection of objects or situations adjacent to the vehicle (e.g., less than 2-10 meters), while others may be used for far (long-range) detection of objects 100 meters (or more or less) in front of the vehicle. Medium-range sensors may also be employed. Multiple sensor units, such as lidar and radar, can be positioned towards the front or rear of the vehicle for long-range object detection. Cameras and other image sensors can also be arranged to provide good visibility around the vehicle. Depending on the configuration, a certain type of sensor may include multiple individual sensors with overlapping fields of view. Alternatively, other sensors can provide a redundant 360° field of view.

[0066] Figure 4 Scenario 400 is illustrated, in which a vehicle uses one or more sensors to detect the presence of water along a road in order to obtain baseline truth data. For example, the baseline truth input may include measurements of water thickness, such as the thickness of a water film on the road surface and / or ice coverage. This can be done at a very granular level, for example, measuring thickness at the micrometer level. In this scenario, the vehicle, including various sensors located at different locations along its exterior, can be configured to operate in an autonomous driving mode (or manual mode). This may include front and / or rear sensor units 402, and a roof-based sensor unit 404, each sensor unit including lidar, radar, optical cameras, acoustic sensors, and / or other sensors. These or other sensor units can be used to collect signals from the environment surrounding the autonomous vehicle.

[0067] As an example, the baseline true value can be collected using sensors designed for water thickness (e.g., water film thickness measurement and / or ice cover) (e.g., front and / or rear sensors 402). This could include, for example, road weather information sensors from Lufft. For instance, the front sensor could collect the data via dashed line 406. F Data is obtained from the scan shown, and then the sensor can obtain it via the dashed line 406. R Data is obtained from the scan shown. Roof-based sensor components can acquire information about objects or conditions around the vehicle, as indicated by dashed line 408. Note that sensors used to collect baseline ground truth data may only be mounted in selected vehicles used to train deep learning models during the development phase. Once such models are deployed on-vehicle in autonomous vehicles, these sensors for measuring road humidity do not need to be mounted on the vehicle.

[0068] The placement of a ground truth collection sensor around a vehicle can vary depending on the vehicle type (e.g., car, truck, motorcycle, etc.) and other factors, provided the sensor has a direct line of sight to the relevant part of the road. Spray from tires or other vehicles can potentially have some impact; therefore, to mitigate this impact, the ground truth sensor should be covered by a protective housing. Furthermore, water droplets crossing the sensor's sensing track can affect optical sensing and thus measurements. However, by avoiding mounting the sensor directly above the tire track, the likelihood of water spray crossing the sensing track is minimal.

[0069] In addition to sensors used for benchmarking the true value Figure 5 Provided with Figure 1B An example 500 of the sensor field of view associated with the illustrated sensor. Here, if the top housing 102 includes a lidar sensor as well as various cameras, radar units, infrared and / or acoustic sensors, each of these sensors can have a different field of view. Thus, as shown, the lidar sensor can provide a 360° FOV 502, while the camera arranged within housing 102 can have a separate FOV 504. The sensor within housing 104 at the front of the vehicle has a forward-facing FOV 506, while the sensor within housing 112 at the rear has a rearward-facing FOV 508. The driver-side and passenger-side housings 106a, 106b of the vehicle can each incorporate lidar, radar, cameras and / or other sensors. For example, the lidar within housings 106a and 106b can have a corresponding FOV 510a or 510b, while the radar units or other sensors within housings 106a and 106b can have a corresponding FOV 511a or 511b. Similarly, each sensor located within housings 108a and 108b facing the rear roof of the vehicle has a corresponding field of view (FOV). For example, a lidar unit within housings 108a and 108b may have a corresponding FOV 512a or 512b, while a radar unit or other sensor within housings 108a and 108b may have a corresponding FOV 513a or 513b. Furthermore, a series of sensor units 116 arranged along the forward-facing direction of the vehicle may have corresponding FOVs 514, 516, and 518. Each of these fields of view is merely exemplary and is not proportional in terms of coverage.

[0070] exist Figure 6A and Figure 6B The diagram shows vehicles used for cargo types (e.g., Figures 1C-1D Examples of lidar, cameras, and radar sensors and their fields of view for vehicles (150). Figure 6AIn Example 600, one or more lidar units may be located in a top sensor housing 602, while other lidar units are located in side sensor housings 604. Specifically, the top sensor housing 602 may be configured to provide a 360° field of view (FOV). A pair of sensor housings 604 may be located on either side of the tractor unit cab, for example, integrated into side mirror assemblies or along the side doors or side panels of the cab. In one scenario, a long-range lidar may be located in the top or upper area along the sensor housings 602 and 604. The long-range lidar may be configured to see above the vehicle's hood. Short-range lidar may be located in other portions of the sensor housings 602 and 604. The perception system can use the short-range lidar to determine whether objects such as another vehicle, pedestrian, cyclist, etc., are in front of or to the side of the vehicle, and to consider this information when determining how to drive or turn. Both types of lidar may coexist in the housing, for example, aligned along a common vertical axis.

[0071] like Figure 6A As illustrated, the lidar in the top sensor housing 602 may have an FOV 606. Here, as shown in area 608, other articulated parts of the trailer or vehicle can provide signal return and may partially or completely obstruct the rear view of the external environment. Long-range lidars located on the left and right sides of the tractor unit have an FOV 610. These may cover important areas along the sides and front of the vehicle. As shown, there may be an overlapping area 612 of their fields of view in front of the vehicle. The overlapping area 612 provides the perception system with additional information about a very important area directly in front of the tractor unit. This redundancy also has safety aspects. If one of the long-range lidar sensors suffers performance degradation, the redundancy will still allow operation in autonomous mode. Short-range lidars on the left and right sides have a smaller FOV 614. For clarity in the figures, space is shown between the different fields of view; however, in reality, there may be no interruption in coverage. The specific placement of the sensor components and fields of view is merely exemplary and can vary depending on, for example, the type of vehicle, the size of the vehicle, FOV requirements, etc.

[0072] Figure 6B Examples include top-shell neutral tractor-trailers (such as...) Figures 1C-1D Example configuration 620 of any one (or both) of the radar and camera sensors on both sides of the vehicle 150. Here, in Figure 6AEach of the sensor housings 602 and 604 may contain multiple radar and / or camera sensors. As shown, sensors may be housed in the top housing having a front FOV 622, side FOVs 624, and rear FOV 626. Similar to area 608, a trailer may affect the sensor's ability to detect objects behind the vehicle. The sensor in sensor housing 604 may have a forward-facing FOV 628 (and side and / or rear fields of view). As mentioned above regarding... Figure 6A Similar to the lidar discussed, Figure 6B The sensors can be arranged such that adjacent fields of view overlap, as shown in overlapping region 630. This overlapping region similarly provides redundancy and has the same benefit if a sensor suffers performance degradation.

[0073] Example Scenario

[0074] like Figure 7 As shown in Example 700, the processing system 702 can receive various inputs from vehicles and other sources. For example, onboard signals received from passenger vehicle 704a or truck 704b may include LiDAR returns, camera images / onboard video, radar returns, audio signals, and reference ground truth values ​​output via a road humidity sensor (e.g., from a sensor configured to detect road weather information including water film height, ice percentage, etc. via spectroscopy or other techniques). Furthermore, outputs from other perception modules / models of the vehicle (e.g., puddle detectors and filtering modules) may also be part of the onboard signals.

[0075] Non-vehicle signals provided by external sources 706 (e.g., 706a and 706b) may include, for example, weather station information, public weather forecasts, road map data, human-marked examples of road humidity baselines, crowdsourced information, and observations from other vehicles in the vicinity (e.g., as part of a convoy) to provide additional context regarding road humidity.

[0076] Figure 8 Example 800 illustrates such on-board factors 802 and off-board factors 804, such as Figure 7 As shown, this can be collected via network 708 and stored as training input 710. Here, for example, weather station information and public weather forecasts can come from third-party sources or external systems 706a. Road map data, human-labeled humidity baseline examples, observations from other vehicles, etc., can come from system 706b.

[0077] As shown in the figure, the processing system 702 includes one or more processors 712, a memory 714 with instruction 716 and data 718, and optional user input 720 and display 722. Each of these can be equivalent to the above regarding... Figure 2and Figures 3A-3B The computing devices and processing systems described herein are configured and operated in a manner consistent with those described. Data 718 may include one or more models 724, such as the DL models described herein.

[0078] Some or all of these signals can be fused together. For example, machine learning can handle fusion from different sources. Machine learning takes input from multiple sensors and builds a model to output a final result. In this modeling process, all inputs (or a selected subset of inputs) are fused together. This can be done by creating special embedding layers in the model that combine inputs in a human-engineered manner, or by directly building an end-to-end architecture that puts all inputs directly into the model. The embedding layers can be human-engineered, or the embeddings can be learned. Sensor data and embeddings can be combined anywhere in the model, initially as raw data, later as embeddings, or in between.

[0079] Different signals can be assigned different weights. For example, one can construct human-designed features from raw sensor inputs, where prior knowledge about which sensor should be emphasized can be encoded into the feature construction. As an example, a system can aggregate LiDAR data in a region into a single value to use as input in a model, but assign different weights to points in different locations within the region when constructing that value. Another approach is to leverage the learning capabilities of deep networks and incorporate the weights of different inputs into the model parameters. The weights of different inputs can then be learned during model training.

[0080] In one scenario, the model learns embeddings and the weights of each embedding. There are two types of weights. First, in the input, different inputs may be weighted differently. This can be done using human-designed embedding layers (e.g., chosen by a systems engineer), or simply computed by the model itself as it trains and converges to different weights for different input channels. For all examples, these weights can typically be the same.

[0081] The second approach is to quantify the importance of different examples to the evaluation of model quality by assigning weights to different examples. For example, in a classification model, examples with a baseline ground truth water film height that is very close to the dry / wet threshold are assigned smaller weights because the dry / wet binarization of such examples is more likely to be ambiguous and / or the baseline ground truth from such examples may be corrupted by measurement noise.

[0082] Roads may exhibit a continuous transition from wet to dry conditions. The system can attempt to identify wet areas, dry areas, and potentially ambiguous regions in between. For example, Figure 9Example 900 illustrates that the rightmost lane portion 902 is wet. This could be due to, for example, puddles or standing water 1.0–4.0 mm deep (or deeper). Dry area 904 may have no standing water (e.g., less than 0.03 mm). And between the wet and dry areas, there may be an area 906 that may have some standing water (e.g., a water film between 0.02–2.0 mm), in which it may be ambiguous whether this should be classified as “wet” or “dry.” In one scenario, as indicated above for the second weighting, information about ambiguous areas can be given lower weight.

[0083] Road moisture values ​​indicate the probability of a road section being wet or to what extent. For classification models, the output is not simply a list of categories (e.g., "wet" or "dry"), but rather the probability that a road area falls into a given category. This probability indicates the system's confidence in the classification result and whether there are any alternative potential categories with lower probabilities. Therefore, for... Figure 9 For example, region 906 may have a high probability (e.g., 60-90%) of being “wet” and a low probability (e.g., 10-30%) of being “dry”.

[0084] Model outputs can have different granularities. For example, a classification model could have only two classes, such as dry / wet, or more classes based on water film height, such as one class for increments of a certain water film height (e.g., every 0.25, 0.5, or 1.0 mm). There could even be regression models that provide continuous estimates of the water film height on the road. When making (autonomous) driving decisions, the granularity can be determined based on needs and requirements. For example, granularity can be useful when deciding whether to drive through or avoid a specific (wet) section of the road.

[0085] While wet and dry are two outputs of the model, as examples only, additional granularities may include “slightly wet” (e.g., damp), where a certain amount of moisture on the road surface is below the “wet” threshold; “ice”, where the water is essentially in the form of ice (e.g., the percentage of ice crystals in the sample exceeds the threshold); “snow”, where water covers selected portions of the road in the form of small white ice crystals; “chemically wet”, where water molecules have not yet turned into ice, for example, due to de-icing chemicals on the road; and / or “other”, where, for example, a particular property of the road condition does not fall into any other category.

[0086] Before building a deep learning (DL) model, statistical analysis can be employed, for example, to identify which onboard signals most effectively correlate with the baseline truth, and to eliminate or weaken any onboard signals or external parameters that do not correlate well with the baseline truth. Relevant statistical parameters include the mean and standard deviation of sensor data. Sensor signal returns can be bucketed based on different conditions (e.g., distance from the autonomous vehicle). This helps determine the useful range and eliminate irrelevant conditions or conditions that affect statistical data. For example, road material, road wear, or road surface type (e.g., ditch) may be irrelevant and exclude returns from outside the road. Temperature, lighting conditions, and other environmental factors may or may not be relevant.

[0087] Another factor can be the placement / positioning of onboard sensors to identify those that provide the most useful information. During the testing phase, sensors can be placed at different locations along the vehicle to see which one gives a stronger signal (e.g., a signal more closely correlated with the measured true value of road humidity).

[0088] The result of this analysis is a very useful subset of data that masks the return of dynamic objects on the road to avoid noise introduced by vehicles, pedestrians, cyclists, and other road users. For example, lidar sensor information can include intensity and reflectivity, and statistical evaluation can show that intensity is more relevant than reflectivity. Therefore, strong signal inputs may be range- and height-limited lidar data. As an example only, the range of lidar points producing the greatest difference between wet and dry road surfaces can be on the order of 30-50 meters from vehicles, and the threshold for separating wet and dry road surfaces based on measured water film thickness can be on the order of 5-20 micrometers. Furthermore, light reflection on water affects the returned intensity, as water alters how much light is reflected back to and from the sensor. This is the primary signal. Height provides geometric information and helps determine which point originates from the road. Elongation and secondary reflection provide additional information about the reflecting surface. Another useful input is road map data from a map, which provides information about points on or off the road.

[0089] The probability of road moisture level is the output of the classification model. A threshold for the probability is given to obtain the dry / wet classification. One example is using a wet probability of 0.5 as the threshold. During model training, this classification is compared to the ground truth as an evaluation of the current model's quality, and the model parameters are adjusted accordingly.

[0090] The model architecture is a deep network, where the exact structure and parameters can be searched using automated machine learning. This can be done through automated machine learning methods such as Neural Networks (NAS), which serve as techniques for automatically designing artificial neural networks rather than human-designed architectures. According to one aspect of this technique, automated machine learning is used to optimize model design. Examples of automated model selection include variants of NAS (such as TuNAS), automated hyperparameter optimization (such as Vizier), and automated data augmentation. In this way, the document can be better understood by a general machine learning audience. An example processing approach would be given a set of basic model architecture elements (such as some representative layers) and use reinforcement learning to search for the optimal combination among these elements.

[0091] Model accuracy can be improved in several ways. These include smoothing measurements from road humidity sensors to obtain a more robust estimate of the baseline, balancing wet and dry examples in the training dataset to avoid a skewed model, and designing a loss function that gives more emphasis to examples with higher confidence levels for either wet or dry conditions. The system can also use a low-pass filter to remove high-frequency noise.

[0092] In all training examples, the loss function can be a weighted sum of the squares of the differences between the ground truth and the model output. In the weighted sum, higher weights are assigned to examples with higher confidence, while examples with lower confidence receive lower weights.

[0093] Figure 10 Example 1000 illustrates a deep learning model architecture for road conditions based on various aspects of this technology. The architecture can be implemented via... Figure 7 This is achieved through a processing system. As shown in box 1002, the signal 1002 comes from the vehicle-mounted sensor. A Non-vehicle signal 1002 B Both are inputs to the system (e.g., Figure 7 The training inputs (710) can be any or all of the types described above, and they are fed into the feature extraction layer 1004. The feature extraction layer takes an initial set of input data and constructs derived features. These features can be dimensionality-reduced, informative, and non-redundant, facilitating subsequent learning and leading to better human interpretation. The extracted features are applied to the pooling layer 1006. The pooling layer can reduce the dimensionality of the data representation and the number of parameters that need to be learned in the model, making the model structure smaller and the learning speed faster.

[0094] The pooling information output from pooling layer 1006 is fed into module 1008, which includes convolutional layer 1010 and activation layer 1012. Convolutional layer 1010 transforms the input image into images of potentially different sizes and parameters, thereby extracting features that may be hidden in the input image. Activation layer 1012 provides non-linearity to the model through different activation functions. As indicated by dashed line 1013, the processing within module 1008 can be repeated multiple times. Repeating such layers increases the depth of the deep learning model and allows us to learn more complex model structures. The exact number of repetitions (e.g., 2, 3, or more times) can be manually designed or searched using NAS.

[0095] Next, the data output from module 1008 is fed into the fully connected layer 1014. The fully connected layer integrates the outputs from the previous layer into a vector of the desired size. This captures the complex relationships within high-level features. Output 1016 is, for example, a classification or continuous estimate of road moisture. Thus, the various layers form a road moisture model, and the model provides output 1016, such as a classification or estimate. Although in Figure 10 Example 1000 shows individual layers 1004, 1006, 1010, 1012, and 1014, but there can be one or more such layers for each of feature extraction, pooling, convolution, activation, and fully connected layers. One or more of these layers may also be absent from the model. For example, in some scenarios, pooling, convolution, and / or activation layers can be omitted.

[0096] The end result of this modeling approach is the ability to provide discrete classifications or continuous regressions / estimates of road moisture, which has numerous useful applications. These include triggering safety precautions (e.g., pulling over to the side of the road for roads that are too wet to handle); causing changes in real-time motion control (e.g., adjusting acceleration / deceleration, braking distance, lane changes, etc.); altering perception systems (e.g., modifying filter thresholds, sensor noise levels, sensor field of view adaptation, sensor validation logic, pedestrian detectors, etc.); influencing how wiper systems (or any sensor cleaning systems) operate; changing models that predict the behavior of other road users (e.g., other vehicles may drive slower, pedestrians or cyclists may move irregularly to avoid puddles, etc.); and altering planner behavior (e.g., where to pick up or drop off, choosing alternative routes or lanes, etc.). Such information can be provided to vehicles across a convoy, such as as part of a general system update or based on current or predicted weather conditions, to assist in convoy scheduling and routing.

[0097] For example, Figure 11AA first scenario 1100 is illustrated, in which a truck 1102 drives over a wet area 1104 of the road. As shown, this results in water splashing 1106 from the truck's tires. In this scenario, a car 1108 can determine that water splashing will occur based on road conditions (e.g., the depth of the water film on the road). Therefore, in response to this determination, and considering other objects along the road such as vehicle 1112, vehicle 1108 can adjust its driving path as shown by dashed line 1110.

[0098] Figure 11B A second scenario 1120 is illustrated, in which vehicle 1122 observes bicycle 1124 approaching a wet area (e.g., a puddle) 1126. Here, based on information from a road wetness model and other factors (such as the observed object being a bicycle), vehicle 1122 can predict that the bicycle will alter its trajectory to avoid the wet area, as shown by dashed line 1128. As a result, vehicle 1122 can brake or stop accelerating to allow bicycle 1124 sufficient space to move around the wet area.

[0099] As described above, this technology is applicable to various types of wheeled vehicles, including passenger cars, buses, motorcycles, RVs, emergency vehicles, and trucks or other freight vehicles.

[0100] In addition to using road condition model information to operate vehicles, this information can also be shared with other vehicles, such as those that are part of a convoy. This facilitates route planning, the collection of additional baseline ground truth data, model updates, and more.

[0101] An example of data sharing is in Figure 12A and Figure 12B shown in . Specifically, Figure 12A and Figure 12B These are schematic diagrams and functional diagrams of example system 1200, which includes multiple computing devices 1202, 1204, 12012, and 1208 and a storage system 1210 connected via network 1216. System 1200 also includes example vehicle 1212 and / or vehicle 1214, which can be respectively connected to... Figure 1A-Figure 1B and Figures 1C-1D Vehicles 100, 150, and / or 170 are configured identically or similarly. Vehicles 1212 and / or 1214 may be part of a convoy. Although only a small number of vehicles and computing devices are depicted for simplicity, a typical system may include significantly more vehicles and computing devices.

[0102] like Figure 12B As shown, each of computing devices 1202, 1204, 1206, and 1208 may include one or more processors, memory, data, and instructions. Such processors, memory, data, and instructions can be configured with the aforementioned... Figure 2or Figures 3A-3B The same configurations as those in [the original text].

[0103] Various computing devices and vehicles can communicate via one or more networks (such as network 1216). Network 1216 and intermediate nodes can include various configurations and protocols, including short-range communication protocols such as Bluetooth. TM Bluetooth LE TM The Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local area networks, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi, and HTTP, as well as various combinations thereof. Such communication can be facilitated by any device capable of transmitting and receiving data from other computing devices (such as modems and wireless interfaces).

[0104] In one example, computing device 1202 may include one or more server computing devices (e.g., a load-balancing server cluster or cloud computing system) having multiple computing devices that exchange information with different nodes in the network for the purpose of receiving, processing, and transmitting data to other computing devices. For example, computing device 1202 may include one or more server computing devices capable of communicating with vehicles 1212 and / or 1214, as well as computing devices 1204, 1206, and 1208, via network 1216. For example, vehicles 1212 and / or 1214 may be part of one or more fleets of vehicles that can be dispatched to various locations by the server computing devices. In this respect, computing device 1202 can serve as a dispatch server computing system, which can be used to dispatch vehicles to different locations for picking up and dropping off passengers or picking up and delivering goods. Furthermore, server computing device 1202 can use network 1216 to transmit and present information to users or vehicle passengers of one of the other computing devices. In this respect, computing devices 1204, 1206, and 1208 can be considered client computing devices.

[0105] like Figure 12A As shown, each client computing device 1204, 1206, and 1208 may be a personal computing device intended for use by the respective user 1218 and has all the components typically used in conjunction with a personal computing device, including one or more processors (e.g., a central processing unit (CPU)), memory for storing data and instructions (e.g., RAM and internal hard disk drives), a display (e.g., a monitor with a screen, touchscreen, projector, television, or other device operable to display information, such as a smartwatch display), and user input devices (e.g., a mouse, keyboard, touchscreen, or microphone). The client computing device may also include a camera for recording video streams, speakers, network interface devices, and all components for connecting these elements to each other.

[0106] While each client computing device may include a full-size personal computing device, they may alternatively include mobile computing devices capable of wirelessly exchanging data with a server over a network such as the Internet. By way of example only, client computing devices 1206 and 1208 may be mobile phones or devices such as wireless-enabled PDAs, tablet PCs, wearable computing devices (e.g., smartwatches), or netbooks capable of accessing information via the Internet or other networks.

[0107] In some examples, the client computing device 1204 may be a remote assistance workstation used by an administrator or operator to communicate with passengers of the dispatched vehicle. Although in Figures 12A-12B Only a single remote assistance workstation 1204 is shown, but any number of such workstations may be included in a given system. Furthermore, although the workstation is described as a desktop computer, it may include various types of personal computing devices, such as laptops, netbooks, tablets, etc.

[0108] Storage system 1210 can be any type of computerized storage capable of storing information accessible by server computing device 1202, such as hard disk drives, memory cards, ROM, RAM, DVDs, CD-ROMs, flash drives, and / or tape drives. Furthermore, storage system 1210 can include a distributed storage system, where data is stored on multiple different storage devices that may be physically located in the same or different geographical locations. Figures 12A-12B As shown, it is connected to a computing device via network 1216, and / or can be directly connected to or integrated into any computing device.

[0109] Storage system 1210 can store various types of information. For example, storage system 1210 can also store autonomous vehicle control software and / or road condition models, which can be used by vehicles such as vehicles 1212 or 1214 to operate such vehicles in autonomous driving mode. Storage system 1210 can store map information, route information, weather condition information, road surface information, vehicle models of vehicles 1212 and 1214, etc. This information can be shared with vehicles 1212 and 1214, for example, to assist in real-time route planning and driving analysis through onboard computer systems.

[0110] The remote assistance workstation 1204 can access stored information and use it to assist the operation of individual vehicles or a convoy. For example, the lead vehicle can detect wet conditions, such as still water, ice, or snow along a road segment, and send information about the wet conditions to the remote assistance workstation 1204. In turn, the remote assistance workstation 1204 can disseminate the information to other vehicles in the convoy so they can adjust their routes.

[0111] When passengers are present, the vehicle or remote assistance can communicate directly or indirectly with the passengers' client computing devices. For example, information can be provided to passengers regarding current driving operations, route changes in response to circumstances, etc.

[0112] Figure 13 Example process 1300 is illustrated, which is a method for generating a deep learning model of road conditions. The method includes, in box 1302, one or more processors receiving sensor data of the environment along a portion of the road from one or more onboard vehicle sensors as a first set of training inputs. In box 1304, the method includes, in box 1304, one or more processors receiving non-vehicle information associated with that portion of the road as a second set of training inputs.

[0113] In box 1306, the method includes evaluating a first set of received training inputs and a second set of received training inputs by one or more processors relative to ground truth data of that section of the road. The ground truth data includes one or more measurements of water thickness, such as water film thickness or ice cover across one or more areas of the road section, to provide a categorical or continuous estimate of humidity along one or more areas of the road section. The evaluation generates road humidity information based on the first and second sets of received training inputs and the ground truth data.

[0114] In box 1308, the method further includes generating a deep learning model of road conditions from road moisture information. And in box 1310, the method stores the generated deep learning model of road conditions in memory. This could be the memory of a backend system, such as… Figures 12A-12B The storage system 1210, or the memory of an autonomous vehicle, such as Figure 2 memory 206 or Figure 3A The memory 306. When stored in the memory of the autonomous vehicle, the model can be used during the vehicle's real-time driving operations. For example, the model can be deployed in, for example, in, Figure 12AThe autonomous vehicles in the illustrated fleet are used to classify / regress road humidity using onboard and / or offboard signals as input, without referencing a baseline true value. This model can be applied to each vehicle to enhance autonomous operation. This can include, for example, modifying current driving actions (e.g., changing lanes, slowing down, changing deceleration rate, accelerating, etc.), modifying planned routes or trajectories, activating onboard cleaning systems (e.g., windshield wiper systems, defroster, defrosters, etc.), etc.

[0115] Unless otherwise stated, any alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be interpreted illustratively rather than as a limitation on the subject matter defined by the claims. Furthermore, the provision of examples described herein and clauses expressed as "such as," "comprising," etc., should not be construed as limiting the subject matter of the claims to the specific examples; rather, the examples are intended to illustrate only one of many possible embodiments. Additionally, the same reference numerals in different figures can identify the same or similar elements. Processes or other operations may be performed in different orders or simultaneously unless expressly indicated herein.

Claims

1. A method for generating a deep learning model of road conditions, the method comprising: One or more processors receive sensor data of the environment along a portion of the road from one or more on-board vehicle sensors as the first set of training inputs; One or more processors receive non-vehicle information associated with said portion of the road as a second set of training inputs; The first set of received training inputs and the second set of received training inputs are evaluated by one or more processors relative to the reference ground truth data of the section of the road to give a classification or continuous estimate of humidity in one or more areas along the section of the road. The reference ground truth data includes one or more measurements of water thickness in one or more areas across the section of the road. The evaluation generates road humidity information based on the first and second sets of received training inputs and the reference ground truth data. Generating a deep learning model of road conditions from road moisture information; and The generated deep learning model of road conditions is stored in memory, and the stored deep learning model of road conditions is used to control the operation of the vehicle in autonomous driving mode.

2. The method according to claim 1, wherein, The received sensor data includes one or more of the following: LiDAR returns, camera still images, camera video, radar returns, audio signals, or outputs from the vehicle's onboard modules.

3. The method according to claim 1, wherein, The non-vehicle information includes one or more of the following: weather station information, public weather forecasts, road map data, crowdsourced information, or observations from one or more other vehicles.

4. The method according to claim 1 further includes performing signal fusion on some or all of the received sensor data and received non-vehicle information.

5. The method of claim 1 further includes applying weighting to different signals of the received sensor data.

6. The method of claim 1 further includes performing statistical analysis to determine which received sensor data is correlated with the baseline ground truth data before constructing the deep learning model of road conditions.

7. The method of claim 6 further includes attenuating any received sensor data that does not meet a threshold related to the reference truth data.

8. The method of claim 1, further comprising masking information about one or more dynamic objects on the road from the received sensor data before constructing a deep learning model of road conditions.

9. The method of claim 1, further comprising, before constructing the deep learning model of road conditions, limiting the received sensor data to a selected range or distance from one or more vehicle sensors.

10. The method of claim 1, further comprising smoothing the benchmark ground value measurement of the benchmark ground value data before constructing the deep learning model of road conditions.

11. The method of claim 1, further comprising applying a deep learning model of road conditions to one or more road areas to identify the probability of humidity in each area or to estimate the water film depth in each area.

12. The method of claim 1, further comprising, based on the output of a deep learning model of road conditions, causing one or more systems of the autonomous vehicle to perform changes to the current driving action, modify the planned route or trajectory, or activate at least one of the onboard cleaning systems of the autonomous vehicle.

13. The method according to claim 1, wherein, Storing the generated deep learning model of road conditions in memory includes storing the generated deep learning model of road conditions in the memory of one or more vehicles that are not equipped with a benchmark ground truth measurement sensor. The stored deep learning model of road conditions is configured to evaluate real-time road humidity based on real-time sensor data and selected off-vehicle information.

14. The method according to claim 1, wherein, The baseline ground truth data includes human-marked examples of road humidity.

15. The method according to claim 1, wherein, The baseline true data includes one or more measurements of water thickness, including one or more measurements of water film thickness or ice coverage in one or more areas across the section of the road.

16. A system configured to operate a vehicle in an autonomous driving mode, the system comprising: The memory stores a deep learning model of road conditions, which involves discrete classification or continuous regression / estimation of road humidity. and One or more processors, operatively coupled to the memory, said one or more processors being configured to: When operating in autonomous driving mode, sensor data is received from one or more sensors of the vehicle's perception system, the one or more sensors being configured to detect objects or conditions in the environment surrounding the vehicle. Use stored models to generate information associated with discrete classification or continuous regression / estimation of road humidity based on received sensor data; as well as Use the generated information to control the vehicle's operation in autonomous driving mode.

17. The system according to claim 16, wherein, A model is formed by evaluating a first set of training inputs of sensor data of the environment along said section of the road from one or more onboard sensors relative to a portion of the road, and a second set of training inputs of non-onboard information associated with said section of the road, said benchmark ground data including one or more measurements of water thickness in one or more areas across said section of the road.

18. The system according to claim 17, wherein, The first set of training inputs for sensor data includes one or more of the following: LiDAR returns, camera still images, camera videos, radar returns, audio signals, or outputs from vehicle onboard modules.

19. The system according to claim 17, wherein, The second set of training inputs for non-vehicle information includes one or more of the following: weather station information, public weather forecasts, road map data, crowdsourced information, or observations from one or more other vehicles.

20. The system according to claim 16, wherein, Using the generated information to control the vehicle's operation in autonomous driving mode includes modifying the current driving action, altering the planned route or trajectory, or activating at least one of the onboard cleaning systems.

21. A vehicle configured to operate in an autonomous driving mode, the vehicle comprising: The system according to claim 16; and Perception system.

Images