Method, system and vehicle for excluding a model of a vehicle from a sensor field of view

Abstract

The disclosed technology involves developing a highly accurate understanding of the sensor field of view relative to the vehicle itself. A training phase is used to collect sensor data in a variety of situations and scenarios, and a modeling phase takes such information and identifies self-return signals and other signals (730) that should be excluded or accounted for from analysis during real-time driving to avoid false positives (Figures 8A-8B). The result is a sensor field of view model for a specific vehicle (Figure 6B) that can be extended to other similar makes and models of that vehicle. This approach enables the vehicle to determine when sensor data is coming from the vehicle or other information. Thus, detailed modeling allows the onboard computing system to make driving decisions and take other actions based on accurate sensor information.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Application No. 16 / 225,745, filed December 19, 2018, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to methods, systems, and vehicles for excluding a model of a vehicle from a sensor field of view. Background Art

[0004] Vehicles configured to operate in an autonomous mode (such as a vehicle that does not require a human driver) may be used to help transport passengers or cargo from one location to another. This may include operating in a fully autonomous mode, where a passenger may provide some initial input, such as a destination, and the vehicle will maneuver itself to that destination without further driving input from the passenger. It may also include operating in a semi-autonomous mode, where certain driving operations are managed by the vehicle and other operations are managed by the driver.

[0005] In order for a vehicle to drive in a fully or semi-autonomous mode, the vehicle may rely heavily on systems that are able to determine its location at any given time and detect and identify objects outside the vehicle (such as other vehicles, stop lights, signs, pedestrians, etc.). As examples, these systems may include sensors, such as laser scanning devices (Lidar), radars, and cameras, which may be mounted at various locations on the vehicle. Such sensors may all have a specific field of view (FOV) of the vehicle's external environment. However, obstacles may cause signals to affect the FOV of a given sensor. This may result in false positive signals, which may affect the understanding of objects in the external environment. And this may affect driving decisions and other actions when the vehicle is operating in an autonomous mode. Summary of the invention

[0006] The technology involves a more accurate understanding of the field of view of the vehicle's sensors relative to the vehicle itself. This enables the vehicle to know when sensor data is coming from the vehicle or other information. In this way, a highly detailed model of a given vehicle, or type of vehicle, is developed. While a rough bounding box can be used given the general size and shape of a specific make or model, this approach may not account for different vehicle configurations or valid operating conditions. In contrast, a fine 3D map of the vehicle developed based on this technology can incorporate different configurations or conditions. This 3D map is used to eliminate signals in the sensor data that are due to the vehicle itself, allowing the system to make driving decisions and take other actions based on accurate sensor information.

[0007] Aspects of the technology include a method for generating a sensor field of view (FOV) model of a vehicle. The method includes acquiring sensor data of an external environment of the vehicle from one or more sensors mounted on the vehicle; filtering out sensor data that does not meet a predetermined range condition to acquire resultant sensor data; projecting the resultant sensor data onto a range image by one or more processors, wherein the data in the range image includes a plurality of points, each of the plurality of points having a respective range value associated therewith; aggregating sensor data associated with different sets of acquired sensor data by the one or more processors; and creating a 3D mesh representation of each of the one or more sensors by the one or more processors, the 3D mesh representation being based on the range image.

[0008] The 3D mesh representation of each of the one or more sensors may also be surrounded by points that do not exceed the coarse bounding box vehicle model.

[0009] In one scenario, the method further includes transforming the 3D mesh representation from a vehicle coordinate system (frame) to a sensor coordinate system using an external transformation specific to the vehicle; and projecting the 3D coordinates into the sensor coordinate system to create a model range image. In this case, for each new 3D point acquired by a given sensor, the method may include projecting the new 3D point onto the model range image; and filtering out the one or more selected points if the range of the one or more selected points is not within a given range.

[0010] The method may include generating a vehicle model from a combination of 3D mesh representations of each of the one or more sensors. Here, the vehicle model is arranged to exclude the 3D mesh representation when detecting objects external to the vehicle. Here, the method may include applying the vehicle model to a generic vehicle brand or model to obtain a generic vehicle model. In this case, the method may also include sending the generic vehicle model to the fleet for use during driving operations in the autonomous driving mode.

[0011] Acquiring sensor data may include operating the vehicle in the actual environment in one or more scenarios, and collecting sensor data of the vehicle's external environment during operation in the one or more scenarios. In this case, the one or more scenarios may include at least one of the following: a turning scenario in which one or more wheels of the vehicle are turned to a maximum extent; a stationary scenario in which at least one of a door, a trunk, or a hood of the vehicle is in a partially or fully open position; or a scenario in which at least one window of the vehicle is open and an object is extended out of the window.

[0012] In yet another example, creating the 3D mesh representation of each of the one or more sensors includes performing a transformation from sensor coordinate system coordinates to vehicle coordinate system coordinates.

[0013] In another example, the method includes incorporating an intensity channel into the 3D mesh representation to account for different colors or lighting ranges of acquired sensor data. This may include applying the intensity channel to assess whether a sensor on a vehicle is covered in debris, damaged, or submerged in water.

[0014] According to other aspects of the technology, a processing system includes a memory configured to store acquired sensor data, and one or more processors operatively coupled to the memory. The one or more processors are configured to receive sensor data of an external environment of a vehicle from one or more sensors of the vehicle. They are also configured to filter out sensor data that does not meet a predetermined range condition to obtain result sensor data and project the result sensor data into a range image. The data in the range image includes a plurality of points, wherein each of the plurality of points has a respective range value associated therewith. The one or more processors are also configured to aggregate sensor data associated with different sets of acquired sensor data, and create a 3D mesh representation for each of the one or more sensors, the 3D mesh representation being based on the range image.

[0015] The 3D mesh representation of each of the one or more sensors may also be surrounded by points that do not exceed the coarse bounding box vehicle model.

[0016] In one example, the one or more processors are further configured to transform the 3D mesh representation into a sensor coordinate system using a vehicle-specific external transformation, and project the 3D coordinates into the sensor coordinate system to create a model range image. In this case, the one or more processors may also be configured to, for each new 3D point acquired by a given sensor, project the new 3D point onto the model range image, and filter out the one or more selected points if the range of the one or more selected points is not within a given range.

[0017] In another example, the one or more processors are further configured to incorporate an intensity channel into the 3D mesh representation to account for different colors or lighting ranges of the acquired sensor data.

[0018] In another example, the one or more processors are further configured to generate a vehicle model from a combination of the 3D mesh representations of each of the one or more sensors. The vehicle model is arranged to exclude the 3D mesh representation when detecting objects external to the vehicle. In this case, the one or more processors may also be configured to apply the vehicle model to a generic vehicle brand or model to obtain a generic vehicle model.

[0019] According to a further aspect of the technology, a vehicle includes a drive system including a steering subsystem, an acceleration subsystem, and a deceleration subsystem that control the drive of the vehicle, and a plurality of wheels configured to contact a driving surface. It also includes a perception system including one or more sensors arranged along the vehicle. At least one of the one or more sensors has a respective detection field of view (FOV). The vehicle also includes a control system including one or more processors. The control system is operatively coupled to the drive system and the perception system, and is configured to operate the vehicle along the driving surface in one or more scenarios; during operation in one or more scenarios, obtain sensor data of the vehicle's external environment from one or more sensors; send the obtained sensor data to a remote system; receive a vehicle model from the remote system based on the field of view of the one or more sensors; and operate the vehicle along a road in response to the received vehicle model.

[0020] In one example, the one or more scenes include at least one of: a turning scene in which one or more wheels of the vehicle are turned to a maximum extent; a stationary scene in which at least one of the vehicle's doors, trunk, or hood is in a partially or fully open position; or a scene in which at least one window of the vehicle is open and an object is extended out of the window. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A vehicle having a sensor according to aspects of the present disclosure.

[0022] Figure 2 is a system diagram of a vehicle according to aspects of the present disclosure.

[0023] Figure 3A-3C Examples of sensor FOV and signals according to aspects of the present disclosure are shown.

[0024] Figure 4 Sensor data according to aspects of the present disclosure is shown.

[0025] Figure 5A-Figure 5B are pictorial and functional diagrams of example systems according to aspects of the present disclosure.

[0026] Figure 6A-6B According to aspects of the present disclosure, a bounding box approach is contrasted with a modeling approach.

[0027] Figure 7A-7B Comparative exemplary signal conditions according to aspects of the present disclosure are shown.

[0028] Figure 8A-8B is an example flow chart according to aspects of the present disclosure. DETAILED DESCRIPTION

[0029] Overview

[0030] This technology involves collecting sensor information during actual driving data collection of the vehicle and using the collected sensor information to create a 3D map of the vehicle during the training phase. During driving, it may be difficult to determine whether the sensor is "seeing" the world correctly, or whether there are certain signals due to the vehicle itself or related to the vehicle, such as dust mist or water spray caused by the vehicle's tires when driving on the road. Therefore, sensor data should be collected under various conditions and situations. The training phase extracts the collected sensor data into a model that defines what each sensor itself is generally expected to see (and / or a trusted range of sensor readings). After the model (or (multiple) models) are trained and developed, the vehicle's on-board system is able to determine whether the sensor data is due to its own returns or from another object, and make operational decisions accordingly, such as during the operation phase.

[0031] Example System

[0032] like Figure 1 As shown, a vehicle 100 according to various aspects of the present disclosure includes various sensors for detecting objects in the vehicle's surroundings. Although certain aspects of the present disclosure are particularly useful for specific types of vehicles, the vehicle can be any type of vehicle, including but not limited to a car, a small or large truck, a motorcycle, a bus, a recreational vehicle, etc. In this example, the roof housing 110 supports a sensor unit 112, such as a lidar sensor, which can be part of a sensor assembly 114 including other sensors such as a camera and / or a radar unit. In addition, a housing 120 located at the front end of the vehicle 100 and housings 130a, 130b located on the driver's side and the passenger side of the vehicle can each include a lidar sensor and / or other sensors. Other housings 140a and 140b can be placed along other parts of the vehicle, such as shown toward the upper rear of the vehicle 100. In this example, the housing 130a is located in front of the driver's door 150 and can be located near the driver's side front tire 160, the hood 170, and the rearview mirror unit 180. Additional sensor units (not shown) may be located at the front and rear ends of vehicle 100 and / or at other locations along the roof or roof shell, such as along area 190 .

[0033] Figure 2 A block diagram 200 of various systems of the vehicle 100 is shown. For example, as shown, the vehicle may have one or more computing devices, such as a computing device 202 that includes one or more processors 204, memory 206, and other components typically found in a general purpose computing device.

[0034] Memory 206 stores information accessible by one or more processors 204, including instructions 208 and data 210 that can be executed or otherwise used by processor 204. Memory 206 can be any non-transitory type capable of storing information accessible by a processor, including computing device readable media, or other media that stores data that can be read by electronic devices, such as hard disk actuators, memory cards, ROM, RAM, optical disks, and other writable and read-only memories. Systems and methods may include different combinations of the above, where different portions of instructions and data are stored on different types of media.

[0035] Instructions 208 may be any set of instructions that are executed by a processor directly (such as machine code) or indirectly (such as a script). For example, instructions may be stored as computing device code on a computing device readable medium. In this regard, the terms "instructions" and "programs" may be used interchangeably herein. Instructions may be stored in an object code format for direct processing by a processor, or in any other computing device language, including scripts or collections of independent source code modules that are interpreted on demand or pre-compiled. The functions, methods, and routines of instructions will be described in more detail below.

[0036] Data 210 may be retrieved, stored, or modified by processor 204 in accordance with instructions 208. For example, although the claimed subject matter is not limited to any particular data structure, the data may be stored in a computing device register, in a relational database, as a table with multiple different fields and records, an XML document, or a flat file. The data may also be formatted in any format readable by a computing device.

[0037] The one or more processors 204 may be any conventional processor, such as a commercially available CPU. Alternatively, the one or more processors may be a dedicated device, such as an ASIC or other hardware-based processor. Figure 2 The processor, memory, and other elements of computing device 202 are functionally shown in the same block, but one of ordinary skill in the art will appreciate that a processor, computing device, or memory may actually include multiple processors, computing devices, or memories, which may or may not be stored in the same physical housing. For example, a memory may be a hard disk actuator or other storage medium located in a different housing than computing device 202. Therefore, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories, which may or may not operate in parallel.

[0038] The computing device 202 may include all components typically used in conjunction with a computing device such as the processor and memory described above. The computing device may include or be associated with a user interface subsystem 212, which may include, for example, a mouse, a keyboard, a touch screen, and / or a microphone, as well as various electronic displays (e.g., a monitor with a screen or any other electronic device operable to display information). For example, a vehicle may include an interior electronic display and one or more speakers to provide information or an audio-visual experience. In this regard, the interior electronic display may be located within the cabin of the vehicle 100 and may be used by the computing device 202 to provide information to passengers within the vehicle 100.

[0039] The communication system 214 may include one or more wireless or wired connections to facilitate communication with other computing devices, such as passenger computing devices (e.g., mobile phones, smart watches, laptops or tablet computers, etc.) and remote server computing devices described further below. Examples of wireless connections include short-range communication protocols such as Bluetooth, Bluetooth low energy (LE), and 802.11, cellular connections, and various configurations and protocols including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local area networks, private networks using one or more company-proprietary communication protocols, Ethernet, WiFi, and HTTP, and various combinations of the above.

[0040] In one example, computing device 202 may be part of an autonomous driving computing system incorporated into vehicle 100. The autonomous driving computing system may be capable of communicating with various components of the vehicle in order to operate vehicle 100 in a fully autonomous driving mode and / or a semi-autonomous driving mode. Figure 2 In order to control the movement, speed, etc. of the vehicle 100 according to the instructions 208 of the memory 206, the computing device 202 can communicate with various systems of the vehicle 100, such as a deceleration system 216, an acceleration system 218, a steering system 220, a signal system 222, a navigation system 224, a positioning system 226, a perception system 228, and a power system 230 (e.g., a gasoline or diesel engine, an electric engine, or a hybrid engine). The wheels 232 are coupled to the transmission system 234 and can also communicate with the computing device 202. Although these systems are shown as being external to the computing device 110, in reality, different systems can also be incorporated into the computing device 202, for example, as part of an autonomous driving computing system for controlling the vehicle 100 in a fully or partially autonomous driving mode.

[0041] As an example, computing device 202 may interact with deceleration system 216, acceleration system 218, and transmission system 234 to control the speed and related aspects of the vehicle. Similarly, steering system 220 may be used by computing device 202 to control the direction of vehicle 100. For example, if vehicle 100 is configured for use on a road, such as a car or truck, the steering system may include components that control the angle of the wheels to turn the vehicle. Signal system 222 may be used by computing device 202 to signal the vehicle's intentions to other drivers or vehicles, such as by illuminating turn signals or brake lights when necessary.

[0042] Navigation system 224 may be used by computing device 202 to determine and follow a route to a location. In this regard, navigation system 224 and / or data 210 may store detailed map information, such as a highly detailed map that identifies the shape and elevation of roads, lane markings, intersections, crosswalks, speed limits, traffic signals, buildings, signs, real-time traffic information, vegetation, or other such objects and information. In other words, such detailed map information may define the geometry of the vehicle's expected environment, including roads and speed limits (legal speed limits) on those roads. In addition, the map information may include information about traffic controls, such as traffic lights, stop signs, yield signs, etc., which, combined with real-time information received from perception system 228, may be used by computing device 202 to determine which directions of traffic have the right of way at a given location.

[0043] The perception system 228 also includes one or more components for detecting objects outside the vehicle, such as other vehicles, bicycles, pedestrians, road obstacles, traffic signals, signs, trees, etc. For example, the perception system 228 may include one or more lidar sensors 236, radar sensors 238, cameras or other image sensors 240, and / or other sensors 242 (e.g., sonar devices) that can detect objects in the environment surrounding the vehicle. Other sensors 242 may also include any other detection devices that record data about the vehicle, which data may be processed by the computing device 202 (such as accelerometers, gyroscopes, etc.). Other sensors 242 may also include sensors that detect the state or configuration of the vehicle 100. For example, other sensors 242 may sense that a door is partially or fully open, a trunk or hood is partially or fully open, and / or one or more windows are partially or fully rolled down.

[0044] The sensors of the perception system can detect objects in the environment outside the vehicle and generate sensor data describing the characteristics of such objects, such as location, orientation, size, shape, type, direction, and speed of movement. The raw sensor data from the sensors and / or the above characteristics can be quantized or arranged as a description function or vector and sent to the computing device 202 for further processing. As discussed in further detail below, the computing device 202 can use the positioning system 226 to determine the location and attitude of the vehicle, and use the perception system 228 to detect and respond to objects when necessary to safely reach the location.

[0045] LiDAR sensor 236, radar sensor 238, image sensor 240, and / or other sensors 242 of perception system 228 may each have its own FOV. And each such sensor may be arranged on the vehicle such that all or part of a given sensor's FOV overlaps at least a portion of the field of view of one or more other sensors. Figure 3A An example of this is shown in , which shows a scenario where there is another vehicle in front of a vehicle that is configured to operate in an autonomous driving mode. Figure 3A A vehicle 300 is shown having a roof sensor assembly 302 and a front sensor assembly 304. The vehicle 300 is located behind another vehicle 306. The roof sensor assembly 302 has a sensor FOV 308, while the front sensor assembly 304 has a sensor FOV 310. As shown, the sensor FOV 308 may be arranged to detect the upper rear of the vehicle 306, while the sensor FOV 310 may detect the lower rear of the vehicle 306, and there may be some overlap in the fields of view.

[0046] For example, the roof sensor assembly 302 and / or the front sensor assembly 304 may include one or more lidar sensors. The roof sensor assembly 302 and the front sensor assembly may detect objects in the external environment by performing one or more scans of the area surrounding the vehicle. Figure 3B As shown, the lidar sensor of the roof sensor assembly can perform multiple scans of an area of ​​interest. Here, line 310 indicates an example of a separate scan of the environment. For example, each scanning cycle may be performed by a given sensor 10 times (or more or less) separate scans. This may include adjusting the sensor's FOV up or down, left or right, for example, with a generator or other actuator. Individual scans can be selected to cover a specific portion of the sensor's FOV or a selected area around the vehicle.

[0047] However, the sensor may also receive feedback from the vehicle itself (self-reflection). Figure 3CAs shown, the scan may generate returns from various parts of the vehicle, such as returns from the roof 312, returns from side mirrors or side sensors 314, returns from wheels 316 (e.g., when the vehicle turns left or right), returns from a partially or fully opened trunk 318, and returns from a partially or fully opened door 320. Other parts of the vehicle may also generate returns depending on the configuration of the vehicle, the type of sensor, the FOV of the sensor, etc. When operating, such as autonomous driving, if the range of a given return is greater than the range of the model (at that angle) plus some increment (e.g., 10-50 cm), the system can infer that the given return is a self-return.

[0048] Figure 4 A scene with an example of closed sensor data is shown. As shown here, vehicle 400 includes a pair of front side sensors 402a, 402b, a lower front side sensor 404, and a pair of rear mounted sensors 406a, 406b. Self-returns 408 (side panel or sensor housing), 410 (door), 412 (side mirror), and / or 414 (wheel) can be detected by front side sensor 402. Self-return signal 416 (front bumper) can be detected by front sensor 404. And signals 418 (rear side panel or trunk) and 420 (dust or water spray) can be detected by rear mounted sensor 406.

[0049] To account for this information (eg, to determine that it is not caused by an object in close proximity to the vehicle) and to provide a more robust approach than a rough bounding box, this technique is used in both the training and modeling phases as described above.

[0050] Example Method

[0051] Various aspects of the technology involve collecting sensor data and using that data to build robust models to understand which sensor data is relevant and which can be ignored. While it is possible to manually build a detailed 3D model of a given vehicle, this is labor intensive and does not scale when a model is needed for each individual vehicle or fleet of similar vehicles. Models may be provided by vehicle brands based on the vehicle as “designed.” However, these may not be sufficient, particularly for vehicles with sensors or aftermarket components added to them later (e.g., roof racks, tow hooks, bike racks, fishing rod holders, etc.). It is also difficult to manually account for all possible vehicle states, such as when the wheels are turned, doors or windows are open, etc.

[0052] The collection of sensor data includes actual driving of the vehicle in various scenarios. For example, the vehicle can be driven along a closed route or selected roads. This can be done without nearby objects, such as other vehicles, bicycles, pedestrians, street signs or building signs, vegetation, etc. As the vehicle drives, the sensors (e.g., LiDAR, radar, imaging, etc.) collect data based on their respective fields of view. This information will be used to develop a 3D mesh for each sensor, which will collectively form an overall model of the specific vehicle.

[0053] As discussed here, points inside the grid (those with smaller range than the model's range image predicts) are likely real objects. Therefore, they are not filtered out; instead, they are passed to the perception pipeline (e.g., Figure 2 In contrast, points that fall outside the grid (or outside the model prediction range) are filtered out. This is because such points are occluded by the vehicle itself and therefore must come from the vehicle itself or are spurious returns.

[0054] When driving the vehicle to obtain actual data, different vehicle settings are evaluated, such as to take into account "deformable" parts of the vehicle, such as parts that can be turned, opened or closed, or have different positions. For example, the wheels can be turned left and right to the maximum extent, such as making a U-turn or a multi-point turn (e.g., a K-turn) to check the maximum angle, etc. When the vehicle is parked, the trunk, doors, and possibly the hood can be partially or completely opened. And the side mirrors can be facing or away from the doors. Lidar and other sensors can be used to detect obstacles created by such vehicles, which may generate self-returns. Sensors can also be used to detect other situations, such as when a passenger extends his arm out of the window, or when dust, water, or snow fog from the tires, etc. Different models can be developed or maintained for specific configurations or situations. Therefore, in one example, a given model can be shared between all vehicles of the same brand and / or the same model. And in another example, a model is established for each unique vehicle (one model for each vehicle identification number).

[0055] For example, actual driving can take into account different driving environments and conditions, such as dusty roads, gravel or other debris sprayed by tires, exhaust emissions, etc. It can also take into account the inaccuracy of sensors. Dust, gravel, debris, water or snow fog, etc. may be confined to certain areas of the vehicle, such as areas near tires or wheel wells. In one example, the resulting information should not be considered part of the vehicle when developing a 3D mesh and a resulting vehicle model. However, in another instance, such information can be incorporated into a given model. One reason for including such information in a given model is that it can better represent the information that a given sensor usually sees, so this information should be ignored during driving. However, this may make it difficult for the vehicle to react to real obstacles in areas where dust, gravel, debris, etc. are prone to appear. Including such information can distinguish between transient items (e.g., dust, gravel, debris, water or snow fog, etc.) and pedestrians, bicycles, or other objects near the vehicle.

[0056] Actual driving may include one or both of manual driving and fully automated driving. This helps to obtain sensor data obtained by both driving modes. For example, manual driving may involve "corner case" states that are rarely encountered during automated driving (e.g., multi-point turns, waving to the window while driving, etc.). Driving can be constituted, for example, by the vehicle or driver performing certain actions, such as turning, braking, accelerating or decelerating in specific situations. Driving can also be constituted so that data is captured at different times of the day and / or under different lighting conditions, such as at dawn or dusk, where lighting may affect the results detected by certain sensors. Similarly, data can be captured for different weather conditions (e.g., snow, sleet or rain, and clear and dry weather) and different seasons. Sensor data collection can be implemented as a specific set of one or more independent training operations. Alternatively, sensor data collection can be performed as part of a calibration process, or can be performed during non-calibration driving operations.

[0057] Typically, information acquired from various scans and scenes from a given vehicle is collected and provided to a computing system for analysis and modeling. The sensor data point cloud may be stored in an organized format, such as a linearized matrix of points. Some of the collected sensor data may initially be filtered out and not stored because they cannot be part of the vehicle due to not meeting a predetermined range condition. For example, a ground point (sensor data on the ground where the vehicle is located) is not part of the vehicle. This may include not meeting a selected height threshold associated with the vehicle or sensor location. Other sensor data may be filtered out because even using a coarse A geometric model is used to identify sensor data points that are too far away to be part of the vehicle. For example, a sphere or other geometric shape may be associated with a particular sensor. Here, if points fall outside a given shape, they may be automatically excluded from further consideration when developing a 3D mesh. In another case, the coarse geometric model may be a bounding box of the vehicle, which is specified in the vehicle coordinate system and shared among all sensors and will mean that each sensor has different range conditions at different angles. One or more such coarse model-based filtering operations may be performed continuously or in parallel. The computing system obtains the resulting sensor information and projects the aggregated sensor data into a range image (e.g., a 2D grid from the perspective of each sensor). The minimum range of all aggregated scans in the data, as well as statistical information such as range variance, return intensity, etc., are recorded for each cell of the range image. From the 2D range image and the calibrated external pose of a given sensor, the system is able to generate a 3D mesh in vehicle coordinate system coordinates. In other words, the system performs a transformation from sensor coordinate system coordinates to vehicle coordinate system coordinates. The 3D mesh may be stored, for example, as an image (2D mesh) with a channel for x-coordinates, a channel for y-coordinates, and a channel for z-coordinates. The 3D meshes associated with the various sensors may be integrated to form an overall vehicle model for a given vehicle, or used independently on a per-sensor basis.

[0058] The purpose of converting to 3D vehicle coordinate system coordinates is to account for differences in sensor position / orientation when the model is loaded onto or otherwise used on different vehicles. A separate calibration procedure can be used to determine the position / orientation of vehicle-specific sensors. Once the 3D mesh is loaded into the vehicle coordinate system of the new vehicle, it is reprojected into a 2D range image in the sensor coordinate system of the new vehicle using the individually calibrated sensor position / orientation. This can then be checked against the 2D mesh during operation (operation phase). This approach is very computationally efficient compared to trying to determine the position of each point on a 3D grid.

[0059] Other aspects that this approach takes into account include intensity. For example, an intensity channel (e.g., stored as a grayscale image corresponding to the stored 3D mesh face pixels) can be included in the 3D model to account for different colors or lighting ranges detected by the sensor. This helps, for example, to distinguish between a dust cloud and a pedestrian standing next to a vehicle (if dust returns are included in the vehicle model), or to distinguish between a vehicle and a pedestrian standing next to a vehicle.

[0060] Likewise, by comparing the observed data to a model of the vehicle shape and appearance (including mesh geometry and strength), it can be detected if a sensor is dirty, submerged in water, or obscured by debris during operation.

[0061] Data from other sensors in the vehicle can be used to detect different vehicle configurations. For example, various sensors may indicate whether a door is open, the wheel angles, and any other articulation states of the vehicle. This allows the system to use the appropriate model in a specific state, such as when performing self-return filtering tasks or sensor obstruction detection tasks. For example, there may be a specific model for "sliding door open" that may be different from the model for "sliding door closed."

[0062] During modeling, the system can create "hard negatives." Hard negatives are examples that pop up during online operation where the model fails to filter out part of a vehicle (e.g., when turning a corner, some tires appear as an obstacle, and the vehicle brakes hard and gets stuck behind an imaginary obstacle). In this case, the system or a human operator can verify that no obstacle is actually there. Based on this, the system adds this data back to the model's dataset (along with its previous data). In this way, hard negatives can be used to improve the model to better handle similar situations.

[0063] The 3D model can be developed by a computing system using machine learning methods. For example, a deep neural network can be used to perform a binary classification task for self-return filtering. During learning, the labeled instances are decomposed into training and test sets, and the training and test sets are fed back to optimization (such as stochastic gradient descent using cross entropy loss) to select the best classifier parameters. These parameters are then fixed and used to estimate the "true or false" of new data encountered during the operation phase. The model can be trained using the acquired scan data, such as instances with positive labels, where obstacles exist due to deformable parts of the vehicle, dust or debris, etc.; and instances with negative labels, where no obstacles exist. Training can also include, for example, placing pedestrians or other objects around the vehicle, or evaluating arbitrary log data, and using a labeler to identify which laser returns near the vehicle are real objects that the system should detect and which are not.

[0064] In addition to performing vehicle-specific analyses and transferring models from one vehicle to another, the modeling phase can also merge information acquired from many different vehicles of the same make and / or model. Here, in addition to the range image of the initial vehicle (the "canonical" range image), the system also records an external transformation from the sensor coordinate system to the vehicle coordinate system. Then, for a new vehicle with a slightly different external transformation (Tv→s), the system converts the standard range image into a vehicle-specific image by applying the following transformation:

[0065]

[0066] This transformation may be calculated once at initialization. Once a 3D model (or (s) of) vehicles of the same make and / or model are developed, the 3D model will be provided to such vehicles for use during future fully or partially automated driving.

[0067] The system performing the modeling can be part of a system, e.g. Figure 5A-Figure 5B shown. Figure 5A-Figure 5B 5 and 6 are schematic diagrams and functional diagrams of an example system 500, respectively, which includes multiple computing devices 510, a database 520, and a workstation 530 connected via a network 540. The system 500 also includes various vehicles 100A, 100B, 100C...100N, which may be of the same general type (e.g., the same brand and / or model). Although only some vehicles and computing devices are described for simplicity, a typical system may include more vehicles and computing devices.

[0068] like Figure 5B As shown, each computing device in computing device 510 may include one or more processors, memory, data, and instructions. Such processors, memory, data, and instructions may be configured similarly to one or more processors 204, memory 206, instructions 208, and data 210 of computing device 202.

[0069] The network 540 and intermediate nodes may include a variety of configurations and protocols, including short-range communication protocols such as Bluetooth, Bluetooth LE, 802.11, the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, a private network using one or more company-proprietary communication protocols, Ethernet, WiFi, and HTTP, and various combinations of the above. Such communications may be facilitated by any device capable of sending data to and from other computing devices such as modems and wireless interfaces.

[0070] In one example, the one or more computing devices 510 may include one or more server computing devices having multiple computing devices (e.g., a load balanced server farm) that exchange information with different nodes of the network to receive, process, and send data from and to other computing devices. For example, the one or more computing devices 510 may include one or more server computing devices that can communicate with the computing device 202 of each vehicle 100 and the workstation 530 via the network 540. For example, vehicles 100A...100N may be part of a fleet that can be dispatched to various locations by the server computing device 510 or the workstation 530. In this regard, the server computing device 510 can be used as a modeling computing system that can be used to generate a 3D model of a vehicle. In addition, the server computing device 510 can use the network 540 to send and present information to a user such as a user of the workstation 530 on a display 532 of the computing device 530. In this regard, the computing device (workstation) 530 can be considered a client computing device.

[0071] like Figure 5B As shown in , workstation 530 can be a personal computing device intended for use by a user and has all components commonly used in association with a personal computing device, including one or more processors (e.g., a central processing unit (CPU)), memory (e.g., RAM and internal hard drive) to store data and instructions, a display, such as display 532 (e.g., a monitor with a screen, touch screen, projector, television, or other device operable to display information), and a user input device (e.g., a mouse, keyboard, touch screen, or microphone). Workstation 530 may also include a camera for recording a video stream, speakers, a network interface device, and all components for connecting these elements to each other.

[0072] Although workstation 530 may include a full-size personal computing device, it may alternatively include a mobile computing device capable of wirelessly exchanging data with a server over a network such as the Internet. By way of example only, workstation 530 may be a mobile phone or a device such as a wireless-enabled PDA, tablet computer, or laptop PC, or a netbook capable of acquiring information via the Internet or other network. For example, a user may input information using a keypad, keys, a microphone, visual signals using a camera, or a touch screen.

[0073] A user of workstation 530 can control or modify aspects of the 3D model generation. For example, a user can label or classify a given set of sensor data as self-returns from a deformable portion of a vehicle, or indicate that the sensor data is actually a dust cloud, water or snow fog, etc. Here, the user can identify positive and negative examples of self-returns. Alternatively or in addition, a user of workstation 530 can set scenarios and conditions for data collection during the training phase.

[0074] Once a 3D model of a vehicle has been developed and shared with one or more vehicles, new query points can be evaluated using the model to determine if they are self-returns (or transients, etc.). For example, if the range of the observed data point is within the tolerance of the 3D model's predicted range, the vehicle's onboard system might infer that it is a self-return (and filter it out). If its range is outside the model, the system might infer that it should be occluded by the vehicle and therefore must be noise or a false artifact (and filter it out). However, if it is closer than the model, then it is likely a return from an external object. Other recorded features can also be used to decide if the return is from the vehicle itself. This may include return intensity, per-pixel range variation (e.g., for articulated parts of the vehicle, such as wheels), etc.

[0075] The end result is a robust global model (or ensemble of models) that excludes sensor data about the vehicle (self-returns) and excludes or accounts for other sensor data, such as transient conditions due to dust, water, snow, fog, etc. Figures 6-7 contrast the bounding box approach with such modeling. For example, Fig. 6A An example bounding box 600 is shown as a rectangular box arrangement around the vehicle. In contrast, Figure 6B A model 610 according to the above techniques is shown. It can be seen that the model 610 has stricter constraints on the vehicle.

[0076] Figure 7A-7B The advantages of this modeling are shown. In particular, Fig. 7A A scene 700 of a vehicle 710 is shown with a bounding box 720. Here, a set of self-return data 730 (shown as three arc-shaped dashed lines located on the driver's side mirror) falls within the bounding box 720. The figure also shows another set of return data 740 from an object such as a basketball. This return data is partially within the bounding box 720 and partially outside the bounding box 720. In this bounding box approach, the onboard system may not be able to discern whether the return data 730 and / or 740 should be ignored or considered when analyzing the external environment.

[0077] In contrast, Figure 7BExample 750 includes a vehicle model 760. Vehicle model 760 has been developed to account for self-returns and other signals. For example, any object that appears in the free space between a given sensor mounted on the side of the vehicle (e.g., the rear quarter panel) and a protruding side-view mirror will be within the grid. This may be a real object (e.g., a pedestrian). In contrast, any object behind the side mirror is blocked from view from the perspective of the given sensor and therefore cannot be a real object. And any object that is exactly on the side mirror will be exactly on the grid surface and will be a self-return and should also be filtered out.

[0078] Thus, in this example, return data 730 would be marked as self-returned and filtered out of the received sensor data set (and thus marked as excluded or otherwise ignored by the vehicle's perception or planning system). In contrast, since return data 740 is not within the scope of model 760, the object associated with the return data would be detected by the sensor system and that information would be taken into account by the planning system in the operation of the vehicle.

[0079] Other benefits of this modeling approach are the ability to verify proper functionality of vehicle sensors during operation, detect sensor blockages (such as dirty or damaged lidar windows), verify sensor calibration, help perform calibrations, and identify vehicle defects such as body damage.

[0080] Fig. 8A 800 is a flowchart of an example process for generating a field of view model according to the methods discussed herein. When the items are shown in a particular order, the process may be performed in parallel or in a different order unless explicitly stated herein. In box 802, sensor data of a given vehicle's external environment is acquired. The sensor data is generated by one or more sensors on the vehicle, such as a lidar, radar, and / or image sensor, each of which may have a specific field of view. The sensor may directly sense the range of an object based on the sensor data. In box 804, the sensor data is filtered so that data that does not meet a predetermined range condition is excluded from further consideration. As described above, a ground point or other points that are too far away may be excluded. The synthesized sensor data is then further analyzed and processed.

[0081] For example, at box 806, the sensor data is projected into a range image. This can be done on a per-sensor basis. The data in the projected range image includes a plurality of points. Each such point has a respective range value associated with it. The range value can be a minimum or maximum range value, or a set of ranges [minimum, maximum]. At box 808, the sensor data associated with the different sets acquired are aggregated. For example, a given lidar sensor can collect a set of multiple sets of results as the vehicle drives in the environment. Or various tests can be performed at different times (e.g., at night, during the day, in different weather conditions, etc.). Aggregation can produce a minimum value in the aggregated data set. At box 810, the system creates a 3D mesh representation of each sensor in one or more sensors based on the range image. Creating a 3D mesh representation of each sensor in one or more sensors can include performing a conversion from a sensor coordinate system coordinate (sensor coordinate system) to a vehicle coordinate system coordinate (vehicle coordinate system). The 3D mesh representation of each sensor can also be surrounded by selected points that do not exceed the rough bounding box vehicle model.

[0082] Figure 8B A flowchart 820 of an example method for filtering points based on a 3D grid representation is shown. At box 822, the 3D grid representation of a given sensor is converted to a sensor coordinate system. The 3D grid representation has 3D coordinates in a vehicle coordinate system. The conversion is done using a previously calibrated, vehicle-specific external transformation. At box 824, the 3D coordinates are projected into the sensor coordinate system to create a model range image. At box 826, for each new 3D point sensed by a given sensor (e.g., a laser / lidar sensor), the point is projected onto the model range image. For example, this will produce pixel coordinates [i, j] and range (r). Then at box 828, if the range (r) is equal to or greater than the model range [i, j] minus a certain difference (ε), the point is filtered out to account for the small range error reported by the sensor (where (ε) is greater than zero).

[0083] Thus, when detecting objects external to the vehicle during future driving operations, the system is able to exclude sensor data outside of the 3D mesh representation. For example, the system is able to identify self-returns and other returns and drive the vehicle in an automated mode or otherwise operate accordingly.

[0084] According to other aspects of the technology, the system can build models dynamically, such as when driving in autonomous mode. When the system can confidently determine (from other sensors, environmental knowledge and / or received information, etc.) that there are no external objects around the vehicle, the model can be updated. In this case, the training, modeling, and operation phases can be viewed as parallel.

[0085] For example, when a vehicle is driving at 60 mph on a highway, anything in the space that the vehicle just passed through must be empty. This determination is made with confidence because it can be reasonably assumed based on the available information that there is no tailgate approaching the vehicle, and / or because the system would detect an impending collision if the space was occupied when the vehicle passed through it. Or, alternatively, there may be another vehicle in the vicinity that can see the vehicle and determine using its onboard sensors that there are no objects nearby, and transmit this information to the vehicle.

[0086] The techniques and methods discussed above provide a very accurate mesh or other representation of what is the “vehicle” versus what is the “environment.” This, in turn, allows the onboard systems to make effective decisions during autonomous driving.

[0087] Unless otherwise stated, the above alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the above features can be utilized without departing from the subject matter defined by the claims, the description of the above embodiments should be made by way of illustration rather than by limitation of the subject matter defined by the claims. In addition, the provisions of the examples described herein and clauses with the wording "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; rather, the examples are intended to illustrate only one of many possible embodiments. In addition, the same reference numbers in different figures may identify the same or similar elements.

Claims

1. A method for generating a sensor field of view (FOV) model of a vehicle, the method comprising: acquiring sensor data of an environment external to the vehicle from one or more sensors mounted on the vehicle; filtering, by one or more processors, sensor data that does not satisfy a predetermined range condition to obtain result sensor data; discarding, by the one or more processors, the filtered sensor data; storing, by the one or more processors, resulting sensor data; projecting, by the one or more processors, the resulting sensor data into a range image, wherein the range image is created by projecting the 3D coordinates into a sensor coordinate system, and the data in the range image includes a plurality of points, each point in the plurality of points having a respective range value associated therewith; aggregating, by the one or more processors, sensor data associated with different sets of acquired sensor data; creating, by the one or more processors, a 3D mesh representation of each of the one or more sensors, the 3D mesh representation being based on the range image; and A sensor field of view model of the vehicle is provided based on at least one 3D mesh representation of the one or more 3D mesh representations.

2. The method according to claim 1, wherein: The 3D mesh representation of each of the one or more sensors is also surrounded by points that do not exceed the coarse bounding box vehicle model.

3. The method according to claim 1, further comprising: The 3D mesh representation is transformed from the vehicle coordinate system to the sensor coordinate system using a vehicle-specific extrinsic transformation.

4. The method according to claim 3, further comprising: For each new 3D point acquired by a given sensor, project the new 3D point onto the model range image; and If the range of one or more selected points is not within the given range, the one or more selected points are filtered out.

5. The method of claim 1 , further comprising generating a vehicle model from a combination of 3D mesh representations of each of the one or more sensors, wherein: The vehicle model is arranged to exclude the 3D mesh representation when detecting objects external to the vehicle.

6. The method according to claim 1, wherein: Obtaining sensor data includes: Operate the vehicle in a real-world environment in one or more scenarios; and Sensor data of an environment external to the vehicle is collected during operation in the one or more scenarios.

7. The method according to claim 6, wherein: The one or more scenarios include at least one of the following: Steering scenarios, where one or more wheels of the vehicle are turned to the maximum extent; A stationary scene where at least one of the vehicle's doors, trunk, or hood is in a partially or fully open position; or A scene in which at least one window of a vehicle is open and an object is extended out of the window.

8. The method according to claim 1, wherein: Creating a 3D mesh representation of each of the one or more sensors includes performing a transformation from sensor coordinate system coordinates to vehicle coordinate system coordinates.

9. The method of claim 1, further comprising incorporating an intensity channel into the 3D mesh representation to account for different colors or lighting ranges of the acquired sensor data.

10. The method of claim 9, further comprising applying an intensity channel to assess whether a sensor on the vehicle is covered with debris, damaged, or submerged in water.

11. The method of claim 5, further comprising applying the vehicle model to a generic vehicle make or model to obtain a generic vehicle model.

12. The method of claim 11 further comprising sending the generic vehicle model to a fleet of vehicles for use in an autonomous driving mode during driving operations.

13. A processing system comprising: a memory configured to store acquired sensor data; and One or more processors operatively coupled to the memory, the one or more processors configured to: receiving sensor data of an environment external to the vehicle from one or more sensors of the vehicle; filtering out sensor data that does not satisfy a predetermined range condition to obtain result sensor data; Discard the filtered sensor data; storing the resulting sensor data; projecting the resulting sensor data into a range image, wherein the range image is created by projecting the 3D coordinates into a sensor coordinate system, the data in the range image comprising a plurality of points, each point in the plurality of points having a respective range value associated therewith; aggregating sensor data associated with different sets of acquired sensor data; and A 3D mesh representation of each of the one or more sensors is created, the 3D mesh representation being based on the range image.

14. The processing system of claim 13, wherein: The 3D mesh representation of each of the one or more sensors is also surrounded by points that do not exceed the coarse bounding box vehicle model.

15. The processing system of claim 13, wherein: The one or more processors are further configured to: The 3D mesh representation is transformed into the sensor coordinate system using a vehicle-specific extrinsic transformation.

16. The processing system of claim 15, wherein: The one or more processors are further configured to: For each new 3D point acquired by a given sensor, project the new 3D point onto the model range image; and If the range of one or more selected points is not within the given range, the one or more selected points are filtered out.

17. The processing system of claim 13, wherein: The one or more processors are further configured to incorporate an intensity channel into the 3D mesh representation to account for different colors or lighting ranges of the acquired sensor data.

18. The processing system of claim 13, wherein: The one or more processors are further configured to generate a vehicle model from a combination of the 3D mesh representations of each of the one or more sensors, wherein the vehicle model is arranged to exclude the 3D mesh representation when detecting objects external to the vehicle.

19. The processing system of claim 18, wherein: The one or more processors are further configured to apply the vehicle model to a generic vehicle make or model to obtain a generic vehicle model.

20. A vehicle comprising: The drive system includes a steering subsystem, an acceleration subsystem, and a deceleration subsystem for controlling the driving of the vehicle; a plurality of wheels configured to contact the driving surface; a perception system including one or more sensors arranged along the vehicle, at least one of the one or more sensors having a respective detection field of view (FOV); and A control system, including one or more processors, the control system is operably coupled to the drive system and the perception system, the control system is configured to: operating the vehicle along a driving surface in one or more scenarios; acquiring sensor data of an environment external to the vehicle from the one or more sensors during operation in the one or more scenarios; receiving, from a remote system, a vehicle model based on sensor data collected from one or more vehicles; and operating the vehicle along the road in response to the received vehicle model, Wherein, operating the vehicle along the road in response to the received vehicle model comprises: filtering out sensor data that does not satisfy a predetermined range condition to obtain result sensor data; and Discard the filtered sensor data.

21. The vehicle of claim 20, wherein: The one or more scenarios include at least one of the following: Steering scenarios, where one or more wheels of the vehicle are turned to the maximum extent; A stationary scene where at least one of the vehicle's doors, trunk, or hood is in a partially or fully open position; or At least one window of the vehicle is open and the object is extended out of the window.

Images