Augmenting Reality Sensor Recordings with Simulated Sensor Data

Abstract

Raw sensor data is received from one or more sensors of the vehicle. Free space around the vehicle is identified based on the sensor data, such as by identifying areas whose data points have heights below a threshold. A location for an object model is selected from the available space. A plane is fitted to the sensor data around the location, and the object model is oriented according to the orientation of the plane. The object model is simulated by the sensors of the vehicle to obtain simulated data, and then the simulated data is added to the raw sensor data. Sensor data corresponding to objects that would be obscured by the object model is removed from the raw sensor data. The augmented sensor data may be used to validate a control algorithm or train a machine learning model.

Description

Technical Field

[0001] The present invention relates to the sensing of simulated scenes using one or more electronic sensors. Background Art

[0002] The challenges in testing and validating driver assistance technologies and autonomous vehicles come from the large number of test cases and very rare cases. While a wide variety of situations are encountered in real-world testing, certain situations are very rare in the real world and too risky to perform on a proving ground. Detailed simulations are often used to demonstrate these situations. However, due to the lack of perfect sensor models and traffic models, the data generated from these simulations are very realistic but still cannot mimic all the imperfections of the real world.

[0003] The systems and methods disclosed herein provide an improved way to generate scenarios by augmenting recorded sensor data with simulated sensor data. Summary of the invention

[0004] According to the present invention, a method is provided, comprising performing the following operations by a computing device:

[0005] receiving sensor data from a controller of the vehicle;

[0006] identifying available space around the vehicle in the sensor data;

[0007] placing an object model at a position relative to the vehicle;

[0008] simulating sensing of the object model to obtain simulated sensor data; and

[0009] The simulated sensor data is added to the sensor data to obtain augmented sensor data.

[0010] According to one embodiment of the present invention, it includes:

[0011] identifying, by the computing device, a portion of the sensor data that cannot be detected due to placement of the object model at the relative position; and

[0012] The portion of the sensor data is removed from the augmented sensor data by the computing device.

[0013] According to one embodiment of the present invention, it also includes:

[0014] defining a shadow cone of the object model by the computing device relative to a sensor position on the vehicle; and

[0015] The portion of the sensor data that cannot be detected is searched within the shadow cone by the computing device.

[0016] According to one embodiment of the present invention, it also includes:

[0017] identifying, by the computing device, an area of ​​available space surrounding the relative position;

[0018] fitting, by the computing device, a plane to the region; and

[0019] The object model is transformed, by the computing device, according to the orientation of the plane.

[0020] According to one embodiment of the present invention, it also includes:

[0021] identifying, by the computing device, the area as an area covered by the object model;

[0022] Determining, by the computing device, (a) insufficient points of the sensor data to cover the area to define data points of the plane; and

[0023] In response to determining (a), the region is expanded, by the computing device, until the region covers enough data points to fit the plane.

[0024] According to an embodiment of the present invention, identifying the available space around the vehicle in the sensor data comprises:

[0025] evaluating, by the computing device, a height associated with a data point in the sensor data; and

[0026] An area satisfying the following condition is identified as the available space by the computing device: a height associated with a data point in the area is less than a threshold height.

[0027] According to an embodiment of the invention, wherein evaluating the height associated with a data point in the sensor data comprises defining, by the computing device, a height relative to a horizontal plane of the vehicle.

[0028] According to an embodiment of the present invention, the sensor data includes at least one of LIDAR (Light Detection and Ranging), RADAR (Radio Detection and Ranging), and image data.

[0029] According to one embodiment of the present invention, the method further includes testing one of a detection algorithm and a control algorithm according to the amplified sensor data by the computing device.

[0030] According to one embodiment of the present invention, the method further includes training a machine learning model based on the augmented training data by the computing device to perform at least one of the following: detecting obstacles and removing objects.

[0031] According to the present invention, there is provided a system comprising one or more processing devices and one or more memory devices operably connected to the one or more processing devices, the one or more memory devices storing executable code, the executable code effectively causing the one or more processing devices to perform the following operations:

[0032] receiving sensor data from a vehicle controller;

[0033] identifying available space around the vehicle in the sensor data;

[0034] placing an object model at a position relative to the vehicle;

[0035] simulating sensing of the object model to obtain simulated sensor data; and

[0036] The simulated sensor data is added to the sensor data to obtain augmented sensor data.

[0037] According to an embodiment of the present invention, the executable code further effectively causes the one or more processing devices to perform the following operations:

[0038] identifying a portion of the sensor data that cannot be detected due to placement of the object model at the relative location; and

[0039] The portion of the sensor data is removed from the augmented sensor data.

[0040] According to an embodiment of the present invention, the executable code further effectively causes the one or more processing devices to perform the following operations:

[0041] defining a shadow cone of the object model relative to a sensor position on the vehicle; and

[0042] The portion of the sensor data that cannot be detected is searched within the shadow cone.

[0043] According to an embodiment of the present invention, the executable code further effectively causes the one or more processing devices to perform the following operations:

[0044] identifying an area of ​​available space surrounding the relative position;

[0045] fitting a plane to the region; and

[0046] The object model is transformed according to the orientation of the plane.

[0047] According to an embodiment of the present invention, the executable code further effectively causes the one or more processing devices to perform the following operations:

[0048] identifying the area as an area covered by the object model;

[0049] assessing whether (a) the area covers insufficient data points from the sensor data to define a plane; and

[0050] If (a), expand the region until it covers enough data points to fit the plane.

[0051] According to an embodiment of the present invention, the executable code further effectively causes the one or more processing devices to perform the following operations:

[0052] evaluating an altitude associated with a data point in the sensor data; and

[0053] identifying an area that satisfies the following conditions as the available space: a height associated with a data point in the area is less than a threshold height;

[0054] According to one embodiment of the invention, the executable code is further effective to cause one or more processing devices to evaluate the height associated with a data point in the sensor data by defining the height relative to a horizontal plane of the vehicle.

[0055] According to an embodiment of the present invention, the sensor data includes at least one of LIDAR (Light Detection and Ranging), RADAR (Radio Detection and Ranging), and image data.

[0056] According to an embodiment of the present invention, the executable code is further effective to cause the one or more processing devices to test at least one of a detection algorithm and a control algorithm based on the amplified sensor data.

[0057] According to one embodiment of the present invention, the executable code is further effective to cause the one or more processing devices to train a machine learning model based on the augmented training data to perform at least one of the following: detecting obstacles and removing objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to facilitate an understanding of the advantages of the present invention, a more particular description of the invention briefly described above will be presented by reference to specific embodiments shown in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope, the present invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0059] Figure 1A and 1B is a schematic block diagram of a system for implementing an embodiment of the present invention;

[0060] Figure 2 is a schematic block diagram of an example computing device suitable for implementing a method according to an embodiment of the present invention;

[0061] Figure 3 is a process flow diagram of a method for adding simulated sensor outputs of a plant model to actual sensor data according to an embodiment of the present invention;

[0062] Figure 4A and 4B is a schematic block diagram of a scene sensed using an actual sensor;

[0063] Figure 5A and 5B The embodiment of the present invention is shown Figure 4A and 4B Amplification of sensor data;

[0064] Figure 6 is a process flow diagram of a method for augmenting LIDAR data according to an object model according to an embodiment of the present invention; and

[0065] Figures 7A to 7C It shows that according to Figure 6 The method augments the image of LIDAR data. DETAILED DESCRIPTION

[0066] Referring to FIG. 1 , a computing environment 100 may include a server system 102 that may host or access a database 104 storing data for use according to the methods disclosed herein. In particular, the database 104 may store sensor data 106 received from sensors of a vehicle while driving in an actual environment. The sensor data may include one or more of visual data 108a (e.g., outputs of one or more visible light cameras or infrared cameras), LIDAR data 108b output by a LIDAR (light detection and ranging) sensor of the vehicle, and RADAR data 108c output by a RADAR (radio detection and ranging) sensor. Other types of sensor data 106 may also be stored, such as outputs of microphones, ultrasonic sensors, or other types of sensors. The sensor data 108a-108c of each sensor may include a series of data sets output over a period of time during which the vehicle crosses a path.

[0067] The database 104 may further store one or more sensor models 110. The sensor model 110 defines data sufficient to enable a three-dimensional (3D) model to simulate the sensing of an actual sensor. For example, a camera model 112a may define data that defines distortion, color shift, frame rate, resolution, zoom, field of view, orientation, or other artifacts of a camera that will affect scene sensing.

[0068] The LIDAR model 112b may define the scan rate, point density, scanning laser wavelength, beam characteristics, detector sensitivity, field of view, etc. The RADAR sensor model 112c may define the limitations and properties of the RADAR system, such as wavelength, signal amplitude, angular scan speed, antenna gain, field of view, etc. In the presence of other types of sensor data 106, other types of sensor models 110 may be defined for those types of sensors. For example, for a microphone, the sensor model 110 may include gain, signal-to-noise ratio, sensitivity distribution (sensitivity versus frequency), etc.

[0069] The data generated can be used for a variety of purposes: training deep neural networks (see paragraph 17), testing and validating autonomous driving algorithms in the field of sensing, scene understanding, object detection, mission planning, path planning and control. These tests can be designed as open-loop or closed-loop tests.

[0070] The database 104 may also include an object model 114. The object model 114 may include 3D models of objects that the vehicle may encounter (e.g., other vehicles 116a, pedestrians 116b, and other objects 116c such as animals, debris, etc.). The object model 114 may be defined using any 3D modeling format known in the art. In some embodiments, the object model 114 may be represented as a point cloud, a triangular mesh, or other representations of the object outline. The object model 114 may also define material properties associated with real-world analogs of the object model detected by one or more types of sensors. For example, for modeling detection of a camera, attributes such as color, texture, reflectivity, etc. may be specified for points on the surface of the object model 114. For modeling detection of a LIDAR sensor, the object model 114 may define the reflectivity of points on the surface of the object model. For modeling detection of a RADAR sensor, the characteristics may include reflectivity according to the wavelength emitted by the model 112c of the RADAR sensor.

[0071] The database 104 may also store a machine learning model 118 trained using simulated sensor data generated according to the methods disclosed herein. The machine learning model 118 may be trained for any purpose. For example, the machine learning model may be trained to detect obstacles, especially obstacles detected according to the methods disclosed herein. The machine learning model 118 may be a deep neural network, a Bayesian network, or other types of machine learning models.

[0072] In another example, the machine learning model 118 can be trained to estimate the outline of a location without an object when an object is present during the detection scene. For example, a street can be detected using LIDAR that has no vehicles traveling along it or parked next to it. An example application can include obtaining sensor data from an empty street, simulating a model for sensing vehicles along an empty street according to the methods described herein, and then training the machine learning model 118 to generate a point cloud corresponding to the empty street by removing points corresponding to the added vehicles.

[0073] The server system 102 may execute a training engine 120. The training engine 120 may include a locator module 122a. The locator module 122a may be programmed to identify coordinates in a spatial region including sensor data that is not currently occupied by obstacles. As described below, this may include identifying a ground plane, such as a street, side wall, parking lot, or other surface.

[0074] The training engine 120 may include an insertion module 122b. The insertion module 122b selects a location for one or more object models 114 within one or more unoccupied areas identified by the locator module 122a. In some embodiments, the location may be manually specified by a user to create a desired scene. In other embodiments, the insertion module 122b may randomly select a location within one or more unoccupied areas. A non-randomly automatically selected location (e.g., using a vehicle driver's implementation model) may also be used.

[0075] The training engine 120 may include a simulation module 122c. The simulation module 122c simulates the detection of the inserted object model 114 and the adjustment of the sensor data to be consistent with the presence of the object model 114. As described below, this may include removing sensor data that would be obscured if a real-world analogue of the object model 114 actually existed.

[0076] The training engine 120 can execute a training module 122d that trains the machine learning model 118 using the sensor data modified according to the methods disclosed herein. As is known in the art, a large amount of training data is required to train a machine learning model. Therefore, hundreds or thousands of sets of sensor data can be modified and used to train the machine learning model 118.

[0077] refer to Figure 1B, the machine learning model 118 can be used to perform obstacle detection in the illustrated system 124, which can be incorporated into a vehicle (e.g., an autonomous or manually operated vehicle). For example, the system 124 may include a controller 126 housed within the vehicle. The vehicle may include any vehicle known in the art. The vehicle may have all the structure and features of any vehicle known in the art, including wheels, a drive train connected to the wheels, an engine connected to the drive train, a steering system, a braking system, and other systems known in the art contained in a vehicle.

[0078] As discussed in more detail herein, the controller 126 may perform autonomous navigation and collision avoidance. In particular, one or more of LIDAR data, image data, RADAR data, and one or more types of sensor data may be analyzed to identify obstacles.

[0079] The controller 126 may receive one or more image streams from one or more imaging devices 128. For example, one or more cameras may be mounted to the vehicle and output image streams that are received by the controller 126. The controller 126 may receive one or more audio streams from one or more microphones 130. For example, one or more microphones or microphone arrays may be mounted to the vehicle and output audio streams that are received by the controller 126. The microphone 130 may include a directional microphone having a sensitivity that varies with angle.

[0080] In some embodiments, the system 124 may include other sensors 132 connected to the controller 126, such as one or more of LIDAR, RADAR, sonar, and ultrasonic sensors, etc. The positions and orientations of the sensing devices 128, 130, 132 may correspond to those modeled in the sensor model 110 used to train the machine learning model 118.

[0081] The controller 126 may execute a collision avoidance module 134 that receives the outputs of the sensing devices 128, 130, 132, detects obstacles in the outputs, and takes actions to avoid the obstacles. The collision avoidance module 134 may include or be embodied as a machine learning model 118 trained using modified sensor data generated according to the methods described herein. The collision avoidance module 134 may also be a programmed control system tuned or tested using augmented sensor data generated according to the methods disclosed herein.

[0082] The collision avoidance module 134 may include an obstacle identification module 136a that receives the output of the sensing devices 128, 130, 132 and outputs estimates of the locations of obstacles and classifications of possible obstacles (vehicles, pedestrians, buildings, animals, etc.).

[0083] The collision avoidance module 134 may further execute a collision prediction module 136b that predicts which obstacle images may collide with the vehicle based on the vehicle's current trajectory or current expected path. The collision prediction module 136b may assess the likelihood of collision with an object identified by the obstacle identification module 136a.

[0084] Decision module 136c can then make a decision to stop, accelerate, turn, etc. to avoid the obstacle. The manner in which collision prediction module 132c predicts a potential collision and the manner in which decision module 132d takes action to avoid a potential collision can be based on any method or system known in the field of autonomous vehicles.

[0085] The decision module 136c can control the trajectory of the vehicle by driving one or more actuators 138 that control the direction and speed of the vehicle. For example, the actuators 138 can include a steering actuator 140a, an accelerator actuator 140b, and a brake actuator 140c. The configuration of the actuators 140a-140c can be according to any implementation of such actuators known in the field of autonomous vehicles.

[0086] Figure 2 2 is a block diagram illustrating an example computing device 200. Computing device 200 may be used to perform various processes, such as those discussed herein. Server system 102 and controller 126 may have some or all of the attributes of computing device 200.

[0087] The computing device 200 includes one or more processors 202, one or more memory devices 204, one or more interfaces 206, one or more mass storage devices 208, one or more input / output (I / O) devices 210, and a display device 230, all of which are connected to a bus 212. The processor 202 includes one or more processors or controllers that execute instructions stored in the memory device 204 and / or the mass storage device 208. The processor 202 may also include various types of computer-readable media, such as cache memory.

[0088] Memory device 204 includes various computer-readable media such as volatile memory (e.g., random access memory (RAM) 214) and / or nonvolatile memory (e.g., read-only memory (ROM) 216). Memory device 204 may also include a rewritable ROM such as flash memory.

[0089] The mass storage device 208 includes various computer-readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., flash memory), etc. Figure 2As shown, the particular mass storage device is a hard drive 224. Various drives may also be included in the mass storage device 208 to enable reading from and / or writing to various computer-readable media. The mass storage device 208 includes removable media 226 and / or non-removable media.

[0090] I / O devices 210 include various devices that allow data and / or other information to be input into or retrieved from computing device 200. Example I / O devices 210 include cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCD (charge coupled devices) or other image capture devices, and the like.

[0091] Display device 230 includes any type of device capable of displaying information to one or more users of computing device 200. Examples of display device 230 include a monitor, a display terminal, a video projection device, and the like.

[0092] Interfaces 206 include various interfaces that allow computing device 200 to interact with other systems, devices, or computing environments. Example interfaces 206 include any number of different network interfaces 220, such as interfaces to a local area network (LAN), a wide area network (WAN), a wireless network, and the Internet. Other interfaces include a user interface 218 and a peripheral device interface 222. Interfaces 206 may also include one or more peripheral interfaces, such as interfaces for a printer, a pointing device (mouse, track pad, etc.), a keyboard, etc.

[0093] The bus 212 allows the processor 202, memory device 204, interface 206, mass storage device 208, I / O device 210, and display device 230 to communicate with each other and with other devices or components connected to the bus 212. The bus 212 represents one or more of several types of bus structures, such as a system bus, a PCI (Peripheral Controller Interface) bus, an IEEE (Institute of Electrical and Electronics Engineers) 1394 bus, a USB (Universal Serial Bus) bus, and the like.

[0094] For purposes of illustration, programs and other executable program components are shown herein as discrete boxes, but it is understood that such programs and components may reside in different storage components of computing device 200 at different times and be executed by processor 202. Alternatively, the systems and processes described herein may be implemented in hardware or a combination of hardware, software, and / or firmware. For example, one or more application specific integrated circuits (ASICs) may be programmed to perform one or more of the systems and processes described herein.

[0095] refer to Figure 3, the method 300 shown can be performed by the server system 102 to generate simulated sensor data for training purposes. The augmented sensor data can also be used to test and verify control algorithms or other types of systems or methods. In another example, the augmented sensor data can be used to test or verify a detection algorithm for sensing an object using the sensor data. The detection algorithm can be tested alone or in combination with a control algorithm that makes decisions based on the output of the detection algorithm. The method 300 may include receiving 302 raw sensor data. The raw sensor data may include a series of sensor outputs received from actual sensors of the vehicle over a period of time as the vehicle traverses a trajectory. Thus, the sensor data will include data indicating the sensor's sensing of commonly encountered structures, such as one or more of streets, buildings, other vehicles, pedestrians, animals, debris, etc.

[0096] The output of each sensor will include a series of data sets, each data set in the series represents the output of the sensor at one point in time or one period of the sensor refresh rate. Where multiple sensors are used, the raw sensor data will include a series of data sets for each sensor which may have the same or different refresh rates or sampling rates.

[0097] The method 300 may include extracting 304 a driving trajectory of the vehicle. This may include receiving a series of GPS (Global Positioning System) coordinates from the raw sensor data 302. The GPS coordinates may be time-stamped or recorded at a known sampling rate so that the time associated with each coordinate is known and may be simultaneously associated with the output of another sensor (i.e., the time closest to the sampling / refresh rate being unequal or misaligned). The extraction of the trajectory may also be defined by an algorithm that defines the driving trajectory using data from one or more sensors over time. Known concepts for this purpose are, for example, odometry, simultaneous localization and mapping (SLAM), or structure from motion.

[0098] Method 300 may also include extracting 306 free space over time. This may include identifying portions of the vehicle environment that capture unoccupied raw sensor data. For example, for LIDAR data, the points in the point cloud may be in or associated with the orientation of the vehicle. Therefore, the height of the points relative to the vertical axis of the vehicle is known. Those points above a specified threshold height may be determined as potential heights, but it may be determined that points below the threshold may be ground level, such as a road surface. Areas of points below the threshold may then be identified. A cluster of continuous points containing an area that does not include any points above the threshold may be determined as an unoccupied area. In other examples, free space is not limited to being at ground level. For example, a bird in motion, a drone, an overhanging branch, a vehicle, or a person may be positioned in free space above the ground due to an accident or other structure. Therefore, free space may be identified as a volume of space without points above the threshold. The threshold may depend on the application, such as being higher in the case where the object to be added is not on the ground.

[0099] Note that the vertical height of a point above the horizontal plane can be defined with reference to the horizontal plane of the vehicle rather than a plane perpendicular to the direction of gravity. In this way, when the vehicle is located on a hillside, points on the hillside will not be incorrectly interpreted as being above the threshold.

[0100] Since sensor data consists of a series of datasets, the available space can be extracted from a single dataset or multiple consecutive datasets.

[0101] In the case of visual data, the color and texture of the image can be analyzed to identify free space, for example, the color and texture of concrete, asphalt, gravel or other driving surfaces can be identified as free space. Portions of the image that include objects that are identifiable as vehicles, buildings, pedestrians, animals, etc. can be excluded from the free space. In the case where a 3D imaging system can use images from multiple cameras to build a 3D model from one or more 2D images, the 3D locations of these free spaces can be determined from the 3D model. In some embodiments, sensor fusion can be used to combine insights from LIDAR points to create an actual 3D scene combined with the imagery. In such an embodiment, free space can be identified based on the color of areas of the 3D scene (asphalt, concrete, etc.) and the height of points in the scene, as outlined above with respect to LIDAR data.

[0102] The method 300 may include selecting 308 an available space from those identified in step 306. For example, the GPS coordinates of the trajectory may be mapped to the coordinate system of a point cloud of a LIDAR system or a 3D imaging system. An available space may be selected due to the space being located on the trajectory. In many applications, step 308 may be performed manually and may simply include receiving a user selection of an available space identified in step 306. For example, the sensor data may be represented graphically with the available space highlighted and associated with a selectable user interface element to receive a user selection of the available space.

[0103] The method 300 may also include selecting 310 an object model. For example, for the space selected at step 308, an object model having a footprint that is less than or equal to the space may be automatically selected. Thus, the object model 114 may include the size of the footprint of the object model to enable the comparison. For step 308, step 310 may also be performed manually by receiving a user selection of an object model 114 from the set of multiple object models 114.

[0104] Steps 312-318 may be repeated for different data sets corresponding to different time steps to provide a series of augmented data sets that may be used to train a machine learning model. Steps 312-318 may also be performed on the output of multiple sensors.

[0105] The method 300 may also include orienting 312 the object model 114 within the coordinate system of the raw sensor data. For example, for the region selected at step 308, a plane may be fitted to points in the data set identifying the selected region. The object may then be oriented so that the base of the object model (e.g., the bottom of a vehicle tire, a foot of a pedestrian or animal model, etc.) is parallel to and co-located with the plane. The position of the object model 114 within the selected space may be selected to be located on the trajectory of step 304 so that if the object model 114 actually existed, the vehicle would collide with the object model 114.

[0106] As described above, the raw sensor data may include multiple data sets at multiple time points. Thus, step 312 may be performed for multiple data sets so that objects are consistently oriented 312 relative to the coordinate system of each data set, either with respect to stationary objects or objects that move in accordance with their real-world analogs (e.g., the walking speed of a pedestrian, the driving speed of a vehicle, etc.). For example, as described above, the coordinate system of each data set may be associated with the GPS coordinates of the vehicle. Thus, the GPS coordinates of the vehicle that is closest in time to each data set (or approximated for each data set by interpolation) may be used to transform the orientation of the object model from one data set to the next to simulate stationary objects or objects that move according to a desired trajectory relative to the vehicle trajectory.

[0107] The method 300 may then include generating 314 simulated sensor outputs that simulate sensing of the object model from the perspective of the vehicle's sensors. This may include using the corresponding sensor model 110 to simulate sensing of the object model at the location and orientation from step 312. For example, for LIDAR, a point cloud of the surface conforming to the object model may be generated, where the density and reflectivity of the points are determined according to the sensor model 112b of the LIDAR sensor.

[0108] In another example, a rendering of the object model at its position and orientation at step 312 can be generated 314. Techniques for performing this operation may include using any computer animation techniques known in the art, which are commonly used to add computer-generated images to movies and television. In particular, the lighting of the original scene can be approximated to simulate the lighting of the object model 114 and provide a realistic rendering. In addition, the 3D environment (including local lights) can be used to create complex shadows. For example, 3D data from LIDAR point clouds and camera data can be used to detect structures and calculate shadows that will be cast by such structures and object models. The locations of lights and structures that can cast shadows can be obtained from a pre-existing 3D map of the area.

[0109] In yet another example, reflection and detection of radio waves from the RADAR sensor may be simulated for the object model at the location and orientation of step 312 according to the RADAR sensor model 112 c .

[0110] The method 300 may then include adding 316 the simulated sensor data to the raw sensor data 302 to obtain augmented sensor data. Note that any number of objects may be inserted into the scene and the corresponding sensor data may be added to the raw sensor data. For LIDAR, this may include adding a point cloud of the object model to the point cloud of a given data frame (i.e., the data frame analyzed in steps 306-312). For RADAR, the reflections of the RADAR signal simulated in step 314 may be added to the reflections in the raw sensor output within a given time step. For image data, a rendering of the object model may be superimposed on the image or added to a 3D model obtained from the image within a given time step.

[0111] Method 300 may further include removing 318 raw sensor data (if present, the raw sensor data would be blocked by the object model). One advantage of RADAR is that the wave can even "pass through" objects (e.g., between the ground and the lower end of the car), so an object behind another object is not necessarily hidden from the RADAR device, while it may be hidden from the LIDAR or camera (although the car window may provide visibility through the object). Therefore, for RADAR data, these properties of the RADAR wave can be simulated when determining whether the raw data is actually blocked. In the case of adding multiple objects, the raw data blocked by each object can be determined. Portions of the simulated data of the object model blocked by other added object models can also be calculated and removed. Alternatively, in step 314, a simulation of the sensing of the added object model can be performed using multiple object models with relative orientations and positioning from steps 308 and 310. In this way, any blocking will be considered in the simulation of the sensing object model group.

[0112] refer to Figure 4A and 4B as well as Figure 5A and 5B This process and other steps of method 300 may be understood. The augmented sensor data with the data removed according to step 318 may then be output and used for training purposes or any other application in which it may be beneficial.

[0113] like Figure 4A As shown, a typical driving scene includes a road 400 having areas occupied by multiple vehicles 402 and one or more unoccupied areas 404. A host vehicle 406 (i.e., a vehicle sensing the scene) is located on the road 400 and senses the scene (including other vehicles 402) using sensors such as cameras 408a-408c, RADAR sensors 410a-410b, and LIDAR sensors 412. Other sensors may also be used, such as microphones, ultrasonic sensors, etc.

[0114] refer to Figure 5A and 5B In the example shown, an object model 500 of a motorcyclist cutting a gap can be added to the unoccupied area 404. Other example locations include in front of the host vehicle 406, so that the object model 500 would be hit if it were actually there. Simulated sensing of the object model 500 by one or more of the sensors 408a-408c, 410b, 412 can then be performed based on the models of these sensors and the locations of these sensors on the host vehicle 406.

[0115] like Figure 5A and 5BAs is apparent in the figure, the object model 500 is inserted between some of the sensors 408a-408c, 410b, 412 and one other vehicle 402. Therefore, the sensor output corresponding to the blocked vehicle can be removed from the sensor output in the manner described above. In some embodiments, to reduce computational requirements, the blocked sensor data can be evaluated within a cone 502 originating from the location of the sensor (in the example shown, the LIDAR sensor 412) and defining the object model 500. Figure 5A As shown, the cone may be conical or frustoconical, and may be sized such that the object model 500 is completely within the cone 502. For example, the included angle of the cone 502 may be selected such that the cone 502 is tangent to the object model 500 at one or more points, while no point of the object model 500 is outside the cone 502.

[0116] Sensor data having coordinates within cone 502 and generally behind the object may then be evaluated to determine if the point is detectable given the presence of object model 500. For example, in one embodiment, a 3D cylinder may be defined around the object in addition to cone 502. Only those points that are within the cylinder or behind the cylinder relative to the vehicle are then evaluated for removal.

[0117] If not, the point is found to be undetectable. The point can then be removed from the augmented sensor data. As described below, the object model 500 can be represented as a set of triangles. The ray emitted from the sensor position to the point in the original sensor data can be evaluated to determine whether the ray intersects with one of these triangles. If so, the point can be eliminated from the augmented sensor data. This method is useful for LIDAR data and for determining which pixels are covered for a 3D image generated from an image or a combination of one or more images and LIDAR data. For RADAR data, a simulation of the propagation of RADAR waves around the inserted object model can be performed to determine whether the reflection from the original sensor data will receive the transmitted wave and whether the reflection can still reach the sensor.

[0118] refer to Figure 6 , the method 600 shown is a more detailed application of the method 300, which is particularly applicable to LIDAR sensor data. The method 600 can be performed for each frame of the LIDAR data over a period of time. As described above, the position of the object model can be adjusted for each iteration of the method 600 to simulate a stationary object or simulate the desired trajectory of the object model.

[0119] Method 600 may include receiving 602 raw LIDAR sensor output in the same manner as step 302 of method 300. Method 600 may also include selecting a 2D location of an augmented reality (AR) object model. In a typical LIDAR system, the two coordinate axes (X, Y) are parallel to the horizontal plane of the vehicle, for example, parallel to a flat surface supporting the vehicle. Therefore, step 604 includes selecting X and Y coordinates (dx, dy) to place the AR (augmented reality) object model. As described above with respect to step 308, the process may be automatic or may be performed manually. For example, the coordinates dx, dy may be selected from the available space identified in the manner described above with respect to steps 304-308.

[0120] In cases where the AR object model is not placed on the ground, other parameters may be received from the user or automatically selected, such as the vertical position (dz) and orientation along the Z axis, for example to simulate an aerial object.

[0121] The method 600 may include calculating 606 an area under the AR object. For example, this may include identifying the size of a rectangle within which all X and Y coordinates of points on the AR object fit. The method 600 may then include evaluating 608 whether the area includes enough points to define a plane. Specifically, whether there are enough points in the LIDAR point cloud with X and Y coordinates within a rectangle centered at the 2D location selected in step 604.

[0122] The number of points sufficient to define a plane can be a predetermined value specified by the designer. The sufficient number depends on the method used to fit the plane. Any method for finding a plane that approximates an array of points can be used, such as the least squares method, the RANSAC (random sample consensus) algorithm, the Hough transform, or other methods known in the art. For example, three points are sufficient to define a plane. However, this number may be needed many times to improve accuracy, such as a number from 6 to 300.

[0123] If the number of points is insufficient, the region may be incrementally enlarged 610 according to predefined increments or percentages until a sufficient number of points is found 608 to accurately define the plane of the region.

[0124] Then, method 600 may include rotating 612 the AR object model so that it is correctly oriented relative to the plane approximating the region calculated in steps 606-610 using any of the plane definition techniques outlined above. In particular, the AR object model may define a local coordinate system that defines a horizontal plane. A transformation may be applied to the AR object model so that the horizontal plane is parallel to the plane of the region in steps 606-610. The AR object model may define a reference point, such as a center point in the local horizontal plane of the model or some other predefined point. In step 612, the point may be located at the 2D position of step 604.

[0125] The method 600 may further include defining 614 a simulated point cloud of the AR object at the location of step 604 and the orientation of step 614 relative to the vehicle when the raw LIDAR sensor output was captured. This may include simulating a scan of the AR object model using the point density, wavelength, and other properties of the LIDAR sensor model 112b and the reflectivity and size of the AR object model (see the description of the object model 114 above). The points of the point cloud at step 614 may be added 616 to the raw LIDAR sensor output to obtain an augmented sensor output.

[0126] A triangular mesh may also be defined 618 for the AR object model. The triangular mesh defines triangles whose vertices are on the surface of the AR object model. These points may be points of the AR object model itself or points from the point cloud of step 616. The mesh may be calculated automatically, such as a greedy surface triangulation algorithm or other methods known in the art.

[0127] The method 600 may also include calculating 619 a shadow cone for the AR object model. As described above with respect to method 300, this may include defining a truncated cone shape that is tangent to the point of the AR object model, but such that the point of the AR object model is not outside the truncated cone shape. For points of the raw LIDAR sensor output 602 that are within the shadow cone, the method may include performing 620 a ray-triangle intersection test. In particular, if a line extending from the position of the sensor to the point passes through one of the triangles of the AR object model, the point is removed 622 from the augmented sensor output.

[0128] Figures 7A to 7C An example use of method 600 is shown. Fig. 7A The raw LIDAR sensor output is shown. Figure 7B A shadow cone 700 of an AR object model 702 is shown superimposed on the raw LIDAR sensor output. Figure 7C The final augmented sensor output is shown where the simulated point cloud for the object model 702 has been added and the points in the shadow cone 700 have been removed.

[0129] In the above disclosure, reference has been made to the accompanying drawings that form a part of the present invention, and specific embodiments in which the present disclosure can be practiced are shown by way of illustration. It should be understood that other embodiments can be utilized and structural changes can be made without departing from the scope of the present disclosure. References to "one embodiment", "embodiment", "example embodiment" and the like in the specification indicate that the described embodiment may include specific features, structures or characteristics, but each embodiment may not necessarily include specific features, structures or characteristics. Moreover, these phrases do not necessarily refer to the same embodiment. In addition, when specific features, structures or characteristics are described in conjunction with an embodiment, it is proposed that whether or not explicitly described, it is within the knowledge of those skilled in the art to affect these features, structures or characteristics in conjunction with other embodiments.

[0130] Embodiments of the systems, devices, and methods disclosed herein may include or utilize a dedicated or general-purpose computer including computer hardware, for example, one or more processors and system memory as discussed herein. Embodiments within the scope of the present disclosure may also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by a general or dedicated computer system. A computer-readable medium storing computer-executable instructions is a computer storage medium (device). A computer-readable medium carrying computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of the present disclosure may include at least two distinct computer-readable media: a computer storage medium (device) and a transmission medium.

[0131] Computer storage media (devices) include RAM, ROM, EEPROM (electrically erasable programmable read-only memory), CD-ROM (compact disc read-only drive), solid-state drive ("SSD") (e.g., RAM-based), flash memory, phase change memory ("PCM"), other memory types, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired program code in the form of computer-executable instructions or data structures and that can be accessed by a general or special purpose computer.

[0132] The embodiments of the equipment, system and method disclosed herein can communicate through a computer network. "Network" is defined as one or more data links capable of transmitting electronic data between a computer system and / or module and / or other electronic devices. When information is transmitted or provided to a computer through a network or other communication connection (hardwired, wireless or a combination of hardwired or wireless), the computer correctly regards the connection as a transmission medium. The transmission medium may include a network and / or a data link, which may be used to carry a desired program code device in the form of a computer executable instruction or a data structure and may be accessed by a general or special purpose computer. The above combination should also be included in the scope of a computer-readable medium.

[0133] Computer executable instructions include, for example, instructions and data that, when executed at a processor, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions. Computer executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, or even source code. Although the subject matter is described in a language dedicated to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features and actions are disclosed as example forms of implementing the claims.

[0134] Those skilled in the art will appreciate that the present disclosure can be practiced in a network computing environment with many types of computer system configurations, including in-dash vehicle computers, personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs (personal digital assistants), tablet computers, pagers, routers, switches, various storage devices, etc. The present disclosure can also be implemented in a distributed system environment, where local and remote computer systems linked by a network (by a hardwired data link, a wireless data link, or a combination of hardwired and wireless data links) all perform tasks. In a distributed system environment, program modules can be located in local and remote memory storage devices.

[0135] In addition, where appropriate, the functions described herein may be performed in one or more of the following: hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) may be programmed to perform one or more of the systems and processes described herein. Certain terms are used throughout the specification and claims to refer to specific system components. As will be appreciated by those skilled in the art, components may be referenced by different names. This document is not intended to distinguish between components that have different names but the same function.

[0136] It should be noted that the sensor embodiments discussed above may include computer hardware, software, firmware, or any combination thereof to perform at least a portion of their functions. For example, the sensor may include computer code configured to be executed in one or more processors, and may include hardware logic / circuitry controlled by the computer code. These exemplary devices are provided herein for illustrative purposes, not restrictive. As will be known to those skilled in the relevant art, embodiments of the present disclosure may be implemented in other types of devices. At least some embodiments of the present disclosure relate to a computer program product including such logic (e.g., in the form of software) stored on any computer-usable medium. When executed in one or more data processing devices, such software causes the device to operate as described herein.

[0137] Computer program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may be executed entirely on a computer system as a stand-alone software package, executed on a separate hardware unit, partially on a remote computer separated from the computer by a distance, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the computer through any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] The present invention is described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustration and / or block diagram and the combination of blocks in the flowchart illustration and / or block diagram can be implemented by computer program instructions or codes. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, so that instructions executed via the processor of the computer or other programmable data processing device create means for implementing the functions / actions specified in the flowchart and / or block diagram blocks.

[0139] These computer program instructions may also be stored in a non-transitory computer-readable medium, which can instruct a computer or other programmable data processing device to act in a specific manner, so that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means, which implements the functions / actions specified in the flowchart and / or block diagram blocks.

[0140] The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus so that a series of operational steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in the flowchart and / or block diagram blocks.

[0141] Although various embodiments of the present disclosure have been described above, it should be understood that they are presented only by way of example, not by way of limitation. It will be apparent to those skilled in the relevant art that various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure. Therefore, the breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should only be limited according to the appended claims and their equivalents. The foregoing description has been presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. In view of the above teachings, many modifications and variations are possible. In addition, it should be noted that any or all of the above-described alternative embodiments may be used in any desired combination to form another hybrid embodiment of the present disclosure.

Claims

1. A method comprising performing the following operations by a computing device: receiving sensor data from a controller of the vehicle; identifying available space around the vehicle in the sensor data; placing an object model at a position relative to the vehicle; simulating sensing of the object model to obtain simulated sensor data; as well as The simulated sensor data is added to the sensor data to obtain augmented sensor data.

2. The method according to claim 1, comprising: identifying, by the computing device, a portion of the sensor data that cannot be detected due to placement of the object model at the relative position; as well as The portion of the sensor data is removed from the augmented sensor data by the computing device.

3. The method according to claim 2, further comprising: defining a shadow cone of the object model by the computing device relative to a sensor position on the vehicle; as well as The portion of the sensor data that cannot be detected is searched within the shadow cone by the computing device.

4. The method according to claim 1, further comprising: identifying, by the computing device, an area of ​​available space surrounding the relative position; fitting a plane to the region by the computing device; as well as The object model is transformed, by the computing device, according to the orientation of the plane.

5. The method according to claim 4, further comprising: identifying, by the computing device, the area as an area covered by the object model; Determining, by the computing device, that (a) the area covers insufficient data points of the sensor data to define the plane; as well as In response to determining (a), the region is expanded, by the computing device, until the region covers enough data points to fit the plane.

6. The method of claim 1 , wherein identifying the available space around the vehicle in the sensor data comprises: evaluating, by the computing device, a height associated with a data point in the sensor data; as well as identifying, by the computing device, an area satisfying the following conditions as the available space: the height associated with the data point in the area is less than a threshold height; Wherein assessing the height associated with a data point in the sensor data comprises defining, by the computing device, the height relative to a horizontal plane of the vehicle.

7. The method of claim 1, wherein the sensor data comprises at least one of LIDAR (Light Detection and Ranging), RADAR (Radio Detection and Ranging), and image data.

8. The method of claim 1 , further comprising using the augmented training data for at least one of: testing the detection algorithm by means of the computing device; testing a control algorithm by means of the computing device; training a machine learning model by the computing device to detect obstacles; and The machine learning model is trained, by the computing device, to remove objects from the sensor output.

9. A system comprising one or more processing devices and one or more memory devices operably connected to the one or more processing devices, the one or more memory devices storing executable code effective to cause the one or more processing devices to: receiving sensor data from a vehicle controller, the sensor data comprising at least one of LIDAR (light detection and ranging), RADAR (radio detection and ranging), and image data; identifying available space around the vehicle in the sensor data; placing an object model at a position relative to the vehicle; simulating sensing of the object model to obtain simulated sensor data; as well as The simulated sensor data is added to the sensor data to obtain augmented sensor data.

10. The system of claim 9, wherein the executable code is further effective to cause the one or more processing devices to: identifying a portion of the sensor data that cannot be detected due to placement of the object model at the relative location; and The portion of the sensor data is removed from the augmented sensor data.

11. The system of claim 10, wherein the executable code is further effective to cause the one or more processing devices to: defining a shadow cone of the object model relative to a sensor position on the vehicle; and The portion of the sensor data that cannot be detected is searched within the shadow cone.

12. The system of claim 9, wherein the executable code is further effective to cause the one or more processing devices to: identifying an area of ​​available space surrounding the relative position; fitting a plane to the region; and The object model is transformed according to the orientation of the plane.

13. The system of claim 12, wherein the executable code is further effective to cause the one or more processing devices to: identifying the area as an area covered by the object model; evaluating whether (a) the region covers insufficient data points of the sensor data to define the plane; and If (a), expand the region until it covers enough data points to fit the plane.

14. The system of claim 9, wherein the executable code is further effective to cause the one or more processing devices to: evaluating an altitude associated with a data point in the sensor data; and identifying an area satisfying the following condition as the available space: the height associated with the data point in the area is less than a threshold height; in, The executable code is further effective to cause the one or more processing devices to evaluate the height associated with a data point in the sensor data by defining the height relative to a horizontal plane of the vehicle.

15. The system of claim 9, wherein the executable code is further effective to cause the one or more processing devices to use the augmented sensor data for at least one of: Testing detection algorithms; Testing control algorithms; Training machine learning models to detect obstacles; and The machine learning model is trained to remove objects from the sensor output.

Images