Simulating LiDAR Devices and Systems

Abstract

A system and method for generating simulated lidar using radar and image data is provided. Deep learning techniques, such as loss functions, are used to train an algorithm to generate simulated lidar data using radar and image data. Once trained, the algorithm can be implemented in a system, such as a vehicle, equipped with radar and image sensors to generate simulated lidar data that describes the system's environment. The vehicle control system can use this simulated lidar data to determine, generate, and implement modified driving maneuvers.

Description

Technical Field

[0001] The present disclosure relates generally to sensor systems, and particularly to sensor systems for motor vehicles. Background Art

[0002] Transportation methods have changed significantly in recent years. This change is driven in part by concerns about the limited availability of natural resources, the proliferation of personal technology, and a societal shift toward more environmentally friendly transportation solutions. These concerns have encouraged the development of many new types of flexible-fuel vehicles, hybrid electric vehicles, and electric vehicles.

[0003] These changes have led to the development of autonomous and semi-autonomous vehicles, which rely on a variety of sensors to collect data about the vehicle's environment to determine driving maneuvers. For example, the need to create a detailed visual representation of the vehicle's environment has been recognized. Against this backdrop, lidar has become one of the most critical components for higher levels of autonomous vehicle operation. The need for lidar sensors is crucial for accurately estimating the 3D structure of the environment. While lidar point clouds are quite sparse, they are still sufficient for most perception problems. However, the main challenge with lidar systems is their high manufacturing cost, typically ranging from several thousand to tens of thousands of dollars, which significantly increases the cost of vehicle manufacturing. Another challenge is that most systems require more than one lidar system to fully model the 3D scene, which incurs further expense and complexity.

[0004] Another common system used to acquire 3D information is radar. Unlike lidar, radar points are extremely sparse. However, radar systems are much cheaper to manufacture. Furthermore, advances in radar technology have enabled higher point density imaging, making radar a more affordable, albeit noisier, alternative to lidar. Another key advantage of radar over lidar is that radar systems have longer operating ranges and are better able to operate in low-visibility conditions, such as cloudy weather or nighttime operations. However, radar data alone is often insufficient to generate the type of high-quality 3D images required for autonomous vehicle operation.

[0005] Therefore, it would be advantageous for the system to use less expensive radar equipment to simulate high-quality lidar data. The present disclosure relates to a learning-based architecture that learns to generate simulated lidar (pseudo lidar or imitation lidar) data from traditional image data (including night vision camera data) and radar point clouds. By combining images rich in textual and semantic information with sparse and noisy 3D scene structures acquired from radar, deep learning algorithms can be trained to generate simulated lidar data that can be used in real time. In essence, the present disclosure relates to using much cheaper radar and image sensors to generate point clouds similar to those generated using expensive lidar equipment.

[0006] Various techniques have been tried to generate simulated lidar data. These techniques include those described in the following references, which are incorporated herein by reference:

[0007] [1] Wang, Yan, et al, “Pseudo-lidar from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.

[0008] [2] You, Yurong et al., “Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving,” arXiv preprint arXiv:1906.06310 (2019).

[0009] [3] Weng, Xinshuo, and Kris Kitani, “Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud,” arXiv preprint arXiv:1903.09847 (2019).

[0010] [4] Zhou, Tinghui et al., “Unsupervised learning of depth and ego-motion from video”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.

[0011] [5] Godard, Clément, Oisin Mac Aodha, and Gabriel J. Brostow, “Unsupervised monocular depth estimation with left-right consistency,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.

[0012] [6] “Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3DObject Detection for Autonomous Driving”, https: / / patrick-llgc.github.io / Learning-Deep- Learning / paper_notes / pseudo_lidar.html

[0013] For example, Wang and You attempted to compute depth maps using stereo images or monocular frames, and Zhou and Godard used adjacent frames to teach similar techniques. In these techniques, a spatial transformer network-based architecture is used to regress the transformation between frames, which is then used to predict depth. Most of these approaches aim to estimate a depth map (the 3D value of each pixel in the image) from an image (mainly using image consistency or stereo settings). The depth map is then used to generate a lidar point cloud. Although the pseudo-lidar approach of Wang, You, and Godard is reasonable, it has three main drawbacks.

[0014] First, setting up stereo cameras with a wide baseline (as used by Wang and You) on a vehicle is very challenging because calibrating the stereo cameras is cumbersome and definitely not safety-critical for production. As a result, most stereo camera setups require cumbersome autocalibration modules and other precautions to make the system safe, which is tedious. For this reason, most camera-based autonomous solution providers avoid using stereo camera setups.

[0015] On the other hand, methods using a monocular camera (multiple frames), while exempt from most of the problems mentioned above, are subject to a slightly different set of challenges. Extracting pose information from consecutive pairs is equally challenging, if not more challenging. Any algorithm that performs this task must account for independently moving objects and, if possible, model the motion of that object well, while also modeling the motion of the scene. Most approaches to this problem, such as those used by Zhou and Godard (which typically use image consistency), suffer from the aforementioned problems and have yet to provide a reliable solution.

[0016] Furthermore, when using a monocular camera, the estimated depth is only valid up to an arbitrary scale, and the true scale is unknown to the system. Although the relative scale and relative Euclidean geometry between the points are maintained, the actual scale remains unknown unless it is externally injected using a sensor. Therefore, all algorithms built on such a monocular approach will work under the arbitrary scale assumption. Summary of the Invention

[0017] The present disclosure may provide a comprehensive solution to the above problems. The present disclosure may include algorithms that can be trained using either a multi-view / multi-frame setup (stereo or multiple adjacent monocular frames) or a single frame. At inference time, the algorithm only requires a single camera frame and the radar point cloud of that frame to generate comprehensive simulated lidar data. The method can regress the point cloud at the true scale of the scene rather than at an arbitrary scale. Furthermore, the method can use a physically based rendering engine to generate customizable lidar point clouds. The physically based rendering engine of the present disclosure can operate like a lidar beam, allowing the system to generate simulated point clouds specific to sensors such as Velodyne (rotating) or Aeva (solid state), which have greater control over the number of beams and point density than conventional lidar point clouds. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A vehicle according to an embodiment of the present disclosure is shown;

[0019] Figure 2 shows a plan view of a vehicle according to at least some embodiments of the present disclosure;

[0020] Figure 3 is a block diagram of an embodiment of a communication environment of a vehicle according to an embodiment of the present disclosure;

[0021] Figure 4 An embodiment of an instrument panel of a vehicle according to one embodiment of the present disclosure is shown;

[0022] Figure 5is a block diagram of an embodiment of a communication subsystem of a vehicle;

[0023] Figure 6 is a block diagram of a computing environment associated with the embodiments presented herein;

[0024] Figure 7 is a block diagram of a computing device associated with one or more components described herein;

[0025] Figure 8A shows a graphical representation of sensor information detected by a first sensor of a vehicle over time according to an embodiment of the present disclosure;

[0026] Figure 8B shows a graphical representation of sensor information detected by a second sensor of the vehicle over time according to an embodiment of the present disclosure;

[0027] Figure 8C shows a graphical representation of sensor information detected by a third sensor of the vehicle over time according to an embodiment of the present disclosure;

[0028] Figure 9 showing a graphical representation of compared sensor information between sensors of a vehicle according to an embodiment of the present disclosure;

[0029] Figure 10 showing a graphical representation of compared sensor information between sensors of a vehicle according to an embodiment of the present disclosure;

[0030] Figure 11A A schematic diagram illustrating imaging sensor information detected by an imaging system of a vehicle at a first traveling moment according to an embodiment of the present disclosure;

[0031] Figure 11B A schematic diagram illustrating imaging sensor information detected by an imaging system of a vehicle at a second traveling moment according to an embodiment of the present disclosure;

[0032] Figure 11C A schematic diagram illustrating imaging sensor information detected by an imaging system of a vehicle at a third traveling moment according to an embodiment of the present disclosure;

[0033] Figure 12 is a block diagram of an embodiment of a communication environment of a vehicle according to an embodiment of the present disclosure;

[0034] Figure 13 is a flow chart illustrating a method for training a simulated lidar system according to an embodiment of the present disclosure; and

[0035] Figure 14 A flow chart illustrating operation of a vehicle according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0036] Embodiments of the present disclosure will be described in conjunction with vehicles, and in some embodiments, electric vehicles, rechargeable electric vehicles, and / or hybrid electric vehicles, and associated systems.

[0037] Figure 1 A perspective view of a vehicle 100 according to an embodiment of the present disclosure is shown. The electric vehicle 100 includes a vehicle front 110, a vehicle tail or rear 120, a roof 130, at least one vehicle side 160, a vehicle chassis 140, and a vehicle interior 150. In any case, the vehicle 100 can include a frame 104 and one or more body panels 108 mounted or attached thereto. The vehicle 100 can include one or more interior components (e.g., components within the interior space 150 of the vehicle 100 or a user's space, etc.), exterior components (e.g., components outside the interior space 150 of the vehicle 100 or a user's space, etc.), a drive system, a control system, structural components, etc.

[0038] Although shown in the form of a car, it should be understood that the vehicle 100 described herein may include any vehicle or type of vehicle designed to move one or more tangible objects, such as people, animals, cargo, etc. The term "vehicle" does not require that the vehicle move or be capable of moving. Typical vehicles may include, but are not limited to, cars, trucks, motorcycles, buses, automobiles, trains, rail vehicles, boats, ships, marine vehicles, submarine vehicles, airplanes, space shuttles, aircraft, human transport vehicles, etc.

[0039] In some embodiments, the vehicle 100 may include a plurality of sensors, devices, and / or systems that can assist in driving operations. Examples of various sensors and systems may include, but are not limited to, one or more of the following: cameras (e.g., independent images, stereo images, combined images, etc.), infrared (IR) sensors, radio frequency (RF) sensors, ultrasonic sensors (e.g., transducers, transceivers, etc.), radar sensors (e.g., object detection sensors and / or systems), ranging sensors and / or devices (e.g., encoders, etc.), orientation sensors (e.g., accelerometers, gyroscopes, magnetometers, etc.), navigation sensors and systems (e.g., GPS, etc.), and other ranging, imaging, and / or object detection sensors. The sensors may be disposed in the interior space 150 of the vehicle 100 and / or on the exterior of the vehicle 100. In some embodiments, the sensors and systems may be disposed in one or more parts of the vehicle 100 (e.g., the frame 104, body panels, compartments, etc.).

[0040] The vehicle sensors and systems may be selected and / or configured to suit the level of operation associated with the vehicle 100. Among other things, the number of sensors used in the system may be varied to increase or decrease the information available to the vehicle control system (e.g., affecting the control capabilities of the vehicle 100). Additionally or alternatively, the sensors and systems may be part of one or more advanced driver assistance systems (ADAS) associated with the vehicle 100. In any case, the sensors and systems may be used to provide driving assistance at any level of operation (e.g., from fully manual operation to fully autonomous operation, etc.), as described herein.

[0041] The various control and / or operation levels of the vehicle can be described as corresponding to the level of autonomy achieved for vehicle driving operations associated with vehicle 100. For example, in Level 0, or fully manual driving operation, the driver (e.g., a human driver) may be responsible for all driving control operations associated with the vehicle (e.g., steering, acceleration, braking, etc.). Level 0 may be referred to as a "no automation" level. In Level 1, the vehicle may be responsible for a limited number of driving operations associated with the vehicle, while the driver remains responsible for most driving control operations. Examples of Level 1 vehicles include vehicles in which throttle control and / or braking operations (e.g., cruise control, etc.) can be controlled by the vehicle. Level 1 may be referred to as a "driver assistance" level. In Level 2, the vehicle may collect (e.g., via one or more driver assistance systems, sensors, etc.) information about the vehicle's environment (e.g., surrounding area, road, traffic, environmental conditions, etc.) and use this collected information to control driving operations associated with the vehicle (e.g., steering, acceleration, braking, etc.). In Level 2 autonomous vehicles, the driver may be required to perform other aspects of driving operations that are not under the vehicle's control. Level 2 may be referred to as a "partial automation" level. It will be appreciated that Levels 0 to 2 all involve the driver monitoring the driving operation of the vehicle.

[0042] At Level 3, the driver can disengage from control of all driving operations of the vehicle except when the vehicle requests operator action or intervention to control one or more driving operations. In other words, the driver can disengage from control of the vehicle unless the driver is required to take over the vehicle. Level 3 can be referred to as the "conditional automation" level. At Level 4, the driver can disengage from control of all driving operations of the vehicle, and the vehicle can control driving operations even when the user fails to respond to an intervention request. Level 4 can be referred to as the "high automation" level. At Level 5, the vehicle can control all driving operations associated with the vehicle in all driving modes. A Level 5 vehicle can continuously monitor traffic, vehicles, roads, and / or environmental conditions while driving the vehicle. At Level 5, no human driver interaction is required in any driving mode. Therefore, Level 5 can be referred to as the "full automation" level. It should be understood that at Levels 3 to 5, the vehicle and / or one or more automated driving systems associated with the vehicle monitor the vehicle's driving operations and the driving environment.

[0043] like Figure 1 As shown, the vehicle 100 may include, for example, at least one of a ranging and imaging system 112, imaging sensors 116A, 116F (e.g., camera, IR, etc.), a radio object detection and ranging system sensor 116B (e.g., radar, RF, etc.), an ultrasonic sensor 116C, and / or other object detection sensors 116D, 116E. In some embodiments, the sensors may be mounted on a roof 130 of the vehicle 100. In one embodiment, the radar sensor 116B may be disposed at least at the front 110, rear 120, or side 160 of the vehicle 100. Among other things, the radar sensor may be used to monitor and / or detect the location of other vehicles, pedestrians, and / or other objects near or approaching the vehicle 100. While shown as being associated with one or more areas of the vehicle 100, it should be understood that Figure 1 and Figure 2 Any of the sensors and systems 116A- 116K, 112 shown in FIG. 1 may be disposed in, on, and / or around the vehicle 100 in any location, area, and / or zone of the vehicle 100 .

[0044] Now refer to Figure 2 , depicts a plan view of a vehicle 100 according to an embodiment of the present disclosure. In particular, Figure 2A vehicle sensing environment 200 is shown, at least in part, defined by sensors and systems 116A-116K, 112 disposed in, on, and / or around the vehicle 100. Each sensor 116A-116K may include an operational detection range R and an operational detection angle α. The operational detection range R may define the effective detection limit or distance of the sensor 116A-116K. In some cases, this effective detection limit may be defined as the distance from a portion of the sensor 116A-116K (e.g., a lens, a sensing surface, etc.) to a point in space offset from the sensor 116A-116K. The effective detection limit may define a distance beyond which the sensing capabilities of the sensor 116A-116K degrade, become inoperative, or become unreliable. In some embodiments, the effective detection limit may define a distance within which the sensing capabilities of the sensor 116A-116K are able to provide accurate and / or reliable detection information. The operational detection angle α may define at least one angle between the span or horizontal and / or vertical limits of the sensor 116A-116K. As can be appreciated, the operational detection limits and the operational detection angle α of the sensors 116A-116K together can define the effective detection zones 216A-216D (eg, effective detection areas and / or volumes, etc.) of the sensors 116A-116K.

[0045] In some embodiments, vehicle 100 may include a ranging and imaging system 112. Ranging and imaging system 112 may be configured to detect visual information in the environment surrounding vehicle 100. Visual information detected in the environment surrounding ranging and imaging system 112 may be processed (e.g., via one or more sensors and / or a system processor, etc.) to generate a complete 360-degree view of environment 200 surrounding the vehicle. Ranging and imaging system 112 may be configured to generate a changing 360-degree view of environment 200 in real time, for example, while vehicle 100 is driving. In some cases, ranging and imaging system 112 may have an effective detection limit 204, which is a certain distance outward from the center of vehicle 100 over 360 degrees. Effective detection limit 204 of ranging and imaging system 112 defines an observation zone 208 (e.g., an area and / or volume, etc.) around vehicle 100. Any objects outside observation zone 208 are in an undetected zone 212 and will not be detected by ranging and imaging system 112 of vehicle 100.

[0046] Sensor data and information may be collected by one or more sensors or systems 116A to 116K, 112 of the vehicle 100 monitoring the vehicle sensing environment 200. This information may be processed (e.g., via a processor, a computer vision system, etc.) to determine targets (e.g., objects, signs, people, markings, roads, road conditions, etc.) within one or more detection zones 208, 216A to 216D associated with the vehicle sensing environment 200. In some cases, information from multiple sensors 116A to 116K may be processed to form composite sensor detection information. For example, a first sensor 116A and a second sensor 116F may correspond to a first camera 116A and a second camera 116F aimed in the forward direction of travel of the vehicle 100. In this example, the images collected by the cameras 116A and 116F may be combined to form stereoscopic image information. This composite information may improve the capabilities of individual sensors in the one or more sensors 116A to 116K by, for example, increasing the ability to determine depth associated with targets in the one or more detection zones 208, 216A to 216D. Similar image data may be collected by a rearview camera (eg, sensors 116G, 116H) of the vehicle 100 aimed in the rearward direction of travel.

[0047] In some embodiments, multiple sensors 116A to 116K can be effectively joined to increase the sensing area and provide increased sensing coverage. For example, multiple radar sensors 116B arranged on the front 110 of the vehicle can be joined to provide a coverage area 216B that spans the entire front 110 of the vehicle. In some cases, multiple radar sensors 116B can cover a detection area 216B that includes one or more other sensor detection areas 216A. These overlapping detection areas can provide redundant sensing, enhanced sensing, and / or provide more sensing details within a specific portion (e.g., area 216A) of a larger area (e.g., area 216B). Additionally or alternatively, the sensors 116A to 116K of the vehicle 100 can be arranged to produce complete coverage via one or more sensing areas 208, 216A to 216D around the vehicle 100. In some areas, the sensing areas 216C of two or more sensors 116D, 116E can intersect at an overlap area 220. In some areas, the angles and / or detection limits of two or more sensing regions 216C, 216D (eg, of two or more sensors 116E, 116J, 116K) may meet at a virtual intersection point 224 .

[0048] The vehicle 100 may include a plurality of sensors 116E, 116G, 116H, 116J, 116K disposed near the rear 120 of the vehicle 100. These sensors may include, but are not limited to, imaging sensors, cameras, IR, radio object detection and ranging sensors, radar, RF, ultrasonic sensors, and / or other object detection sensors. These sensors 116E, 116G, 116H, 116J, 116K may, among other things, detect objects near or approaching the rear of the vehicle 100. For example, another vehicle approaching the rear 120 of the vehicle 100 may be detected by one or more of the ranging and imaging system 112, the rearview cameras 116G, 116H, and / or the rear-facing radar sensors 116J, 116K. As described above, the images from the rearview cameras 116G, 116H may be processed to generate a stereoscopic view of objects visible to both cameras 116G, 116H (e.g., to provide depth associated with an object or environment, etc.). As another example, the vehicle 100 may be driving and one or more of the ranging and imaging system 112, the front-facing cameras 116A, 116F, the front-facing radar sensor 116B, and / or the ultrasonic sensor 116C may detect an object in front of the vehicle 100. This approach can provide critical sensor information to the vehicle control system in at least one of the aforementioned autonomous driving levels. For example, when the vehicle 100 is driving autonomously (e.g., Level 3, Level 4, or Level 5) and detects another vehicle stopped in its path of travel, the sensor detection information can be sent to the vehicle control system of the vehicle 100 to control driving operations associated with the vehicle 100 (e.g., braking, deceleration, etc.) (in this example, slowing the vehicle 100 to avoid colliding with the other stopped vehicle). As yet another example, the vehicle 100 may be operating and one or more of the ranging and imaging system 112 and / or the side-facing sensors 116D, 116E (e.g., radar, ultrasonic, camera, a combination thereof, and / or other types of sensors) may detect an object located to the side of the vehicle 100. It should be understood that the sensors 116A-116K can detect objects at both the sides 160 and the front 110 of the vehicle 100 (e.g., disposed at a diagonal angle to the centerline of the vehicle 100 traveling from the front 110 of the vehicle 100 to the rear 120 of the vehicle). Additionally or alternatively, the sensors 116A-116K can detect objects at both the sides 160 and the rear 120 of the vehicle 100, or both (e.g., disposed at a diagonal angle to the centerline of the vehicle 100).

[0049] Figure 33 is a block diagram of an embodiment of a communication environment 300 for a vehicle 100 according to an embodiment of the present disclosure. The communication system 300 may include one or more vehicle driving sensors and systems 304, a sensor processor 340, a sensor data storage 344, a vehicle control system 348, a communication subsystem 350, control data 364, a computing device 368, a sensor cleaning system 370, a display device 372, and other components 374 that may be associated with the vehicle 100. These associated components may be electrically and / or communicatively coupled to each other via at least one bus 360. In some embodiments, one or more of the associated components may send and / or receive signals to at least one of a navigation source 356A, a control source 356B, or some other entity 356N via a communication network 352.

[0050] According to at least some embodiments of the present disclosure, the communication network 352 may include any type of known communication medium or collection of communication media and may use any type of protocol, such as SIP, TCP / IP, SNA, IPX, AppleTalk, etc., to transmit messages between endpoints. The communication network 352 may include wired and / or wireless communication technologies. The Internet is an example of a communication network 352, which constitutes an Internet Protocol (IP) network, which is composed of many computers, computing networks, and other communication devices located around the world, which are connected by many telephone systems and other means. Other examples of the communication network 104 include, but are not limited to, standard plain old telephone systems (POTS), integrated services digital networks (ISDN), public switched telephone networks (PSTN), local area networks (LANs) such as Ethernet, token ring networks, and / or the like, wide area networks (WANs), including but not limited to virtual private networks ("VPNs"); the Internet, intranets, extranets, cellular networks, infrared networks; wireless networks (e.g., those implemented in the IEEE 802.9 or 802.11 protocol suites, as known in the art) The communication network 352 may include any type of network operating under any of the IEEE 802.11a and 802.11b protocols and / or any other wireless protocols), as well as any other type of packet-switched or circuit-switched network known in the art and / or any combination of these and / or other networks. Additionally, it will be appreciated that the communication network 352 need not be limited to any one type of network, but may include many different networks and / or network types. The communication network 352 may include a number of different communication media, such as coaxial cables, copper cables / wires, fiber optic cables, antennas for transmitting / receiving wireless messages, and combinations thereof.

[0051] The driving vehicle sensors and systems 304 may include at least one navigation sensor or system 308 (e.g., a global positioning system (GPS) or the like), an orientation sensor or system 312, a range sensor or system 316, a radar sensor or system 324, an ultrasonic sensor or system 328, a camera sensor or system 332, an infrared (IR) sensor or system 336, and / or other sensors or systems 338. These driving vehicle sensors and systems 304 may be combined with Figure 1 and Figure 2 The sensors and systems 116A to 116K, 112 are described as being similar, if not identical.

[0052] Navigation sensor 308 may include one or more sensors having a receiver and an antenna configured to utilize a satellite-based navigation system including a network of navigation satellites capable of providing geographic location and time information to at least one component of vehicle 100. Examples of navigation sensor 308 described herein may include, but are not limited to, at least one of the following: GLO TM Series of GPS and GLONASS combination sensors, GPS 15x TM Series sensors, GPS 16x TM A series of sensors with high sensitivity receivers and antennas, GPS 18xOEM series high-sensitivity GPS sensors, Dewetron DEWE-VGPS series GPS sensors, GlobalSat 1-Hz series GPS sensors, other industry equivalent navigation sensors and / or systems, and may use any known or future developed standards and / or architectures to perform navigation and / or geolocation functions.

[0053] Orientation sensor 312 may include one or more sensors configured to determine the orientation of vehicle 100 relative to at least one reference point. In some embodiments, orientation sensor 312 may include at least one pressure transducer, stress / strain gauge, accelerometer, gyroscope, and / or geomagnetic sensor. Examples of the navigation sensors 308 described herein may include, but are not limited to, at least one of the following: Bosch Sensortec BMX 160 series low-power absolute orientation sensor, Bosch Sensortec BMX055 9-axis sensor, Bosch Sensortec BMI055 6-axis inertial sensor, Bosch Sensortec BMI160 6-axis inertial sensor, Bosch Sensortec BMF055 9-axis inertial sensor (accelerometer, gyroscope, and magnetometer) with integrated Cortex M0+ microcontroller, Bosch Sensortec BMP280 absolute barometric pressure sensor, Infineon TLV493D-A1B6 3D magnetic sensor, Infineon TLI493D-W1B6 3D magnetic sensor, Infineon TL series 3D magnetic sensor, Murata Electronics SCC2000 series combined gyroscope sensor and accelerometer, Murata Electronics The SCC1300 series combines gyroscope sensors with accelerometers, other industry equivalent orientation sensors, and / or systems that may use any known or future developed standards and / or architectures to perform orientation detection and / or determination functions.

[0054] The ranging sensor and / or system 316 may include one or more components configured to determine the change in the position of the vehicle 100 over time. In some embodiments, the ranging system 316 may utilize data from one or more other sensors and / or systems 304 to determine the position (e.g., distance, location, etc.) of the vehicle 100 relative to a previously measured position of the vehicle 100. Additionally or alternatively, the ranging sensor 316 may include one or more encoders, Hall effect sensors, and / or other measurement sensors / devices configured to measure wheel speed, rotation, and / or revolutions over time. Examples of ranging sensors / systems 316 as described herein may include, but are not limited to, at least one of the following: Infineon TLE4924 / 26 / 27 / 28C high performance speed sensors, Infineon TL4941plusC(B) single chip differential Hall effect wheel speed sensors, Infineon TL5041plusC giant magnetoresistance (GMR) effect sensors, Infineon TL series magnetic sensors, EPC 25SP model Accu-CoderPro TMIncremental shaft encoders, EPC 30M compact incremental encoders with advanced magnetic sensing and signal processing technology, EPC 925 absolute shaft encoders, EPC 958 absolute shaft encoders, EPC MA36S / MA63S / SA36S absolute shaft encoders, Dynapar TM F18 commutation optical encoder, Dynapar TM HS35R Series phased array encoder sensors, other industrial equivalent distance measuring sensors and / or systems, and may use any known or future developed standards and / or architectures to perform position change detection and / or determine a change in functionality.

[0055] The radar sensor 324 may include one or more radio components configured to detect objects / targets in the environment of the vehicle 100. In some embodiments, the radar sensor 324 may determine the distance, position, and / or motion vector (e.g., angle, velocity, etc.) associated with the target over time. The radar sensor 324 may include a transmitter configured to generate and transmit electromagnetic waves (e.g., radio, microwave, etc.), and a receiver configured to detect the returned electromagnetic waves. In some embodiments, the radar sensor 324 may include at least one processor configured to interpret the returned electromagnetic waves and determine the position characteristics of the target. Examples of the radar sensor 324 as described herein may include, but are not limited to, at least one of the following: an Infineon RASIC TM The RTN7735PL transmitter and RRN7745PL / 46PL receiver sensor, Autoliv ASP vehicle radar sensor, Delphi L2C0051TR 77GHz ESR electronically scanned radar sensor, Fujitsu Ten Ltd. automotive compact 77GHz 3D electronically scanned millimeter wave radar sensor, other industrial equivalent radar sensors and / or systems, and can perform radio target and / or obstacle detection in the environment surrounding the vehicle 100 using any known or future developed standards and / or architectures.

[0056] The ultrasonic sensor 328 may include one or more components configured to detect objects / targets in the environment of the vehicle 100. In some embodiments, the ultrasonic sensor 328 may determine the distance, position, and / or motion vector (e.g., angle, velocity, etc.) associated with the target over time. The ultrasonic sensor 328 may include an ultrasonic transmitter and receiver, or transceiver, configured to generate and transmit ultrasonic waves and interpret the return echoes of those waves. In some embodiments, the ultrasonic sensor 328 may include at least one processor configured to interpret the returning ultrasonic waves and determine the positional characteristics of the target. Examples of the ultrasonic sensor 328 described herein may include, but are not limited to, at least one of the following: Texas Instruments TIDA-00151 Automotive Ultrasonic Sensor Interface IC sensor, MB8450 ultrasonic proximity sensor, ParkSonar TM -EZ ultrasonic proximity sensor, Murata Electronics MA40H1S-R open structure ultrasonic sensor, Murata Electronics MA40S4R / S open structure ultrasonic sensor, Murata Electronics MA58MF14-7N waterproof ultrasonic sensor, other industry equivalent ultrasonic sensors and / or systems, and may use any known or future developed standards and / or architectures to perform ultrasonic detection of targets and / or obstacles in the environment surrounding the vehicle 100.

[0057] The camera sensor 332 may include one or more components configured to detect image information associated with the environment of the vehicle 100. In some embodiments, the camera sensor 332 may include a lens, a filter, an image sensor, and / or a digital image processor. One aspect of the present disclosure is that multiple camera sensors 332 may be used together to produce stereo images, thereby providing depth measurements. Examples of camera sensors 332 as described herein may include, but are not limited to, at least one of the following: MT9V024 global shutter VGA GS CMOS image sensor, Teledyne DALSAFalcon2 camera sensor, CMOSIS CMV50000 high-speed CMOS image sensor, other industrial equivalent camera sensors and / or systems, and may use any known or future developed standards and / or architectures to perform visual object and / or obstacle detection in the environment surrounding the vehicle 100 .

[0058] The infrared (IR) sensor 336 may include one or more components configured to detect image information associated with the environment of the vehicle 100. The IR sensor 336 may be configured to detect targets in low light, dark, or poorly lit environments. The IR sensor 336 may include an IR light emitting element (e.g., an IR light emitting diode (LED), etc.) and an IR photodiode. In some embodiments, the IR photodiode may be configured to detect returned IR light of the same or approximately the same wavelength as the wavelength emitted by the IR light emitting element. In some embodiments, the IR sensor 336 may include at least one processor configured to interpret the returned IR light and determine positional characteristics of the target. The IR sensor 336 may be configured to detect and / or measure a temperature associated with a target (e.g., an object, a pedestrian, another vehicle, etc.). Examples of the IR sensor 336 as described herein may include, but are not limited to, at least one of the following: a photodiode lead salt IR array sensor, a photodiode OD-850 near infrared LED sensor, a photodiode SA / SHA727 steady state IR emitter and IR detector, LS microbolometer sensor, TacFLIR 380-HD InSb MWIR FPA with HD MWIR thermal sensor, VOx 640x480 pixel detector sensor, DelphiIR sensor, other industry equivalent IR sensor and / or system, and performs IR visual target and / or obstacle detection in the environment around the vehicle 100 using any known or future developed standard and / or architecture.

[0059] In some embodiments, the driving vehicle sensors and system 304 may include other sensors 338 and / or combinations of the aforementioned sensors 308 through 336. Additionally or alternatively, one or more of the aforementioned sensors 308 through 336 may include one or more processors configured to process and / or interpret signals detected by one or more of the sensors 308 through 336. In some embodiments, processing of at least some sensor information provided by the vehicle sensors and system 304 may be handled by at least one sensor processor 340. Raw and / or processed sensor data may be stored in a sensor data memory 344 storage medium. In some embodiments, the sensor data memory 344 may store instructions used by the sensor processor 340 to process the sensor information provided by the sensors and system 304. In any case, the sensor data memory 344 may be a disk drive, an optical storage device, a solid-state storage device such as a random access memory ("RAM") and / or a read-only memory ("ROM"), which may be programmable, flash-updatable, and / or the like.

[0060] The vehicle control system 348 can receive processed sensor information from the sensor processor 340 and determine whether to control certain aspects of the vehicle 100. Controlling certain aspects of the vehicle 100 may include presenting information via one or more display devices 372 associated with the vehicle, sending commands to one or more computing devices 368 associated with the vehicle, and / or controlling the driving operation of the vehicle. In some embodiments, the vehicle control system 348 may correspond to one or more computing systems that control the driving operation of the vehicle 100 according to the aforementioned driving autonomy level. In one embodiment, the vehicle control system 348 can manipulate the speed of the vehicle 100 by controlling output signals to the vehicle's accelerometer and / or braking system. In this example, the vehicle control system 348 can receive sensor data describing the environment surrounding the vehicle 100 and, based on the received sensor data, determine whether to adjust the acceleration, power output, and / or braking of the vehicle 100. The vehicle control system 348 may also control the steering and / or other driving functions of the vehicle 100.

[0061] The vehicle control system 348 can communicate with the driving sensors and system 304 in real time, thereby forming a feedback loop. In particular, upon receiving sensor information describing target conditions in the environment surrounding the vehicle 100, the vehicle control system 348 can autonomously change the driving operation of the vehicle 100. The vehicle control system 348 can then receive subsequent sensor information describing any changes to the target conditions detected in the environment as a result of the changed driving operation. This continuous cycle of observation (e.g., via sensors, etc.) and action (e.g., selected control or non-control of vehicle operation, etc.) allows the vehicle 100 to operate autonomously in the environment.

[0062] In some embodiments, one or more components of the vehicle 100 (e.g., driving vehicle sensors 304, vehicle control systems 348, display devices 372, etc.) can communicate with one or more entities 356A to 356N via a communication subsystem 350 of the vehicle 100 over a communication network 352. Figure 5 Embodiments of the communication subsystem 350 are described in greater detail. For example, the navigation sensor 308 can receive global positioning, location, and / or navigation information from a navigation source 356A. In some embodiments, the navigation source 356A can be a global navigation satellite system (GNSS) similar to (if not identical to) NAVSTAR GPS, GLONASS, EU Galileo, and / or the BeiDou Navigation Satellite System (BDS), to name a few.

[0063] In some embodiments, the vehicle control system 348 can receive control information from one or more control sources 356B. The control sources 356 can provide vehicle control information, including autonomous driving control commands, vehicle operation override control commands, etc. The control sources 356 can correspond to autonomous vehicle control systems, traffic control systems, administrative control entities, and / or some other control server. One aspect of the present disclosure is that the vehicle control system 348 and / or other components of the vehicle 100 can exchange information with the control sources 356 via the communication network 352 and via the communication subsystem 350.

[0064] Information associated with controlling the driving operation of the vehicle 100 may be stored in the control data memory 364 storage medium. The control data memory 364 may store instructions, historical control information, autonomous driving control rules, etc., used by the vehicle control system 348 to control the driving operation of the vehicle 100. In some embodiments, the control data memory 364 may be a disk drive, an optical storage device, a solid-state storage device such as a random access memory ("RAM") and / or a read-only memory ("ROM"), which may be programmable, flash-updatable, and / or the like.

[0065] In addition to the mechanical components described herein, the vehicle 100 may also include multiple user interface devices. The user interface device receives human input and converts it into mechanical motion or electrical signals or stimuli. The human input can be one or more of the following: motion (e.g., body motion, body part motion in two or three-dimensional space, etc.), voice, touch, and / or physical interaction with components of the vehicle 100. In some embodiments, the human input can be configured to control one or more functions of the vehicle 100 and / or the systems of the vehicle 100 described herein. The user interface may include, but is not limited to, at least one graphical user interface of the following: a display device, a steering wheel or steering mechanism, a gear lever or button (e.g., including a parking position, a neutral position, a reverse position, and / or a drive position, etc.), an accelerator control pedal or mechanism, a brake control pedal or mechanism, a power control switch, a communication device, etc.

[0066] Figure 4One embodiment of an instrument panel 400 of a vehicle 100 is shown. The instrument panel 400 of the vehicle 100 includes a steering wheel 410, a vehicle operational display 420 (e.g., configured to present and / or display driving data such as speed, measured air drag, vehicle information, entertainment information, etc.), one or more auxiliary displays 424 (e.g., configured to present and / or display information separate from the operational display 420, entertainment applications, movies, music, etc.), a heads-up display 434 (e.g., configured to display any of the information previously described, including but not limited to, guidance information such as a route to a destination, obstacle warning information for warning of a potential collision, or some or all of the primary vehicle operational data such as speed, drag, etc.), a power management display 428 (e.g., configured to display data corresponding to the power level, backup power, charging status, etc. of the vehicle 100), and an input device 432 (e.g., a controller, touch screen, or other interface device configured to interface with one or more displays in the instrument panel or component of the vehicle 100. The input device 432 may be configured as a joystick, mouse, touchpad, tablet computer, 3D gesture capture device, etc.). In some embodiments, input device 432 may be used to manually maneuver a portion of vehicle 100 into a charging position (eg, move a charging plate to a desired separation distance, etc.).

[0067] While one or more displays of instrument panel 400 may be touchscreen displays, it should be understood that vehicle operational displays may be displays that are incapable of receiving touch input. For example, operational display 420, which spans interior centerline 404 and spans first zone 408A and second zone 408B, may be isolated from receiving touch input, particularly input from a passenger. In some cases, displays that provide vehicle operational or critical system information and interfaces may be constrained from receiving touch input and / or configured as non-touch displays. This type of configuration can prevent dangerous errors in providing touch input, which could result in an accident or unintended control.

[0068] In some embodiments, one or more displays of the instrument panel 400 can be a mobile device and / or an application resident on a mobile device such as a smartphone. Additionally or alternatively, any information described herein can be presented to one or more portions 420A to 420N of the operational display 420 or other displays 424, 428, 434. In one embodiment, one or more displays of the instrument panel 400 can be physically separated or detached from the instrument panel 400. In some cases, the detachable display can remain tethered to the instrument panel.

[0069] The portions 420A to 420N of the operational display 420 can be dynamically reconfigured and / or resized to accommodate any information display described. Additionally or alternatively, the number of portions 420A to 420N used to visually present information via the operational display 420 can be dynamically increased or decreased as needed and is not limited to the configuration shown.

[0070] Figure 5 A hardware diagram illustrating communication components that may optionally be associated with vehicle 100 according to an embodiment of the present disclosure is shown.

[0071] The communication components may include one or more wired or wireless devices, such as (one or more) transceivers and / or modems that allow communication not only between the various systems disclosed herein but also with other devices such as devices on a network and / or devices on a distributed network such as the Internet and / or in the cloud and / or with (one or more) other vehicles.

[0072] The communication subsystem 350 may also include inter-vehicle and intra-vehicle communication capabilities, such as hotspot and / or access point connections for any one or more of vehicle occupant and / or vehicle-to-vehicle communications.

[0073] In addition, although not specifically shown, the communication subsystem 350 may include one or more communication links (which may be wired or wireless) and / or communication buses (managed by the bus manager 574) including one or more of the following: CANbus, OBD-II, ARCINC 429, Byteflight, CAN (Controller Area Network), D2B (Domestic Digital Bus), FlexRay, DC-BUS, IDB-1394, IEBus, I2C, ISO 9141-1 / -2, J1708, J1587, J1850, J1939, ISO 11783, Keyword Protocol 2000, LIN (Local Interconnect Network), MOST (Media Oriented Systems Transport), Multifunction Vehicle Bus, SMARTwireX, SPI, VAN (Vehicle Area Network), etc., or generally any communication protocol and / or standard(s).

[0074] The various protocols and communications may be communicated wirelessly and / or over one or more of a transmission medium such as single wire, twisted pair, fiber optic, IEEE 1394, MIL-STD-1553, MIL-STD-1773, power line communication, etc. (all of the above standards and protocols are incorporated herein by reference in their entirety).

[0075] As discussed, the communication subsystem 350 enables communications between any inter-vehicle systems and subsystems as well as with non-collocated resources, such as those available over a network such as the Internet.

[0076] In addition to well-known components (omitted for clarity), the communication subsystem 350 includes interconnect elements, including one or more of the following: one or more antennas 504, an interleaver / deinterleaver 508, an analog front end (AFE) 512, memory / storage / cache 516, a controller / microprocessor 520, a MAC circuitry 522, a modulator / demodulator 524, an encoder / decoder 528, a plurality of connectivity managers 534, 558, 562, 566, a GPU 540, an accelerator 544, a multiplexer / demultiplexer 552, a transmitter 570, a receiver 572, and a radio 578 component, such as a Wi-Fi module 580, Wi-Fi / BT MAC module 584, transmitter 588, and receiver 592. The various elements in device 350 are connected via one or more links / buses 5 (also not shown for clarity).

[0077] The device 350 may have one or more antennas 504 for wireless communications, such as multiple-input multiple-output (MIMO) communications, multi-user multiple-input multiple-output (MU-MIMO) communications, and multiple-user multiple-input multiple-output (MU-MIMO) communications. LTE, 4G, 5G, near field communication (NFC), etc., and is generally used for any type of wireless communication. (One or more) antennas 504 may include, but are not limited to, one or more of the following: a directional antenna, an omnidirectional antenna, a monopole antenna, a patch antenna, a loop antenna, a microstrip antenna, a dipole antenna, and any other antenna (one or more) suitable for communication transmission / reception. In an exemplary embodiment, transmission / reception using MIMO may require specific antenna spacing. In another exemplary embodiment, MIMO transmission / reception can achieve spatial diversity, thereby allowing different channel characteristics at each antenna. In yet another embodiment, MIMO transmission / reception can be used to allocate resources to multiple users, for example, within the vehicle 100 and / or in another vehicle.

[0078] The antenna(s) 504 typically interact with an analog front end (AFE) 512, which is necessary to properly process received modulated signals and perform signal conditioning on transmitted signals. The AFE 512 can be functionally located between the antenna and the digital baseband system to convert analog signals to digital signals for processing and vice versa.

[0079] Subsystem 350 may also include a controller / microprocessor 520 and a memory / storage / cache 516. Subsystem 350 may interact with memory / storage / cache 516, which may store information and operations necessary for configuration and transmitting or receiving information as described herein. Memory / storage / cache 516 may also be used in conjunction with controller / microprocessor 520 to execute application programming or instructions and may be used for temporary or long-term storage of program instructions and / or data. By way of example, memory / storage / cache 520 may include a computer-readable device, RAM, ROM, DRAM, SDRAM, and / or other storage device(s) and media.

[0080] The controller / microprocessor 520 may include a general-purpose programmable processor or controller for executing application programming or instructions associated with the subsystem 350. In addition, the controller / microprocessor 520 may perform operations for configuration and transmitting / receiving information, as described herein. The controller / microprocessor 520 may include multiple processor cores and / or implement multiple virtual processors. Alternatively, the controller / microprocessor 520 may include multiple physical processors. As an example, the controller / microprocessor 520 may include a specially configured application-specific integrated circuit (ASIC) or other integrated circuit, a digital signal processor (one or more), a controller, a hard-wired electronic or logic circuit, a programmable logic device or gate array, a special-purpose computer, etc.

[0081] Subsystem 350 may also include a transmitter 570 and a receiver 572, which may use one or more antennas 504 and / or links / buses to transmit and receive signals to and from other devices, subsystems, and / or other destinations, respectively. Subsystem 350 circuitry includes medium access control or MAC circuitry 522. MAC circuitry 522 provides for controlling access to the wireless medium. In an exemplary embodiment, MAC circuitry 522 may be arranged to contend for the wireless medium and configure frames or packets transmitted over the wired / wireless medium.

[0082] Subsystem 350 may also optionally include a security module (not shown). This security module may contain information about, but not limited to, security parameters required to connect the device to one or more other devices or other available network(s), and may include WEP or WPA / WPA-2 (optionally with AES and / or TKIP) security access keys, network keys, etc. A WEP security access key is a security password used by Wi-Fi networks. Knowing this code enables a wireless device to exchange information with an access point and / or another device. Information exchange may be performed via coded messages, where a WEP access code is typically selected by a network administrator. WPA is an additional security standard also used in conjunction with network connections, where encryption is stronger than WEP.

[0083] In some embodiments, the communication subsystem 350 also includes a GPU 540, an accelerator 544, a Wi-Fi / BT / BLE PHY module 580 and a Wi-Fi / BT / BLE MAC module 584, as well as a wireless transmitter 588 and a receiver 592. In some embodiments, the GPU 540 may be a graphics processing unit or a visual processing unit that includes at least one circuit and / or chip that manipulates and modifies memory to accelerate the creation of images in a frame buffer for output to at least one display device. The GPU 540 may include one or more of the following: a display device connection port, a printed circuit board (PCB), a GPU chip, a metal oxide semiconductor field effect transistor (MOSFET), memory (e.g., single data rate random access memory (SDRAM), double data rate random access memory (DDR) RAM, etc., and / or combinations thereof), auxiliary processing chips (e.g., to handle video output capabilities, processing and / or other functions beyond the GPU chip), capacitors, a heat sink, a temperature control or cooling fan, motherboard connections, shielding, etc.

[0084] Various connectivity managers 534, 558, 562, 566 manage and / or coordinate communications between subsystem 350 and one or more systems disclosed herein and one or more other devices / systems. Connectivity managers 534, 558, 562, 566 include a charging connectivity manager 534, a vehicle database connectivity manager 558, a teleoperation system connectivity manager 562, and a sensor connectivity manager 566.

[0085] The charging connectivity manager 534 can not only coordinate the physical connectivity between the vehicle 100 and the charging device / vehicle, but can also communicate with one or more of a power management controller, one or more third parties, and optionally, a billing system(s). As an example, the vehicle 100 can establish communication with the charging device / vehicle to do one or more of the following: coordinate the interconnectivity between the two (e.g., by spatially aligning a charging receptacle on the vehicle with a charger on the charging vehicle), and optionally share navigation information. Once charging is complete, the amount of charge provided can be tracked and optionally forwarded to, for example, a third party for billing. In addition to being able to manage connectivity for exchanging power, the charging connectivity manager 534 can also communicate information, such as billing information, to the charging vehicle and / or third party. This billing information can include, for example, the vehicle's owner, the vehicle's driver / occupant(s), company information, or generally any information that can be used to bill the appropriate entity for the power received.

[0086] The vehicle database connectivity manager 558 allows the subsystems to receive and / or share information stored in the vehicle database. This information can be shared with other vehicle components / subsystems and / or other entities such as third parties and / or charging systems. This information can also be shared with one or more vehicle occupant devices, such as apps on a mobile device used by the driver to track information about the vehicle 100 and / or dealership or service / maintenance provider. In general, any information stored in the vehicle database can optionally be shared with any one or more other devices, optionally subject to any privacy or confidentiality constraints.

[0087] Teleoperation system connectivity manager 562 facilitates communications between vehicle 100 and any one or more autonomous vehicle systems. These communications may include one or more of: navigation information, vehicle information, other vehicle information, weather information, occupant information, or generally any information related to the remote operation of vehicle 100.

[0088] The sensor connectivity manager 566 facilitates communication between any one or more vehicle sensors (e.g., driving vehicle sensors and systems 304, etc.) and any one or more other vehicle systems. The sensor connectivity manager 566 can also facilitate communication between any one or more sensors and / or vehicle systems and any other destination (such as a service company, an application, or generally any destination requiring sensor data).

[0089] According to an exemplary embodiment, any of the communications discussed herein can be transmitted via the conductor(s) used for charging. One exemplary protocol that can be used for these communications is power line communication (PLC). PLC is a communication protocol that uses wires to carry both data and alternating current (AC) power transmission or distribution. It is also known as power line carrier, power line digital subscriber line (PDSL), power communication, power line communication, or power line networking (PLN). For DC environments in vehicles, PLC can be used in conjunction with the CAN bus, the LIN bus over power line (DC-LIN), and the DC-BUS.

[0090] The communication subsystem may also optionally manage one or more identifiers associated with the vehicle and one or more of the systems or subsystems or components therein, such as IP (Internet Protocol) addresses(es). These identifiers may be used in conjunction with any one or more connectivity managers discussed herein.

[0091] Figure 6 A block diagram of a computing environment 600 that can be used as a server, user computer, or other system as provided and described herein is shown. The computing environment 600 includes one or more user computers or computing devices, such as a vehicle computing device 604, a communication device 608, and / or more devices 612. The computing devices 604, 608, 612 may include general-purpose personal computers (including, by way of example only, Microsoft Windows running various versions of Microsoft and / or Apple's operating systems) and / or running various commercially available or a workstation computer running any of a UNIX-like operating system. These computing devices 604, 608, 612 may also have any of a variety of applications, including, for example, database client and / or server applications and web browser applications. Alternatively, the computing devices 604, 608, 612 may be any other electronic device capable of communicating via the network 352 and / or displaying and navigating web pages or other types of electronic documents, such as a thin client computer, an Internet-enabled mobile phone, and / or a personal digital assistant. Although an exemplary computing environment 600 is shown with two computing devices, any number of user computers or computing devices may be supported.

[0092] The computing environment 600 may also include one or more servers 614, 616. In this example, the server 614 is shown as a web server and the server 616 is shown as an application server. The web server 614 can be used to process requests for web pages or other electronic documents from the computing devices 604, 608, 612. The web server 614 can run an operating system, including any of those discussed above and any commercially available server operating system. The web server 614 can also run a variety of server applications, including SIP (Session Initiation Protocol) servers, HTTP servers, FTP servers, CGI servers, database servers, Java servers, etc. In some cases, the web server 614 can publish available operations as one or more network services.

[0093] The computing environment 600 may also include one or more file and / or application servers 616, which, in addition to an operating system, may include one or more applications accessible by clients running on one or more of the computing devices 604, 608, 612. The server(s) 616 and / or 614 may be one or more general-purpose computers capable of executing programs or scripts in response to the computing devices 604, 608, 612. As an example, the servers 616, 614 may execute one or more web applications. The web applications may be implemented in a language such as Java™, C, or C++ and / or any scripting language such as Perl, Python or TCL, as well as any combination of programming / scripting languages. The application server(s) 616 may also include a database server, including but not limited to a database server that can be accessed from These data block servers, such as those commercially available, can process requests from database clients running on computing devices 604 , 608 , 612 .

[0094] Web pages created by servers 614 and / or 616 may be forwarded to computing devices 604, 608, 612 via web (file) servers 614, 616. Similarly, web server 614 may be capable of receiving web page requests, web service calls, and / or input data from computing devices 604, 608, 612 (e.g., user computers, etc.) and may forward the web page requests and / or input data to web (application) server 616. In further embodiments, server 616 may function as a file server. Although for ease of description, Figure 6A separate network server 614 and file / application server 616 are shown, but those skilled in the art will recognize that the functions described with respect to servers 614, 616 may be performed by a single server and / or multiple dedicated servers, depending on the requirements and parameters of the specific implementation. The computer systems 604, 608, 612, the network (file) server 614 and / or the network (application) server 616 may be used as Figures 1 to 6 The system, equipment or component described in.

[0095] The computing environment 600 may also include a database 618. The database 618 may reside in a variety of locations. As an example, the database 618 may reside on a storage medium that is local to (and / or resides in) one or more of the computers 604, 608, 612, 614, 616. Alternatively, the database may be remote from any or all of the computers 604, 608, 612, 614, 616 and in communication with one or more of these computers (e.g., via the network 352). The database 618 may reside in a storage area network ("SAN") familiar to those skilled in the art. Similarly, any necessary files for performing the functions attributed to the computers 604, 608, 612, 614, 616 may be stored locally on the respective computers and / or remotely, as appropriate. The database 618 may be a relational database, such as Oracle Database 11g, suitable for storing, updating, and retrieving data in response to SQL-formatted commands.

[0096] Figure 7 One embodiment of a computer system 700 is shown on which the above-described servers, user computers, computing devices, or other systems or components may be deployed or executed. The computer system 700 is shown as including hardware elements that may be electrically coupled via a bus 704. The hardware elements may include one or more central processing units (CPUs) 708; one or more input devices 712 (e.g., a mouse, keyboard, etc.); and one or more output devices 716 (e.g., a display device, a printer, etc.). The computer system 700 may also include one or more storage devices 720. By way of example, the storage device(s) 720 may be a disk drive, an optical storage device, a solid-state storage device such as a random access memory ("RAM") and / or a read-only memory ("ROM"), which may be programmable, flash-updatable, and / or the like.

[0097] The computer system 700 may further include a computer-readable storage medium reader 724; a communication system 728 (e.g., a modem, a network card (wireless or wired), an infrared communication device, etc.); and a working memory 736, which may include the RAM and ROM devices described above. The computer system 700 may also include a processing acceleration unit 732, which may include a DSP, a special-purpose processor, and / or the like.

[0098] Computer-readable storage media reader 724 may also be connected to computer-readable storage media, which together (and optionally, in conjunction with storage device(s) 720) comprehensively represent remote, local, fixed, and / or removable storage devices plus storage media for temporarily and / or more permanently containing computer-readable information. Communication system 728 may allow data to be exchanged with the network and / or any other computer described above with respect to the computer environment described herein. Furthermore, as disclosed herein, the term "storage media" may refer to one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, magnetic core memory, magnetic disk storage media, optical storage media, flash memory devices, and / or other machine-readable media for storing information.

[0099] The computer system 700 may also include software elements, shown as currently located within working memory 736, including an operating system 740 and / or other code 744. It should be understood that alternative embodiments of the computer system 700 may have many variations different from those described above. For example, custom hardware and / or specific elements that can be implemented in hardware, software (including portable software, such as applets), or both may also be used. In addition, connections to other computing devices, such as network input / output devices, may be employed.

[0100] Examples of the processors 340, 708 described herein may include, but are not limited to, at least one of the following: 800 and 801, with 4G LTE integration and 64-bit operation 620 and 615, with 64-bit architecture A7 processor, M7 motion coprocessor, series, CoreTM series processors, series processors, AtomTM series processors, Intel series processors, i5-4670K and i7-4770K 22nm Haswell, i5-3570K 22nm Ivy Bridge, FXTM series processors, FX-4300, FX-6300 and FX-8350 32nm Vishera, Kaveri processor, Texas Jacinto C6000TM automotive infotainment processor, Texas OMAPTM automotive-grade mobile processors, CortexTM-M processor, Cortex-A and ARM926EJ-STM processors, other industry equivalent processors; and may use any known or future developed standard, instruction set, library, and / or architecture to perform computing functions.

[0101] Figures 8A to 8C Graphical representations 801 to 803 of sensor information detected by different sensors of the vehicle 100 (e.g., a first sensor, a second sensor, a third sensor, etc.) over time are shown. As provided herein, the term sensor may be used to refer to any sensor of the vehicle 100 (e.g., driving vehicle sensors and systems 304, ranging and imaging system 112, sensors 116A to 116K, etc.) or combination of sensors. Figures 8A to 8C As shown, each graphical representation 801 to 803 includes a chart having an origin 804 (or first detection point), an endpoint 806 (or last detection point), an output (or many outputs) 808A to 808C over time (e.g., a waveform), a vertical axis 812 representing the intensity associated with the detected sensor information, and a horizontal axis 816 representing the time associated with each output.

[0102] First detection point 804 may correspond to a point in time at which a sensor begins detecting (e.g., providing detection information, etc.). In some cases, first detection point 804 may correspond to the first point in the analyzed portion of the detected sensor information. In this example, first detection point 804 may not be a point in time at which a sensor begins detecting, but rather may be a point defined or automatically selected by sensor processor 340 for determining the sensor's ability to measure the environment of vehicle 100.

[0103] The last detection point 806 may correspond to a point in time at which the sensor ceased detection (e.g., ceased providing detection information). In some cases, the last detection point 806 may correspond to the last point in the analyzed portion of the detected sensor information. In this example, the last detection point 806 may not be the point in time at which the sensor ceased detection, but rather may be the last point defined or automatically selected by the sensor processor 340 for determining the sensor's ability to measure the vehicle 100's environment.

[0104] Outputs 808A-808C over time may correspond to the intensity or measurement units of sensor information detected over time (e.g., range, distance, speed, time of flight, etc.). In some embodiments, outputs 808A-808C over time may include multiple outputs at times within the first detection point 804 and the last detection point 806. As the sensing environment changes over time (e.g., as the target moves relative to the sensor), the intensity of the detected sensor information may change. For example, as vehicle 100 approaches an intersection and a target in front 110 of vehicle 100, radar sensor 116B may determine the distance to the target by recording the intensity or measurement units at multiple different times while vehicle 100 is operating. In this example, as vehicle 100 approaches the target, the distance to the target decreases over time. If the output intensity or measurement units are time of flight associated with the emission and sensed values, graphical representations 801-803 would indicate that the output at a first time is higher (e.g., indicating that the sensor emission took a long time to return to the sensor) than the output at a second or subsequent time (e.g., when vehicle 100 and the sensor are closer to the target). Where the output strength or unit of measurement is the strength of the returned sensor signal, graphical representations 801 to 803 would indicate that the output at a first moment in time is shorter (e.g., indicating that the sensor signal strength measured from the returned signal has decreased by an amount from the transmitted signal strength because the target was further away at the first moment in time) than the height of the output at a second or subsequent moment in time (e.g., when the vehicle 100 and the sensor are closer to the target and the sensor signal strength measured from the returned signal is closer to the transmitted signal strength). In any case, changes in the range to the target can be shown as sensor outputs that change in strength in association with outputs 808A to 808C over time.

[0105] Figure 8A A graphical representation 801 of the output 808A of a first sensor of the vehicle 100 over time is shown. Figure 8A As shown, the output 808A over time may include different levels of intensity or measurement units along the vertical axis 812 for one or more moments in time in the horizontal axis 816 . Figure 8B A graphical representation 802 of the output 808B of a second sensor of the vehicle 100 over time is shown. Figures 8A to 8BAs shown, the outputs 808A, 808B of the first sensor and the second sensor over time are substantially similar (if not identical). In some embodiments, the first sensor and the second sensor may be the same type of sensor (e.g., imaging and ranging system 112, sensors 116A to 116K, radar 324, ultrasound 328, camera 332, infrared (IR) 336, and / or other sensors or systems 338). For example, the first sensor and the second sensor may be radar sensor 324. Whenever the outputs 808A, 808B over time are substantially similar between sensors of the same type and / or location on vehicle 100, it can be determined that the sensors have the same operating characteristics or capabilities. In one embodiment, this similarity may indicate that the sensors are not obscured by any objects.

[0106] Figure 8C A graphical representation 803 of the output 808C of a third sensor of the vehicle 100 over time is shown. In some embodiments, one or more of the first and / or second sensors may be different from a third type of sensor of the vehicle 100. For example, the third sensor of the same type may be an imaging sensor 116A, 116F, 332, 336 (e.g., a camera, etc.) of the vehicle 100. Although the output 808C over time is schematically represented as a 2D waveform, it should be understood that visual data need not be so limited. In any case, the output 808C of the third sensor output over time is shown as changing over time (e.g., changing in intensity and / or measurement units, as described above).

[0107] Figure 9 A graphical representation 901 is shown showing an example of overlapping outputs 808B, 808C, 908 of three sensors of the vehicle 100 over time during the same time period. As mentioned above, the methods and systems described herein can utilize Figures 8A to 9 Similar (if not identical) sensor detection information as shown in (e.g., associated with one or more sensors to detect sensor occlusion, including objects on, in contact with, or as part of a sensor surface. For example, Figure 9 The graphical representation 901 shows an example of the output 1008 of a first sensor that is blocked over time. The output 808B of a similar sensor (e.g., a second sensor) over the same time period can be determined. Figure 9 The first sensor in FIG is blocked. The output of the second sensor over time 908B is shown as a dashed line, which overlaps with the output of the blocked first sensor over time 908. In this exemplary graphical representation 901, the difference between the outputs over time 908, 808B can be determined or observed in the inconsistent area 902 of the graph. Figure 9As shown, the first sensor and the second sensor provide similar (if not identical) detected sensor information for a first portion of the time period (e.g., similar outputs over time 908, 808B over the time portion). However, for the same remaining portion of the time period, the first sensor provides different detected sensor information (e.g., output over time 908) over the inconsistency region 902 when compared to the output over time 808B of the second sensor.

[0108] In some embodiments, the difference between the outputs 908, 808B over time can indicate that at least one sensor of the vehicle 100 is blocked and / or non-functional. One aspect of the present disclosure is that one or more specific sensors can be identified as the blocked sensor or sensors. This identification can include reference to signal characteristics, strengths, measurements, etc. associated with the sensors of the vehicle 100. In one embodiment, the sensor processor 340 can determine that the detected sensor information from one of the sensors has not changed based on a predicted or preset threshold. The lack of such a change in the detected sensor information can indicate that the sensor was completely blocked or obscured for a period of time or for a period of time.

[0109] In some embodiments, the sensor processor 340 may determine that the detected sensor information from one of the sensors has not changed when compared to the detected sensor information of at least one other sensor (e.g., a different type of sensor, a third sensor, etc.), based on a change characteristic of the at least one other sensor. For example, a third sensor providing output 808C over time may be oriented on the vehicle 100 at a similar location as the first and / or second sensors providing outputs 908 and 808B over time (e.g., sensing movement, objects, or changes in the environment surrounding the vehicle 100, at a particular side, area, or zone of the vehicle 100, etc.). In this example, the sensor processor 340 may determine that the second sensor's output 808B over time and the third sensor's output 808C over time, while not necessarily identical, indicate related changes in the sensed environment over time. Additionally or alternatively, the sensor processor 340 may determine that the first sensor's output 1008 over time and the third sensor's output 808C over time have no relationship, at least for a portion of time (e.g., the inconsistency region 902). In some embodiments, the sensor processor 340 may use this relationship in change information to uniquely identify an occluded sensor from among the one or more sensors.

[0110] Figure 10A graphical representation 1001 is shown showing an example of output 1008, 808B of multiple sensors of a vehicle 100 over the same time period. As described above, the methods and systems described herein can utilize Figures 8A to 10 , similar (if not identical) sensor detection information as shown in (e.g., associated with one or more sensors to detect sensor occlusion, including objects on, in contact with, or as part of the sensor surface. In one example, a sensor may be obscured by dust or debris and may provide information that is impaired in some way (e.g., not as accurate as a clean sensor, the sensor's range is limited, etc.). For example, Figure 10 The graphical representation 1001 shows an example of an output 1008 of a first sensor that is obscured over time. The output 808B of a similar sensor (e.g., a second sensor) over time can be determined by comparing the output 1008 of the first sensor over time with the output 808B of the first sensor over time during the same time period. Figure 10 The first sensor in FIG is obscured. The output of the second sensor over time 808B is shown as a dashed line, which overlaps with the output of the obscured first sensor over time 1008. In this exemplary graphical representation 1001, the difference between the outputs 1008, 808B over time can be determined or observed as an inconsistent area 1002 or measurement change in the graph. Figure 10 As shown, the first sensor and the second sensor provide similar (if not identical) detected sensor information (e.g., similar outputs 1008, 808B over time) over the time period. However, the first sensor provides a scaled (e.g., reduced, zoomed out, less accurate, etc.) measurement variation 1002 of the detected sensor information (e.g., output 1008 over time) over the time period when compared to the output 808B of the second sensor over time over the same time period. In some embodiments, the measurement variation 1002 can be attributed to the proximity of the target to the particular sensor, but when the measurement variation 1002 (e.g., measurement offset, etc.) is substantially consistent across all measurements taken over a period of time, it can be determined that the sensor is (at least partially) obscured. Figure 10 For example, a sensor that provides a decreasing intensity or measurement value (eg, a shrinking, decreasing, or decreasing value, etc.) may be determined to be an occluded sensor.

[0111] Figures 11A to 11CSchematic diagrams of imaging sensor information 1101-1103 detected by at least one imaging system of vehicle 100 are shown, describing a visual environment (e.g., at a point on or around vehicle 100) that changes over time (T1-T3) (e.g., while vehicle 100 is driving, etc.). In some embodiments, the imaging system may be one or more of the imaging sensors 116A, 116F, 332, 336 (e.g., cameras, etc.) described above. Schematic diagrams 11A-11C illustrate computer-generated images 1101-1103 that include one or more objects 1104A-1104E detected as varying in shape, size, range, and / or geometry as vehicle 100 operates along a path 1106 or roadway.

[0112] Figure 11A A schematic diagram illustrates imaging sensor information 1101 detected by an imaging system of a vehicle 100 at a first travel time T1, according to an embodiment of the present disclosure. In some embodiments, the vehicle 100 may be driving along a street, road, or other driving path 1104. While the vehicle 100 is driving, the imaging system may visually detect objects within the imaging system's sensing area that describe (e.g., visually) the environment outside the vehicle 100. The environment may include a first object 1104A on a road 1106 (e.g., another vehicle, a pedestrian, an object, etc.), and / or one or more other objects 1104B-1104E (e.g., buildings, landmarks, signs, markings, etc.).

[0113] As vehicle 100 moves along path 1106 , the visual characteristics associated with targets 1104A- 1104E may change at a second time T2 . Figure 11B FIG. 1 is a schematic diagram showing imaging sensor information 1102 detected by an imaging system of a vehicle 100 at a second traveling time T2 according to an embodiment of the present disclosure. Figure 11B , the ranges to all objects 1104A to 1104E have changed. For example, the size and shape of vehicle object 1104 and building objects 1104B, 1104D, 1104E have increased in size at the second time instant T2, while building object 1104C is shown as moving out of the image.

[0114] As vehicle 100 continues to move along path 1106 at subsequent times, the visual characteristics associated with objects 1104A-1104E may continue to change. Figure 11C , a schematic diagram of imaging sensor information 1103 detected by an imaging system of a vehicle 100 at a third traveling time T3 according to an embodiment of the present disclosure is shown. Figure 11CThe object information is shown to have changed to include larger shapes and sizes associated with some objects, while other objects have moved completely outside the image. For example, the shape and size of vehicle object 1104 and building object 1104B have increased, while building object 1204D is shown as moving outside the image and building objects 1104C, 1104E have moved completely outside the image. In some embodiments, Figures 11A to 11C Imaging sensor information is shown as a function of time (eg, T1 to T3 ) for an unobstructed imaging system and / or sensor.

[0115] Figure 12 is a block diagram of an embodiment of aspects of a communication environment 1200 of a vehicle 100 including a system for generating simulated lidar data according to an embodiment of the present disclosure. The communication system 1200 may include Figure 3 The components shown are shown in Figure 1, with several components omitted for simplicity. However, it should be understood that Figure 3 All components in the Figure 12 The features of the graph are integrated or otherwise used in conjunction with the features of the graph.

[0116] like Figure 3 As shown, the driving vehicle sensors may include radar 324 and camera 332. Camera 332 may be any type of sensor configured to collect image data, including a night vision camera or an infrared camera. Sensor processor system 340 includes a lidar autoencoder 1215, which may include a convolutional neural network autoencoder described in more detail below, and a rendering engine 1220, which may include a custom lidar rendering engine described in more detail below. Lidar autoencoder 1215 receives data from radar 324 and camera 332 and outputs the data to rendering engine 1220. Rendering engine 1220 outputs simulated lidar data to vehicle control system 348.

[0117] The vehicle control system 348 can receive the simulated lidar data from the rendering engine 1220 and determine whether to control certain aspects of the vehicle 100. Controlling certain aspects of the vehicle 100 may include presenting information via one or more display devices 372 associated with the vehicle, sending commands to one or more computing devices 368 associated with the vehicle, and / or controlling the driving operation of the vehicle. In some embodiments, the vehicle control system 348 may correspond to one or more computing systems that control the driving operation of the vehicle 100 according to the aforementioned driving autonomy level. In one embodiment, the vehicle control system 348 can manipulate the speed of the vehicle 100 by controlling output signals to the vehicle's accelerometer and / or braking system. In this example, the vehicle control system 348 can receive sensor data describing the environment surrounding the vehicle 100 and, based on the received sensor data, determine whether to adjust the acceleration, power output, and / or braking of the vehicle 100. The vehicle control system 348 may also control the steering and / or other driving functions of the vehicle 100. In another embodiment, the vehicle control system 348 may calculate modified driving maneuvers and generate control signals based on those modified driving maneuvers to control one or more actuators 1250 to implement the modified driving maneuvers.

[0118] It should be understood that the operation of the vehicle control system 300 and the modified communication environment 1200 may be as described above with reference to Figure 3 Runs as described.

[0119] Figure 13 is a flow chart illustrating a training operation for a lidar simulation algorithm. The training method utilizes (1) a lidar autoencoder 1215, such as a convolutional neural network autoencoder, which takes as input image data (I) from camera 332 and radar data from radar 324 and outputs a predicted depth (D); (2) a rendering engine 1220 configured to simulate any type of lidar data using any number of beams or a single grating in the case of a solid-state lidar system; and (3) a series of loss functions (along with additional networks for training, not shown) that involve minimizing geometric loss and error between distributions. Thus, at step 1305, the autoencoder 1215 receives sparse radar data and image data. At step 1310, the autoencoder 1215 applies a convolutional neural network autoencoding operation. At step 1315, the autoencoder 1215 outputs a predicted dense depth map or point cloud. At step 1320, a first set of loss functions is applied to the data. At step 1325, the custom lidar rendering engine 1220 renders the dense depth map or point cloud into a sparse depth map or point cloud. At step 1330, the rendering engine 1220 outputs the sparse depth map or point cloud. At step 1335, a second set of loss functions is applied.

[0120] At step 1320, several loss functions may be used including the following:

[0121] Sum of Squares Error: Given an image I (X×Y×3) and a radar point cloud R (N×3, (x, y, z)), the autoencoder 1215 outputs a depth map D (X×Y×1). The sum of squares loss is the average of the sum of the squared differences between the nearest z value at (x, y) in the radar point cloud R and the depth value D corresponding to (x, y) and the probability map P representing the confidence of the depth value. The purpose of this loss function is to force the network to learn the true scale of the scene acquired from the radar. Using radar points via the loss function can also enable the network to learn to ignore noisy radar points, thereby reducing or eliminating the need for a denoising module.

[0122] Image consistency loss: A second autoencoder 1215B (not shown) can be configured identically to autoencoder 1215. Autoencoder 1215B takes as input the predicted depth map D (the output of autoencoder 1215) and outputs a rendered version of image I, denoted as I'. Cycle consistency is achieved using a cycle consistency loss (L2 norm or sequential loss) ||I-I'||2. This loss function aims to ensure that the image and depth map are consistent—smoother regions in the image are also smoother in the depth map, and vice versa, and that nearby regions share the same depth with the same texture in the image (to implicitly combat the sparsity of radar point clouds).

[0123] Cross-entropy loss: Similar to the image consistency loss, the cross-entropy loss operates to ensure that the entropy is relatively consistent between the image and the depth map in a local neighborhood. This approach again helps ensure that the depth map remains consistent across texture-similar regions in the image, but not within a local neighborhood. This loss also enables smoothness constraints to be enforced in the predicted depth map.

[0124] The loss function applied at step 1335 may include the following:

[0125] Adversarial Loss: Once the depth map is rendered as lidar data, the algorithm can learn a penalty if the generated lidar data and the actual lidar data belong to the same distribution. This is done using a discriminator network G (a generative adversarial network architecture) that computes the distance between the generated lidar points and the underlying distribution of actual lidar points (Wasserstein distance). Although the discriminator is tested using lidar points, this does not constitute supervision using lidar (as contemplated by the present disclosure) and enables the algorithm to be trained without supervision using lidar. Arbitrary lidar samples can be taken at this stage and fed as ground truth to the discriminator. For example, if the algorithm wishes to generate simulated Velodyne 64 beam lidar images, the rendering engine 1220 is configured to do so, and any existing Velodyne 64 beam dataset can be collected from any dataset to be used for training the adversarial loss.

[0126] Semantic loss: This loss function ensures that the rendered point cloud is not only beautiful and similar to the real point cloud, but can also be used as a real lidar point cloud to solve perception problems. For example, in the presence of object segmentation annotations in 2D images or object detection annotations in 2D images, the generated lidar data can be used to perform object detection and / or point segmentation, and the simulated lidar generation network is trained using the localization loss or cross entropy loss (in the case of segmentation). This ensures that the simulated lidar network learns to generate not only semantically meaningful point clouds, but also learns to generate obvious cues of object points that are critical for object detection.

[0127] A further description of the autoencoder 1215 will now be provided. Unlike typical autoencoders, the autoencoder 1215 has two convolutional heads (encoders)—one for image data from the camera 332 and another for radar data from the radar 324. The output is a dense depth map D as described above. To create a stronger dependency between the autoencoder 1215 and the radar point cloud (to ensure that the radar data is well utilized), the autoencoder 1215 uses skip connections between the radar convolutional head (encoder) and the decoder. It also regresses depth maps at multiple scales (at multiple layers of the decoder) and minimizes the loss between the radar data and the predicted depth map at each scale, thereby creating a stronger dependency on the radar point cloud.

[0128] The autoencoder 1215 can optionally use a parallel architecture to perform iterative regression of depth. The goal of the autoencoder 1215 is to increase the density of the radar point cloud. Therefore, the autoencoder 1215 can be iteratively fed by selecting only points of the depth map D whose confidence, denoted by P, exceeds a certain threshold and feeding them as input to the autoencoder 1215 at each step. This allows the autoencoder 1215 to iteratively increase the density of the radar point cloud. However, it should be noted that the techniques described herein can be implemented using a non-iterative architecture to improve ease of real-time implementation.

[0129] In contrast, the autoencoder B (described above) can be a typical autoencoder with a depth map D as input and I' as output. D and I are matrices of the same size.

[0130] Similarly, the discriminator G used to calculate the adversarial loss is a typical convolutional neural network with convolutional, nonlinear fully connected layers followed by a softmax function that provides the probability that the generated data and the true value data belong to the same distribution. This can be achieved via a minimax decision rule or Wasserstein loss.

[0131] The training method can also be extended to a multi-frame setting by adding an additional image consistency loss function between two adjacent image frames. In this method, the algorithm runs in "independent" mode for each image and generates a separate depth map for each instance. Subsequently, the scene transformation can be estimated simultaneously with the two independent depth maps, allowing the estimation of independently moving 3D points / regions. This actually provides the distance between two independently moving 3D points, which can also be used as velocity information. This is an advantage compared to known expensive multi-view image reconstruction techniques.

[0132] The operation of rendering engine 1220 will now be described. Given a depth map D and an optical center O(x, y, z), rendering engine 1220 simulates a lidar by algorithmically generating a lidar point cloud based on the sensor's input data and configuration. For example, to simulate the orientation and tilt of a Velodyne 64, engine 1220 places 64 different optical centers above and below the sensor's optical center, generates a 360-degree beam, and extracts a corresponding depth point for each projection of the beam from each of the 64 optical centers. Similarly, for any solid-state sensor such as the Aeva, engine 1220 performs raster rendering to generate multiple points rather than a 360-degree beam.

[0133] Furthermore, the algorithm can operate directly with the estimated depth map without the need for a rendering engine, but a rendering engine provides a way to mimic lidar data with the same level of sparsity and similar properties as real lidar. Using a rendering engine is also necessary for the adversarial loss function, as the true and generated data must have the same distribution, but otherwise the generated depth map is at least as good, if not better, for use in most perception pipelines.

[0134] Figure 14 is a flow chart illustrating the operation of the on-site lidar simulation system 1405 and the vehicle control system 348 after the algorithm has been trained. Figure 13 The same steps are shown, except that the loss function is now omitted because the algorithm is fully trained. Therefore, at step 1415, the autoencoder 1215 receives the sparse radar data from the radar 324 and the image data from the camera 332. At step 1420, the autoencoder 1215 applies a convolutional neural network autoencoding operation. At step 1425, the autoencoder 1215 outputs a predicted dense depth map or point cloud. At step 1430, the custom lidar rendering engine 1220 renders the dense depth map or point cloud into a sparse depth map or point cloud. At step 1435, the rendering engine 1220 outputs the sparse depth map or point cloud.

[0135] The simulated lidar data (e.g., a sparse depth map or point cloud) is output to the vehicle control system 348, which receives the data at step 1440. At step 1445, the vehicle control system 348 determines and generates a modified driving operation signal based on the received simulated lidar data. At step 1450, the vehicle control system 348 outputs the modified driving operation signal to operate the vehicle, for example, by controlling one or more actuators to implement the generated modified driving operation signal. The vehicle control system is described above with respect to Figure 3 and Figure 12 Operate as described.

[0136] Any of the steps, functions and operations discussed herein may be performed continuously and automatically.

[0137] The exemplary systems and methods of the present disclosure have been described with respect to vehicle systems and electric vehicles. However, to avoid unnecessarily obscuring the present disclosure, the foregoing description omits many known structures and devices. This omission should not be construed as limiting the scope of the claimed disclosure. Numerous specific details are set forth to provide an understanding of the present disclosure. However, it should be understood that the present disclosure may be practiced in a variety of ways beyond the specific details set forth herein.

[0138] Furthermore, while the exemplary embodiments described herein illustrate various components of the system in conjunction with one another, certain components of the system may be remotely located, located at a remote portion of a distributed network such as a LAN and / or the Internet, or located within a dedicated system. Therefore, it should be understood that the components of the system may be combined into one or more devices, such as servers, communication devices, or collocated at specific nodes of a distributed network, such as an analog and / or digital telecommunications network, a packet-switched network, or a circuit-switched network. It will be understood from the foregoing description, and for computational efficiency reasons, that the components of the system may be arranged at any location within a distributed component network without affecting the operation of the system.

[0139] Furthermore, it should be understood that the various links connecting the elements may be wired or wireless links or any combination thereof, or any other known or subsequently developed element(s) capable of providing and transmitting data to and from the connected elements. These wired or wireless links may also be secure links and may be capable of transmitting encrypted information. For example, the transmission medium used as the link may be any suitable carrier for electrical signals, including coaxial cable, copper wire, and optical fiber, and may take the form of sound or light waves, such as those generated during radio wave and infrared data communications.

[0140] While the flowcharts have been discussed and illustrated with respect to a particular sequence of events, it should be understood that changes, additions, and omissions to this sequence may occur without materially affecting the operation of the disclosed embodiments, configurations, and aspects.

[0141] Many variations and modifications of the disclosure may be used. Some features of the disclosure may be provided without others.

[0142] In another embodiment, the system and method of the present disclosure can be implemented in combination with a special-purpose computer, a programmed microprocessor or microcontroller and (one or more) peripheral integrated circuit components, an ASIC or other integrated circuit, a digital signal processor, a hard-wired electronic device or logic circuit (such as a discrete element circuit), a programmable logic device or gate array (such as a PLD, PLA, FPGA, PAL), a special-purpose computer, any suitable device, etc. In general, any (one or more) devices or devices capable of implementing the methods shown herein can be used to implement the various aspects of the present disclosure. Exemplary hardware that can be used for the present disclosure includes computers, handheld devices, phones (e.g., cellular, Internet, digital, analog, hybrid, etc.), and other hardware known in the art. Some of these devices include processors (e.g., single or multiple microprocessors), memory, non-volatile memory, input devices, and output devices. In addition, alternative software implementations can also be constructed, including but not limited to distributed processing or component / object distributed processing, parallel processing, or virtual machine processing to implement the methods described herein.

[0143] In yet another embodiment, the disclosed method can be readily implemented in conjunction with software using an object or object-oriented software development environment that provides portable source code that can be used on a variety of computer or workstation platforms. Alternatively, the disclosed system can be implemented partially or completely in hardware using standard logic circuits or VLSI designs. Whether software or hardware is used to implement a system according to the present disclosure depends on the speed and / or efficiency requirements of the system, the specific functionality, and the specific software or hardware system or microprocessor or microcomputer system being used.

[0144] In yet another embodiment, the disclosed method may be implemented in part in software, which may be stored on a storage medium and executed on a programmed general purpose computer, a special purpose computer, a microprocessor, etc., in cooperation with a controller and memory. In these cases, the systems and methods of the present disclosure may be implemented as a program (such as an applet, or CGI scripts), resources resident on a server or computer workstation, routines embedded in a dedicated measurement system, system components, etc. The system may also be implemented by physically incorporating the system and / or method into a software and / or hardware system.

[0145] Although this disclosure describes the components and functions implemented in the embodiments with reference to specific standards and protocols, this disclosure is not limited to such standards and protocols. Other similar standards and protocols not mentioned herein exist and are considered to be included in this disclosure. Moreover, the standards and protocols mentioned herein, as well as other similar standards and protocols not mentioned herein, are regularly replaced by faster or more efficient equivalents having substantially the same functions. Such alternative standards and protocols having the same functions are considered to be equivalents included in this disclosure.

[0146] The present disclosure includes, in various embodiments, configurations, and aspects, components, methods, processes, systems, and / or apparatus substantially as depicted and described herein, including various embodiments, subcombinations, and subsets thereof. Those skilled in the art will understand how to make and use the systems and methods disclosed herein after understanding the present disclosure. The present disclosure, in various embodiments, configurations, and aspects, includes providing apparatus and processes, for example, to improve performance, facilitate implementation, and / or reduce implementation costs, including in the absence of matters not depicted and / or described herein, or in its various embodiments, configurations, or aspects, including in the absence of such matters that may have been used in previous apparatus or processes.

[0147] The foregoing discussion of the present disclosure has been presented for purposes of illustration and description. The foregoing is not intended to limit the present disclosure to the form or forms disclosed herein. In, for example, the above-described detailed description, various features of the present disclosure are combined in one or more embodiments, configurations, or aspects for the purpose of streamlining the present disclosure. Features of the embodiments, configurations, or aspects of the present disclosure may be combined in alternative embodiments, configurations, or aspects other than those discussed above. This approach to the present disclosure should not be interpreted as reflecting an intention that the claimed disclosure requires more features than are expressly recited in each claim. Rather, as reflected in the appended claims, inventive aspects rely on less than all of the features of a single, foregoing disclosed embodiment, configuration, or aspect. Accordingly, the appended claims are hereby incorporated into this detailed description, with each claim relying on itself as an independent preferred embodiment of the present disclosure.

[0148] Moreover, although the description of the present disclosure has included descriptions of one or more embodiments, configurations, or aspects and certain variations and modifications, other variations, combinations, and modifications are also within the scope of the present disclosure, for example, as may be within the skill and knowledge of those skilled in the art after understanding this disclosure. It is intended to include, to the extent permitted, alternative embodiments, configurations, or aspects, including alternative, interchangeable, and / or equivalent structures, functions, ranges, or steps to those claimed, regardless of whether such alternative, interchangeable, and / or equivalent structures, functions, ranges, or steps are disclosed herein, and it is not intended to publicly dedicate any patentable subject matter.

[0149] An embodiment includes a system comprising: a first sensor configured to sense a sparse signal; a second sensor configured to sense an image signal; an encoder coupled to the first sensor and the second sensor and configured to receive at least one of the sparse signals and at least one of the image signals; a processor coupled to the encoder; and a computer-readable medium coupled to the processor and including instructions stored thereon that cause the processor to generate a simulated lidar output based on the at least one of the sparse signals and the at least one of the image signals.

[0150] Aspects of the above-mentioned system include: wherein the first sensor comprises a radar array and the sparse signal comprises a radar signal; wherein the second sensor comprises a camera; wherein at least one of the sparse signals consists of a radar point cloud; wherein at least one of the image signals consists of a single frame; wherein the simulated lidar output comprises a simulated lidar point cloud; and wherein the simulated lidar output comprises a depth map.

[0151] Embodiments include a vehicle comprising: a simulated lidar imaging system comprising: a first sensor configured to sense a sparse signal; a second sensor configured to sense an image signal; an encoder coupled to the first sensor and the second sensor and configured to receive at least one of the sparse signals and at least one of the image signals; a first processor coupled to the encoder; and a first computer-readable medium coupled to the first processor and comprising instructions stored thereon that cause the first processor to generate a simulated lidar output based on the at least one of the sparse signals and the at least one of the image signals; a second processor configured to receive the simulated lidar output; and a second computer-readable medium coupled to the second processor and comprising instructions stored thereon that cause the second processor to perform steps comprising: generating at least one modified driving operation signal in response to the simulated lidar image; and transmitting the modified driving operation signal to one or more actuators to control one or more vehicle components to perform a selected operation in response to the modified driving operation signal.

[0152] Aspects of the above-mentioned vehicle include: wherein the first sensor comprises a radar array and the sparse signal comprises a radar signal; wherein the second sensor comprises a camera; wherein at least one of the sparse signals consists of a radar point cloud; wherein at least one of the image signals consists of a single frame; wherein the simulated lidar output comprises a simulated lidar point cloud; and wherein the simulated lidar output comprises a depth map.

[0153] An embodiment includes a method comprising: sensing at least one sparse signal by a first sensor; sensing at least one image signal by a second sensor; receiving the at least one sparse signal and the at least one image signal by an encoder coupled to the first sensor and the second sensor; and generating, by a processor, a simulated lidar output based on the at least one sparse signal and the at least one image signal.

[0154] Aspects of the above method include: wherein the first sensor comprises a radar array and the sparse signals comprise radar signals; wherein at least one of the sparse signals consists of a radar point cloud; wherein at least one of the image signals consists of a single frame; wherein the simulated lidar output comprises a simulated lidar point cloud; and wherein the simulated lidar output comprises a depth map.

[0155] Any one or more of the aspects / embodiments as substantially as disclosed herein.

[0156] Any one or more aspects / embodiments as substantially disclosed herein may optionally be combined with any one or more other aspects / embodiments as substantially disclosed herein.

[0157] One or apparatus adapted to perform any one or more of the above aspects / embodiments as substantially disclosed herein.

[0158] The phrases "at least one," "one or more," "or," and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions "at least one or two of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," "A, B and / or C," and "A, B, or C" means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0159] The term "a" or "an" entity refers to one or more of that entity. Thus, the terms "a" (or "an"), "one or more," and "at least one" are used interchangeably herein. It should also be noted that the terms "including," "comprising," and "having" are used interchangeably.

[0160] As used herein, the term "automatic" and its variations refer to any process or operation that is performed without significant human input, typically on a continuous or semi-continuous basis, when the process or operation is performed. However, a process or operation may be automatic if input is received prior to the execution of the process or operation, even if the execution of the process or operation utilizes significant or insignificant human input. Human input is considered significant if it affects the manner in which the process or operation is performed. Human input that consents to the execution of the process or operation is not considered "significant."

[0161] Aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." Any combination of one or more computer-readable medium(s) may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.

[0162] A computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media would include the following: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0163] A computer-readable signal medium may include a propagated data signal in which a computer-readable program code is embedded, for example, in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and that can convey, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embedded on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or the like, or any suitable combination thereof.

[0164] As used herein, the terms "determine," "calculate," "infer," and variations thereof are used interchangeably and include any type of methodology, process, mathematical operation, or technique.

[0165] The term "electric vehicle" (EV) is also referred to herein as an electrically driven vehicle that can be propelled using one or more electric motors or traction motors. An electric vehicle can be powered by electricity from an off-board source through a collector system, or can contain its own batteries or generators to convert fuel into electricity. Electric vehicles generally include a rechargeable electrical storage system (RESS) (also referred to as a full electric vehicle (FEV)). Power storage methods can include chemical energy stored in the vehicle's on-board battery (e.g., a battery electric vehicle or BEV), on-board kinetic energy storage devices (e.g., a flywheel), and / or static energy (e.g., via an on-board double-layer capacitor). Rechargeable on-board electrical storage devices can be in the form of batteries, double-layer capacitors, and flywheel energy storage devices.

[0166] The term "hybrid electric vehicle" refers to a vehicle that can combine a conventional (usually fossil fuel-powered) powertrain with some form of electric propulsion. Most hybrid electric vehicles combine a conventional internal combustion engine (ICE) propulsion system with an electric propulsion system (hybrid vehicle driveline). In a parallel hybrid, both the ICE and the electric motor are connected to a mechanical transmission and can usually transmit power simultaneously through the conventional transmission to drive the wheels. In a series hybrid, only the electric motor drives the powertrain, and the smaller ICE is used as a generator to power the electric motor or recharge the battery. Power-split hybrids have both series and parallel characteristics. A full hybrid, sometimes also called a strong hybrid, is a vehicle that can run on just the engine, just the battery, or a combination of the two. A moderate hybrid is a vehicle that cannot be driven by its electric motor alone, because the electric motor does not have enough power to propel the vehicle on its own.

[0167] The term "rechargeable electric vehicle" or "REV" refers to a vehicle having an onboard rechargeable energy storage device, including electric vehicles and hybrid electric vehicles.

Claims

1. A system for simulating a laser radar, comprising: A first sensor for sensing a sparse signal; a second sensor, configured to sense an image signal; an encoder coupled to the first sensor and the second sensor for receiving at least one of the sparse signals and at least one of the image signals; the encoder comprising a convolutional neural network autoencoder; a rendering engine coupled to the encoder; a processor coupled to the encoder and the rendering engine; as well as a computer-readable medium coupled to the processor and comprising instructions stored thereon, the instructions causing the processor to: applying, by the encoder, a convolutional neural network auto-encoding operation; outputting, by the encoder, at least one of a predicted point cloud and a predicted depth map based on the at least one of the sparse signals and the at least one of the image signals; rendering, by the rendering engine, at least one of the predicted point cloud and the predicted depth map into at least one of a second point cloud and a second depth map having a lower density than the predicted point cloud; as well as The rendering engine generates a simulated lidar output capable of replacing the non-simulated lidar output based on at least one of the second point cloud and the second depth map.

2. The system of claim 1, wherein: The first sensor comprises a radar array, and the sparse signal comprises a radar signal.

3. The system of claim 1, wherein: The second sensor includes a camera.

4. The system of claim 1, wherein: The at least one of the sparse signals consists of a radar point cloud.

5. The system of claim 1, wherein: The at least one of the image signals consists of a single frame.

6. The system of claim 1, wherein: The simulated lidar output includes a simulated lidar point cloud.

7. The system of claim 1, wherein: The simulated lidar output includes a depth map.

8. A vehicle comprising: Simulate a lidar imaging system, including: a first sensor that senses a sparse signal; a second sensor, the second sensor sensing an image signal; an encoder coupled to the first sensor and the second sensor, receiving at least one of the sparse signals and at least one of the image signals, the encoder comprising a convolutional neural network autoencoder; a rendering engine coupled to the encoder; a first processor coupled to the encoder and the rendering engine; and A first computer-readable medium is coupled to the first processor and includes instructions stored thereon that cause the first processor to: applying, by the encoder, a convolutional neural network auto-encoding operation; outputting, by the encoder, at least one of a predicted point cloud and a predicted depth map based on the at least one of the sparse signals and the at least one of the image signals; rendering, by the rendering engine, at least one of the predicted point cloud and the predicted depth map into at least one of a second point cloud and a second depth map having a lower density than the predicted point cloud; and generating, by the rendering engine, a simulated lidar output capable of replacing a non-simulated lidar output based on at least one of the second point cloud and the second depth map; a second processor, the second sensor receiving the simulated lidar output; and a second computer-readable medium coupled to the second processor and comprising instructions stored thereon that cause the second processor to: generating at least one modified driving operation signal in response to the simulated lidar output; and The modified driving operation signal is communicated to one or more actuators to control one or more vehicle components to perform a selected operation in response to the modified driving operation signal.

9. The vehicle according to claim 8, wherein: The first sensor comprises a radar array, and the sparse signal comprises a radar signal.

10. The vehicle of claim 8, wherein: The second sensor includes a camera.

11. The vehicle of claim 8, wherein: The at least one of the sparse signals consists of a radar point cloud.

12. The vehicle of claim 8, wherein: The at least one of the image signals consists of a single frame.

13. The vehicle of claim 8, wherein: The simulated lidar output includes a simulated lidar point cloud.

14. The vehicle of claim 8, wherein: The simulated lidar output includes a depth map.

15. A method for simulating a laser radar, comprising: sensing at least one sparse signal by a first sensor; sensing at least one image signal by a second sensor; receiving the at least one sparse signal and the at least one image signal by an encoder coupled to the first sensor and the second sensor, the encoder comprising a convolutional neural network autoencoder; applying, by the encoder, a convolutional neural network auto-encoding operation; outputting, by the encoder, at least one of a predicted point cloud and a predicted depth map based on at least one of the sparse signals and at least one of the image signals; rendering, by a rendering engine, at least one of the predicted point cloud and the predicted depth map into at least one of a second point cloud and a second depth map having a lower density than the predicted point cloud; as well as The rendering engine generates a simulated lidar output capable of replacing the non-simulated lidar output based on at least one of the second point cloud and the second depth map.

16. The method of claim 15, wherein: The first sensor comprises a radar array, and the sparse signal comprises a radar signal.

17. The method of claim 15, wherein: The at least one of the sparse signals consists of a radar point cloud.

18. The method of claim 15, wherein: The at least one of the image signals consists of a single frame.

19. The method of claim 15, wherein: The simulated lidar output includes a simulated lidar point cloud.

20. The method of claim 15, wherein: The simulated lidar output includes a depth map.

Images