Dynamic control and configuration of autonomous navigation systems

Abstract

This disclosure generally relates to systems and methods for generating, processing, and correlating data from multiple sensors in an autonomous navigation system, and more particularly to the utilization of configurable and dynamic sensor modules within optical detection and ranging systems, which enables improved correlation between sensor data and the configurability and responsiveness of the system to its surrounding environment.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to USSN 16 / 241,825, filed January 7, 2019, which is incorporated herein by reference in its entirety. Background Technology

[0003] A. Technical Field

[0004] This disclosure generally relates to systems and methods for generating, processing, and correlating data from multiple sensors in an autonomous navigation system, and more particularly to the utilization of configurable and dynamic sensor modules within an optical detection and ranging (hereinafter "LIDAR") system, which enables improved correlation between sensor data and the configurability and responsiveness of the system to its surrounding environment.

[0005] B. Background Technology

[0006] Those skilled in the art will understand the importance of the accuracy and timely analysis of sensor data within autonomous navigation systems. Autonomous navigation requires computerized systems to receive data from sensors, form a sufficiently accurate representation of their environment, and make decisions based on that data in real time. Any errors in the interpretation of sensor data or delays in initiating timely responses to that sensor data can have undesirable consequences. Modern autonomous systems must process data from multiple discrete sensor systems and interpret their combined outputs. The volume of data from these various sensors can be enormous, and simply processing and correlating the outputs from different sensors can require significant processing power and time.

[0007] The implementation of LiDAR or hybrid sensor systems may have different operational requirements, such as field of view, range, and response rate. Adapting these systems to different types of autonomous navigation systems and vehicles can be challenging because the system's environment and intended use may impose different performance requirements. Simply positioning and integrating sensors within different types of vehicles can introduce problems for many existing sensor-based navigation systems. For example, sensors cannot be easily juxtaposed because the data is subject to parallax caused by data acquired from different vantage points. Furthermore, these sensors may also have different data collection rates. Improper correlation of this sensor data can lead to motion errors or conflicts across sensor data. Therefore, before any autonomous navigation decisions can be made, the autonomous system must first process and interpret data from relevant sensors and then correlate them with each other. Any unnecessary delays within the decision-making process can lead to malfunctions in the autonomous driving system.

[0008] Therefore, what is needed are systems and methods that provide configurable, accurate, timely and efficient solutions for receiving and processing sensor data across multiple sensors installed within an autonomous navigation system. Attached Figure Description

[0009] Reference will be made to embodiments of the invention, examples of which are illustrated in the accompanying drawings. These drawings are intended to be illustrative and not restrictive. Although the invention is generally described in the context of these embodiments, it should be understood that the scope of the invention is not intended to be limited to these specific embodiments. Items in the figures are not to scale.

[0010] Figure 1 (“Figure”) illustrates the operation of a LiDAR system according to an embodiment of this document.

[0011] Figure 2A The illustration shows the operation of a LiDAR system and multiple return light signals according to an embodiment of this document.

[0012] Figure 2B A LiDAR system with an oscillating mirror according to an embodiment of this document is described.

[0013] Figure 3A A distributed sensor system installed in an automobile according to an embodiment of this document is described, which utilizes a set of sensors coupled to a microcontroller (hereinafter "MCU").

[0014] Figure 3B The framework of a sensor system according to embodiments of the present disclosure is described.

[0015] Figure 3C The operation of an MCU in an autonomous driving system utilizing a sensor module and a sensor bus according to embodiments of the present disclosure is described.

[0016] Figure 3D and Figure 3E The illustration depicts a method for dynamically configuring different sensors and sensor types within an autonomous navigation system according to an embodiment of the present disclosure.

[0017] Figure 3F The illustration shows a method for updating calibration parameters in a calibration engine according to an embodiment of the present disclosure.

[0018] Figure 4A and Figure 4B A configurable sensor architecture according to embodiments of the present disclosure is described.

[0019] Figure 4C illustrates the Lissajous scanning mode and resolution according to an embodiment of the present disclosure. Figure 4D, Figure 4E and Figure 4F The illustration shows the scanning resolution of the field of view (FOV) according to an embodiment of the present disclosure.

[0020] Figure 4G The illustration shows a specific scanning pattern of a sensor module including eight sensors according to an embodiment of the present disclosure.

[0021] Figure 4H and Figure 4I Exemplary square and pie wedge configurations of sensors according to embodiments of the present disclosure are illustrated.

[0022] Figure 4J The illustration shows a sensor system according to an embodiment of the present disclosure, which supports object detection using various sensor types including LIDAR, infrared radiation (IR), and ambient light modality, to detect distance, reflectivity, temperature, and color, respectively.

[0023] Figure 5 A simplified block diagram of a computing device / information processing system for automotive applications, according to embodiments of this document, is depicted. Detailed Implementation

[0024] In the following description, specific details are set forth for purposes of explanation in order to provide an understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without these details. Furthermore, those skilled in the art will recognize that the embodiments of the invention described below can be implemented in various ways, such as processes, apparatuses, systems, devices, or methods on tangible computer-readable media.

[0025] The components or modules illustrated in the figures are illustrative of exemplary embodiments of the invention and are intended to avoid obscuring the invention. It should also be understood that throughout this discussion, components can be described as individual functional units, which may include subunits; however, those skilled in the art will recognize that various components or portions thereof may be divided into individual components or may be integrated together, including within a single system or component. It should be noted that the functions or operations discussed herein can be implemented as components. Components can be implemented using software, hardware, or a combination thereof.

[0026] Furthermore, the connections between components or systems within the diagram are not intended to be limited to direct connections. Rather, data between these components can be modified, reformatted, or otherwise altered by intermediate components. Additionally, additional or fewer connections may be used. It should also be noted that the terms “coupling,” “connection,” or “communicably coupled” should be understood to include direct connections, indirect connections via one or more intermediate devices, and wireless connections.

[0027] References to "an embodiment," "preferred embodiment," "an embodiment," or "embodiment" in the specification mean that a particular feature, structure, characteristic, or function described in connection with that embodiment is included in at least one embodiment of the invention, and may be included in more than one embodiment. Furthermore, the foregoing phrases appearing in various places in the specification do not necessarily refer to the same one or more embodiments.

[0028] The use of certain terms in various places in this specification is for illustrative purposes and should not be construed as limiting. Services, functions, or resources are not limited to individual services, functions, or resources; the use of these terms may refer to groups of related services, functions, or resources, which may be distributed or aggregated.

[0029] The terms “comprising,” “including,” “containing,” and “comprising” should be understood as open-ended terms, and any subsequent list is illustrative and not intended to limit us to the listed items. Any headings used herein are for organizational purposes only and should not be used to limit the scope of the specification or claims. Each reference mentioned in this patent document is incorporated herein by reference in its entirety.

[0030] In addition, those skilled in the art should recognize that: (1) certain steps may be performed optionally; (2) the steps may not be limited to the specific order set forth herein; (3) certain steps may be performed in a different order; and (4) certain steps may be performed simultaneously.

[0031] Light detection and ranging systems, such as LIDAR systems, can be tools for measuring the shape and contours of the environment surrounding the system. LIDAR systems can be applied to a wide range of applications, including autonomous navigation and aerial mapping of surfaces. A LIDAR system emits light pulses, which are then reflected from objects within the environment in which the system operates. The time it takes for each pulse to travel from emission to reception (i.e., time of flight, "TOF") can be measured to determine the distance between the object and the LIDAR system. The science is based on the physical properties of light and optics.

[0032] In a LiDAR system, light is emitted from a rapidly firing laser. The laser travels through a medium and is reflected from points on surfaces in the environment, such as buildings, tree branches, and vehicles. The reflected light returns to the LiDAR transceiver (detector), where it is recorded and used to map the environment.

[0033] Figure 1Operation of a LiDAR component 102 and data analysis and interpretation 109 according to embodiments of this disclosure is depicted. The LiDAR component 102 may include a transmitter 104 for transmitting emitted light signals 110, a receiver 106 including a detector, and a system control and data acquisition 108. The LiDAR component 102 may be referred to as a LiDAR transceiver. The emitted light signal 110 propagates through a medium and is reflected from an object 112. The returned light signal 114 propagates through a medium and is received by the receiver 106. The system control and data acquisition 108 may control the light emission of the transmitter 104, and the data acquisition may record the returned light signal 114 detected by the receiver 106. The data analysis and interpretation 109 may receive output from the system control and data acquisition 108 via a connection 116 and perform data analysis functions. The connection 116 may be implemented using wireless or contactless communication methods. The transmitter 104 and receiver 106 may include optical lenses and mirrors (not shown). The transmitter 104 may emit a laser beam having multiple pulses in a specific sequence. In some embodiments, the light detection and ranging component 102 and the data analysis and interpretation 109 comprise a LiDAR system. The design elements of the receiver 106 are the horizontal field of view (hereinafter “FOV”) and the vertical FOV. Those skilled in the art will recognize that the FOV effectively defines the visible area associated with a particular LiDAR system. The horizontal and vertical FOVs may be defined by a single LiDAR sensor, or may involve multiple configurable sensors (which may be exclusively LiDAR sensors or may include different types of sensors). The FOV can be considered as the scanning area of ​​the LiDAR system. A scanning mirror can be used to obtain the scanned FOV.

[0034] Figure 2A The illustration shows the operation 200 of a LiDAR system 202 according to an embodiment of this document, including multiple return optical signals: (1) return signal 203 and (2) return signal 205. Due to the divergence of the laser beam, a single laser shot often hits multiple objects, resulting in multiple returns. The optical detection and ranging system 202 can analyze these multiple returns and can report the strongest return, the last return, or two returns. According to... Figure 2A The optical detection and ranging system 202 emits lasers in directions near the wall 204 and far the wall 208. As shown, most of the beam hits the near wall 204 at region 206, resulting in a return signal 203, while another portion of the beam hits the far wall 208 at region 210, resulting in a return signal 205. Compared to return signal 205, return signal 203 can have a shorter time-of-flight (TOF) and a stronger received signal strength. In both single-return and multi-return LIDAR systems, it is important that the return signal is accurately correlated with the transmitted optical signal in order to calculate an accurate TOF.

[0035] Some embodiments of LIDAR systems can capture spacing data in a 2D (i.e., single-plane) point cloud manner. These LIDAR systems are often used in industrial applications and may frequently be reused for surveying, mapping, autonomous navigation, and other purposes. Some embodiments of these devices rely on using a single laser emitter / detector pair combined with some type of moving mirror to achieve scanning across at least one plane. This mirror not only reflects the emitted light from the diode but also reflects the returned light back to the detector. Using an oscillating mirror in this application can be a means of achieving 90–180–360 degree azimuth (horizontal) views while simplifying both system design and manufacturability. Many applications require more data than just a single 2D plane. 2D point clouds can be expanded to form 3D point clouds, where multiple 2D clouds are used, each pointing at a different elevation (vertical) angle. The design elements of the receiver in the light detection and ranging system 202 include horizontal and vertical FOVs.

[0036] Figure 2B A LIDAR system 250 with an oscillating mirror according to an embodiment of this document is depicted. The LIDAR system 250 employs a single laser emitter / detector combined with an oscillating mirror to efficiently scan across a plane. Spacing measurements performed by such a system are actually two-dimensional (i.e., planar), and the captured spacing points are presented as a 2D (i.e., single-plane) point cloud. In some embodiments, but not limited thereto, the oscillating mirror oscillates at a very high speed (e.g., thousands of cycles per minute).

[0037] LIDAR system 250 includes laser electronics 252, which includes a single light emitter and a photodetector. The emitted laser signal 251 can be guided to a fixed mirror 254, which reflects the emitted laser signal 251 to an oscillating mirror 256. When the oscillating mirror 256 "oscillates," the emitted laser signal 251 can be reflected from an object 258 in its propagation path. The reflected signal 253 can be coupled to the detector in laser electronics 252 via the oscillating mirror 256 and the fixed mirror 254. The receiver design elements of LIDAR system 250 include a horizontal field of view (FOV) and a vertical field of view (FOV), which define the scanning area.

[0038] Figure 3AA distributed sensor system 300 installed in an automobile according to an embodiment of the present disclosure is depicted. The distributed sensor system 300 utilizes a set of sensors coupled to an MCU 302. The set of sensors includes sensor modules 304, 306, 308, 310, and 312. The term "sensor module" is intended to be broadly defined and includes implementations of both single-sensor modules and multi-sensor modules. Furthermore, the type of sensor(s)(one or more) within a sensor module may vary depending on the system configuration. In some instances, a sensor module may include a single sensor (hereinafter "single-sensor module") such as a LiDAR sensor or multiple sensors (hereinafter "multi-sensor module"). A multi-sensor module may include multiple integrated sensors, multiple discrete sensors, or combinations thereof. A multi-sensor module may also include multiple LiDAR sensors or multiple sensors of different types associated within the module. Figure 3A As shown, this sensor module suite can be distributed across various locations on the vehicle. Correlated sensor data from the various sensor modules is provided to the MCU 302 for analysis and decision processing. The connectivity between the sensor modules and the MCU 302 is provided by a sensor bus, which can serially transmit different sensor data (other embodiments may exist in which sensor data is transmitted on a parallel bus).

[0039] As previously described, the sensor module may include a single sensor or multiple sensors and supports various types of sensors, such as LIDAR transceivers, thermal / far-IR sensors, visible / near-IR sensors, or other types of sensors known to those skilled in the art. The sensor structure can have various shapes, including rectangular or wedge-shaped modular designs that can be laid flat and / or stacked and allow for designs that can bypass corners. These different sensor shapes allow for the configurability of the sensor module, including configurability of the field of view (FOV), sensor range, etc. Different scanning modes and resolutions can be achieved based on the specific configuration of the sensor module and the corresponding FOV.

[0040] MCU 302 may be coupled to autonomous driving system control unit (hereinafter “ADSCU”) 301. In some embodiments, ADSCU 301 may provide sensor commands and information to MCU 302.

[0041] Figure 3BThe framework of a sensor system 320 according to an embodiment of this disclosure is depicted. The sensor system 322 may be supported by an MCU 324 and its associated software. The sensor system 322 may include a scanning mirror 326, an ASIC 328, firmware 330, and a sensor 332. In some embodiments, the scanning mirror 326 may be a dual-axis resonant scanning mirror. In some embodiments, the sensor 332 may support a combination of sensor modules as described above and may include various sensor types, including LIDAR, color (RGB), thermal (far IR), or other sensor types known to those skilled in the art. The sensor system 320 is capable of receiving data signals from the combination of sensor modules, correlating sensor data, and processing the correlated sensor data in a timely manner to make timely decisions based thereon.

[0042] To enable autonomous vehicles to perceive their surroundings and react accordingly, multiple technologies can be applied to sensor systems to co-locate data from multiple sensor modules. In particular, it may be necessary to co-locate data from sensor modules for dynamic and spatial analysis / inference, meaning their differences are decoupled, and digital information can be transmitted, stored, and computed in a way that allows the vehicle and its operating system to efficiently process and act on different sensor data. In this regard, data from distributed sensors can be multiplexed to provide unified data packets and coupled to a microcontroller via a sensor bus.

[0043] Figure 3C The operation of an MCU 348 in an autonomous driving system 340 utilizing sensor module 352 and bus 358 according to an embodiment of this disclosure is depicted. As shown, an object 341 within the autonomous navigation environment is detected by one or more sensor modules 352. As previously described, the structure and type of the sensors(s) within sensor module 352 can vary based on design and / or preference.

[0044] Based on the number, type, and location of sensor modules 352 installed around the vehicle, the autonomous driving system 340 can support various configurations and redundancies. Sensor modules 352 can be activated based on application and external conditions. For example, fewer sensors and / or sensor modules can be activated when the car is being driven on an open highway compared to when the car is being driven in heavy traffic. Additionally, sensors and / or sensor modules can be activated based on the specific mode in which the car is operating. For example, specific sensors can be activated if the vehicle is operating in driver mode compared to autonomous mode. This dynamic activation of sensors is another aspect of sensor network configurability, allowing the system to dynamically adapt to its environment both during installation and during operation.

[0045] One or more sensor modules 352 may detect objects 341 across multiple sensors and individually couple their detected data signals (shown as data streams 1 to n) 354 to a multiplexer 356. The multiplexer 356 combines channels of different sensed data and generates a unified data packet that correlates the data from each sensor. In some embodiments, the unified data packet includes distance and reflectivity data from a LIDAR transceiver, color / RGB data from a camera, and temperature data from a far-infrared detector. In other embodiments, other sensor types from other electromagnetic spectrum regions, such as acoustic, radar, or sonar, may be included. Those skilled in the art will recognize that sensor module 352 may include various combinations of sensor modules, sensor types, and sensor configurations. The unified data packet is coupled to a bus 358, which is typically serial but can also be parallel in nature.

[0046] Data from multiple sensors and / or sensor modules can be multiplexed and coupled to a microcontroller MCU 348 via bus 358. The MCU 348 interacts with the autonomous driving system control unit (hereinafter “ADSCU”) 342 to receive configuration and parameters from the sensors for data acquisition.

[0047] In some embodiments, MCU 348 can receive external conditions and information about the car's motion. MCU 348 includes data processing element 349, demultiplexer 350, calibration engine 351, and driver 353. In some embodiments where the bus is serial, demultiplexer 350 receives data serially from multiple sensor modules and transforms the data using calibration parameters from the calibration engine as if it were coming directly from the sensors (i.e., on a sensor channel basis). Calibration engine 351 provides transformations between different sensors and / or sensor modules. In some examples, these transformations are initialized to factory settings and continuously updated over time. Data processing element 349 includes one or more embedded algorithms for calculating information such as object detection, speed estimation, road localization, and external mapping. Driver 353 is responsible for activating the sensors and / or sensor modules of interest and also provides clock triggering.

[0048] Demultiplexer 350 demultiplexes the unified serial data packets of sensor data and associates the data with the corresponding sensor and / or sensor module. This data is then provided to calibration engine 351, which generates transformation information based on calibration parameters received from ADSCU 342. Demultiplexer 350 also receives spatial transformation information and integrates it with the demultiplexed sensor data's unified serial data packets into a specific format, such as a point cloud format.

[0049] As previously Figure 3A As noted, the ADSCU 342 can provide sensor commands to the MCU 302. In some embodiments, the ADSCU 342 is a computer in an automobile and a component manufactured into the vehicle. Figure 3C As shown, the ADSCU 342 receives input in the form of a point cloud from the data processing component 349—the MCU 348. In some embodiments, odometer readings, lighting conditions, and external mapping can be provided to the MCU 348 from another device within the vehicle. The ADSCU 342 can also generate a sensor configuration 346, including sensor type configuration, field of view, frame rate, and region of interest (ROI). ROI can be, for example, a pedestrian crossing or driving lane. Through ROI identification methods, the autonomous driving system 340 can filter out a large amount of unwanted raw data for actual tracking. In effect, the MCU 348 homogenizes and decouples different types of sensor data. Utilizing dynamic feedback from the ADSCU 342 in the form of calibration parameters and sensor configurations, the MCU 348 can dynamically configure sensors and / or sensor modules across different configurations and spaces within an autonomous vehicle environment.

[0050] Figure 3D and Figure 3E The figures illustrate methods 360 and 361 for dynamically configuring a multi-sensor module across different types of sensors and spaces according to embodiments of the present disclosure, including the following steps:

[0051] Receive sensor configuration parameters at the MCU (driver) and calibration parameters from ADSCU at the MCU (calibration engine) (step 362).

[0052] Send configuration parameters from the MCU (driver) to activate the selective single-sensor module or multi-sensor module (step 364).

[0053] Data is acquired from objects in the environment using one or more selected sensor modules (step 366).

[0054] Multiplex sensor data to generate a unified data packet (step 368)

[0055] Send a unified data packet to the MCU via the bus (step 370)

[0056] In the MCU, the unified data is grouped and demultiplexed into homogeneous sensor data (step 372).

[0057] In the MCU, homogeneous sensor data is sent to the calibration engine (step 374).

[0058] In the MCU, transformation information is generated in the calibration engine and sent to the demultiplexer (step 376).

[0059] In the MCU, the demultiplexer integrates the homogeneous sensor data and the transformed data, and sends them to the data processing unit (step 378).

[0060] The MCU generates a point cloud containing homogeneous sensor data and transformation data through data processing, and sends it to the ADSCU (step 380).

[0061] In the ADSCU, vehicle control is determined / adjusted based on the point cloud, and updated sensor configuration and calibration parameters are generated (step 382).

[0062] Repeat step 362

[0063] Figure 3F The figure illustrates a method 390 for updating calibration parameters in a calibration engine according to an embodiment of the present disclosure, including the following steps:

[0064] Receive homogeneous sensor data from the demultiplexer (step 392)

[0065] Receive calibration parameter data from ADSCU (step 394)

[0066] Update the calibration parameters in the calibration engine and generate transformation information (step 396).

[0067] The above description illustrates the configurability of system-level autonomous navigation, including the activation of certain sensors and / or sensor modules and data association across these sensors and sensor modules. In another aspect of the invention, each sensor module can be configured to operate according to a preferred set of parameters.

[0068] Figure 4A The sensor module 400 is described, and Figure 4B A sensor module 402 is depicted, from which configurable operating parameters can be defined. This configurability allows not only FOV definition but also sensor type configuration within the sensor module. Additionally, this configurability can be implemented during installation or in real-time during system operation. According to various embodiments, the directionality of one or more sensors within the sensor module can be defined by using the physical structure of the sensors, or by including directional elements (e.g., wedges) that define the orientation of the corresponding sensors. Figure 4BAs shown, sensor module 402 may include multiple sensors 406-413 coupled together in a specific architecture, such that the combination of individual sensor FOVs is stitched together to create a wider FOV for the sensor module. This configurability of the sensor module allows users to efficiently build unique sensor modules by combining different sensors into diverse architectures. The configurability is further enhanced by the ability to include different sensor types within the sensor module to improve performance relative to the environmental characteristics in which the module operates.

[0069] Sensor module 402 has a horizontal FOV and a vertical FOV corresponding to the combination of sensors 406-413. The operating characteristics of each sensor 406-413 within module 402 are combined to provide an enhanced modular FOV. These operating characteristics include the sensor's directivity, sensor range, sensor FOV, sensor type, and other characteristics known to those skilled in the art. In some embodiments, specific sensors within the module may be activated or deactivated depending on the environment in which the system is operating. Furthermore, in the event of one or more sensors failing or becoming temporarily inoperable, specific sensors may be used as redundant elements. The FOV of the sensor module depends not only on the specific operating characteristics of each sensor but also on how the data from these sensors are correlated and combined.

[0070] Figure 4C illustrates a specific example Lissajous scan pattern and resolution 432 based on different vertical FOVs of the sensor according to embodiments of the present disclosure. Scan 432 illustrates vertical and horizontal scans generated by different vertical FOV configurations of the sensor.

[0071] The charts on the right side of Figure 4C illustrate the scan resolution for different FOVs. Figure 4D, vFOV 434, illustrates the scan resolution with a 2.5-degree FOV. Figure 4E, vFOV 436, illustrates the scan resolution with a 5-degree FOV. Figure 4F Figure 438 illustrates a scan resolution with a 10-degree FOV. A resolution achieved with a 2.5-degree FOV is twice as dense as a resolution achieved with a 5-degree FOV. Similarly, a resolution achieved with a 5-degree FOV is twice as dense as a resolution achieved with a 10-degree FOV. This example illustrates the configurability of the sensor and its combined effect on scan modes and resolution. Those skilled in the art will recognize that a variety of modes and resolutions can be achieved by configuring the sensor according to aspects of this disclosure.

[0072] The configurability of the sensor module is further enhanced not only by the specific operating parameters of one or more of the sensors, but also by the combination of one or more sensors within the module. Figure 4GAn exemplary scan pattern 440 of a sensor system according to an embodiment of the present disclosure is illustrated. This sensor system includes eight sensors within a sensor module. Scan pattern 440 can be obtained using a sensor module architecture 402, in which data sensed across the eight sensors is combined to provide enhanced resolution and field of view. Scan pattern 440 includes scans 446, 447, 448, 449, 450, 451, 452, and 453, which are correlated and processed to generate the pattern. In this example, the total field of view for the sensor module architecture 402 is approximately 40 to 120 degrees. Those skilled in the art will recognize that a variety of modular FOVs and other module performance characteristics can be achieved by modifying the way the sensors are coupled together, the specific parameters of the sensors, and the methods by which the sensor data is correlated and analyzed.

[0073] Figure 4H and Figure 4I Sensor module configurations 461 and 462, according to various embodiments of the present invention, are illustrated respectively. These configurations are intended to be exemplary and do not limit the scope of the invention. In one embodiment, the sensor module configuration may be square or rectangular in shape, as illustrated in configuration 461, wherein the shape of individual sensors is configured to provide specific operational characteristics within the module. Configuration 461 includes two stacked sets of sensors, wherein the physical structure defines the field of view (FOV) for each sensor. For example, the physical size and orientation of the sensors can provide different angular and spatial scanning characteristics for use within the sensor module. As a result, the sensor shape and the relative position of the sensors provide a specific scan resolution and FOV. In another configuration, the sensor module configuration may be wedge-shaped, as illustrated in configuration 462, wherein the physical wedge element defines the orientation of the sensors within the module. These two examples illustrate to those skilled in the art a large number of configurable combinations of sensors within a sensor module. In one example, the sensor is a LiDAR sensor with corresponding operational characteristics, which allows the MCU to build enhanced scan patterns with preferred resolution. In some embodiments, the performance of the sensor system can be further enhanced by including different sensor types within the sensor module.

[0074] LiDAR sensors primarily provide unique capabilities for autonomous driving based on the rate and accuracy of their operation. These LiDAR sensors create accurate mappings that can be processed quickly and decisively, enabling rapid navigation decisions with minimal error. However, some embodiments of the invention support non-LiDAR sensors, which can be included within the sensor module to supplement LiDAR sensor data. This multi-sensor module employing different types of sensors presents unique challenges in correlating the sensed data across these sensors. Different types of sensors may have different data collection rates, making correlation across time more difficult. Additionally, different sensors closely juxtaposed within the module may be subject to parallax because data is acquired from different vantage points. Therefore, using different types of sensors within a single sensor module further complicates the previously described correlation problem and introduces additional complexity within the system's data analysis and response processing.

[0075] Various embodiments of the present invention provide a more efficient way to correlate sensor data across diverse sensor types by physically combining different sensors within a single module package. This multi-sensor module, employing different sensors, ensures a 1:1 correspondence between data points from various sensors. Sensor data streams can be presented to the autonomous system in the form of data from various sensor types already combined into correlated data packets. The autonomous system's bandwidth can then be focused on the navigation task, rather than the preprocessing and correlation of mixed datasets.

[0076] In one embodiment, consider a LiDAR system that returns a single point from the environment. This single data point is both a spacing measurement (distance) and a measure of the reflectance of an object with active illumination. As a further enhancement, the LiDAR detector can also passively measure ambient light from the scene to effectively represent the passive grayscale value associated with each LiDAR channel. In real-world navigation scenarios, the color of an object carries important information about its relevance. For example, stop signs and stop lights are red, yellow means caution, and green can mean "information" or safe passage, etc. Providing a unified grouping of data points in which spacing, reflectance, and color provide the autonomous system with additional, immediate information about the relevance of objects in the field of view.

[0077] Another key aspect of the real world is that it is teeming with living organisms. General algorithms exist that attempt to classify detected objects based on size, shape, and velocity. However, errors in such algorithms have been proven and can lead to errors within sensor systems. A key characteristic of most living animal organisms that autonomous systems may encounter is that they are warm-blooded and generally have a temperature different from their surrounding environment. This characteristic makes it possible to monitor the temperature of objects using various thermal detection techniques. In the case of incorporating thermal sensors into a LiDAR sensor, for each data point reported by the sensor, another data type can be incorporated into a single data group—the temperature of the associated object. The ability to immediately classify objects as living organisms has significant benefits for rapid decision-making in autonomous systems. The fact that data naturally correlates with real-world physical objects greatly improves both the reaction time and determinism of object identification. In some embodiments, the correlation of diverse sensor data can be used to derive confidence factors for identified objects, allowing the processed response to consider the possibility that the object is one type of object compared to another.

[0078] Thermal sensors provide real-time 3D thermal space information, enabling smarter machine vision. For example, but without limitation, an array of photodetectors sensitive to long-IR electromagnetic radiation, operating in conjunction with a scanning LiDAR system, can simultaneously locate objects in a 3D environment and distinguish warm objects (such as living organisms) from other objects in a typical automotive environment. Active tracking systems can deliver real-time digital information about the position and temperature of warm objects to the vehicle control system (as opposed to passive tracking systems that deliver trigger signals). By implementing a rapid scanning mechanism, a single detector can provide data over a large area. Large and dense channel arrays can provide high-resolution thermal space data in all directions. Furthermore, detectors can be arranged such that the data is correlated with LiDAR channels both temporally and spatially.

[0079] Those skilled in the art will recognize that various combinations of sensor types can be included within a sensor module and used to improve the performance of a sensor system. In some examples, these different sensor types can be used to enhance the performance of a LiDAR system and provide greater accuracy based on certain correlations between the sensed data and the LiDAR data.

[0080] Figure 4JThe illustration depicts a sensor system 480 according to various embodiments of the present invention, which supports the detection of an object 482 using different types of sensors within a sensor module 484. In this example, the sensor module 484 may include various combinations of LiDAR sensors, thermal / far-infrared radiation (IR) sensors, visible / near-IR sensors, and other sensor types known to those skilled in the art. The sensor module 484 receives signals from different sensor types relative to the sensed object 482. Sensor data from each different type of sensor is captured and provided along corresponding channels 490-494 to a multiplexer 488. This data can then be represented on a single cloud point for further processing.

[0081] In a specific example, sensor 484a (thermal / far IR channel) may include an array of photodetectors sensitive to long IR electromagnetic radiation. Sensor 484a can simultaneously locate objects in a 3D environment and distinguish warm objects (such as living organisms) from other objects in a typical automotive environment. Sensor 484b (visible / near IR channel) detects the RGB color characteristics of ambient light and may also include sensors for detecting other light sources such as near-infrared light. Sensor 484d may also include sensors for another region of the electromagnetic spectrum, such as acoustic, radar, or sonar. These sensors 484a, 484b, and 484d complement LiDAR sensor 484c to provide enhanced sensor system performance.

[0082] Data multiplexer 488 generates unified data packets 495, which represent correlated data from different sensors 484a-d. This data is correlated because it was acquired from the same point in space (or nearly the same point, and is unified because it is bundled into a single data packet).

[0083] In embodiments, aspects of this patent document may be directed to or implemented on an information processing system / computing system. For the purposes of this disclosure, a computing system may include any means or aggregation of means operable to compute, calculate, determine, classify, process, transmit, receive, retrieve, initiate, route, switch, store, display, communicate, manifest, detect, record, reproduce, dispose of, or utilize information, intelligence, or data of any form for commercial, scientific, control, or other purposes. For example, a computing system may be a personal computer (e.g., a laptop computer), a tablet computer, a tablet device, a personal digital assistant (PDA), a smartphone, a smartwatch, a smart bag, a server (e.g., a blade server or a rack server), a network storage device, or any other suitable device, and may vary in size, shape, performance, functionality, and price. A computing system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, ROM, and / or other types of memory. Additional components of a computing system may include: one or more disk drives; one or more network ports for communicating with external devices; and various input and output (I / O) devices such as a keyboard, mouse, touchscreen, and / or video display. The computing system may also include one or more buses operable to transmit communication between various hardware components.

[0084] Figure 5 A simplified block diagram of a computing device / information processing system (or computing system) according to embodiments of the present disclosure is depicted. It will be understood that the functionality shown for system 500 can operate to support various embodiments of the information processing system—although it should be understood that the information processing system can be configured differently and include different components.

[0085] like Figure 5 As illustrated in the diagram, system 500 includes one or more central processing units (CPUs) 501 that provide computing resources and control the computer. CPU 501 may be implemented using a microprocessor or the like, and may also include one or more graphics processing units (GPUs) 517 and / or floating-point coprocessors for mathematical calculations. System 500 may also include system memory 502, which may be in the form of random access memory (RAM), read-only memory (ROM), or both.

[0086] Multiple controllers and peripheral devices can also be provided, such as Figure 5As shown in the diagram. Input controller 503 represents an interface to one or more various input devices 504, such as a keyboard, mouse, or stylus. A wireless controller 505 communicating with a wireless device 506 may also be present. System 500 may also include a storage controller 507 for interfacing with one or more storage devices 508, each storage device 508 including a storage medium, such as magnetic tape or a disk, or an optical medium that can be used to record instruction programs for operating systems, utilities, and applications, said instruction programs may include embodiments of programs implementing various aspects of the invention. One or more storage devices 508 may also be used to store processed or unprocessed data according to the invention. System 500 may also include a display controller 509 for providing an interface to a display device 511, which may be a cathode ray tube (CRT), thin-film transistor (TFT) display, or other type of display. Computing system 500 may also include an automotive signal controller 512 for communicating with an automotive system 513. The communication controller 514 can interface with one or more communication devices 515, enabling the system 500 to connect to remote devices via any of a variety of networks, including the Internet, cloud resources (e.g., Ethernet cloud, Fibre Channel over Ethernet (FCoE) / Data Center Bridge (DCB) cloud, etc.), local area network (LAN), wide area network (WAN), storage area network (SAN), or via any suitable electromagnetic carrier signal including infrared signals.

[0087] In the illustrated system, all major system components can be connected to bus 516, which can represent more than one physical bus. However, the various system components may or may not be physically close to each other. For example, input data and / or output data can be remotely transmitted from one physical location to another. Furthermore, programs implementing various aspects of the invention can be accessed from a remote location (e.g., a server) via a network. Such data and / or programs can be transmitted through any of a variety of machine-readable media, including but not limited to: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices specifically configured to store or store and execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, and ROM and RAM devices.

[0088] Embodiments of the present invention may utilize instructions for one or more processors or processing units encoded on one or more non-transitory computer-readable media to cause steps to be performed. It should be noted that the one or more non-transitory computer-readable media should include volatile and non-volatile memory. It should be noted that alternative implementations are possible, including hardware implementations or software / hardware implementations. The functionality of a hardware implementation can be implemented using one or more ASICs, programmable arrays, digital signal processing circuits, etc. Therefore, the term "means" in any claim is intended to cover both software and hardware implementations. Similarly, the term "one or more computer-readable media" as used herein includes software and / or hardware, or a combination thereof, on which a program of instructions is embodied. In consideration of these implementation alternatives, it will be understood that the figures and accompanying description provide functional information that a person skilled in the art would need to write program code (i.e., software) and / or manufacture circuitry (i.e., hardware) to perform the desired processing.

[0089] It should be noted that embodiments of the present invention may further relate to computer products having a non-transitory tangible computer-readable medium having computer code thereon for performing operations of various computer implementations. The medium and computer code may be those specifically designed and constructed for the purposes of the present invention, or they may be of a type known or available to those skilled in the art. Examples of tangible computer-readable media include, but are not limited to: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices specifically configured to store or store and execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, and ROM and RAM devices. Examples of computer code include machine code (such as that generated by a compiler), and files containing high-level code executed by a computer using an interpreter. Embodiments of the present invention may be implemented wholly or partially as machine-executable instructions, which may be in program modules executed by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In a distributed computing environment, program modules may be physically located in a local, remote, or both setting.

[0090] Those skilled in the art will recognize that no computing system or programming language is essential to the practice of this invention. They will also recognize that the aforementioned elements can be physically and / or functionally separated into sub-modules or combined together.

[0091] Those skilled in the art will appreciate that the foregoing examples and embodiments are exemplary and not limiting of the scope of this disclosure. All substitutions, enhancements, equivalents, combinations, and modifications thereof intended to be apparent to those skilled in the art upon reading the specification and studying the accompanying drawings are included within the true spirit and scope of this disclosure. It should also be noted that elements of any claim can be arranged differently, including having multiple dependencies, configurations, and combinations.

Claims

1. A controller for use within a sensor system, the controller being programmed to perform operations including: Receive sensor data from multiple sensor modules; Receive calibration data; Provides transformation data indicating multiple transformations between multiple sensors within the multiple sensor modules; The series of sensor data streams are demultiplexed by using transformed data to homogenize and decouple multiple different types of sensor data from a series of sensor data streams from the multiple sensor modules; Processing demultiplexed sensor data to generate a response includes integrating the demultiplexed sensor data with spatially transformed data to generate a point cloud; and The calibration data is updated based on the received sensor data and external condition data, and the updated calibration data is used to generate transformation data.

2. The controller of claim 1, wherein the operation further comprises activating at least one of the plurality of sensors.

3. The controller of claim 1, wherein the operation further includes receiving external condition data.

4. The controller according to claim 3, wherein the external condition data includes at least one of odometer reading, lighting conditions, vehicle speed reading, external mapping, and congestion indicator.

5. The controller of claim 3, wherein the operation further includes receiving sensor configuration data associated with at least one sensor module among the plurality of sensor modules.

6. The controller of claim 5, wherein the sensor configuration data includes at least one of sensor type, field of view, frame rate, or region of interest associated with at least one of the plurality of sensor modules.

7. The controller of claim 6, wherein the operation further comprises updating calibration data based on the external condition data and the sensor configuration data.

8. The controller of claim 1, wherein the received sensor data is correlated across a subset of sensors within the plurality of sensors.

9. The controller of claim 8, wherein the subset of sensors within the plurality of sensors comprises at least two different types of sensors.

10. The controller of claim 1, wherein the plurality of sensors includes at least one LiDAR sensor.

11. The controller of claim 10, wherein the plurality of sensors includes at least one thermal sensor.

12. The controller of claim 10, wherein the plurality of sensors includes at least one color sensor.

13. The controller of claim 1, wherein the controller is coupled to an autonomous driving system control unit, the autonomous driving system control unit being operable to monitor a plurality of external conditions related to the autonomous driving of the vehicle.

14. A method comprising: The controller receives sensor data from multiple sensor modules. Receive calibration data at the controller; The controller provides transformation data indicating multiple transformations between multiple sensors within the multiple sensor modules; The controller demultiplexes the series of sensor data streams by homogenizing and decoupling multiple different types of sensor data from the series of sensor data streams from the multiple sensor modules using transformed data. The controller processes demultiplexed sensor data to generate a response, which includes integrating the demultiplexed sensor data with spatial transformation data to generate a point cloud; as well as The controller updates the calibration data based on the received sensor data and external condition data, and uses the updated calibration data to generate transformation data.

15. The method of claim 14, further comprising activating at least one of the plurality of sensors.

16. The method of claim 14, further comprising providing external condition data to the controller.

17. The method of claim 16, wherein the external condition information includes at least one of odometer readings, lighting conditions, vehicle speed readings, external mapping, and congestion indicators.

18. The method of claim 16, further comprising receiving sensor configuration data associated with at least one of the plurality of sensor modules.

19. The method of claim 18, wherein the sensor configuration data includes at least one of sensor type, field of view, frame rate, or region of interest associated with at least one of the plurality of sensor modules.

Images