Sensor fusion for determining reliability of autonomous vehicle operation

Abstract

A method for an autonomous vehicle includes: receiving first object data from a first sensor module; receiving second object data from a second sensor module; comparing the first object data to the second object data; determining whether the first object data corresponds to the second object data based on the comparison of the first object data to the second object data; and performing an action on the autonomous vehicle in response to determining that the first object data does not correspond to the second object data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application Ser. No. 16 / 194,722, filed on Nov. 19, 2018, entitled “SENSOR FUSION TO DETERMINE RELIABILITY OF AUTONOMOUS VEHICLE OPERATION,” the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] At least some embodiments disclosed herein relate generally to computing systems for autonomous vehicles, and more particularly, but not limited to, using sensor fusion to determine the operational reliability of sensors of an autonomous vehicle. Background Art

[0004] Autonomous vehicles typically include multiple sensors to help control the autonomous vehicle. In some cases, sensor failure can lead to an accident, collision, or near-collision with the vehicle.

[0005] Multi-sensor fusion is used in computing systems to manage non-ideal inputs from the environment. Sensor fusion can use techniques such as artificial intelligence (AI), pattern recognition, digital signal processing, control theory, and statistical estimation. Data from multiple microelectromechanical systems (MEMS) can be fused to provide applications with increased response and accuracy.

[0006] Advances in sensor technology and processing, combined with improved hardware, have made it possible to fuse sensor data in real time. For example, a system can combine and integrate MEMS accelerometers, gyroscopes, pressure sensors, magnetic sensors, and microphones into a multi-sensor combination with onboard processing and wireless connectivity.

[0007] Data fusion can be performed in different ways depending on the application, typically related to the level of complexity and the number of components being measured. Feature-level fusion involves extracting features from observations or measurements made by different sensors and combining them into a concatenated feature vector. Decision-level fusion takes information from each sensor after each sensor has individually measured or evaluated the target.

[0008] Sensor fusion can also be centralized or decentralized, depending on where the data fusion occurs. In centralized sensor fusion, data is forwarded to a central location for correlation and fusion. In decentralized sensor fusion, each sensor or platform has some degree of autonomy in the decisions resulting from the fusion.

[0009] In centralized fusion, clients forward all data to a central location, and entities at the central location correlate and fuse the data. In decentralized fusion, clients handle all data fusion. Combinations of centralized and decentralized systems also exist. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] These embodiments are illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which like references indicate similar elements.

[0011] Figure 1 An autonomous vehicle is shown including sensor modules providing data to a central processing unit according to one embodiment.

[0012] Figure 2 A sensor module according to one embodiment is shown that includes sensors that provide raw data to a processor.

[0013] Figure 3 An autonomous vehicle is shown including a central processing device that receives data from sensor modules according to one embodiment.

[0014] Figure 4 A vehicle is shown collecting data during operation according to one embodiment.

[0015] Figure 5 A vehicle is shown according to one embodiment including sensor modules that provide data to a central processing unit and control system.

[0016] Figure 6 A server according to one embodiment is shown that stores event data received from vehicles.

[0017] Figure 7 An example of a vehicle including sensor modules and configured using an artificial neural network (ANN) model is shown according to one embodiment.

[0018] Figure 8 A method of determining whether object data from a first sensor module corresponds to object data from a second sensor module according to one embodiment is shown.

[0019] Figure 9 An autonomous vehicle is shown that is controlled and / or configured in response to determining that object data from a first sensor module of the vehicle does not correspond to object data from a second sensor module of the vehicle, according to one embodiment.

[0020] Figure 10is a block diagram of an autonomous vehicle including one or more various components and / or subsystems, each of which may be updated to configure the vehicle and / or perform other actions associated with the vehicle in various embodiments.

[0021] Figure 11 is a block diagram of a centralized autonomous vehicle operating system, according to various embodiments. DETAILED DESCRIPTION

[0022] Correct operation of autonomous vehicle systems is crucial in a variety of situations. For example, correct vehicle navigation is crucial to avoid collisions with people or other vehicles.

[0023] Practices for safe vehicle system operation are becoming more standardized. For example, the automotive industry is adopting standardized practices for designing and testing products. One example is ISO 26262, which provides an automotive-specific international standard focused on safety-critical components. ISO 26262, a derivative of IEC 61508, is a generic functional safety standard for electrical and electronic (E / E) systems.

[0024] The increasing complexity of the automotive industry is driving efforts to deliver systems that meet safety requirements. For example, modern vehicles use drive-by-wire systems, such as throttle-by-wire. When the driver depresses the accelerator pedal, a sensor on the pedal sends a signal to an electronic control unit. This control unit analyzes several factors, such as engine speed, vehicle speed, and pedal position. It then transmits a command to the throttle body. However, testing and validating systems like throttle-by-wire presents technical challenges.

[0025] Existing vehicles often utilize full system redundancy to improve operational reliability and comply with the ISO 26262 automotive safety and reliability standard. For example, in many cases, vehicles utilize dual or triple full system redundancy to enhance system reliability and safety. In one example, existing vehicles utilize dual or triple redundancy within the same components. For example, two or three cameras are used to detect any camera malfunctions; two or three radars are used to detect any radar malfunctions; and two or three lidars are used to detect any lidar malfunctions.

[0026] However, the aforementioned full system redundancy has the technical problem that hardware requirements and costs are significantly increased due to full system redundancy. In addition, since multiple redundant systems need to be maintained, the complexity of maintenance is also increased.

[0027] Various embodiments disclosed herein address the aforementioned technical issues by using smart sensors that perform local sensor data analysis within the sensor itself and then report detected objects. Additionally, a central sensor health analysis component (e.g., a processor) determines the health status of the smart sensors by comparing detected objects between sensors (e.g., comparing a first sensor to a second sensor).

[0028] Although sensors such as cameras, radars, and lidars will not always detect all objects simultaneously, it is expected that there will be statistical matches between objects detected by the sensors (e.g., when looking at the data sample size taken across multiple object detection events). For example, during daytime operation, it is expected that cameras, radars, and lidars will detect and report the same objects in their field of view (e.g., cars and pedestrians) most of the time (e.g., greater than a threshold percentage of events). Further, for example, during nighttime operation, radars and lidars are expected to reliably detect and report objects detected in the same field of view (e.g., cars).

[0029] Determining that a statistical detection relationship (e.g., a statistical correlation within predetermined limits) between sensors (e.g., smart sensors) fails to maintain can indicate the presence of a fault in one of the sensors or in one of the sensor fusion processing components (e.g., a processor that performs statistical correlation of object data). In response to determining the presence of such a fault, various actions can be taken (e.g., the operation of the vehicle can be altered). In one example, a safety hazard signal is triggered (e.g., providing an alert to passengers and / or sending a communication to a central server that monitors vehicle operation of multiple vehicles).

[0030] In one example, the action taken is to disable the vehicle's autonomous operating mode. In one example, a person is required to take control of the vehicle. In one example, the action is to automatically stop the vehicle and then disable autonomous mode.

[0031] Various embodiments herein offer cost savings by avoiding full system redundancy. Rather than duplicating the entire vehicle operating system, a lower-cost health analysis solution is provided for the sensor and sensor fusion portion of the system. This solution does not require sensor and sensor fusion redundancy. For example, in addition to the sensor and sensor fusion processing components, only that portion of the system requires redundancy.

[0032] In one example, one advantage can be reduced vehicle wiring costs. For example, providing object data results from sensor modules without the need to transmit high-speed raw data signals from the sensors (e.g., as required by redundant raw video data feeds) reduces vehicle wiring requirements. For example, only detected objects need to be transmitted (rather than the raw data used to detect the objects). This significantly reduces the bit rate data bandwidth required for vehicle wiring (e.g., it avoids the need for high-speed wiring to connect smart sensor modules to central processing units). This saves additional costs at the vehicle system level.

[0033] In one example, another advantage provided is a smaller size central processing unit. For example, object data comparison requires significantly less processing power than processing raw sensor data feeds in the central processing unit.

[0034] In various embodiments, system-wide redundancy is avoided based on statistical correlations of similar components (e.g., redundancy in components that are statistically expected to generate the same or similar results under certain conditions). For example, three different components (e.g., a camera, a radar, and a lidar) are expected to have correlations in the objects they identify and report under certain types of conditions, but not under other types of conditions (e.g., a sensor module reports the same or similar data for objects such as cars and pedestrians during the day, but does not report the same or similar data for objects at night and / or for other different types of objects). For example, the expected statistical correlation may vary depending on the type of object (e.g., radar or light reflecting objects versus non-reflective objects). The object types (used to provide result data from the sensor modules) can be stored in a database, for example, in the sensor module that processes the raw sensor data (e.g., from the lidar).

[0035] Various embodiments disclosed herein relate to systems and methods for autonomous vehicles. The vehicle can be, for example, any of several types of autonomous vehicles (e.g., a car, truck, airplane, drone, boat, etc.).

[0036] In one embodiment, a method for an autonomous vehicle includes receiving first object data from a first sensor module and receiving second object data from a second sensor module. Comparing the first object data and the second object data. For example, the comparison can be performed by a processor (e.g., a CPU, a GPU, or an FPGA). Based on the comparison of the first object data and the second object data, determining whether the first object data corresponds to the second object data. For example, the determination can be based on comparing a statistical correlation with a threshold value (e.g., whether the correlation is within one or more predetermined limits). In response to determining that the first object data does not correspond to the second object data (e.g., due to a failure of a sensor component or sensor fusion processing), performing an action for the autonomous vehicle. For example, the action can be a change in control of an operating characteristic of the vehicle, such as a change in navigation or engine control.

[0037] In some embodiments, the operation or configuration of the vehicle is controlled based on the determination of a sensor and / or sensor fusion failure. For example, one or more systems of the vehicle may be controlled by a host system. Data regarding the operation of the vehicle may be collected by a memory device. The collected data is generated by one or more systems. After collection, the data is stored in the memory device.

[0038] In one embodiment, an autonomous vehicle includes sensors that collect sensor data. The vehicle's non-volatile memory device is used to receive various data to be stored (e.g., raw sensor data and / or results of processing the raw sensor data, such as data regarding the type or location of detected objects). One or more computing devices can control the operation of the vehicle. At least one of these computing devices is used to control the collection of data generated by the vehicle during operation via the non-volatile memory device. This data may include sensor data. For example, the computing device is a controller of the memory device, and the controller executes instructions in the firmware of the memory device. After collecting the data, the memory device stores the data (e.g., using non-volatile memory such as 3DXP memory).

[0039] Figure 1 An autonomous vehicle 610 is shown according to one embodiment and includes sensor modules 612, 614 providing data to a central processing device 618. Each sensor module 612, 614 provides object data to the central processing device 618. One or more other sensors 616 provide sensor data to the central processing device 618. In one embodiment, the sensors 616 provide raw sensor data and / or object data.

[0040] Central processing unit 618 compares the object data received from each sensor module 612, 614. In one example, the comparison is based on the detection of a single object in the field of view of sensor module 612 and sensor module 614. In one example, the comparison is based on a correlation determined based on a number of events, each event associated with the detection of a different object. Central processing unit 618 uses the comparison of the object data to determine whether the object data from sensor module 612 corresponds to the object data from sensor module 614.

[0041] If the object data matches, the central processing unit 618 determines that the system operation is reliable and that each sensor module 612, 614 is operating correctly. If the object data does not match, the central processing unit 618 determines that a system fault exists. For example, the fault may exist in either or both of the sensor modules 612 and 614.

[0042] In one example, object data from sensor modules 612, 614 is determined to be a match if the data satisfies a statistical correlation. In one example, a correlation is determined and compared to a predetermined limit or threshold.

[0043] In response to determining that the object data from each sensor module 612, 614 does not match, the central processing device 618 sends a signal to the host processing device 620. The signal indicates that a fault has been identified.

[0044] In one embodiment, host processing device 620 performs one or more actions in response to receiving the signal indicating fault identification. In various embodiments, host processing device 620 sends a control signal to one or more vehicle systems 622 that alters one or more aspects of the operation of autonomous vehicle 610. In one example, navigation, brake control, and / or engine control are altered. In another example, the configuration of one or more vehicle systems 622 is updated. In one example, the firmware of one of vehicle systems 622 is updated. In one example, at least a portion of the firmware is downloaded from a central server using wireless communication.

[0045] In one embodiment, the action performed by host processing device 620 is to initiate the execution of one or more diagnostic tests. For example, diagnostic tests of sensor modules 612 and / or 614 may be executed. In another example, diagnostic tests of central processing unit 618 may be executed, alternatively and / or additionally.

[0046] In one embodiment, sensors 616 provide data to central processing unit 618 that is used to determine the operating context of autonomous vehicle 610. For example, sensors 616 may provide data indicating whether the vehicle is currently operating during the day or at night. This provides contextual data that central processing unit 618 can use to compare object data from sensor modules 612 and 614.

[0047] In one example, the central processing device 618 is a system on a chip (SOC), a field programmable gate array (FPGA), a CPU, or a graphics processing unit (GPU). In one example, the central processing device 618 stores runtime data in a volatile memory device (e.g., a DRAM device). In one example, the central processing device 618 may include logic circuitry that implements at least part of the various embodiments described herein.

[0048] In one example, host processing device 620 is a system on a chip (SOC), a field programmable gate array (FPGA), a CPU, or a graphics processing unit (GPU). In one example, host processing device 620 stores data in a volatile memory device. For example, the stored data is received from vehicle system 622. Host processing device 620 can be, for example, a processing core of a processor, an execution unit, or the like.

[0049] In one example, the central processing unit 618 and / or the host processing unit 620 may include hardware such as one or more integrated circuits and / or discrete components, buffer memory, cache memory, or a combination thereof. The host or central processing unit may be a microcontroller, a dedicated logic circuit (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.), or other suitable processor.

[0050] In one embodiment, each sensor module 612, 614 is a camera. Each camera shares a common field of view. During normal operation, it is expected that the cameras will detect the same objects most of the time. If the object data does not match when compared to a predetermined threshold, a system failure is determined. In one embodiment, if the system is operating correctly, the compared object data is expected to exceed a threshold percentage (e.g., 60%). In one embodiment, the threshold percentage is 99% for events occurring during the day and 70% for events occurring at night.

[0051] Figure 2Sensor module 612 is shown according to one embodiment and includes sensor 624 that provides raw data to processor 626. In one example, sensor module 614 is the same as or similar to sensor module 612. In another example, sensor 616 is similar to sensor module 612. In one example, sensor 624 is a camera. In another example, sensor 624 is a radar or lidar sensor. Processor 626 is, for example, a microprocessor, a GPU, and / or an FPGA.

[0052] Processor 626 executes one or more processes to evaluate the raw data from sensor 624. As a result of this processing, processor 626 provides object data. The object data is transmitted to a central processing unit. The object data includes, for example, location data and / or object type data. It may also include other metadata characterizing the detected object.

[0053] During operation, the processor 626 accesses the memory 628. The memory 628 is, for example, a volatile memory and / or a non-volatile memory.

[0054] The memory 628 includes an object database 630. The database 630 includes a library of object types. When provided with the results of evaluating the raw data from the sensor 624, the processor 626 can retrieve predetermined object types from the library.

[0055] In one embodiment, neural network 632 is stored in memory 628. Neural network 632 can be used to provide one or more inputs to processor 626 when evaluating raw data from sensor 624. In one embodiment, neural network 632 receives input from sensor 616, which is used to determine the operating context of autonomous vehicle 610, as output of neural network 632. The output is provided to processor 626 for use in evaluating the raw data from sensor 624.

[0056] Figure 3 Autonomous vehicle 170 is shown including central processing device 618 that receives data from sensor module 178, according to one embodiment. Sensor module 178 is an example of sensor modules 612, 614.

[0057] In one embodiment, central processing device 618 receives object data from sensor module 178. Based on a comparison of the object data, central processing device 618 sends a signal to host system 172 indicating that a system fault has been detected. Host system 172 is an example of host processing device 620. In response to receiving the signal indicating the system fault, host system 172 sends a signal to vehicle system 176 via bus 174, which causes vehicle system 176 to change an operational aspect of autonomous vehicle 170. In another embodiment, host system 172 may change the operation and / or update the configuration of vehicle system 185. In another embodiment, data used to update the software and / or configuration of vehicle system 185, host system 172, and / or vehicle system 176 is downloaded from server 188 using wireless interface 182 and stored in memory device 180.

[0058] Data is generated by various systems of vehicle 170, including, for example, vehicle systems 176, 183, and 185. For example, vehicle systems 176 and 183 utilize one or more sensor modules 178. Each system may process the sensor data and then transmit the resulting data (e.g., object data) to another component of the vehicle, and / or may pass the raw sensor data to another component.

[0059] In one example, data from the vehicle system 176 is sent to the host system 172 via the communication bus 174. The host system 172 controls various operations of the vehicle system 176 (e.g., image processing and recognition systems). The host system 172 sends commands to the vehicle system 176 via the bus 174. For example, the commands may be used to control engine operation or to implement steering control.

[0060] Host system 172 may also control other systems, such as vehicle system 183 and / or vehicle system 185. Host system 172 may receive responses and / or data from these systems via one or more communication paths (eg, bus 174) of vehicle 170.

[0061] In one embodiment, memory device 180 monitors communications on bus 174. Memory device 180 can collect selected data from bus 174. In one example, memory device 180 listens for certain predetermined types of data extracted from bus 174. The extracted data is stored by memory device 180.

[0062] In one embodiment, memory device 180 stores the collected data at a predetermined physical location 186. Physical location 186 may correspond to the physical location or position of vehicle 170. In other cases, physical location 186 may correspond to a specific predetermined component or component type (e.g., a specific type of memory storage medium).

[0063] In one embodiment, memory device 180 collects data stored in volatile memory 184. In one example, host system 172 uses volatile memory 184. For example, host system 172 can collect data from various systems of vehicle 170. This data can be stored in volatile memory 184. In one embodiment, memory device 180 collects data from sensor module 178.

[0064] In one embodiment, after the data is collected by the memory device 180, the data is stored in the memory device 180. The stored data may be later transmitted to the server 188 via the wireless interface 182.

[0065] In one embodiment, server 188 analyzes the collected data received from memory device 180. Based on the analysis, one or more configurations of vehicle 170 may be implemented or updated. For example, server 188 may send commands to memory device 180 via wireless interface 182 regarding configuration changes to be implemented in vehicle 170. Memory device 180 may control the implementation of the configuration changes.

[0066] In another embodiment, based on analysis of the collected data, the server 188 sends a communication to the host system 172. In one example, the communication may be sent via a wireless interface (not shown). The communication may be, for example, a command to update the configuration of the vehicle 170. The update may be performed under the control of the host system 172. In one embodiment, the update is performed in response to a determination by the central processing unit 618 of a system failure.

[0067] In one embodiment, the collected data is stored in a non-volatile memory device, such as memory device 180. After storage, the data is removed from the non-volatile memory device and analyzed, for example, by a server, such as a server in the cloud, such as server 188.

[0068] In one embodiment, analysis of the stored data is used to identify unsafe aspects or designs of the software operating the vehicle (e.g., software executing on the host system 172 to control navigation). In response to the analysis, control and / or configuration actions can be performed on the vehicle (e.g., via the wireless interface 182). In some cases, the same or similar control and / or configuration actions can be performed on other vehicles (e.g., vehicles of the same type and / or using the same or similar software). In various embodiments, the vehicle can be, for example, any of several types of autonomous vehicles (e.g., cars, trucks, airplanes, drones, boats, etc.).

[0069] In various embodiments, the collected data can be stored by any of several non-volatile memory systems of the vehicle (e.g., at physical location 186). An example of a memory system is a storage system, such as a solid-state drive (SSD). In some embodiments, the memory system is a hybrid memory / storage system. Generally, a vehicle can utilize a memory system that includes one or more memory devices. The memory device can include a medium. The medium can be a non-volatile memory device, such as a NAND.

[0070] In one embodiment, a cloud service (sometimes simply referred to as the cloud) is used to receive data collected from a vehicle and then analyze the data. For example, when a vehicle's automatic emergency braking system is activated, event data (e.g., location) of the braking event is transmitted to, for example, a server or other computing device in the cloud service.

[0071] In one embodiment, event data collected by memory device 180 is received from the vehicle and analyzed. For example, pattern recognition can be used on the received event data. In one embodiment, machine learning is used to identify patterns or regularities in the data. In some cases, the pattern recognition system can be trained from labeled training data (e.g., supervised learning). In other cases, when no labeled data is available, other algorithms can be used to identify previously unknown patterns (e.g., unsupervised learning).

[0072] In one embodiment, an event occurring on a vehicle is detected based on a measurement of the vehicle's brake pedal. For example, an event can be identified based on foot pressure or movement compared to a predetermined threshold. In another embodiment, an event is identified based on the vehicle's deceleration rate. For example, if the deceleration rate exceeds a predetermined threshold, an event (e.g., a near-crash) is identified. In another example, the deceleration rate is observed over a predetermined time period (e.g., the deceleration rate is averaged over the time period). In response to detecting the event, system memory data is downloaded. Detection of such an event can trigger the collection of data by memory device 180.

[0073] In one embodiment, at least one action is performed in response to analyzing the event data. For example, a communication may be sent to the vehicle 170 (e.g., from which the collected data was obtained) to configure the vehicle's software and / or disable one or more features of the vehicle or software.

[0074] In one embodiment, in response to identifying unsafe software (e.g., determined based on pattern recognition or other analysis of collected data, such as object data from sensor module 178), the server can perform one or more actions. For example, the server can send a communication to the current vehicle. In one embodiment, the communication can cause the current vehicle to take corrective action, such as terminating autonomous navigation mode, braking or changing course, or updating the vehicle's firmware (e.g., via an over-the-air firmware update).

[0075] In one embodiment, in response to receiving a communication from a server that has analyzed the data collected by memory device 180 , vehicle 170 may turn off its autonomous driving mode, use a backup system, and / or activate a braking system to stop the vehicle.

[0076] In another embodiment, the cloud service may send a communication to a server or other computing device (eg, a central monitoring service) that monitors the operating status of other vehicles. The communication may, for example, identify that a software component is unsafe.

[0077] In one embodiment, it may be determined whether the vehicle has experienced a braking event and / or has been involved in an accident based on data from one or more sensors or sensor modules of the vehicle. For example, data from a vehicle accelerometer may indicate rapid deceleration of the vehicle (e.g., deceleration exceeding a threshold). In another instance, the data may indicate that an emergency system of the vehicle has been activated, such as an airbag, an emergency braking system, or the like. In some embodiments, any one and / or combination of the aforementioned events may be considered an event for transmitting downloaded data to a server. Data associated with the aforementioned events may be collected by memory device 180.

[0078] In one embodiment, one or more sensor modules (e.g., sensor module 178) on the current vehicle are used to obtain data about braking events and / or objects in the vehicle environment while the vehicle is traveling. Object data from the sensor modules and / or data generated based on analysis of other sensor data and / or other types of data can be wirelessly transmitted (e.g., using a 3G, 4G, or 5G network or other radio-based communication system) to a cloud service, for example, after being automatically collected by memory device 180.

[0079] In one embodiment, in response to identifying unsafe software determined based on the collected data, one or more actions of the vehicle are configured. For example, an over-the-air firmware update may be sent to the vehicle to update the firmware of the vehicle's computing device (e.g., such that the update causes the vehicle to avoid the identified unsafe functionality). In one example, the firmware updates the vehicle's navigation system.

[0080] Figure 4Vehicle 200 is shown collecting data during operation according to one embodiment. Vehicle 200 is an example of autonomous vehicle 610.

[0081] For example, autonomous vehicle 200 detects various objects while driving. Vehicle 200 also detects certain events that trigger the downloading of data (e.g., from system memory) to a non-volatile memory device. In one example, vehicle 200 detects a collision with an object (e.g., another vehicle). In response to detecting the collision, the computing system of vehicle 200 causes the system memory data to be downloaded to the non-volatile memory device. This is done to allow for analysis of the downloaded data after the collision. In one example, any of the aforementioned data can be automatically collected by memory device 180.

[0082] More specifically, in one embodiment, Figure 4 A sensing system is shown located on a vehicle 200 for collecting information about the operation of the vehicle. Any of this collected data may be downloaded from system memory and / or other volatile memory, for example, in response to a detected event, or may be part of data routinely collected during normal vehicle operation.

[0083] For example, a sensing system can be used to collect object and event data to be sent to a cloud service for storage and analysis. As mentioned above, object data can also be sent to a central processing device for comparison with other object data.

[0084] In various embodiments, the sensing system may generally include one or more sensor modules 220, a processor 230, a memory 240, and a communication device (eg, a transmitter 250). In addition to object data, other data may also be collected to be sent to a cloud service.

[0085] In various embodiments, the sensing system may include one or more sensor modules 220 (sometimes referred to herein simply as sensors 220) configured to collect information regarding operational aspects of autonomous vehicle 200, such as velocity, vehicle speed, vehicle acceleration, braking force, braking deceleration, etc. Representative sensors configured to collect information regarding operational driving characteristics may include, but are not limited to, a tachometer, a vehicle speed sensor such as a vehicle speed sensor or a wheel speed sensor, a brake pressure sensor, a fuel flow sensor, a steering angle sensor, etc.

[0086] In various embodiments, the sensing system may additionally or alternatively include one or more sensors 220 configured to collect information about the static environment in which autonomous vehicle 200 operates, such as the presence and contents of physical objects surrounding the vehicle. Physical objects include, for example, signs and traffic signals (e.g., stop signs, construction zones, speed limit signs, stop lights), road lane dividers (e.g., solid and dashed lane markings), and the like. Representative sensors configured to collect such static operating environment information may include outward-facing cameras positioned and oriented such that their respective fields of view capture the information they are configured to collect. For example, a camera configured to capture surrounding signs may be configured to face toward the front or top of autonomous vehicle 200 and face forward (e.g., straight ahead or possibly tilted to the side by up to approximately 45 degrees) so as to capture roadside and overhead signs / traffic signals within its field of view as autonomous vehicle 200 travels forward.

[0087] As another example, cameras configured to capture road lane dividers can be positioned on the sides or outside of the front / rear quarters of autonomous vehicle 200 and can be oriented slightly downward to capture road lane dividers on both sides of autonomous vehicle 200. Additional representative sensors for collecting static operating environment information can include a receiver configured to receive wireless signals from a base station or other transmitter that convey information typically found on signs or otherwise related to the static operating environment of autonomous vehicle 200. Similarly, a global positioning system (GPS) or other location-related sensors can be used to collect information about the static environment in which autonomous vehicle 200 operates, such as which street autonomous vehicle 200 is traveling on, whether the street is a traffic artery (e.g., a highway) or other type, and whether the location is in an urban or rural area.

[0088] In various embodiments, the sensing system may additionally or alternatively include one or more sensors 220 configured to collect information about the dynamic environment in which autonomous vehicle 200 operates, such as information about the presence of moving objects such as other nearby vehicles (e.g., along with the location, direction of travel, speed, and acceleration / deceleration rates of such vehicles), and similar information about the presence of nearby pedestrians. In one embodiment, during the evaluation, sensor modules 612, 614 may use any of the aforementioned information to generate object data.

[0089] Representative sensors configured to collect such dynamic operating environment information may include outward-facing cameras positioned and oriented so that their respective fields of view can capture the information they are configured to collect. For example, outward-facing cameras may be positioned around the perimeter of autonomous vehicle 200 (e.g., at the front, rear, top, sides, and / or quarters) to capture images to which image processing techniques such as vehicle recognition algorithms may be applied. Additionally or alternatively, one or more optical sensors (e.g., lidar, infrared), acoustic wave sensors (e.g., sonar, ultrasonic), or similar detection sensors may be positioned around the vehicle to measure dynamic operating environment information, such as distance, relative velocity, relative acceleration, and similar characteristics of the motion of nearby human or autonomous vehicles.

[0090] In various embodiments, the sensing system may utilize sensors typically found in most autonomous vehicles as sensors 220, such as, but not limited to, sensors configured to measure speed, RPM, fuel consumption, and other characteristics of vehicle operation, as well as sensors configured to detect the presence of other vehicles or obstacles in the vicinity of the vehicle. Sensors 220 may additionally or alternatively include aftermarket sensors installed on autonomous vehicle 200 to facilitate collecting additional information related to driving style.

[0091] In various embodiments, the sensing system of vehicle 200 may further include an onboard processor 230, onboard memory 240, and an onboard transmitter 250. Generally speaking, in various embodiments, processor 230 may be configured to execute instructions stored on memory 240 to process information collected by sensors 200 for subsequent transmission outside of vehicle 200. In various embodiments, onboard processor 230 may additionally or alternatively be configured to execute instructions stored on memory 240 to process information from two or more sensors 220 to generate further information regarding characteristics of objects detected by autonomous vehicle 200. For example, in one embodiment, processor 230 may process operating characteristics (such as braking deceleration) and dynamic environmental characteristics (such as following distance) to avoid debris or animals that suddenly appear in the roadway. It should be appreciated that this is merely an illustrative example, and one of ordinary skill in the art will recognize other ways in which sensor data may be processed by the processor to generate further information regarding objects detected by autonomous vehicle 200.

[0092] In various embodiments, the processor 230 can be configured to pre-process information from the sensors 220 for subsequent off-vehicle transmission via the transmitter 250. The pre-processing activities can include filtering, organizing, and packaging the information from the sensors 220 into one or a combination of a format and communication protocol for efficient wireless transmission. In such embodiments, the pre-processed information can then be transmitted by the transmitter 250 to the off-vehicle vehicle 200 in real-time or at periodic intervals, where it can be received by a nearby manned or autonomous vehicle and / or a remote server. It should be understood that according to various embodiments described later, when configured to transmit the pre-processed information directly to a nearby manned or autonomous vehicle, the transmitter 250 can utilize short-range wireless signals (e.g., Wi-Fi, Bluetooth), and when transmitting the pre-processed information directly to a remote server, the transmitter 250 can utilize long-range signals (e.g., cellular, satellite).

[0093] In one embodiment, processor 230 may be a processor of a memory device (eg, memory device 180 ) that operates independently of a host system to automatically collect data from vehicle 200 .

[0094] Like the sensor 220, in various embodiments, the processor 230 and / or the vehicle-mounted transmitter 250 may be integrally mounted in the vehicle 200 (e.g., a car computer, a connected vehicle), while in other embodiments, the processor 230 and / or the transmitter 250 may be added as an aftermarket feature.

[0095] Figure 5 A vehicle according to one embodiment is shown including a sensor module 737 that provides data to the central processing unit 618 and the control system 731. The sensor module 737 is an example of the sensor modules 612, 614.

[0096] According to one embodiment, in one embodiment, the control system 731 is configured to download data in response to detecting an event associated with the vehicle 711. In one example, the detected event is a system fault determined based on a signal received from the central processing unit 618. In one example, the central processing unit 618 receives object data from each sensor module 737.

[0097] In one embodiment, the control system 731 includes Figure 3 The downloaded data is stored in the memory device 180 , and the downloaded data is collected by the memory device 180 .

[0098] The control system 731 also controls at least one operation of the vehicle (e.g., navigation and / or braking). The control system 731 includes one or more processors 733, non-volatile memory 735, and system memory 729 including volatile memory 727. The system memory 729 may also include data of the vehicle 711 stored in other volatile memory devices (not shown), for example.

[0099] The volatile memory 727 may be, for example, a dynamic random access memory device. The volatile memory 727 is used to store data used by the software during vehicle operation, such as sensor data (e.g., object data) and data regarding the operating state of the control system 731. The object data is collected, for example, by the sensor module 737.

[0100] Non-volatile memory 735 is used to store data from volatile memory 727 after an event is detected. Examples of non-volatile memory 735 include 3D XPoint memory and NAND flash memory, as well as removable memory devices such as solid-state drives.

[0101] As described above, the vehicle 711 uses the sensor module 737 and the control system 731 to detect various events. These events can include, for example, a collision of the vehicle 711 with an object. Other events can be the detection of system activity, such as activation of an emergency braking system or sensor data exceeding a threshold. When an event is detected, the data stored in the volatile memory 727 is downloaded to the non-volatile memory 735. Examples of stored data include sensor data, such as data about acceleration, vehicle speed, and braking status, lidar and video inputs, position data, and control system states, such as program counters indicating the operating state of software executed during the event. In some embodiments, the downloading of volatile data to non-volatile data is performed within a specific duration (e.g., less than five to ten seconds) after the control system 731 detects the event.

[0102] In one embodiment, the control system 731 of the vehicle 711 can be configured to extract event data from non-volatile memory and analyze the data. The event data is extracted and downloaded as an event is detected. For example, the event data can be wirelessly transmitted to a central server that monitors the movement and operation of the vehicle 711.

[0103] In one embodiment, only a predetermined portion of system memory 729 is downloaded. Processor 733 may determine the predetermined portion to download after detecting an event. In another embodiment, data from system memory 729 may be downloaded portion by portion in an order based on the priority of the data. The priority may be determined by the type of event detected by control system 731.

[0104] Figure 6A server 101 is shown storing event data received from a vehicle according to one embodiment. In one example, the event data corresponds to a determination by a central processing unit that object data from a sensor module of a vehicle 111 does not match. For example, vehicle 111 may detect object 155 using a sensor module that sends the object data to the central processing unit.

[0105] In one example, if a sensing system failure of the vehicle 111 is detected, the event data 160 (e.g., stored in Figure 5 volatile memory 727) from system memory 729 to non-volatile memory 735. Event data 160 may include data such as sensor data 103 (obtained from a sensor module of vehicle 111), location data 163, data related to activation of an emergency braking system, or data output from an artificial neural network. During operation, vehicle 111 collects data about detected objects, such as object 155 and object 157.

[0106] Event data 160 is retrieved from non-volatile memory and communicated to server 101 via wireless communication network 102. Server 101 analyzes event data 160 to determine the cause of the sensor failure (e.g., a software failure). For example, server 101 can use simulation to determine whether the software executing on vehicle 111 is operating correctly. Server 101 can select the type of event data 160 to download for analysis (e.g., sensor data or control system status). One example of wireless communication network 102 is a cellular telephone network.

[0107] The collected event data received from the vehicles 111 by the server 101 is analyzed. For example, the analysis may include pattern recognition or other data analysis (eg, determining associations of the event data with other data).

[0108] In response to identifying a software fault, at least one action is performed. For example, the server 101 may send a communication to the vehicle 111 causing the vehicle to reconfigure its software.

[0109] In one example, a vehicle can communicate with server 101 to submit event data via a wireless connection 115 to an access point (or base station) 105. Wireless connection 115 can be implemented via a wireless local area network, a cellular communication network, and / or a communication link 107 to a satellite 109 or a communication balloon.

[0110] Optionally, vehicle 111 has self-learning capabilities. After a period of driving on the road, vehicle 111 can reconfigure its software. In one embodiment, centralized server 101 can be operated by the factory, manufacturer, or manufacturer of vehicle 111 or a supplier of autonomous driving and / or advanced driver assistance systems for vehicle 111.

[0111] Figure 7 An example of a vehicle 111 is shown that includes a sensor module 137 and is configured using an artificial neural network (ANN) model according to one embodiment. The sensor module 137 provides object data to a computer 131. The computer 131 includes a processor and / or software processing that performs the role of a central processing unit.

[0112] The vehicle 111 includes an infotainment system 149, a communication device 139, one or more sensors 137, and a computer 131 connected to some controls of the vehicle 111, such as steering control 141 for the direction of the vehicle 111, braking control 143 for stopping the vehicle 111, acceleration control 145 for the speed of the vehicle 111, etc.

[0113] The computer 131 of the vehicle 111 includes one or more processors 133 , memory 135 storing firmware (or software) 127 , the ANN model 119 , and other data 129 .

[0114] In one example, firmware 127 is updated via an over-the-air update in response to a communication from server 101 sent in response to identifying faulty software. Alternatively and / or additionally, other firmware of various computing devices or systems of vehicle 111 may be updated.

[0115] One or more sensors 137 may include a visible light camera, an infrared camera, a lidar, a radar or sonar system, and / or peripheral sensors configured to provide sensor input to the computer 131. Modules of the firmware (or software) 127 executed in the processor 133 apply the sensor input to the ANN defined by the model 119 to generate output that identifies or classifies an event or object (e.g., an image or video clip) captured in the sensor input. As discussed above, data from this identification and / or classification may be included in the data collected by the memory device (e.g., memory device 180) and sent from the vehicle to the server 101.

[0116] Alternatively and / or additionally, the autonomous drive module of the firmware (or software) 127 can use the identification of the unsafe software to generate a response. The response can be a command to activate and / or adjust one of the vehicle controls 141, 143, and 145. In one embodiment, the response is an action performed by the vehicle, wherein the action has been configured based on an update command from the server 101 (e.g., the update command can be generated by the server 101 in response to determining a software failure of the vehicle 111 based on analysis of event data). In one embodiment, the vehicle is configured before generating the control response. In one embodiment, the configuration of the vehicle is performed by updating the firmware of the vehicle 111. In one embodiment, the configuration of the vehicle includes updating a computer model stored in the vehicle 111 (e.g., the ANN model 119).

[0117] In one embodiment, the server 101 stores the received sensor input as part of the sensor data for subsequent further training or updating of the ANN model 119 using the supervised training module 117. When an updated version of the ANN model 119 is available in the server 101, the vehicle 111 can use the communication device 139 to download the updated ANN model 119 for installation in the memory 135 and / or for replacing the previously installed ANN model 119. These actions can be performed in response to determining that the vehicle 111 failed to properly detect an object and / or in response to identifying unsafe software.

[0118] In one example, during autonomous driving, the output of the ANN model 119 may be used to control (eg, 141 , 143 , 145 ) the acceleration of the vehicle (eg, 111 ), the speed of the vehicle 111 , and / or the direction of the vehicle 111 .

[0119] In one example, data obtained from a vehicle's sensors can include images of objects captured using a camera that uses visible light or infrared light, or a sonar, radar, or lidar system. In one embodiment, image data obtained from at least one of the vehicle's sensors is part of the data collected from the vehicle being analyzed. In some cases, an ANN model is configured for a specific vehicle based on the sensor and other collected data.

[0120] Figure 8 A method for determining whether object data from a first sensor module corresponds to object data from a second sensor module according to one embodiment is shown. For example, the first sensor module is sensor module 612 and the second sensor module is sensor module 614. For example, Figure 8 The method can be Figure 1-7 implemented in the system.

[0121] Figure 8The method may be performed by processing logic, which may include hardware (e.g., a processing device, circuitry, dedicated logic, programmable logic, microcode, device hardware, integrated circuits, etc.), software (e.g., instructions running or executed on a processing device), or a combination thereof. In some embodiments, Figure 8 The method is at least partially performed by the central processing unit 618.

[0122] Although shown in a particular order or sequence, unless otherwise specified, the order of the processes may be modified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. Additionally, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are also possible.

[0123] In block 601 , first object data is received from a first sensor module of an autonomous vehicle. For example, the central processing device 618 receives object data regarding a location and / or type of an object from the sensor module 612 .

[0124] In block 603 , second object data is received from a second sensor module of the autonomous vehicle. For example, the central processing unit 618 receives object data regarding the location and / or type of an object from the sensor module 614 .

[0125] In block 605 , the first object data is compared to the second object data. For example, the central processing device 618 compares the object data from the sensor modules 612 , 614 based on statistical correlation.

[0126] In block 607, based on the comparison of the object data, it is determined whether the first object data corresponds to the second object data. For example, central processing unit 618 compares the statistical correlation to a predetermined threshold or limit to determine whether the object data from sensor module 612 corresponds to (e.g., matches) the object data from sensor module 614.

[0127] In response to determining that the first object data does not correspond to the second object data, an action associated with operation of the autonomous vehicle is performed in block 609. For example, central processing device 618 sends a signal to host processing device 620 to cause a change in operation of one or more vehicle systems 622.

[0128] In one embodiment, a method for an autonomous vehicle (e.g., autonomous vehicle 610) includes: receiving first object data from a first sensor module (e.g., sensor module 612); receiving second object data from a second sensor module (e.g., sensor module 614); comparing the first object data with the second object data; determining whether the first object data corresponds to the second object data based on the comparison of the first object data with the second object data; and performing an action on the autonomous vehicle in response to determining that the first object data does not correspond to the second object data.

[0129] In one embodiment, the first sensor module comprises processing means configured to process raw data from the sensor.

[0130] In one embodiment, the first object data includes a location and an object type of an object detected by the first sensor module, and the second object data includes a location and an object type of an object detected by the second sensor module.

[0131] In one embodiment, comparing the first object data to the second object data includes determining an association between the first object data and the second object data.

[0132] In one embodiment, a non-transitory computer storage medium stores instructions that, when executed on a computing device (e.g., central processing unit 618), cause the computing device to perform a method for an autonomous vehicle, the method comprising: receiving first object data from a first sensor module; receiving second object data from a second sensor module; comparing the first object data to the second object data; determining whether the first object data corresponds to the second object data based on the comparison of the first object data to the second object data; and in response to determining that the first object data does not correspond to the second object data, performing an action associated with the autonomous vehicle.

[0133] In one embodiment, a processing device controls navigation of an autonomous vehicle, and performing an action includes causing the processing device to change a navigation path of the autonomous vehicle.

[0134] In one embodiment, the autonomous vehicle includes a first sensor module including a first sensor (e.g., sensor 624) and a first processing device (e.g., processor 626), wherein the first processing device evaluates sensor data from the first sensor (e.g., Figure 2the first object data corresponding to the second object data; a second sensor module comprising a second sensor and a second processing device, wherein the second processing device evaluates sensor data from the second sensor to provide second object data; a central processing device configured to receive the object data provided by the sensor module of the autonomous vehicle; and a memory storing instructions configured to instruct the central processing device to: receive the first object data; receive the second object data; compare the first object data with the second object data; based on the comparison, determine whether the first object data corresponds to the second object data; and in response to determining that the first object data does not correspond to the second object data, perform an action associated with operation of the autonomous vehicle.

[0135] In one embodiment, the first object data comprises an object type selected from a plurality of object types.

[0136] In one embodiment, the first sensor module further includes a first memory (eg, memory 628 ) storing data for a neural network (eg, neural network 632 ), and the first processing device evaluates the sensor data from the first sensor using the neural network.

[0137] In one embodiment, comparing the first object data to the second object data includes performing a statistical correlation of the first object data and the second object data.

[0138] In one embodiment, the instructions are further configured to direct the central processing device to determine a context of the autonomous vehicle based on sensor data from at least one sensor other than the first sensor and the second sensor, and determining whether the first object data corresponds to the second object data is based in part on the context.

[0139] In one embodiment, the autonomous vehicle further includes a host processing device, performing the action includes sending a signal to the host processing device, and sending the signal causes the host processing device to change a configuration of a vehicle system of the autonomous vehicle.

[0140] In one embodiment, the first sensor is a lidar or radar sensor and the second sensor is a camera.

[0141] In one embodiment, the first processing device is a system on chip, a field programmable gate array, a graphics processing unit, or an application specific integrated circuit.

[0142] In one embodiment, the first object data comprises a position of an object detected by a first sensor module, and the second object data comprises a position of an object detected by a second sensor module.

[0143] In one embodiment, determining whether the first object data corresponds to the second object data includes comparing a number of times the first object data matches the second object data to a threshold.

[0144] In one embodiment, the instructions are further configured to instruct the central processing device to determine whether the current time of the autonomous vehicle corresponds to daytime or nighttime, and the threshold is a first threshold during daytime and a second threshold during nighttime.

[0145] In one embodiment, the threshold is a predetermined percentage of the total number of comparisons.

[0146] In one embodiment, performing an action associated with operation of the autonomous vehicle includes sending a signal that causes a host processing device of the autonomous vehicle to perform a diagnostic test of at least one of the first sensor module or the second sensor module.

[0147] In one embodiment, the first object data comprises a first type of detection object, wherein the second object data comprises a second type of detection object, and comparing the first object data to the second object data comprises comparing the first type to the second type.

[0148] In one example, the central or host processing device may be one or more general-purpose processing devices such as microprocessors, central processing units, and the like. More specifically, the central or host processing device may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets, or a processor that implements a combination of instruction sets. The central or host processing device may also be one or more special-purpose processing devices such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, and the like. The central or host processing device is configured to execute instructions for performing the operations and steps discussed herein.

[0149] Figure 9 An autonomous vehicle 303 is shown being controlled and / or configured in response to determining that object data from a first sensor module of the vehicle does not correspond to object data from a second sensor module of the vehicle, according to one embodiment. Sensor module 306 provides object data for comparison to central processing unit 618. Sensor module 306 is an example of sensor modules 612, 614.

[0150] Based on comparing the object data from the sensor module 306, the central processing unit 618 sends a signal to the controller 307. The controller 307 is an example of the host processing unit 620. In one example, in response to receiving the signal, the controller 307 causes a display to be displayed to the user on one or more display devices 308. In one example, the display presents information in a user interface indicating to the user that a fault has been detected. In one example, the display presents an indication of the component identified as faulty. The display device 308 is an example of the vehicle system 622.

[0151] Memory 309 stores computer model 312 and training data 314. Computer model 312 is an example of neural network 632. In one example, computer model 312 and / or training data 314 are used to configure and / or update one or more sensor modules 306. In one example, updates to computer model 312 and / or training data 314 are received from server 301.

[0152] The systems of vehicle 303 include a display device 308 or other devices, systems, or components. In one example, a controller 307 is included in memory device 180 and controls the collection of data from various systems of vehicle 303 .

[0153] The server 301 may store event data 160 that is collected by, for example, the controller 307 and sent to the server 301. The event data may include object data provided by the central processing device 618.

[0154] Server 301 can use event data 160 and / or other data provided by central processing device 618 to determine that sensor components and / or sensor software (e.g., software executed on central processing device 618) of vehicle 303 are operating incorrectly. In response to this determination, server 301 can cause controller 307 to terminate the autonomous navigation mode. Other actions can be performed in response to this determination, including, for example, configuring vehicle 303 by updating firmware 304, updating computer model 312, updating data in database 310, and / or updating training data 314.

[0155] Controller 307 can receive data collected by one or more sensor modules 306. Sensor modules 306 can be installed in autonomous vehicle 303, for example. Sensor modules 306 can include, for example, cameras, microphones, motion detectors, and / or cameras. At least some of the sensor modules can provide data associated with newly detected objects while vehicle 303 is traveling.

[0156] The sensor module 306 may provide various types of data for collection by the controller 307 and / or the central processing unit 618. For example, the collected data may include image data from a camera and / or audio data from a microphone.

[0157] In one embodiment, the central processing unit 618 and / or the controller 307 analyzes the data collected from the sensor module 306. The analysis of the collected data includes providing some or all of the collected data as one or more inputs to the computer model 312. The computer model 312 can be, for example, an artificial neural network trained via deep learning. In one example, the computer model is a machine learning model trained using training data 314. The computer model 312 and / or the training data 314 can be stored, for example, in the memory 309. The output from the computer model 312 can be transmitted to the server 301 as part of the object data.

[0158] In one embodiment, memory 309 stores a database 310 that may include data collected by sensor module 306 and / or data received by communication interface 305 from a computing device (e.g., server 301). In one example, the communication may be used to wirelessly transmit the collected data from sensor module 306 to server 301. The received data may include configuration, training, and other data used to configure controller 307 to control display device 308 or other vehicle systems. The received data may also be used to update the configuration of a machine learning model stored as computer model 312 in memory 309.

[0159] In one embodiment, firmware 304 controls the operation of, for example, controller 307 in controlling components of vehicle 303. Controller 307 may also, for example, execute firmware 304 to perform operations in response to communications from server 301. Autonomous vehicle 303 includes volatile dynamic random access memory (DRAM) 311 for storing runtime data and instructions used by controller 307. The runtime data and / or instructions may be part of the data downloaded in response to detecting an event.

[0160] At least some embodiments of the systems and methods disclosed herein can be implemented using computer instructions (e.g., firmware) executed by a controller or processor. In some cases, hardware circuitry can be used to implement at least some of the functionality of the firmware. The firmware can initially be stored in a non-volatile storage medium or another non-volatile device and loaded into volatile DRAM and / or intra-processor cache for execution. In one example, the firmware can be configured to control the collection of data from the vehicle.

[0161] Figure 10is a block diagram of an autonomous vehicle (e.g., vehicle 610) that includes one or more various components and / or subsystems, each of which, in various embodiments, can be updated to configure the vehicle and / or perform other actions associated with the vehicle (e.g., change the operation of a vehicle system, update a configuration, and / or perform other actions in response to identifying a sensor-related fault based on a comparison of object data provided by a sensor module). Figure 10 The system shown in can be installed completely within the vehicle.

[0162] The system includes an autonomous vehicle subsystem 402. Subsystem 402 is Figure 1 4. An example of a vehicle system 622 is shown. In the illustrated embodiment, autonomous vehicle subsystem 402 includes a map database 402A, a radar device 402B, a lidar device 402C, a digital camera 402D, a sonar device 402E, a GPS receiver 402F, and an inertial measurement unit 402G. Each component of autonomous vehicle subsystem 402 comprises standard components found in most current autonomous vehicles. In one embodiment, map database 402A stores multiple high-definition three-dimensional maps for routing and navigation. Radar device 402B, lidar device 402C, a digital camera 402D, sonar device 402E, a GPS receiver 402F, and an inertial measurement unit 402G may comprise respective devices mounted at various locations throughout the autonomous vehicle, as is known in the art. For example, these devices may be mounted along the perimeter of the autonomous vehicle to provide location awareness, collision avoidance, and other standard autonomous vehicle functions.

[0163] Additionally, vehicle subsystem 406 is included within the system. Vehicle subsystem 406 includes various anti-lock braking systems 406A, an engine control unit 402B, and a transmission control unit 402C. These components can be used to control the operation of the autonomous vehicle in response to streaming data generated by autonomous vehicle subsystem 402. Standard autonomous vehicle interactions between autonomous vehicle subsystem 402 and vehicle subsystem 406 are well known in the art and will not be described in detail herein.

[0164] The processing side of the system includes one or more processors 410 , short-term memory 412 , an RF subsystem 414 , a graphics processing unit (GPU) 416 , long-term storage 418 , and one or more interfaces 420 .

[0165] One or more processors 410 may include a central processing unit, an FPGA, or any range of processing devices required to support autonomous vehicle operation. Memory 412 may include DRAM or other suitable volatile RAM for temporarily storing data required by processor 410. RF subsystem 414 may include a cellular transceiver and / or a satellite transceiver. Long-term storage 418 may include one or more high-capacity solid-state drives (SSDs). Generally speaking, long-term storage 418 may be used to store, for example, high-definition maps, route data, and any other data that requires permanent or semi-permanent storage. GPU 416 may include one or more high-throughput GPU devices for processing data received from autonomous vehicle subsystem 402. Finally, interface 420 may include various display units (e.g., built-in screens) located within the autonomous vehicle.

[0166] In addition, the system includes a reporting subsystem 404 that performs data collection (e.g., data obtained from vehicle sensors used to drive the vehicle). Reporting subsystem 404 includes sensor monitoring 404A connected to bus 408 and records sensor data transmitted on bus 408 as well as any log data transmitted on the bus. In addition, reporting subsystem 404 may include one or more endpoints to allow system components to transmit log data directly to reporting subsystem 404.

[0167] In addition, the reporting subsystem 404 includes a packager 404B. In one embodiment, the packager 404B retrieves data from the sensor monitoring 404A or endpoint and packages the raw data for transmission to a central system (e.g., Figure 11 In some embodiments, the packetizer 404B may be configured to packetize data at periodic time intervals. Alternatively, or in combination with the foregoing, the packetizer 404B may transmit data in real time and may compress the data to facilitate real-time communication with the central system.

[0168] Additionally, reporting subsystem 404 includes a batch processor 404C. In one embodiment, batch processor 404C is configured to perform any pre-processing on the logged data prior to transmission. For example, batch processor 404C may perform compression on the data prior to packaging by packager 404B. In another embodiment, batch processor 404C may be configured to filter the logged data to remove irrelevant data prior to packaging or transmission. In another embodiment, batch processor 404C may be configured to perform data cleansing on the logged data to conform the raw data to a format suitable for further processing by the central system.

[0169] Each device is connected via bus 408. Bus 408 is Figure 3174 . In one embodiment, bus 408 may comprise a controller area network (CAN) bus. In some embodiments, other bus types (e.g., FlexRay or MOST bus) may be used. Additionally, each subsystem may include one or more additional buses to handle inter-subsystem communications (e.g., a LIN bus for lower bandwidth communications).

[0170] Figure 11 6 is a block diagram of a centralized autonomous vehicle operating system according to various embodiments. As shown, the system includes multiple autonomous vehicles 502A-502E. Each vehicle is an instance of vehicle 610.

[0171] In one embodiment, each autonomous vehicle may include a Figure 10 . Each autonomous vehicle 502A-502E can communicate with a central system 514 via a network 516. In one embodiment, network 516 comprises a global network such as the Internet. In one example, central system 514 is implemented using one or more of servers 101 or 301.

[0172] In addition, the system includes a plurality of client devices 508A, 508B. In the illustrated embodiment, the client devices 508A, 508B may include any personal computing device (e.g., a laptop, tablet, mobile phone, etc.). The client devices 508A, 508B may issue data requests from the central system 514. In one embodiment, as previously described, the client devices 508A, 508B send data requests to support mobile applications or web page data.

[0173] In one embodiment, the central system 514 includes a plurality of servers 504A. In one embodiment, the servers 504A include a plurality of front-end web servers configured to provide responses to the client devices 508A, 508B. In addition, the servers 504A may include one or more application servers configured to perform various operations to support one or more vehicles.

[0174] Furthermore, in one embodiment, central system 514 includes multiple models 504B. In one embodiment, models 504B may store one or more neural networks for classifying autonomous vehicle objects. Additionally, models 504B may include models for predicting future events. In some embodiments, models 504B may store a combination of neural networks and other machine learning models.

[0175] In addition, the central system 514 includes one or more databases 504C. The database 504C may include database records for vehicles 504D, personalities 504E, and raw data 504F. As previously described, the raw data 504F may include an unstructured database for storing raw data received from sensors and logs.

[0176] The present disclosure includes methods and apparatus for performing the above methods, including data processing systems for performing these methods, and computer-readable media containing instructions that, when executed on a data processing system, cause the system to perform these methods.

[0177] The server and / or computing device of the vehicle described above may be implemented as one or more data processing systems. A typical data processing system may include an interconnect (e.g., a bus and system core logic) interconnecting a microprocessor and memory. The microprocessor is typically coupled to a cache memory.

[0178] The interconnect interconnects the microprocessor and memory together and also interconnects them to input / output (I / O) devices via I / O controllers. The I / O devices may include a display device and / or peripheral devices such as a mouse, keyboard, modem, network interface, printer, scanner, camera, and other devices known in the art. In one embodiment, when the data processing system is a server system, some I / O devices (such as a printer, scanner, mouse, and / or keyboard) are optional.

[0179] The interconnection can include one or more buses connected to each other through various bridges, controllers and / or adapters. In one embodiment, the I / O controller includes a USB adapter for controlling USB (Universal Serial Bus) peripheral devices, and / or an IEEE-1394 bus adapter for controlling IEEE-1394 peripheral devices.

[0180] The memory may include one or more of: ROM (Read Only Memory), volatile RAM (Random Access Memory), and non-volatile memory such as a hard disk, flash memory, and the like.

[0181] Volatile RAM is typically implemented as dynamic RAM (DRAM), which requires a continuous supply of power to refresh or retain the data in the memory. Nonvolatile memory is typically a magnetic hard drive, a magnetic optical drive, an optical drive (e.g., DVD RAM), or other types of memory systems that retain data even after the system loses power. Nonvolatile memory can also be random access memory.

[0182] The non-volatile memory may be a local device directly coupled to the rest of the data processing system. Non-volatile memory that is remote from the system may also be used, such as a network storage device coupled to the data processing system through a network interface such as a modem or an Ethernet interface.

[0183] In this disclosure, some functions and operations are described as being performed or caused by software codes to simplify the description. However, such expressions are also used to specify functions resulting from execution of codes / instructions by a processor such as a microprocessor.

[0184] Alternatively, or in combination, the functions and operations described herein may be implemented using dedicated circuitry (with or without software instructions), such as using an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Embodiments may be implemented using hardwired circuitry without software instructions, or in combination with software instructions. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any particular source for the instructions executed by the data processing system.

[0185] While one embodiment can be implemented on fully functional computers and computer systems, various embodiments can be distributed as computer products in various forms and can be used regardless of the particular type of machine or computer readable medium used to actually implement the distribution.

[0186] At least some aspects disclosed herein may be embodied at least in part in software. That is, the techniques may be implemented in a computer system or other data processing system in response to its processor (such as a microprocessor) executing sequences of instructions contained in a memory (such as ROM, volatile RAM, non-volatile memory, cache, or remote storage device).

[0187] The routines executed to implement the embodiments may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions referred to as a "computer program." A computer program typically comprises one or more instructions that are located at different times in the various memories and storage devices of a computer and, when read and executed by one or more processors in the computer, cause the computer to perform the operations necessary to implement the elements involved in the various aspects.

[0188] Machine-readable media can be used to store software and data that, when executed by a data processing system, causes the system to perform various methods. Executable software and data can be stored in various locations, including, for example, ROM, volatile RAM, non-volatile memory, and / or cache. Portions of the software and / or data can be stored in any of these storage devices. Furthermore, data and instructions can be obtained from a centralized server or peer-to-peer network. Different portions of the data and instructions can be obtained from different centralized servers and / or peer-to-peer networks at different times, in different communication sessions, or within the same communication session. The data and instructions can all be obtained before the application is executed. Alternatively, portions of the data and instructions can be obtained dynamically and in a timely manner when execution is required. Therefore, it is not required that all data and instructions be on the machine-readable medium at a particular moment.

[0189] Examples of computer-readable media include, but are not limited to, non-transitory, recordable and non-recordable types of media, such as volatile and non-volatile memory devices, read-only memory (ROM), random access memory (RAM), flash memory devices, floppy disks and other removable disks, magnetic disk storage media, optical storage media (e.g., compact disk read-only memory (CD ROM), digital versatile disks (DVD), etc.), etc. Computer-readable media can store instructions.

[0190] The instructions may also be implemented in digital and analog communication links using electrical, optical, acoustic or other forms of propagated signals (such as carrier waves, infrared signals, digital signals, etc.). However, propagated signals (such as carrier waves, infrared signals, digital signals, etc.) are not tangible machine-readable media and are not configured to store instructions.

[0191] Generally, a machine-readable medium includes any mechanism that provides (i.e., stores and / or transmits) information in a form accessible to a machine (e.g., a computer, a network device, a personal digital assistant, a manufacturing tool, any device with a set of one or more processors, etc.).

[0192] In various embodiments, hardwired circuitry may be used in conjunction with software instructions to implement the technology.Thus, the technology is not limited to any specific combination of hardware circuitry and software, nor to any specific source for the instructions executed by the data processing system.

[0193] The above description and accompanying drawings are illustrative and should not be construed as restrictive. Numerous specific details are described to provide a thorough understanding. However, in some cases, well-known or conventional details are not described to avoid obscuring the description. References to one or more embodiments in this disclosure are not necessarily references to the same embodiment; rather, such references mean at least one embodiment.

[0194] In the foregoing specification, the present disclosure has been described with reference to specific exemplary embodiments thereof. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope set forth in the appended claims. The specification and drawings are, therefore, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. An autonomous vehicle, comprising: a first sensor module comprising a first sensor and a first processing device, wherein the first processing device evaluates sensor data from the first sensor to provide first object data; a second sensor module comprising a second sensor and a second processing device, wherein the second processing device evaluates sensor data from the second sensor to provide second object data; a central processing unit configured to receive object data provided by a sensor module of the autonomous vehicle; as well as a memory storing instructions configured to instruct the central processing unit to: receiving the first object data; receiving the second object data; determining an object type based on at least one of the first object data or the second object data; Based on the object type, determining an expected association; comparing the first object data to the second object data, the comparing comprising performing a statistical correlation of the first object data and the second object data to provide a result, and comparing the result to the expected correlation; determining whether the first object data corresponds to the second object data based on the comparison of the first object data with the second object data, wherein the determining comprises comparing a number of times the first object data matches the second object data to a threshold, and the threshold is a first threshold during daytime and a second threshold during nighttime; as well as In response to determining that the first object data does not correspond to the second object data, an action associated with operation of the autonomous vehicle is performed. 2 . The autonomous vehicle of claim 1 , wherein the first object data comprises the object type selected from a plurality of object types.

3. The autonomous vehicle of claim 2 , wherein the first sensor module further comprises a first memory storing data for a neural network, and wherein the first processing device uses the neural network to evaluate the sensor data from the first sensor.

4. The autonomous vehicle of claim 1 , wherein the instructions are further configured to instruct the central processing unit to determine a context of the autonomous vehicle based on sensor data from at least one sensor other than the first sensor and the second sensor, and wherein determining whether the first object data corresponds to the second object data is based in part on the context.

5. The autonomous vehicle of claim 1 , further comprising a host processing device, wherein performing the action comprises sending a signal to the host processing device, and wherein sending the signal causes the host processing device to change a configuration of a vehicle system of the autonomous vehicle.

6. The autonomous vehicle of claim 1, wherein the first sensor is a lidar or radar sensor and the second sensor is a camera.

7. The autonomous vehicle of claim 1 , wherein the first processing device is a system on a chip, a field programmable gate array, a graphics processing unit, or an application specific integrated circuit.

8. The autonomous vehicle of claim 1, wherein the first object data comprises a location of an object detected by the first sensor module, and the second object data comprises a location of an object detected by the second sensor module.

9. The autonomous vehicle of claim 1 , wherein the threshold is a predetermined percentage of a total number of comparisons.

10. The autonomous vehicle of claim 1, wherein performing the action associated with operation of the autonomous vehicle comprises sending a signal that causes a host processing device of the autonomous vehicle to perform a diagnostic test of at least one of the first sensor module or the second sensor module.

11. The autonomous vehicle of claim 1 , wherein the first object data comprises a first type of detected object, wherein the second object data comprises a second type of detected object, and wherein comparing the first object data to the second object data comprises comparing the first type to the second type.

12. A method for an autonomous vehicle, the method comprising: receiving first object data from a first sensor module; receiving second object data from a second sensor module; determining an object type based on at least one of the first object data or the second object data; Based on the object type, determining an expected association; comparing the first object data to the second object data, the comparing comprising performing a statistical correlation of the first object data and the second object data to provide a result, and comparing the result to the expected correlation; determining whether the first object data corresponds to the second object data based on a comparison of the first object data and the second object data, wherein the determining comprises comparing a number of times the first object data matches the second object data to a threshold, and the threshold is a first threshold during daytime and a second threshold during nighttime; as well as In response to determining that the first object data does not correspond to the second object data, an action is performed for the autonomous vehicle.

13. The method of claim 12, wherein the first sensor module comprises a processing device configured to process raw data from the sensor.

14. The method of claim 12, wherein the first object data comprises a position and an object type of an object detected by the first sensor module, and the second object data comprises a position and an object type of an object detected by the second sensor module. 15 . The method of claim 12 , wherein the first object data corresponds to an object detected by a sensor of the first sensor module, and the second object data corresponds to an object detected by a sensor of the second sensor module. 16 . The method of claim 12 , wherein determining whether the first object data corresponds to the second object data comprises determining whether a statistical detection relationship holds between the first object data and the second object data.

17. The method of claim 16, wherein performing the action for the autonomous vehicle comprises at least one of: Triggering safety red flags; Providing alerts to passengers; sending a communication to a central server that monitors vehicle operation of a plurality of vehicles including the autonomous vehicle; disabling an autonomous operating mode of the autonomous vehicle; requiring a human to take over control of the autonomous vehicle; or Automatically stopping the autonomous vehicle.

Images