Multi-modality data analysis engine for defect detection

Abstract

A system and method for fault detection related to vehicle driving operations includes collecting multiple modality input data streams from multiple different types of vehicle sensors, extracting one or more features from the input data streams using a grid-based feature extractor, and obtaining spatial attributes of an object located in any of multiple cells of the grid-based feature extractor. One or more anomalies are detected based on residual scores generated by each of the mutual attention-based anomaly detection and the time series-based anomaly detection. One or more faults are identified based on a generated total fault score identified by integrating residual scores of the mutual attention-based anomaly detection and the time series-based anomaly detection that exceed a preset fault score threshold. The vehicle driving operations are controlled based on the identified one or more faults.

Description

[Technical field]

[0001] The present invention relates to a multi-modality data analysis engine for vehicle sensors, and more particularly to improving the accuracy of real-time fault detection in autonomous, semi-autonomous and / or notification-assisted driving of a vehicle based on analysis of multiple different types of data collected by vehicle sensors during driving operations of the vehicle. [Background technology]

[0002] Conventional autonomous, semi-autonomous and / or notification-assisted vehicles use multiple cameras positioned at various locations on the vehicle (e.g., front, rear, left, right, etc.) to build a complete 360-degree view of the surrounding area during vehicle operation to collect data related to autonomous driving. Although conventional camera-based self-driving systems accurately depict the view captured by each camera, it may be difficult or impossible to determine features related to the distance to a particular object when using such systems. Furthermore, such camera-based self-driving systems generally do not perform well in conditions of poor visibility (e.g., at night, in fog, in rain, in snow, etc.), which may result in reduced accuracy in analyzing data or reduced performance of the vehicle's driving operation tasks (e.g., accelerating, braking, notifying obstacles, etc.). Summary of the Invention

[0003] According to one aspect of the present invention, there is provided a method for fault detection related to vehicle driving operations, comprising: collecting input data streams of multiple modalities from multiple different types of vehicle sensors; extracting one or more features from the input data streams using a grid-based feature extractor; and obtaining spatial attributes of an object located in any of multiple cells of the grid-based feature extractor. One or more anomalies are detected based on residual scores generated by each of mutual attention-based anomaly detection and time series-based anomaly detection. One or more faults are identified based on a generated total fault score identified by integrating residual scores of the mutual attention-based anomaly detection and the time series-based anomaly detection that exceed a preset fault score threshold. The vehicle driving operations are controlled based on the identified one or more faults.

[0004] According to another aspect of the present invention, there is provided a system for fault detection related to vehicle driving operations, comprising a processor device configured to collect input data streams of multiple modalities from multiple different types of vehicle sensors, extract one or more features from the input data streams using a grid-based feature extractor, and obtain spatial attributes of objects located in any of multiple cells of the grid-based feature extractor. One or more anomalies are detected based on residual scores generated by each of mutual attention based anomaly detection and time series based anomaly detection. One or more faults are identified based on a generated total fault score identified by integrating residual scores of the mutual attention based anomaly detection and the time series based anomaly detection that exceed a preset fault score threshold. The vehicle driving operations are controlled based on the identified one or more faults.

[0005] According to another aspect of the present invention, there is provided a non-transitory computer-readable storage medium including content configured to cause a computer to execute a method for fault detection related to vehicle driving operation, the method including: collecting input data streams of multiple modalities from multiple different types of vehicle sensors; extracting one or more features from the input data streams using a grid-based feature extractor; and obtaining spatial attributes of objects located in any of multiple cells of the grid-based feature extractor. One or more anomalies are detected based on residual scores generated by each of mutual attention-based anomaly detection and time series-based anomaly detection. One or more faults are identified based on a generated total fault score identified by integrating residual scores of the mutual attention-based anomaly detection and the time series-based anomaly detection that exceed a preset fault score threshold. The vehicle driving operation is controlled based on the identified one or more faults.

[0006] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments which are to be read in conjunction with the accompanying drawings.

[0007] In the present disclosure, preferred embodiments are described in detail with reference to the following drawings, as described below. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an exemplary block diagram illustrating a representative processing system to which the present invention may be applied, in accordance with an embodiment of the present invention.

[0009] [Diagram 2] FIG. 2 illustrates an exemplary high-level view of a system and method for detecting faults based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle, according to an embodiment of the present disclosure.

[0010] [Diagram 3] FIG. 3 is an exemplary diagram illustrating a system and method for mutual attention-based fault detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle, according to an embodiment of the present disclosure.

[0011] [Figure 4] FIG. 4 is a block / flow diagram illustrating an exemplary high-level method of mutual attention-based fault detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle, in accordance with an embodiment of the present disclosure.

[0012] [Diagram 5] FIG. 5 is an exemplary diagram illustrating a grid-based feature acquisition method for extracting features from multi-dimensional sensor data for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle, in accordance with an embodiment of the present invention.

[0013] [Figure 6] FIG. 6 is an exemplary diagram illustrating a high-level view of a system and method for mutual attention-based fault detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle in accordance with an embodiment of the present invention.

[0014] [Figure 7] FIG. 7 is an exemplary diagram illustrating a system for mutual attention-based fault detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle in accordance with an embodiment of the present invention.

[0015] [Figure 8] FIG. 8 is an exemplary diagram illustrating a system for anomaly detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle in accordance with an embodiment of the present invention.

[0016] [Figure 9] FIG. 9 is a block / flow diagram illustrating an exemplary mutual attention based fault detection method based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle in accordance with an embodiment of the present invention.

[0017] [Figure 10] FIG. 10 illustrates an exemplary system for a representative vehicle utilizing mutual attention-based fault detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of the vehicle in accordance with an embodiment of the present invention.

[0018] [Figure 11] FIG. 11 is an exemplary diagram illustrating a high-level system for mutual attention-based fault detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] According to embodiments of the present invention, systems and methods are provided for autonomous, semi-autonomous and / or notification-assisted operation of a vehicle with improved accuracy of real-time fault detection during vehicle operation based on analysis of multiple different types of data collected from vehicle sensors.

[0020] In accordance with aspects of the invention, in various embodiments, a number of different types of sensors (e.g., cameras, radar, proximity sensors, LIDAR, GPS, etc.) may be installed and utilized on a vehicle (e.g., an automobile, an aircraft, a drone, a boat, a rocket-powered spacecraft, etc.) for autonomous, semi-autonomous, and / or notification-assisted operation of the vehicle. For ease of explanation, such vehicles capable of autonomous, semi-autonomous, and / or notification-assisted operation in accordance with embodiments of the invention are hereinafter referred to as "autonomous vehicles."

[0021] In various embodiments, an autonomous vehicle equipped with multiple different types of sensors can collect sensor data in multiple formats (e.g., "multi-modality") and can integrate multiple data modalities (e.g., different types of data from different types of sensors) from different sensors for cross-attention based fault detection during vehicle driving operations. For many data analysis tasks (fault detection, autonomous driver assistance system (ADAS) fault detection, ADAS video search, etc.), conventional systems typically rely on only a single data modality, resulting in low accuracy.

[0022] Utilizing multi-modality data for controlling the driving operations of an autonomous vehicle in accordance with embodiments of the present invention can improve the accuracy of real-time data analysis tasks (e.g., data analysis of vehicle and / or external conditions for autonomous control of various functions of the autonomous vehicle). Furthermore, in accordance with aspects of the present invention, such utilization and analysis of multi-modality data improves the accuracy and reliability of any of a number of autonomous tasks during the driving operations of the autonomous vehicle.

[0023] As described in further detail below, in various embodiments, the present invention can be utilized to solve various problems that cannot be adequately addressed by conventional autonomous driving systems. For example, when collecting multi-modality data using multiple vehicle sensors, the raw sensor data can be dynamic and noisy, reducing the accuracy of data analysis tasks. According to aspects of the present invention, the present invention utilizes multi-modality sensor data as input for mutual attention based fault detection during vehicle driving operations.

[0024] Furthermore, utilizing only a single modality to determine analysis results, as in conventional systems, can be difficult, at least in part, because a single modality is limited in scope and does not provide sufficient data for accurate analysis and determination of autonomous vehicle control tasks. In accordance with aspects of the present invention, the present invention can improve the accuracy of such data analysis for anomaly and fault detection, at least in part, by using multiple modalities of data from multiple different types of sensors to obtain a complete and accurate view of the vehicle and surrounding environment.

[0025] In various embodiments, multi-modality data can be collected from different sources and sensors, each of which can collect data and describe different aspects of the monitored system. When performing an anomaly detection task, applying the influence of one modality to other modalities can be important for computational speed and accuracy. According to aspects of the invention, the invention uses a mutual attention based anomaly detection engine to detect anomalies from multi-modality data and can further utilize the application of the anomaly detection engine for fault analysis in any Autonomic Driving Assistant System (ADAS) of multiple types of vehicles.

[0026] In some embodiments, the input of the ADAS is a set of multimodal sequences, e.g., from different sensors mounted on the vehicle, and the acquired data includes system data and / or environmental data. The system data collected using the sensors includes data on multiple types of system states (speed, acceleration, turning angle, braking performance, transmission performance, etc.) of the vehicle (electric car, airplane, boat, position, etc.), which can be collected from one or more sensors (CANBus sensor, GPS sensor, etc.) before or during the vehicle's driving operation.

[0027] The environmental data collected using sensors includes data representative of the environment around the vehicle (e.g., object detection, lane detection, road hazard detection, etc.) and can be collected from one or more sensors (e.g., LIDAR sensors, cameras, proximity sensors, temperature sensors, radar, etc.). According to aspects of the invention, in some embodiments, such data may be collected in an irregular time series format (in other embodiments, it may be collected in a regular time series of fixed dimensions), and the data collected in the irregular time series format may be converted and transformed into a regular time series format for further processing. It is noted that the LIDAR sensor data of a vehicle is highly dynamic and noisy. Also, according to aspects of the invention, the invention includes specialized processing tools to obtain useful features determined from the multi-modality data, and a difference-based system can be used to effectively detect ADAS malfunctions before and / or during vehicle operation.

[0028] According to aspects of the invention, in some embodiments, the output of the ADAS fault task (which may be determined based on collected system and / or environmental data) is a fault score over time (e.g., in a time series format), and if the score is greater than a predefined threshold, the fault may be reported to an end user and / or the controller may automatically adjust the navigation (or other vehicle task) to take the fault into account. A fault is an incorrect driving behavior of the vehicle depending on the environment (not avoiding an obstacle, changing lanes to avoid an obstacle when one is not present, driving off the road, etc.), and conventional systems that only use changes in patterns / values ​​in a single mode of data often fail to identify the fault due to limited collection and / or use of multi-modal data, at least during the vehicle's driving operation. For example, if the vehicle actually slows down because it needs to slow down when entering a junction, it should not be identified as a fault, even if the speed value changes. However, according to aspects of the invention, if the vehicle does not perform a braking action when entering a junction and the speed value does not change, this should be identified as a fault and reported to the user or corrective action should be initiated during the vehicle's driving operation.

[0029] In various embodiments, the system and method can integrate data from multiple modalities and make decisions based on changes in multiple modalities for fault detection and / or navigation of an autonomous vehicle. Existing state-of-the-art anomaly detection algorithms cannot effectively detect such faults because they only work in a single or few dimensions. In fact, conventional systems can only detect anomalies / outliers of one or a few sensors, but cannot detect faults (such as reaction errors) based on a given environment.

[0030] In contrast, the present invention, in various embodiments, uses a mutual attention-based analysis engine for multi-modality data and applies it to ADAS fault detection on vehicle data, thereby realizing high accuracy and high speed fault identification during vehicle driving operation. Attention corresponds to a measurement of environmental changes. The mutual attention-based mechanism can apply the effects of environmental changes to the system data of a vehicle (autonomous vehicle, ego car, etc.) and build a model that records the vehicle's normal reaction in various environments. According to aspects of the present invention, in some embodiments, in the online testing step, the fault detection engine can first determine the vehicle's current environment (e.g., environment awareness) and then select a model corresponding to fault detection on the system data.

[0031] In various embodiments, the mutual attention mechanism can be used to perform environment awareness based anomaly / fault detection, and the present invention is superior to conventional systems and methods, at least in part, in terms of accuracy and speed of fault calculation and identification, since conventional anomaly detection systems function only in one or very few surrounding environments. According to aspects of the present invention, the present invention can be utilized to consider multiple types of data from multiple dynamic environments, and can achieve relatively high accuracy in detecting complex fault patterns during vehicle driving operations.

[0032] The embodiments described herein may be implemented entirely in hardware, entirely in software, or contain both hardware and software elements, in a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0033] An embodiment may include a computer program product accessible from a computer usable or computer readable medium that provides program code for use by or in connection with a computer or any instruction execution system. A computer usable or computer readable medium may include any device that stores, conveys, propagates, or transfers a program for use by or in connection with an instruction execution system, device, or apparatus. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or device or apparatus), or a propagation medium. The medium may include computer readable media such as semiconductor or solid state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disk, and optical disk.

[0034] Each computer program is tangibly stored on a machine-readable recording medium or device (such as a program memory or magnetic disk) readable by a general-purpose or special-purpose programmable computer, and configures and controls the operation of the computer when the recording medium or device is read by the computer to execute the procedures described herein. The system of the present invention may also be considered embodied in a computer-readable recording medium configured with a computer program, the recording medium so configured causing the computer to operate in a specific, predefined manner to perform the functions described herein.

[0035] A data processing system suitable for storing and / or executing program code may include at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory, bulk memory devices, and cache memory for temporarily storing at least some program code used during actual execution of the program code to reduce the number of times code is retrieved from bulk memory devices during execution of a process. Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may also be connected to the system, either directly or through I / O controllers.

[0036] The system may also have network adapters attached to it that enable the data processing system to be connected to other data processing systems or remote printers and storage devices through intervening private and / or public networks. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters.

[0037] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, systems and computer program products according to embodiments of the invention. It should be noted that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0038] The flowcharts and block diagrams in the figures show the architecture, functionality and operation of the implementation of the system, method and computer program product according to various embodiments of the present invention. Each block in the flowchart or block diagram may represent a module, segment or part of code including one or more executable instructions for implementing a specified logical function, and in some alternative implementations of the present invention, the functions described in the blocks may occur in a different order than described in the figures. For example, two blocks shown in succession may actually be executed substantially simultaneously, in a reverse order, or in another order, depending on the functionality of the specific embodiment.

[0039] It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be realized by special purpose hardware systems performing particular functions / operations, or combinations of special purpose hardware and computer instructions in accordance with the principles of the present invention.

[0040] Referring now in detail to the drawing, FIG. 1, in which like numerals represent the same or similar elements, there is illustrated, by way of example, a representative processing system 100 to which the principles of the present invention may be applied, in accordance with an embodiment of the present invention.

[0041] In some embodiments, processing system 100 includes at least one processor (CPU) 104 operatively connected to other components via a system bus 102. Operatively connected to system bus 102 are cache 106, read-only memory (ROM) 108, random access memory (RAM) 110, input / output (I / O) adapter 120, sound adapter 130, network adapter 140, user interface adapter 150, and display adapter 160.

[0042] The first storage device 122 and the second storage device 124 are operably connected to the system bus 102 by the I / O adapter 120. The first storage device 122 and the second storage device 124 may be disk storage devices (e.g., magnetic or optical disk storage devices), solid state magnetic devices, etc. The first storage device 122 and the second storage device 124 may be the same type of storage device or different types of storage devices.

[0043] Speakers 132 are operably connected to the system bus 102 by a sound adapter 130. Transceiver 142 is operably connected to the system bus 102 by a network adapter 140. A display device 162 is operably connected to the system bus 102 by a display adapter 160. In accordance with aspects of the present invention, one or more sensors 164 (e.g., cameras, proximity sensors, LIDAR data, GPS data, time series signal detectors, etc.) may further be connected to the system bus 102 by any suitable connection system or method (e.g., Wi-Fi, wired, network adapters, etc.).

[0044] A first user input device 152, a second user input device 154, and a third user input device 156 are operatively connected to the system bus 102 by a user interface adapter 150. The user input devices 152, 154, and 156 may be any of a keyboard, a mouse, a keypad, an image capture device, a motion sensing device, a microphone, or a device incorporating the functionality of at least two of these devices. Of course, other types of input devices may be used while maintaining the spirit of the principles of the present invention. The user input devices 152, 154, and 156 may be the same type of user input device or different types of user input devices. The user input devices 152, 154, and 156 are used to input information to the processing system 100 and output information from the processing system 100.

[0045] Of course, processing system 100 may include other elements (not shown) or omit certain elements, as would be readily apparent to one of ordinary skill in the art. For example, processing system 100 may include various other types of input and / or output devices, depending on the particular implementation, as would be readily apparent to one of ordinary skill in the art. For example, various wireless and / or wired input and / or output devices may be used. Furthermore, various configurations of additional processors, controllers, memory, etc. may be used, as would be readily apparent to one of ordinary skill in the art. These and other variations of processing system 100 will be readily apparent to one of ordinary skill in the art from the teachings of the present principles provided herein.

[0046] It should further be noted that systems 200, 300, 600, 700, 800, 1000 and 1100 described below in relation to Figures 2, 3, 6, 7, 8, 10 and 11 are systems for implementing respective embodiments of the present invention. In accordance with aspects of the present invention, some or all of the processing system 100 may be implemented in one or more of the elements of systems 200, 300, 600, 700, 800, 1000 and 1100.

[0047] It is further understood that the processing system 100 is capable of performing at least a portion of the methods described herein, such as at least a portion of the methods 200, 300, 400, 500 and 900 described below in connection with Figures 2, 3, 4, 5 and 9, respectively. Similarly, in accordance with aspects of the present invention, some or all of the systems 200, 300, 600, 700, 800, 1000 and 1100 may be used to perform at least a portion of the methods 200, 300, 400, 500 and 900 of Figures 2, 3, 4, 5 and 9, respectively.

[0048] As used herein, the term "hardware processor subsystem" or "hardware processor" may refer to a processor, memory, software, or combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution units, etc.). The one or more data processing elements may include a central processing unit, a graphics processing unit, and / or individual processors or computing element-based controllers (e.g., logic gates, etc.). The hardware processor subsystem may include one or more on-board memories (e.g., caches, dedicated memory arrays, read-only memories, etc.). In any embodiment, the hardware processor subsystem may include one or more memories that may be on-board or off-board, or may be dedicated for use with the hardware processor subsystem (e.g., ROM, RAM, Basic Input / Output System (BIOS), etc.).

[0049] In some embodiments, the hardware processor subsystem may include and execute one or more software elements, which may include an operating system and / or one or more applications and / or specific code for achieving a particular result.

[0050] In other embodiments, the hardware processor subsystem may include dedicated circuitry to perform one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0051] These and other variations of the hardware processor subsystem are contemplated, in accordance with embodiments of the present invention.

[0052] Referring to FIG. 2, FIG. 2 illustrates an exemplary high-level diagram of a system and method 200 for detecting faults based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous, and / or notification-assisted driving of a vehicle, in accordance with an embodiment of the present invention.

[0053] In one embodiment, the vehicle 202 (e.g., an autonomous vehicle, airplane, boat, etc.) may include one or more sensors 210 (e.g., LIDAR, GPS, radar, cameras, microphones, etc.) that may collect a number of different types of data regarding environmental conditions 212 during vehicle operation of the vehicle 202. According to aspects of the invention, the environmental condition data may be stored on a computer readable storage medium 204 and analyzed (e.g., for anomalies and / or malfunctions) using a processor device 206, and an automated vehicle controller 208 may automatically control the vehicle 202 (e.g., accelerate, brake, turn, enable / disable lights, and / or perform other vehicle functions), for example, based on the detected malfunction. According to aspects of the invention, vehicle system data 218 (e.g., speed, acceleration, braking, etc.) may be collected using processor device 206, the system data 218 may be analyzed (e.g., for anomalies and / or malfunctions) using processor device 206, and vehicle 202 may automatically control (e.g., accelerate, brake, turn, enable / disable lights, and / or perform other vehicle functions) using automatic vehicle controller 208, for example, based on the detected malfunction.

[0054] In some embodiments, features are extracted from vehicle system data 218 in block 220 and features are extracted from environmental condition data 212 in block 214. Attention is calculated based on the environment and cross-applied to system measurements (e.g., system data 218) using cross-attention in block 216. According to aspects of the invention, weights can be identified and / or applied in block 222 to generate weighted features in block 224. Anomaly detection engine 226 can determine whether the received weighted features 224 include “no anomaly” 228, “known anomaly” 232 and / or “unknown anomaly” 236 and can recommend and / or execute corresponding commands based on the determination. In various embodiments, if no anomaly is identified in block 228, the controller 208 takes no action, if a known anomaly is identified in block 232, the controller 208 takes corrective action (e.g., change lanes, brake, accelerate, etc.) in block 234, and if an unknown anomaly is identified in block 236, the controller 208 directs the processor device 206 to perform further analysis of the weighted features 224 before taking action. In some embodiments, if an unknown anomaly is identified in block 236, the controller 208 may gradually slow the vehicle down and pull over to a stop until an anomaly is identified, while in other embodiments, if the controller 208 determines that the unknown anomaly 236 is not an imminent danger to the vehicle 202 or its occupants, in accordance with aspects of the present invention, the controller 208 may continue operating the vehicle during further analysis in block 238.

[0055] Reference is now made to FIG. 3, which illustratively depicts a diagram illustrating a high-level view of a system and method 300 for mutual attention-based fault detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle, in accordance with an embodiment of the present invention.

[0056] In one embodiment, a driving state 304 (e.g., lane changing, braking, accelerating, etc.) of a vehicle 304 (e.g., an autonomous vehicle) may be identified and vehicle system data 306 may be identified and collected in block 306 and used for anomaly detection by an anomaly detection engine 308. Detection of environmental conditions / driving behavior may be performed in block 310.

[0057] In block 312, appropriate behavior of the vehicle 302 may be identified, including, for example, performing a lane change maneuver to avoid the obstacle 301 in the event of an environmental change (e.g., if the obstacle 301 is detected en route) in block 314, or remaining in a new lane of the road without reacting to the obstacle in block 316. In accordance with aspects of the present invention, inappropriate behavior (e.g., malfunction) related to the driving behavior of the vehicle 302 is determined in block 318. Inappropriate behavior includes, for example, changing lanes in inappropriate situations, such as performing a lane change maneuver when another vehicle 303 is already in the desired lane in block 320, or performing a lane change maneuver into a prohibited area (e.g., shoulder, off road, etc.) when attempting to overtake in block 322. In this exemplary embodiment, for ease of explanation, a lane change driving maneuver is shown in blocks 314, 316, 320, and 322, however, it should be understood that any type of driving behavior or detection of environmental conditions may be performed in accordance with various aspects of the present invention.

[0058] In some embodiments, the environmental condition / driving behavior detection data of block 310 is analyzed using mutual attention in block 324 and the vehicle system data 306 is input to an anomaly detection engine 308 that can detect one or more anomalies in the vehicle system data 306 and / or the environmental condition / driving behavior detection data of block 310. According to aspects of the invention, the detected anomaly is determined to be a fault in block 326 and a fault score is identified and / or output in block 328 for use in any of a number of autonomous driving tasks.

[0059] In the following, the system and method 300 is described with respect to controlling an autonomous vehicle, however, in accordance with aspects of the present invention, it should be understood that the principles of the present invention can also be applied to other cyber-physical systems, such as smart cities (cameras, video, temperature sensors, etc.), smart houses, etc.

[0060] Referring to FIG. 4, FIG. 4 illustratively illustrates a block / flow diagram illustrating a high level method 400 of mutual attention based fault detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle, in accordance with an embodiment of the present invention.

[0061] According to various embodiments, multi-modality data (e.g., environmental data and vehicle data of various types / formats) may be captured and / or received at block 402 and feature extraction may be performed therefrom at block 404. According to aspects of the invention, one or more anomalies and / or malfunctions in the vehicle system data and / or environmental condition data are identified at block 406 and a malfunction score is generated at block 408, as described in further detail below with reference to Figures 5, 6, 7 and 8.

[0062] Referring to FIG. 5 , FIG. 5 exemplarily illustrates a diagram illustrating a grid-based feature acquisition method 500 for extracting features from multi-dimensional sensor data for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle, in accordance with an embodiment of the present invention.

[0063] In some embodiments, input data streams are monitored from one or more sensors (e.g., LIDAR sensors, video cameras, proximity sensors, infrared sensors, microphones, speed sensors, etc.) located on the vehicle 520 and data is collected and / or received to extract features from the sensor data using the grid-based feature acquisition method 500. For ease of explanation, the sensor data described below is LIDAR sensor data, however, it should be understood that other types of sensor data may be used in accordance with various aspects of the present invention.

[0064] It should be noted that the primary sensor that a vehicle can use to sense the surrounding environment is a LIDAR sensor. Such a sensor can detect surrounding objects and lanes by reflecting laser radar signals. The format of LIDAR data can be described as a sequence of detected objects, which include object attributes such as the speed, size, acceleration, and position of the vehicle 520. A problem in utilizing LIDAR data for vehicle driving behavior is that the objects detected at each timestamp are usually not constant. For example, at timestamp T1, there may be 20 objects around the vehicle 520, and at timestamp T2, there may be 30 objects around the ego-vehicle. According to an aspect of the present invention, the present invention can obtain a constant number of features from dynamically changing object detection data as a first step.

[0065] In some embodiments, the grid-based feature acquisition method 500 can be utilized for feature extraction by dividing the spatial region into a grid having nine cells (502, 504, 506, 508, 510, 512, 514, 516, and 518) and placing the vehicle 520 in cell 510 (e.g., the center cell). Thus, in this embodiment, there are eight cells surrounding the vehicle 520. According to aspects of the invention, each cell has a predefined length and width, and only detected objects located within the cell (e.g., 501, 505, 509, and 511) are considered and analyzed, while detected objects located outside the nine cells (e.g., 503, 507, 513, and 515) are determined to be sufficiently far away from the vehicle 520 that they may be ignored during grid-based acquisition and may not be considered or analyzed for feature extraction.

[0066] According to aspects of the invention, in some embodiments, for each cell (502, 504, 506, 508, 510, 512, 514, 516, and 518), spatial attributes of the object and the object are captured and may include, for example, object number, size of the closest object, distance to the closest object, speed of the closest object, etc., so that regardless of changes in the number of objects detected or the total number, a constant number of features are always obtained from the nine cells.

[0067] Referring to FIG. 6, FIG. 6 shows an exemplary diagram illustrating a system and method 600 for mutual attention-based fault detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle, in accordance with an embodiment of the present invention.

[0068] According to aspects of the invention, in various embodiments, two detectors can be used for anomaly and / or fault detection. One detector is a mutual attention based detector 602, which can use both environmental data 606 and system data 610 as inputs to perform mutual attention 608. According to aspects of the invention, one or more anomalies are detected in block 613, and a residual score (residual A) is determined and output in block 614. Another detector utilized is a long short-term memory (LSTM) based time series detector 604, which in some embodiments uses only system data 620 as input for anomaly detection in block 622, and determines another residual score (residual V) to output in block 624.

[0069] It should be noted that the residual scores (residual A 614 and residual V 624) are the difference between the predicted and actual values ​​of the system data 610, 620, and will be described in more detail below with reference to FIG. 7 and FIG. 8. The mutual attention based detector 602 and the LSTM based detector 604 can be trained with normal data to predict values, and the difference between the predicted and actual values ​​during the training step can be minimized. If a relatively large difference (e.g., a high residual score) is observed during the online testing step, it may indicate that a different behavior (e.g., an anomaly) is occurring, but the detection of such an anomaly does not necessarily mean that a defect exists. According to aspects of the present invention, in some embodiments, the residual A 614 and the residual V 624 are input and combined into a score aggregator 626, and the defect score generator 628 can use the combined data from the score aggregator 626 to determine whether a defect exists based on a defect score generated from the comparison result of both residuals in block 628.

[0070] In some embodiments, the score aggregator 626 may compare the residual scores (e.g., block 754 of FIG. 7 and block 814 of FIG. 8) and analyze them to determine a final defect score in block 628. According to an aspect of the invention, the final defect score 628 may be calculated as follows:

number

[0071] Referring to FIG. 7, FIG. 7 shows an exemplary diagram illustrating a system 700 for mutual attention-based fault detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle, in accordance with an embodiment of the present invention.

[0072] According to aspects of the present invention, in one embodiment, a mutual attention based fault detection system 700 includes two main stages: an attention computation stage 701 and a residual generation stage 703. In the attention computation stage 701, environmental data (X) 702 is collected over time and / or received as input (e.g., X1 704, X2 706, ..., X t-1 708). Environmental Data (X1704, X2706, ..., X t-1 708) is encoded by the LSTM encoder 710 into h1714, h2716, ..., h t-1 718. According to an embodiment of the present invention, the environmental data (X t ) 730 is passed through an LSTM encoder 732 and used as a query 734. The query is generated by applying the key (h t )736, α1724, α2726, ...α t-1 728.

[0073] In some embodiments, the residual generation stage 703 performs mutual attention in block 738 to set the environmental attention weights 722 as y1 744, y2 746, ..., y t-1 748 may be cross-applied to system data 742(Y) (e.g., real-time or historical system data). t-1 The weights of 708 are y1744, y2746, …, y t-1 748, and the attention weighting system data (y1 744, y2 746, ..., y t-1 748) is the prediction module loss function (yt -y t ') 2 750 may be used to predict a value (yt) at time t and used to generate a predicted value (yt') 752. t andy t ' can be output as residual A in block 754. In the model training stage, the parameters of the LSTM encoder, the attention module over time, and the prediction module are adjusted to obtain the loss function (y t -y t ') 2 750 can be minimized, where y t is the actual value, and y t ' is the predicted value (y t According to an embodiment of the present invention, in the online testing stage, (y t -y t The difference between the residuals A and B can be output as a residual A score 754 for further processing and / or analysis.

[0074] 6, 7 and 11, and now referring to FIG. 8, FIG. 8 shows an exemplary diagram illustrating a system 800 for anomaly detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle, in accordance with an embodiment of the present invention.

[0075] In some embodiments, an anomaly detector (e.g., LTSM) can be utilized in a manner similar to the mutual attention based detection residual generation stage (shown in FIG. 7 at block 703), although in this embodiment, the anomaly detector does not include a mutual attention stage. t-1 808) (e.g., real-time or historical) data is collected and / or received as input, and a prediction module loss function (y t -y t ') 810, and is used to t " 812, where y t andy t ' is calculated as the residual V

number

[0076] In some embodiments, a score aggregator (illustrated by block 626 of FIG. 6 and block 1114 of FIG. 11) may compare the residual scores from block 754 of FIG. 7 and block 814 of FIG. 8 and analyze them to determine a final defect score. According to an aspect of the invention, the final defect score 628 may be calculated as follows:

number

[0077] Referring to FIG. 9, there is shown an exemplary block / flow diagram illustrating a method 900 of mutual attention based fault detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle, in accordance with an embodiment of the present invention.

[0078] According to aspects of the invention, in various embodiments, at block 902, multi-modality input data streams (e.g., environmental data and / or vehicle system data) are collected from one or more vehicle sensors (e.g., video cameras, sensors, LIDAR, GPS, microphones, etc.) and / or received as input data (e.g., environmental data, road data, etc.) via any suitable transmitting / receiving means. At block 904, latent features are extracted from the one or more input data streams using a grid-based feature extractor.

[0079] At block 906, spatial attributes of one or more objects in each cell in the grid-based feature extractor are obtained (e.g., object number, size of nearest object, speed, distance to vehicle, etc.). At block 908, anomaly and / or defect detection, defect score generation, and / or model training are performed. According to aspects of the invention, this may include mutual attention based anomaly detection at block 910, LSTM time series based anomaly detection at block 912, defect score integration at block 914, and / or total defect score generation at block 916.

[0080] In some embodiments, one of a number of driving operations of the vehicle (e.g., accelerating, turning, braking, adjusting lighting or other vehicle functions, etc.) can be automatically controlled in block 918 based on the detected anomalies and / or the generated total fault score. According to aspects of the invention, the collection in block 902, the extraction in block 904, the acquisition in block 906, the fault detection / fault score generation / model training in block 908 (including blocks 910, 912, 914, and 916), and the controlled driving operation of the vehicle in block 918 can be performed repeatedly before, during, and / or after the vehicle is driven to detect and / or report additional faults and adjust the automated control of the vehicle in block 918 to account for the newly detected faults and / or anomalies.

[0081] Referring to FIG. 10, FIG. 10 illustrates an exemplary diagram of an exemplary system 1000 including an exemplary vehicle that utilizes mutual attention-based fault detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of the vehicle in accordance with an embodiment of the present invention.

[0082] The system 1000 may include an autonomous vehicle 12. In one embodiment, the autonomous vehicle 12 may be an automobile. In other embodiments, the autonomous vehicle 12 may include a boat, an airplane, a helicopter, a truck, a boat, or the like. The autonomous vehicle 12 may include a propulsion system 18. For flying embodiments, the propulsion system 18 may include a propeller or other engine for flying the autonomous vehicle 12. In another embodiment, the propulsion system 18 may include wheels or tracks. In another embodiment, the propulsion system 18 may include a jet engine or hover technology. The propulsion system 18 may include one or more motors, including an internal combustion engine, an electric motor, or the like.

[0083] Autonomous vehicle 12 may include a power source 20. Power source 20 may include or use one or more batteries, liquid fuels (gasoline, alcohol, diesel, etc.), or other energy sources. In another embodiment, power source 20 may include one or more solar cells or one or more fuel cells. In another embodiment, power source 20 may include combustion gases (e.g., hydrogen).

[0084] The autonomous vehicle 12 may be equipped with computing capabilities and control. The autonomous vehicle 12 may include a processor 22. The autonomous vehicle 12 may include a transceiver 24. In one embodiment, the transceiver 24 may be connected to a Global Positioning System (GPS) to generate and alert the location of the autonomous vehicle 12 relative to other vehicles in a common coordinate system. The transceiver 24 may be equipped to communicate with a cellular network system. In this manner, the location of the autonomous vehicle may be calculated based on triangulation between cell towers based on signal strength, etc. The transceiver 24 may include a WIFI or equivalent wireless system. The processor 22, transceiver 24, and location information are available to a guidance and control system 26 of the autonomous vehicle 12.

[0085] Autonomous vehicle 12 may include memory device 28. Memory device 28 may include solid state storage or soft storage and may interface with other systems of autonomous vehicle 12, record data, execute algorithms or programs, control the vehicle, etc. Memory device 28 may include read only memory (ROM), random access memory (RAM), or other types of memory useful for the present application.

[0086] The autonomous vehicle 12 includes one or more sensors 14 (e.g., cameras, proximity sensors, LIDAR, radar, GPS, etc.) for collecting a number of different data types before, during, and / or after use of the autonomous vehicle 12. The one or more sensors 14 can view the area around the autonomous vehicle 12 and input the sensor data to an autonomous driver assistance system (ADAS) data processing and analysis engine 30, as well as a guidance and control system 26, of the autonomous vehicle 12. The one or more sensors 14 can detect objects around the autonomous vehicle 12 (e.g., other vehicles, buildings, light poles, pedestrians 16, trees, etc.) and / or internal vehicle features and / or vehicle component states. According to various aspects of the present invention, data obtained by the one or more sensors 14 is processed by the ADAS engine 30 of the autonomous vehicle 12 and utilized by the guidance and control system 26, for example, to adjust the propulsion system 18 of the autonomous vehicle 12 to avoid objects around the autonomous vehicle 12.

[0087] Referring to FIG. 11, FIG. 11 illustrates an exemplary diagram illustrating a system 1100 for mutual attention-based fault detection based on analysis of multiple different types of data from vehicle sensors for autonomous, semi-autonomous and / or notification-assisted driving of a vehicle, in accordance with an embodiment of the present invention.

[0088] According to aspects of the present invention, in some embodiments, data is collected from one or more sensors 1102 (e.g., LIDAR, GPS, smart sensors, cameras, IoT devices, etc.) and the data stream from the sensors 1102 is transmitted over a computing network 1104 (e.g., WiFi, wireless, 4G, 5G, CAN bus, LAN, WAN, wired, etc.) and analyzed using one or more processor devices 1120 deployed in the vehicle 1118 or remotely from the vehicle 1118. The feature extractor 1106 can extract features from the data collected and / or received from the sensors 1102. The features are further processed by the object / spatial attribute acquisition device 1108 and anomalies and / or faults are identified using a mutual attention based anomaly detector 1110 and / or an LSTM time series based anomaly detector 1112.

[0089] According to aspects of the invention, in various embodiments, anomalies detected by the mutual attention based anomaly detector 1110 and / or the LSTM time series based anomaly detector 1112 are output to a fault score aggregator 1114, which can combine the data received from the detectors 1110 and / or 1112 and use it as input to a fault score generator 1116 to generate one or more fault scores. A neural network training device 1122 can be used to further improve the accuracy and speed of anomaly and / or fault detection, for example, by repeatedly training the neural network with new data acquired by one or more sensors 1102.

[0090] According to aspects of the invention, in various embodiments, one or more controller devices 1124 may be utilized to adjust any of a number of vehicle driving operations (e.g., acceleration, braking, lights, etc.) in response to determining a malfunction and / or a particular event identified during the vehicle's driving operations in which the malfunction and / or malfunction score exceeds a user-selectable pre-set malfunction score threshold.

[0091] In the embodiment shown in FIG. 11, the elements are interconnected by a bus 1101. However, in other embodiments, other types of connections may be used. Furthermore, in one embodiment, at least one of the elements of the system 1100 is processor-based and / or logic circuitry and may include one or more processor devices 1120. Furthermore, while one or more elements may be shown as separate elements, in other embodiments, these elements may be combined into one element. The reverse is also applicable, where one or more elements may be part of another element, while in other embodiments, one or more elements may be implemented as stand-alone elements. These and other variations of the elements of the system 1100 are readily determined by one of ordinary skill in the art given the teachings of the present principles provided herein, while maintaining the spirit of the present principles.

[0092] Reference herein to "one embodiment" or "an embodiment" of the invention, as well as other variations, means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment of the invention. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment" in various places herein, as well as other variations, do not necessarily all refer to the same embodiment. However, given the teachings of the invention provided herein, it should be understood that features of one or more embodiments may be combined.

[0093] For example, the use of any of "A / B," "A and / or B," and "at least one of A and B" should be understood to be intended to include the selection of only the first listed option (A), the selection of only the second listed option (B), or the selection of both options (A and B). By way of further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to include the selection of only the first listed option (A), the selection of only the second listed option (B), the selection of only the third listed option (C), the selection of only the first and second listed options (A and B), the selection of only the first and third listed options (A and C), the selection of only the second and third listed options (B and C), or the selection of all three options (A, B, and C). The above examples may be expanded depending on the number of items listed.

[0094] The foregoing should be understood in all respects as illustrative and exemplary, and not restrictive, and the scope of the invention disclosed herein should be determined not from the detailed description, but from the claims which are to be interpreted in accordance with the broadest possible interpretation permitted by the Patent Law. It should be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and that various modifications may be made by those skilled in the art without departing from the scope and spirit of the invention. Various other feature combinations may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Although aspects of the invention have been described above with the fine detail and particularity required by the Patent Law, the scope of the claims which are sought to be protected by Letters Patent are set forth in the appended claims.

Claims

1. 1. A method for detecting faults related to driving operation of a vehicle, comprising: collecting multiple modality input data streams from multiple different types of vehicle sensors; extracting one or more features from the input data stream using a grid-based feature extractor; obtaining spatial attributes of an object located in any of a plurality of cells of the grid-based feature extractor; detecting one or more anomalies based on residual scores generated by each of the mutual attention based anomaly detection and the time series based anomaly detection; Identifying one or more defects based on a generated total defect score identified by combining residual scores of the mutual attention based anomaly detection and the time series based anomaly detection that exceed a pre-defined defect score threshold; controlling driving operations of the vehicle based on the identified fault or faults; The method according to claim 1,

2. The method of claim 1 , wherein the mutual attention based anomaly detection uses spatial attributes of the object and vehicle system data, and the time series based anomaly detection uses the vehicle system data during detection.

3. The method of claim 1 , wherein the object is an environmental object that represents one or more hazardous situations.

4. The method of claim 1 , wherein the grid-based feature extractor includes nine cells and the vehicle is positioned within a central cell of the grid-based feature extractor.

5. 2. The method of claim 1, wherein additional malfunctions are continually detected in real time during operation of the vehicle by iteratively repeating the collecting, extracting, obtaining, detecting, and identifying steps during operation of the vehicle.

6. The mutual attention based anomaly detection comprises: generating environmental attention weights in an attention computation stage by encoding the received environmental data to generate one or more keys that are matched to the query in a longitudinal attention stage; cross-applying the environmental attention weights to historical system data for the vehicle to generate a prediction of values ​​at a next time step; training the model by adjusting one or more parameters of the prediction to minimize a loss function between the actual values ​​and the predicted values; The method of claim 1 further comprising:

7. The total defect score is: [0010] 2. The method of claim 1 , wherein residual A represents the output of the mutual attention based anomaly detection and residual V represents the output of the time series based anomaly detection.

8. A system for detecting a malfunction related to a driving operation of a vehicle, comprising: one or more processors operably connected to a computer-readable non-transitory storage medium; The processor, collecting multiple modality input data streams from multiple different types of vehicle sensors; extracting one or more features from the input data stream using a grid-based feature extractor; obtaining spatial attributes of an object located in any of a plurality of cells of the grid-based feature extractor; detecting one or more anomalies based on residual scores generated by each of the mutual attention based anomaly detection and the time series based anomaly detection; Identifying one or more defects based on a generated total defect score identified by combining residual scores of the mutual attention based anomaly detection and the time series based anomaly detection that exceed a pre-defined defect score threshold; controlling driving operations of the vehicle based on the identified fault or faults; A system configured to run

9. 10. The system of claim 8, wherein the mutual attention based anomaly detection uses spatial attributes of the object and vehicle system data, and the time series based anomaly detection uses the vehicle system data during detection.

10. The system of claim 8 , wherein the object is an environmental object that represents one or more hazardous situations.

11. 9. The system of claim 8, wherein the grid-based feature extractor includes nine cells, and the vehicle is positioned within a central cell of the grid-based feature extractor.

12. 9. The system of claim 8, wherein additional malfunctions are continually detected in real time during operation of the vehicle by iteratively repeating the collecting, extracting, acquiring, detecting and identifying steps during operation of the vehicle.

13. The mutual attention based anomaly detection comprises: generating environmental attention weights in an attention computation stage by encoding the received environmental data to generate one or more keys that are matched to the query in a longitudinal attention stage; cross-applying the environmental attention weights to historical system data for the vehicle to generate a prediction of values ​​at a next time step; training the model by adjusting one or more parameters of the prediction to minimize a loss function between the actual values ​​and the predicted values; The system of claim 8 further comprising:

14. The total defect score is: [0025] 10. The system of claim 8, wherein residual A represents an output of the mutual attention based anomaly detection and residual V represents an output of the time series based anomaly detection.

15. A non-transitory computer-readable recording medium including a computer-readable program, operably connected to a processor device for detecting a malfunction related to a driving operation of a vehicle, the non-transitory computer-readable recording medium comprising: When the computer readable program is executed on the computer, the computer collecting multiple modality input data streams from multiple different types of vehicle sensors; extracting one or more features from the input data stream using a grid-based feature extractor; obtaining spatial attributes of an object located in any of a plurality of cells of the grid-based feature extractor; detecting one or more anomalies based on residual scores generated by each of the mutual attention based anomaly detection and the time series based anomaly detection; Identifying one or more defects based on a generated total defect score identified by combining residual scores of the mutual attention based anomaly detection and the time series based anomaly detection that exceed a pre-defined defect score threshold; controlling driving operations of the vehicle based on the identified fault or faults; A non-transitory computer-readable recording medium for executing the method of the present invention.

16. 16. The non-transitory computer-readable storage medium of claim 15, wherein the mutual attention based anomaly detection uses spatial attributes of the object and vehicle system data, and the time series based anomaly detection uses the vehicle system data during detection.

17. The computer-readable non-transitory storage medium of claim 15 , wherein the grid-based feature extractor includes nine cells, and the vehicle is positioned within a central cell of the grid-based feature extractor.

18. 16. The non-transitory computer readable storage medium of claim 15, wherein the collecting, extracting, obtaining, detecting, and identifying steps are repeated iteratively during operation of the vehicle such that additional malfunctions are continually detected in real time during operation of the vehicle.

19. The mutual attention based anomaly detection comprises: generating environmental attention weights in an attention computation stage by encoding the received environmental data to generate one or more keys that are matched to the query in a longitudinal attention stage; cross-applying the environmental attention weights to historical system data for the vehicle to generate a prediction of values ​​at a next time step; training the model by adjusting one or more parameters of the prediction to minimize a loss function between the actual values ​​and the predicted values; 16. The non-transitory computer readable storage medium of claim 15, further comprising:

20. The total defect score is: [0030] and a residual A represents an output of the mutual attention based anomaly detection, and a residual V represents an output of the time series based anomaly detection.

Images