Abnormal detection device

Abstract

To determine existence of an abnormality of periphery recognition performance at an early stage on the basis of the sign being deterioration of the periphery recognition performance in an automatic driving system.SOLUTION: There is provided an abnormality determination device that determines the presence / absence of any abnormality in an automatic driving system of an own vehicle. The abnormality determination device comprises a performance deterioration determination unit configured to determine, on the basis of a plurality of indices obtained by detection results of sensors of the own vehicle, whether or not deterioration of periphery recognition performance of the automatic driving system for each of the indices, during automatic driving of the own vehicle. The abnormality determination device facilitates determination of existence of the abnormality of the periphery recognition performance of the automatic driving system easier when the performance deterioration determination unit determines that the deterioration of the periphery recognition performance has occurred, compared to the case of not determining that the deterioration of the periphery recognition performance has occurred.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an abnormality determination device.

Background Art

[0002] Conventionally, Japanese Unexamined Patent Application Publication No. 2022-032109 is known as a technical document regarding an abnormality determination device. In this publication, in a collision prevention system that determines the reliability of the movement characteristics of an object to be determined for collision using detection results obtained from a plurality of different types of detectors, it is shown that, based on a predetermined index (the deviation between the posture of the target object and the velocity vector), it is determined that the error in the calculated posture or velocity vector of the target object is large and the reliability is low.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional system, the reliability is determined using detection results obtained from a plurality of different types of detectors, but no device for early determination of an abnormality such that the reliability is significantly reduced is disclosed. Particularly in an automatic driving system, early detection of an abnormality is important. Therefore, an object of the present invention is to provide a device capable of performing early determination of an abnormality in an automatic driving system.

Means for Solving the Problems

[0005] One aspect of the present invention is an abnormality determination device for determining whether or not there is an abnormality in the autonomous driving system of a vehicle, comprising a performance degradation determination unit that determines whether or not a decrease in the surrounding recognition performance of the autonomous driving system has occurred for each indicator based on a plurality of indicators obtained from the detection results of the vehicle's sensors during autonomous driving of the vehicle, and when the performance degradation determination unit determines that a decrease in surrounding recognition performance has occurred, it is made easier to determine that there is an abnormality in the surrounding recognition performance of the autonomous driving system compared to when it is not determined that a decrease in surrounding recognition performance has occurred.

[0006] One aspect of the present invention is an abnormality determination device for determining whether or not there is an abnormality in the autonomous driving system of a vehicle, comprising a performance degradation determination unit that determines whether or not a decrease in the surrounding recognition performance of the autonomous driving system has occurred for each indicator based on a plurality of indicators obtained from the detection results of the vehicle's sensors during autonomous driving of the vehicle, and when the performance degradation determination unit determines that a decrease in surrounding recognition performance has occurred, it is made easier to determine that there is an abnormality in the surrounding recognition performance of the autonomous driving system compared to when it is not determined that a decrease in surrounding recognition performance has occurred. The multiple indicators include at least two of the following: the reliability of the vehicle's external cameras, the reliability of the vehicle's radar sensors, the reliability of the autonomous driving system's external object recognition, the reliability of the autonomous driving system's lane marking recognition, the continuity of the autonomous driving system's surrounding area recognition results, the consistency of the recognition results from the vehicle's external cameras and radar sensors, the consistency between the surrounding area recognition results obtained from surrounding vehicles via vehicle-to-vehicle communication and the surrounding area recognition results from the autonomous driving system, the number or frequency of abnormal behavior of the vehicle during autonomous driving by the autonomous driving system, the number or frequency of driver intervention operations on the vehicle during autonomous driving, the number or frequency of abnormal behavior of the vehicle towards surrounding vehicles during autonomous driving, the number or frequency of abnormal behavior of the vehicle towards lane markings during autonomous driving, and the number or frequency of abnormal behavior of surrounding vehicles towards the vehicle during autonomous driving. . [Effects of the Invention]

[0008] According to each aspect of the present invention, in an autonomous driving system, it is possible to determine the presence of an abnormality in the surrounding recognition performance at an early stage, based on the warning sign of a decline in surrounding recognition performance. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing an automated driving system (abnormality detection device) according to one embodiment. [Figure 2] This flowchart shows an example of a performance degradation detection process. [Figure 3] This flowchart shows an example of an anomaly detection process. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings.

[0011] Figure 1 is a block diagram showing an automated driving system (anomaly detection device) 100 according to one embodiment. The automated driving system 100 shown in Figure 1 is a system that performs automated driving of its own vehicle 1. Automated driving is a vehicle control system that allows the vehicle 1 to automatically drive along a preset route or along the road it is currently traveling on, without the driver performing any driving operations. Automated driving corresponds to, for example, automated driving level 2 or higher certified by the Society of Automotive Engineers (SAE). Automated driving may be limited to automated driving level 3 or higher.

[0012] The automated driving system 100 starts automated driving of its own vehicle 1 when the driver initiates the automated driving start operation. If the driver intervenes (driver operation during automated driving) while the automated driving system 100 is in automated driving mode, it switches to an override state. In the override state, automated driving continues, while the driver's operation is reflected in the driving of its own vehicle 1.

[0013] The automated driving system 100 is configured to include an abnormality detection device 50. The abnormality detection device 50 is a device that determines whether or not there is an abnormality in the surrounding recognition performance of the automated driving system 100. Surrounding recognition performance refers to the performance of the vehicle 1 in terms of surrounding recognition (recognition of the external environment). Note that the abnormality detection device 50 does not necessarily have to be part of the automated driving system 100, and may be provided independently of the automated driving system 100.

[0014] The autonomous driving system 100 is equipped with an autonomous driving ECU 30 [Electronic Control Unit]. The autonomous driving ECU 30 is an electronic control unit having a CPU [Central Processing Unit] and a memory unit. The memory unit consists of, for example, ROM [Read Only Memory], RAM [Random Access Memory], EEPROM [Electrically Erasable Programmable Read-Only Memory], etc. The autonomous driving ECU 30 realizes various functions by executing programs stored in the memory unit with the CPU. The autonomous driving ECU 30 may be composed of multiple electronic units.

[0015] As shown in Figure 1, the autonomous driving ECU 30 is connected to a GNSS receiver 21, an external camera 22, a radar sensor 23, an internal sensor 24, a map database 25, and a driving operation detection unit 26.

[0016] The GNSS receiver 21 measures the position of the vehicle 1 (for example, the latitude and longitude of the vehicle 1) by receiving signals from positioning satellites. The GNSS receiver 21 transmits the measured position information of the vehicle 1 to the automatic driving ECU 30.

[0017] The external camera 22 is an imaging device that captures images of the external conditions of the vehicle 1. The external camera 22 is installed, for example, behind the windshield of the vehicle 1 and captures images of the area in front of the vehicle 1. The external camera 22 may be installed to the side or rear of the vehicle 1 and configured to capture images of the area around the vehicle 1. The external camera 22 transmits the captured images of the area outside the vehicle 1 to the automatic driving ECU 30.

[0018] The radar sensor 23 is a detection device that uses radio waves (e.g., millimeter waves) or light to detect objects around the vehicle 1. The radar sensor 23 includes, for example, millimeter-wave radar or lidar [LIDAR: Light Detection and Ranging] installed in multiple directions around the vehicle 1. The radar sensor 23 detects objects by transmitting radio waves or light around the vehicle and receiving the radio waves or light reflected by the objects. The radar sensor 23 transmits information about the detected objects to the autonomous driving ECU 30. Objects include fixed obstacles such as guardrails and buildings, as well as moving obstacles such as pedestrians, bicycles, and other vehicles.

[0019] The internal sensor 24 is a detection device that detects the driving state of the vehicle 1. The internal sensor 24 includes a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. The vehicle speed sensor is a detector that detects the speed of the vehicle 1. As the vehicle speed sensor, for example, a wheel speed sensor is used, which is provided on the wheels of the vehicle 1 or on a drive shaft that rotates integrally with the wheels, and detects the rotational speed of the wheels. The vehicle speed sensor transmits the detected vehicle speed information (wheel speed information) to the automatic driving ECU 30.

[0020] An acceleration sensor is a detector that detects the acceleration of the vehicle 1. The acceleration sensor includes, for example, a longitudinal acceleration sensor that detects the longitudinal acceleration of the vehicle 1 and a lateral acceleration sensor that detects the lateral acceleration of the vehicle 1. The acceleration sensor transmits the acceleration information of the vehicle 1 to the automatic driving ECU 30. A yaw rate sensor is a detector that detects the yaw rate (rotational angular velocity) of the vehicle 1 around the vertical axis of its center of gravity. For example, a gyro sensor can be used as the yaw rate sensor. The yaw rate sensor transmits the detected yaw rate information of the vehicle 1 to the automatic driving ECU 30.

[0021] The map database 25 is a database that stores map information. The map database 25 is formed, for example, in a storage device such as an HDD [Hard Disk Drive] mounted on the host vehicle 1. The map information includes road position information, road shape information (such as the type of curve, straight section, curvature of the curve, etc.), intersection and branch point position information, and structure position information. Note that the map database 25 may be formed in a server that can communicate with the host vehicle 1.

[0022] The driving operation detection unit 26 detects the driving operation of the host vehicle 1 by the driver. The driving operation detection unit 26 includes, for example, a steering sensor, an accelerator sensor, and a brake sensor. The driving operation detection unit 26 transmits operation amount information regarding the detected operation amount of the driver to the automatic driving ECU 30.

[0023] Next, the functional configuration of the automatic driving ECU 30 will be described. As shown in FIG. 1, the automatic driving ECU 30 includes a surrounding recognition unit 31, a vehicle control unit 32, and an abnormality determination device 50.

[0024] The surrounding recognition unit 31 performs surrounding recognition (recognition of the external environment) of the host vehicle 1 in order to execute the automatic driving of the host vehicle 1. The surrounding recognition unit 31 performs surrounding recognition of the host vehicle 1 based on at least one of the captured images of the external camera 22 and the detection results of the radar sensor 23. The surrounding recognition unit 31 may perform surrounding recognition of the host vehicle 1 by so-called sensor fusion.

[0025] The surrounding recognition unit 31 recognizes the lane lines of the road on which the host vehicle 1 is traveling and external objects (other vehicles, pedestrians, etc.) around the host vehicle 1. The surrounding recognition unit 31 may recognize objects such as guardrails, road signs, and road markings. The surrounding recognition unit 31 may obtain surrounding recognition results from surrounding vehicles that are other vehicles traveling around the host vehicle 1 through vehicle-to-vehicle communication. The surrounding recognition result is the result of surrounding recognition such as information on objects around the host vehicle 1. The surrounding recognition unit 31 may recognize the road shape, road width, position of the lane lines, etc. based on the position information of the host vehicle 1 recognized by the GNSS receiving unit 21 and the map information of the map database 25.

[0026] The vehicle control unit 32 performs automatic driving of the vehicle 1 based on the surrounding area recognition results from the surrounding area recognition unit 31 and the detection results from the internal sensors 24. The vehicle control unit 32 performs automatic driving by controlling the movement of the vehicle 1, for example, by transmitting control signals to engine actuators, brake actuators, and steering actuators (not shown).

[0027] If the vehicle control unit 32 detects an abnormality in the surrounding recognition performance of the automated driving system 100 by the abnormality detection device 50 while the vehicle 1 is being driven autonomously, it will warn the driver of the abnormality. If the vehicle control unit 32 is unable to continue autonomous driving, it will notify the driver that autonomous driving cannot be continued and then terminate the autonomous driving.

[0028] The abnormality detection device 50 includes an indicator recognition unit 51, a performance degradation detection unit 52, and an abnormality detection unit 53. The indicator recognition unit 51 recognizes multiple indicators used to determine a degradation in the surrounding recognition performance of the autonomous driving system 100 of the vehicle 1. During autonomous driving of the vehicle 1, the indicator recognition unit 51 recognizes multiple indicators based on the detection results of the vehicle 1's sensors (external camera 22, radar sensor 23, and driving operation detection unit 26, etc.).

[0029] The multiple indicators include at least two of the following: the reliability of the external camera 22, the reliability of the radar sensor 23, the reliability of the autonomous driving system 100's external object recognition, the reliability of the autonomous driving system 100's lane marking recognition, the continuity of the autonomous driving system 100's surrounding recognition results, the consistency of the recognition results of the external camera 22 and the radar sensor 23, the consistency between the surrounding recognition results obtained from surrounding vehicles located around the vehicle 1 via inter-vehicle communication and the surrounding recognition results of the autonomous driving system 100 of the vehicle 1, the number or frequency of abnormal behavior of the vehicle 1 during autonomous driving by the autonomous driving system 100, the number or frequency of driver intervention operations on the vehicle 1 during autonomous driving, the number or frequency of abnormal behavior of the vehicle 1 towards surrounding vehicles during autonomous driving, the number or frequency of abnormal behavior of the vehicle 1 towards lane markings during autonomous driving, and the number or frequency of abnormal behavior of surrounding vehicles towards the vehicle 1 during autonomous driving.

[0030] The reliability of the external camera 22 refers to the reliability of the images captured by the external camera 22. The reliability of the external camera 22 is calculated as a lower value the more continuous the images are blacked out or noisy. The reliability of the external camera 22 may also be calculated as a lower value the longer the time elapsed since maintenance was performed. The reliability of the external camera 22 may also be calculated as a lower value in environments such as rain, snow, fog, and dust compared to environments with clear skies. In addition, the reliability of the external camera 22 can be calculated using various well-known methods. Furthermore, the various reliability values ​​described later can also be calculated using various well-known methods. The reliability of the external camera 22 may also be calculated according to the orientation of the camera (front of the vehicle, rear of the vehicle, right side of the vehicle, left side of the vehicle, etc.).

[0031] The reliability of the radar sensor 23 is the reliability of the detection results of the radar sensor 23. The reliability of the radar sensor 23 is calculated as a low value when, for example, objects approaching the vehicle 1 in motion are continuously misdetected due to adhering substances such as dirt or snow. The reliability of the radar sensor 23 may also be calculated as a lower value the longer the time elapsed since maintenance was performed. The reliability of the radar sensor 23 may also be calculated as a lower value in environments such as rain, snow, fog, and dust compared to environments with clear skies. The reliability of the radar sensor 23 may also be calculated separately depending on the orientation of the radar sensor 23 (front of the vehicle, rear of the vehicle, right side of the vehicle, left side of the vehicle, etc.).

[0032] The reliability of external object recognition in the autonomous driving system 100 is the reliability of the recognition result of the surrounding recognition unit 31 of the vehicle 1 to external objects. The reliability of external object recognition is calculated as a lower value when, for example, the surrounding recognition unit 31 recognizes an abnormal state of an external object such as another vehicle, compared to when it does not recognize an abnormal state of an external object. An abnormal state is, for example, another vehicle driving in the wrong direction in the lane, another vehicle overlapping with a building or guardrail, or a pedestrian stopping in the middle of the lane. When the surrounding recognition unit 31 recognizes an abnormal behavior of an external object, there is a possibility that the external object is actually performing an abnormal behavior, as well as a possibility that an abnormality has occurred in the external camera 22, radar sensor 23, or autonomous driving ECU 30. The reliability of external object recognition may be calculated as a lower value the lower the reliability of the external camera 22 or the radar sensor 23.

[0033] The reliability of lane marking recognition in the autonomous driving system 100 is the reliability of the recognition result of the surrounding recognition unit 31 for the lane markings of the vehicle 1's driving lane. The reliability of lane marking recognition is calculated as a lower value if, for example, the lane markings are worn and cannot be clearly recognized, compared to when the lane markings are not worn. The reliability of lane marking recognition may also be calculated as a lower value if the reliability of the external camera 22 or the radar sensor 23 is low.

[0034] The continuity of the surrounding recognition results of the autonomous driving system 100 refers to the degree of continuity in the surrounding recognition results of the vehicle 1 by the surrounding recognition unit 31. The continuity of the surrounding recognition results is calculated as a higher value the longer the time for which the vehicle 1 can continuously recognize the position changes of surrounding vehicles. The continuity of the surrounding recognition results is calculated as a lower value when the vehicle 1 loses its sense of surrounding vehicles or when it suddenly recognizes a pedestrian within the detection range that it had not previously recognized. In addition, the continuity of the surrounding recognition results can be calculated using various well-known methods. Surrounding vehicles may be limited to vehicles traveling in front of the vehicle 1, or limited to vehicles traveling behind, or limited to vehicles traveling alongside.

[0035] The degree of consistency between the recognition results of the external camera 22 and the radar sensor 23 is the degree of agreement between the object information recognized from the image captured by the external camera 22 and the object information detected by the radar sensor 23. The object information includes the size of the object, the type of object, the position information of the object, the direction of movement of the object, and the speed of movement of the object. The position information of the object may be relative to the vehicle 1. The direction of movement and the speed of movement of the object can be similarly determined. The degree of consistency between the recognition results of the external camera 22 and the radar sensor 23 is calculated as a lower value the greater the discrepancy between the position information of the surrounding vehicle recognized from the image captured by the external camera 22 and the position information of the surrounding vehicle detected by the radar sensor 23. In addition, the degree of consistency between the recognition results of the external camera 22 and the radar sensor 23 can be calculated using various well-known methods.

[0036] The degree of consistency between the surrounding recognition results obtained from surrounding vehicles via vehicle-to-vehicle communication and the surrounding recognition results of the autonomous driving system 100 of the vehicle 1 is the degree of agreement between the object information obtained from surrounding vehicles via vehicle-to-vehicle communication and the object information obtained from the autonomous driving system 100 as surrounding recognition results. The indicator recognition unit 51 calculates the degree of consistency as a low value when, for example, the autonomous driving system 100 fails to recognize another vehicle that is recognized by surrounding vehicles but is not in a blind spot within a certain distance from the vehicle 1. In addition, the degree of consistency can be calculated using various well-known methods.

[0037] The number of abnormal behaviors of the vehicle 1 during autonomous driving refers to the number of times the vehicle 1 performed a predetermined abnormal behavior during autonomous driving. Abnormal behaviors include at least one of the following: sudden acceleration, sudden braking, sudden steering, unnecessary vehicle stopping, and violation of traffic rules. The indicator recognition unit 51 may use frequency (the number of times an abnormal behavior occurs in a certain period of time) instead of the number of abnormal behaviors of the vehicle 1 during autonomous driving. Abnormal behavior of the vehicle 1 may also be caused by a malfunction of the internal sensor 24 rather than by a malfunction in the surrounding recognition performance.

[0038] The number of driver intervention operations on the vehicle 1 during autonomous driving refers to the number of times the driver performed an intervention operation during autonomous driving. The indicator recognition unit 51 may use frequency (the number of intervention operations that occurred in a certain period of time) instead of the number of driver intervention operations. If the driver performs intervention operations frequently, there is a higher possibility that the surrounding recognition performance of the autonomous driving system 100 is deteriorating.

[0039] The number of abnormal behaviors of vehicle 1 towards surrounding vehicles during autonomous driving refers to the number of times vehicle 1, during autonomous driving, performed abnormal behaviors such as rapidly approaching surrounding vehicles. Abnormal behaviors include at least one of the following: rapidly approaching, lane changes that obstruct the path of surrounding vehicles, and vehicle 1 stopping while excessively close to surrounding vehicles. The indicator recognition unit 51 may use frequency (the number of occurrences of abnormal behavior in a certain period of time) instead of the number of abnormal behaviors of vehicle 1 towards surrounding vehicles during autonomous driving. Note that abnormal behaviors of vehicle 1 towards surrounding vehicles during autonomous driving may be included in the abnormal behaviors of vehicle 1 during autonomous driving described above.

[0040] The number of abnormal behaviors of the vehicle 1 with respect to the lane markings during autonomous driving refers to the number of times the vehicle 1, while autonomous driving, exhibited abnormal behaviors such as serpentine driving, repeatedly approaching and moving away from the lane markings. Abnormal behaviors include not only serpentine driving but also driving while straddling the lane markings. The indicator recognition unit 51 may use frequency (the number of occurrences of abnormal behavior in a certain period of time) instead of the number of abnormal behaviors of the vehicle 1 with respect to the lane markings during autonomous driving. Note that abnormal behaviors of the vehicle 1 with respect to the lane markings during autonomous driving may be included in the abnormal behaviors of the vehicle 1 during autonomous driving described above.

[0041] The number of abnormal behaviors of surrounding vehicles relative to the autonomous vehicle 1 during autonomous driving refers to the number of times surrounding vehicles detected within a certain distance from the autonomous vehicle 1 during autonomous driving exhibited abnormal behavior. Abnormal behavior includes at least one of the following: sudden acceleration, sudden braking, sudden steering, unnecessary vehicle stopping, and violation of traffic rules. The indicator recognition unit 51 may use frequency (the number of occurrences of abnormal behavior of surrounding vehicles in a certain period of time) instead of the number of abnormal behaviors of surrounding vehicles. The number or frequency of abnormal behaviors of surrounding vehicles may be integrated into the reliability of the external object recognition of the autonomous driving system 100.

[0042] Each of the multiple indicators is associated with at least one other indicator. This association is used to change the ease of anomaly detection, as will be described later. For example, the reliability of lane marking recognition is associated with the reliability of external object recognition. The reliability of the external camera 22 and the reliability of the radar sensor 23 may be associated with all other indicators.

[0043] The number or frequency of abnormal behavior of the vehicle 1 with respect to lane markings during autonomous driving may be associated with at least one of the reliability of lane marking recognition and the reliability of the external camera 22. If the abnormal behavior of the vehicle 1 with respect to lane markings increases, there may be a problem with lane marking recognition by the external camera 22 or the like.

[0044] The number or frequency of abnormal behaviors of the vehicle 1 in relation to surrounding vehicles during autonomous driving may be associated with at least one of the reliability of the external camera 22 and the reliability of the radar sensor 23. If the number of abnormal behaviors of the vehicle 1 in relation to surrounding vehicles such as preceding vehicles increases, there may be a problem with the detection of surrounding vehicles by the external camera 22 or radar sensor 23.

[0045] The number or frequency of driver interventions on the vehicle during autonomous driving may be correlated with the reliability of lane marking recognition. For example, an increase in driver deceleration interventions while autonomous driving is cornering may indicate a problem with lane marking recognition, such as a shortened recognition distance for lane markings.

[0046] The association of indicators may be changed according to the driving conditions of the vehicle 1. The indicator recognition unit 51 may associate the number or frequency of driver intervention operations with the reliability of lane marking recognition when the vehicle 1 is driving on a curve during autonomous driving, and may not associate the number or frequency of driver intervention operations with the reliability of lane marking recognition when the vehicle 1 is not driving on a curve. In this case, the intervention operations may be limited to deceleration operations such as pressing the brake pedal.

[0047] Similarly, the indicator recognition unit 51 may associate the number or frequency of driver intervention operations with the reliability of external object recognition for a certain period of time after the vehicle 1 performs a lane change during autonomous driving, and may not associate the number or frequency of driver intervention operations with the reliability of external object recognition outside of the aforementioned certain period of time. If the driver intervenes, such as accelerating the vehicle 1, after the vehicle 1 performs a lane change during autonomous driving, there may be problems in recognizing the relative distance and relative speed between the vehicle 1 and the following vehicle in the lane to which the vehicle changed. The intervention operation at this time may be limited to acceleration operations such as pressing the accelerator pedal. Instead of the reliability of external object recognition, the reliability of the radar sensor 23 that detects the rear of the vehicle 1 may be associated with the driver intervention operations. In addition, regardless of the driving conditions of the vehicle 1, the number or frequency of driver intervention operations may be associated with at least one of the reliability of external object recognition and the reliability of the radar sensor 23 that detects the rear.

[0048] Furthermore, in an external camera system 22 consisting of multiple cameras installed according to the orientation of the vehicle 1, the reliability of all external cameras 22 may be correlated with each other. Since the external cameras 22 often perform feature extraction processing using a common DNN (Deep Neural Network), if the reliability of one camera decreases, problems may occur in the other cameras as well.

[0049] The performance degradation determination unit 52 determines, for each indicator, whether or not a degradation in the surrounding recognition performance of the autonomous driving system 100 has occurred, based on a plurality of indicators obtained from the detection results of the sensors of the vehicle 1 (external camera 22, radar sensor 23, and driving operation detection unit 26, etc.) while the vehicle 1 is driving autonomously.

[0050] The performance degradation determination unit 52 determines a decrease in peripheral recognition performance using thresholds. The various thresholds described later are thresholds with predetermined values. For example, if the reliability of the external camera 22 is below the external camera threshold, the performance degradation determination unit 52 determines that a decrease in peripheral recognition performance related to the external camera 22 has occurred. Similarly, if the reliability of the radar sensor 23 is below the radar sensor threshold, the performance degradation determination unit 52 may determine that a decrease in peripheral recognition performance related to the radar sensor 23 has occurred.

[0051] The performance degradation determination unit 52 may determine that a degradation in peripheral recognition performance related to external object recognition has occurred if the reliability of the external object recognition of the automated driving system 100 is below the external object threshold. The performance degradation determination unit 52 may also determine that a degradation in peripheral recognition performance related to lane marking recognition has occurred if the reliability of the lane marking recognition of the automated driving system 100 is below the lane marking threshold.

[0052] The performance degradation determination unit 52 may determine that a degradation in peripheral recognition performance has occurred if the continuity of the peripheral recognition results of the autonomous driving system 100 is less than the continuity threshold. The performance degradation determination unit 52 may also determine that a degradation in peripheral recognition performance has occurred with respect to the external camera 22 or radar sensor 23 if the consistency of the recognition results of the external camera 22 and radar sensor 23 is less than the first consistency threshold.

[0053] The performance degradation determination unit 52 may determine that a degradation in surrounding recognition performance has occurred if the degree of consistency between the surrounding recognition results obtained from surrounding vehicles via inter-vehicle communication and the surrounding recognition results of the autonomous driving system 100 of the vehicle 1 is less than the second consistency threshold. The performance degradation determination unit 52 may also determine that a degradation in surrounding recognition performance has occurred if the number or frequency of abnormal behavior of the vehicle 1 during autonomous driving by the autonomous driving system 100 is equal to or greater than the first behavior threshold.

[0054] The performance degradation determination unit 52 may determine that a degradation in surrounding recognition performance has occurred if the number or frequency of driver intervention operations on the vehicle 1 during autonomous driving is equal to or greater than the intervention operation threshold. The performance degradation determination unit 52 may also determine that a degradation in surrounding recognition performance has occurred if the number or frequency of abnormal behavior of the vehicle 1 towards surrounding vehicles during autonomous driving is equal to or greater than the second behavior threshold.

[0055] The performance degradation determination unit 52 may determine that a degradation in surrounding recognition performance has occurred if the number or frequency of abnormal behavior of the vehicle 1 with respect to the lane markings during autonomous driving is equal to or greater than the third behavior threshold. The performance degradation determination unit 52 may also determine that a degradation in surrounding recognition performance has occurred if the number or frequency of abnormal behavior of surrounding vehicles with respect to the vehicle 1 during autonomous driving is equal to or greater than the fourth behavior threshold. Note that the performance degradation determination unit 52 does not necessarily need to make a determination for all indicators recognized by the indicator recognition unit 51.

[0056] The abnormality determination unit 53 determines whether or not there is an abnormality in the surrounding recognition performance of the autonomous driving system 100 based on the detection results of the sensors of the vehicle 1 (external camera 22, radar sensor 23, and driving operation detection unit 26, etc.). The abnormality determination unit 53 determines whether or not there is an abnormality in the surrounding recognition performance of the autonomous driving system 100 for each indicator, for example, based on the above-mentioned multiple indicators.

[0057] The anomaly detection unit 53 determines, for example, that a decrease in the peripheral recognition performance of the external camera 22 has occurred if the reliability of the external camera 22 is below the external camera anomaly threshold. Similarly, the anomaly detection unit 53 may determine that a decrease in the peripheral recognition performance of the radar sensor 23 has occurred if the reliability of the radar sensor 23 is below the radar sensor anomaly threshold.

[0058] The abnormality determination unit 53 may determine that a decrease in peripheral recognition performance related to external object recognition has occurred if the reliability of the external object recognition of the automated driving system 100 is below the external object abnormality threshold. The abnormality determination unit 53 may also determine that an abnormality in peripheral recognition performance related to lane marking recognition has occurred if the reliability of the lane marking recognition of the automated driving system 100 is below the lane marking abnormality threshold.

[0059] The abnormality determination unit 53 may determine that an abnormality in the surrounding recognition performance has occurred if the continuity of the surrounding recognition results of the autonomous driving system 100 is less than the continuity abnormality threshold. The abnormality determination unit 53 may also determine that an abnormality in the surrounding recognition performance related to the external camera 22 or the radar sensor 23 has occurred if the consistency of the recognition results of the external camera 22 and the radar sensor 23 is less than the first consistency abnormality threshold.

[0060] The abnormality determination unit 53 may determine that an abnormality in surrounding recognition performance has occurred if the degree of consistency between the surrounding recognition results obtained from surrounding vehicles via inter-vehicle communication and the surrounding recognition results of the autonomous driving system 100 of the vehicle 1 is less than the second consistency abnormality threshold. The abnormality determination unit 53 may also determine that an abnormality in surrounding recognition performance has occurred if the number or frequency of abnormal behaviors of the vehicle 1 during autonomous driving by the autonomous driving system 100 is equal to or greater than the first behavior abnormality threshold.

[0061] The abnormality determination unit 53 may determine that an abnormality in surrounding recognition performance has occurred if the number or frequency of driver intervention operations on the vehicle 1 during autonomous driving is equal to or greater than the intervention operation abnormality threshold. The abnormality determination unit 53 may also determine that an abnormality in surrounding recognition performance has occurred if the number or frequency of abnormal behavior of the vehicle 1 towards surrounding vehicles during autonomous driving is equal to or greater than the second behavior abnormality threshold.

[0062] The abnormality determination unit 53 may determine that an abnormality in surrounding recognition performance has occurred if the number or frequency of abnormal behavior of the vehicle 1 with respect to the lane markings during autonomous driving is equal to or greater than the third behavioral abnormality threshold. The abnormality determination unit 53 may also determine that an abnormality in surrounding recognition performance has occurred if the number or frequency of abnormal behavior of surrounding vehicles with respect to the vehicle 1 during autonomous driving is equal to or greater than the fourth behavioral abnormality threshold.

[0063] Furthermore, the abnormality determination unit 53 does not necessarily need to make a determination on all indicators recognized by the indicator recognition unit 51. The abnormality determination unit 53 may make a determination on only some of the indicators. The indicators used by the abnormality determination unit 53 for determination may overlap with or differ from the indicators used by the performance degradation determination unit 52 for determination.

[0064] If the performance degradation determination unit 52 determines that a degradation in peripheral recognition performance has occurred based on any indicator, the abnormality determination unit 53 makes it easier to determine that there is an abnormality in the peripheral recognition performance of the autonomous driving system 100 compared to when no degradation in peripheral recognition performance has been determined.

[0065] The anomaly detection unit 53 makes it easier to determine if there is an anomaly in the surrounding recognition performance by changing the values ​​of the thresholds described above. For example, the anomaly detection unit 53 can make it easier to determine if there is an anomaly in the surrounding recognition performance by increasing the values ​​of the external camera anomaly threshold, external object anomaly threshold, lane line anomaly threshold, continuity anomaly threshold, first consistency anomaly threshold, and second consistency anomaly threshold. For the first behavior anomaly threshold, intervention operation anomaly threshold, second behavior anomaly threshold, third behavior anomaly threshold, and fourth behavior anomaly threshold, the anomaly detection unit 53 can make it easier to determine if there is an anomaly in the surrounding recognition performance by decreasing the values ​​of the thresholds.

[0066] The abnormality determination unit 53 may, when the performance degradation determination unit 52 determines that a degradation in peripheral recognition performance has occurred, make it easier to determine that there is an abnormality in peripheral recognition performance based on other indicators associated with the indicator used in the said determination, without changing the ease with which abnormalities in peripheral recognition performance based on other indicators not associated with the said indicator are determined.

[0067] Specifically, if the performance degradation determination unit 52 determines that a degradation in peripheral recognition performance has occurred based on the reliability of lane marking recognition, the abnormality determination unit 53 makes it easier to determine that there is an abnormality in peripheral recognition performance based on the reliability of external object recognition, which is associated with the reliability of lane marking recognition. The abnormality determination unit 53 makes it easier to determine that there is an abnormality in peripheral recognition performance based on the reliability of external object recognition by changing the external object abnormality threshold to a larger value compared to when no degradation in peripheral recognition performance has been determined. The abnormality determination unit 53 does not change the ease with which it can determine an abnormality in peripheral recognition performance based on other indicators that are not associated with the reliability of lane marking recognition.

[0068] Similarly, if the abnormality determination unit 53 determines that a decrease in surrounding recognition performance has occurred based on the number or frequency of abnormal behavior of the vehicle 1 with respect to lane markings during autonomous driving, it may be made easier to determine that there is an abnormality in surrounding recognition performance based on the reliability of lane marking recognition or the reliability of the external camera 22 associated with the said indicator.

[0069] If the abnormality determination unit 53 determines that a decrease in surrounding recognition performance has occurred based on the number or frequency of abnormal behavior of the vehicle 1 towards surrounding vehicles during autonomous driving, it may make it easier to determine that there is an abnormality in surrounding recognition performance based on the reliability of the external camera 22 or the radar sensor 23 associated with the said indicator. If the abnormality determination unit 53 determines that a decrease in surrounding recognition performance has occurred based on the number or frequency of driver intervention operations, it may make it easier to determine that there is an abnormality in surrounding recognition performance based on the reliability of lane marking recognition associated with the said indicator.

[0070] If the abnormality detection unit 53 determines that an abnormality has occurred in the surrounding recognition performance of the automated driving system 100, it transmits an abnormality signal to the vehicle control unit 32. The vehicle control unit 32, for example, warns the driver of the abnormality and terminates automated driving. The vehicle control unit 32 gives the driver time to prepare to switch to manual driving. Depending on the type of abnormality detected, the vehicle control unit 32 does not necessarily have to terminate automated driving; it may simply warn the driver.

[0071] Next, the control method of the abnormality detection device 50 according to this embodiment will be described with reference to the drawings. Figure 2 is a flowchart showing an example of the performance degradation detection process. The performance degradation detection process is executed when the vehicle 1 is in automatic driving mode.

[0072] As shown in Figure 2, the anomaly detection device 50, as S1, recognizes multiple indicators using the indicator recognition unit 51. These multiple indicators are used to determine a decrease in the surrounding recognition performance of the autonomous driving system 100. The indicator recognition unit 51 recognizes indicators such as the reliability of the external camera 22 and the reliability of the radar sensor 23 using well-known methods.

[0073] In S2, the abnormality detection device 50 uses the performance degradation determination unit 52 to determine whether or not a degradation has occurred in the surrounding recognition performance of the autonomous driving system 100 for each indicator. If the abnormality detection device 50 determines that a degradation has occurred in the surrounding recognition performance of the autonomous driving system 100 (S2: YES), it proceeds to S3. If the abnormality detection device 50 does not determine that a degradation has occurred in the surrounding recognition performance of the autonomous driving system 100 (S2: NO), it terminates the performance degradation determination process.

[0074] In S3, the abnormality detection device 50 makes it easier for the abnormality detection unit 53 to determine that there is an abnormality in the peripheral recognition performance of the automated driving system 100. The abnormality detection unit 53 makes it easier for the abnormality in peripheral recognition performance to be determined based on other indicators associated with the indicator used by the performance degradation detection unit 52 to determine the degradation of peripheral recognition performance. Specifically, the abnormality detection unit 53 makes it easier for the abnormality in peripheral recognition performance to be determined by changing the value of the threshold used for the determination. The abnormality detection unit 53 does not change the ease with which it determines an abnormality in peripheral recognition performance based on other indicators not associated with the said indicator. After that, the abnormality detection device 50 terminates the current performance degradation determination process.

[0075] Figure 3 is a flowchart showing an example of the abnormality detection process. The abnormality detection process is executed when vehicle 1 is in autonomous driving mode.

[0076] As shown in Figure 3, the abnormality detection device 50, in step S10, recognizes multiple indicators using the indicator recognition unit 51. These multiple indicators are used to determine abnormalities in the surrounding recognition performance of the automated driving system 100.

[0077] In S11, the abnormality determination device 50 instructs the abnormality determination unit 53 to determine whether or not there is an abnormality in the surrounding recognition performance of the automated driving system 100. The abnormality determination unit 53 determines, for example, whether or not an abnormality in the surrounding recognition performance has occurred for each indicator based on multiple indicators. If the abnormality determination device 50 determines that an abnormality in the surrounding recognition performance has occurred (S11: YES), it proceeds to S12. If the abnormality determination device 50 does not determine that an abnormality in the surrounding recognition performance has occurred (S11: NO), it terminates the current abnormality determination process.

[0078] In S12, the abnormality detection device 50 transmits an abnormality occurrence signal to the vehicle control unit 32 via the abnormality detection unit 53. After that, the abnormality detection device 50 terminates the current abnormality detection process.

[0079] According to the abnormality detection device 50 (autonomous driving system 100) of this embodiment described above, when it is determined that a decrease in the surrounding recognition performance of the autonomous driving system has occurred based on multiple indicators, it is easier to determine that there is an abnormality in the surrounding recognition performance of the autonomous driving system compared to when it is not determined that a decrease in surrounding recognition performance has occurred. Therefore, the existence of an abnormality in surrounding recognition performance can be determined early based on the warning sign of a decrease in surrounding recognition performance.

[0080] According to the anomaly detection device 50, multiple indicators that are linked when an anomaly occurs are pre-associated, making it easier to determine that there is an anomaly in the surrounding recognition performance based on the indicator that was the determining factor for the deterioration of the surrounding recognition performance and other indicators associated with it, thereby enabling early detection of the existence of an anomaly in the surrounding recognition performance. For example, if the anomaly detection device 50 determines that a deterioration in the surrounding recognition performance has occurred based on the reliability of lane marking recognition, it makes it easier to determine that there is an anomaly in the surrounding recognition performance based on the reliability of external object recognition, which is related to the reliability of lane marking recognition, thereby enabling early detection of the existence of an anomaly in the surrounding recognition performance.

[0081] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. The present invention can be implemented in various forms, starting with the embodiments described above, by making various changes and improvements based on the knowledge of those skilled in the art.

[0082] The abnormality detection unit 53 may determine whether there is an abnormality in the surrounding recognition performance of the autonomous driving system 100 without using the multiple indicators described above. For example, without calculating the reliability of the external camera 22, the abnormality detection unit 53 may determine that there is an abnormality in the surrounding recognition performance of the autonomous driving system 100 if the image captured by the external camera 22, which is necessary for autonomous driving, remains black for a certain period of time or longer. Similarly, without calculating the reliability of the radar sensor 23, the abnormality detection unit 53 may determine that there is an abnormality in the surrounding recognition performance of the autonomous driving system 100 if the radar sensor 23, which is necessary for autonomous driving, remains silent for a certain period of time or longer. The abnormality detection unit 53 can make it easier to determine an abnormality by shortening the value of the certain period of time. In addition, the abnormality detection unit 53 may determine whether there is an abnormality in the surrounding recognition performance of the autonomous driving system 100 using various well-known methods.

[0083] The multiple indicators mentioned above do not need to be associated with any other indicators. It is even possible for no indicators to be associated with each other at all. In this case, if the performance degradation determination unit 52 determines that a degradation in peripheral recognition performance has occurred, the anomaly determination unit 53 will be more likely to determine an anomaly in peripheral recognition performance based on all indicators and other factors. [Explanation of Symbols]

[0084] 1...Vehicle, 22...External camera, 23...Radar sensor, 30...Automated driving ECU, 50...Anomaly detection device, 51...Indicator recognition unit, 52...Performance degradation detection unit, 53...Anomaly detection unit, 100...Automated driving system.

Claims

1. An abnormality determination device that determines whether or not there is an abnormality in the automatic driving system of the vehicle based on the detection results of the vehicle's sensors, During the autonomous driving of the vehicle, the system includes a performance degradation determination unit that determines whether or not an abnormality in the surrounding recognition performance of the autonomous driving system has occurred for each of the multiple indicators obtained from the detection results of the vehicle's sensors. If the performance degradation determination unit determines that a degradation in the surrounding recognition performance has occurred based on any of the indicators, it will be easier to determine that there is an abnormality in the surrounding recognition performance of the autonomous driving system compared to when no degradation in the surrounding recognition performance has been determined. An abnormality detection device, wherein the plurality of indicators include at least two of the following: the reliability of the vehicle's external camera, the reliability of the vehicle's radar sensor, the reliability of the autonomous driving system's external object recognition, the reliability of the autonomous driving system's lane marking recognition, the continuity of the autonomous driving system's surrounding recognition results, the consistency of the recognition results of the vehicle's external camera and radar sensor, the consistency between the surrounding recognition results obtained from surrounding vehicles located around the vehicle via vehicle-to-vehicle communication and the surrounding recognition results of the autonomous driving system, the number or frequency of abnormal behavior of the vehicle during autonomous driving by the autonomous driving system, the number or frequency of driver intervention operations on the vehicle during autonomous driving, the number or frequency of abnormal behavior of the vehicle towards surrounding vehicles during autonomous driving, the number or frequency of abnormal behavior of the vehicle towards lane markings during autonomous driving, and the number or frequency of abnormal behavior of surrounding vehicles towards the vehicle during autonomous driving.

2. Whether or not there is an abnormality in the surrounding recognition performance of the autonomous driving system is determined for each of the multiple indicators based on the above indicators. Each of the aforementioned multiple indicators is associated with at least one other aforementioned indicator, An abnormality determination device according to claim 1, wherein, when the performance degradation determination unit determines that a degradation in the peripheral recognition performance has occurred, the device makes it easier to determine that there is an abnormality in the peripheral recognition performance based on other indicators associated with the indicator used in the determination, without changing the ease with which an abnormality in the peripheral recognition performance can be determined based on other indicators not associated with the said indicator.

3. Of the aforementioned indicators, the reliability of the lane line recognition is related to the reliability of the external object recognition. The abnormality determination device according to claim 2, wherein if the performance degradation determination unit determines that a degradation in the surrounding recognition performance has occurred based on the reliability of the road mark recognition, it makes it easier to determine that there is an abnormality in the surrounding recognition performance based on the reliability of the external object recognition.

Images