Abnormality determination device

Abstract

To determine existence of an abnormality of periphery recognition performance at an early stage on the basis of the sign being deterioration of the periphery recognition performance in an automatic driving system.SOLUTION: There is provided an abnormality determination device that determines the presence / absence of any abnormality in an automatic driving system of an own vehicle. The abnormality determination device comprises a performance deterioration determination unit configured to determine, on the basis of a plurality of indices obtained by detection results of sensors of the own vehicle, whether or not deterioration of periphery recognition performance of the automatic driving system for each of the indices, during automatic driving of the own vehicle. The abnormality determination device facilitates determination of existence of the abnormality of the periphery recognition performance of the automatic driving system easier when the performance deterioration determination unit determines that the deterioration of the periphery recognition performance has occurred, compared to the case of not determining that the deterioration of the periphery recognition performance has occurred.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an abnormality determination device. [Background technology]

[0002] Conventionally, JP 2022-032109 A is known as a technical document related to an abnormality determination device. This publication describes a collision prevention system that uses detection results obtained from multiple different types of detectors to determine the reliability of the movement characteristics of an object that is to be determined as a collision, and determines that the error in the calculated posture or velocity vector of the target object is large and the reliability is reduced based on a predetermined index (the deviation between the posture and velocity vector of the target object). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-032109 A Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional systems, reliability is determined using detection results obtained from multiple different types of detectors, but no ingenuity has been disclosed for early detection of abnormalities that would significantly reduce reliability. Early detection of abnormalities is particularly important in autonomous driving systems. Therefore, the present invention aims to provide a device that can perform early detection of abnormalities in autonomous driving systems. [Means for solving the problem]

[0005] One aspect of the present invention is an abnormality determination device that determines whether or not there is an abnormality in the autonomous driving system of a vehicle, and is equipped with a performance degradation determination unit that determines whether or not a degradation in the surrounding recognition performance of the autonomous driving system has occurred for each of a plurality of indicators obtained from the detection results of the vehicle's sensors during autonomous driving of the vehicle.When the performance degradation determination unit determines that a degradation in surrounding recognition performance has occurred, it makes it easier to determine that there is an abnormality in the surrounding recognition performance of the autonomous driving system compared to when it is not determined that a degradation in surrounding recognition performance has occurred.

[0006] According to an abnormality determination device of one embodiment of the present invention, when it is determined that a decline in the peripheral recognition performance of an autonomous driving system has occurred based on multiple indicators, it is easier to determine that there is an abnormality in the peripheral recognition performance of the autonomous driving system compared to when it is not determined that a decline in the peripheral recognition performance has occurred. Therefore, it is possible to determine the existence of an abnormality in the peripheral recognition performance early on based on the signs of a decline in the peripheral recognition performance.

[0007] In an abnormality determination device according to one aspect of the present invention, the multiple indicators may include at least two of the following: reliability of the external camera of the host vehicle, reliability of the radar sensor of the host vehicle, reliability of external object recognition by the autonomous driving system, reliability of lane marking recognition by the autonomous driving system, continuity of the surrounding recognition results of the autonomous driving system, consistency of the recognition results of the external camera and radar sensor of the host vehicle, consistency between the surrounding recognition results obtained via vehicle-to-vehicle communication from surrounding vehicles located around the host vehicle and the surrounding recognition results of the autonomous driving system, the number or frequency of abnormal behavior of the host vehicle during autonomous driving by the autonomous driving system, the number or frequency of driver intervention operations regarding the host vehicle during autonomous driving, the number or frequency of abnormal behavior of the host vehicle relative to surrounding vehicles during autonomous driving, and the number or frequency of abnormal behavior of the host vehicle relative to lane marks during autonomous driving, and the number or frequency of abnormal behavior of surrounding vehicles relative to the host vehicle during autonomous driving. Effect of the Invention

[0008] According to each aspect of the present invention, in an autonomous driving system, it is possible to determine early on whether there is an abnormality in surrounding recognition performance based on the indication of a decline in surrounding recognition performance. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an automatic driving system (an abnormality determination device) according to an embodiment. FIG. [Diagram 2] 13 is a flowchart illustrating an example of a performance degradation determination process. [Diagram 3] 5 is a flowchart illustrating an example of an abnormality determination process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] FIG. 1 is a block diagram showing an autonomous driving system (abnormality determination device) 100 according to an embodiment. The autonomous driving system 100 shown in FIG. 1 is a system that executes autonomous driving of a host vehicle 1. Autonomous driving is vehicle control that automatically drives the host vehicle 1 along a preset route or along a road on which the host vehicle 1 is traveling, without the driver performing any driving operation. Autonomous driving corresponds to, for example, autonomous driving level 2 or higher control certified by the Society of Automotive Engineers (SAE). Autonomous driving may be limited to autonomous driving level 3 or higher control.

[0012] When the driver performs an operation to start automatic driving, the automatic driving system 100 starts automatic driving of the host vehicle 1. When the driver performs an intervention operation (driving operation during automatic driving) during automatic driving, the automatic driving system 100 transitions to an override state. In the override state, automatic driving continues while the driver's driving operation is reflected in the traveling of the host vehicle 1.

[0013] The autonomous driving system 100 is configured to include an abnormality determination device 50. The abnormality determination device 50 is a device that determines whether or not there is an abnormality in the surrounding recognition performance of the autonomous driving system 100. The surrounding recognition performance is performance related to surrounding recognition (recognition of the external environment) of the host vehicle 1. Note that the abnormality determination device 50 does not necessarily need to constitute a part of the autonomous driving system 100, and may be provided independently of the autonomous driving system 100.

[0014] The autonomous driving system 100 includes an autonomous driving ECU 30 [Electronic Control Unit]. The autonomous driving ECU 30 is an electronic control unit having a CPU [Central Processing Unit] and a storage unit. The storage unit is composed of, for example, a ROM [Read Only Memory], a RAM [Random Access Memory], and an EEPROM [Electrically Erasable Programmable Read-Only Memory]. The autonomous driving ECU 30 realizes various functions by executing programs stored in the storage unit with the CPU. The autonomous driving ECU 30 may be composed of multiple electronic units.

[0015] As shown in FIG. 1, the autonomous driving ECU 30 is connected to a GNSS receiver 21, an external camera 22, a radar sensor 23, an internal sensor 24, a map database 25, and a driving operation detector 26.

[0016] The GNSS receiver 21 receives signals from positioning satellites to measure the position of the vehicle 1 (for example, the latitude and longitude of the vehicle 1). The GNSS receiver 21 transmits the measured position information of the vehicle 1 to the autonomous driving ECU 30.

[0017] The external camera 22 is an imaging device that captures an image of an external situation of the host vehicle 1. The external camera 22 is provided, for example, on the back side of the windshield of the host vehicle 1, and captures an image in front of the host vehicle 1. The external camera 22 may be provided on the side or rear of the host vehicle 1, and configured to be able to capture an image of the surroundings of the host vehicle 1. The external camera 22 transmits the captured image of the outside of the host vehicle 1 to the autonomous driving ECU 30.

[0018] The radar sensor 23 is a detection device that detects objects around the vehicle 1 by using radio waves (e.g., millimeter waves) or light. The radar sensor 23 includes, for example, a millimeter wave radar or a lidar (LIDAR: Light Detection and Ranging) provided in multiple directions of the vehicle 1. The radar sensor 23 detects objects by transmitting radio waves or light to the surroundings of the vehicle and receiving the radio waves or light reflected by the objects. The radar sensor 23 transmits information on the detected objects to the autonomous driving ECU 30. The objects include fixed obstacles such as guardrails and buildings, as well as moving obstacles such as pedestrians, bicycles, and other vehicles.

[0019] The internal sensor 24 is a detection device that detects the traveling state of the host vehicle 1. The internal sensor 24 includes a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. The vehicle speed sensor is a detector that detects the speed of the host vehicle 1. As the vehicle speed sensor, for example, a wheel speed sensor that is provided on the wheels of the host vehicle 1 or a drive shaft that rotates integrally with the wheels and detects the rotation speed of the wheels is used. The vehicle speed sensor transmits the detected vehicle speed information (wheel speed information) to the autonomous driving ECU 30.

[0020] The acceleration sensor is a detector that detects the acceleration of the host vehicle 1. The acceleration sensor includes, for example, a longitudinal acceleration sensor that detects the acceleration in the longitudinal direction of the host vehicle 1, and a lateral acceleration sensor that detects the lateral acceleration of the host vehicle 1. The acceleration sensor transmits, for example, acceleration information of the host vehicle 1 to the automatic driving ECU 30. The yaw rate sensor is a detector that detects the yaw rate (rotational angular velocity) around the vertical axis of the center of gravity of the host vehicle 1. For example, a gyro sensor can be used as the yaw rate sensor. The yaw rate sensor transmits the detected yaw rate information of the host vehicle 1 to the automatic driving ECU 30.

[0021] The map database 25 is a database that stores map information. The map database 25 is formed, for example, in a storage device such as a HDD (Hard Disk Drive) mounted on the vehicle 1. The map information includes road position information, road shape information (for example, curves, types of straight sections, curvature of curves, etc.), intersection and branch point position information, and structure position information. The map database 25 may be formed in a server that can communicate with the vehicle 1.

[0022] The driving operation detection unit 26 detects the driving operation of the vehicle 1 by the driver. The driving operation detection unit 26 includes, for example, a steering sensor, an accelerator sensor, and a brake sensor. The driving operation detection unit 26 transmits operation amount information related to the detected driver's operation amount to the autonomous driving ECU 30.

[0023] Next, a description will be given of the functional configuration of the autonomous driving ECU 30. As shown in FIG.

[0024] The surroundings recognition unit 31 recognizes the surroundings of the vehicle 1 (recognizes the external environment) in order to execute autonomous driving of the vehicle 1. The surroundings recognition unit 31 recognizes the surroundings of the vehicle 1 based on at least one of an image captured by the external camera 22 and a detection result of the radar sensor 23. The surroundings recognition unit 31 may recognize the surroundings of the vehicle 1 by so-called sensor fusion.

[0025] The surroundings recognition unit 31 recognizes the dividing lines of the road on which the host vehicle 1 is traveling and external objects (other vehicles, pedestrians, etc.) around the host vehicle 1. The surroundings recognition unit 31 may recognize objects such as guardrails, road signs, and road markings. The surroundings recognition unit 31 may acquire surroundings recognition results from surrounding vehicles, which are other vehicles traveling around the host vehicle 1, through vehicle-to-vehicle communication. The surroundings recognition results are the results of surroundings recognition such as information on objects around the host vehicle 1. The surroundings recognition unit 31 may recognize the road shape, road width, position of dividing lines, etc. based on the position information of the host vehicle 1 recognized by the GNSS receiving unit 21 and the map information of the map database 25.

[0026] The vehicle control unit 32 executes automatic driving of the host vehicle 1 based on the surrounding recognition result by the surrounding recognition unit 31 and the detection result of the internal sensor 24. The vehicle control unit 32 executes automatic driving by controlling the traveling of the host vehicle 1, for example, by transmitting control signals to an engine actuator, a brake actuator, and a steering actuator (not shown).

[0027] When the abnormality determination device 50 determines that there is an abnormality in the surrounding recognition performance of the automatic driving system 100 during automatic driving of the vehicle 1, the vehicle control unit 32 warns the driver of the occurrence of the abnormality. When the automatic driving cannot be continued, the vehicle control unit 32 notifies the driver that the automatic driving cannot be continued, and then ends the automatic driving.

[0028] The abnormality determination device 50 has an indicator recognition unit 51, a performance degradation determination unit 52, and an abnormality determination unit 53. The indicator recognition unit 51 recognizes a plurality of indicators used to determine degradation of the surrounding recognition performance of the autonomous driving system 100 of the host vehicle 1. The indicator recognition unit 51 recognizes a plurality of indicators based on detection results of sensors (such as the external camera 22, the radar sensor 23, and the driving operation detection unit 26) of the host vehicle 1 during autonomous driving of the host vehicle 1.

[0029] The multiple indicators include at least two of the following: reliability of the external camera 22, reliability of the radar sensor 23, reliability of external object recognition of the autonomous driving system 100, reliability of lane marking recognition of the autonomous driving system 100, continuity of the surrounding recognition results of the autonomous driving system 100, consistency of the recognition results of the external camera 22 and the radar sensor 23, consistency between the surrounding recognition results obtained from surrounding vehicles located around the host vehicle 1 via vehicle-to-vehicle communication and the surrounding recognition results of the autonomous driving system 100 of the host vehicle 1, the number or frequency of abnormal behavior of the host vehicle 1 during autonomous driving by the autonomous driving system 100, the number or frequency of driver intervention operations regarding the host vehicle 1 during autonomous driving, the number or frequency of abnormal behavior of the host vehicle 1 toward surrounding vehicles during autonomous driving, the number or frequency of abnormal behavior of the host vehicle 1 toward lane marks during autonomous driving, and the number or frequency of abnormal behavior of surrounding vehicles toward the host vehicle 1 during autonomous driving.

[0030] The reliability of the external camera 22 is the reliability of the image captured by the external camera 22. The reliability of the external camera 22 is calculated as a lower value, for example, the longer the image is filled with black or noise. The reliability of the external camera 22 may be calculated as a lower value, the longer the time has passed since maintenance was performed. The reliability of the external camera 22 may be calculated as a lower value in an environment of rain, snow, fog, dust, etc., compared to an environment of fine weather. In addition, the reliability of the external camera 22 can be calculated by various well-known methods. Note that various reliabilities described later can also be calculated by various well-known methods. The reliability of the external camera 22 may be calculated according to the orientation of the camera (front of the vehicle, rear of the vehicle, right side of the vehicle, left side of the vehicle, etc.).

[0031] The reliability of the radar sensor 23 is the reliability of the detection result of the radar sensor 23. The reliability of the radar sensor 23 is calculated as a low value when, for example, an object close to the vehicle 1 while traveling is continuously erroneously detected due to adhesion of dirt, snow, or the like. The reliability of the radar sensor 23 may be calculated as a lower value the longer the elapsed time since maintenance was performed. The reliability of the radar sensor 23 may be calculated as a lower value in an environment of rain, snow, fog, dust, or the like compared to an environment of fine weather. The reliability of the radar sensor 23 may be calculated according to the orientation of the radar sensor 23 (front of the vehicle, rear of the vehicle, right side of the vehicle, left side of the vehicle, etc.).

[0032] The reliability of the external object recognition of the autonomous driving system 100 is the reliability of the recognition result of the external object of the host vehicle 1 by the periphery recognition unit 31. For example, when the periphery recognition unit 31 recognizes an abnormal state of an external object such as another vehicle, the reliability of the external object recognition is calculated as a lower value than when the abnormal state of the external object is not recognized. The abnormal state is, for example, a state in which another vehicle is running in the wrong direction in a lane, a state in which another vehicle overlaps with a building or a guardrail, or a pedestrian stops in the center of a lane. When the periphery recognition unit 31 recognizes an abnormal behavior of an external object, in addition to the possibility that the external object is actually performing an abnormal behavior, there is a possibility that an abnormality has occurred in the external camera 22, the radar sensor 23, or the autonomous driving ECU 30. The reliability of the external object recognition may be calculated as a lower value as the reliability of the external camera 22 or the reliability of the radar sensor 23 is lower.

[0033] The reliability of lane marking recognition by the autonomous driving system 100 is the reliability of the recognition result of the lane markings of the lane in which the vehicle 1 is traveling by the surrounding recognition unit 31. The reliability of lane marking recognition is calculated to be a lower value, for example, when the lane markings are rubbed and cannot be clearly recognized, compared to when the lane markings are not rubbed. The reliability of lane marking recognition may be calculated to be a lower value as the reliability of the external camera 22 or the reliability of the radar sensor 23 is lower.

[0034] The continuity of the surrounding recognition result of the autonomous driving system 100 is the degree of continuity of the surrounding recognition result of the host vehicle 1 by the surrounding recognition unit 31. The continuity of the surrounding recognition result is calculated as a higher value, for example, the longer the time during which the position change of the surrounding vehicles of the host vehicle 1 can be continuously recognized. The continuity of the surrounding recognition result is calculated as a lower value when the surrounding vehicles of the host vehicle 1 are lost or when a pedestrian within the detection range that could not be recognized is suddenly recognized. In addition, the continuity of the surrounding recognition result can be calculated by various well-known methods. The surrounding vehicles may be limited to the preceding vehicle traveling in front of the host vehicle 1, may be limited to the following vehicle, or may be limited to the vehicle traveling parallel to the host vehicle.

[0035] The consistency of the recognition results of the external camera 22 and the radar sensor 23 is the degree of agreement between the information of the object recognized from the image captured by the external camera 22 and the information of the object detected by the radar sensor 23. The information of the object includes the size of the object, the type of the object, the position information of the object, the moving direction of the object, and the moving speed of the object. The position information of the object may be relative position information based on the vehicle 1. The moving direction and the moving speed of the object may be similar. The consistency of the recognition results of the external camera 22 and the radar sensor 23 is calculated as a value that is lower as the position information of the surrounding vehicle recognized from the image captured by the external camera 22 and the position information of the surrounding vehicle detected by the radar sensor 23 diverge, for example. In addition, the consistency of the recognition results of the external camera 22 and the radar sensor 23 can be calculated by various well-known methods.

[0036] The consistency between the surrounding recognition result acquired from the surrounding vehicles through vehicle-to-vehicle communication and the surrounding recognition result of the autonomous driving system 100 of the vehicle 1 is the degree of agreement between the object information as the surrounding recognition result acquired from the surrounding vehicles through vehicle-to-vehicle communication and the object information as the surrounding recognition result of the autonomous driving system 100. For example, when the autonomous driving system 100 cannot recognize another vehicle that is not in a blind spot within a certain distance from the vehicle 1 and is recognized by the surrounding vehicles, the indicator recognition unit 51 calculates the consistency as a low value. In addition, the consistency can be calculated by various well-known methods.

[0037] The number of abnormal behaviors of the vehicle 1 during autonomous driving is the number of times the vehicle 1 performs a predetermined abnormal behavior during autonomous driving. The abnormal behaviors include at least one of sudden acceleration, sudden braking, sudden steering, unnecessary vehicle stops, violation of traffic rules, etc. The indicator recognition unit 51 may use the frequency (the number of times abnormal behaviors occur within a certain period of time) instead of the number of abnormal behaviors of the vehicle 1 during autonomous driving. The abnormal behavior of the vehicle 1 may occur due to an abnormality in the internal sensor 24 rather than the surrounding recognition performance.

[0038] The number of times the driver performs an intervention operation on the vehicle 1 during autonomous driving is the number of times the driver performs an intervention operation during autonomous driving. The indicator recognition unit 51 may use the frequency (the number of times an intervention operation occurs within a certain period of time) instead of the number of times the driver performs an intervention operation. When the driver frequently performs an intervention operation, there is a high possibility that the surrounding recognition performance of the autonomous driving system 100 is degraded.

[0039] The number of times that the host vehicle 1 behaves abnormally with respect to surrounding vehicles during autonomous driving is the number of times that the host vehicle 1 behaves abnormally with respect to surrounding vehicles, such as approaching suddenly while driving autonomously. Abnormal behavior includes at least one of the following, in addition to approaching suddenly: changing lanes that obstruct the path of surrounding vehicles; and stopping the host vehicle 1 while excessively approaching surrounding vehicles. The indicator recognition unit 51 may use a frequency (the number of times abnormal behavior occurs in a certain period of time) instead of the number of times that the host vehicle 1 behaves abnormally with respect to surrounding vehicles during autonomous driving. Note that abnormal behavior of the host vehicle 1 with respect to surrounding vehicles during autonomous driving may be included in the abnormal behavior of the host vehicle 1 during autonomous driving described above.

[0040] The number of times that the vehicle 1 exhibits abnormal behavior with respect to a marking line during autonomous driving is the number of times that the vehicle 1 exhibits abnormal behavior, such as meandering, which repeatedly approaches and moves away from a marking line during autonomous driving. Abnormal behavior includes, in addition to meandering, driving while straddling a marking line. The indicator recognition unit 51 may use a frequency (the number of times abnormal behavior occurs in a certain period of time) instead of the number of times that the vehicle 1 exhibits abnormal behavior with respect to a marking line during autonomous driving. Note that abnormal behavior of the vehicle 1 exhibiting abnormal behavior with respect to a marking line during autonomous driving may be included in the abnormal behavior of the vehicle 1 exhibiting abnormal driving described above.

[0041] The number of abnormal behaviors of surrounding vehicles with respect to the host vehicle 1 during autonomous driving is the number of times that surrounding vehicles detected within a certain distance from the host vehicle 1 during autonomous driving exhibit abnormal behavior. Abnormal behaviors include at least one of sudden acceleration, sudden braking, sudden steering, unnecessary vehicle stops, violation of traffic rules, etc. The indicator recognition unit 51 may use the frequency (the number of times abnormal behaviors of surrounding vehicles occur in a certain period of time) instead of the number of abnormal behaviors of surrounding vehicles. Note that the number or frequency of abnormal behaviors of surrounding vehicles may be integrated into the reliability of external object recognition by the autonomous driving system 100.

[0042] Each of the multiple indices is associated with at least one other indices. This association is used to change the ease of abnormality determination, as described below. For example, the reliability of lane marking recognition is associated with the reliability of external object recognition. The reliability of the external camera 22 and the reliability of the radar sensor 23 may be associated with all other indices.

[0043] The number or frequency of abnormal behavior of the vehicle 1 with respect to the lane markings during autonomous driving may be associated with at least one of the reliability of the lane marking recognition and the reliability of the external camera 22. If the number of abnormal behaviors of the vehicle 1 with respect to the lane markings increases, there may be a problem with the lane marking recognition by the external camera 22 or the like.

[0044] The number or frequency of abnormal behavior of the vehicle 1 with respect to surrounding vehicles during autonomous driving may be associated with at least one of the reliability of the external camera 22 and the reliability of the radar sensor 23. If the number of abnormal behaviors of the vehicle 1 with respect to surrounding vehicles such as a preceding vehicle increases, there may be a problem with the detection of surrounding vehicles by the external camera 22, the radar sensor 23, etc.

[0045] The number or frequency of driver interventions in the vehicle 1 during autonomous driving may be associated with the reliability of lane marking recognition. For example, an increase in the driver's intervention in deceleration operations while traveling around a curve in autonomous driving may indicate a problem with lane marking recognition, such as a shortened recognition distance of the lane markings.

[0046] The association of indicators may be changed depending on the driving conditions of the vehicle 1. The indicator recognition unit 51 may be configured to associate the number or frequency of the driver's intervention operations with the reliability of the lane marking recognition when the vehicle 1 in autonomous driving is traveling around a curve, and not to associate the number or frequency of the driver's intervention operations with the reliability of the lane marking recognition when the vehicle 1 is not traveling around a curve. The intervention operation in this case may be limited to a deceleration operation of depressing the brake pedal.

[0047] Similarly, the indicator recognition unit 51 may be configured to associate the number or frequency of the driver's intervention with the reliability of the external object recognition for a certain time after the host vehicle 1 in automatic driving has changed lanes, and not associate the number or frequency of the driver's intervention with the reliability of the external object recognition outside the certain time. If the driver intervenes, such as accelerating the host vehicle 1, after the host vehicle 1 in automatic driving has changed lanes, there may be a problem in recognizing the relative distance and relative speed between the following vehicle in the lane to which the lane is changed and the host vehicle 1. The intervention at this time may be limited to an acceleration operation of depressing the accelerator pedal. Instead of the reliability of the external object recognition, the reliability of the radar sensor 23 that detects the rear of the host vehicle 1 may be associated. Note that the number or frequency of the driver's intervention may be associated with at least one of the reliability of the external object recognition and the reliability of the radar sensor 23 that detects the rear, regardless of the driving conditions of the host vehicle 1.

[0048] In addition, the reliability of all the external cameras 22 may be associated with each other in the external cameras 22 consisting of a plurality of cameras installed according to the orientation of the vehicle 1. Since the external cameras 22 often perform feature extraction processing using a common DNN [Deep Neural Network], if the reliability of any camera decreases, problems may occur in the other cameras.

[0049] During autonomous driving of the vehicle 1, the performance degradation determination unit 52 determines whether or not a degradation in the surrounding recognition performance of the autonomous driving system 100 has occurred for each of multiple indicators obtained from the detection results of the vehicle 1's sensors (such as the external camera 22, radar sensor 23, and driving operation detection unit 26).

[0050] The performance degradation determination unit 52 determines degradation of the periphery recognition performance using a threshold value. Various threshold values ​​described later are preset threshold values. For example, when the reliability of the external camera 22 is less than the external camera threshold value, the performance degradation determination unit 52 determines that degradation of the periphery recognition performance related to the external camera 22 has occurred. Similarly, when the reliability of the radar sensor 23 is less than the radar sensor threshold value, the performance degradation determination unit 52 may determine that degradation of the periphery recognition performance related to the radar sensor 23 has occurred.

[0051] The performance degradation determination unit 52 may determine that a degradation in periphery recognition performance regarding external object recognition has occurred when the reliability of the external object recognition of the autonomous driving system 100 is less than an external object threshold. The performance degradation determination unit 52 may determine that a degradation in periphery recognition performance regarding lane line recognition has occurred when the reliability of the lane line recognition of the autonomous driving system 100 is less than an external object threshold.

[0052] The performance degradation determination unit 52 may determine that a degradation in periphery recognition performance has occurred when the continuity of the periphery recognition results of the autonomous driving system 100 is less than a continuity threshold. The performance degradation determination unit 52 may determine that a degradation in periphery recognition performance has occurred regarding the external camera 22 or the radar sensor 23 when the consistency of the recognition results of the external camera 22 and the radar sensor 23 is less than a first consistency threshold.

[0053] The performance degradation determination unit 52 may determine that a degradation in periphery recognition performance has occurred when the consistency between the periphery recognition result acquired from the surrounding vehicles through vehicle-to-vehicle communication and the periphery recognition result of the autonomous driving system 100 of the host vehicle 1 is less than a second consistency threshold. The performance degradation determination unit 52 may determine that a degradation in periphery recognition performance has occurred when the number or frequency of abnormal behaviors of the host vehicle 1 during autonomous driving by the autonomous driving system 100 is equal to or greater than a first behavior threshold.

[0054] The performance degradation determination unit 52 may determine that a degradation in periphery recognition performance has occurred when the number or frequency of the driver's intervention operations for the vehicle 1 during autonomous driving is equal to or greater than an intervention operation threshold. The performance degradation determination unit 52 may determine that a degradation in periphery recognition performance has occurred when the number or frequency of the abnormal behavior of the vehicle 1 during autonomous driving with respect to surrounding vehicles is equal to or greater than a second behavior threshold.

[0055] The performance degradation determination unit 52 may determine that a degradation in periphery recognition performance has occurred when the number or frequency of abnormal behavior of the host vehicle 1 with respect to a lane marking during autonomous driving is equal to or greater than a third behavior threshold. The performance degradation determination unit 52 may determine that a degradation in periphery recognition performance has occurred when the number or frequency of abnormal behavior of surrounding vehicles with respect to the host vehicle 1 during autonomous driving is equal to or greater than a fourth behavior threshold. Note that the performance degradation determination unit 52 does not necessarily need to make a determination for all indicators recognized by the indicator recognition unit 51.

[0056] The abnormality determination unit 53 determines whether or not there is an abnormality in the surrounding recognition performance of the autonomous driving system 100 based on the detection results of the sensors (such as the external camera 22, the radar sensor 23, and the driving operation detection unit 26) of the host vehicle 1. The abnormality determination unit 53 determines whether or not there is an abnormality in the surrounding recognition performance of the autonomous driving system 100 for each index based on, for example, the multiple indexes described above.

[0057] For example, when the reliability of the external camera 22 is less than the external camera abnormality threshold, the abnormality determination unit 53 determines that a deterioration in the surrounding recognition performance regarding the external camera 22 has occurred. Similarly, when the reliability of the radar sensor 23 is less than the radar sensor abnormality threshold, the abnormality determination unit 53 may determine that a deterioration in the surrounding recognition performance regarding the radar sensor 23 has occurred.

[0058] The abnormality determination unit 53 may determine that a deterioration in peripheral recognition performance regarding external object recognition has occurred when the reliability of the external object recognition of the autonomous driving system 100 is less than the external object abnormality threshold. The abnormality determination unit 53 may determine that an abnormality in peripheral recognition performance regarding lane line recognition has occurred when the reliability of the lane line recognition of the autonomous driving system 100 is less than the lane line abnormality threshold.

[0059] The abnormality determination unit 53 may determine that an abnormality has occurred in the surrounding recognition performance when the continuity of the surrounding recognition results of the autonomous driving system 100 is less than a continuity abnormality threshold. The abnormality determination unit 53 may determine that an abnormality has occurred in the surrounding recognition performance regarding the external camera 22 or the radar sensor 23 when the consistency of the recognition results of the external camera 22 and the radar sensor 23 is less than a first consistency abnormality threshold.

[0060] The abnormality determination unit 53 may determine that an abnormality in the surrounding recognition performance has occurred when the consistency between the surrounding recognition result acquired from the surrounding vehicles through vehicle-to-vehicle communication and the surrounding recognition result of the autonomous driving system 100 of the host vehicle 1 is less than a second consistency abnormality threshold. The abnormality determination unit 53 may determine that an abnormality in the surrounding recognition performance has occurred when the number or frequency of abnormal behavior of the host vehicle 1 during autonomous driving by the autonomous driving system 100 is equal to or greater than a first behavior abnormality threshold.

[0061] The abnormality determination unit 53 may determine that an abnormality in the surrounding recognition performance has occurred when the number or frequency of the driver's intervention operation for the host vehicle 1 during autonomous driving is equal to or greater than an intervention operation abnormality threshold. The abnormality determination unit 53 may determine that an abnormality in the surrounding recognition performance has occurred when the number or frequency of the abnormal behavior of the host vehicle 1 during autonomous driving with respect to surrounding vehicles is equal to or greater than a second behavior abnormality threshold.

[0062] The abnormality determination unit 53 may determine that an abnormality in the surrounding recognition performance has occurred when the number or frequency of abnormal behavior of the host vehicle 1 with respect to a marking line during autonomous driving is equal to or greater than a third behavior abnormality threshold. The abnormality determination unit 53 may determine that an abnormality in the surrounding recognition performance has occurred when the number or frequency of abnormal behavior of a surrounding vehicle with respect to the host vehicle 1 during autonomous driving is equal to or greater than a fourth behavior abnormality threshold.

[0063] The abnormality determination unit 53 does not necessarily need to perform a determination for all the indices recognized by the index recognition unit 51. The abnormality determination unit 53 may perform a determination for some of the indices. The indices used by the abnormality determination unit 53 for the determination may overlap with or be different from the indices used by the performance degradation determination unit 52 for the determination.

[0064] When the performance degradation determination unit 52 determines that a degradation in the surrounding recognition performance has occurred based on any of the indicators, the abnormality determination unit 53 makes it easier to determine that there is an abnormality in the surrounding recognition performance of the autonomous driving system 100 compared to when it is not determined that a degradation in the surrounding recognition performance has occurred.

[0065] The abnormality determination unit 53 changes the values ​​of the above-mentioned thresholds to make it easier to determine that there is an abnormality in the periphery recognition performance. For example, the abnormality determination unit 53 can make it easier to determine that there is an abnormality in the periphery recognition performance by increasing the threshold values ​​for the external camera abnormality threshold, the external target abnormality threshold, the lane marking abnormality threshold, the continuity abnormality threshold, the first consistency abnormality threshold, and the second consistency abnormality threshold. The abnormality determination unit 53 can make it easier to determine that there is an abnormality in the periphery recognition performance by decreasing the threshold values ​​for the first behavior abnormality threshold, the intervention operation abnormality threshold, the second behavior abnormality threshold, the third behavior abnormality threshold, and the fourth behavior abnormality threshold.

[0066] The abnormality judgment unit 53 may be configured, when the performance degradation judgment unit 52 judges that a degradation in peripheral recognition performance has occurred, to make it easier to judge that there is an abnormality in peripheral recognition performance based on other indices associated with the index used in the judgment, but not to change the ease of judging an abnormality in peripheral recognition performance based on other indices not associated with the index used in the judgment.

[0067] Specifically, when the performance degradation determination unit 52 determines that a degradation in periphery recognition performance has occurred based on the reliability of lane line recognition, the abnormality determination unit 53 makes it easier to determine that there is an abnormality in the periphery recognition performance based on the reliability of external object recognition associated with the reliability of lane line recognition. The abnormality determination unit 53 makes it easier to determine that there is an abnormality in the periphery recognition performance based on the reliability of external object recognition by changing the external object abnormality threshold to a larger value compared to when it is not determined that a degradation in periphery recognition performance has occurred. The abnormality determination unit 53 does not change the ease of determining an abnormality in the periphery recognition performance based on other indices not associated with the reliability of lane line recognition.

[0068] Similarly, when the performance degradation judgment unit 52 judges that a degradation in the surrounding recognition performance has occurred based on the number or frequency of abnormal behavior of the vehicle 1 toward marking lines during autonomous driving, the abnormality judgment unit 53 may make it easier to judge that there is an abnormality in the surrounding recognition performance based on the reliability of the marking line recognition associated with the indicator or the reliability of the external camera 22.

[0069] When the performance degradation determination unit 52 determines that a degradation in periphery recognition performance has occurred based on the number or frequency of abnormal behavior of the vehicle 1 against surrounding vehicles during autonomous driving, the abnormality determination unit 53 may make it easier to determine that there is an abnormality in the periphery recognition performance based on the reliability of the external camera 22 or the reliability of the radar sensor 23 associated with the indicator. When the performance degradation determination unit 52 determines that a degradation in periphery recognition performance has occurred based on the number or frequency of intervention operations by the driver, the abnormality determination unit 53 may make it easier to determine that there is an abnormality in the periphery recognition performance based on the reliability of lane marking recognition associated with the indicator.

[0070] When the abnormality determination unit 53 determines that an abnormality has occurred in the surrounding recognition performance of the autonomous driving system 100, it transmits an abnormality occurrence signal to the vehicle control unit 32. The vehicle control unit 32, for example, issues a warning to the driver of the occurrence of the abnormality and terminates the autonomous driving. The vehicle control unit 32 gives the driver a preparation time to switch to manual driving. Note that the vehicle control unit 32 does not necessarily need to terminate the autonomous driving depending on the type of abnormality determination, and may simply issue a warning to the driver.

[0071] Next, a control method of the abnormality determination device 50 according to this embodiment will be described with reference to the drawings. Fig. 2 is a flowchart showing an example of a performance degradation determination process. The performance degradation determination process is executed when the host vehicle 1 is in an autonomous driving state.

[0072] 2, the abnormality determination device 50, as S1, recognizes a plurality of indexes by an index recognition unit 51. The plurality of indexes are used to determine a deterioration in the surrounding recognition performance of the autonomous driving system 100. The index recognition unit 51 recognizes indexes such as the reliability of the external camera 22 and the reliability of the radar sensor 23 by a well-known method.

[0073] In S2, the abnormality determination device 50 determines whether or not a degradation in the surrounding recognition performance of the autonomous driving system 100 has occurred for each index using the performance degradation determination unit 52. If it is determined that a degradation in the surrounding recognition performance of the autonomous driving system 100 has occurred (S2: YES), the abnormality determination device 50 proceeds to S3. If it is not determined that a degradation in the surrounding recognition performance of the autonomous driving system 100 has occurred (S2: NO), the abnormality determination device 50 ends the current performance degradation determination process.

[0074] In S3, the abnormality determination device 50 makes it easier for the abnormality determination unit 53 to determine that there is an abnormality in the surrounding recognition performance of the autonomous driving system 100. The abnormality determination unit 53 makes it easier to determine that there is an abnormality in the surrounding recognition performance based on another index associated with the index used to determine the deterioration of the surrounding recognition performance by the performance degradation determination unit 52. Specifically, the abnormality determination unit 53 makes it easier to determine that there is an abnormality in the surrounding recognition performance by changing the value of the threshold used for the determination. The abnormality determination unit 53 does not change the ease of determining an abnormality in the surrounding recognition performance based on another index not associated with the index. Thereafter, the abnormality determination device 50 ends the current performance degradation determination process.

[0075] 3 is a flowchart showing an example of the abnormality determination process, which is executed when the host vehicle 1 is in an autonomous driving state.

[0076] 3, in step S10, the abnormality determination device 50 recognizes a plurality of indexes using the index recognition unit 51. The plurality of indexes are used to determine whether or not the autonomous driving system 100 has an abnormality in its surrounding recognition performance.

[0077] In S11, the abnormality determination device 50 causes the abnormality determination unit 53 to determine whether or not there is an abnormality in the surrounding recognition performance of the autonomous driving system 100. The abnormality determination unit 53 determines whether or not there is an abnormality in the surrounding recognition performance for each index based on, for example, a plurality of indexes. If it is determined that there is an abnormality in the surrounding recognition performance (S11: YES), the abnormality determination device 50 proceeds to S12. If it is not determined that there is an abnormality in the surrounding recognition performance (S11: NO), the abnormality determination device 50 ends the current abnormality determination process.

[0078] In S12, the abnormality determination device 50 transmits an abnormality occurrence signal to the vehicle control unit 32 via the abnormality determination unit 53. Thereafter, the abnormality determination device 50 ends the current abnormality determination process.

[0079] According to the abnormality determination device 50 (autonomous driving system 100) of this embodiment described above, when it is determined that a decline in the peripheral recognition performance of the autonomous driving system has occurred based on multiple indicators, it is easier to determine that there is an abnormality in the peripheral recognition performance of the autonomous driving system compared to when it is not determined that a decline in the peripheral recognition performance has occurred. Therefore, it is possible to determine the presence of an abnormality in the peripheral recognition performance early on based on the precursor of a decline in the peripheral recognition performance.

[0080] According to the abnormality determination device 50, by previously associating multiple indices that are linked when an abnormality occurs and making it easier to determine that there is an abnormality in the peripheral recognition performance based on other indices associated with the indices that were the determining factors for the deterioration of the peripheral recognition performance, it is possible to determine the presence of an abnormality in the peripheral recognition performance at an early stage. For example, when the abnormality determination device 50 determines that a deterioration in the peripheral recognition performance has occurred based on the reliability of the lane marking recognition, it is possible to determine the presence of an abnormality in the peripheral recognition performance at an early stage by making it easier to determine that there is an abnormality in the peripheral recognition performance based on the reliability of the external object recognition related to the reliability of the lane marking recognition.

[0081] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-mentioned embodiment. The present invention can be embodied in various forms including the above-mentioned embodiment and various modifications and improvements based on the knowledge of those skilled in the art.

[0082] The abnormality determination unit 53 may determine whether or not there is an abnormality in the periphery recognition performance of the autonomous driving system 100, for example, without using the above-mentioned multiple indexes. The abnormality determination unit 53 may determine that there is an abnormality in the periphery recognition performance of the autonomous driving system 100, when a state in which the captured image of the external camera 22 necessary for autonomous driving is blacked out continues for a certain period of time or more, without calculating the reliability of the external camera 22, for example. Similarly, the abnormality determination unit 53 may determine that there is an abnormality in the periphery recognition performance of the autonomous driving system 100, when a state in which there is no detection signal of the radar sensor 23 necessary for autonomous driving continues for a certain period of time or more, without calculating the reliability of the radar sensor 23. The abnormality determination unit 53 can make it easier to determine that there is an abnormality by shortening the value of the certain period of time. In addition, the abnormality determination unit 53 may determine the presence or absence of an abnormality in the periphery recognition performance of the autonomous driving system 100 by various well-known methods.

[0083] It is not necessary that all of the multiple indices described above are associated with any of the other indices. In addition, the indices may not be associated with each other at all. In this case, when the performance degradation determination unit 52 determines that the peripheral recognition performance has been degraded, the abnormality determination unit 53 makes it easier to determine an abnormality in the peripheral recognition performance based on all of the indices and other factors. [Explanation of symbols]

[0084] 1... host vehicle, 22... external camera, 23... radar sensor, 30... autonomous driving ECU, 50... abnormality determination device, 51... indicator recognition unit, 52... performance degradation determination unit, 53... abnormality determination unit, 100... autonomous driving system.

Claims

1. An abnormality determination device that determines whether or not there is an abnormality in an autonomous driving system of a vehicle based on a detection result of a sensor of the vehicle, a performance degradation determination unit that determines whether or not an abnormality has occurred in the surrounding recognition performance of the autonomous driving system for each of a plurality of indicators obtained from detection results of the sensor of the autonomous driving system during autonomous driving of the autonomous driving system; An abnormality determination device that makes it easier to determine that there is an abnormality in the surrounding recognition performance of the autonomous driving system when the performance degradation determination unit determines that a degradation in the surrounding recognition performance has occurred based on any of the indicators, compared to when it is not determined that a degradation in the surrounding recognition performance has occurred.

2. 2. The abnormality determination device according to claim 1, wherein the multiple indicators include at least two of: a reliability of the external camera of the host vehicle; a reliability of the radar sensor of the host vehicle; a reliability of external object recognition of the autonomous driving system; a reliability of lane marking recognition of the autonomous driving system; a continuity of the surrounding recognition results of the autonomous driving system; a consistency of the recognition results of the external camera and radar sensor of the host vehicle; a consistency between the surrounding recognition results obtained by vehicle-to-vehicle communication from surrounding vehicles located around the host vehicle and the surrounding recognition results of the autonomous driving system; a number or frequency of abnormal behavior of the host vehicle during autonomous driving by the autonomous driving system; a number or frequency of driver intervention operations with respect to the host vehicle during the autonomous driving; a number or frequency of abnormal behavior of the host vehicle with respect to the surrounding vehicles during the autonomous driving; and a number or frequency of abnormal behavior of the host vehicle with respect to lane marks during the autonomous driving.

3. The presence or absence of an abnormality in the peripheral recognition performance of the autonomous driving system is determined for each of the indexes based on the multiple indexes, Each of the plurality of indices is associated with at least one other of the indices; The abnormality determination device of claim 2, wherein when the performance degradation determination unit determines that a degradation in the peripheral recognition performance has occurred, the device makes it easier to determine that there is an abnormality in the peripheral recognition performance based on other indicators associated with the indicator used in the determination, but does not change the ease of determining that there is an abnormality in the peripheral recognition performance based on other indicators that are not associated with the indicator used in the determination.

4. Among the indicators, a reliability of the lane marking recognition is associated with a reliability of the external object recognition, The abnormality determination device according to claim 3, wherein when the performance degradation determination unit determines that a degradation in the surrounding recognition performance has occurred based on the reliability of the lane marking recognition, it is easier to determine that there is an abnormality in the surrounding recognition performance based on the reliability of the external object recognition.

Images