Abnormal detection device, abnormal detection method, and computer program for abnormal detection

Abstract

To provide an abnormality detection device capable of improving detection accuracy of an abnormality disturbing normal traveling of a vehicle.SOLUTION: An abnormality detection device comprises: an extraction section 31 that extracts feature amount by inputting an image showing the surroundings of a vehicle 10 to a feature extractor preliminarily trained so as to extract a feature amount showing a state of a road surface; a detection section 32 that detects an abnormal state in which the vehicle 10 cannot normally travel when the feature amount is not included in a normal range representing an allowable range in which the vehicle 10 can normally travel; and a change section 35 that changes the normal range on the basis of the distribution of the normal travel feature amount representing the state of the road surface, which is extracted from each of a plurality of images obtained in a period in which the vehicle 10 normally travels.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an abnormality detection device, an abnormality detection method, and an abnormality detection computer program that detects abnormalities based on an image showing the surroundings of a vehicle. [Background technology]

[0002] In order to safely control automatic driving of a vehicle, it is preferable to be able to determine the condition of the road surface on which the vehicle is traveling. To address this, a technology for detecting the condition of the road surface has been proposed (see Patent Document 1).

[0003] The road surface condition detection device described in Patent Document 1 detects the shape of a target road surface based on distance information indicating the distance between each measurement point on the target road surface and the measurement device, and if the target road surface includes a concave or convex area, determines whether that area is in an abnormal state. The road surface condition detection device determines whether the area is in an abnormal state based on the shape of the area in a normal state. The road surface condition detection device also determines whether the area is in an abnormal state by comparing an image obtained by capturing an image of the area with a reference image that indicates the normal state of the area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-82377 Summary of the Invention [Problem to be solved by the invention]

[0005] There may be obstacles such as falling objects on the path of a vehicle. Such obstacles may be objects whose color, shape, and size are not specified in advance. Therefore, it may not be possible to accurately detect such obstacles even using an identifier for detecting a predetermined object from an image. As a result, it may be difficult to accurately determine whether the vehicle can travel on the road surface on the path of the vehicle.

[0006] Therefore, an object of the present invention is to provide an abnormality detection device capable of improving the detection accuracy of an abnormality that hinders normal traveling of a vehicle.

Means for Solving the Problems

[0007] According to one embodiment, an abnormality detection device is provided. This abnormality detection device includes an extraction unit that extracts a feature amount by inputting an image representing the surroundings of the vehicle to a feature extractor that has been pre-learned to extract a feature amount representing the road surface condition, and a detection unit that detects an abnormal situation where the vehicle cannot normally travel when the feature amount is not included in a normal range representing an allowable range in which the vehicle can normally travel, and a change unit that changes the normal range based on the distribution of normal traveling feature amounts representing the road surface condition extracted from each of a plurality of images obtained during a period when the vehicle has traveled normally.

[0008] This abnormality detection device preferably further includes a storage unit that stores a plurality of reference feature amounts representing the road surface condition included in a preset normal range. And the change unit preferably changes the normal range so as to approximate the distribution of the plurality of reference feature amounts and the plurality of normal traveling feature amounts.

[0009] According to another embodiment, an abnormality detection method is provided. This abnormality detection method extracts a feature amount by inputting an image representing the surroundings of a vehicle into a feature extractor that has been pre-trained to extract a feature amount representing the road surface condition, and when the feature amount is not included in a normal range representing an allowable range in which the vehicle can normally travel, it detects an abnormal situation in which the vehicle cannot normally travel, and includes changing the normal range based on the distribution of normal travel feature amounts representing the road surface condition extracted from each of a plurality of images obtained during the period in which the vehicle has traveled normally.

[0010] According to still another embodiment, a computer program for abnormality detection is provided. This computer program for abnormality detection extracts a feature amount by inputting an image representing the surroundings of a vehicle into a feature extractor that has been pre-trained to extract a feature amount representing the road surface condition, and when the feature amount is not included in a normal range representing an allowable range in which the vehicle can normally travel, it detects an abnormal situation in which the vehicle cannot normally travel, and includes instructions for causing a processor mounted on the vehicle to change the normal range based on the distribution of normal travel feature amounts representing the road surface condition extracted from each of a plurality of images obtained during the period in which the vehicle has traveled normally.

Advantages of the Invention

[0011] The abnormality detection device according to the present disclosure has the effect of being able to improve the detection accuracy of an abnormality that hinders the normal travel of a vehicle.

Brief Description of the Drawings

[0012] [Figure 1] It is a schematic configuration diagram of a vehicle control system in which an abnormality detection device is implemented. [Figure 2] It is a hardware configuration diagram of an electronic control unit which is one embodiment of the abnormality detection device. [Diagram 3] It is a functional block diagram of a processor of an electronic control unit regarding vehicle control processing including abnormality detection processing. [Figure 4] It is a diagram for explaining an example of changing the normal range according to the present embodiment. [Figure 5]This is a diagram for explaining another example of the change in the normal range according to the present embodiment. [Figure 6] This is an operation flowchart of vehicle control processing including abnormality detection processing. [Figure 7] This is an operation flowchart of the processing related to the change in the normal range in the abnormality detection processing.

Mode for Carrying Out the Invention

[0013] Hereinafter, with reference to the drawings, an abnormality detection device, an abnormality detection method and an abnormality detection computer program implemented in the abnormality detection device will be described. By inputting an image representing the surroundings of the vehicle generated by an imaging unit provided in the vehicle into a feature extractor, this abnormality detection device extracts a feature amount representing the road surface condition. Further, this abnormality detection device determines whether the extracted feature amount is included in a normal range representing an allowable range in which the vehicle can normally travel, and when the feature amount deviates from the normal range, it detects an abnormal situation in which the vehicle cannot normally travel in its own lane. Furthermore, this abnormality detection device changes the normal range so as to approximate the distribution of the feature amounts extracted from each of a plurality of images obtained during a predetermined period in which the vehicle could normally travel.

[0014] Note that the vehicle being able to normally travel means that the vehicle can travel without decelerating at a predetermined deceleration or more or steering at a predetermined steering amount or more to avoid contact with any obstacles. Also, the obstacles are, for example, any three-dimensional structures that should not originally exist on the road surface, such as fallen objects on the road surface, or road surface defects with steps, such as potholes formed on the road surface.

[0015] Hereinafter, an example in which the abnormality detection device is applied to a vehicle control device will be described.

[0016] FIG. 1 is a schematic diagram of a vehicle control system in which an anomaly detection device is implemented. The vehicle control system 1 is mounted on a vehicle 10 and controls the vehicle 10. To this end, the vehicle control system 1 includes a camera 2 and an electronic control unit (ECU) 3, which is an example of an anomaly detection device. The camera 2 and the ECU 3 are communicatively connected to each other via an in-vehicle network conforming to a communication standard, such as a controller area network. The vehicle control system 1 may also include a ranging sensor (not shown), such as a LiDAR or radar, that measures the distance from the vehicle 10 to objects around the vehicle 10. The vehicle control system 1 may also include a positioning device (not shown), such as a GPS receiver, that determines the position of the vehicle 10 based on signals from a satellite. The vehicle control system 1 may also include a navigation device (not shown) that searches for a planned driving route to a destination. The vehicle control system 1 may also include a storage device (not shown) that stores map information referenced in the autonomous driving control of the vehicle 10.

[0017] Camera 2 is an example of an imaging unit that generates an image showing the surroundings of vehicle 10. Camera 2 has a two-dimensional detector configured with an array of photoelectric conversion elements, such as a CCD or C-MOS, that are sensitive to visible light, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. Camera 2 is attached, for example, inside the passenger compartment of vehicle 10 so as to face forward of vehicle 10. Camera 2 photographs the area ahead of vehicle 10 at predetermined photographing intervals (for example, 1 / 30 to 1 / 10 seconds) and generates an image showing the area ahead. The image obtained by camera 2 may be a color image or a grayscale image. Note that vehicle 10 may be provided with two or more cameras with different photographing directions or focal lengths.

[0018] Every time the camera 2 generates an image, it outputs the generated image to the ECU 3 via the in-vehicle network.

[0019] The ECU 3 is configured to perform automatic driving control of the vehicle 10 under predetermined circumstances.

[0020] Fig. 2 is a hardware configuration diagram of an ECU 3, which is an example of an anomaly detection device. As shown in Fig. 2, the ECU 3 has a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 may each be configured as separate circuits, or may be configured integrally as a single integrated circuit.

[0021] The communication interface 21 has an interface circuit for connecting the ECU 3 and the camera 2. Every time the communication interface 21 receives an image from the camera 2, it passes the received image to the processor 23.

[0022] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a nonvolatile semiconductor memory. The memory 22 stores various data used in the vehicle control process, including the anomaly detection process, executed by the processor 23 of the ECU 3. For example, the memory 22 stores parameters for identifying a classifier used to detect the road area and parameters for identifying a feature extractor used to extract features representing the road surface condition. The memory 22 also stores a preset normal range (hereinafter referred to as the pre-normal range), multiple reference features included in the pre-normal range, and a changed normal range. The memory 22 also temporarily stores images received from the camera 2. The memory 22 also temporarily stores various data generated during the vehicle control process, such as multiple features obtained while the vehicle 10 is traveling and used to change the normal range.

[0023] The processor 23 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logic unit, a numerical calculation unit, or a graphics processing unit. The processor 23 executes vehicle control processing for the vehicle 10.

[0024] 3 is a functional block diagram of processor 23 related to vehicle control processing including anomaly detection processing. Processor 23 has an extraction unit 31, a detection unit 32, a vehicle control unit 33, a collection unit 34, and a change unit 35. Each of these units in processor 23 is a functional module realized by, for example, a computer program running on processor 23. Alternatively, each of these units in processor 23 may be a dedicated arithmetic circuit provided in processor 23. Furthermore, among these units, the units other than vehicle control unit 33 are related to anomaly detection processing.

[0025] The extraction unit 31 extracts features representing road surface conditions (hereinafter, these features may be referred to as road surface condition features) from the latest images received by the ECU 3 from the camera 2 at predetermined intervals. To do this, the extraction unit 31 inputs the images to a feature extractor that has been trained in advance to extract road surface condition features. The feature extractor is configured with a deep neural network (DNN) having a convolutional neural network (CNN) architecture, such as VGG16, VGG19, or ResNet, with multiple convolutional layers and one or more fully connected layers on the output side of the convolutional layers and an output layer that performs softmax operations, etc. However, the feature map output from the most output convolutional layer (or from the most output convolutional layer and any convolutional layer on the input side of that layer) is used as the road surface condition feature output by the feature extractor. The road surface condition feature output by the feature extractor is represented, for example, as a feature vector having one or more element values.

[0026] The feature extractor is pre-trained according to a predetermined learning method such as the error backpropagation method using a large number of images representing any of various types of objects, not limited to road surfaces, so that these objects can be classified. As a result, the feature map used as the road surface condition feature amount condenses various information represented in the image. Therefore, the feature extractor can output a road surface condition feature amount representing the condition of the road surface of the road on which the vehicle 10 is traveling. That is, the feature extractor can output a road surface condition feature amount having different values when the condition of the road surface of the road on which the vehicle 10 is traveling is abnormal and when it is not.

[0027] Alternatively, as the feature extractor, a DNN pre-trained by so-called unsupervised learning such as an Auto-Encoder or Stacked What-Where Auto-Encoders may be used. In this case, the feature extractor has, in order from the input side, an encoder that outputs a feature amount with a reduced dimension compared to the input data (in this embodiment, an image), and a decoder to which the feature amount output from the encoder is input. Then, the feature extractor is pre-trained using a large number of images as described above so that the data input to the encoder and the data output from the decoder are the same. Then, by inputting an image to the pre-trained feature extractor, the feature amount output by the encoder can be obtained as the road surface condition feature amount. By using such a feature extractor, the extraction unit 31 can obtain a road surface condition feature amount that appropriately represents the condition of the road surface of the road on which the vehicle 10 is traveling even when there are obstacles with indefinite colors, shapes, and sizes within the shooting range of the camera 2.

[0028] According to the modification example, the extraction unit 31 may identify a road region in which the lane on which the vehicle 10 is traveling on the image (hereinafter sometimes referred to as the host vehicle lane) is represented, and input the identified road region into the feature extractor to extract the road surface condition feature amount. Thereby, since the influence of the situation around the host vehicle lane on the extraction of the road surface condition feature amount is suppressed, the extraction unit 31 can extract the road surface condition feature amount more appropriately. In this case, the extraction unit 31 identifies the road region by inputting the image acquired from the camera 2 into an identifier that has been pre-trained to identify the road region. As such an identifier, the extraction unit 31 can use a DNN having a CNN-type architecture. More specifically, as the identifier, a DNN for semantic segmentation that identifies the object represented by each pixel, for example, a Fully Convolutional Network (FCN) or a U-net, is used for each pixel. Alternatively, the extraction unit 31 may use an identifier for semantic segmentation based on a machine learning method other than a neural network, such as a random forest, as the identifier. The identifier is pre-trained according to a predetermined learning method such as the error backpropagation method using a large number of teacher images in which the road region is represented. The extraction unit 31 sets the set of pixels output by the identifier and representing the host vehicle lane as the road region.

[0029] Alternatively, instead of using the above-described identifier, the extraction unit 31 may detect two lane dividing lines that demarcate the host vehicle lane from the image by image analysis, and use the area sandwiched between the two detected lane dividing lines as the driving lane area. Generally, lane dividing lines have a brighter color (white or yellow) than the surrounding road surface. Therefore, on the image, the luminance value of the pixels representing the lane dividing lines is higher than the luminance value of the pixels representing the surrounding road surface. Thus, the extraction unit 31 extracts pixels whose luminance value is equal to or higher than a predetermined value within a range where the road surface is assumed to be represented on the image. Alternatively, the extraction unit 31 may extract the pixel with the higher luminance value when the luminance difference between two adjacent pixels in the horizontal direction is equal to or greater than a predetermined threshold value. Then, the extraction unit 31 detects each lane dividing line by approximating the set of the extracted pixels with a straight line, and sets the lane dividing line closest to the center of the image on the left and right sides of the image as the lane dividing line that demarcates the host vehicle lane.

[0030] When the feature extractor extracts the road surface condition feature amount based only on the driving lane area on the image, the extraction unit 31 masks the area outside the driving lane area by replacing the value of each pixel outside the driving lane area in the image with a predetermined value. Then, the extraction unit 31 may input the image with the area outside the driving lane area masked to the feature extractor. Alternatively, the extraction unit 31 may cut out the driving lane area from the image and input the cut-out driving lane area to the feature extractor. In this case, the extraction unit 31 may perform preprocessing such as upsampling, downsampling, or padding on the cut-out driving lane area so that the cut-out driving lane area becomes an area having a predetermined shape and size. Then, the extraction unit 31 may input the preprocessed driving lane area to the feature extractor. Thereby, since the shape and size of the driving lane area input to the feature extractor are constant, it becomes possible to simplify the feature extractor.

[0031] The extraction unit 31 passes the extracted road surface condition feature amount to the detection unit 32.

[0032] The detection unit 32 determines whether the road surface condition feature amount received from the extraction unit 31 is included in the normal range read from the memory 22. As described above, the normal range represents the allowable range within which the vehicle 10 can normally travel with respect to the road surface condition feature amount. The detection unit 32 uses the pre-stored pre-normal range in the memory 22 until the normal range is changed by the change unit 35, and uses the changed normal range after the normal range is changed by the change unit 35. Then, when the road surface condition feature amount extracted by the extraction unit 31 is not included in the normal range, the detection unit 32 detects an abnormal situation in which the vehicle 10 cannot normally travel in its own lane. On the other hand, when the road surface condition feature amount is included in the normal range, the detection unit 32 does not detect such an abnormal situation.

[0033] The detection unit 32 notifies the vehicle control unit 33 of the determination result as to whether an abnormal situation has been detected.

[0034] When notified by the detection unit 32 that an abnormal situation has been detected, the vehicle control unit 33 controls each part of the vehicle 10 so that no danger occurs to the vehicle 10 due to the abnormal situation. For example, when notified that an abnormal situation has been detected, the vehicle control unit 33 decelerates the vehicle 10 at a predetermined deceleration.

[0035] The vehicle control unit 33 sets the accelerator opening or the brake amount so as to achieve the set deceleration. Then, the vehicle control unit 33 obtains the fuel injection amount according to the set accelerator opening, and outputs a control signal corresponding to the fuel injection amount to the fuel injection device of the engine of the vehicle 10. Alternatively, the vehicle control unit 33 controls the power supply device to the motor for driving the vehicle 10 so as to supply power corresponding to the set accelerator opening to the motor. Alternatively, the vehicle control unit 33 outputs a control signal corresponding to the set brake amount to the brake of the vehicle 10.

[0036] Furthermore, the vehicle control unit 33 may notify the driver of the detection of an abnormal situation via a notification device provided in the vehicle cabin of the vehicle 10. For example, if a display device is provided as an example of the notification device, the vehicle control unit 33 may cause the display device to display a warning message or icon indicating that an abnormal situation has been detected. If a speaker is provided as another example of the notification device, the vehicle control unit 33 may cause the speaker to output an audio warning indicating that an abnormal situation has been detected. If a vibrator is provided on the driver's seat or steering wheel as another example of the notification device, the vehicle control unit 33 may vibrate the vibrator. If one or more light sources are provided as another example of the notification device, the vehicle control unit 33 may turn on or flash one of the light sources that indicates that an abnormal situation has been detected. If multiple notification devices are provided in the vehicle cabin, the vehicle control unit 33 may notify the driver of the detection of an abnormal situation via two or more of the multiple notification devices.

[0037] Alternatively, the vehicle control unit 33 may reduce the level of the automatic driving control applied to the vehicle 10. For example, when the so-called level 3 automatic driving control defined by the Society of Automotive Engineers (SAE) is applied to the vehicle 10, the vehicle control unit 33 reduces the level of the automatic driving control applied to the vehicle 10 to any level between level 0 and level 2. Also, when the level of the automatic driving control applied to the vehicle 10 is the level 2 driving control, the vehicle control unit 33 reduces the level of the automatic driving control applied to the vehicle 10 to level 0 or level 1. Alternatively, the vehicle control unit 33 may request the driver of the vehicle 10 to hold the steering. Also in this case, the vehicle control unit 33 notifies the driver of the change in the level of the automatic driving control via the notification device provided in the vehicle interior of the vehicle 10. That is, when a display device is provided as an example of the notification device, the vehicle control unit 33 causes the display device to display a notification message or icon indicating the change in the level of the applied automatic driving control and the level after the change. Also, when a speaker is provided as another example of the notification device, the vehicle control unit 33 causes the speaker to output a notification voice indicating the change in the level of the applied automatic driving control and the level after the change. Further, as another example of the notification device, when a vibrator is provided on the driver seat or the steering, the vehicle control unit 33 vibrates the vibrator in a manner corresponding to the change in the level of the applied automatic driving control. Furthermore, as another example of the notification device, when one or more light sources are provided, the vehicle control unit 33 turns on or blinks the light source corresponding to the level of the automatic driving control after the change among the respective light sources. Also, when a plurality of notification devices are provided in the vehicle interior, the vehicle control unit 33 may notify the change in the level of the applied automatic driving control via two or more of those plurality of notification devices.

[0038] Alternatively, the vehicle control unit 33 may neither decelerate the vehicle 10 nor change the level of the applied automatic driving control, and may only notify the driver of a warning indicating that an abnormal situation has been detected via the notification device provided in the vehicle interior.

[0039] The collection unit 34 collects, as normal driving feature amounts to be used for changing the normal range, each of the feature amounts extracted by the extraction unit 31 from each of the multiple images generated by the camera 2 during a period when the vehicle 10 was able to drive normally. Hereinafter, the period when the vehicle 10 was able to drive normally may be referred to as the normal driving period. The normal driving period is set to a period having a predetermined length of time, for example, from several seconds to 10 and a few seconds.

[0040] For example, the collection unit 34 identifies the normal driving period based on a sensor signal representing the behavior of the vehicle 10 during a period in which the driver is manually driving the vehicle 10 before automatic driving control is applied to the vehicle 10 (hereinafter referred to as the manual driving period). The sensor signal representing the behavior of the vehicle 10 may be, for example, a sensor signal representing the acceleration / deceleration of the vehicle 10 or a sensor signal representing the steering angle of the vehicle 10. To this end, the collection unit 34 acquires measured values of the acceleration / deceleration of the vehicle 10 from an acceleration sensor (not shown) mounted on the vehicle 10 during the manual driving period. Furthermore, the collection unit 34 acquires measured values of the steering angle of the vehicle 10 from a steering angle sensor (not shown) mounted on the vehicle 10 during the manual driving period. The collection unit 34 then determines that a period in which the absolute value of the maximum acceleration / deceleration of the vehicle 10 is less than a predetermined deceleration threshold and the maximum steering angle of the vehicle 10 is less than a predetermined steering angle threshold continues for a predetermined length of time or longer is the normal driving period.

[0041] The collection unit 34 regards each feature extracted by the extraction unit 31 from each of the multiple images generated by the camera 2 during the normal driving period as a normal driving feature. The collection unit 34 then stores the normal driving feature in the memory 22.

[0042] After the end of the normal driving period, the changing unit 35 determines the changed normal range by changing the pre-change normal range so as to approximate the distribution of the normal driving feature amounts. For example, the changing unit 35 reads each reference feature amount and each normal driving feature amount from the memory 22. Then, the changing unit 35 calculates the average value of the read feature amounts and the Euclidean distance from the average value to the feature amount that is the farthest from the average value among the feature amounts. The changing unit 35 sets the range centered on the calculated average value and having the calculated Euclidean distance as the radius as the normal range after the change. Alternatively, the changing unit 35 calculates the average value of each normal driving feature amount and the Euclidean distance from the average value of each normal driving feature amount to the normal driving feature amount that is the farthest from the average value of each normal driving feature amount. The changing unit 35 may obtain the normal range after the change by adding, as one of the normal ranges, the range centered on the calculated average value of each normal driving feature amount and having the calculated Euclidean distance as the radius.

[0043] Alternatively, the changing unit 35 may approximate the distribution of the feature amounts including each reference feature amount and each normal driving feature amount with a probability model such as a normal distribution or a mixture normal distribution. At this time, the changing unit 35 obtains the parameters of the probability model for approximating the distribution of the feature amounts according to a maximum likelihood estimation method such as the expectation maximization method. Then, the changing unit 35 sets the range within a predetermined Mahalanobis distance (for example, 2 to 3) or less from the position where the probability is maximized in the probability model (in the case where the probability model is a normal distribution, the average value of the normal distribution; in the case of a mixture normal distribution, the average value of each normal distribution included in the mixture normal distribution) as the normal range after the change.

[0044] By changing the normal range as described above, the changing unit 35 can change the normal range to a more appropriate range according to the road conditions during which the vehicle 10 is running.

[0045] FIG. 4 is a diagram for explaining an example of the change of the normal range according to the present embodiment. In FIG. 4, each round mark 401 represents a reference feature amount, and each star mark 402 represents a normal running feature amount. Also, in FIG. 4, for convenience, each feature amount is represented in three dimensions, but the dimension of each feature amount (that is, the number of elements each feature amount has) may be 2 or less, or may be 4 or more.

[0046] The pre-change normal range 410 is set to include the distribution of the reference feature amounts 401 and not to include feature amounts deviating from the distribution. However, the distribution of the normal running feature amounts 402 obtained during the normal running period in which the vehicle 10 could run normally does not coincide with the distribution of the reference feature amounts 401. Therefore, the normal range is changed so as to approximate the distributions of each reference feature amount 401 and each normal running feature amount 402. At that time, as shown as the changed normal range 420, the changed normal range 420 is obtained so as to include the distribution of the reference feature amounts 401 and the distribution of the normal running feature amounts 402 as one range. Alternatively, as shown as the changed normal range 430, a range including the distribution of the reference feature amounts 401 and a range including the distribution of the normal running feature amounts 402 may be provided separately. Thus, in the example shown in FIG. 4, by changing the normal range so as to include the distribution of the normal running feature amounts 402, the changed normal range is expanded compared to the pre-change normal range.

[0047] FIG. 5 is a diagram for explaining another example of the change of the normal range according to the present embodiment. In FIG. 5, each round mark 501 represents a reference feature amount, and each star mark 502 represents a normal running feature amount. Similar to FIG. 4, in FIG. 5, for convenience, each feature amount is represented in three dimensions, but the dimension of each feature amount may be 2 or less, or may be 4 or more.

[0048] Even in the example shown in FIG. 5, the pre-normal range 510 is set to include the distribution of the reference feature amount 501 and not to include feature amounts deviating from that distribution. Also, in this example, the range in which the normal driving feature amounts 502 obtained during the normal driving period when the vehicle 10 could drive normally are distributed is limited to a part of the pre-normal range 510. Therefore, in the changed normal range 520, a plurality of ranges are set so that the distributions of the individual reference feature amounts 501 and the individual normal driving feature amounts 502 are more appropriately approximated. Alternatively, the normal range may be changed so that only the range in which the reference feature amount 501 and the normal driving feature amount 502 are concentrated, such as the changed normal range 530, is included in the normal range. Thus, in the example shown in FIG. 5, by changing the normal range to include the distribution of the normal driving feature amounts 502, the changed normal range is more limited than the pre-normal range.

[0049] FIG. 6 is an operation flowchart of vehicle control processing including abnormality detection processing executed by the processor 23. The processor 23 executes vehicle control processing according to the following operation flowchart at a predetermined cycle.

[0050] The extraction unit 31 of the processor 23 extracts road surface condition feature amounts from the latest image obtained by the camera 2 (step S101). Then, the detection unit 32 of the processor 23 determines whether the road surface condition feature amounts are included in the normal range (step S102). When the road surface condition feature amounts are not included in the normal range (step S102 - No), the detection unit 32 detects an abnormal situation in which the vehicle 10 cannot drive normally (step S103). Then, the vehicle control unit 33 of the processor 23 decelerates the vehicle 10 or reduces the level of the automatic driving control applied to the vehicle 10 so that no danger occurs to the vehicle 10 due to the detected abnormal situation (step S104).

[0051] On the one hand, in step S102, when the road surface condition feature amount is included in the normal range (step S102 - Yes), the detection unit 32 does not detect an abnormal situation where the vehicle 10 cannot travel normally. Then, the vehicle control unit 33 continues the control of the vehicle 10 that is currently applied to the vehicle 10 (step S105).

[0052] After step S104 or S105, the processor 23 ends the vehicle control process.

[0053] FIG. 7 is an operation flowchart of a process related to the change of the normal range in the abnormality detection process executed by the processor 23. Before the automatic driving control is applied to the vehicle 10, the processor 23 executes the process related to the change of the normal range according to the following operation flowchart.

[0054] The collection unit 34 of the processor 23 specifies the normal driving period during which the vehicle 10 can travel normally based on the sensor signal representing the behavior of the vehicle 10 (step S201). Then, the collection unit 34 stores, in the memory 22 as the normal driving feature amount, each of the individual feature amounts extracted by the extraction unit 31 from each of the plurality of images generated by the camera 2 during the normal driving period (step S202).

[0055] The change unit 35 of the processor 23 reads out the plurality of reference feature amounts and the plurality of normal driving feature amounts from the memory 22. Then, the change unit 35 changes the normal range so as to approximate the distribution of these feature amounts (step S203). After step S203, the processor 23 ends the process related to the change of the normal range.

[0056] As described above, this abnormality detection device extracts road surface condition feature amounts by inputting an image representing the surroundings of the vehicle, which is generated by an imaging unit provided in the vehicle, into a feature extractor. Then, this abnormality detection device determines whether or not the extracted road surface condition feature amounts are included in a normal range indicating that the vehicle can travel normally, and when the feature amounts deviate from the normal range, it detects an abnormal situation in which the vehicle cannot travel normally in its own lane. Further, this abnormality detection device changes the normal range so as to approximate the distribution of the feature amounts extracted from each of a plurality of images obtained during a normal traveling period. As a result, since the normal range is adjusted according to the condition of the road on which the vehicle has actually traveled, this abnormality detection device can improve the detection accuracy of an abnormality that hinders the normal traveling of the vehicle.

[0057] A computer program for realizing the functions of the processor 23 of the ECU 3 according to the above-described embodiment or modification example may be provided in a form recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

[0058] As described above, those skilled in the art can make various changes according to the implemented forms within the scope of the present invention.

Explanation of Signs

[0059] 1 Vehicle control system 10 Vehicle 2 Camera 3 Electronic control unit (ECU, abnormality detection device) 21 Communication interface 22 Memory 23 Processor 31 Extraction unit 32 Detection unit 33 Vehicle control unit 34 Collection unit 35 Change unit

Claims

1. An extraction unit that extracts the feature amount by inputting an image representing the surroundings of the vehicle into a feature extractor that has been pre-trained to extract feature amounts representing the road surface conditions; A detection unit that detects an abnormal situation in which the vehicle cannot normally travel when the feature amount is not included in a normal range representing an allowable range in which the vehicle can normally travel; A change unit that changes the normal range based on the distribution of normal travel feature amounts representing the road surface conditions, extracted from each of a plurality of the images obtained during a period in which the vehicle has traveled normally; An abnormality detection device having the above.

2. Further comprising a storage unit that stores a plurality of reference feature amounts representing the road surface conditions, which are included in the preset normal range; The abnormality detection device according to claim 1, wherein the change unit changes the normal range so as to approximate the distribution of the plurality of reference feature amounts and the plurality of normal travel feature amounts.

3. The normal travel feature amounts extracted from each of the plurality of the images obtained during the period are those extracted by the feature extractor, The abnormality detection device according to claim 2, wherein the change unit approximates the distribution of the plurality of reference feature amounts and the plurality of normal travel feature amounts with a predetermined probability model, and changes the normal range so that a predetermined range centered on a point where the probability is maximized in the probability model is the normal range.

4. Extracting the feature amount by inputting an image representing the surroundings of the vehicle into a feature extractor that has been pre-trained to extract feature amounts representing the road surface conditions; Detecting an abnormal situation in which the vehicle cannot normally travel when the feature amount is not included in a normal range representing an allowable range in which the vehicle can normally travel; Changing the normal range based on the distribution of normal travel feature amounts representing the road surface conditions, extracted from each of a plurality of the images obtained during a period in which the vehicle has traveled normally; An abnormality detection method including the above.

5. Extracting the feature amount by inputting an image representing the surroundings of the vehicle into a feature extractor that has been pre-trained to extract feature amounts representing the road surface conditions; Detecting an abnormal situation in which the vehicle cannot normally travel when the feature amount is not included in a normal range representing an allowable range in which the vehicle can normally travel; Changing the normal range based on the distribution of normal travel feature amounts representing the road surface conditions, extracted from each of a plurality of the images obtained during a period in which the vehicle has traveled normally; An anomaly detection computer program for causing a processor mounted on the vehicle to execute the above.

Images