Abnormality diagnosis device

Abstract

To provide an abnormality diagnostic device that can set an abnormal area generated in a sensor in a short time.SOLUTION: An abnormality diagnostic device comprises: a target reliability determination unit 400 which determines reliability of a target based on a detection result in a common imaging area; and a single abnormal image area determination unit 500 which determines an abnormal image area in a monocular area using a tracking result of the target in the monocular area. The single abnormal image area determination unit 500 changes an anomaly score of an anomaly score map given to the abnormal image area depending on reliability of the target.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an abnormality diagnosis device that appropriately diagnoses an abnormal area that occurs in a sensor installed in a mobile body. [Background technology]

[0002] In recent years, autonomous driving and driver assistance technologies have required the proper recognition of the surrounding environment from sensor information installed on moving vehicles. If an abnormal area appears on the sensor mounted on a moving vehicle due to dirt or scratches on the lens, the surrounding environment may be incorrectly recognized, so it is necessary to detect the abnormal area early.

[0003] Conventionally, there is a method of determining an area where it is no longer possible to track a target object as an abnormal area, and Patent Document 1 states that "it is detected whether or not the object tracking process is in an abnormal state." For example, when a pedestrian is being tracked in Fig. 13(a), the area where it was possible to track is determined as normal, and the area where it was not possible to track is determined as abnormal, and the abnormality score map shown in Fig. 13(b) is updated. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-272436 Summary of the Invention [Problem to be solved by the invention]

[0005] However, conventional technology such as that disclosed in Patent Document 1 cannot properly estimate an abnormal area when it erroneously detects dirt or scratches on the lens as a target object. Figures 13(c) and (d) show problematic scenes. When dirt is erroneously detected as a target as in Figure 13(c), the value of the abnormality score map for the detected area is updated to normal as in Figure 13(d). One way to deal with such false detections is to determine the abnormal area from detection results at multiple times, but this takes time to confirm the abnormal area.

[0006] The present invention has been made in view of the above circumstances, and has as its object to provide an abnormality diagnosis device that can quickly identify an abnormality area that has occurred in a sensor. [Means for solving the problem]

[0007] In order to solve the above problems, one aspect of the abnormality diagnosis device of the present invention comprises a sensor unit in which multiple cameras are arranged so that their fields of view overlap, an object detection unit that detects objects from sensor information from the sensor unit, an object reliability determination unit that determines the reliability of the object based on the detection results in a common imaging area where the multiple cameras share a field of view, and a single abnormal image area determination unit that determines an abnormal image area in a monocular area consisting of only the field of view of a single camera among the multiple cameras using the tracking results of the object within the monocular area, and the single abnormal image area determination unit changes the abnormal score in an abnormality score map that is assigned to the abnormal image area according to the reliability of the object.

[0008] Another aspect of the abnormality diagnosis device according to the present invention comprises a sensor unit in which multiple sensors are arranged so that their observation areas overlap; an object detection unit that detects an object from sensor information from the sensor unit; an object reliability determination unit that determines the reliability of the object based on the detection results in a common sensing area in which the multiple sensors share an observation area; and an individual abnormality area determination unit that determines an abnormal area in the individual sensing area by using the tracking results of the object within an individual sensing area that is composed only of the observation area of a single sensor among the multiple sensors, and the individual abnormality area determination unit changes the abnormality score in an abnormality score map that is assigned to the abnormal area in accordance with the reliability of the object. [Effects of the Invention]

[0009] According to the present invention, an abnormal area occurring in a sensor can be identified in a short period of time.

[0010] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the overall configuration of an abnormality diagnosis device according to a first embodiment of the present invention. [Figure 2] 1A and 1B are diagrams showing an example of the configuration of a sensor unit 100 according to a first embodiment of the present invention, where FIG. 1A is a perspective view of one example and FIG. 1B is a perspective view of another example. [Figure 3] 1A and 1B are diagrams illustrating an example of operation of the object detection unit 200 according to the first embodiment of the present invention, in which (a) is a template (another vehicle), (b) is a template (pedestrian), and (c) is an explanatory diagram of a driving scene. [Figure 4] 1A and 1B show examples of common normal areas of the common normal area calculation unit 300 according to the first embodiment of the present invention, where (a) is an explanatory diagram of an image acquired by the left front camera 110 and (b) is an explanatory diagram of an image acquired by the right front camera 120 (both images are normal). [Figure 5] 1A and 1B show examples of common normal areas of the common normal area calculation unit 300 according to the first embodiment of the present invention, where (a) is an image acquired by the left front camera 110 and (b) is an explanatory diagram of an image acquired by the right front camera 120 (the right image is abnormal). [Figure 6] FIG. 3 is a process flow diagram of the common normal region calculation unit 300 according to the first embodiment of the present invention. [Figure 7] FIG. 4 is a process flow diagram of an object reliability determination unit 400 according to the first embodiment of the present invention. [Figure 8] 5A and 5B are diagrams illustrating an example of the operation of the single abnormal image region determining unit 500 according to the first embodiment of the present invention, in which FIG. 5A is a diagram illustrating object detection information, and FIG. 5B is a diagram illustrating an abnormality score map for FIG. [Figure 9] FIG. 3 is a process flow diagram of the single abnormal image region determining unit 500 according to the first embodiment of the present invention. [Figure 10] FIG. 10 is a block diagram showing an example of the overall configuration of an abnormality diagnosis device according to a second embodiment of the present invention. [Figure 11] FIG. 11 is a bird's-eye view showing an example of the configuration of a sensor unit 1100 according to a second embodiment of the present invention. [Figure 12] FIG. 10 is a process flow diagram of an object reliability determination unit 1400 according to the second embodiment of the present invention. [Figure 13]This is an explanatory diagram of the problem, where (a) is the object detection information (pedestrian, dirt present), (b) is the anomaly score map for (a) (pedestrian, dirt present), (c) is the object detection information (dirt present), and (d) is the anomaly score map for (c) (dirt present). DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In each drawing, parts having the same function or configuration are designated by the same reference numerals, and repeated description may be omitted.

[0013] [Example 1] In the first embodiment, multiple sensors installed on a moving body for observing the surrounding environment are composed of cameras, and the normal / abnormal determination of the common imaging area of the multiple sensors (cameras) is performed by matching identical pixels.

[0014] FIG. 1 shows an overall configuration diagram of an abnormality diagnosis device according to a first embodiment of the present invention.

[0015] The abnormality diagnosis device 1 of this embodiment is mounted on, for example, a vehicle and used, and includes a sensor unit 100, an object detection unit 200, a common normal area calculation unit 300, an object reliability determination unit 400, an individual abnormal image area determination unit 500, and a display / alarm / control unit 600.

[0016] The sensor unit 100 is composed of multiple imaging devices (cameras). The multiple cameras are arranged so that their fields of view overlap. FIG. 1 shows an example in which the sensor unit 100 is composed of three cameras: a left front camera 110, a right front camera 120, and a side camera 130. As shown in FIG. 2(a), the three cameras are arranged so that there is a common imaging area that can be imaged by the two cameras, the left front camera 110 and the right front camera 120 (the two cameras, the left front camera 110 and the right front camera 120, share a field of view), and a monocular area that can be imaged only by the single camera, the side camera 130 (consisting of only the field of view of the single camera, the side camera 130).

[0017] In the following, the camera system shown in Fig. 2(a) is used as an example, but other configurations may be used as long as the camera system has both a common imaging area and a monocular area. For example, as shown in Fig. 2(b), a camera system may be used in which two cameras, side camera 140 and side camera 150, are installed so that their fields of view overlap.

[0018] The sensor unit 100 outputs image information obtained by capturing an image of the surrounding environment as sensor information to the object detection unit 200 and the common normal area calculation unit 300.

[0019] The object detection unit 200 detects objects to be used for estimating an abnormal area from image information as sensor information acquired by the sensor unit 100. The objects include other vehicles, motorcycles, pedestrians, signs, billboards, and other features.

[0020] One detection method is to use template images of vehicles or pedestrians, as shown in Figures 3(a) and 3(b). As shown in Figure 3(c), the template image is scanned against an image acquired by the sensor unit 100 while driving, and similar areas are detected as targets. Furthermore, the method is not limited to such template matching methods, and targets may be detected using any algorithm.

[0021] When the object detection unit 200 detects an object from an image acquired by the sensor unit 100, it outputs the position and feature amount of the object in the image to the object reliability determination unit 400. The feature amount may be an HOG (Histograms of Oriented Gradients) feature amount, which is a histogram of the brightness gradient direction of the detected object area, but the feature amount may also be calculated using any other algorithm.

[0022] Furthermore, if the object detection unit 200 can calculate the direction of movement in the image, the distance to the object, and the movement speed in three-dimensional space, it may also output this information to the object reliability determination unit 400. Furthermore, since the object detection unit 200 may output an erroneous detection result when detecting at a single time, it may also output a time-series detection result obtained by analyzing the object detection results over multiple frames to the object reliability determination unit 400.

[0023] Also, characteristic areas in the image, such as edges and corners, may be included as targets.

[0024] The common normal area calculation unit 300 checks for the presence or absence of adhesions (dirt, raindrops, cloudiness, icing, etc.) on the lens and lens abnormalities (cracks, scratches, distortions, etc.) in the common imaging area where the fields of view of the left front camera 110 and the right front camera 120 overlap (share a field of view), and determines whether the common imaging area is normal or abnormal. The common normal area calculation unit 300 outputs the determination result to the object reliability determination unit 400 or the display / alarm / control unit 600.

[0025] The common normal area calculation unit 300 determines whether the common imaging area is normal or abnormal by searching for identical pixels in images captured by two cameras having a common imaging area. As shown in FIGS. 4(a) and 4(b), when neither the left front camera 110 nor the right front camera 120 has an abnormal condition such as adhesions to the lens or lens abnormalities (in other words, when both the image acquired by the left front camera 110 and the image acquired by the right front camera 120 are normal), identical pixels are calculated in the shaded area in FIGS. 4(a) and 4(b). On the other hand, as shown in FIGS. 5(a) and 5(b), when an abnormality occurs in one of the images (here, the right image acquired by the right front camera 120), identical pixels are calculated in the shaded area in FIGS. 5(a) and 5(b). The common normal area calculation unit 300 saves the area where these identical pixels are calculated as a common normal area. That is, the common normal area calculation unit 300 calculates the same pixels that match between the cameras that share the field of view (left front camera 110 and right front camera 120) as the common normal area.

[0026] The processing flow of the common normal area calculation unit 300 is shown in Fig. 6. Fig. 6 shows an example of a camera configuration in which the left front camera 110 and the right front camera 120 have a common imaging area, but similar processing can be performed with any cameras that have a common imaging area.

[0027] In steps S301 and S302, images captured by the left front camera 110 and the right front camera 120, which have a common imaging area, are acquired.

[0028] In the geometric correction of step S303, optical characteristics such as lens distortion are corrected for the acquired image.

[0029] The geometrically corrected image is searched for identical pixels. In step S304, a local region of the image captured by the left front camera 110 (left image) is cut out as a template. Furthermore, in step S305, a region similar to the cut-out template is searched for in the image captured by the right front camera 120 (right image). In step S306, if a similar region is found in the right image (matching is achieved), a normal label is assigned to the corresponding pixel in step S307. In step S306, if a similar region is not found in the right image (matching is not achieved), an abnormal label is assigned to the corresponding pixel in step S308. By performing this process on the entire image, a common normal region is determined.

[0030] The search for identical pixels is not limited to such a template matching method, and identical pixels may be searched for using any algorithm.

[0031] The object reliability determination unit 400 receives the results from the object detection unit 200 and the common normal region calculation unit 300, and assigns a reliability to each object detected by the object detection unit 200. The object reliability determination unit 400 outputs the results, to which the reliability for each object has been assigned, to the single abnormal image region determination unit 500.

[0032] The processing flow of the object reliability determination unit 400 is shown in Fig. 7. Fig. 7 shows an example using the left front camera 110, but the present invention is not limited to this, and similar processing can be performed with any camera that has common normal area information.

[0033] First, in step S401, object information (detection information) of the left front camera 110 detected by the object detection unit 200 is acquired.

[0034] Furthermore, in step S402, the common normal area information of the left front camera 110 calculated by the common normal area calculation unit 300 is acquired.

[0035] In step S403, if the position of the object detected by the object detection unit 200 is included in the common normal area (in other words, the area of the object detected by the object detection unit 200 is assigned a normal label), a first reliability is assigned to the object in step S404. In step S403, if the position of the object detected by the object detection unit 200 is not included in the common normal area (in other words, the area of the object detected by the object detection unit 200 is not assigned a normal label), a second reliability is assigned to the object in step S405. That is, the object reliability determination unit 400 assigns a first reliability to the object detected in the common normal area (step S404) and a second reliability to the object detected outside the common normal area (step S405). In other words, the object reliability determination unit 400 assigns different reliability to the object detected in the common normal area and the object detected outside the common normal area.

[0036] Since it is desirable to set a higher reliability for objects detected in the common normal area than for objects detected in other areas, a predetermined value is set in advance so that the first reliability is larger (higher) than the second reliability. Furthermore, if it is possible to acquire detection results in a time series obtained by analyzing the detection results of objects over multiple frames from the object detection unit 200, the first reliability and second reliability may be calculated by including information on past detection results.

[0037] The single abnormal image region determination unit 500 determines an abnormal image region in the monocular region based on the reliability of each object calculated by the object reliability determination unit 400, and outputs the result to the display / alarm / control unit 600.

[0038] 8(a) and (b) show an overview of the processing performed by the single abnormal image area determination unit 500. While the side camera 130 is used as an example in FIGS. 8(a) and (b), similar processing is possible for other monocular areas. Object 1 in FIG. 8(a) represents a pedestrian who was detected by (the image acquired by) the left front camera 110 and then moved into the image (monocular area) acquired by the side camera 130. Assuming that the left front camera 110 is normal, object 1 is assigned a first reliability. Object 2 in FIG. 8(a) represents a pedestrian who was detected for the first time by (the image acquired by) the side camera 130, and is assigned a second reliability because it was detected outside the common normal area.

[0039] FIG. 8(b) shows an anomaly score map, which stores an anomaly score corresponding to the side camera 130. The anomaly score can take a value between 0 and 1, with the closer to 0 being more normal and the closer to 1 being more abnormal, but any other numerical value may be used to store the normal / abnormal state. The anomaly score map is updated based on the detection information of target 1 and target 2; if detected, it is updated to approach 0 (normal), and if detection is no longer possible, it is updated to approach 1 (abnormal). The amount of update of the anomaly score in the anomaly score map at that time is changed according to the reliability of the target, and the higher the reliability of the target, the greater the anomaly score in the anomaly score map is updated.

[0040] The processing flow of the single abnormal image region determining unit 500 is shown in FIG.

[0041] First, in step S501, the appearance position of the object detected by the object detection unit 200 is predicted. The predicted appearance position may be around the position at the previous time, or if the movement direction of the object has been calculated, the predicted appearance position may be determined from that information, or if the movement amount of the host vehicle is known, that information may be used.

[0042] In S502, it is determined whether an object has appeared at the predicted position based on the feature values of the object. In addition, in S503 and S506, the reliability assigned to the tracked object is confirmed. If the object has appeared at the predicted position in S502 and a first reliability has been assigned to the object in S503, the abnormality score in the anomaly score map is decreased by a first update amount in step S504. If a second reliability has been assigned to the object in S503, the abnormality score in the anomaly score map is decreased by a second update amount in step S505. If the object has not appeared at the predicted position in S502 and a first reliability has been assigned to the object in S506, the abnormality score in the anomaly score map is increased by a first update amount in step S507. If a second reliability has been assigned to the object in S506, the abnormality score in the anomaly score map is increased by a second update amount in step S508. That is, when the target is detected (tracked) in the monocular region, the single abnormal image region determination unit 500 decreases (lowers) the abnormality score in the abnormality score map of the detected location (by the first update amount or the second update amount) (steps S504, S505).Furthermore, when the target is lost (cannot be tracked) in the monocular region, the single abnormal image region determination unit 500 increases (raise) the abnormality score in the abnormality score map of the lost location (by the first update amount or the second update amount) (steps S507, S508).

[0043] When objects overlap, the camera cannot capture the object behind it, which may prevent the object from being detected at the predicted position. In this case, to prevent the value of the anomaly score map from being increased erroneously, the overlap of objects may be detected from the relative positions of the objects, and objects determined to be hidden behind may not be used for updating the anomaly score map. If the distance and three-dimensional coordinates of the objects are known, this information may also be used to determine whether the objects overlap.

[0044] Since it is desired to update the abnormality score of the abnormality score map to a larger value for targets assigned the first reliability than for targets assigned the second reliability, a predetermined value is set in advance so that the first update amount is larger than the second update amount.

[0045] After updating the abnormality scores in the abnormality score map using all target information, normal / abnormal regions are determined in step S509. A predetermined threshold is set for the abnormality scores in the updated abnormality score map, and regions below the predetermined threshold are determined as normal regions, and regions above the predetermined threshold are determined as abnormal regions.

[0046] In addition, the abnormality score map may also be updated for areas that have been determined to be normal or abnormal, and if the score exceeds a predetermined threshold, it may be determined that an abnormality has occurred in the normal area (such as dirt being deposited while driving), or if the score falls below the predetermined threshold, it may be determined that the abnormal area has recovered (such as dirt being removed while driving or by a cleaning operation).

[0047] The display / warning / control unit 600 acquires information on the normal area or abnormal area calculated by the common normal area calculation unit 300 or the individual abnormal image area determination unit 500, and performs display or warning to the driver and / or control to resolve the abnormal state.

[0048] When the display / warning / control unit 600 receives information that the camera's imaging area is normal, it displays a message to the driver that the installed camera is normal. Also, when camera information is used for an automatic driving or driving assistance system, it may display a message that the system is operating normally.

[0049] On the other hand, when the display / warning / control unit 600 receives information that an abnormality has occurred in the camera's imaging area, it displays a message to the driver that the installed camera is abnormal. Also, when camera information is used in an automated driving or driving assistance system, the system operation may be stopped and information that the system is not operating may be displayed to the driver. Also, the automated driving or driving assistance system may be degraded in stages. For example, it may be possible to stop the vehicle on the shoulder using only cameras that are not experiencing abnormalities, or to stop the vehicle on the shoulder using only normal areas of the imaging area.

[0050] The display / warning / control unit 600 may also display a message requesting the driver to check the status of the camera in which the abnormality occurred. The display / warning / control unit 600 may also output a command to the camera in which the abnormality occurred to perform an operation to resolve the abnormality, such as operating the wipers or injecting windshield washer fluid or compressed air.

[0051] As described above, the abnormality diagnosis device 1 of this embodiment 1 comprises a sensor unit 100 in which multiple cameras (which observe the surrounding environment) are arranged so that their fields of view overlap, an object detection unit 200 that detects objects from sensor information (image information) of the sensor unit 100, an object reliability determination unit 400 that determines the reliability of the object based on the detection results in a common imaging area where the multiple cameras share a field of view, and a single abnormal image area determination unit 500 that determines an abnormal image area in a monocular area formed by only the field of view of a single camera among the multiple cameras using the tracking results of the object within the monocular area, and the single abnormal image area determination unit 500 changes the abnormal score in the abnormality score map to be assigned to the abnormal image area according to the reliability of the object.

[0052] The device further includes a common normal area calculation unit 300 that determines the state (normal / abnormal) of the common imaging area based on the detection results in the common imaging area, and the object reliability determination unit 400 determines the reliability of the object based on the state (normal / abnormal) of the common imaging area.

[0053] According to this embodiment, reliability is assigned to the target based on the detection result of the common imaging area, and the update amount of the abnormality score map of the monocular area is changed based on the reliability of the target, so that abnormal areas that occur in the sensor (camera) can be determined in a short time without erroneous judgment.

[0054] [Example 2] In Example 2, multiple sensors installed on a moving body to observe the surrounding environment are not limited to cameras, and a normal / abnormal judgment is made of the common sensing area of multiple sensors (including millimeter wave radar, LiDAR, etc. other than cameras) based on the type and positional relationship of the target.

[0055] FIG. 10 shows an overall configuration diagram of an abnormality diagnosis device according to a second embodiment of the present invention.

[0056] The abnormality diagnosis device 2 of this embodiment is mounted on, for example, a vehicle and used, and includes a sensor unit 1100, an object detection unit 1200, an object reliability determination unit 1400, an individual abnormality area determination unit 1500, and a display / alarm / control unit 1600.

[0057] The sensor unit 1100 is composed of multiple sensors. The multiple sensors are arranged so that their observation areas (also called sensing areas) overlap. In FIG. 10, the sensor unit 1100 is exemplified by two sensors, sensor 1110 and sensor 1120. As shown in FIG. 11, the two sensors are exemplified by a configuration of a front camera 1160 and a LiDAR 1170. However, other configurations are also possible as long as there is a common sensing area that can be sensed by two sensors (two sensors sharing an observation area) and an individual sensing area that can only be sensed by one (single) sensor (consisting only of the observation area of one (single) sensor), and different types of sensors such as sonar or millimeter-wave radar may be combined.

[0058] The sensor unit 1100 outputs sensing information (detection information) obtained by sensing the surrounding environment to the object detection unit 1200 as sensor information.

[0059] The object detection unit 1200 detects objects to be used for estimating an abnormal area from the detection information as sensor information acquired by the sensor unit 1100. The objects include other vehicles, motorcycles, pedestrians, signs, billboards, and other features.

[0060] As a detection method, an arbitrary algorithm suitable for each sensor is selected and used.

[0061] The object reliability determination unit 1400 receives the results from the object detection unit 1200 and assigns a reliability to each object detected by the object detection unit 1200. The object reliability determination unit 1400 outputs the results, with the reliability assigned to each object, to the single abnormal area determination unit 1500.

[0062] The processing flow of the target reliability determination unit 1400 is shown in Fig. 12. Fig. 12 shows an example using the front camera 1160 and the LiDAR 1170, but the present invention is not limited to this, and similar processing can be performed with any sensor that has common sensing area information.

[0063] First, in step S1401, object information (detection information) of the front camera 1160 detected by the object detection unit 1200 is acquired.

[0064] In step S1403, all the acquired detection targets are matched to determine whether the same target exists among the targets (detection information) detected by the LiDAR 1170. Whether the targets are the same is determined based on information such as the position, type, and movement direction of the target information (detection information).

[0065] In step S1403, if the same object exists among the objects detected by the LiDAR 1170, a first reliability is assigned to the detected object in step S1404. In step S1403, if the same object does not exist among the objects detected by the LiDAR 1170, a second reliability is assigned to the detected object in step S1405. That is, the object reliability determination unit 1400 assigns a first reliability to an object that could be observed (as the same object) by two sensors that share an observation area (step S1404), and assigns a second reliability to an object that could only be observed by one of the two sensors that share an observation area (step S1405).

[0066] The single abnormal region determination unit 1500 determines an abnormal region in the single sensing region based on the reliability of each target calculated by the target reliability determination unit 1400 and outputs the result to the display / alarm / control unit 1600. Similar to FIG. 9 , the processing flow of the single abnormal region determination unit 1500 decreases the abnormality score in the abnormality score map if the target is detected at the predicted position according to the reliability of the target being tracked, and increases the abnormality score in the abnormality score map if the target is not detected at the predicted position. That is, if the target is detected (tracked) in the single sensing region, the single abnormal region determination unit 1500 decreases (lowers) the abnormality score in the abnormality score map for the detected location (by the first or second update amount). Furthermore, if the target is lost (cannot be tracked) in the single sensing region, the single abnormal region determination unit 1500 increases (raises) the abnormality score in the abnormality score map for the lost location (by the first or second update amount).

[0067] After updating the abnormality scores of the abnormality score map using all target information, a predetermined threshold is set for the abnormality scores of the updated abnormality score map, and areas below the predetermined threshold are determined to be normal areas, and areas above the predetermined threshold are determined to be abnormal areas.

[0068] In addition, the abnormality score map may also be updated for areas that have been determined to be normal or abnormal, and if the score exceeds a predetermined threshold, it may be determined that an abnormality has occurred in the normal area (such as dirt being deposited while driving), or if the score falls below the predetermined threshold, it may be determined that the abnormal area has recovered (such as dirt being removed while driving or by a cleaning operation).

[0069] The display / warning / control unit 1600 acquires information on the normal region or abnormal region calculated by the single abnormal region determination unit 1500, and performs display or warning to the driver and / or control to eliminate the abnormal state.

[0070] As described above, the abnormality diagnosis device 2 of this embodiment 2 comprises a sensor unit 1100 in which multiple sensors (that observe the surrounding environment) are arranged so that their observation areas overlap, an object detection unit 1200 that detects an object from sensor information (detection information) of the sensor unit 1100, an object reliability determination unit 1400 that determines the reliability of the object based on the detection results in a common sensing area in which the multiple sensors share an observation area, and a single abnormal area determination unit 1500 that determines an abnormal area in the single sensing area by using the tracking results of the object within a single sensing area that is composed only of the observation area of a single sensor among the multiple sensors, and the single abnormal area determination unit 1500 changes the abnormal score in the abnormality score map that is assigned to the abnormal area according to the reliability of the object.

[0071] According to this embodiment, reliability is assigned to the target based on the detection result of the common sensing area, and the update amount of the anomaly score map of the individual sensing area is changed based on the reliability of the target, so that an abnormal area that has occurred in a sensor (camera, LiDAR, millimeter-wave radar, etc.) can be determined in a short time without erroneous judgment.

[0072] It should be noted that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0073] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a storage device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0074] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0075] 1. Abnormality diagnosis device (Example 1) 100 Sensor unit 200 Target detection unit 300 Common normal area calculation unit 400 Object reliability determination unit 500 Single abnormal image area determination unit 600 Display / Alarm / Control Unit 2. Abnormality diagnosis device (Example 2) 1100 Sensor unit 1200 Target detection unit 1400 Object reliability determination unit 1500 Single abnormal area determination unit 1600 Display / Alarm / Control Unit

Claims

1. a sensor unit in which a plurality of cameras are arranged so that their fields of view overlap; an object detection unit that detects an object based on sensor information from the sensor unit; an object reliability determination unit that determines the reliability of the object based on a detection result in a common imaging area that is shared by the plurality of cameras; a single abnormal image area determination unit that determines an abnormal image area in a monocular area formed by only a field of view of a single camera among the plurality of cameras, using a tracking result of the object in the monocular area, The abnormality diagnosis device is characterized in that the single abnormal image region determination unit changes the abnormality score in the abnormality score map assigned to the abnormal image region depending on the reliability of the target.

2. 2. The abnormality diagnosis device according to claim 1, a common normal area calculation unit that determines a state of the common imaging area based on a detection result in the common imaging area, The abnormality diagnosis device, wherein the object reliability determination unit determines the reliability of the object based on a state of the common imaging area.

3. 3. The abnormality diagnosis device according to claim 2, The common normal area calculation unit calculates identical pixels that match between cameras that share a field of view as the common normal area.

4. 4. The abnormality diagnosis device according to claim 3, The object reliability determination unit assigns a first reliability to an object detected in the common normal area, and assigns a second reliability different from the first reliability to an object detected outside the common normal area.

5. 5. The abnormality diagnosis device according to claim 4, The abnormality diagnosis device, wherein the object reliability determination unit sets the first reliability to a value higher than the second reliability.

6. 2. The abnormality diagnosis device according to claim 1, The abnormality diagnosis device is characterized in that the single abnormal image area determination unit lowers the abnormality score of the detected location in the abnormality score map when the object is detected in the monocular area.

7. 2. The abnormality diagnosis device according to claim 1, The abnormality diagnosis device is characterized in that the single abnormal image area determination unit increases the abnormality score of the lost location in the abnormality score map when the target is lost in the monocular area.

8. 2. The abnormality diagnosis device according to claim 1, The single abnormal image region determination unit determines, as an abnormal region, a region where the value of the abnormality score map exceeds a predetermined threshold, or determines, as a normal region, a region where the value of the abnormality score map is below a predetermined threshold.

9. 2. The abnormality diagnosis device according to claim 1, The abnormality diagnosis device is characterized in that the single abnormal image region determination unit updates the abnormality score of the abnormality score map to a larger value for an object with a higher reliability.

10. a sensor unit in which a plurality of sensors are arranged so that their observation areas overlap; an object detection unit that detects an object based on sensor information from the sensor unit; an object reliability determination unit that determines the reliability of the object based on a detection result in a common sensing area that is shared by the plurality of sensors; a single abnormal region determination unit that determines an abnormal region in a single sensing region using a tracking result of the target in the single sensing region that is configured only by an observation region of a single sensor among the plurality of sensors, The single abnormality region determination unit changes the abnormality score in the abnormality score map assigned to the abnormal region according to the reliability of the target.

11. The abnormality diagnosis device according to claim 10, The object reliability determination unit assigns a first reliability to an object that can be observed by multiple sensors that share an observation area, and assigns a second reliability different from the first reliability to an object that can be observed only by a single sensor among the multiple sensors that share the observation area.

12. The abnormality diagnosis device according to claim 11, The abnormality diagnosis device, wherein the object reliability determination unit sets the first reliability to a value higher than the second reliability.

13. The abnormality diagnosis device according to claim 10, The anomaly diagnosis device is characterized in that the single abnormality area determination unit lowers the abnormality score in the abnormality score map of the detected location when the object is detected in the single sensing area.

14. The abnormality diagnosis device according to claim 10, The anomaly diagnosis device is characterized in that the single abnormality area determination unit increases the abnormality score in the abnormality score map of the lost location when the target is lost in the single sensing area.

15. The abnormality diagnosis device according to claim 10, The single abnormality region determination unit determines an area where the value of the abnormality score map exceeds a predetermined threshold as an abnormal region, or determines an area where the value of the abnormality score map is below a predetermined threshold as a normal region.

Images