Anomaly detection device, anomaly detection program, and anomaly detection method

Abstract

To provide an anomaly detection device that suppresses cost increases. [Solution] An anomaly detection device for detecting abnormalities in objects outside a vehicle that are imaged from a vehicle comprises: an image acquisition unit capable of acquiring images of objects captured by a camera mounted on the vehicle; an image data acquisition unit that acquires image data of the object from the images acquired by the image acquisition unit; a data distribution learning unit that learns the normal data distribution of the object captured in the image by unsupervised learning of the image data acquired by the image data acquisition unit; and a distribution anomaly determination unit that determines that there is an abnormality in the object if the image data acquired by the image data acquisition unit deviates from the data distribution.

Description

Technical Field

[0001] The present disclosure relates to an abnormality detection device, an abnormality detection program, and an abnormality detection method for detecting an abnormality of an object outside a vehicle imaged from the vehicle.

Background Art

[0002] For example, Patent Document 1 discloses a technique for acquiring image information from a camera mounted on a vehicle and detecting the presence or absence of an abnormality in road facilities by comparing the state of road facilities obtained by analyzing this image information with basic information indicating the state of road facilities in a normal state. [[ID=I3]]

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technique described in Patent Document 1, it is necessary to prepare in advance basic information determined by a human to be correct data (a state in a normal state) for each road facility and for each imaging position imaged by a camera. For this reason, a very large amount of basic information is required, increasing the cost of the abnormality detection device.

[0005] The present disclosure has been made in view of the above problems, and an object thereof is to provide an abnormality detection device, an abnormality detection program, and an abnormality detection method in which an increase in cost is suppressed.

Means for Solving the Problems

[0006] To achieve the above objective, the anomaly detection device according to the present disclosure is an anomaly detection device for detecting an anomaly of an object outside the vehicle that is imaged from the vehicle, and comprises: an image acquisition unit capable of acquiring an image of the object captured by a camera mounted on the vehicle; an image data acquisition unit that acquires image data of the object from the image acquired by the image acquisition unit; a data distribution learning unit that learns the normal data distribution of the object captured in the image by unsupervised learning of the image data acquired by the image data acquisition unit; and a distribution anomaly determination unit that determines that there is an anomaly in the object if the image data acquired by the image data acquisition unit deviates from the data distribution.

[0007] To achieve the above objective, the anomaly detection program relating to this disclosure is an anomaly detection program for detecting anomalies in objects outside a vehicle that are imaged from the vehicle, and causes a computer to execute: an image acquisition process capable of acquiring an image of the object captured by a camera mounted on the vehicle; an image data acquisition process that acquires image data of the object from the image acquired by the image acquisition process; a data distribution learning process that learns the normal data distribution of the object captured in the image by unsupervised learning of the image data acquired by the image data acquisition process; and a distribution anomaly determination process that determines that there is an anomaly in the object if the image data acquired by the image data acquisition process deviates from the data distribution.

[0008] To achieve the above objective, the anomaly detection method relating to this disclosure is an anomaly detection method for detecting an anomaly in an object outside a vehicle that has been photographed from the vehicle, and comprises: an image acquisition step capable of acquiring an image of the object taken by a camera mounted on the vehicle; an image data acquisition step for acquiring image data of the object from the image acquired by the image acquisition step; a data distribution learning step for learning the normal data distribution of the object captured in the image by unsupervised learning of the image data acquired by the image data acquisition step; and a distribution anomaly determination step for determining that there is an anomaly in the object if the image data acquired by the image data acquisition step deviates from the data distribution. [Effects of the Invention]

[0009] According to the anomaly detection device, anomaly detection program, and anomaly detection method of this disclosure, cost increases can be suppressed. [Brief explanation of the drawing]

[0010] [Figure 1] This diagram schematically shows the configuration of an anomaly detection device according to one embodiment. [Figure 2] This is a schematic functional block diagram of a mobile terminal according to one embodiment. [Figure 3] This is a diagram illustrating an example of the operation of a mobile terminal according to one embodiment. [Figure 4] This figure shows an example of the screen display of a mobile device according to one embodiment. [Figure 5] This is a flowchart of an anomaly detection method according to one embodiment. [Modes for carrying out the invention]

[0011] Hereinafter, an anomaly detection device, an anomaly detection program, and an anomaly detection method according to embodiments of this disclosure will be described with reference to the drawings. Such embodiments represent one aspect of this disclosure and are not limiting, and can be modified at will within the scope of the technical idea of ​​this disclosure.

[0012] <Anomaly detection device> (composition) The anomaly detection device according to this disclosure detects anomalies in objects outside the vehicle that are imaged from the vehicle. Figure 1 is a schematic diagram showing the configuration of an anomaly detection device 1 according to one embodiment. In one embodiment, as illustrated in Figure 1, the anomaly detection device 1 is, for example, a mobile terminal 1A(1) such as a smartphone or tablet terminal, which is brought into the vehicle 100 by the user of the mobile terminal 1A. The mobile terminal 1A has a built-in camera 50 capable of capturing video and still images, and is fixed to the vehicle body by a fixing device (not shown) so that the front of the vehicle 100 is captured by the camera 50. The mobile terminal 1A includes a screen 52, which is configured as, for example, a touch panel. The mobile terminal 1A is equipped with a GPS 54 for acquiring location information X of the mobile terminal 1A, and is capable of acquiring location information X of the mobile terminal 1A when the camera 50 is capturing video or still images.

[0013] The object Ob is not particularly limited as long as it is imaged from a moving vehicle 100, and may be an artificial or natural object. This disclosure describes an example in which the object Ob includes a road Ob1(Ob) and a sign Ob2(Ob). In some embodiments, the object Ob includes at least one of the road and the sign. In some embodiments, the object Ob includes at least one of the sign, a lighting pole, a fence, vegetation, and a utility pole.

[0014] The anomaly detection device 1 (mobile terminal 1A) is a computer such as an electronic control unit, and includes a processor such as a CPU or GPU (not shown), memory such as ROM or RAM, and an I / O interface. The anomaly detection device 1 realizes each of its functional units by having the processor operate (calculate, etc.) according to the instructions of the program loaded into memory. The functional units of the mobile terminal 1A will be described with reference to Figure 2.

[0015] Figure 2 is a schematic functional block diagram of a mobile terminal 1A according to one embodiment. As shown in Figure 2, the mobile terminal 1A includes an image acquisition unit 2, an image data acquisition unit 4, a data distribution learning unit 6, and a distribution anomaly determination unit 8. In one embodiment, the mobile terminal 1A further includes a difference acquisition unit 10, a first image selection unit 12, a positioning unit 14, a text acquisition unit 16, a text anomaly determination unit 18, a drawing unit 20, and a determination result display unit 22. Although not shown in Figure 2, the mobile terminal 1A includes control units that control the camera 50, the screen 52, and the GPS 54, respectively.

[0016] The image acquisition unit 2 acquires an image Im of the road Ob1 and sign Ob2 captured by the camera 50 of the mobile terminal 1A mounted on the vehicle 100. In one embodiment, the image Im is a video. The image Im includes a first image Im1 capturing the road Ob1 and sign Ob2, and a second image Im2 capturing the road Ob1 and sign Ob2 at a later time than when the first image Im1 was captured. In some embodiments, the second image Im2 is captured in real time. The camera 50 that captures the first image Im1 and the camera 50 that captures the second image Im2 may be the same or different.

[0017] The image data acquisition unit 4 acquires image data D1(D) of road Ob1 and image data D2(D) of sign Ob2 from the image Im acquired by the image acquisition unit 2. In other words, the image data acquisition unit 4 selects one video frame from a plurality of video frames (still images) that make up the image Im (video), and acquires image data D1 of road Ob1 and image data D2 of sign Ob2 from this selected video frame. In one embodiment, each of the image data D1 of road Ob1 and the image data D2 of sign Ob2 includes location information X of the mobile terminal 1A indicating the location where the image Im was captured.

[0018] The image data D is, for example, the R value, G value, and B value included in the pixels constituting a video frame. Each of the R value, G value, and B value is a numerical value between 0 and 255. In some embodiments, the image data is the C value, Y value, M value, and K value. The image data may be other than color, for example, brightness or chroma.

[0019] (Out-of-distribution detection) The data distribution learning unit 6 learns the normal data distribution of the road Ob1 and the sign Ob2 imaged in the image Im (the first image Im1 and the second image Im2) by performing unsupervised learning on the image data D1 of the road Ob1 and the image data D2 of the sign Ob2 acquired by the image data acquisition unit 4. Specifically, the data distribution learning unit 6 acquires the distribution of the RGB values of the road Ob1 and the distribution of the RGB values of the sign Ob2. For example, each of the R value, G value, and B value of the road Ob1 whose color is close to black is near 0, and each of the R value, G value, and B value of the sign Ob2 whose color is close to white is near 255. Then, the data distribution learning unit 6 learns the normal data distribution of the RGB values of the road Ob1 with outliers removed from the acquired distribution of the RGB values of the road Ob1 by a predetermined method, and continues to update the normal data distribution of the RGB values of the road Ob1. Similarly, the data distribution learning unit 6 learns the data distribution of the normal distribution of the RGB values of the sign Ob2 with outliers removed from the acquired distribution of the RGB values of the sign Ob2 by a predetermined method, and continues to update the data distribution of the normal distribution of the RGB values of the sign Ob2.

[0020] FIG. 3 is a diagram for explaining the operation of the mobile terminal 1A according to an embodiment, and shows a second image Im2 in which a road Ob1 and a sign Ob2 are represented. In the form illustrated in FIG. 3, the road Ob1 includes a roadway 60, a road shoulder 62, a center line 64, and an outer lane line 66. The data distribution learning unit 6 learns the normal data distribution of each of the roadway 60, the road shoulder 62, the center line 64, and the outer lane line 66 by performing unsupervised learning on the RGB values (image data D1) of the road Ob1. Similarly, the data distribution learning unit 6 learns the normal data distribution of the sign Ob2 by performing unsupervised learning on the RGB values (image data D2) of the sign Ob2. In some embodiments, the data distribution learning unit 6 learns the shape and position of the roadway 60, the road shoulder 62, the center line 64, and the outer lane line 66 based on the normal data distribution of each of them.

[0021] When the image data D1 of the road Ob1 acquired by the image data acquisition unit 4 deviates from the normal data distribution of the road Ob1, the distribution abnormality determination unit 8 determines that there is an abnormality in the road Ob1. When the image data D2 of the sign Ob2 acquired by the image data acquisition unit 4 deviates from the normal data distribution of the sign Ob2, the distribution abnormality determination unit 8 determines that there is an abnormality in the sign Ob2. Specifically described while referring to FIG. 3, if a crack Dm is formed in the roadway 60, the color of the crack Dm and the color of the roadway 60 are significantly different from each other. Therefore, the distribution abnormality determination unit 8 detects an abnormal region R (constituted by one or more pixels) indicating the RGB values of the roadway 60 that deviates from the normal data distribution. Then, when the distribution abnormality determination unit 8 detects the abnormal region R, it determines that there is an abnormality in the road Ob1. Thus, by including the data distribution learning unit 6 and the distribution abnormality determination unit 8, the mobile terminal 1A performs out-of-distribution detection for detecting an abnormality of the object Ob depending on whether the image data D deviates from the normal data distribution.

[0022] (Difference Detection) Using marker Ob2 as an example, the difference detection by the difference acquisition unit 10, the first image selection unit 12, and the alignment unit 14 will be explained. The difference acquisition unit 10 acquires the difference between marker Ob2a (Ob2) captured in the first image Im1 and marker Ob2b (Ob2) captured in the second image Im2. In one embodiment, the difference acquisition unit 10 acquires the difference between marker Ob2a in the first image Im1 and marker Ob2b in the second image Im2, which have been aligned by the alignment unit 14. In some embodiments, the difference acquisition unit 10 acquires the difference between marker Ob2a in the first image Im1 and marker Ob2b in the second image Im2, which have been selected by the first image selection unit 12.

[0023] To explain the difference acquisition unit 10, Figure 3 shows the marker Ob2a in the first image Im1 as a dotted line. The marker Ob2a in the first image Im1 is in a normal state, while the marker Ob2b in the second image Im2 is in an abnormal state with its support bent. In one embodiment, the difference acquisition unit 10 superimposes the marker Ob2a in the first image Im1 onto the marker Ob2b in the second image Im2, and determines that there is an abnormality in the marker Ob2 if the non-overlapping portion (set of pixels) is greater than a certain amount. In some embodiments, the difference acquisition unit 10 superimposes the marker Ob2a in the first image Im1 onto the marker Ob2b in the second image Im2, and determines that there is an abnormality in the marker Ob2 if there are pixels with a difference of RGB values ​​greater than a certain amount.

[0024] The first image selection unit 12 selects the first image Im1 which contains the same position information X as the second image Im2. In one embodiment, the first image selection unit 12 selects a video frame of the first image Im1 (video) from among a plurality of video frames that constitute the first image Im1 (video) which contains the same position information X as the video frame of the second image Im2.

[0025] The alignment unit 14 uses a feature object 70 common to the video frames of the first image Im1 and the second image Im2 selected by the first image selection unit 12 to align the marker Ob2a captured in the first image Im1 with the Ob2b captured in the second image Im2. The feature object 70 is a stationary artificial or natural object, such as a road shoulder 62, a center line 64, or a roadway outer line 66. The feature object 70 is formed by a predetermined number of pixels or more. In this disclosure, the case where the road shoulder 62 is the feature object 70 will be explained as an example. The alignment unit 14 moves the road shoulder 62 in the video frame of the first image Im1 to the road shoulder 62 in the video frame of the second image Im2 to perform alignment.

[0026] (VLM) Using marker Ob2 as an example, the VLM (Vision and Language Model) by the text acquisition unit 16 and the text anomaly determination unit 18 will be explained. The text acquisition unit 16 acquires the state of marker Ob2 in text from the image Im (first image Im1 and second image Im2) acquired by the image acquisition unit 2. The text anomaly determination unit 18 then determines whether or not there is an anomaly in marker Ob2 based on the text acquired by the text acquisition unit 16. Referring to Figure 3, the text acquisition unit 16 acquires the state of marker Ob2b in the second image Im2 in text, such as "bent" or "white". The text anomaly determination unit 18 then determines that there is an anomaly in marker Ob2 if it detects "bent" in the text acquired by the text acquisition unit 16.

[0027] Figure 4 shows an example of the display on the screen 52 of a mobile terminal 1A according to one embodiment. In one embodiment, as illustrated in Figure 4, the mobile terminal 1A is configured to display a map 80 on the screen 52. The drawing unit 20 draws a symbol 82 on the map 80 that indicates the location where the image Im was captured, based on the location information X. When the symbol 82 is selected, the determination result display unit 22 displays on the screen 52 the determination result, which determines whether or not there is an abnormality in the marker Ob2 in the image Im corresponding to the symbol 82. In some embodiments, the mobile terminal 1A further includes a storage unit that stores the image Im and the determination result.

[0028] (Effects / Actions) The operation and effects of a mobile terminal 1A according to one embodiment will be described. According to one embodiment, since the mobile terminal 1A performs out-of-distribution detection, the normal data distribution of road Ob1 and sign Ob2 captured in image Im is generated by unsupervised learning, and ground truth data indicating that road Ob1 and sign Ob2 are normal is not required. If ground truth data is used, for example, it is necessary to prepare ground truth data in advance for each road Ob1 and sign Ob2 and for each imaging position captured by camera 50, which would incur significant costs. However, according to one embodiment, as described above, ground truth data is not required, so the increase in cost of the mobile terminal 1A can be suppressed.

[0029] Out-of-distribution detection is suitable for detecting damage or dirt on roads Ob1 and signs Ob2, but unsuitable for detecting deformation of roads Ob1 and signs Ob2. Out-of-distribution detection alone may not be able to identify, for example, a bent sign Ob2 as abnormal. In contrast, according to one implementation, the mobile terminal 1A also performs differential detection. Therefore, the accuracy of detecting deformation of roads Ob1 and signs Ob2 can be improved. Furthermore, since differential detection does not require ground truth data, the cost increase of the mobile terminal 1A can be suppressed.

[0030] According to one embodiment, the difference between marker Ob2a in the first image Im1 and marker Ob2b in the second image Im2, whose position information X is equal to each other, is obtained, thereby improving the detection accuracy of deformation of marker Ob2. According to one embodiment, the position of marker Ob2a in the first image Im1 and marker Ob2b in the second image Im2 is aligned using the road shoulder 62 (feature object 70), and then the difference between marker Ob2a in the first image Im1 and marker Ob2b in the second image Im2 is obtained. Therefore, the detection accuracy of deformation of marker Ob2 can be improved.

[0031] According to one embodiment, the mobile terminal 1A also performs VLM (Visual Memory Learning). This improves the accuracy of detecting deformations of roads Ob1 and signs Ob2. Furthermore, since VLM does not require ground truth data, it is possible to suppress an increase in the cost of the mobile terminal 1A.

[0032] According to one embodiment, a symbol 82 is drawn on the map 80 displayed on the screen 52 of the mobile terminal 1A. As a result, the user of the mobile terminal 1A can quickly find out the location of the marker Ob2 for which an abnormality has been determined. In addition, the user of the mobile terminal 1A can quickly find out the result of the abnormality determination of marker Ob2 by selecting the symbol 82.

[0033] Furthermore, this disclosure does not limit the anomaly detection device 1 to a mobile terminal 1A. In some embodiments, the anomaly detection device 1 is a server separate from the camera mounted on the vehicle 100. The server is electrically connected to the camera, for example, via the internet, and is capable of acquiring images from the camera.

[0034] The anomaly detection device 1 may perform out-of-distribution detection on all of the objects Ob in the image Im, or it may perform out-of-distribution detection on only some of the objects Ob in the image Im. The anomaly detection device 1 may perform only out-of-distribution detection on the objects Ob in the image Im, or it may perform out-of-distribution detection in addition to at least one of difference detection and VLM.

[0035] <Anomaly detection program> The anomaly detection program described herein causes a computer such as a smartphone, tablet, or server to execute: an image acquisition process capable of acquiring an image Im of an object Ob captured by a camera mounted on a vehicle 100; an image data acquisition process that acquires image data D of the object Ob from the image Im acquired by the image acquisition process; a data distribution learning process that learns the normal data distribution of the object Ob captured in the image Im by unsupervised learning of the image data D acquired by the image data acquisition process; and a distribution anomaly determination process that determines that there is an anomaly in the object Ob if the image data D acquired by the image data acquisition process deviates from the data distribution.

[0036] In some embodiments, the image Im includes a first image Im1 capturing a specific object Ob from among the objects Ob, and a second image Im2 capturing the specific object Ob after the timing at which the first image Im1 was captured. The anomaly detection program then causes the computer to perform a difference acquisition process to obtain the difference between the specific object Ob captured in the first image Im1 and the specific object Ob captured in the second image Im2.

[0037] In some embodiments, the anomaly detection program causes the computer to further execute a text acquisition process that obtains the state of the object Ob in text from the image Im acquired by the image acquisition process, and a text anomaly determination process that determines whether or not there is an anomaly in the object Ob from the text acquired by the text acquisition process.

[0038] <Anomaly detection method> Figure 5 is a flowchart of an anomaly detection method according to one embodiment. The anomaly detection method detects anomalies in an object Ob outside the vehicle 100, which is imaged from the vehicle 100. As shown in Figure 5, the anomaly detection method includes an image acquisition step S1, an image data acquisition step S2, a data distribution learning step S3, and a distribution anomaly determination step S4.

[0039] In the image acquisition step S1, an image (Im) of the object Ob captured by a camera (50) mounted on the vehicle (100) is acquired. In the image data acquisition step S2, image data (D) of the object Ob is acquired from the image (Im) acquired in the image acquisition step S1. In the data distribution learning step S3, the normal data distribution of the object Ob captured in the image (Im) is learned by unsupervised learning of the image data (D) acquired in the image data acquisition step S2. In the distribution anomaly determination step S4, if the image data (D) acquired in the image data acquisition step S2 deviates from the data distribution, it is determined that there is an anomaly in the object Ob.

[0040] In the embodiment illustrated in Figure 5, image Im includes a first image Im1 capturing a specific object Ob from among the objects Ob, and a second image Im2 capturing the specific object Ob after the timing of capturing the first image Im1. The anomaly detection method further includes a difference acquisition step S5 that acquires the difference between the specific object Ob captured in the first image Im1 and the specific object Ob captured in the second image Im2, a text acquisition step S6 that acquires the state of the object Ob in text from the image Im acquired in the image acquisition step S1, and a text anomaly determination step S7 that determines whether or not there is an anomaly in the object Ob from the text acquired in the text acquisition step S6. In some embodiments, the difference acquisition step S5 is performed simultaneously with or before the distribution anomaly determination step S4. In some embodiments, the text anomaly determination step S7 is performed simultaneously with or before the distribution anomaly determination step S4.

[0041] The contents described in each of the above embodiments can be understood, for example, as follows:

[0042] [1] The anomaly detection device (1) relating to this disclosure is an anomaly detection device that detects an anomaly of an object (Ob) outside the vehicle that is imaged from the vehicle (100), An image acquisition unit (2) capable of acquiring an image (Im) of the object captured by a camera (50) mounted on the vehicle, An image data acquisition unit (4) acquires image data (Im) of the object from the image acquired by the image acquisition unit, A data distribution learning unit (6) learns the normal data distribution of the object captured in the image by performing unsupervised learning on the image data acquired by the image data acquisition unit, The system includes a distribution anomaly determination unit (8) that determines that there is an abnormality in the object if the image data acquired by the image data acquisition unit deviates from the data distribution.

[0043] According to the configuration described in [1] above, the normal data distribution of objects captured in the image is generated by unsupervised learning, and ground truth data indicating that the objects are normal is not required. If ground truth data is used, for example, it would be necessary to prepare ground truth data in advance for each object and for each imaging position captured by the camera, which would incur significant costs. However, according to the configuration described in [1] above, ground truth data is not required as described above, so the increase in the cost of the anomaly detection device can be suppressed.

[0044] [2] In some embodiments, in the configuration described in [1] above, The aforementioned image includes a first image (Im1) of a specific object among the aforementioned objects, and a second image (Im2) of the same specific object taken at a time after the first image was taken. The system further includes a difference acquisition unit (10) that acquires the difference between the specific object captured in the first image and the specific object captured in the second image.

[0045] A method for determining anomalies in an object based on whether or not it is outside the data distribution (out-of-distribution detection) is suitable for detecting damage or dirt on an object, but unsuitable for detecting deformation. Out-of-distribution detection alone may not be able to determine, for example, that a bent road sign is abnormal. In contrast, the configuration described in [2] above also employs a method to acquire the difference between a specific object in the first image and a specific object in the second image, which are captured at different timings (difference detection). This improves the accuracy of detecting deformation of the object.

[0046] [3] In some embodiments, in the configuration described in [2] above, The aforementioned image data includes location information (X) indicating the location where the image was captured. The system further includes a first image selection unit (12) that selects the first image containing the same position information as the second image.

[0047] According to the configuration described in [3] above, the difference between a specific object in the first image and a specific object in the second image, whose positional information is equal to each other, is obtained, thereby improving the accuracy of detecting deformation of the object.

[0048] [4] In some embodiments, in the configuration described in [3] above, The system further includes an alignment unit (14) that uses a feature (70) common to the first image and the second image selected by the first image selection unit to align the specific object captured in the first image with the specific object captured in the second image. The difference acquisition unit acquires the difference between the object in the first image, which has been aligned by the alignment unit, and the object in the second image.

[0049] The configuration described in [4] above can improve the accuracy of detecting deformation of the object compared to the configuration described in [3] above.

[0050] [5] In some embodiments, in the configuration described in any one of [1] to [4] above, A text acquisition unit (16) acquires the state of the object in text from the image acquired by the image acquisition unit, The system further includes a text abnormality determination unit (18) that determines whether or not there is an abnormality in the object based on the text acquired by the text acquisition unit.

[0051] According to the configuration described in [5] above, a method (VLM) for determining whether or not there is an abnormality in the object from the text will also be performed. This makes it possible to improve the accuracy of detecting deformation of the object.

[0052] [6] In some embodiments, in the configuration described in any one of [1] to [5] above, The aforementioned image data includes location information (X) indicating the location where the image was captured. A screen (52) capable of displaying a map (80), A drawing unit (20) that draws a symbol (82) indicating the location where the image was captured based on the location information on the map displayed on the screen, The system further includes a determination result display unit (22) that, when the symbol is selected, displays on the screen the determination result of whether or not there is an abnormality in the object in the image corresponding to the symbol.

[0053] According to the configuration described in [6] above, the user of the anomaly detection device can quickly find out the location of the object for which an anomaly was determined, and the result of the determination of whether or not the object has an anomaly.

[0054] [7] In some embodiments, in the configuration described in [1] above, The aforementioned object includes at least one of a road (Ob1) and a sign (Ob2).

[0055] According to the configuration described in [7] above, an abnormality in an object is determined by whether or not it is outside the data distribution (out-of-distribution detection), so abnormalities such as road damage and dirt on signs can be detected with high accuracy.

[0056] [8] In some embodiments, in the configuration described in [2] above, The aforementioned object includes at least one of the following: signs, lighting poles, fences, plants, and utility poles.

[0057] According to the configuration described in [8] above, a method (difference detection) is also performed to acquire the difference between a specific object in the first image and a specific object in the second image, which are captured at different timings. Therefore, abnormalities such as deformation of signs, lighting poles, fences, plants, and utility poles can be detected with high accuracy.

[0058] [9] In some embodiments, in the configuration described in [5] above, The aforementioned object includes at least one of the following: signs, lighting poles, fences, plants, and utility poles.

[0059] According to the configuration described in [9] above, a method for determining whether or not there is an abnormality in an object from the text (VLM) is also used, so abnormalities such as deformation of signs, lighting poles, fences, plants, and utility poles can be detected with high accuracy.

[0060]

[10] An anomaly detection program relating to the present disclosure is an anomaly detection program for detecting an anomaly in an object (Ob) outside the vehicle that is imaged from the vehicle (100), Computer (1), Image acquisition processing capable of acquiring an image (Im) of the object captured by a camera (50) mounted on the vehicle, The aforementioned image acquisition process includes an image data acquisition process that acquires image data (D) of the object from the image acquired by the aforementioned image acquisition process, The image data acquisition process includes a data distribution learning process that learns the normal data distribution of the object captured in the image by unsupervised learning of the image data acquired by the image data acquisition process, If the image data acquired by the image data acquisition process deviates from the data distribution, the system executes a distribution anomaly determination process that determines that there is an abnormality in the object.

[0061] According to the configuration described in

[10] above, the normal data distribution of objects captured in the image is generated by unsupervised learning, and ground truth data indicating that the objects are normal is not required. If ground truth data is used, for example, it would be necessary to prepare ground truth data in advance for each object and for each imaging position captured by the camera, which would incur significant costs. However, according to the program described in

[10] above, ground truth data is not required as described above, so the increase in the cost of the anomaly detection program can be suppressed.

[0062]

[11] An anomaly detection method relating to the present disclosure is an anomaly detection method for detecting an anomaly in an object (Ob) outside the vehicle that is imaged from the vehicle (100), An image acquisition step that can acquire an image (Im) of the object captured by a camera (50) mounted on the vehicle, The image acquisition step includes an image data acquisition step in which image data (D) of the object is acquired from the image acquired, The image data acquisition step includes a data distribution learning step which learns the normal data distribution of the object captured in the image by unsupervised learning of the image data acquired in the image data acquisition step, The system includes a distribution anomaly determination step, which determines that there is an abnormality in the object if the image data acquired in the image data acquisition step deviates from the data distribution.

[0063] According to the method described in

[11] above, the normal data distribution of objects captured in the image is generated by unsupervised learning, and ground truth data indicating that the objects are normal is not required. If ground truth data is used, for example, it would be necessary to prepare ground truth data in advance for each object and for each imaging position captured by the camera, which would incur significant costs. However, as described above, the method described in

[11] above does not require ground truth data, so it is possible to suppress the increase in the cost of the anomaly detection method. [Explanation of symbols]

[0064] 1. Anomaly detection device 1A Mobile device 2 Image acquisition unit 4. Image Data Acquisition Unit 6. Data Distribution Learning Unit 8 Distribution anomaly determination section 10 Difference acquisition part 12. First Image Selection Section 14 Alignment section 16 Text acquisition unit 18 Text Anomaly Detection Unit 20 Drawing section 22 Judgment result display section 50 Cameras 52 screens 60 Roadway 62 Road shoulder 64 Center Line 66 Outer edge line of the carriageway 70 Characteristic Features 80 Maps 82 Symbols 100 vehicles D Image data D1 Road Image Data Image data of D2 sign Dm Hibi Im Image Im1 First Image Image 2 (second image) Ob object Ob1 road Ob2 indicator R abnormal area S1 Image acquisition step S2 Image Data Acquisition Step S3 Data Distribution Learning Steps S4 Distribution Anomaly Detection Step S5 Difference Acquisition Step S6 Text Acquisition Step S7 Text Anomaly Detection Step X Location information

Claims

1. An anomaly detection device for detecting anomalies in objects outside the vehicle that are imaged from the vehicle, An image acquisition unit capable of acquiring an image of the object captured by a camera mounted on the vehicle, An image data acquisition unit acquires image data of the object from the image acquired by the image acquisition unit, A data distribution learning unit learns the normal data distribution of the object captured in the image by unsupervised learning of the image data acquired by the image data acquisition unit, The system includes a distribution anomaly determination unit that determines that there is an abnormality in the object if the image data acquired by the image data acquisition unit deviates from the data distribution. Anomaly detection device.

2. The aforementioned image includes a first image of a specific object among the aforementioned objects, and a second image of the same specific object taken at a time after the first image was taken. The system further includes a difference acquisition unit that acquires the difference between the specific object captured in the first image and the specific object captured in the second image. An anomaly detection device according to claim 1.

3. The aforementioned image data includes location information indicating the location where it was captured. The system further includes a first image selection unit that selects the first image containing the same position information as the second image, An anomaly detection device according to claim 2.

4. The system further includes an alignment unit that uses a feature common to the first image and the second image selected by the first image selection unit to align the specific object captured in the first image with the specific object captured in the second image, The difference acquisition unit acquires the difference between the object in the first image and the object in the second image, which has been aligned by the alignment unit. An anomaly detection device according to claim 3.

5. A text acquisition unit that acquires the state of the object in text format from the image acquired by the image acquisition unit, The system further comprises a text abnormality determination unit that determines whether or not there is an abnormality in the object based on the text acquired by the text acquisition unit, An anomaly detection device according to any one of claims 1 to 4.

6. The aforementioned image data includes location information indicating the location where it was captured. A screen capable of displaying a map, A drawing unit that draws a symbol indicating the location where the image was captured based on the location information on the map displayed on the screen, The system further includes a determination result display unit that, when the symbol is selected, displays on the screen the determination result of whether or not there is an abnormality in the object in the image corresponding to the symbol. An anomaly detection device according to any one of claims 1 to 4.

7. The aforementioned object includes at least one of a road and a sign. An anomaly detection device according to claim 1.

8. The aforementioned object includes at least one of the following: signs, lighting poles, fences, plants, and utility poles. An anomaly detection device according to claim 2.

9. The aforementioned object includes at least one of the following: signs, lighting poles, fences, plants, and utility poles. An anomaly detection device according to claim 5.

10. An anomaly detection program for detecting anomalies in objects outside the vehicle that are imaged from the vehicle, On the computer, Image acquisition processing capable of acquiring an image of the object captured by a camera mounted on the vehicle, The image acquisition process includes an image data acquisition process that acquires image data of the target object from the acquired image, The image data acquisition process includes a data distribution learning process that learns the normal data distribution of the object captured in the image by unsupervised learning of the image data acquired by the image data acquisition process, If the image data acquired by the image data acquisition process deviates from the data distribution, the system executes a distribution anomaly determination process that determines that there is an abnormality in the object. Anomaly detection program.

11. An anomaly detection method for detecting anomalies in objects outside the vehicle that are imaged from the vehicle, An image acquisition step that allows acquisition of an image of the object captured by a camera mounted on the vehicle, The image acquisition step includes an image data acquisition step that acquires image data of the object from the acquired image, The image data acquisition step includes a data distribution learning step which learns the normal data distribution of the object captured in the image by unsupervised learning of the image data acquired in the image data acquisition step, The system includes a distribution anomaly determination step, which determines that there is an abnormality in the object if the image data acquired in the image data acquisition step deviates from the data distribution. Anomaly detection method.

Images