Abnormality detection device, abnormality detection method, and program

Abstract

An abnormality detection device according to the present invention generates a plurality of altered images by altering each of a plurality of captured images that are in a time sequence. Alteration of the captured images includes masking processing of masking one or more partial regions included in the captured images, and restoration processing of restoring the masked partial regions using data from other than the partial regions. For each of the plurality of captured images, the abnormality detection device calculates difference data representing difference between the captured image and the altered image generated from the captured image, and determines whether or not the captured images represent an abnormal state on the basis of a plurality of pieces of the calculated difference data.

Description

Anomaly detection device, anomaly detection method, and program

[0001] The present disclosure relates to an abnormality detection device, an abnormality detection method, and a program.

[0002] Technologies for detecting anomalies through image analysis have been developed. For example, Patent Literature 1 discloses a technology for detecting defective products through image analysis. The system in Patent Literature 1 masks a portion of an image of an item such as a product, then restores the image and compares the original image with the restored image to detect defects in the item.

[0003] Special Publication No. 2020-525940

[0004] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu, "SimMIM: A Simple Framework for Masked Image Modeling," [online], November 18, 2021, [Retrieved January 16, 2023], Internet, <URL: https: / / arxiv.org / pdf / 2111.09886.pdf>

[0005] The technology disclosed in Patent Document 1 does not take into account changes in the situation of the target of anomaly detection. The present disclosure has been made in consideration of the above-mentioned problems, and one of its purposes is to provide a new technology for detecting anomalies using images.

[0006] The anomaly detection device of the present disclosure includes a generating unit that generates a plurality of processed images by processing each of a plurality of captured images in time series, a calculating unit that calculates, for each of the plurality of captured images, difference data representing the difference between the captured image and the processed image generated from the captured image, and a determining unit that determines whether the captured image represents an abnormal situation based on the calculated plurality of difference data. The processing performed on the captured images includes a masking process that masks one or more partial regions included in the captured image, and a restoration process that restores the masked partial regions using data other than the partial regions.

[0007] The anomaly detection method according to the present disclosure is executed by a computer. The anomaly detection method includes a generating step of generating a plurality of processed images by processing each of a plurality of captured images in a time series, a calculating step of calculating, for each of the captured images, difference data representing a difference between the captured image and the processed image generated from the captured image, and a determining step of determining whether the captured image represents an abnormal situation based on the calculated difference data. The processing performed on the captured images includes a masking process of masking one or more partial regions included in the captured image and a restoration process of restoring the masked partial regions using data other than the partial regions.

[0008] The program of the present disclosure causes a computer to execute the anomaly detection method of the present disclosure.

[0009] According to the present disclosure, a new technique for detecting anomalies using images is provided.

[0010] FIG. 1 is a diagram illustrating an overview of an anomaly detection device. FIG. 2 is a diagram illustrating an overview of processing performed on a captured image. FIG. 3 is a block diagram illustrating a functional configuration of an anomaly detection device. FIG. 4 is a block diagram illustrating a hardware configuration of a computer that realizes the anomaly detection device. FIG. 5 is a flowchart illustrating a flow of processing executed by the anomaly detection device. FIG. 6 is a diagram illustrating a mask processing using a mask image. FIG. 7 is a diagram illustrating a case in which two mask images are applied alternately. FIG. 8 is a diagram illustrating a case in which a normal region is excluded from a region to be masked. FIG. 9 is a diagram illustrating a functional configuration of an anomaly detection device having an output unit. FIG. 10 is a diagram illustrating an observation screen including a display indicating that the observation range is in an abnormal situation.

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0012] <Overview> Fig. 1 is a diagram illustrating an example of an overview of an anomaly detection device 2000 according to an embodiment. Fig. 1 is a diagram for facilitating understanding of the overview of the anomaly detection device 2000, and the operation of the anomaly detection device 2000 is not limited to that shown in Fig. 1.

[0013] The anomaly detection device 2000 acquires a plurality of captured images 10 and determines whether the captured images 10 represent an abnormal situation. The plurality of captured images 10 acquired by the anomaly detection device 2000 are time-series image data generated by a camera 20. For example, the camera 20 is configured to generate video data by repeatedly capturing images. In this case, each captured image 10 is a video frame that constitutes the video data.

[0014] The anomaly detection device 2000 performs processing on each captured image 10 to generate a processed image 30 from the captured image 10. FIG. 2 is a diagram illustrating an example of an outline of the processing performed on the captured image 10. The processing performed on the captured image 10 includes a masking process and a restoration process. The masking process is a process of generating an intermediate image 50 by masking at least one partial region included in the captured image 10. In FIG. 2, the masked image region is represented by a dot pattern. The restoration process is a process of generating a processed image 30 by restoring the masked partial region from the intermediate image 50 (i.e., the captured image 10 whose partial region has been masked).

[0015] Here, the process of restoring a masked partial area in a certain intermediate image 50 is a process of inferring the content of the partial area by using data of image areas other than the masked partial area that are included in the intermediate image 50. Therefore, some difference may occur between the processed image 30 and the captured image 10 from which the processed image 30 is based.

[0016] For example, in FIG. 2 , a captured image 10 is an image obtained by capturing an image of a road. A fallen object 60 is captured in the captured image 10. In the intermediate image 50, an image region representing the fallen object 60 is masked. Here, the fallen object 60 is not restored in a restoration process based on the unmasked image region. Therefore, the fallen object 60 is not included in the processed image 30 obtained by the restoration process. As a result, a difference occurs between the captured image 10 and the intermediate image 50.

[0017] The abnormality detection device 2000 calculates, for each captured image 10, difference data 40 that represents the difference between the captured image 10 and a processed image 30 generated from that captured image 10. Then, the abnormality detection device 2000 uses the calculated plurality of difference data 40 to perform a process of determining whether or not the captured image 10 represents an abnormal situation (hereinafter, referred to as an abnormality determination process).

[0018] Here, the abnormal situation represented by the captured image 10 is, for example, "a situation in which an object that is not normally present is present in the imaging range of the camera 20." An object that is not normally present is, for example, a foreign object such as a fallen object, or an abandoned object (hereinafter referred to as an abandoned object). Hereinafter, the imaging range of the camera 20 is also referred to as the "observation range."

[0019] For example, assume that the camera 20 is a camera used to monitor road conditions. In this case, a foreign object on the road is an object that does not normally exist in the road, which is the observation range. Therefore, if such a foreign object is captured in the captured image 10, the captured image 10 represents an abnormal situation.

[0020] Alternatively, for example, assume that the camera 20 is a camera used for monitoring facilities such as an airport. In this case, an abandoned object is an object that is not normally present within the observation range. Therefore, if such an abandoned object is captured in the captured image 10, the captured image 10 represents an abnormal situation.

[0021] <Example of Effects> According to the abnormality detection device 2000, processing including masking and restoration is performed on each of a plurality of captured images 10 in time series, thereby obtaining a processed image 30. Furthermore, for each of the plurality of captured images 10, difference data 40 is generated that represents the difference between the captured image 10 and the processed image 30 generated from that captured image 10. Then, by using the plurality of difference data 40, it is determined whether or not the captured image 10 represents an abnormal situation.

[0022] Here, a method may be considered in which whether or not a captured image 10 represents an abnormal situation is determined based only on the difference between one captured image 10 and a processed image 30 generated from that captured image 10. However, with this method, even if the difference between the captured image 10 and the processed image 30 is caused by the influence of temporary noise, the captured image 10 may be determined to represent an abnormal situation. In this regard, the abnormality detection device 2000 obtains multiple difference data 40 using multiple captured images 10 and detects an abnormality using these multiple difference data 40, thereby preventing erroneous detection of an abnormal situation due to the influence of temporary noise.

[0023] The abnormality detection device 2000 of this embodiment will be described in more detail below.

[0024] <Example of Functional Configuration> Fig. 3 is a block diagram illustrating an example of the functional configuration of the anomaly detection device 2000 of this embodiment. The anomaly detection device 2000 has a generation unit 2020, a calculation unit 2040, and a determination unit 2060. The generation unit 2020 generates a plurality of processed images 30 by performing processing on each of a plurality of captured images 10. The calculation unit 2040 calculates, for each of the plurality of captured images 10, difference data 40 representing the difference between the captured image 10 and the processed image 30 generated from that captured image 10. The determination unit 2060 determines whether the scene represented by the captured image 10 represents a predetermined situation based on the calculated plurality of difference data 40.

[0025] <Example of Hardware Configuration> Each functional component of the abnormality detection device 2000 may be realized by hardware that realizes the functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a case where each functional component of the abnormality detection device 2000 is realized by a combination of hardware and software will be further described.

[0026] 4 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the anomaly detection device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 1000 may be a portable computer such as a smartphone or a tablet terminal. The computer 1000 may be a dedicated computer designed to realize the anomaly detection device 2000, or may be a general-purpose computer.

[0027] For example, by installing a predetermined application on the computer 1000, each function of the anomaly detection device 2000 is realized on the computer 1000. The application is configured as a program for realizing each functional component of the anomaly detection device 2000. Note that any method for acquiring the program is possible. For example, the program can be acquired from a storage medium (such as a DVD disc or USB memory) on which the program is stored. Alternatively, the program can be acquired by downloading it from a server device that manages the storage device on which the program is stored.

[0028] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection.

[0029] The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, a read only memory (ROM), or the like.

[0030] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0031] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0032] The storage device 1080 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the error detection device 2000. The processor 1040 reads this program into the memory 1060 and executes it, thereby realizing each functional component of the error detection device 2000.

[0033] The anomaly detection device 2000 may be realized by one computer 1000, or may be realized by multiple computers 1000. In the latter case, the configurations of the computers 1000 do not need to be the same, and can be different from each other.

[0034] <Processing Flow> Fig. 5 is a flowchart illustrating the processing flow executed by the anomaly detection device 2000 of this embodiment. The generation unit 2020 acquires a plurality of captured images 10 (S102). The calculation unit 2040 generates a processed image 30 from each captured image 10 (S104). The calculation unit 2040 calculates difference data 40 for each captured image 10, representing the difference from the corresponding processed image 30 (S106). The determination unit 2060 uses the calculated plurality of difference data 40 to determine whether the scene represented by the captured images 10 represents a predetermined situation (S108).

[0035] <Acquisition of Captured Images 10: S102> The generation unit 2020 acquires the captured images 10 (S102). Here, various methods can be used to acquire the captured images generated by the camera. For example, the camera 20 stores each captured image 10 in a storage unit accessible by the anomaly detection device 2000. The generation unit 2020 acquires each captured image 10 from this storage unit. Alternatively, for example, the camera 20 may be configured to transmit the captured images 10 to the anomaly detection device 2000. In this case, the generation unit 2020 acquires the captured images 10 by receiving the captured images 10 transmitted from the camera 20.

[0036] Here, the generation unit 2020 may acquire the captured images 10 one by one, or may acquire a plurality of captured images 10 at once. In the latter case, for example, the generation unit 2020 periodically accesses a storage unit in which the captured images 10 are stored, and acquires all of the captured images 10 that have not yet been acquired.

[0037] <Generation of processed image 30: S104> The generation unit 2020 generates a processed image 30 from each captured image 10 (S104). As described above, the processing for generating the processed image 30 from the captured image 10 includes masking and restoration. The masking and restoration processes will be described below.

[0038] <<Mask Processing>> The generation unit 2020 masks one or more partial regions of the captured image 10. Hereinafter, the region to be masked will also be referred to as the mask target region. Various methods can be used to mask a specific region on an image. For example, the generation unit 2020 generates an intermediate image 50 in which each mask target region is masked by changing the value of each pixel in the mask target region in the captured image 10 to a predetermined value (e.g., 0 or 1). Note that a method for identifying the mask target region will be described later. Alternatively, for example, the generation unit 2020 may generate the intermediate image 50 by performing a process of overlaying the captured image 10 with a mask image in which the arrangement of the mask target region is determined. Here, existing technology can be used to perform mask processing on a specific image using a mask image. Note that the mask image will be described later.

[0039] Various methods can be used to identify the mask target region. For example, the generation unit 2020 randomly identifies one or more partial regions from the captured image 10 and treats each identified partial region as a mask target region. The number of mask target regions to be identified from one captured image 10, the shape of the mask target region, and the size of the mask target region are, for example, predetermined. Hereinafter, the number of mask target regions is denoted as Nm. In this case, the generation unit 2020 randomly identifies Nm positions from the captured image 10. Furthermore, for each of the identified Nm positions, the generation unit 2020 identifies a partial region of a predetermined shape and a predetermined size that uses that position as a reference position (e.g., the center or the upper left corner). Then, the generation unit 2020 treats each of the identified Nm partial regions as a mask target region.

[0040] Here, the multiple mask target regions may be identified so as to allow overlapping with each other, or so as not to overlap with each other. In the latter case, for example, the generation unit 2020 sequentially and randomly identifies Nm mask target regions. Then, if a newly identified mask target region overlaps with an already identified mask target region, the generation unit 2020 performs the process of randomly identifying the new mask target region again, so that the new mask target region does not overlap with the already identified mask target region.

[0041] Note that one or more of the number of mask target regions, the shape of the mask target region, and the size of the mask target region may be determined randomly.

[0042] Alternatively, the mask target region may be identified using a mask image that represents the layout of the mask target region. For example, in the mask image, the value of each pixel in the image region treated as the mask target region is 0, and the value of each pixel in the other image regions is 1.

[0043] 6 is a diagram illustrating an example of mask processing using a mask image. In Fig. 6, the area to be masked is represented by a dot pattern. In the mask image 70 in Fig. 6, the area to be masked is defined by a checkered pattern.

[0044] The mask image and the captured image 10 may have the same size or different sizes. In the latter case, the generation unit 2020 enlarges or reduces the mask image so that the size matches the size of the captured image 10.

[0045] The mask target area may be specified based on a predetermined rule (hereinafter referred to as a mask rule). For example, a rule such as "divide the captured image 10 vertically into Mv parts and horizontally into Mh parts, and set the partial areas of odd-numbered rows and columns as mask target areas, and the partial areas of even-numbered rows and columns as mask target areas" may be used. According to this rule, the arrangement of the mask target areas is represented by a checkerboard pattern.

[0046] The number of divisions in the vertical and horizontal directions may be predetermined or may be determined randomly. Alternatively, for example, the number of divisions in the vertical and horizontal directions may be determined based on the size of a predetermined partial region. Specifically, if the horizontal size of the captured image 10 is Wc and the horizontal size of the partial region is Wp, the number of divisions in the horizontal direction is the smallest integer equal to or greater than Wc / Wp. The number of divisions in the vertical direction can also be calculated using a similar method.

[0047] Note that instead of the above-mentioned rule that "subregions in odd-numbered rows and odd-numbered columns and subregions in even-numbered rows and even-numbered columns are treated as regions to be masked," a rule that "subregions in odd-numbered rows and even-numbered columns and subregions in even-numbered rows and odd-numbered columns are treated as regions to be masked" may be used. Furthermore, the rule for determining the regions to be masked is not limited to the rule representing a check pattern, and any rule may be used.

[0048] The generation unit 2020 may or may not make the mask target region common to all captured images 10. In the former case, for example, the generation unit 2020 performs mask processing on all captured images 10 using the same mask image or mask rule.

[0049] When the mask target region is not common to all captured images 10, for example, the generation unit 2020 randomly identifies the mask target region for each captured image 10. Alternatively, for example, the generation unit 2020 alternately applies two mask images or mask rules to the captured images 10 in time series.

[0050] 7 is a diagram illustrating a case where two mask images are applied alternately. Here, the reference number for the i-th captured image 10 in chronological order is represented as "10-i." In FIG. 7, mask processing using mask image 70-1 is performed on the captured images 10 that are even-numbered in chronological order. On the other hand, mask processing using mask image 70-2 is performed on the captured images 10 that are odd-numbered in chronological order.

[0051] Here, it is preferable that the two mask images 70-1 and 70-2 satisfy the following relationship: "A partial region treated as a mask target region in mask image 70-1 is not treated as a mask target region in mask image 70-2, and a partial region not treated as a mask target region in mask image 70-1 is treated as a mask target region in mask image 70-2." When the mask images are realized as binary images, mask image 70-2 is obtained by performing a process of inverting 0s and 1s on mask image 70-1. By ensuring that mask images 70-1 and 70-2 satisfy this relationship, the frequency with which each partial region of captured image 10 is masked can be made equal.

[0052] The number of mask images and mask rules is not limited to two and may be three or more. For example, when three mask images are used, a mask image M1 to be applied to the (3k-2)th captured image 10 in chronological order, a mask image M2 to be applied to the (3k-1)th captured image 10 in chronological order, and a mask image M3 to be applied to the 3kth captured image 10 in chronological order are prepared in advance, where k is a natural number.

[0053] The generation unit 2020 may detect an object that is normally included in the captured image 10 (in other words, captured by the camera 20) from the captured image 10, and may exclude an image region representing the object from the mask target region. Hereinafter, an object that is normally included in the captured image 10 will also be referred to as a "normal object." Furthermore, an image region representing a normal object will also be referred to as a "normal region."

[0054] For example, suppose that the camera 20 is a camera that monitors a road. In this case, a normal object would be a vehicle or the like. Alternatively, suppose that the camera 20 is a camera that monitors the inside of a facility used by people. In this case, a normal object would be a person or the like.

[0055] Here, existing technology can be used to detect a specific type of object from an image, and the type of object to be detected as a normal object is assumed to be predetermined.

[0056] 8 is a diagram illustrating a case where a normal region is excluded from the region to be masked. In this example, the normal object is a vehicle. Therefore, a normal region 82 representing the vehicle is detected from the captured image 10.

[0057] The generation unit 2020 generates a new mask image 90 by superimposing a mask image 70 prepared in advance on an image 80 representing the arrangement of normal regions 82. The region represented by the mask image 90 as a region to be masked is a region that is a region to be masked in the mask image 70 and is not included in the normal region 82. The generation unit 2020 performs mask processing on the captured image 10 using the mask image 90.

[0058] The method of excluding the normal region from the mask target region is not limited to the method using the mask image. For example, the generation unit 2020 may specify the mask target region by randomly specifying a partial region from the image region excluding the normal region.

[0059] Excluding the normal region from the region to be masked has the effect of reducing erroneous determinations by the determination unit 2060. When a restoration process is performed on the intermediate image 50 in which the normal region is masked, there is a possibility that the normal region cannot be accurately restored. If the normal region cannot be accurately restored, there is a possibility that the captured image 10 will be erroneously determined to represent an abnormal situation due to the difference in the normal region between the captured image 10 and the processed image 30. By excluding the normal region from the region to be masked, it is possible to prevent such erroneous determinations caused by the inability to correctly restore the masked normal region.

[0060] <<Regarding Restoration Processing>> The generation unit 2020 generates the processed image 30 by performing restoration processing on the intermediate image 50. The restoration processing is performed using, for example, a machine learning model such as a neural network. Hereinafter, the machine learning model used in the restoration processing will also be referred to as a restoration model. The restoration model is trained in advance so that, in response to an input of an image including a masked image region, it outputs an image in which the image region has been restored. The generation unit 2020 inputs the intermediate image 50 to the restoration model and uses the image output from the restoration model as the processed image 30.

[0061] The reconstruction model is trained using multiple training data. The training data includes a ground-truth image before masking and training input images obtained by masking one or more subregions in the ground-truth image. A device for training the reconstruction model (hereinafter referred to as a training device) inputs the training input images to the reconstruction model and calculates a loss based on the image output from the reconstruction model and the ground-truth image. The training device then trains the reconstruction model by updating trainable parameters of the reconstruction model based on the loss.

[0062] The training device may not include the normal region (region representing a normal object such as a vehicle) in the loss calculation. Specifically, when calculating the loss using the values ​​of each pixel in the training input image and the image output from the restoration model, the training device excludes the values ​​of each pixel included in the normal region from the calculation.

[0063] Here, the training input images used to train the restoration model may be images generated by the camera 20 or may be images generated by another camera.

[0064] For example, SimMIM disclosed in Non-Patent Document 1 can be used as the restoration model. However, the restoration model is not limited to SimMIM, and various machine learning models can be used.

[0065] <Calculation of Difference Data 40: S106> For each of the multiple captured images 10, the calculation unit 2040 calculates difference data 40 representing the difference between that captured image 10 and the processed image 30 generated from that captured image 10 (S106). There are various methods for calculating the difference data 40 representing the difference between the two images. For example, the calculation unit 2040 calculates the difference value of pixel values ​​between corresponding pixels between the captured image 10 and the processed image 30. Then, the calculation unit 2040 calculates the sum of the difference values ​​calculated for each pixel as the difference data 40.

[0066] Here, if the captured image 10 and the processed image 30 are single-channel images (e.g., grayscale images), the pixel value of each pixel is a scalar value, and therefore, the difference value between corresponding pixels can be obtained by calculating the difference between the two scalar values.

[0067] On the other hand, if the captured image 10 and the processed image 30 are multi-channel images (e.g., RGB images), the value of each pixel is a vector composed of values ​​for each channel. Therefore, the difference value between corresponding pixels can be obtained by calculating the norm of the two vectors. For example, suppose the pixel value at coordinates (x, y) in the captured image 10 is (r1, g1, b1), and the pixel value at coordinates (x, y) in the processed image 30 is (r2, g2, b2). In this case, the difference value calculated for the pixel at coordinates (x, y) is expressed as the norm of the vectors (r1, g1, b1) and (r2, g2, b2).

[0068] The difference data 40 between the captured image 10 and the processed image 30 is not limited to the sum of pixel values, but may be, for example, the average value of pixel values.

[0069] The calculation unit 2040 may exclude the normal region from the calculation of the difference data 40. In this way, the difference between the captured image 10 and the processed image 30 in the normal region is not taken into consideration in the determination by the determination unit 2060. Therefore, even if the masked normal region cannot be accurately restored, it is possible to accurately determine whether or not the captured image 10 represents an abnormal situation.

[0070] Note that, by excluding the normal region from the region to be masked or by excluding the normal region from the calculation target of the difference data 40, the difference between the captured image 10 and the processed image 30 in the normal region can be prevented from being taken into consideration in the determination by the determination unit 2060. Therefore, when the normal region is excluded from the region to be masked, it is not necessary to exclude the normal region from the calculation target of the difference data 40.

[0071] <Abnormality Detection: S108> The determination unit 2060 determines whether the captured images 10 represent an abnormal situation by using the difference data 40 calculated for each of the multiple captured images 10 (S108). To do this, for example, the determination unit 2060 calculates, for each of multiple time points, an index value that represents the degree of change in the situation before and after that time point. Hereinafter, this index value will also be referred to as the "degree of change in situation." Furthermore, when the degree of change in situation is calculated for a certain time point, that time point will also be referred to as the reference time point for that degree of change in situation.

[0072] The determination unit 2060 determines whether the degree of change in the situation calculated for each reference time point is equal to or greater than a threshold value. If the degree of change in the situation calculated for a certain reference time point is equal to or greater than the threshold value, the determination unit 2060 determines that the captured image 10 at that reference time point represents an abnormal situation.

[0073] The degree of change in the situation at a certain reference time point is calculated, for example, based on difference data 40 calculated for a first period of a predetermined length that includes a time point before the reference time point, and difference data 40 calculated for a second period of a predetermined length that includes a time point after the reference time point. The reference time point may be included in either the first period or the second period, or may be included in both.

[0074] For example, the degree of change in the situation is calculated by the following formula (1). Here, D(r) represents the degree of change in the situation at reference time r. In equation (1), the time points are represented by discrete values ​​(e.g., frame numbers) assigned to each captured image 10 in ascending order of the time points of generation. V1(r) represents the average value of the difference data 40 calculated for each of the (L1+1) captured images 10 included in a period of length L1 that ends at time r. V2(r) represents the average value of the difference data 40 calculated for each of the (L2+1) captured images 10 included in a period of length L2 that begins at time r. Si represents the difference data 40 calculated for the captured image 10 at time i.

[0075] L1 and L2 may be the same or different from each other. When L1=L2, the total value of the difference data 40 may be used for V1(r) and V2(r) instead of the average value of the difference data 40.

[0076] Here, V1(r) represents the magnitude of the difference between the captured image 10 and the processed image 30 in the near past, based on time r. On the other hand, V2(r) represents the magnitude of the difference between the captured image 10 and the processed image 30 in the near future, based on time r. Therefore, when V2(r) is sufficiently larger than V1(r), it indicates that the difference between the captured image 10 and the processed image 30 has increased at or around time r, and that this large difference continues. Therefore, when V2(r) is sufficiently larger than V1(r), it indicates that the situation represented by the captured image 10 (i.e., the situation of the location captured by the camera 20) has changed at or around time r.

[0077] For example, suppose a foreign object falls onto the road at a certain time point r. In this case, there is no foreign object on the road in the past closer to time point r, but there is a foreign object on the road in the future closer to time point r. Therefore, the difference between the captured image 10 and the processed image 30 is small in the period before time point r, but the difference between the captured image 10 and the processed image 30 is large in the period after time point r.

[0078] In another example, suppose that a package is left behind at time r. In this case, there is no left-behind object in the past closer to time r, but there is an left-behind object in the future closer to time r. Therefore, the difference between the captured image 10 and the processed image 30 is small in the period before time r, but the difference between the captured image 10 and the processed image 30 is large in the period after time r.

[0079] Therefore, by comparing the degree of change in the situation with a threshold, an abnormal situation can be detected.

[0080] Here, instead of using the integrated value of the difference data 40 at multiple points in time as shown in equation (1), another method is possible in which the difference data 40 at a reference point in time is compared with the difference data 40 at the point in time immediately before that point in time (e.g., determining whether S_i / S_(i-1) is equal to or greater than a threshold value). However, this method is susceptible to the influence of temporary noise. On the other hand, as shown in equation (1), the method of integrating and comparing the difference data 40 over a certain period of time before and after the reference point in time is less susceptible to the influence of temporary noise. Therefore, the above-described method makes it possible to accurately determine whether the captured image 10 represents an abnormal situation.

[0081] Here, the larger the value of L2, the length of the near future from the reference time point r, the longer the future situation will be considered. Therefore, it is preferable to make L2 large enough so that the observation range is not judged to be in an abnormal situation when the situation in the observation range has only changed for a short period of time.

[0082] For example, suppose that an abnormal situation in the observation range is the presence of an abandoned object. In this case, it is preferable that a situation in which a person temporarily places their belongings on the ground is not detected as an abnormal situation. Therefore, it is preferable to set the length L2 to an appropriate value so that an object temporarily placed on the ground can be distinguished from an abandoned object. Furthermore, the presence of an abandoned object can be more accurately detected by combining this method with other methods, such as analyzing the captured image 10 to associate people with objects.

[0083] <Output of Information Related to Anomaly Detection> It is preferable that the anomaly detection device 2000 outputs information related to the determination result by the determination unit 2060. Hereinafter, the functional configuration unit that outputs information related to the determination result will be referred to as an output unit. Fig. 9 is a diagram illustrating an example of the functional configuration of the anomaly detection device 2000 having the output unit 2080.

[0084] For example, the output unit 2080 displays on an arbitrary display device a screen on which the captured images 10 generated by the camera 20 are displayed in chronological order. Hereinafter, this screen will be referred to as an observation screen. If the camera 20 is a video camera, the observation screen displays video data generated by the camera 20.

[0085] The observation screen displays various information in addition to the captured image 10. For example, if the determination unit 2060 determines that the captured image 10 represents an abnormal situation (in other words, that the observation range is in an abnormal situation), some kind of display is added to the observation screen so that the user of the anomaly detection device 2000 can perceive the result of the determination. For example, the output unit 2080 includes on the observation screen a predetermined message or mark indicating that the observation range is in an abnormal situation. Hereinafter, the user of the anomaly detection device 2000 will also be simply referred to as the user.

[0086] 10 is a diagram illustrating an example of an observation screen including a display indicating that the observation range is in an abnormal state. The observation screen 100 includes an image display area 110. In the image display area 110, captured images 10 generated by the camera 20 are displayed in chronological order.

[0087] 10, the observation range is a road, and the abnormal situation is a situation in which a foreign object (e.g., a fallen object) exists on the road.

[0088] When a foreign object appears in the observation range, the degree of change in the situation, with the reference time being the time when the foreign object appeared or a time around that time, becomes equal to or exceeds the threshold. As a result, the determination unit 2060 determines that "the road being observed is in a situation where a foreign object is present." In response to this determination, the output unit 2080 displays a message 120 saying "foreign object present" on the observation screen 100.

[0089] The output unit 2080 may include on the observation screen a display indicating the position of an object (hereinafter, an abnormal object) that contributes to the observation range being in an abnormal state, such as the foreign object in the example of Fig. 10. For example, on the observation screen 100 of Fig. 10, a mark 130 indicating the position of the detected foreign object is displayed on the captured image 10.

[0090] In the captured image 10, an image region representing an abnormal object is an image region where there is a large difference between the captured image 10 and the processed image 30. As described above, when calculating the difference data 40 for each captured image 10, a difference value between corresponding pixels in that captured image 10 and the processed image 30 generated from that captured image 10 is calculated.

[0091] Therefore, the output unit 2080 identifies difference values ​​whose magnitude is equal to or greater than the threshold from the difference values ​​between pixels calculated for the captured image 10 in which the degree of situation change is equal to or greater than the threshold and the processed image 30 generated from the captured image 10. Then, the output unit 2080 identifies an image region made up of pixels whose difference values ​​are equal to or greater than the threshold as an image region representing an abnormal object (hereinafter referred to as an abnormal region).

[0092] Here, to prevent noise from being recognized as an abnormal region, a lower limit value for the size of the abnormal region may be set. In this case, the output unit 2080 identifies, as an abnormal region, an image region that is configured by pixels whose difference value between the captured image 10 and the processed image 30 is equal to or greater than a threshold value and whose size is equal to or greater than a predetermined lower limit value.

[0093] In order to allow the user to easily recognize an abnormal object, it is preferable that the output unit 2080 continues to display the message 120 or the mark 130 (hereinafter, referred to as the message 120, etc.) for a certain period of time on the observation screen 100. For example, in response to determining that the observation range is in an abnormal situation, the output unit 2080 displays the message 120, etc. on the observation screen 100 for a predetermined period of time.

[0094] Alternatively, for example, the output unit 2080 may determine whether to continue displaying the message 120 or the like on the observation screen 100 based on the degree of change in the situation. Specifically, when the output unit 2080 determines that the degree of change in the situation at a certain reference point in time is equal to or greater than a threshold, it identifies an abnormal area and records the association between the abnormal area and the degree of change in the situation. Furthermore, the output unit 2080 attenuates the degree of change in the situation corresponding to the abnormal area over time. The degree of change in the situation attenuated over time represents the degree to which it is necessary to continue to notify the message 120 or the like. Therefore, the degree of change in the situation attenuated over time is also expressed as the degree of notification necessity.

[0095] The output unit 2080 determines whether or not to include the message 120 or the like corresponding to the abnormal region on the observation screen 100, depending on the notification necessity level corresponding to the abnormal region. For example, the output unit 2080 displays the message 120 or the like corresponding to the abnormal region on the observation screen 100 while the notification necessity level corresponding to the abnormal region is equal to or greater than a threshold value.

[0096] Alternatively, for example, the output unit 2080 may compare the degree of notification necessity corresponding to the abnormal region with the newly calculated degree of change in the situation. If the degree of notification necessity corresponding to the abnormal region is greater than the newly calculated degree of change in the situation, the output unit 2080 displays the message 120 or the like corresponding to the abnormal region on the observation screen 100. On the other hand, if the degree of notification necessity corresponding to the abnormal region is equal to or less than the newly calculated degree of change in the situation, the output unit 2080 terminates the display of the message 120 or the like corresponding to the abnormal region.

[0097] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0098] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0099] (Supplementary Note 1) An anomaly detection device comprising: a generating means for generating a plurality of processed images by processing each of a plurality of captured images in time series; a calculating means for calculating, for each of the plurality of captured images, difference data representing the difference between the captured image and the processed image generated from the captured image; and a determining means for determining whether the captured image represents an abnormal situation based on the calculated plurality of difference data, wherein the processing performed on the captured image includes a masking process for masking one or more partial regions included in the captured image, and a restoration process for restoring the masked partial regions using data other than the partial regions. (Supplementary Note 2) The anomaly detection device according to Supplementary Note 1, wherein the calculating means detects, from the captured image, an image region representing a predetermined type of object whose inclusion in the captured image is not abnormal, and calculates the difference between the captured image and the processed image generated from the captured image for image regions excluding the detected image region, thereby calculating the difference data. (Supplementary Note 3) The anomaly detection device according to Supplementary Note 1, wherein the calculating means detects, from the captured image, an image region representing a predetermined type of object, and excludes the detected image region from being masked. (Supplementary Note 4) The anomaly detection device according to any one of Supplementary Notes 1 to 3, further comprising a restoration model trained to output an image in which a masked image region is restored in response to an input of an image including the masked image region, wherein the calculation means performs the restoration process by inputting the captured image in which the partial region is masked by the masking process to the restoration model, and wherein, in training the restoration model, image regions representing a predetermined type of object whose inclusion in the captured image is not abnormal are not included in the calculation of loss.(Supplementary Note 5) The anomaly detection device according to any one of Supplementary Notes 1 to 3, wherein the determination means calculates, for each of a plurality of reference time points, a situation change degree representing a ratio of a statistical value of the difference data calculated for each of a plurality of the captured images generated in a period after the reference time point to a statistical value of the difference data calculated for each of the plurality of captured images generated in a period before the reference time point, and determines that the captured image generated at the reference time point represents an abnormal situation if the situation change degree calculated for the reference time point is equal to or greater than a threshold. (Supplementary Note 6) The anomaly detection device according to Supplementary Note 5, further comprising: output means for outputting a screen including the captured image, wherein, after it is determined that the situation change degree calculated for the reference time point is equal to or greater than a threshold, the output means attenuates the situation change degree over time, and includes on the screen a display indicating that the captured image represents an abnormal situation while the attenuated situation change degree is equal to or greater than the situation change degree calculated for a newly acquired captured image. (Supplementary Note 7) An anomaly detection method executed by a computer, comprising: a generating step of generating a plurality of processed images by processing each of a plurality of captured images in time series; a calculating step of calculating, for each of the plurality of captured images, difference data representing the difference between the captured image and the processed image generated from the captured image; and a determining step of determining whether the captured image represents an abnormal situation based on the calculated plurality of difference data, wherein the processing performed on the captured image includes a masking process of masking one or more partial regions included in the captured image, and a restoration process of restoring the masked partial region using data other than the partial region. (Supplementary Note 8) The anomaly detection device according to Supplementary Note 7, wherein in the calculating step, an image region representing a predetermined type of object whose inclusion in the captured image is not abnormal is detected from the captured image, and the difference data is calculated by calculating the difference between the captured image and the processed image generated from the captured image for image regions excluding the detected image region.(Supplementary Note 9) The anomaly detection method according to Supplementary Note 7, wherein in the calculation step, an image region representing a predetermined type of object is detected from the captured image, and the detected image region is excluded from targets for masking. (Supplementary Note 10) The anomaly detection method according to any one of Supplementary Notes 7 to 9, wherein the computer has a restoration model trained to output an image in which the masked image region is restored in response to input of an image including the masked image region, and in the calculation step, the restoration process is performed by inputting the captured image in which the partial region is masked by the mask process to the restoration model, and in training the restoration model, image regions representing a predetermined type of object whose inclusion in the captured image is not abnormal are not included in targets for calculating a loss. (Supplementary Note 11) The anomaly detection method according to any one of Supplementary Notes 7 to 9, wherein in the determining step, for each of a plurality of reference time points, a situation change degree is calculated that represents a ratio of a statistical value of the difference data calculated for each of a plurality of the captured images generated in a period after the reference time point to a statistical value of the difference data calculated for each of the plurality of captured images generated in a period before the reference time point, and when the situation change degree calculated for the reference time point is equal to or greater than a threshold, the captured image generated at the reference time point is determined to represent an abnormal situation. (Supplementary Note 12) The anomaly detection method according to Supplementary Note 11, further comprising an output step of outputting a screen including the captured image, wherein in the output step, after it is determined that the situation change degree calculated for the reference time point is equal to or greater than a threshold, the situation change degree is attenuated over time, and a display indicating that the captured image represents an abnormal situation is included on the screen while the attenuated situation change degree is equal to or greater than the situation change degree calculated for a newly acquired captured image.(Supplementary Note 13) A program causing a computer to execute the following steps: generating a plurality of processed images by processing each of a plurality of captured images in time series; calculating, for each of the plurality of captured images, difference data representing the difference between the captured image and the processed image generated from the captured image; and determining whether the captured image represents an abnormal situation based on the calculated plurality of difference data; wherein the processing performed on the captured image includes a masking process that masks one or more partial regions included in the captured image and a restoration process that restores the masked partial regions using data other than the partial regions. (Supplementary Note 14) The program according to Supplementary Note 13, wherein, in the calculation step, an image region representing a predetermined type of object whose inclusion in the captured image is not abnormal is detected from the captured image, and the difference data is calculated by calculating the difference between the captured image and the processed image generated from the captured image for image regions excluding the detected image region. (Supplementary Note 15) The program according to Supplementary Note 13, wherein, in the calculation step, an image region representing a predetermined type of object is detected from the captured image, and the detected image region is excluded from being masked. (Supplementary Note 16) The program according to any one of Supplementary Notes 13 to 15, including a restoration model trained to output an image in which a masked image region is restored in response to input of an image including the masked image region, wherein in the calculation step, the restoration process is performed by inputting the captured image in which the partial region is masked by the masking process to the restoration model, and in training the restoration model, image regions representing a predetermined type of object that is not abnormal to be included in the captured image are not included in the calculation of loss.(Supplementary Note 17) The program according to any one of Supplementary Notes 13 to 15, wherein in the determining step, for each of a plurality of reference time points, a situation change degree is calculated that represents a ratio of a statistical value of the difference data calculated for each of a plurality of the captured images generated in a period after the reference time point to a statistical value of the difference data calculated for each of the plurality of captured images generated in a period before the reference time point, and when the situation change degree calculated for the reference time point is equal to or greater than a threshold, the captured image generated at the reference time point is determined to represent an abnormal situation. (Supplementary Note 18) The program according to Supplementary Note 17, wherein, after determining that the situation change degree calculated for the reference time point is equal to or greater than a threshold, the situation change degree is attenuated over time, and a display indicating that the captured image represents an abnormal situation is included on the screen while the attenuated situation change degree is equal to or greater than the situation change degree calculated for a newly acquired captured image.

[0100] This application claims priority based on Japanese Patent Application No. 2023-009536, filed on January 25, 2023, the disclosure of which is incorporated herein in its entirety by reference.

[0101] REFERENCE SIGNS LIST 10 Captured image 20 Camera 30 Processed image 40 Difference data 50 Intermediate image 60 Falling object 70 Mask image 80 Image 82 Normal area 90 Mask image 100 Observation screen 110 Image display area 120 Message 130 Mark 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface 2000 Anomaly detection device 2020 Generation unit 2040 Calculation unit 2060 Determination unit 2080 Output unit

Claims

1. a generating means for generating a plurality of processed images by processing each of a plurality of captured images in time series; a calculation means for calculating, for each of the plurality of captured images, difference data representing a difference between the captured image and the processed image generated from the captured image; and a determination means for determining whether or not the captured image represents an abnormal situation based on the calculated plurality of pieces of difference data, The processing performed on the captured image includes a masking process that masks one or more partial areas contained in the captured image, and a restoration process that restores the masked partial areas using data other than those partial areas.

2. The calculation means detecting an image area from the captured image that represents a predetermined type of object that is not abnormal to be included in the captured image; The anomaly detection device according to claim 1 , wherein the difference data is calculated by calculating a difference between the captured image and the processed image generated from the captured image for an image area excluding the detected image area.

3. The anomaly detection device according to claim 1 , wherein the calculation means detects an image area representing a predetermined type of object from the captured image, and excludes the detected image area from the target of masking.

4. a restoration model trained to output an image in which the masked image region is restored in response to an input of an image including the masked image region; the calculation means performs the restoration process by inputting the captured image in which the partial region is masked by the mask process into the restoration model; 4. The anomaly detection device according to claim 1, wherein, in training the restoration model, image regions representing predetermined types of objects that are not abnormal to be included in the captured images are not included in the calculation of loss.

5. The determination means calculating, for each of a plurality of reference time points, a situation change degree that represents a ratio of a statistical value of the difference data calculated for each of a plurality of the captured images generated in a period after the reference time point to a statistical value of the difference data calculated for each of the plurality of the captured images generated in a period before the reference time point; 4. The abnormality detection device according to claim 1, wherein if the degree of change in the situation calculated for the reference time point is equal to or greater than a threshold value, it is determined that the captured image generated at the reference time point represents an abnormal situation.

6. an output means for outputting a screen including the captured image; 6. The abnormality detection device according to claim 5, wherein, after it is determined that the degree of change in the situation calculated for the reference time point is equal to or greater than a threshold value, the output means attenuates the degree of change in the situation over time, and includes on the screen a display indicating that the captured image represents an abnormal situation while the attenuated degree of change in the situation is equal to or greater than the degree of change in the situation calculated for a newly acquired captured image.

7. a generating step of generating a plurality of processed images by processing each of the plurality of captured images in time series; a calculation step of calculating, for each of the plurality of captured images, difference data representing a difference between the captured image and the processed image generated from the captured image; and a determining step of determining whether or not the captured image represents an abnormal situation based on the calculated plurality of pieces of difference data, The anomaly detection method executed by a computer, wherein the processing performed on the captured image includes a masking process that masks one or more partial areas included in the captured image, and a restoration process that restores the masked partial areas using data other than those partial areas.

8. In the calculation step, detecting an image area from the captured image that represents a predetermined type of object that is not abnormal to be included in the captured image; The anomaly detection device according to claim 7 , wherein the difference data is calculated by calculating a difference between the captured image and the processed image generated from the captured image for an image area excluding the detected image area.

9. a generating step of generating a plurality of processed images by processing each of the plurality of captured images in time series; a calculation step of calculating, for each of the plurality of captured images, difference data representing a difference between the captured image and the processed image generated from the captured image; a determination step of determining whether or not the captured image represents an abnormal situation based on the calculated plurality of difference data; The processing performed on the captured image includes a masking process that masks one or more partial areas included in the captured image, and a restoration process that restores the masked partial areas using data other than those partial areas.

10. In the calculation step, detecting an image area from the captured image that represents a predetermined type of object that is not abnormal to be included in the captured image; The program according to claim 9 , wherein the difference data is calculated by calculating a difference between the captured image and the processed image generated from the captured image for an image area excluding the detected image area.

Images