Anomaly detection device

Abstract

An abnormality detection device according to the present invention generates a plurality of altered images by altering each of a plurality of captured images that are in a time sequence. Alteration of the captured images includes masking processing of masking one or more partial regions included in the captured images, and restoration processing of restoring the masked partial regions using data from other than the partial regions. For each of the plurality of captured images, the abnormality detection device calculates difference data representing difference between the captured image and the altered image generated from the captured image, and determines whether or not the captured images represent an abnormal state on the basis of a plurality of pieces of the calculated difference data.

Description

Technical Field

[0001] The present disclosure relates to an abnormality detection device, an abnormality detection method, and a program.

Background Art

[0002] Techniques for performing abnormality detection by image analysis have been developed. For example, Patent Document 1 discloses a technique for detecting defective products by image analysis. The system of Patent Document 1 masks a part of an image of an item such as a product, then restores the image, and compares the original image with the image obtained by restoration to detect a defect of the item.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Patent Document 1 does not assume that the situation of the object of anomaly detection will change. This disclosure has been made in view of the above problem, and one of its purposes is to provide a new technology for anomaly detection using images. [Means for solving the problem]

[0006] The anomaly detection device of this disclosure includes: generation means for generating a plurality of processed images by processing each of a plurality of time-series captured images; calculation means for calculating difference data representing the difference between each of the plurality of captured images and the processed image generated from that captured image; and determination means for determining whether the captured image represents an abnormal situation based on the calculated plurality of difference data. The processing performed on the captured image includes a masking process for masking one or more partial regions included in the captured image, and a restoration process for restoring the masked partial region using data other than that partial region.

[0007] The anomaly detection method of this disclosure is performed by a computer. The anomaly detection method includes a generation step of generating multiple processed images by processing each of a plurality of time-series captured images; a calculation step of calculating difference data for each of the plurality of captured images, representing the difference between the captured image and the processed image generated from that captured image; and a determination step of determining whether the captured image represents an abnormal situation based on the calculated plurality of difference data. The processing performed on the captured image includes a masking process of masking one or more partial regions included in the captured image, and a restoration process of restoring the masked partial region using data other than that partial region.

[0008] The program in this disclosure causes a computer to execute the anomaly detection method in this disclosure. [Effects of the Invention]

[0009] This disclosure provides a new technology for anomaly detection using images. [Brief explanation of the drawing]

[0010] [Figure 1] This diagram illustrates an example of an anomaly detection device. [Figure 2] This diagram illustrates an overview of the processing steps performed on captured images. [Figure 3] This is a block diagram illustrating the functional configuration of an anomaly detection device. [Figure 4] This is a block diagram illustrating the hardware configuration of a computer that implements an anomaly detection device. [Figure 5] This flowchart illustrates the processing flow performed by an anomaly detection device. [Figure 6] This diagram illustrates mask processing using a mask image. [Figure 7] This diagram illustrates a case where two mask images are applied alternately. [Figure 8] This diagram illustrates a case where a normal area is excluded from the area to be masked. [Figure 9] This diagram illustrates the functional configuration of an anomaly detection device having an output unit. [Figure 10] This diagram illustrates an observation screen that includes a display indicating that the observation area is in an abnormal state. [Modes for carrying out the invention]

[0011] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and redundant explanations are omitted as necessary for clarity. Unless otherwise specified, predetermined values ​​such as specified values ​​and thresholds are stored in advance in a storage device accessible from the device that uses those values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0012] <Overview> FIG. 1 is a diagram illustrating an overview of the abnormality detection apparatus 2000 according to an embodiment. Here, FIG. 1 is a diagram for facilitating understanding of the overview of the abnormality detection apparatus 2000, and the operation of the abnormality detection apparatus 2000 is not limited to that shown in FIG. 1.

[0013] The abnormality detection apparatus 2000 acquires a plurality of captured images 10 and determines whether the captured images 10 represent an abnormal situation. The plurality of captured images 10 acquired by the abnormality detection apparatus 2000 are time-series image data generated by the camera 20. For example, the camera 20 is configured to generate video data by repeatedly performing imaging. In this case, each captured image 10 is a video frame constituting the video data.

[0014] The abnormality detection apparatus 2000 generates a processed image 30 from the captured image 10 by performing a processing operation on each captured image 10. FIG. 2 is a diagram illustrating an overview of the processing operation performed on the captured image 10. The processing operation performed on the captured image 10 includes a masking process and a restoration process. The masking process is a process of generating an intermediate image 50 by masking at least one partial region included in the captured image 10. In FIG. 2, the masked image region is represented by a dot pattern. The restoration process is a process of generating the processed image 30 by restoring the masked partial region for the intermediate image 50 (that is, the captured image 10 in which the partial region is masked).

[0015] Here, the process of restoring the masked partial region in a certain intermediate image 50 is a process of estimating the content of the partial region by using the data of the image region other than the partial region included in the intermediate image 50. Therefore, some difference may occur between the processed image 30 and the captured image 10 that is the source of the processed image 30.

[0016] For example, in FIG. 2, the captured image 10 is an image obtained by capturing a road. The falling object 60 is captured in the captured image 10. In the intermediate image 50, the image area representing the falling object 60 is masked. Here, in the restoration process based on the unmasked image area, the falling object 60 is not restored. Therefore, the processed image 30 obtained by the restoration process does not include the falling object 60. From this, a difference occurs between the captured image 10 and the intermediate image 50.

[0017] The abnormality detection device 2000 calculates difference data 40 representing the difference between the processed image 30 generated from each captured image 10 and the captured image 10. Then, the abnormality detection device 2000 performs a process (hereinafter referred to as an abnormality determination process) of determining whether the captured image 10 represents an abnormal situation by using the calculated plurality of difference data 40.

[0018] Here, the abnormal situation represented by the captured image 10 is, for example, "a situation where an object that normally does not exist exists in the imaging range of the camera 20". Objects that normally do not exist are, for example, foreign objects such as falling objects and abandoned objects (hereinafter referred to as abandoned objects). Hereinafter, the imaging range of the camera 20 is also referred to as an "observation range".

[0019] For example, assume that the camera 20 is a camera used for monitoring the road condition. In this case, for example, foreign objects on the road are objects that normally do not exist on the road, which is the observation range. Therefore, when such foreign objects are captured in the captured image 10, the captured image 10 represents an abnormal situation.

[0020] In addition, for example, assume that the camera 20 is a camera used for monitoring inside a facility such as an airport. In this case, for example, an abandoned object is an object that normally does not exist in the observation range. Therefore, when such an abandoned object is captured in the captured image 10, the captured image 10 represents an abnormal situation.

[0021] <Examples of effects> According to the anomaly detection device 2000, a processing process including masking and restoration is performed on each of the multiple time-series captured images 10 to obtain a processed image 30. Furthermore, for each of the multiple captured images 10, difference data 40 is generated that represents the difference between that captured image 10 and the processed image 30 generated from it. Then, by using the multiple difference data 40, it is determined whether or not the captured image 10 represents an abnormal situation.

[0022] One possible method is to determine whether an image 10 represents an abnormal situation based solely on the difference between that image 10 and the processed image 30 generated from it. However, with this method, even if the difference between the image 10 and the processed image 30 is due to temporary noise, the image 10 may still be judged as representing an abnormal situation. In this respect, the anomaly detection device 2000 obtains multiple difference data 40 using multiple image 10s, and anomaly detection is performed using these multiple difference data 40, thus preventing false detection of an abnormal situation due to the influence of temporary noise.

[0023] The abnormality detection device 2000 of this embodiment will be described in more detail below.

[0024] <Example of functional configuration> Figure 3 is a block diagram illustrating the functional configuration of the anomaly detection device 2000 of this embodiment. The anomaly detection device 2000 includes a generation unit 2020, a calculation unit 2040, and a determination unit 2060. The generation unit 2020 generates multiple processed images 30 by performing processing on each of the multiple captured images 10. The calculation unit 2040 calculates difference data 40 for each of the multiple captured images 10, representing the difference between that captured image 10 and the processed image 30 generated from it. The determination unit 2060 determines, based on the calculated difference data 40, whether the scene represented by the captured image 10 represents a predetermined situation.

[0025] <Example of hardware configuration> Each functional component of the anomaly detection device 2000 may be implemented by hardware (e.g., hardwired electronic circuits) or by a combination of hardware and software (e.g., a combination of an electronic circuit and a program to control it). The following will further explain the case where each functional component of the anomaly detection device 2000 is implemented by a combination of hardware and software.

[0026] Figure 4 is a block diagram illustrating the hardware configuration of computer 1000, which implements the anomaly detection device 2000. Computer 1000 is any computer. For example, computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, computer 1000 could be a portable computer such as a smartphone or a tablet terminal. Computer 1000 may be a dedicated computer designed to implement the anomaly detection device 2000, or it may be a general-purpose computer.

[0027] For example, by installing a predetermined application on computer 1000, the various functions of the anomaly detection device 2000 are realized on computer 1000. The above application consists of programs for realizing each functional component of the anomaly detection device 2000. The method of obtaining the above program is arbitrary. For example, the program can be obtained from a storage medium (such as a DVD disk or USB memory) on which it is stored. Alternatively, the program can be obtained by downloading it from a server device that manages the storage device on which it is stored.

[0028] Computer 1000 includes a bus 1020, a processor 1040, memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 to send and receive data from each other. However, the method of connecting the processor 1040 and other components is not limited to bus connection.

[0029] The processor 1040 is a variety of processors such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or FPGA (Field-Programmable Gate Array). The memory 1060 is the main memory, implemented using RAM (Random Access Memory), etc. The storage device 1080 is the auxiliary storage, implemented using a hard disk, SSD (Solid State Drive), memory card, or ROM (Read Only Memory), etc.

[0030] The input / output interface 1100 is an interface for connecting the computer 1000 with input / output devices. For example, input devices such as keyboards and output devices such as display devices are connected to the input / output interface 1100.

[0031] The network interface 1120 is an interface for connecting computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0032] The storage device 1080 stores programs that implement each functional component of the anomaly detection device 2000 (programs that implement the aforementioned applications). The processor 1040 reads these programs into memory 1060 and executes them to implement each functional component of the anomaly detection device 2000.

[0033] The anomaly detection device 2000 may be implemented using one computer 1000 or multiple computers 1000. In the latter case, the configuration of each computer 1000 does not need to be the same and can be different.

[0034] <Processing flow> Figure 5 is a flowchart illustrating the processing flow performed by the anomaly detection device 2000 of this embodiment. The generation unit 2020 acquires multiple captured images 10 (S102). The calculation unit 2040 generates a processed image 30 from each captured image 10 (S104). The calculation unit 2040 calculates difference data 40 representing the difference between each captured image 10 and the corresponding processed image 30 (S106). The determination unit 2060 uses the calculated difference data 40 to determine whether the scene represented by the captured image 10 represents a predetermined situation (S108).

[0035] <Acquisition of captured image 10: S102> The generation unit 2020 acquires the captured image 10 (S102). Here, various methods can be used to acquire the captured image generated by the camera. For example, the camera 20 stores each captured image 10 in a storage unit accessible from the anomaly detection device 2000. The generation unit 2020 acquires each captured image 10 from this storage unit. Alternatively, for example, the camera 20 may be configured to transmit the captured image 10 to the anomaly detection device 2000. In this case, the generation unit 2020 acquires the captured image 10 by receiving the captured image 10 transmitted from the camera 20.

[0036] Here, the generation unit 2020 may acquire the captured images 10 one by one, or it may acquire multiple captured images 10 at once. In the latter case, for example, the generation unit 2020 periodically accesses the storage unit where the captured images 10 are stored and acquires the unacquired captured images 10 all at once.

[0037] <Generation of processed image 30: S104> The generation unit 2020 generates processed images 30 from each captured image 10 (S104). As mentioned above, the processing to generate processed images 30 from captured images 10 includes masking and restoration. The masking and restoration processes will be described below.

[0038] <<About masking>> The generation unit 2020 masks one or more sub-regions of the captured image 10. Hereafter, the region to be masked will also be referred to as the masked region. Various methods can be used to mask a specific region on the image. For example, the generation unit 2020 generates an intermediate image 50 in which each masked region is masked by changing the value of each pixel in the masked region of the captured image 10 to a predetermined value (e.g., 0 or 1). The method for identifying the masked region will be described later. Alternatively, for example, the generation unit 2020 may generate the intermediate image 50 by superimposing the captured image 10 with a mask image in which the arrangement of the masked regions is defined. Here, existing technologies can be used for the technique of performing masking on a specific image using a mask image. The mask image will be described later.

[0039] Various methods can be employed to identify the areas to be masked. For example, the generation unit 2020 randomly identifies one or more sub-regions from the captured image 10 and treats each identified sub-region as an area to be masked. The number of areas to be masked to be identified from a single captured image 10, the shape of the areas to be masked, and the size of the areas to be masked are predetermined, for example. Let the number of areas to be masked be Nm. In this case, the generation unit 2020 randomly identifies Nm positions from the captured image 10. Furthermore, for each of the Nm positions identified, the generation unit 2020 identifies a sub-region of a predetermined shape and size, with that position as the reference position (for example, the center or the upper left corner). The generation unit 2020 then treats each of the identified Nm sub-regions as an area to be masked.

[0040] Here, multiple mask target regions may be identified in a way that allows for overlap with each other, or they may be identified in a way that does not allow for overlap. In the latter case, for example, the generation unit 2020 identifies Nm mask target regions in order and randomly. Then, if the newly identified mask target region overlaps with an already identified mask target region, the generation unit 2020 performs the process of randomly identifying the new mask target region again so that the new mask target region does not overlap with an already identified mask target region.

[0041] Furthermore, one or more of the following may be determined randomly: the number of areas to be masked, the shape of the areas to be masked, and the size of the areas to be masked.

[0042] Alternatively, for example, the area to be masked may be identified using a mask image that shows the arrangement of the areas to be masked. For example, in the mask image, the value of each pixel in the image area to be treated as the area to be masked is 0, and the value of each pixel in the other image areas is 1.

[0043] Figure 6 illustrates masking using a mask image. In Figure 6, the area to be masked is represented by a dot pattern. In the mask image 70 of Figure 6, the area to be masked is defined by a checkered pattern.

[0044] The sizes of the mask image and the captured image 10 may be the same or different. In the latter case, the generation unit 2020 enlarges or reduces the mask image so that its size matches that of the captured image 10.

[0045] The areas to be masked may be identified based on a predetermined rule (hereinafter referred to as the mask rule). For example, a rule such as "divide the captured image 10 vertically into Mv sections and horizontally into Mh sections, and in odd-numbered rows, the sub-regions of odd-numbered columns will be the areas to be masked, and in even-numbered rows, the sub-regions of even-numbered columns will be the areas to be masked" may be used. According to this rule, the arrangement of the areas to be masked is represented by a check pattern.

[0046] The number of divisions in the vertical and horizontal directions may be predetermined or determined randomly. Alternatively, the number of divisions in the vertical and horizontal directions may be determined based on the size of a predetermined sub-region. Specifically, if the horizontal size of the captured image 10 is Wc and the horizontal size of the sub-region is Wp, then the number of divisions in the horizontal direction will be the smallest integer greater than or equal to Wc / Wp. The number of divisions in the vertical direction can be calculated in a similar manner.

[0047] Furthermore, instead of the aforementioned rule "treat the subregions with odd rows and odd columns and the subregions with even rows and even columns as the areas to be masked," the rule "treat the subregions with odd rows and even columns and the subregions with even rows and odd columns as the areas to be masked" may be used. Also, the rule that defines the areas to be masked is not limited to the rule that represents the check pattern, but can be any rule.

[0048] The generation unit 2020 may or may not use the same mask target area for all captured images 10. In the former case, for example, the generation unit 2020 performs masking on all captured images 10 using the same mask image or masking rule.

[0049] If the mask target area is not the same for all captured images 10, for example, the generation unit 2020 randomly identifies the mask target area for each captured image 10. Alternatively, for example, the generation unit 2020 may alternately apply two mask images or mask rules to the time-series captured images 10.

[0050] Figure 7 illustrates a case where two mask images are applied alternately. Here, the code for the i-th captured image 10 in chronological order is represented as "10-i". In Figure 7, masking is performed using mask image 70-1 for even-numbered captured images 10 in chronological order. On the other hand, masking is performed using mask image 70-2 for odd-numbered captured images 10 in chronological order.

[0051] Here, it is preferable that the two mask images 70-1 and 70-2 satisfy the relationship that "a subregion treated as a mask target region in mask image 70-1 is not treated as a mask target region in mask image 70-2, and a subregion not treated as a mask target region in mask image 70-1 is treated as a mask target region in mask image 70-2." When the mask image is implemented as a binary image, mask image 70-2 is obtained by performing a process of inverting 0s and 1s in mask image 70-1. 7 By ensuring that 0-2 satisfies this relationship, the frequency with which each subregion of the captured image 10 is masked can be made equal.

[0052] Note that the number of mask images and mask rules is not limited to two; there may be three or more. For example, if three mask images are used, mask image M1 is applied to the (3k-2)th captured image 10 in chronological order, mask image M2 is applied to the (3k-1)th captured image 10 in chronological order, and mask image M3 is applied to the 3kth captured image 10 in chronological order. Here, k is a natural number.

[0053] The generation unit 2020 may detect objects that are normally included in the captured image 10 (in other words, that are normally captured by the camera 20) and exclude the image region representing such objects from the mask target region. Hereinafter, objects that are normally included in the captured image 10 will also be referred to as "normal objects." The image region representing a normal object will also be referred to as a "normal region."

[0054] For example, suppose camera 20 is a camera that monitors roads. In this case, normal objects would be vehicles, etc. Alternatively, suppose camera 20 is a camera that monitors the inside of a facility used by people. In this case, normal objects would be people, etc.

[0055] Here, existing technologies can be used to detect specific types of objects from images. It should be assumed that the types of objects to be detected as normal are predetermined.

[0056] Figure 8 illustrates a case where a normal area is excluded from the masked area. In this example, the normal object is a vehicle. Therefore, a normal area 82 representing the vehicle is detected from the captured image 10.

[0057] The generation unit 2020 generates a new mask image 90 by superimposing a pre-prepared mask image 70 with an image 80 representing the arrangement of the normal region 82. The region represented as the mask target region in the mask image 90 is the region that is the mask target region in the mask image 70 and is not included in the normal region 82. The generation unit 2020 uses the mask image 90 to perform masking on the captured image 10.

[0058] The method for excluding normal areas from the masked area is not limited to using a mask image. For example, the generation unit 2020 may identify the masked area by randomly selecting a sub-region from the image area excluding the normal area.

[0059] Excluding normal areas from the masked area has the effect of reducing erroneous judgments by the determination unit 2060. When restoration processing is performed on the intermediate image 50 in which the normal area has been masked, there is a possibility that the normal area cannot be accurately restored. If the normal area cannot be accurately restored, there is a possibility that the captured image 10 may be incorrectly judged to represent an abnormal situation due to the difference in the normal area between the captured image 10 and the processed image 30. By excluding normal areas from the masked area, it is possible to prevent such erroneous judgments caused by the inability to correctly restore the masked normal boundary.

[0060] <<Regarding the recovery process>> The generation unit 2020 generates a processed image 30 by performing a restoration process on the intermediate image 50. The restoration process is performed using a machine learning model, such as a neural network. Hereinafter, the machine learning model used for the restoration process will also be referred to as the restoration model. The restoration model is pre-trained to output an image in which a masked image region has been restored, in response to an image containing that masked image region being input. The generation unit 2020 inputs the intermediate image 50 to the restoration model and uses the image output from the restoration model as the processed image 30.

[0061] The reconstruction model is trained using multiple training data sets. The training data includes the ground-truth image before masking and the training input image obtained by masking one or more subregions in the ground-truth image. The device that trains the reconstruction model (hereinafter referred to as the training device) inputs the training input image into the reconstruction model and calculates a loss based on the image output from the reconstruction model and the ground-truth image. The training device then trains the reconstruction model by updating the trainable parameters of the reconstruction model based on this loss.

[0062] Furthermore, the training device does not need to include the normal region (the region representing normal objects such as vehicles) in the loss calculation. Specifically, when the training device calculates the loss using the values ​​of each pixel in the training input image and the image output from the reconstruction model, it excludes the values ​​of each pixel included in the normal region from the calculation.

[0063] Here, the training input images used to train the reconstruction model may be images generated by camera 20 or images generated by other cameras.

[0064] For the reconstruction model, for example, SimMIM disclosed in Non-Patent Document 1 can be used. However, the reconstruction model is not limited to SimMIM, and various machine learning models can be used.

[0065] <Calculation of difference data 40: S106> The calculation unit 2040 calculates difference data 40 for each of the multiple captured images 10, representing the difference between the captured image 10 and the processed image 30 generated from the captured image 10 (S106). There are various methods for calculating the difference data 40 that represents the difference between the two images. For example, the calculation unit 2040 calculates the difference in pixel values ​​between the captured image 10 and the processed image 30 for each corresponding pixel. Then, the calculation unit 2040 calculates the sum of the difference values ​​calculated for each pixel as the difference data 40.

[0066] Here, if the captured image 10 and the processed image 30 are single-channel images (for example, grayscale images), the pixel value of each pixel is a scalar value. Therefore, the difference between corresponding pixels can be obtained by calculating the difference between the two scalar values.

[0067] On the other hand, if the captured image 10 and the processed image 30 are multi-channel images (for example, RGB images), the value of each pixel is a vector composed of values ​​for each channel. Therefore, the difference between corresponding pixels can be obtained by calculating the norm of the two vectors. For example, suppose the pixel value at coordinate (x,y) in the captured image 10 is (r1,g1,b1), and the pixel value at coordinate (x,y) in the processed image 30 is (r2,g2,b2). In this case, the difference value calculated for the pixel at coordinate (x,y) is expressed as the norm of the vectors (r1,g1,b1) and (r2,g2,b2).

[0068] The difference data 40 between the captured image 10 and the processed image 30 is not limited to the sum of pixel values. For example, the difference data 40 between the captured image 10 and the processed image 30 may be the average value of the pixel values.

[0069] The calculation unit 2040 may exclude the aforementioned normal region from the calculation of the difference data 40. In this way, the difference between the captured image 10 and the processed image 30 in the normal region is not considered in the determination by the determination unit 2060. Therefore, even if the masked normal region cannot be accurately restored, it is possible to accurately determine whether or not the captured image 10 represents an abnormal situation.

[0070] Furthermore, by either excluding the normal region from the mask target region or excluding the normal region from the calculation of difference data 40, the difference between the captured image 10 and the processed image 30 in the normal region will not be considered in the determination by the determination unit 2060. Therefore, if the normal region is excluded from the mask target region, it is not necessary to exclude the normal region from the calculation of difference data 40.

[0071] <Anomaly detected: S108> The determination unit 2060 uses the difference data 40 calculated for each of the multiple captured images 10 to determine whether or not the captured image 10 represents an abnormal situation (S108). To this end, for example, the determination unit 2060 calculates an index value for each of the multiple time points that represents the degree of change in the situation before and after that time point. Hereinafter, this index value will also be referred to as the "degree of change in situation". Furthermore, when the degree of change in situation is calculated for a certain time point, that time point will also be referred to as the reference time point for that degree of change in situation.

[0072] The determination unit 2060 determines whether the degree of change in the situation calculated for each reference time point is equal to or greater than a threshold. If the degree of change in the situation calculated for a certain reference time point is equal to or greater than a threshold, the determination unit 2060 determines that the captured image 10 at that reference time point represents an abnormal situation.

[0073] The degree of change in the situation at a certain reference point is calculated, for example, based on differential data 40 calculated for a first period of predetermined length that includes time points before the reference point, and differential data 40 calculated for a second period of predetermined length that includes time points after the reference point. The reference point may be included in either the first period or the second period, or in both.

[0074] For example, the degree of change in the situation is calculated using the following formula (1).

number

[0075] L1 and L2 may be the same or different. If L1=L2, V1(r) and V2(r) may use the sum of the difference data 40 instead of the mean of the difference data 40.

[0076] Here, V1(r) represents the magnitude of the difference between the captured image 10 and the processed image 30 in the near past, with respect to time r. On the other hand, V2(r) represents the magnitude of the difference between the captured image 10 and the processed image 30 in the near future, with respect to time r. Therefore, if V2(r) is sufficiently larger than V1(r), it indicates that the difference between the captured image 10 and the processed image 30 has increased, and that this large difference continues, with respect to time r or a time around it. Thus, if V2(r) is sufficiently larger than V1(r), it indicates that the situation represented by the captured image 10 (i.e., the situation of the location being captured by the camera 20) has changed with respect to time r or a time around it.

[0077] For example, suppose a foreign object falls onto the road at a certain point in time r. In this case, the foreign object does not exist on the road in the near past than point in time r, but it does exist on the road in the near future than point in time r. Therefore, the difference between the captured image 10 and the processed image 30 is small in the period before point in time r, while the difference between the captured image 10 and the processed image 30 is large in the period after point in time r.

[0078] For example, suppose a package is left behind at time r. In this case, the abandoned object does not exist in the past more than time r, but it does exist in the future more than time r. Therefore, the difference between the captured image 10 and the processed image 30 is small in the period before time r, while the difference between the captured image 10 and the processed image 30 is large in the period after time r.

[0079] Therefore, by comparing the degree of change in the situation with a threshold, it is possible to detect abnormal situations.

[0080] Here, instead of using the integrated value of the difference data 40 at multiple time points as shown in equation (1), another method is to compare the difference data 40 at the reference time with the difference data 40 at the immediately preceding time point (for example, determining whether S_i / S_(i-1) is greater than or equal to a threshold). However, this method is susceptible to the influence of temporarily occurring noise. On the other hand, as shown in equation (1), if the difference data 40 over a period of a certain length before and after the reference time is integrated and compared, it becomes less susceptible to the influence of temporary noise. Therefore, using the method described above, it is possible to accurately determine whether the captured image 10 represents an abnormal situation.

[0081] Here, the larger the value of L2, which represents the length of the near future from the reference time r, the longer the future situation is considered. Therefore, in order to prevent the observation range from being judged as having an abnormal situation when the situation within the observation range has only changed for a short period of time, it is preferable to make the magnitude of L2 sufficiently large.

[0082] For example, suppose we consider the situation of "an abandoned object being present" as an abnormal situation within the observation range. In this case, it is preferable that the situation of "a person temporarily placing their belongings on the ground" is not detected as an abnormal situation. Therefore, it is preferable to set the length L2 to an appropriate value so that objects temporarily placed on the ground and abandoned objects can be distinguished. Furthermore, by using this method in combination with other techniques, such as analyzing the captured image 10 to associate people with objects, the presence of abandoned objects can be detected more accurately.

[0083] <Output of information regarding anomaly detection> The anomaly detection device 2000 preferably outputs information related to the determination result made by the determination unit 2060. Hereinafter, the functional component that outputs information related to the determination result will be referred to as the output unit. Figure 9 is a diagram illustrating the functional configuration of the anomaly detection device 2000 having an output unit 2080.

[0084] For example, the output unit 2080 causes a screen in which the captured images 10 generated by the camera 20 are displayed sequentially in chronological order on any display device. Hereinafter, this screen will be referred to as the observation screen. If the camera 20 is a video camera, the observation screen will display the video data generated by the camera 20.

[0085] The observation screen displays various information in addition to the captured image 10. For example, if the determination unit 2060 determines that the captured image 10 represents an abnormal situation (in other words, that the observation range is in an abnormal situation), some kind of display is added to the observation screen so that the user of the anomaly detection device 2000 can perceive the result of the determination. For example, the output unit 2080 includes a predetermined message or mark on the observation screen that indicates that the observation range is in an abnormal situation. Hereinafter, the user of the anomaly detection device 2000 will also be simply referred to as the user.

[0086] Figure 10 illustrates an observation screen that includes a display indicating that the observation area is in an abnormal state. The observation screen 100 includes an image display area 110. The image display area 110 displays the captured images 10 generated by the camera 20 in chronological order.

[0087] In the example shown in Figure 10, the observation area is the road. The abnormal situation is the presence of foreign objects (e.g., fallen objects) on the road.

[0088] When a foreign object appears within the observation area, the degree of change in the situation, using the time of the foreign object's appearance or a time in the vicinity as the reference point, exceeds a threshold. As a result, the determination unit 2060 determines that "the road under observation is in a state where a foreign object is present." In response to this determination, the output unit 2080 displays the message 120 "Foreign object present" on the observation screen 100.

[0089] The output unit 2080 may include a display on the observation screen that indicates the location of an object (hereinafter referred to as an abnormal object) that contributes to an abnormal situation in the observation range, such as a foreign object in the example of Figure 10. For example, in the observation screen 100 of Figure 10, a mark 130 indicating the location of the detected foreign object is displayed on the captured image 10.

[0090] In the captured image 10, the image region representing an abnormal object is the image region with a large difference between the captured image 10 and the processed image 30. As mentioned above, when calculating the difference data 40 for each captured image 10, the difference value between corresponding pixels is calculated between the captured image 10 and the processed image 30 generated from that captured image 10.

[0091] The output unit 2080 then identifies the difference values ​​between pixels that are greater than or equal to the threshold value from among the difference values ​​between the captured image 10 in which the degree of change in the situation is greater than or equal to the threshold value and the processed image 30 generated from the captured image 10. The output unit 2080 then identifies the image region composed of each pixel whose difference value is greater than or equal to the threshold value as the image region representing an abnormal object (hereinafter referred to as the abnormal region).

[0092] Here, in order to prevent noise from being recognized as an abnormal region, a lower limit value for the size of the abnormal region may be defined. In this case, the output unit 2080 identifies an image region as an abnormal region if it is an image region composed of each pixel whose difference value between the captured image 10 and the processed image 30 is greater than or equal to a threshold value, and whose size is greater than or equal to a predetermined lower limit value.

[0093] To enable users to easily perceive abnormal objects, it is preferable that the output unit 2080 continues to display the message 120 and mark 130 (hereinafter referred to as message 120, etc.) on the observation screen 100 for a certain period of time. For example, the output unit 2080 displays the message 120, etc. on the observation screen 100 for a predetermined time in response to the determination that the observation range is in an abnormal state.

[0094] In addition, for example, the output unit 2080 may decide whether or not to continue displaying the message 120, etc., on the observation screen 100 based on the degree of change in the situation. Specifically, if the output unit 2080 determines that the degree of change in the situation at a certain reference point is above a threshold, it identifies an abnormal area and records the correspondence between that abnormal area and the degree of change in the situation. Furthermore, the output unit 2080 gradually reduces the degree of change in the situation corresponding to the abnormal area over time. The degree of change in the situation that has been reduced over time represents the degree to which it is necessary to continue sending notifications such as the message 120. Therefore, the degree of change in the situation that has been reduced over time can also be expressed as the notification necessity level.

[0095] The output unit 2080 determines whether or not to include the message 120, etc., corresponding to the abnormal area on the observation screen 100, according to the notification necessity level for that abnormal area. For example, the output unit 2080 displays the message 120, etc., corresponding to the abnormal area on the observation screen 100 as long as the notification necessity level for that abnormal area is above a threshold.

[0096] In addition, for example, the output unit 2080 may compare the notification necessity level corresponding to the abnormal area with a newly calculated status change level. If the notification necessity level corresponding to the abnormal area is greater than the newly calculated status change level, the output unit 2080 displays the message 120, etc., corresponding to that abnormal area on the observation screen 100. On the other hand, if the notification necessity level corresponding to the abnormal area is less than or equal to the newly calculated status change level, the output unit 2080 terminates the display of the message 120, etc., corresponding to that abnormal area.

[0097] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made, which can be understood by those skilled in the art within the scope of the present invention.

[0098] In the above examples, the program includes a set of instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more of the functions described in the embodiments. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrical, optical, acoustic or other forms of propagating signals.

[0099] (Note 1) A generation means that generates multiple processed images by processing each of multiple time-series captured images, A calculation means for calculating difference data representing the difference between each of the multiple captured images and the processed image generated from that captured image, The system includes a determination means for determining whether the captured image represents an abnormal situation based on a plurality of calculated difference data, An anomaly detection device, wherein the processing performed on the captured image includes a masking process that masks one or more subregions included in the captured image, and a restoration process that restores the masked subregions using data other than those subregions. (Note 2) The calculation means is From the captured image, an image region representing a predetermined type of object that is not abnormal to be included in the captured image is detected. An anomaly detection device according to Appendix 1, which calculates the difference data by calculating the difference between the captured image and the processed image generated from the captured image for the image region excluding the detected image region. (Note 3) The anomaly detection device according to Appendix 1, wherein the calculation means detects an image region representing a predetermined type of object from the captured image and excludes the detected image region from the target of the mask. (Note 4) The system has a reconstruction model that, upon receiving an image containing a masked image region, outputs an image in which the masked image region has been reconstructed. The calculation means performs the restoration process by inputting the captured image, in which the partial region has been masked by the masking process, into the restoration model. An anomaly detection device according to any one of the appendices 1 to 3, wherein, in training the reconstruction model, image regions representing a predetermined type of object that is not an anomaly when included in the captured image are not included in the loss calculation. (Note 5) The determination means is, For each of the multiple reference time points, a degree of change in the situation is calculated, which represents the ratio of the statistical value of the difference data calculated for each of the multiple captured images generated during the period after the reference time point to the statistical value of the difference data calculated for each of the multiple captured images generated during the period before the reference time point. An anomaly detection device according to any one of the appendices 1 to 3, which determines that the captured image generated at the reference time represents an abnormal situation if the degree of change in the situation calculated for the reference time is equal to or greater than a threshold. (Note 6) The system has output means that outputs a screen containing the aforementioned captured image, An anomaly detection device as described in Appendix 5, wherein the output means, after it is determined that the degree of change in the situation calculated for the reference time is equal to or greater than a threshold, attenuates the degree of change in the situation over time, and while the attenuated degree of change in the situation is equal to or greater than the degree of change in the situation calculated for the newly acquired captured image, includes a display on the screen indicating that the captured image represents an abnormal situation. (Note 7) A generation step that generates multiple processed images by processing each of multiple time-series captured images, A calculation step for each of the multiple captured images, which involves calculating difference data representing the difference between the captured image and the processed image generated from that captured image, The system includes a determination step of determining whether the captured image represents an abnormal situation based on a plurality of calculated difference data, An anomaly detection method performed by a computer, wherein the processing performed on the captured image includes a masking process that masks one or more subregions included in the captured image, and a restoration process that restores the masked subregions using data other than those subregions. (Note 8) In the calculation step, From the captured image, an image region representing a predetermined type of object that is not abnormal to be included in the captured image is detected. An anomaly detection device according to Appendix 7, which calculates the difference data by calculating the difference between the captured image and the processed image generated from the captured image for the image region excluding the detected image region. (Note 9) The anomaly detection method according to Appendix 7, wherein in the calculation step, an image region representing a predetermined type of object is detected from the captured image, and the detected image region is excluded from the target of the mask. (Note 10) The computer has a restoration model that, upon receiving an image containing a masked image region, outputs an image in which the masked image region has been restored. In the calculation step, the restoration process is performed by inputting the captured image, in which the partial region has been masked by the masking process, into the restoration model. An anomaly detection method according to any one of the appendices 7 to 9, wherein, in training the reconstruction model, image regions representing a predetermined type of object that is not an anomaly when included in the captured image are not included in the loss calculation. (Note 11) In the determination step, For each of the multiple reference time points, a degree of change in the situation is calculated, which represents the ratio of the statistical value of the difference data calculated for each of the multiple captured images generated during the period after the reference time point to the statistical value of the difference data calculated for each of the multiple captured images generated during the period before the reference time point. An anomaly detection method according to any one of the appendices 7 to 9, wherein if the degree of change in the situation calculated for the aforementioned reference time is equal to or greater than a threshold, it is determined that the captured image generated at that reference time represents an abnormal situation. (Note 12) The system has an output step that outputs a screen containing the aforementioned captured image, An anomaly detection method according to Appendix 11, wherein, in the output step, after it is determined that the degree of change in the situation calculated for the reference time is equal to or greater than a threshold, the degree of change in the situation is attenuated over time, and while the attenuated degree of change in the situation is equal to or greater than the degree of change in the situation calculated for the newly acquired captured image, a display indicating that the captured image represents an abnormal situation is included on the screen. (Note 13) A generation step that generates multiple processed images by processing each of multiple time-series captured images, A calculation step for each of the multiple captured images, which involves calculating difference data representing the difference between the captured image and the processed image generated from that captured image, The computer is instructed to perform a determination step of determining whether the captured image represents an abnormal situation based on the multiple difference data calculated, The processing performed on the captured image is a program that includes a masking process for masking one or more subregions included in the captured image, and a restoration process for restoring the masked subregions using data other than those subregions. (Note 14) In the calculation step, From the captured image, an image region representing a predetermined type of object that is not abnormal to be included in the captured image is detected. The program described in Appendix 13, which calculates the difference data by calculating the difference between the captured image and the processed image generated from the captured image for the image region excluding the detected image region. (Note 15) The program described in Appendix 13, wherein in the calculation step, it detects an image region representing a predetermined type of object from the captured image and excludes the detected image region from the mask. (Note 16) The system includes a reconstruction model trained to output an image in which the masked image region is reconstructed, in response to an input image containing the masked image region. In the calculation step, the restoration process is performed by inputting the captured image, in which the partial region has been masked by the masking process, into the restoration model. In training the reconstruction model, image regions representing a predetermined type of object that is not abnormal to be included in the captured image are not included in the loss calculation, as described in any one of the appendices 13 to 15 of the program. (Note 17) In the determination step, For each of the multiple reference time points, a degree of change in the situation is calculated, which represents the ratio of the statistical value of the difference data calculated for each of the multiple captured images generated during the period after the reference time point to the statistical value of the difference data calculated for each of the multiple captured images generated during the period before the reference time point. A program according to any one of the appendices 13 to 15, which determines that the captured image generated at the reference time represents an abnormal situation if the degree of change in the situation calculated for the reference time is equal to or greater than a threshold. (Note 18) The computer is instructed to perform an output step that outputs a screen containing the aforementioned captured image. The program as described in Appendix 17, wherein in the output step, after it is determined that the degree of change in the situation calculated for the reference time is equal to or greater than a threshold, the degree of change in the situation is attenuated over time, and while the attenuated degree of change in the situation is equal to or greater than the degree of change in the situation calculated for the newly acquired captured image, a display indicating that the captured image represents an abnormal situation is included on the screen.

[0100] This application claims priority based on Japanese Patent Application No. 2023-009536, filed on 25 January 2023, and incorporates all of its disclosures herein. [Explanation of symbols]

[0101] 10 Acquired images 20 cameras 30 processed images 40 Differential data 50 Intermediate Images 60 Falling objects 70 Mask Images 80 images 82 Normal area 90 Mask Images 100 Observation screen 110 Image display area 120 messages 130 marks 1000 computers 1020 Bus 1040 processor 1060 memory 1080 Storage Devices 1100 Input / Output Interface 1120 Network Interface 2000 Anomaly detection device 2020 Generation Department 2040 Calculation Unit 2060 Judgment section 2080 Output Section

Claims

1. A generation means that generates multiple processed images by processing each of multiple time-series captured images, A calculation means for calculating difference data representing the difference between each of the multiple captured images and the processed image generated from that captured image, The system includes a determination means for determining whether the captured image represents an abnormal situation based on a plurality of calculated difference data, The processing performed on the captured image includes a masking process that masks one or more subregions included in the captured image, and a restoration process that restores the masked subregions using data other than those subregions. The calculation means is From the captured image, an image region representing a predetermined type of object that is not abnormal to be included in the captured image is detected. An anomaly detection device that calculates difference data by calculating the difference between the captured image and the processed image generated from the captured image for the image region excluding the detected image region.

2. A generation means for generating multiple processed images by processing each of multiple time-series captured images, A calculation means for calculating difference data representing the difference between each of the multiple captured images and the processed image generated from that captured image, The system includes a determination means for determining whether the captured image represents an abnormal situation based on a plurality of calculated difference data, The processing performed on the captured image includes a masking process that masks one or more subregions included in the captured image, and a restoration process that restores the masked subregions using data other than those subregions. The calculation means is an anomaly detection device that detects an image region representing a predetermined type of object from the captured image and excludes the detected image region from the target of the mask.

3. A generation means for generating multiple processed images by processing each of multiple time-series captured images, A calculation means for calculating difference data representing the difference between each of the multiple captured images and the processed image generated from that captured image, The system includes a determination means for determining whether the captured image represents an abnormal situation based on a plurality of calculated difference data, The processing performed on the captured image includes a masking process that masks one or more subregions included in the captured image, and a restoration process that restores the masked subregions using data other than those subregions. The system has a reconstruction model that, upon receiving an image containing a masked image region, outputs an image in which the masked image region has been reconstructed. The calculation means performs the restoration process by inputting the captured image, in which the partial region has been masked by the masking process, into the restoration model. An anomaly detection device in which, during the training of the reconstruction model, image regions representing a predetermined type of object that is not abnormal to be included in the captured image are not included in the loss calculation.

4. A generation means for generating multiple processed images by processing each of multiple time-series captured images, A calculation means for calculating difference data representing the difference between each of the multiple captured images and the processed image generated from that captured image, The system includes a determination means for determining whether the captured image represents an abnormal situation based on a plurality of calculated difference data, The processing performed on the captured image includes a masking process that masks one or more subregions included in the captured image, and a restoration process that restores the masked subregions using data other than those subregions. The determination means is For each of the multiple reference time points, a degree of change in the situation is calculated, which represents the ratio of the statistical value of the difference data calculated for each of the multiple captured images generated during the period after the reference time point to the statistical value of the difference data calculated for each of the multiple captured images generated during the period before the reference time point. An anomaly detection device that determines that the captured image generated at the reference time represents an abnormal situation if the degree of change in the situation calculated for the reference time is equal to or greater than a threshold.

5. The system has output means that outputs a screen containing the aforementioned captured image, An anomaly detection device according to claim 4, wherein the output means, after it is determined that the degree of change in the situation calculated for the reference time is equal to or greater than a threshold, attenuates the degree of change in the situation over time, and while the attenuated degree of change in the situation is equal to or greater than the degree of change in the situation calculated for the newly acquired captured image, includes a display on the screen indicating that the captured image represents an abnormal situation.

Images