Information processing device, information processing method, and program

Abstract

The objective is to provide an information processing device, an information processing method, and a program that can suppress false detection of anomalies. [Solution] The information processing device according to the embodiment comprises a storage means, a first acquisition means, a generation means, a detection means, a correction means, and an output means. The storage means stores an anomaly detection model generated by learning a first image. The first acquisition means acquires a second image. The generation means uses the anomaly detection model to calculate an anomaly score representing the degree of anomaly occurring in the environment in which the second image was captured, and generates an anomaly score map based on the calculated anomaly score. The detection means detects a first region containing objects present in the environment in which the second image was captured from the second image. The correction means corrects the anomaly score map based on the first region. The output means outputs the corrected anomaly score map.

Description

Technical Field

[0006] , , ,

[0001] Embodiments of the present invention relate to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Generally, monitoring of a predetermined environment (hereinafter referred to as a target environment) is performed by a user visually checking an image (video) captured by a surveillance camera installed for crime prevention purposes or security measures.

[0003] However, there is a limit to the number of images that a user can check. For this reason, in recent years, it has been considered to automatically detect an abnormality that has occurred in a target environment from an image captured by a surveillance camera. In this case, for example, by preparing an anomaly detection model that has learned an image captured in a target environment in a normal state (a target environment in which no anomaly has occurred), it is possible to detect (infer) that the state has deviated from the normal state as an anomaly.

[0004] The method using the anomaly detection model as described above can detect an anomaly even if the anomaly is not defined in advance. However, when an object that is not included in the learned image (an object whose appearance is different from the learned image) exists in the target environment or when an object is moving in the target environment, there is a possibility of false detection.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Therefore, the problem that the present invention aims to solve is to provide an information processing device, an information processing method, and a program that can suppress false detection of anomalies. [Means for solving the problem]

[0007] The information processing device according to the embodiment comprises a storage means, a first acquisition means, a generation means, a detection means, a correction means, and an output means. The storage means stores an anomaly detection model that is generated by learning a first image captured in a normal environment and is used to calculate an anomaly score representing the degree of anomaly occurring in the environment. The first acquisition means acquires a second image captured in the environment. The generation means uses the anomaly detection model stored in the storage means to calculate an anomaly score representing the degree of anomaly occurring in the environment in which the acquired second image was captured, and generates an anomaly score map based on the calculated anomaly score. The detection means detects a first region containing an object present in the environment in which the second image was captured from the second image. The correction means corrects the anomaly score map based on the detected first region. The output means outputs the corrected anomaly score map. [Brief explanation of the drawing]

[0008] [Figure 1] A block diagram showing an example of the functional configuration of the information processing device according to the first embodiment. [Figure 2] A diagram showing an example of the hardware configuration of an information processing device. [Figure 3] A diagram showing an example of a normal image. [Figure 4] A flowchart illustrating an example of the inference process steps. [Figure 5] A diagram to specifically explain the inference process. [Figure 6] A diagram illustrating the modification of the object's region. [Figure 7] A block diagram showing an example of the functional configuration of the information processing device according to the second embodiment. [Figure 8] A diagram to explain the training data. [Figure 9] A diagram to explain the training data. [Figure 10] A diagram to explain the training data. [Figure 11] A block diagram showing an example of the functional configuration of the information processing device according to the third embodiment. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings. (First Embodiment) First, the first embodiment will be described. The information processing device according to this embodiment operates as an anomaly detection device for detecting anomalies that occur in a target environment (for example, a monitored area) using images captured in that target environment.

[0010] Figure 1 is a block diagram showing an example of the functional configuration of the information processing device according to this embodiment. As shown in Figure 1, the information processing device 10 includes an image database (DB) 11, a learning processing unit 12, a model storage unit 13, and an inference processing unit 14.

[0011] The image database 11 stores images taken under normal conditions in the target environment (hereinafter referred to as "normal images").

[0012] The learning processing unit 12 trains the anomaly detection model using normal images stored in the image database 11. In other words, the anomaly detection model is generated by learning from normal images stored in the image database 11.

[0013] The anomaly detection model trained by the learning processing unit 12 is stored in the model storage unit 13. The anomaly detection model stored in the model storage unit 13 is used to calculate an anomaly score (hereinafter referred to as the target environment anomaly score) that represents the degree of anomalies occurring in the target environment.

[0014] The inference processing unit 14 is a functional unit that executes processing corresponding to inferring whether an abnormality has occurred in the target environment using an abnormality detection model, and includes an image acquisition unit 141, a prompt acquisition unit 142, an abnormality score map generation unit 143, an object region detection unit 144, an abnormality score map correction unit 145, and an output unit 146.

[0015] When performing the above inference, the image acquisition unit 141 acquires an image (hereinafter referred to as a target image) captured in the target environment. The target image acquired by the image acquisition unit 141 is, for example, specified by a user who uses the information processing apparatus 10. Specifically, for example, when a path indicating the location where the target image is stored is specified by the user, the image acquisition unit 141 can acquire (read) the target image stored at the location indicated by the path.

[0016] The target image may be, for example, an image (data) captured by an imaging device such as a surveillance camera installed near the target environment, and may be acquired from the imaging device.

[0017] The prompt acquisition unit 142 acquires a prompt that indicates an object existing in the target environment. In the present embodiment, the object indicated by the prompt is assumed to be, for example, an object that is not a target for abnormality detection. The prompt is, for example, text and is specified by the user. Specifically, for example, when a path indicating the location where the prompt is stored is specified by the user, the prompt acquisition unit 142 can acquire (read) the prompt stored at the location indicated by the path.

[0018] The prompt may be entered by the user, for example, via a GUI (Graphical User Interface), or it may be obtained by converting the user's voice into text through speech recognition. In this embodiment, the prompt is described as indicating an object, but the object in this embodiment may be anything that can be recognized on the target image described above, and may be a fluid such as smoke or water, or a non-material thing such as fire.

[0019] The anomaly score map generation unit 143 uses the anomaly detection model (anomaly detection model trained by the learning processing unit 12) stored in the model storage unit 13 to calculate the anomaly score of the target environment in which the target image acquired by the image acquisition unit 141 was captured. The anomaly score map generation unit 143 generates an anomaly score map based on the anomaly score of the target environment calculated in this way. In this embodiment, the anomaly score is an index of anomalies in which the value increases as the degree of anomaly increases.

[0020] The object region detection unit 144 detects the region containing the object indicated by the prompt obtained by the prompt acquisition unit 142 from the target image acquired by the image acquisition unit 141 (hereinafter referred to as the object region). The object region detected by the object region detection unit 144 is, for example, a rectangular region, and is represented by the coordinates of the center of the rectangle and the width and height of the rectangle.

[0021] Furthermore, the shape of the object region is not limited to a rectangle. The object region may also be represented as segment information indicating the object's contour or position at the pixel level, or by other methods.

[0022] The abnormal score map correction unit 145 corrects the abnormal score map generated by the abnormal score map generation unit 143 based on the object region detected by the object region detection unit 144. The details of the correction process by the abnormal score map correction unit 145 will be described later, but the abnormal score map correction unit 145 corrects the abnormal score map so that the object indicated by the prompt acquired by the prompt acquisition unit 142 does not affect the detection of abnormalities in the target environment.

[0023] The output unit 146 outputs an abnormal score map corrected by the abnormal score map correction unit 145. The abnormal score map output by the output unit 146 corresponds to the detection result of an anomaly that occurred in the target environment (the inference result of whether or not an anomaly occurred in the target environment).

[0024] Figure 2 shows an example of the hardware configuration of the information processing device 10 shown in Figure 1. The information processing device 10 includes a CPU 10a, non-volatile memory 10b, main memory 10c, and communication device 10d, etc.

[0025] The CPU 10a is a processor for controlling the operation of various components within the information processing device 10. The CPU 10a may be a single processor or may consist of multiple processors. The CPU 10a executes various programs loaded from the non-volatile memory 10b into the main memory 10c. These programs include, for example, an operating system (OS) and application programs.

[0026] The non-volatile memory 10b is a storage medium used as an auxiliary storage device. The main memory 10c is a storage medium used as the main storage device. Although only the non-volatile memory 10b and the main memory 10c are shown in Figure 2, the information processing device 10 may also be equipped with other storage devices.

[0027] The communication device 10d is a device configured to communicate with an external device (for example, a server device).

[0028] In this embodiment, the image database 11 and model storage unit 13 shown in Figure 1 are implemented by, for example, a non-volatile memory 10b or another storage device.

[0029] Furthermore, some or all of the learning processing unit 12 and inference processing unit 14 included in the information processing device 10 shown in Figure 1 are implemented by having the CPU 10a (i.e., the computer of the information processing device 10) execute a predetermined program, that is, by software. This program may be stored and distributed on a computer-readable storage medium, or it may be downloaded to the information processing device 10 via a network. Some or all of the learning processing unit 12 and inference processing unit 14 may be implemented by hardware such as an IC (Integrated Circuit), or by a combination of software and hardware.

[0030] Although not shown in Figure 2, the information processing device 10 may further include input devices such as a mouse and keyboard, and display devices such as a display.

[0031] The operation of the information processing device 10 according to this embodiment will be described below. In this embodiment, processing related to the learning of an anomaly detection model (hereinafter referred to as the learning process) and processing related to inference of whether or not an anomaly has occurred in the target environment using the anomaly detection model (hereinafter referred to as the inference process) are executed.

[0032] First, let me briefly explain the learning process described above. In the learning process, the learning processing unit 12 learns an anomaly detection model using normal images stored in the image database 11.

[0033] Here, if the target environment is, for example, an environment where automobiles travel, such as a highway, the learning processing unit 12 will use images captured in that target environment, as shown in Figure 3, as normal images for learning. In Figure 3, images of a highway with no automobiles and images of a highway with various automobiles are shown as normal images. As described above, when the target environment is an environment where automobiles travel, it is assumed that, for example, the entry of a person or animal that obstructs the movement of the automobile onto the highway will be detected as an anomaly occurring in that target environment.

[0034] In this embodiment, the anomaly detection model is used to calculate an anomaly score of the target environment and utilizes, for example, an autoencoder or a Generative Adversarial Network (GAN), which are types of neural networks. In this case, the anomaly detection model is trained to output an image that reproduces the input image, and such an anomaly detection model can calculate an anomaly score of the target environment based on the reconstruction error between the input image and the output image.

[0035] In this embodiment, it is assumed that an anomaly detection model is generated by training using normal images. However, images taken in an abnormal environment (hereinafter referred to as "abnormal images") may also be used to further train the anomaly detection model.

[0036] Furthermore, while this section has described cases where autoencoders or generative adversarial networks are used as anomaly detection models, these anomaly detection models may be generated according to other machine learning algorithms, as long as they are used to calculate anomaly scores for the target environment (generate an anomaly score map).

[0037] The anomaly detection model generated in the learning process described above is stored in the model storage unit 13 and used in the inference process.

[0038] Next, we will explain an example of the processing steps for the inference process described above, referring to the flowchart in Figure 4.

[0039] First, the image acquisition unit 141 acquires an image captured under the target environment as the target image (step S1). Note that the target image acquired in step S1 may be one or multiple images.

[0040] Furthermore, the prompt acquisition unit 142 acquires prompts that indicate objects present in the target environment (step S2). As mentioned above, the objects indicated by the prompts are objects that do not need to be detected as abnormal. One or more prompts may be acquired in step S2.

[0041] Next, the anomaly score map generation unit 143 generates an anomaly score map using the anomaly detection model stored in the model storage unit 13 (step S3).

[0042] Now, let's explain the process in step S3. The anomaly detection model (e.g., an autoencoder) generated by training using normal images as shown in Figure 3 above is constructed to output an image similar to the normal image when a normal image is input. In other words, when an image different from a normal image (e.g., an anomaly image) is input to the anomaly detection model, it cannot reconstruct an image similar to the original image and will output an image different from the original image.

[0043] According to such an anomaly detection model, an anomaly score can be calculated based on the reconstruction error between the image input to the anomaly detection model (input image) and the image output from the anomaly detection model (output image).

[0044] In this embodiment, the anomaly score map generation unit 143 calculates an anomaly score for the target environment based on the reconstruction error between the target image acquired in step S1 and the image output from the anomaly detection model (i.e., the output image of the anomaly detection model) by inputting the target image into the anomaly detection model. In this embodiment, an anomaly score is calculated for each pixel constituting the target image. Specifically, for example, the absolute value of the difference between the pixel values ​​can be calculated for each corresponding pixel in the target image and the output image of the anomaly detection model, and the absolute value calculated for each pixel can be used as the anomaly score for that pixel.

[0045] In this case, the abnormal score map generation unit 143 generates an abnormal score map by assigning the abnormal score calculated for each pixel constituting the target image to that pixel.

[0046] As described above, the anomaly score assigned to each pixel in the anomaly score map may be normalized so that it is contained within a certain range. Furthermore, the anomaly score may be calculated using other methods.

[0047] Next, the object region detection unit 144 detects object regions from the target image acquired in step S1 based on the prompt acquired in step S2 (step S4). In step S4, object regions are detected using a pre-prepared base model, for example, obtained through self-supervised learning using large-scale images or text.

[0048] The underlying model used for object region detection may be a single model, or it may be implemented using multiple models with different network configurations and parameters. Furthermore, the underlying model may be based on other learning methods, such as unsupervised or supervised learning, rather than self-supervised learning.

[0049] Furthermore, while it is assumed that the base model is constructed to estimate object regions in a target image by inputting the target image and prompts, the configuration may also involve separately executing the process of extracting regions containing a predetermined object (candidate regions) in the target image and the process of identifying whether the object contained in the estimated region is the object indicated by the prompt. That is, for example, candidate regions may be extracted from the target image by performing predetermined image processing, and these candidate regions may be input together with the prompts to the base model (a model that identifies labels representing the type of object on an image-by-image basis) to detect object regions. In addition, object regions may be detected by methods other than the base model.

[0050] Next, the abnormal score map correction unit 145 corrects the portion of the abnormal score map generated in step S3 that corresponds to the object region detected in step S4 (step S5).

[0051] As described above, the anomaly score map is generated by assigning the anomaly score calculated for each pixel to that pixel. In step S5, a process is performed to correct the anomaly score assigned to the pixel corresponding to the object region (i.e., the anomaly score of the object region). Specifically, in step S5, a correction is performed to reduce the anomaly score of the object region (i.e., to reduce the degree of anomaly).

[0052] When the processing in step S5 is executed, the output unit 146 outputs the abnormal score map corrected in step S5 (step S6). In step S6, the abnormal score map may be output to a communication device 10d for transmission to, for example, an external server device of the information processing device 10, or it may be output to a display device (display) for presentation to the user. In this embodiment, the abnormal score map output makes it possible to detect whether an abnormality has occurred in the target environment based on the abnormal score assigned to each pixel constituting the abnormal score map, as described above. Therefore, the abnormal score map corresponds to the inference result of whether or not an abnormality has occurred in the target environment.

[0053] Although this explanation assumes that an anomaly score map is output, this anomaly score map may be processed before output. Alternatively, the maximum anomaly score assigned to each pixel constituting the anomaly score map may be compared with a pre-defined threshold (a threshold for distinguishing between normal and abnormal). If the maximum anomaly score is less than the threshold, an output (notification) indicating that no anomaly has occurred in the target environment may be generated. If the maximum anomaly score is equal to or greater than the threshold, an output (notification) indicating that an anomaly has occurred in the target environment may be generated.

[0054] Furthermore, although Figure 4 was explained assuming that the object indicated by the prompt exists in the target environment (i.e., the object region is detected from the target image), if the object indicated by the prompt does not exist in the target environment and therefore the object region is not detected from the target image, the processing in step S5 is omitted, and the abnormal score map generated in step S3 is output in step S6.

[0055] The inference process described above will be explained in detail below with reference to Figure 5. Here, the inference process is assumed to be performed using an anomaly detection model generated by training with the normal images shown in Figure 3.

[0056] Here, as shown in Figure 5, we assume that a target image Im has been acquired under the target environment in which the automobile is driving. In this case, the target image Im is input to the anomaly detection model, which generates an anomaly score map Am.

[0057] In this embodiment, the autoencoder used as an anomaly detection model, for example, learns the normal range and estimates anything outside that range as abnormal. However, if an image containing a car that differs in position, shape, or color from the normal image is input as the target image Im to an anomaly detection model that has been trained using the normal image shown in Figure 3, as described above, an anomaly score map Am with a high anomaly score may be generated. Furthermore, even if the same normal image as the target image Im is used to train the anomaly detection model, if the car present in the image appears in the normal image at a low frequency, the anomaly score assigned to the anomaly score map Am may be high.

[0058] In the example shown in Figure 5, we assume a case where the region containing the car in the target image Im is not reconstructed in the same way as the target image Im by the anomaly detection model (autoencoder), and the anomaly score map Am includes the region R to which a high anomaly score is assigned.

[0059] However, in this embodiment, if, for example, a person or animal that obstructs the movement of a vehicle enters the highway (target environment) and is detected as an anomaly, then detecting a vehicle present in the target image Im as an anomaly would be a false positive. In other words, it is preferable to avoid detecting anomalies based on vehicles present in the target image Im.

[0060] Therefore, in this embodiment, as shown in Figure 5, for example, the prompt "automobile" is acquired, and the target image Im and the prompt "automobile" are input to the base model, thereby detecting the object region D containing the automobile from the target image Im. In Figure 5, the object region D is shown by a dashed line and is represented, for example, by the coordinates (X,Y) of the top left point and the coordinates (X+W,Y+H) of the bottom right point. In other words, the object region D is a rectangular area with a width of W and a height of H. The object region D may have any shape as long as it encompasses the automobile.

[0061] Next, the anomaly score map Am is partially corrected based on the position of the object region D described above. Specifically, if the anomaly score map Am is generated by assigning the absolute value of the difference in pixel values ​​as an anomaly score to each pixel as described above, a correction is performed to set the anomaly score assigned to each of the multiple pixels corresponding to the object region D in the anomaly score map Am to 0.

[0062] In this embodiment, since the automobile is an object that does not need to be detected as an anomaly, the anomaly score map Am' has an anomaly score of 0 for the portion corresponding to object region D (i.e., the high anomaly score calculated based on the automobile is corrected), thus avoiding situations in which an anomaly is falsely detected as occurring in the target environment due to the automobile included in object region D.

[0063] In this explanation, it is assumed that the anomaly score is corrected to 0, but the correction of the anomaly score (anomaly score map Am) may be performed by other methods. In this embodiment, the anomaly score map Am only needs to be corrected so that an anomaly is not recognized in the object region D, and the anomaly score assigned to each of the multiple pixels corresponding to the object region D should be at least lower than the maximum value of the anomaly score. In this case, for example, the anomaly score assigned to each of the multiple pixels corresponding to the object region D may be corrected to the minimum or median value of the anomaly score.

[0064] Furthermore, in the example shown in Figure 5, it was explained that object region D is detected from the target image Im, but as mentioned above, the region with a high anomaly score may be wider than the object region D. For this reason, the detected object region D may be modified as described above to correct the anomaly score map Am. Specifically, as shown in Figure 6, object region D may be changed to region D' which includes the area surrounding object region D, and the anomaly scores assigned to each of the multiple pixels corresponding to region D' may be corrected. Region D' is, for example, a region obtained by adding a fixed-value buffer around object region D. The buffer added around object region D may be determined (and may vary) according to the size of object region D (width W and height H).

[0065] As described above, the information processing device 10 according to this embodiment uses an anomaly detection model generated by learning normal images (first images) captured in a target environment under normal conditions to calculate an anomaly score (an anomaly score representing the degree of anomaly occurring in the target environment) of the target environment in which the target image (second image) was captured, and generates an anomaly score map based on the calculated anomaly score. Furthermore, the information processing device 10 according to this embodiment detects an object region (first region) containing objects present in the target environment in which the target image was captured from the target image, corrects the anomaly score map based on the detected object region, and outputs the corrected anomaly score map.

[0066] In this embodiment, the above configuration makes it possible to suppress false detection of anomalies without collecting abnormal images.

[0067] Specifically, the information processing device 10 according to this embodiment acquires a prompt indicating an object that does not need to be detected as an anomaly, and detects an object region containing the object indicated by the acquired prompt from the target image. Furthermore, the information processing device 10 according to this embodiment calculates an anomaly score for each pixel constituting the target image, generates an anomaly score map by assigning the calculated anomaly score to the pixel, and corrects the first anomaly score assigned to each of the multiple pixels corresponding to the detected object region as described above to a second anomaly score. In this case, the second anomaly score is set to be lower than the maximum value of the first anomaly score assigned to at least each of the multiple pixels corresponding to the object region (for example, 0).

[0068] With this configuration, it is possible to correct the anomaly score map so as to reduce the anomaly score of the portion of the anomaly score map corresponding to the object region. This avoids the detection (determination) that an anomaly is occurring in the target environment due to the presence of objects that do not need to be detected as an anomaly.

[0069] Furthermore, the second anomaly score described above may be a score greater than or equal to the minimum value of the first anomaly score assigned to each of the multiple pixels corresponding to the object region (for example, the minimum value or median value). With such a configuration, when the corrected anomaly score map is displayed on a display device or the like (i.e., the anomaly score map is visualized), it is possible to prevent the anomaly score of the object region from appearing unnaturally lower than its surroundings.

[0070] Furthermore, in this embodiment, a weight map may be created in which a weight that lowers the first anomaly score is assigned to each of at least several pixels corresponding to the object region, and the anomaly score map may be corrected by multiplying the anomaly score map by the created weight map. In this case, for example, by creating a weight map in which the weight gradually increases from the center of the object region to the outside of the object region, it is possible to eliminate the abrupt change in the anomaly score at the boundary between the inside and outside of the object region (i.e., the unnaturalness when displaying the anomaly score map).

[0071] Furthermore, for example, after correcting the abnormal score map, a smoothing process may be performed on the abnormal scores to eliminate any unnaturalness when displaying the abnormal score map.

[0072] Furthermore, in this embodiment, the abnormal score map may be corrected by modifying the object region detected from the target image. In this case, the object region may be changed to, for example, a region with a buffer added around it. The buffer added to the object region may be determined according to the size of the object region. With such a configuration, even if, for example, the region with a high abnormal score in the abnormal score map is larger than the object region, it becomes possible to appropriately correct the abnormal score map and suppress false detections.

[0073] Furthermore, in this embodiment, it is assumed that object regions are detected using a pre-prepared base model, for example, through self-supervised learning, but the object regions may be detected by other methods. Also, in this embodiment, for example, multiple candidate regions (second regions) containing each of multiple objects may be detected from the target image, and the object region may be detected by identifying whether the object contained in each of the detected candidate regions is the object indicated by the prompt.

[0074] In this embodiment, the information processing device 10 has been described as including an image database 11, a learning processing unit 12, a model storage unit 13, and an inference processing unit 14. However, the information processing device 10 may also be configured to include only a portion of each of the parts 11 to 14 (i.e., a portion of each of the parts 11 to 14 may be omitted). Specifically, the information processing device 10 according to this embodiment may include, for example, an inference processing unit 14, and at least a portion of the image database 11, the learning processing unit 12, and the model storage unit 13 may be located externally.

[0075] Furthermore, although the information processing device 10 has been described as a single device in this embodiment, the information processing device 10 may be implemented as an information processing system or the like, which is made up of multiple devices. Specifically, for example, it may be an information processing system that includes a learning processing device that performs processing equivalent to the learning processing device 12 included in the information processing device 10, and an inference processing device (anomaly detection device) that performs processing equivalent to the inference processing device 14 included in the information processing device 10.

[0076] (Second Embodiment) Next, a second embodiment will be described. In this embodiment, a detailed explanation of the parts that are the same as those in the first embodiment described above will be omitted, and the parts that differ from the first embodiment will be described mainly.

[0077] In the first embodiment described above, a base model was used to detect object regions from a target image. However, this base model is constructed by learning from a large number of images, and therefore has a large number of parameters and is also large in size. While such a base model can be used if computing resources are abundant, it may not be usable when operating an edge device as an information processing device if such computing resources are not abundant.

[0078] Therefore, this embodiment differs from the first embodiment described above in that, instead of using a base model for detecting object regions during inference processing, it uses an object detection model specifically designed to detect object regions from images captured in the target environment.

[0079] Figure 7 is a block diagram showing an example of the functional configuration of the information processing device according to this embodiment. As shown in Figure 7, the information processing device 10 includes an image database 11, a learning processing unit 12, a first model storage unit 13, an inference processing unit 14, and a second model storage unit 15.

[0080] The image database 11 is as described in Figure 1 above, so a detailed explanation will be omitted.

[0081] The learning processing unit 12 includes a first learning unit 121, a prompt acquisition unit 122, a teacher data generation unit 123, and a second learning unit 124.

[0082] As described in Figure 1 above, the first learning unit 121 trains an anomaly detection model using normal images stored in the image database 11. The anomaly detection model trained by the first learning unit 121 is stored in the first model storage unit 13. In other words, the first model storage unit 13 in this embodiment corresponds to the model storage unit 13 described in Figure 1 above.

[0083] The prompt acquisition unit 122 acquires prompts that indicate objects present in the target environment, similar to the prompt acquisition unit 142 described in Figure 1 above.

[0084] The training data generation unit 123 generates training data that includes one or more normal images stored in the image database 11 and region information indicating object regions detected from the normal images (regions containing objects indicated by prompts acquired by the prompt acquisition unit 122).

[0085] Furthermore, the object regions indicated by the region information included in the training data are detected, for example, based on the output of the base model described in the first embodiment above, when a normal image and a prompt acquired by the prompt acquisition unit 122 are input to the base model.

[0086] Furthermore, the region information includes, for example, information indicating the extent of the object region (the position of the object and the size of the object region), and is labeled to represent the objects contained within that object region.

[0087] Furthermore, the normal images used to generate training data may be the same images used to train the anomaly detection model described above, or they may be different images (normal or abnormal images) from the normal images used to train the anomaly detection model.

[0088] The second learning unit 124 trains the object detection model using the training data generated by the training data generation unit 123. When training is performed by the second learning unit 124, an object detection model is generated that is constructed to output object regions (region information indicating object regions) detected from an image taken in the target environment when the image is input. Note that the object detection model may be generated by any method that makes it possible to estimate the object region (object position) from the image. The object detection model thus generated is stored in the second model storage unit 15.

[0089] The inference processing unit 14 includes an image acquisition unit 141, an abnormal score map generation unit 143, an object region detection unit 144, an abnormal score map correction unit 145, and an output unit 146.

[0090] The image acquisition unit 141, the abnormal score map generation unit 143, the abnormal score map correction unit 145, and the output unit 146 are as described in Figure 1 above, so a detailed explanation of them will be omitted. In addition, in this embodiment, the prompt acquisition unit 142 shown in Figure 1 above is omitted from the inference processing unit 14.

[0091] Unlike the first embodiment described above, the object region detection unit 144 in this embodiment uses the object detection model stored in the second model storage unit 15 to detect object regions from the target image acquired by the image acquisition unit 141. As described above, the object detection model in this embodiment is trained to output object regions when a target image is input, so in this embodiment, a prompt is not required during the inference process.

[0092] Although the functional configuration of the information processing device 10 according to this embodiment has been described here, the hardware configuration of the information processing device 10 is the same as that of the first embodiment described above, so a detailed explanation of it will be omitted. In this embodiment, the second model storage unit 15 shown in Figure 7 is realized by, for example, the non-volatile memory 10b shown in Figure 2 or other storage devices.

[0093] The training of the object detection model described above will be explained in detail below. First, the prompt acquisition unit 122 acquires prompts that indicate objects that do not need to be detected as abnormal. For example, if the target environment is a highway where cars are driving, the prompt that indicates objects that do not need to be detected as abnormal is "car".

[0094] Next, the training data generation unit 123 applies one or more normal images and the prompt "automobile" stored in the image database 11 to the base model and detects an object region containing an automobile from the normal image. The training data generation unit 123 generates training data based on the object region detected from the normal image.

[0095] Here, for example, as shown in Figure 8, if a normal image Im1 stored in the image database 11 and the prompt "automobile" are input to the base model, no object region is detected. In this case, training data is generated that includes, for example, the normal image Im1 and information indicating that no object region exists in the normal image Im1.

[0096] Furthermore, as shown in Figure 9, for example, when a normal image Im2 and the prompt "automobile" stored in the image database 11 are input to the base model, the object region D2 is output from the base model (i.e., the object region D2 is detected from the normal image Im2). In this case, training data is generated that includes, for example, the normal image Im2 and region information indicating the object region D2.

[0097] Furthermore, as shown in Figure 10, for example, when a normal image Im3 and the prompt "automobile" stored in the image database 11 are input to the base model, object regions D31 and D32 are output from the base model (i.e., object regions D31 and D32 are detected from the normal image Im3). In this case, training data is generated that includes, for example, the normal image Im3 and region information indicating object regions D31 and D32.

[0098] The second learning unit 124 trains an object detection model using the training data generated by the training data generation unit 123. This training of the object detection model is performed according to an arbitrary machine learning algorithm, but for example, when a normal image included in the training data is input, a process is executed to update the parameters (weights) of the object detection model so that the object region indicated by the region information included in the training data is output (detected).

[0099] The inference process in this embodiment is the same as in the first embodiment described above, except that it does not obtain a prompt and detects the object region from the target image using the object detection model described above. Therefore, a detailed explanation of this process will be omitted.

[0100] As described above, in this embodiment, an object detection model that has learned training data including normal images stored in the image database 11 and region information indicating object regions (third regions including the object indicated by the prompt) detected from the normal images is used to detect object regions from the target image.

[0101] In this embodiment, the object detection model generated by training on training data is specialized in detecting object regions containing objects present in the target environment. Compared to the base model constructed by training on large-scale images as described in the first embodiment, it can be implemented with fewer parameters and a smaller size. In this embodiment, by using such an object detection model during inference, it becomes possible to perform inference processing on, for example, an edge device with limited computing resources, such as the information processing device 10.

[0102] Furthermore, in this embodiment, it is believed that the amount of processing and processing time can be reduced compared to inference processing using the base model, by performing inference processing using the object detection model.

[0103] Furthermore, in this embodiment, by using normal images and the output results of the base model as training data to train the object detection model, it becomes possible to prepare (generate) the object detection model without the effort of generating training data.

[0104] In this embodiment, when using a base model to train an object detection model, the inference of whether or not an anomaly has occurred in the target environment may be performed on an edge device or the like with limited computing resources, while the training of the object detection model may be performed on an information processing device with abundant computing resources, different from the edge device.

[0105] (Third embodiment) Next, a third embodiment will be described. In this embodiment, a detailed explanation of the parts that are the same as those in the first and second embodiments described above will be omitted, and the parts that differ from the first and second embodiments will be described mainly.

[0106] In the second embodiment described above, it is assumed that the object detection model is trained before the information processing device is put into operation (i.e., the object detection model is generated in advance). However, for example, images stored in an image database may not be sufficient for training the object detection model, and it may be difficult to collect the data (images) necessary for training the object detection model before starting operation of the information processing device.

[0107] This embodiment differs from the second embodiment described above in that, taking the above circumstances into consideration, it detects object regions from the target image using a base model immediately after starting operation of the information processing device, and trains an object detection model using training data that includes the target image and region information indicating the object region during the operation of the information processing device.

[0108] Figure 11 is a block diagram showing an example of the functional configuration of the information processing device according to this embodiment. As shown in Figure 11, the information processing device 10 includes an image database 11, a learning processing unit 12, a first model storage unit 13, an inference processing unit 14, and a second model storage unit 15.

[0109] The image database 11 is as described in Figure 1 above, so a detailed explanation will be omitted.

[0110] The learning processing unit 12 includes a first learning unit 121 and a second learning unit 124. The first learning unit 121 has the same functionality as the first learning unit 121 shown in Figure 7, and learns an anomaly detection model using normal images stored in the image database 11. The second learning unit 124 has the same functionality as the second learning unit 124 shown in Figure 7, and learns an object detection model using training data.

[0111] The first model storage section 13 is as described in Figure 7 above, so a detailed explanation will be omitted.

[0112] The inference processing unit 14 includes an image acquisition unit 141, a prompt acquisition unit 142, an abnormal score map generation unit 143, an object region detection unit 144, an abnormal score map correction unit 145, and an output unit 146.

[0113] The image acquisition unit 141, prompt acquisition unit 142, abnormal score map generation unit 143, abnormal score map correction unit 145, and output unit 146 are as described in Figure 1 above, so a detailed explanation of them will be omitted.

[0114] In this embodiment, the object region detection unit 144 operates to detect object regions from a target image using a base model during the initial operation of the information processing device 10, as described in the first embodiment above. When an object region is detected from a target image using the base model in this way, the target image and region information indicating the object region are passed from the inference processing unit 14 to the learning processing unit 12 as training data. The training data passed from the inference processing unit 14 to the learning processing unit 12 is used for training the object detection model by the second learning unit 124, as described above.

[0115] Furthermore, when learning is performed by the second learning unit 124, the object detection model described in the second embodiment described above is generated. When the object detection model generated in this way is stored in the second model storage unit 15, the object region detection unit 144 operates to detect object regions from the target image using the object detection model stored in the second model storage unit 15.

[0116] As described above, in this embodiment, during the initial operation of the information processing device 10, it operates in the same manner as in the first embodiment described above, detecting object regions from a target image using a base model, while simultaneously training the object detection model using the target image and the detection results as training data. As a result, when a target image is acquired after the object detection model has been sufficiently trained (i.e., when inference processing is performed), the object detection model can be used to detect object regions from the target image instead of the base model described above.

[0117] According to the above configuration, it is difficult to use the base model from the standpoint of hardware resources (computational resources) and acceptable inference processing time. However, if it is not possible to collect the data necessary for training the object detection model before operating the information processing device 10, the base model can be used as a bridge until the object detection model is trained (generated). If training is performed using data collected during the operation of the information processing device 10, the base model can be changed to the object detection model. This operational process can be implemented in this way.

[0118] In this embodiment, it has been explained that the object detection model is used in place of the base model after the object detection model has been trained. However, one of the base model and the object detection model may be selectively used based on, for example, the operational policy of the information processing device 10. Alternatively, the decision to use the base model or the object detection model may be made based on, for example, the training status of the object detection model (e.g., training period or training amount). The decision to use the base model or the object detection model may also be made based on, for example, the accuracy of object region detection using the object detection model.

[0119] Furthermore, in this embodiment, it has been explained that training data (target images and region information indicating object regions detected from the target images) is passed from the inference processing unit 14 to the learning processing unit 12 in order to train the object detection model. However, the training data may be stored in a memory unit (not shown) within the learning processing unit 12 or the inference processing unit 14, for example, and the object detection model may be trained once a sufficient number of training data have been accumulated in the memory unit. For example, if estimation processing is performed for each of multiple consecutive target images, such as in a video, all of the target images may be stored in the memory unit as training data. Alternatively, some of the images among the multiple target images (for example, images sampled from multiple target images or images acquired at regular intervals from multiple target images) may be stored in the memory unit as training data. With such a configuration, it is possible to prevent the accumulation of redundant training data, for example, when similar images appear consecutively in a video.

[0120] Furthermore, in this embodiment, the object detection model is trained using training data that includes a target image and region information indicating the object region detected from the target image. However, there may be errors in the region information. Specifically, the position or range of the object region indicated by the region information included in the training data may be different from that of the object, or the object included in the object region may be different from the object indicated by the prompt (i.e., the region information may be labeled to represent an object different from the object indicated by the prompt). In such cases, the object detection model will be trained using training data containing erroneous region information, and the accuracy of object region detection by the object detection model will decrease.

[0121] Therefore, although omitted in Figure 11, the information processing device 10 may further include a modification unit. The modification unit receives instructions from an external source (e.g., user instructions) and modifies the region information (such as the location or size and label of an object region) contained in the training data stored in the memory unit based on those instructions. By modifying the region information contained in the training data in this way, it becomes possible to train the object detection model using training data that contains the correct region information, thereby suppressing a decrease in the accuracy of object region detection by the object detection model (i.e., improving the detection accuracy).

[0122] According to at least one embodiment described above, it is possible to provide an information processing device, an information processing method, and a program that can suppress false detection of anomalies.

[0123] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0124] 10...Information processing device, 10a...CPU, 10b...Non-volatile memory, 10c...Main memory, 10d...Communication device, 11...Image database, 12...Learning processing unit, 13...Model storage unit (First model storage unit), 14...Inference processing unit, 15...Second model storage unit, 121...First learning unit, 122...Prompt acquisition unit, 123...Training data generation unit, 124...Second learning unit, 141...Image acquisition unit, 142...Prompt acquisition unit, 143...Anomaly score map generation unit, 144...Object region detection unit, 145...Anomaly score map correction unit, 146...Output unit

Claims

1. A storage means for storing an anomaly detection model that is generated by learning a first image captured in a normal environment and is used to calculate an anomaly score representing the degree of anomaly occurring in the environment, A first acquisition means for acquiring a second image captured under the aforementioned environment, A generation means that uses an anomaly detection model stored in the storage means to calculate an anomaly score representing the degree of anomaly occurring in the environment in which the acquired second image was captured, and generates an anomaly score map based on the calculated anomaly score, A detection means for detecting a first region containing an object present in the environment in which the second image was captured from the second image, Correction means for correcting the abnormal score map based on the detected first region, Output means for outputting the corrected abnormal score map An information processing device equipped with the following.

2. The system further comprises a second acquisition means for obtaining a prompt indicating the aforementioned object, The detection means detects a first region containing the object indicated by the acquired prompt from the second image. The information processing apparatus according to claim 1.

3. The information processing apparatus according to claim 2, wherein the detection means detects the first region based on the output of the base model when the second image and the acquired prompt are input to the base model.

4. The second image was captured in an environment where multiple objects were present. The detection means detects a plurality of second regions from the second image, each containing one of the plurality of objects, and identifies whether the object contained in each of the detected plurality of second regions is the object indicated by the prompt, thereby detecting the first region. The information processing apparatus according to claim 3.

5. The information processing apparatus according to claim 2, wherein the detection means detects the first region from the second image using an object detection model that has learned training data including the first image and region information indicating a third region including an object indicated by the prompt detected from the first image.

6. The system comprises a learning means for training an object detection model using training data that includes the second image and region information indicating the first region, If the object detection model has been trained and the second image has been acquired, the detection means detects the first region from the second image using the object detection model or the base model. The information processing apparatus according to claim 2.

7. The information processing apparatus according to claim 6, further comprising a correction means for correcting region information included in the training data in response to user operations.

8. The generation means calculates the anomaly score for each pixel constituting the second image, and generates the anomaly score map by assigning the calculated anomaly score to the pixel. The correction means corrects the first anomaly score assigned to each of the plurality of pixels corresponding to the detected first region to a second anomaly score. The information processing apparatus according to any one of claims 1 to 7.

9. The information processing apparatus according to claim 8, wherein the correction means corrects the abnormal score map by changing the detected first region.

10. The information processing apparatus according to claim 9, wherein the first region is changed to a region in which a buffer is added around the first region.

11. The information processing apparatus according to claim 10, wherein the buffer is determined according to the size of the first region.

12. The information processing apparatus according to claim 8, wherein the second anomaly score is lower than the maximum value of the first anomaly scores assigned to each of the plurality of pixels corresponding to at least the first region.

13. The information processing apparatus according to claim 12, wherein the second anomaly score is a score greater than or equal to the minimum value of the first anomaly scores assigned to each of the plurality of pixels corresponding to the first region.

14. The information processing apparatus according to claim 8, wherein the abnormal score map is corrected by multiplying the abnormal score map by a weight map in which weights are assigned to each pixel constituting the second image.

15. The information processing apparatus according to claim 14, wherein the weight map is configured such that each of at least a plurality of pixels corresponding to the first region is assigned a weight that lowers the first anomaly score.

16. An information processing method performed by an information processing device comprising storage means for storing an anomaly detection model, which is generated by learning a first image captured in a normal environment and used to calculate an anomaly score representing the degree of anomaly occurring in the environment, The steps include acquiring a second image captured under the aforementioned conditions, The steps include: calculating an anomaly score representing the degree of anomaly occurring in the environment where the acquired second image was captured using an anomaly detection model stored in the storage means, and generating an anomaly score map based on the calculated anomaly score; The steps include detecting a first region containing an object present in the environment in which the second image was captured from the second image, The steps include correcting the abnormal score map based on the detected first region, The steps include outputting the corrected abnormal score map and An information processing method comprising the following.

17. A program executed by a computer of an information processing device, which includes storage means for storing an anomaly detection model that is generated by learning a first image taken in a normal environment and is used to calculate an anomaly score representing the degree of anomaly occurring in the environment, To the aforementioned computer, The steps include acquiring a second image captured under the aforementioned conditions, The steps include: calculating an anomaly score representing the degree of anomaly occurring in the environment where the acquired second image was captured using an anomaly detection model stored in the storage means, and generating an anomaly score map based on the calculated anomaly score; The steps include detecting a first region containing an object present in the environment in which the second image was captured from the second image, The steps include correcting the abnormal score map based on the detected first region, The steps include outputting the corrected abnormal score map and A program to execute.

Images