Monitoring method, monitoring device, and computer storage medium

Abstract

The system properly detects openings where the grating has been removed. [Solution] A monitoring method for monitoring the inside of a cleanroom in which a removable grating is placed on the floor, comprising the steps of detecting an opening where the grating has been removed based on an image of the cleanroom captured by an imaging module, wherein the detection step includes (A) extracting an opening candidate which is a portion corresponding to a candidate for the opening from the image to be detected, (B) removing the opening candidate which corresponds to a moving object moving inside the cleanroom based on a plurality of images arranged in chronological order including the image to be detected, and (C) narrowing down the opening candidate based on the color of the opening candidate in the image to be detected.

Description

[Technical Field]

[0001] This disclosure relates to a monitoring method, a monitoring device, and a computer storage medium. [Background technology]

[0002] Patent Document 1 discloses a cleanroom monitoring device for monitoring the interior of a cleanroom in which a removable floor surface is installed on the floor. This monitoring device includes a monitoring camera that images the removable floor surface in a passageway through which workers can pass, detects from the image signal obtained by the monitoring camera whether or not there is an opening where the removable floor surface has been removed, and if there is an opening, detects from the image signal of the monitoring camera whether or not there is a worker approaching the opening. Furthermore, this monitoring device includes a monitoring unit that outputs an alarm signal when a worker is detected, and an alarm generating means that receives the alarm signal from the monitoring unit and issues an alarm. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2014-16799 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] The technology described herein appropriately detects openings where grating has been removed. [Means for solving the problem]

[0005] One aspect of the present disclosure is a monitoring method for monitoring the inside of a cleanroom in which a removable grating is placed on the floor, the method comprising the steps of detecting an opening where the grating has been removed based on an image of the cleanroom captured by an imaging module, the detection step comprising: (A) extracting an opening candidate, which is a portion corresponding to a candidate for the opening, from the image to be detected; (B) removing an opening candidate corresponding to a moving object moving inside the cleanroom based on a plurality of images arranged in chronological order, including the image to be detected; and (C) narrowing down the opening candidates by the color of the opening candidate in the image to be detected. [Effects of the Invention]

[0006] According to this disclosure, it is possible to properly detect openings where the grating has been removed. [Brief explanation of the drawing]

[0007] [Figure 1] This is a block diagram showing the schematic configuration of a monitoring system equipped with a monitoring device according to the first embodiment. [Figure 2] This is a diagram illustrating the placement of surveillance cameras. [Figure 3] This is a diagram showing an example of a reference image. [Figure 4] Figure 1 is a functional block diagram of the control unit of the monitoring device related to image-based monitoring of a cleanroom. [Figure 5] This is a flowchart illustrating an example of the monitoring flow using the monitoring device shown in Figure 4. [Figure 6] This figure shows an example of an image to be detected. [Figure 7] This figure shows an example of a binarized difference image related to the image being detected. [Figure 8] This figure shows an example of an image after the first removal. [Figure 9] This figure shows an example of the first replacement image. [Figure 10] This figure shows an example of an image after the second removal. [Figure 11] This figure shows an example of an image after the third removal. [Figure 12] This figure shows an example of an object detected by a monitoring device. [Figure 13] This is a diagram illustrating an example of setting criteria for potential openings. [Figure 14] This figure illustrates another example of a method for extracting aperture candidates from images of the cleanroom to be detected. [Figure 15] This figure shows an example of an operation screen for setting the region to be extracted as a candidate for aperture. [Figure 16] This is a block diagram showing the schematic configuration of a monitoring system equipped with a monitoring device according to the second embodiment. [Figure 17] This is a diagram illustrating the arrangement of means for generating changes in physical quantities. [Figure 18] Figure 16 is a functional block diagram of the control unit of the monitoring device related to cleanroom monitoring. [Modes for carrying out the invention]

[0008] Traditionally, in semiconductor manufacturing lines for producing semiconductor devices, for example, numerous semiconductor manufacturing devices are arranged in a cleanroom with a clean atmosphere. In such cleanrooms, a grating with numerous grid-like ventilation holes is installed on the floor. In addition, pipes, various electrical equipment, pumps, chemical tanks, etc. are housed in the space below the grating. Furthermore, the grating is removable for the purpose of allowing workers to descend into the space below the grating to perform work in the event of installation, maintenance, or malfunction of these pipes, electrical equipment, pumps, etc.

[0009] In this way, with the grating removed, workers can enter the space below the grating and perform their work. However, when the grating is removed, an opening is created in the floor of the cleanroom, which is formed by numerous gratings, and there is a risk of workers falling through this opening into the space below. In particular, workers performing other tasks on the grating may be unaware that an opening has been created, and warnings have traditionally been issued to workers.

[0010] The technology described in Patent Document 1 detects the presence or absence of an opening where the grating has been removed, based on an image signal obtained from a surveillance camera that images the grating. If an opening is present, it detects the presence or absence of a worker, and outputs an alarm signal when a worker is detected.

[0011] However, when detecting openings where gratings have been removed based on images captured by a camera, there is a possibility of misidentifying non-openings as openings.

[0012] Therefore, the technology described herein appropriately detects openings where gratings have been removed, and more specifically, appropriately detects openings where gratings have been removed based on images taken of a cleanroom.

[0013] The monitoring method and monitoring device according to this embodiment will be described below with reference to the drawings. In this specification, elements having substantially the same functional configuration are denoted by the same reference numerals, and redundant explanations will be omitted.

[0014] [First Embodiment] <Monitoring System> Figure 1 is a block diagram illustrating the schematic configuration of a monitoring system equipped with a monitoring device according to the first embodiment. Figure 2 is a diagram illustrating the arrangement of the monitoring cameras described later. Figure 3 is a diagram illustrating an example of a reference image described later.

[0015] The monitoring system 1 shown in Figure 1 comprises a monitoring device 10 that monitors the floor inside the cleanroom and a monitoring camera 20 that acts as an imaging module to capture images of the floor inside the cleanroom.

[0016] The monitoring device 10 includes a main unit 11, a display unit 12, an alarm unit 13, and an input unit 14.

[0017] The main unit 11 is a computer equipped with, for example, a processor such as a CPU and memory, and has a storage unit 11a and a control unit 11b. The storage unit 11a stores various types of information and includes memory such as RAM and storage devices such as HDDs. The storage unit 11a stores programs, including commands for executing the monitoring flow described later. These programs may have been recorded on a computer-readable storage medium H and installed from the storage medium H to the main unit 11. The storage medium H may be temporary or permanent.

[0018] The display unit 12 displays various images and is composed of, for example, a liquid crystal display, an organic EL display, or the like.

[0019] The alarm unit 13 issues an alarm using at least one of either light or sound. The sound alarm may be in the form of emitting an alarm sound or by reading a warning message aloud. This alarm unit 13 is installed, for example, in a semiconductor manufacturing apparatus SA (see Figure 2) in a cleanroom to be monitored.

[0020] The input unit 14 is for the worker to input instructions and includes, for example, a touch panel or a keyboard.

[0021] As shown in Figure 2, the surveillance camera 20 images the floor of the cleanroom to which the grating G is detachably attached. Semiconductor manufacturing equipment SA, such as coating and developing equipment, is placed on the floor of the cleanroom, which is the target of surveillance camera 20's monitoring. The surveillance cameras 20 are positioned to capture images of at least the area on the floor of the cleanroom where the semiconductor manufacturing equipment SA is not placed, i.e., the area where workers pass through. For example, six surveillance cameras 20 are provided to capture images of the floor along each side of the semiconductor manufacturing equipment SA in a plan view and the floor opposite each corner of the semiconductor manufacturing equipment SA in a plan view. In addition, the six surveillance cameras 20 are positioned so that all the gratings G surrounding the semiconductor manufacturing equipment SA are within the field of view of the surveillance cameras 20.

[0022] The surveillance camera 20 is attached, for example, to the semiconductor manufacturing equipment SA. However, at least a portion of the surveillance camera 20 may be attached to places other than the semiconductor manufacturing equipment SA, specifically, for example, to the ceiling or walls of a cleanroom, or to other equipment within the cleanroom.

[0023] In addition, each of the surveillance cameras 20 captures a color image of the cleanroom within its field of view. Images of the cleanroom captured by each of the surveillance cameras 20 (specifically, color images of the cleanroom including the floor) are output to the monitoring device 10 and stored in the storage unit 11a.

[0024] The monitoring device 10 monitors the cleanroom using the images of the cleanroom captured by each of the monitoring cameras 20. While the monitoring device 10 is monitoring, each of the monitoring cameras 20 takes images sequentially. Furthermore, each of the surveillance cameras 20 captures a color image of the cleanroom before the start of monitoring by the monitoring device 10, i.e., prior to the start of monitoring, under the following conditions: • All gratings G within the field of view are installed (i.e., there are no openings within the field of view where gratings G have been removed). • No moving objects (such as workers) are within the field of view within the cleanroom.

[0025] The image I0 captured in this state, as illustrated in Figure 3, is stored in the memory unit 11a of the surveillance camera 20 as a reference image (hereinafter referred to as reference image I0) to serve as the basis for monitoring. In addition to the grating G, the reference image I0 in Figure 3 also shows the semiconductor manufacturing equipment SA on the floor, the wall W of the cleanroom, etc. This reference image I0 is also used for monitoring the cleanroom by the monitoring device 10.

[0026] For the sake of simplicity, in the following explanation, we will assume that there is only one surveillance camera 20.

[0027] <Control unit 11b of monitoring device 10> Figure 4 is a functional block diagram of the control unit 11b of the monitoring device 10 for monitoring based on images of a cleanroom. As shown in Figure 4, the control unit 11b is realized by a processor such as a CPU reading and executing a program stored in the storage unit 11a, and includes an acquisition unit 100, a detection unit 200, and a notification control unit 300.

[0028] The acquisition unit 100 selects and acquires images to be detected by the detection unit 200 from the images of the cleanroom sequentially captured by the surveillance camera 20, which are stored in the storage unit 11a. The acquisition unit 100 selects the images to be detected in chronological order. Specifically, the acquisition unit 100 extracts, i.e., selects and acquires images of the cleanroom sequentially captured by the surveillance camera 20, which are stored in the storage unit 11a, in batches of n (where n is an integer of 1 or more).

[0029] The detection unit 200 detects the opening from which the grating G has been removed based on the image of the cleanroom to be detected (hereinafter sometimes abbreviated as the image to be detected) acquired by the acquisition unit 100. The detection unit 200 includes an extraction unit 210 and a removal unit 220.

[0030] The extraction unit 210 extracts portions corresponding to candidate apertures (hereinafter sometimes abbreviated as "aperture candidates") from the image to be detected. Specifically, based on the aforementioned reference image and the image to be detected, the extraction unit 210 extracts pixels, i.e., portions, where there is a difference in pixel values ​​between the two images as aperture candidates.

[0031] More specifically, the extraction unit 210 converts the color image to be detected into grayscale and generates a grayscale image to be detected. The extraction unit 210 then extracts pixels in the grayscale image to be detected whose luminance value is lower than or equal to a predetermined threshold compared to the reference grayscale image, i.e., darker areas compared to the reference grayscale image, as aperture candidates. The reference grayscale image is a grayscale version of the color reference image, and is, for example, pre-generated and stored in the storage unit 11a. The reason for extracting darker areas as potential apertures compared to the reference grayscale image is that gratings appear relatively bright in the reference grayscale image, while apertures appear relatively dark in the grayscale image being detected when they are formed.

[0032] In the following example, the number of tones for each color in the color detection target image and the color reference image, as well as the number of tones in the detection target grayscale image and the reference grayscale image, are assumed to be 256 each.

[0033] The removal unit 220 removes specific aperture candidates from the aperture candidates in the detection target image extracted by the extraction unit 210. The removal unit 220 includes a moving object removal unit 221, a specific color removal unit 222, and an object-unit removal unit.

[0034] The moving object removal unit 221 removes candidate openings corresponding to moving objects based on a series of cleanroom images arranged in chronological order, including the image to be detected. Specifically, the series of cleanroom images arranged in chronological order are a series of images to be detected sequentially acquired by the acquisition unit 100, including the current image to be detected. Moving objects also include openings that are closed in a short time. This movable object removal unit 221 has a first removal unit 221a and a second removal unit 221b.

[0035] The first removal unit 221a removes portions corresponding to moving objects from the aperture candidates based on the comparison results of multiple cleanroom images arranged in the above time series. Specifically, the first removal unit 221a removes aperture candidates that do not continuously exist across three detection target images arranged in the time series including the current detection target image, as moving objects. In other words, the first removal unit 221a removes non-applicable aperture candidates as moving objects so that only aperture candidates present in all three detection target images arranged in the above time series remain.

[0036] The second removal unit 221b, like the first removal unit 221a, removes aperture candidates corresponding to moving objects based on the comparison results of multiple cleanroom images arranged in the above time series. However, the second removal unit 221b differs from the first removal unit 221a in that the time periods occupied by the multiple cleanroom images arranged in the above time series used for comparison are different.

[0037] Specifically, the time occupied by the multiple cleanroom images arranged in the above time series used for the above comparison in the second removal unit 221b is longer than the time occupied by the three detection target images used in the first removal unit 221a. More specifically, the second removal unit 221b removes an aperture candidate if, after removal by the first removal unit 221a, the specific color removal unit 222 described later, and the object unit removal unit 223, the aperture candidate does not continuously exist across a predetermined number of detection target images corresponding to a predetermined time including the current detection target image.

[0038] The specific color removal unit 222 narrows down the aperture candidates based on the color of the aperture candidate in the image of the cleanroom to be detected. In other words, the specific color removal unit 222 removes aperture candidates that show a specific color in the color image to be detected from among the aperture candidates in the image to be detected. Narrowing down the aperture candidates can be rephrased as limiting the aperture candidates to a small or narrow range, or identifying the aperture. Specifically, in the image of the cleanroom captured by the surveillance camera 20, the openings are shown in a dark color and the gratings are shown in a non-dark color. Therefore, the specific color removal unit 222 removes the opening candidates so that only the opening candidates with a dark color remain in the color detection target image. In other words, the specific color removal unit 222 removes the opening candidates with a white color in the color detection target image, as well as the opening candidates that are neither white nor black, i.e., colored, in the same image.

[0039] In addition, "white" means, for example, that any of the R, G, or B components of the pixel value exceeds a predetermined threshold (e.g., 20). Furthermore, "not white and not black, i.e., colored" means that at least one of the absolute values ​​of the difference between the R and G components of the pixel value, the absolute value of the difference between the R and G components of the pixel value, or the absolute value of the difference between the R and G components of the pixel value exceeds a predetermined threshold (e.g., 100). In the following, "white" means that all the luminance values ​​of the R, G, and B components of the pixel value are 255, and "black" means that all the luminance values ​​of the R, G, and B components of the pixel value are 0.

[0040] The object-based removal unit 223 detects a set of aperture candidates within the target image as an object, and performs a machine learning-based determination of whether the detected object, i.e., the set of aperture candidates, can be an aperture. Based on the determination result, it narrows down the aperture candidates. The object unit removal unit 223 includes an object detection unit 223a, an area removal unit 223b, and a determination unit 223c.

[0041] The object detection unit 223a detects a set of aperture candidates within the target image, that is, a cluster of pixels that are aperture candidates within the target image, as an object. For example, contour detection and blob detection can be used for this object detection.

[0042] The area removal unit 223b removes objects detected by the object detection unit 223a that have an area less than or equal to a predetermined first threshold, and objects whose area is greater than or equal to a predetermined second threshold (> first threshold). This removes clusters of pixels in the target image that are candidates for apertures but whose area is clearly larger or smaller than the aperture in the cleanroom image.

[0043] The determination unit 223c determines, based on a model created in advance by machine learning, whether each object detected by the object detection unit 223a within the target image, i.e., each set of aperture candidates, can be an image of an aperture.

[0044] The object removal unit 223 removes from the detection target image any objects that were not determined by the determination unit 223c to be potential aperture images, i.e., aperture candidates included in the set of aperture candidates.

[0045] If, after removal by the removal unit 220, a candidate aperture remains in the detected image, the notification control unit 300 controls the display unit 12 and the alarm unit 13 to issue a notification, assuming that an aperture has been detected by the detection unit 200. In this case, the alarm unit 13, under the control of the notification control unit 300, issues an alarm using at least one of light or sound. Also in this case, the display unit 12, under the control of the notification control unit 300, displays at least one of either an image indicating that an aperture has been detected or an image indicating the location of the aperture. The image indicating the location of the aperture is generated, for example, based on information about the candidate apertures remaining in the detected image after removal by the removal unit 220.

[0046] <Monitoring Flow> Next, an example of the monitoring flow by the monitoring device 10 will be explained using Figures 5 to 11, with reference to Figure 3. Figure 5 is a flowchart illustrating an example of the monitoring flow by the monitoring device 10. Figure 6 is a diagram showing an example of an image to be detected. Figure 7 is a diagram showing an example of a binarized difference image applied to the image to be detected, as described later. Figure 8 is a diagram showing an example of a first removal image, as described later. Figure 9 is a diagram showing an example of a first replacement image, as described later. Figure 10 is a diagram showing an example of a second removal image, as described later. Figure 11 is a diagram showing an example of a third removal image, as described later. Figure 12 is a diagram showing an example of an object detected in the monitoring flow by the monitoring device 10.

[0047] (Step S1: Acquisition process) First, as shown in Figure 5, the acquisition unit 100 of the control unit 11b of the monitoring device 10 acquires the nth (where n is an integer of 3 or more) detection target image In from the images of the cleanroom sequentially captured by the monitoring camera 20, which are stored in the storage unit 11a, as the next detection target in the subsequent step S2. In the following explanation, the nth detection target image I n As shown in Figure 6, images P1 to P3 are shown that are not present in the reference image I0 in Figure 3. Furthermore, in the following explanation, images P1 and P2 are in chronological order of the detection target image I n The image I that was detected one step earlier, i.e., the (n-1)th image. n-1 , the (n-2)th detected image I n-2 As also shown, image P3 is the image I that is being detected. n-1 Only shown is the detection target image I n-1 It is assumed that this is not shown.

[0048] (Step S2: Detection process) Next, the detection unit 200 detects the detection target image I acquired in step S1. n Based on this, the opening is detected. This detection step S2 includes an extraction step (step S11) and a removal step (step S12).

[0049] (Step S11: Extraction process) In this step, the extraction unit 210 extracts the detection target image I acquired in step S1 n from which aperture candidates are extracted. Specifically, the extraction unit 210 converts the color detection target image I n to grayscale, and generates a detection target grayscale image I g、n . Then, the extraction unit 210 processes the detection target grayscale image I g、n , and extracts pixels whose luminance value is lower than or equal to a predetermined threshold compared to the grayscaled reference image I0, that is, the reference grayscale image I g、0 , as aperture candidates. More specifically, the extraction unit 210 generates a difference image I g、0 between the reference grayscale image I g、n and the detection target grayscale image I d、n . At the same time, the extraction unit 210 binarizes the difference image I d、n such that in the difference image I d、n , only pixels with luminance values equal to or lower than a predetermined threshold, i.e., the aperture candidates, are white and the other pixels are black, so as to generate the binarized difference image I for the detection target image I illustrated in Figure 7 n t、n . That is, the white pixels in the binarized difference image I t、n are aperture candidates.

[0050] (Step S12: Removal Step) In this step, the removal unit 220 removes specific aperture candidates from the aperture candidates in the detection target image I extracted in step S11 n . The removal step of step S12 includes a moving object removal step (step S21), a specific color removal step (step S22), an object unit removal step (step S23), and a time determination step (step S24).

[0051] (Step S21: Moving Object Removal Step) In this step, the first removal unit 221a removes the portion corresponding to the moving object from the aperture candidates based on the comparison result of a plurality of clean room images including the detection target image I n . ​Specifically, the first removal unit 221a removes the current detection target image I n Detected image I including n , I n-1 , I n-2 Within this process, aperture candidates that do not exist consecutively across all three images are removed as moving objects. More specifically, the first removal unit 221a removes the detected image I n Binarized difference image I t、n Among the white pixels, the detection target image I n-1 , I n-2 Binarized difference image I t、n-1 , I t、n-2 Pixels that are black in any of the following ways are blackened to obtain the first removed image I, as illustrated in Figure 8. r1、n Generates the first image after removal. r1、n In this case, white pixels are candidates for aperture.

[0052] (Step S22: Specific color removal process) In this process, the specific color removal unit 222 removes the color from the target image I n Depending on the color in the image I, n The candidates for the opening inside are narrowed down. This specific color removal step S22 includes a colored candidate removal step (step S22a) and a white candidate removal step (step S22b).

[0053] (Step S22a: Colored candidate removal process) In this process, the specific color removal unit 222 detects the target image I n From the aperture candidates within, the color detection target image I n In this context, non-white and non-black aperture candidates, i.e., colored aperture candidates, are removed. Specifically, the specific color removal unit 222 removes the color from the target image I n In the image I, pixels where at least one of the absolute values ​​of the difference between the R and G components of the pixel value, or the absolute value of the difference between the R and G components of the pixel value exceeds a predetermined threshold are detected. n Remove from the list of potential openings within the structure.

[0054] More specifically, for example, first, the specific color removal unit 222 removes the first color from the image I r1、n The pixel values ​​of white pixels in the color detection target image I n Replace with the pixel values, and the first image after removal I r、n The pixel values ​​of the black pixels in the image are replaced with a color having only the R, G, or B component of the highest luminance value (255). As a result, the specific color removal unit 222 produces the first replaced image I illustrated in Figure 9. c1、n Generates. Next, the specific color removal unit 222 processes the first replacement image I c1、n The R, G, and B components are extracted from the first replacement image I to generate an image. In other words, the specific color removal unit 222 generates an image from the first replacement image I c1、n Regarding the R component brightness image I Rc、n , G component brightness image I Gc、n and the luminance image I of component B Bc、n Generates. Subsequently, the specific color removal unit 222 removes the R component from the luminance image I Rc、n and the brightness image of the G component I Gc、n Difference image I R-G、d、n It generates the following. At the same time, the specific color removal unit 222 generates the difference image I R-G、d、n In the difference image I, pixels with a brightness value below a predetermined threshold become white, and all other pixels become black. R-G、d、n The image is binarized, and the binarized difference image I is obtained for the R component and the G component. R-G、t、n Generates. Similarly, the specific color removal unit 222 removes the luminance image I of the G component. Gc、n and the brightness image of the G component I Bc、n Based on this, a binarized difference image I of the G component and the B component. G-B、t、n It generates the following. In addition, the specific color removal unit 222 generates the luminance image I of component B. Bc、n and the R component luminance image I Rc、n Based on this, a binarized difference image I of the B component and the R component. B-G、t、n Generates. Then, the specific color removal unit 222 processes the binarized difference image I R-G、t、n , I G-B、t、n , I R-G、t、nIt extracts pixels that are white in all of them. In other words, the specific color removal unit 222 extracts the second post-removal image I illustrated in Figure 10. r2、n Generates the second image after removal. r2、n This is a binarized difference image I R-G、t、n , I G-B、t、n , I R-G、t、n Only pixels that are white in all aspects appear white, while all other pixels appear black. Also, the second image after removal I r2、n In this image, the white pixels are candidates for aperture, and the first image after removal is I. r1、n This is the result of removing the white pixel group P21, which corresponds to the colored aperture candidates.

[0055] (Step S22b: White candidate removal process) In this process, the specific color removal unit 222 detects the target image I n From the aperture candidates within, the color detection target image I n Remove any white-ish aperture candidates. Specifically, the color image to be detected I n In this process, pixels in which any of the R, G, or B components of the pixel value exceeds a predetermined threshold (e.g., 20) are detected in the target image I. n Remove from the list of potential openings within the structure.

[0056] More specifically, for example, first, the specific color removal unit 222 removes the second color from the image I r2、n The pixel values ​​of white pixels in the color detection target image I n Replace with the pixel values, and the first image after removal I r、n The pixel values ​​of black pixels in the image are replaced with white pixels. As a result, the specific color removal unit 222 generates the second replacement image I c2、n Generates. Next, the specific color removal unit 222 processes the second replacement image I such that pixels where any of the R, G, or B components of the pixel value exceeds a predetermined threshold (e.g., 20) become black, and the others become white. c2、n Convert the image and create the third removed image I, as illustrated in Figure 11. r3、n Generates the third removal image I. r3、n In this image as well, the white pixels are candidates for aperture. Also, the third image after removal I r3、nis the second post-removal image I r2、n from which the white pixel group P22 corresponding to the white-based opening candidate is excluded.

[0057] (Step S23: Object unit removal step) In this step, the object unit removal unit 223 detects a set of opening candidates in the detection target image I n as an object, performs determination by machine learning on whether the detected object, that is, the set of opening candidates, can be an opening, and narrows down the opening candidates based on the determination result. The object unit removal step of step S23 includes an object detection step (step S23a), an area removal step (step S23b), and an opening determination step (step S23c).

[0058] (Step S23a: Object detection step) In this step, the object detection unit 223a detects a set of opening candidates in the detection target image I n as an object. Specifically, the object detection unit 223a performs contour detection and blob detection on the third post-removal image I r3、n , and detects, as an object, a mass of white pixels that are opening candidates in the third post-removal image I r3、n . Thereby, objects B1 and B2 are detected from the third post-removal image I r3、n as shown in, for example, FIG. 12.

[0059] (Step S23b: Area removal step) In this step, the area removal unit 223b removes, from among the objects (that is, the masses of white pixels) in the third post-removal image I r3、n detected in step S23a, those having an area equal to or less than a predetermined first threshold and those having an area equal to or greater than a predetermined second threshold (> the first threshold). Note that if there is no corresponding object, no object removal is performed.

[0060] (Step S23c: Opening determination step) In this step, the determination unit 223c determines, for each object detected in step S23c, whether the object can be an image of an opening based on a model created in advance by machine learning. Specifically, for each object that is detected in step S23a and remains after step S23b, the determination unit 223c determines whether the object can be an image of an opening based on a model created in advance by machine learning. For example, the determination unit 223c determines, for each of the objects B1 and B2 in Figure 12, whether the object can be an image of an opening based on a model created in advance by machine learning

[0061] Then, as a result of the determination, when it is not determined that the image can be an image of an opening, the object unit removal unit 223 removes the corresponding object, that is, the opening candidate included in the set of opening candidates, from the opening candidates in the detection target image I n . On the other hand, as a result of the determination, when it is determined that the image can be an image of an opening, the object unit removal unit 223 does not remove the corresponding object, that is, the opening candidate included in the set of opening candidates, from the opening candidates in the detection target image I n .

[0062] (Step S24: Time determination step) In this step, when an opening candidate remains in the detection target image I even after step S23c, the second removal unit 221b determines whether the opening candidate exists continuously for a predetermined time or more. The predetermined time is at least longer than the time occupied by three time-series detection target images I n , and is, for example, 10 seconds. n、 I n-1 , I n-2 In this step, specifically, when an opening candidate remains in the detection target image I even after step S23c, the second removal unit 221b determines, for example, whether the opening candidate exists continuously over a number of detection target images that are arranged in time series and include the detection target image I and that is equal to or greater than the number corresponding to the predetermined time. In this step, specifically, when an opening candidate remains in the detection target image I even after step S23c, the second removal unit 221b determines, for example, whether the opening candidate exists continuously over a number of detection target images that are arranged in time series and include the detection target image I and that is equal to or greater than the number corresponding to the predetermined time. n In this step, specifically, when an opening candidate remains in the detection target image I even after step S23c, the second removal unit 221b determines, for example, whether the opening candidate exists continuously over a number of detection target images that are arranged in time series and include the detection target image I and that is equal to or greater than the number corresponding to the predetermined time. n In this step, specifically, when an opening candidate remains in the detection target image I even after step S23c, the second removal unit 221b determines, for example, whether the opening candidate exists continuously over a number of detection target images that are arranged in time series and include the detection target image I and that is equal to or greater than the number corresponding to the predetermined time. That is, this determination is made based on the comparison result of a plurality of time-series detection target images including the detection target image I, and the total number of detection target images used for this determination is equal to or greater than the number corresponding to the predetermined time. n That is, this determination is made based on the comparison result of a plurality of time-series detection target images including the detection target image I, and the total number of detection target images used for this determination is equal to or greater than the number corresponding to the predetermined time.

[0063] In this process, more specifically, the second removal unit 221b determines whether the corresponding aperture candidate still exists after step S23c in all of the detection target images, for a number equivalent to or greater than the predetermined time.

[0064] If the determination result does not indicate that any remaining opening candidates exist continuously for a predetermined period of time after step S23c (i.e., NO), the detection unit 200 processes the remaining opening candidates after step S23c as if they had been removed as moving objects by the second removal unit 221b. Then, the detection unit 200 returns the monitoring flow to step S1, assuming that no openings have been detected.

[0065] On the other hand, if the determination in step S24 is found to indicate that the remaining candidate openings after step S23c have been continuously present for a predetermined period of time or longer (in the case of NO), the monitoring flow proceeds to step S3, assuming that candidate openings remain after removal by the removal unit 220 and that the detection unit 200 has detected an opening.

[0066] (Step S3: Hochi) Then, the notification control unit 300 controls the display unit 12 and the alarm unit 13 to notify that an opening has been detected. Specifically, under the control of the notification control unit 300, the alarm unit 13 issues an alarm using at least one of light or sound. Also, under the control of the notification control unit 300, the display unit 12 displays, for example, at least one of the images indicating the location of the opening. The image indicating the location of the opening is, for example, an image of the cleanroom currently being captured by the surveillance camera 20, with the outline of the object determined to be an opening in the opening determination step of step S23c superimposed in a highlighted color.

[0067] <Main effects> As described above, the monitoring method for monitoring a cleanroom in which a grating G is placed on the floor, according to this embodiment, includes a step of detecting an opening where the grating G has been removed, based on an image of the cleanroom captured by a monitoring camera 20. The detection step includes (A) a step of extracting opening candidates from an image of the cleanroom to be detected, i.e., the detection target image; (B) a step of removing opening candidates corresponding to moving objects based on a plurality of images arranged in chronological order including the detection target image; and (C) a step of narrowing down the opening candidates by the color of the opening candidates in the detection target image. Therefore, according to this embodiment, when detecting an opening where the grating G has been removed based on an image of the cleanroom captured by the surveillance camera 20, it is possible to suppress the detection of a moving object as an opening. In other words, according to this embodiment, the opening can be appropriately detected.

[0068] Furthermore, in this embodiment, as a method for removing aperture candidates that correspond to moving objects, a method is employed in which aperture candidates that do not appear consecutively across three detection target images arranged in a time series, including the image of the cleanroom currently being detected, are removed as moving objects. Another method for removing moving objects, different from this embodiment, involves removing the entire set of aperture candidates when the centroid of that set of aperture candidates shifts between detection target images arranged in chronological order. In this method, if a moving object and an aperture overlap, removing the entire set of aperture candidates will also remove the aperture candidate corresponding to the actual aperture. Compared to this method, the method for removing aperture candidates corresponding to moving objects adopted in this embodiment can suppress the removal of aperture candidates that actually correspond to the aperture.

[0069] Furthermore, in the monitoring method according to this embodiment, machine learning is used to determine whether an object could be an image of an aperture, based on a set of aperture candidates, that is, on an object-by-object basis within the image to be detected. Therefore, in this embodiment, the amount of computation required for determination is less compared to the case where machine learning is used to determine whether an aperture exists on an image-by-image basis. Consequently, the monitoring method according to this embodiment can be suitably used for real-time monitoring. Furthermore, when machine learning is used to determine whether an opening exists on a per-image basis, even if an opening is determined to exist, it is difficult to pinpoint the location where the opening is formed. In contrast, as in this embodiment, when machine learning is used to determine whether an object within the image can be an opening, it is easy to pinpoint the location where the opening is formed.

[0070] Furthermore, in the monitoring method according to this embodiment, after narrowing down the candidates for openings by removing candidates corresponding to moving objects, the machine learning judgment described above is performed on the set of candidates for openings, i.e., on individual objects. Therefore, the amount of computation required for the judgment can be reduced compared to the case where the machine learning judgment described above is performed without narrowing down the candidates for openings.

[0071] <Example of setting criteria for potential openings> The criteria for candidate apertures (specifically, for example, at least one of the threshold used in the binarization step in step S11 or the first threshold used in the area removal step in step S23b) are set as follows, for example.

[0072] Specifically, as shown in Figure 13, first, an adjustment jig Z, which is a reference setting component, is placed on the grating Gl furthest from the surveillance camera 20 within the area included in the detection target image, i.e., within the field of view F of the surveillance camera 20. The adjustment jig Z is a thin, sheet-like component and has the same color as the opening. The size of the adjustment jig Z is approximately equal to that of one grating, and the adjustment jig Z is placed so as to cover the entire furthest grating Gl.

[0073] After the adjustment jig Z is placed, the inside of the cleanroom is imaged by the surveillance camera 20. Then, based on the imaging results from the surveillance camera 20, the criteria for the aperture candidates described above are set. Specifically, for example, a threshold used in the binarization step in step S11 is set based on the pixel values ​​of the adjustment jig Z in the image of the cleanroom captured by the surveillance camera 20. Also, for example, a first threshold used in the area removal step in step S23b is set based on the area of ​​the adjustment jig Z in the image of the cleanroom captured by the surveillance camera 20.

[0074] By setting criteria for potential openings in this way, it becomes possible to more appropriately detect openings based on images of the cleanroom captured by the surveillance camera 20, regardless of the installation environment of the surveillance camera 20 (specifically, the illuminance inside the cleanroom, the type of light source used in the cleanroom, and the installation location of the surveillance camera 20).

[0075] <Other examples of methods for extracting aperture candidates from the target image> In the above example, the extraction unit 210 extracted aperture candidates from the entire image to be detected. In other words, in the above example, the region from which the extraction unit 210 extracted aperture candidates was the entire image to be detected. Alternatively, if the image of the cleanroom captured by the surveillance camera 20 is divided into multiple vertical and horizontal sections to form smaller regions, the following may be used: The extraction unit 210, as shown in Figure 14, excludes the pre-selected divided region SRN from the divided region SR that does not contain the grating G, and then extracts the detection target image I n From this, aperture candidates may be extracted. That is, divided region SRNs that do not include the pre-selected grating G may be excluded, or masked, from the region targeted for extraction of aperture candidates by the extraction unit 210. Specifically, in the extraction step of step S11, divided region SRNs that do not include the pre-selected grating G may be excluded from the region in which the extraction unit 210 calculates the difference with the grayscale image to be detected. This reduces the computational load required to extract potential openings. Furthermore, since no processing other than the extraction of potential openings (for example, machine learning-based determination by the determination unit 223c) is performed on the pre-selected divided region SRN that does not include the grating G, the computational load is reduced from this perspective as well.

[0076] The selection of a divided area SRN that does not include the grating G is performed by an operator via, for example, the operation screen U illustrated in Figure 15, specifically, by inputting via the input unit 14 while the operation screen U is displayed on the display unit 12. The operation screen U includes an area U1 that displays the image of the cleanroom captured by the surveillance camera 20 so that each divided area is visible, and an area U2 that displays a toggle button to switch whether or not to exclude each divided area from the area to be extracted as an opening candidate.

[0077] [Second Embodiment] <Monitoring System> Figure 16 is a block diagram illustrating the schematic configuration of a monitoring system equipped with a monitoring device according to the second embodiment. Figure 17 is a diagram illustrating the arrangement of the means for generating changes in physical quantities, which will be described later. Figure 18 is a functional block diagram of the control unit of the monitoring device of Figure 16 related to the monitoring of a clean room.

[0078] The monitoring system 1A in Figure 16 is used when the grating G installed on the floor of a cleanroom, as shown in Figure 17, includes a portion of transparent grating Gt. Similar to the monitoring system 1A in Figure 1, it includes a monitoring device 10A and a monitoring camera 20. However, the transparent grating Gt shows little change in the imaging results from the monitoring camera 20 between when it is installed and when it is removed, creating an opening. Therefore, the monitoring system 1A in Figure 16 further includes a means for generating changes in physical quantities 30 and a detection unit 40 for detecting openings caused by the transparent grating Gt. The grating Gt is made transparent for purposes such as visually inspecting the condition of the area under the floor when the grating Gt is installed.

[0079] As shown in Figure 17, the change generating means 30 is arranged to extend across multiple gratings G, including a transparent grating Gt. The change generating means 30 changes a physical quantity (specifically, an electrical characteristic quantity such as voltage or current) when the transparent grating Gt is lifted upward. The change generating means 30 disclosed in Japanese Patent Application Publication No. 2020-17105 can be used. Furthermore, the change generating means 30 is formed to be thin in plan view so as not to cover the entire transparent grating Gt, that is, so as not to significantly reduce visibility of the area under the floor through the grating Gt.

[0080] The detection unit 40 detects the physical quantity that changes when the change generating means 30 is lifted upward. The detection result of the detection unit 40 is output to the monitoring device 10A.

[0081] Furthermore, the configuration of the control unit of monitoring system 1A differs from that of monitoring system 1. In addition to the configuration of the control unit 11b of monitoring device 10 in Figure 4, the control unit 11Ab of monitoring system 1A further includes a detection unit 400, which is realized by a processor such as a CPU reading and executing a program stored in the storage unit 11a.

[0082] The detection unit 400 detects the opening created by the removal of the transparent grating Gt based on the change in a physical quantity generated by the change generating means 30. The detection unit 400 detects the opening created by the removal of the transparent grating Gt based on the detection result of the detection unit 40 of the physical quantity changed by the change generating means 30. For example, the detection unit 400 detects the opening created by the transparent grating Gt when the transparent grating Gt is lifted and the state in which voltage or current was detected in the closed circuit including the change generating means 30 changes from a state in which it is not detected by the detection unit 40.

[0083] Thus, according to this embodiment, it is possible to detect openings made of transparent grating Gt.

[0084] Furthermore, if there are multiple transparent gratings Gt, the physical quantity that changes when the grating Gt is lifted by the change generating means 30 may be different for each transparent grating Gt. In this case, the position of the opening from which the transparent grating Gt was removed can be identified based on the amount of change in the physical quantity detected by the detection unit 40. For example, in the change generating means 30, electrical resistance members (not shown) with different electrical resistance values ​​for each transparent grating Gt may be provided in the part corresponding to each transparent grating Gt. In this example, when a certain transparent grating Gt is lifted, the amount of change in voltage or current detected by the detection unit 40 in the closed circuit including the change generating means 30 will change for each transparent grating Gt (i.e., for each opening position). Using this, the detection unit 400 can identify the position of the opening from which the transparent grating Gt was removed based on the amount of change in voltage or current corresponding to the position of that opening.

[0085] <Variation> In the above example, the white candidate removal process was performed after the colored candidate removal process, but the colored candidate removal process may be performed after the white candidate removal process. Furthermore, it is not necessary to perform all the steps included in the monitoring flow on a single device. For example, the first half of the monitoring flow may be performed by a control unit on a worker's mobile device, while the second half, which includes machine learning-based decision-making, may be performed by a control unit on a management computer that controls all the equipment in the cleanroom, including the semiconductor manufacturing equipment SA.

[0086] The monitoring method according to this embodiment can be applied even before the semiconductor manufacturing equipment SA is placed within the field of view of the monitoring camera 20 inside the cleanroom.

[0087] The embodiments disclosed herein should be considered in all respects as illustrative and not restrictive. The embodiments described above may be omitted, replaced, or modified in various ways without departing from the scope and spirit of the appended claims. For example, the constituent elements of the embodiments described above can be combined in any way. Such any combination will naturally yield the functions and effects of each constituent element in the combination, as well as other functions and effects that will be apparent to those skilled in the art from the description herein.

[0088] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or instead of the effects described herein.

[0089] Furthermore, the following configuration examples also fall within the technical scope of this disclosure. (1) A monitoring method for monitoring the inside of a cleanroom in which a removable grating is installed on the floor, The process includes detecting the opening where the grating has been removed based on the image of the cleanroom captured by the imaging module, The aforementioned detection step is, (A) A step of extracting an aperture candidate, which is a portion corresponding to a candidate for the aperture, from the image to be detected, (B) A step of removing the candidate openings corresponding to a moving object moving within the cleanroom based on a plurality of images arranged in a time series including the image of the object to be detected, (C) A monitoring method comprising the step of narrowing down the aperture candidates based on the color of the aperture candidates in the image of the object to be detected. (2) The (B) process has (a) and (b) processes, Step (a) and Step (b) respectively remove the portion corresponding to the moving body from the candidate aperture based on the comparison results of the plurality of images. The monitoring method according to (1), wherein the time occupied by the plurality of images used for comparison is different in step (a) and step (b). (3) The monitoring method according to (2), wherein step (a) removes as moving objects any aperture candidates that are not consecutively present across three images of the detection target arranged in a time series including the current image of the detection target. (4) The monitoring method according to (3), wherein the time occupied by the image used in step (b) is longer than the time occupied by the three images used in step (a). (5) The monitoring method according to any one of (1) to (4), wherein step (C) removes aperture candidates whose color is white in the image of the object to be detected, and aperture candidates whose color is neither white nor black in the image of the object to be detected. (6) The monitoring method according to any one of (1) to (5) above, wherein the detection step further includes detecting the set of candidate openings as an object, performing a machine learning determination as to whether the detected object could be the opening, and narrowing down the candidate openings based on the result of the determination. (7) The monitoring method according to any one of (1) to (6), wherein the criteria for the candidate opening are set based on the results of imaging by the imaging module of a member that is located in the grating furthest from the imaging module within the range included in the image and has the same color as the opening. (8) The transparent grating is placed on the floor, A means for generating a change in a physical quantity is arranged extending from the upper surface of the grating, spanning across a plurality of the gratings, including the transparent grating. The aforementioned change generating means changes a physical quantity when the transparent grating is lifted upwards. The monitoring method is, The monitoring method according to any one of (1) to (7), further comprising the step of detecting an opening from which the transparent grating has been removed based on a change in a physical quantity by the change generating means. (9) When the above image is divided into multiple vertical and horizontal sections, the small regions obtained by dividing the image into multiple sections are defined as divided regions, The monitoring method according to any one of (1) to (8), wherein step (A) is to extract the aperture candidates from the image of the object to be detected, from which the divided regions that do not include the grating, which have been pre-selected from the divided regions, have been excluded. (10) A readable computer storage medium containing a program that runs on a computer of a control unit that controls a monitoring device, which causes the monitoring device to perform a monitoring method for monitoring the inside of a cleanroom in which a removable grating is placed on the floor, The monitoring method includes the step of detecting the opening from which the grating has been removed based on the image of the cleanroom captured by the imaging module. The aforementioned detection step is, (A) A step of extracting an aperture candidate, which is a portion corresponding to a candidate for the aperture, from the image to be detected, (B) A step of removing from the candidate opening a portion corresponding to a moving body moving within the cleanroom based on a plurality of images arranged in a time series including the image of the object to be detected, (C) A computer storage medium comprising the step of narrowing down the aperture candidates by the color of the aperture candidates in the image of the object to be detected. (11) A monitoring device for monitoring the inside of a cleanroom in which a removable grating is installed on the floor, The system has a detection unit that detects the opening where the grating has been removed, based on the image of the cleanroom captured by the imaging module. The detection unit, An extraction unit extracts candidate openings corresponding to the openings from the image to be detected, A moving object removal unit removes opening candidates corresponding to moving objects moving within the cleanroom based on a plurality of images arranged in a time series, including the image of the object to be detected. A monitoring device comprising a specific color removal unit that narrows down the aperture candidates based on the color of the aperture candidate in the image of the object to be detected. (12) The moving body removal unit has a first removal unit and a second removal unit, The first removal unit and the second removal unit each remove the portion corresponding to the moving body from the candidate opening based on the comparison results of the plurality of images. The monitoring device according to (11), wherein the time occupied by the plurality of images used for comparison is different between the first removal unit and the second removal unit. (13) The monitoring device according to (12), wherein the first removal unit removes, as the moving body, the aperture candidate that does not exist consecutively across three images of the detection target arranged in a time series including the current image of the detection target. (14) The monitoring device according to (13), wherein the time occupied by the image used in the second removal unit is longer than the time occupied by the three images used in the first removal unit. (15) The monitoring device according to any one of (11) to (14), wherein the specific color removal unit removes the aperture candidates whose color is white in the image of the object to be detected, and the aperture candidates whose color is neither white nor black in the image of the object to be detected. (16) The monitoring device according to any one of (11) to (15), wherein the detection unit further includes an object unit removal unit that detects the set of candidate openings as an object, performs a machine learning determination as to whether the detected object could be the opening, and narrows down the candidate openings based on the result of the determination. (17) The monitoring device according to any one of (11) to (16), wherein the criteria for the candidate opening are set based on the results of imaging by the imaging module of a member that is located in the grating furthest from the imaging module within the range included in the image and has the same color as the opening. (18) The transparent grating is placed on the floor, A means for generating a change in a physical quantity is arranged extending from the upper surface of the grating, spanning across a plurality of the gratings, including the transparent grating. The aforementioned change generating means changes a physical quantity when the transparent grating is lifted upwards. The monitoring device according to any one of (11) to (17), further comprising another detection unit that detects an opening from which the transparent grating has been removed based on a change in a physical quantity by the change generating means. (19) When the above image is divided into multiple vertical and horizontal sections, the small regions obtained by dividing the image into multiple sections are defined as divided regions, The monitoring device according to any one of (11) to (18), wherein the extraction unit extracts the aperture candidates from the image of the object to be detected, from which the pre-selected division areas that do not include the grating have been excluded. [Explanation of Symbols]

[0090] 10, 10A monitoring device 20 surveillance cameras 200 detection unit 210 Extraction part 220 Removal section 221 Mobile object removal unit 222 Specific color removal section I n nth detected image I n-1 n-1st detected image I n-2 (n-2nd) detected image

Claims

1. A monitoring method for monitoring the inside of a cleanroom in which a removable grating is installed on the floor, The process includes detecting the opening where the grating has been removed based on the image of the cleanroom captured by the imaging module, The aforementioned detection step is, (A) A step of extracting an aperture candidate, which is a portion corresponding to a candidate for the aperture, from the image to be detected, (B) A step of removing the candidate openings corresponding to a moving object moving within the cleanroom based on a plurality of images arranged in a time series including the image of the object to be detected, (C) A monitoring method comprising the step of narrowing down the aperture candidates based on the color of the aperture candidates in the image of the object to be detected.

2. The (B) process comprises the (a) process and the (b) process, Step (a) and Step (b) each remove the portion corresponding to the moving body from the candidate aperture based on the comparison results of the plurality of images. The monitoring method according to claim 1, wherein the time occupied by the plurality of images used for comparison is different in step (a) and step (b).

3. The monitoring method according to claim 2, wherein step (a) removes as the moving object any aperture candidate that does not exist consecutively across three images of the detection target arranged in a time series including the current image of the detection target.

4. The monitoring method according to claim 3, wherein the time occupied by the image used in step (b) is longer than the time occupied by the three images used in step (a).

5. The monitoring method according to claim 1 or 2, wherein step (C) removes aperture candidates whose color is white in the image of the object to be detected, and aperture candidates whose color is neither white nor black in the image of the object to be detected.

6. The monitoring method according to claim 1 or 2, wherein the detection step further includes detecting the set of candidate openings as an object, performing a machine learning determination as to whether the detected object could be the opening, and narrowing down the candidate openings based on the result of the determination.

7. The monitoring method according to claim 1 or 2, wherein the criteria for the candidate opening are set based on the results of imaging by the imaging module of a member that is located in the grating furthest from the imaging module within the range included in the image and has the same color as the opening.

8. The transparent grating is placed on the floor, A means for generating a change in a physical quantity is arranged extending from the upper surface of the grating, spanning across a plurality of the gratings, including the transparent grating. The aforementioned change generating means changes a physical quantity when the transparent grating is lifted upwards. The monitoring method is, The monitoring method according to claim 1 or 2, further comprising the step of detecting an opening from which the transparent grating has been removed based on a change in a physical quantity by the change generating means.

9. When the above image is divided into multiple vertically and horizontally, and these small regions are defined as divided regions, The monitoring method according to claim 1 or 2, wherein step (A) is to extract the aperture candidates from the image of the object to be detected, from which the divided regions that do not include the grating, which have been pre-selected, have been excluded.

10. A readable computer storage medium containing a program that runs on a computer of a control unit that controls a monitoring device, which causes the monitoring device to perform a monitoring method for monitoring the inside of a cleanroom in which removable gratings are installed on the floor, The monitoring method includes the step of detecting the opening from which the grating has been removed based on the image of the cleanroom captured by the imaging module. The aforementioned detection step is, (A) A step of extracting an aperture candidate, which is a portion corresponding to a candidate for the aperture, from the image to be detected, (B) A step of removing from the candidate opening a portion corresponding to a moving body moving within the cleanroom based on a plurality of images arranged in a time series including the image of the object to be detected, (C) A computer storage medium comprising the step of narrowing down the aperture candidates based on the color of the aperture candidates in the image of the object to be detected.

11. A monitoring device for monitoring the inside of a cleanroom where a removable grating is installed on the floor, The system has a detection unit that detects the opening where the grating has been removed, based on the image of the cleanroom captured by the imaging module. The detection unit, An extraction unit extracts candidate openings corresponding to the openings from the image to be detected, A moving object removal unit removes opening candidates corresponding to moving objects moving within the cleanroom based on a plurality of images arranged in a time series, including the image of the object to be detected. A monitoring device comprising a specific color removal unit that narrows down the aperture candidates based on the color of the aperture candidate in the image of the object to be detected.

12. The aforementioned moving body removal unit has a first removal unit and a second removal unit. The first removal unit and the second removal unit each remove the portion corresponding to the moving body from the candidate opening based on the comparison results of the plurality of images. The monitoring device according to claim 11, wherein the time occupied by the plurality of images used for comparison is different between the first removal unit and the second removal unit.

13. The monitoring device according to claim 12, wherein the first removal unit removes, as the moving body, the aperture candidates that do not continuously exist across three images of the detection target arranged in a time series including the current image of the detection target.

14. The monitoring device according to claim 13, wherein the time occupied by the image used in the second removal unit is longer than the time occupied by the three images used in the first removal unit.

15. The monitoring device according to claim 11 or 12, wherein the specific color removal unit removes aperture candidates whose color is white in the image of the object to be detected, and aperture candidates whose color is neither white nor black in the image of the object to be detected.

16. The monitoring device according to claim 11 or 12, wherein the detection unit further includes an object-unit removal unit that detects the collection of candidate openings as an object, performs a machine learning determination as to whether the detected object could be the opening, and narrows down the candidate openings based on the result of the determination.

17. The monitoring device according to claim 11 or 12, wherein the criteria for the candidate opening are set based on the results of imaging by the imaging module of a member that is located in the grating furthest from the imaging module within the range included in the image and has the same color as the opening.

18. The transparent grating is placed on the floor, A means for generating a change in a physical quantity is arranged extending from the upper surface of the grating, spanning across a plurality of the gratings, including the transparent grating. The aforementioned change generating means changes a physical quantity when the transparent grating is lifted upwards. The monitoring device according to claim 11 or 12, further comprising another detection unit that detects an opening from which the transparent grating has been removed based on a change in a physical quantity by the change generating means.

19. When the above image is divided into multiple vertically and horizontally, and these small regions are defined as divided regions, The monitoring device according to claim 11 or 12, wherein the extraction unit extracts the aperture candidates from the image of the object to be detected, from which the pre-selected division areas that do not include the grating have been excluded.

Images