Particle appearance monitoring apparatus and particle appearance monitoring method

Abstract

To properly detect a foreign matter even when the size and quantity of the appeared foreign matter and a plurality of kinds of foreign matters exist.SOLUTION: The image processing device includes: a camera (26) provided with a light irradiation unit which performs light irradiation, and a camera body unit which receives light reflected from the irradiated light by a light receiving unit and obtains one frame image of a matrix having a plurality of pixels which convert the light into multi-level pixel data in accordance with intensity of the received light; an image holding unit (31) which holds the image data; an image holding control means (32) which obtains the image data in time series by the camera and holds the image data in the image holding unit; and a controller which, based on comparison between the image data at a predetermined point of time held in the image holding unit (31) and the image data before the predetermined point of time or the image data obtained in advance when a foreign object does not appear, this device is provided with a foreign matter appearance / range detecting means 3 for detecting the appearance and range of the foreign matter and a foreign matter kind detecting means 3 for detecting the kind of the foreign matter existing in the range of the foreign matter detected by the means 3.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a foreign matter appearance monitoring processing device and a foreign matter appearance monitoring processing method. [Background technology]

[0002] Conventionally, in devices that use liquid inside a housing, leak detection is performed using a liquid leak detection tape. For example, as shown in Fig. 1, a liquid leak detection tape 120 is attached to a desired position on a floor surface 110 on which a device 100 is placed to detect leaks.

[0003] However, the leakage 130 spreads from time t1 (Fig. 1(A)) to time t2 (Fig. 1(B)) to time t3 (Fig. 1(C)), and at time t3, the leakage is finally detected. Therefore, it takes time for the leakage 130 to reach the leakage detection tape 120 from time t1 when the leakage started, and the damage may become greater.

[0004] Furthermore, when attempting to simultaneously detect a wide area, the liquid leak detection tape 120 requires the installation of many sensors because it detects leaks by point or line. It is not possible to easily determine the path of the leak in a continuous time series. When there are many types of leaks to be detected, methods that detect leaks that pass through points or lines are not sufficient. Once the leak has run out, it becomes difficult to identify the source of the leak. Water leak detection tape cannot detect oil. Water leak detection tape cannot detect leaks unless water reaches the installation location. It may also fail to detect minute leaks even when water reaches the installation location. Physical changes to the water leak detection tape, such as peeling, cutting, or the attachment of foreign objects outside the detection target, can lead to false detections. Another problem is the inability to determine the amount of leaked liquid. Infrared detection, when the background is black, such as a floor, can sometimes be mistakenly detected as a leak. False detections can also occur due to influences from the surrounding environment, such as reflection or absorption of irradiated light. In addition, in Figure 1(C) at time t3, leakage traces 131 remain, but they are not necessarily clear, and the leakage traces may also disappear, making it difficult to identify the source of the leakage.

[0005] Patent Document 1 discloses a sheet for detecting water leaks. Patent Document 2 discloses a sheet-like water leak detection device that detects the occurrence of water leaks based on the pressing force caused by the expansion of an absorbing expansion member. Patent Document 3 discloses a device that has a water collecting section that accumulates leaked water inside an electrical device, an optical fiber with a part disposed in the water collecting section, and detects water leaks by emitting light from the end of the optical fiber and receiving it with an optical fiber led from the water collecting section, and comparing the incident light with the received light. Patent Document 4 discloses a device that has a water collector and a condensation sensor installed in the water collector to detect water leaks.

[0006] Patent Document 5 discloses an intruder detection device that performs image recognition using image data captured by a camera and is capable of separately detecting intruders, small animals, etc. This device is composed of an imaging means, an A / D, an image storage means that cyclically stores image data, an image area setting means that sets an image area to be recognized, a difference calculation unit that calculates and outputs difference image data between background image data and current monitored image data, an obstacle map generation means that generates a map of obstacles, an abnormal shape extraction means that extracts abnormal shapes, an intruder position detection means that detects position data of an intruder, an actual shape extraction means that corrects and calculates the abnormal shape and extracts actual shape data, an intruder body judgment unit that judges an intruder, an intruder judgment unit, etc.

[0007] Patent Document 6 discloses a liquid leakage detection device that has a liquid leakage detection unit that includes an introduction section for introducing leaked liquid, light projection means for irradiating light toward the introduction section, light reception means for receiving light from the light projection means via the introduction section, and detection means for detecting the liquid leakage based on the amount of light received by the light reception means, and that outputs a detection signal to the outside based on the liquid leakage detection result of the liquid leakage detection unit. This liquid leakage detection device is installed with a gap provided from a floor surface, and the liquid leakage detection unit has a first liquid leakage detection unit and a second liquid leakage detection unit provided at different positions on the underside of the liquid leakage detection device, and the first liquid leakage detection unit is located closer to the periphery and the second liquid leakage detection unit is located closer to the center on a line connecting the center and periphery of the underside of the liquid leakage detection device. The second liquid leakage detection unit is provided in a recess recessed at a position away from the peripheral edge of the underside of the liquid leakage detection device, and an output control means outputs the detection signal when the second liquid leakage detection unit detects the liquid leakage, and outputs the detection signal after a predetermined time when the first liquid leakage detection unit detects the liquid leakage.

[0008] Patent Document 7 discloses an autonomous mobile robot cleaner including an obstacle detection means for detecting obstacles around the device body, a traveling means for causing the device body to travel and turn, a cleaning means for cleaning the area where the device body travels, and a cleaning operation control means for controlling the traveling means and the cleaning means based on the output of the obstacle detection means to cause the device body to travel and clean the area where the device body travels. This device includes a dirt level determination means for determining the dirt level of the area where the device body travels, and the cleaning operation control means performs a basic cleaning operation for causing the device body to travel according to a predetermined traveling rule, and if an area where the dirt level exceeds a reference value is found during the basic cleaning operation based on the output obtained from the dirt level determination means, the device travels through the area where the dirt level exceeds the reference value by the basic cleaning operation, and then performs a local cleaning operation for further traveling locally through the area where the dirt level exceeds the reference value.

[0009] Patent Document 8 discloses an interior foreign object detection means capable of reliably detecting objects ("foreign objects") other than people, animals, and vehicles inside the passenger compartment. This device includes an entire interior sensor capable of detecting the entire interior area of ​​the passenger compartment in real time without any blind spots, and a data processing device that detects foreign objects from the detection data of the entire interior sensor. The entire interior sensor is one or more 3D laser radars. The 3D laser radars are installed so that foreign objects cannot enter between the interior wall and floor of the passenger compartment 2, and are located across the designated parking position for vehicles. The data processing device stores unattended data, which is detection data when no foreign objects are present inside the passenger compartment, and removes the unattended data or the unattended data and vehicle detection data from the detection data of the entire interior sensor, and detects the removed object as a foreign object. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Application Publication No. 7-92054 [Patent Document 2] Japanese Patent Application Laid-Open No. 2015-152313 [Patent Document 3] Japanese Patent Application Publication No. 56-142658 [Patent Document 4] Japanese Utility Model Application Publication No. 4-25248 [Patent Document 5] Japanese Patent Application Laid-Open No. 2001-148077 [Patent Document 6] Japanese Patent Application Laid-Open No. 2011-95249 [Patent Document 7] Japanese Patent Application Laid-Open No. 2005-211365 [Patent Document 8] Japanese Patent Publication No. 2022-40148 Summary of the Invention [Problem to be solved by the invention]

[0011] Conventional foreign object detection devices or water leak detection devices have limitations in that they can only detect objects of a certain quantity and size, and when detecting the type of foreign object, they have the problem of being unable to detect the type of foreign object if there are multiple foreign objects present.

[0012] An embodiment of the present invention aims to provide a foreign object appearance monitoring processing device and a foreign object appearance monitoring processing method that can appropriately detect foreign objects regardless of the size and quantity of the foreign object that has appeared, or even when multiple types of foreign objects are present. [Means for solving the problem]

[0013] The foreign object appearance monitoring processing device according to this embodiment is characterized by comprising: a camera having a light irradiating unit that irradiates light; and a camera main body having a plurality of pixels that receive the light reflected from the irradiated light at a light receiving unit and convert it into multi-level pixel data in accordance with the intensity of the received light, and that obtains one frame image in a matrix form using these plurality of pixels; an image storage unit for storing the image data; image storage control means for obtaining image data in chronological order using the camera and storing it in the image storage unit; foreign object appearance / range detection means for detecting the appearance and range of a foreign object based on a comparison of image data at a predetermined time point stored in the image storage unit with earlier image data or image data previously obtained when no foreign object was present; and foreign object type detection means for detecting the type of foreign object present within the range of the foreign object detected by the foreign object appearance / range detection means based on the image data stored in the image storage unit.

[0014] The foreign object appearance monitoring processing method according to this embodiment is characterized in that, when performing foreign object appearance monitoring processing in a foreign object appearance monitoring processing device that includes a camera having a light irradiation unit that irradiates light, a camera main body unit that has a plurality of pixels that receive light reflected from the irradiated light at a light receiving unit and convert the light reflected by the irradiated light into multi-value pixel data in accordance with the intensity of the received light, and that obtains one frame image in a matrix form using this plurality of pixels, and an image storage unit for storing the image data, the method executes the following steps: an image storage step that obtains image data in chronological order using the camera and stores it in the image storage unit; a foreign object appearance / range detection step that detects the appearance of a foreign object and the range of the foreign object based on a comparison of image data at a predetermined time point stored in the image storage unit with earlier image data or image data previously obtained when no foreign object has appeared; and a foreign object type detection step that detects the type of foreign object present within the range of the foreign object detected in the foreign object appearance / range detection step based on the image data stored in the image storage unit. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 10 is an explanatory diagram of conventional leak detection using a leak detection tape. [Figure 2] 1 is a block diagram of a foreign object appearance monitoring processing device according to a first embodiment of the present invention; [Figure 2A] 3A and 3B are diagrams showing illumination and reflection by a camera main body used in a processing device for monitoring the appearance of a foreign object according to an embodiment of the present invention; [Figure 2B] 3A and 3B are diagrams showing infrared light irradiation and reflection in a camera body used in a processing device for monitoring the appearance of a foreign object according to an embodiment of the present invention; [Figure 2C] 3A and 3B are diagrams showing infrared light irradiation and reflection in a camera body used in a processing device for monitoring the appearance of a foreign object according to an embodiment of the present invention; [Figure 3] 1 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a first embodiment of the present invention; [Figure 4] 1A and 1B are diagrams showing foreign object detection by a handheld foreign object appearance monitoring processing device according to an embodiment of the present invention; [Figure 5] 1 is a perspective view of a mobile robot-type foreign object appearance monitoring processing device according to an embodiment of the present invention; [Figure 6] 4 is a flowchart showing the operation of the processing device for monitoring the appearance of a foreign object according to the first embodiment of the present invention. [Figure 7] 3A and 3B are diagrams showing image data obtained by the operation of the processing device for monitoring the appearance of a foreign substance according to the first embodiment of the present invention, and image data when no foreign substance is present. [Figure 8] FIG. 10 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a second embodiment of the present invention. [Figure 9] 10 is an explanatory diagram of image data in a specific direction relative to the imaging line of sight of a camera in a processing device for monitoring the appearance of a foreign object according to a second embodiment of the present invention. FIG. [Figure 10] 10 is a flowchart showing the operation of a processing device for monitoring the appearance of a foreign object according to a second embodiment of the present invention. [Figure 11] FIG. 10 is an explanatory diagram of a process performed by a processing device 1A for monitoring the appearance of a foreign object according to a second embodiment of the present invention. [Figure 12] FIG. 10 is an explanatory diagram of processing by a foreign matter appearance monitoring processing device 1A according to a modified example of the second embodiment of the present invention. [Figure 13] FIG. 10 is a block diagram of a foreign object appearance monitoring processing device according to a third embodiment of the present invention. [Figure 14]10 is a flowchart showing the operation of a processing device for monitoring the appearance of a foreign object according to a third embodiment of the present invention. [Figure 15] FIG. 10 is an explanatory diagram of a process performed by a processing device 1A for monitoring the appearance of a foreign object according to a third embodiment of the present invention. [Figure 16] 10A and 10B are diagrams showing image data obtained by the operation of a processing device for monitoring the appearance of a foreign object according to a third embodiment of the present invention. [Figure 17] 10A and 10B are diagrams showing the operation of detecting a type of foreign matter by a processing device for monitoring the appearance of a foreign matter according to a third embodiment of the present invention. [Figure 18] FIG. 10 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a fourth embodiment of the present invention. [Figure 19] 10A and 10B are diagrams showing the operation of detecting a type of foreign matter by a processing device for monitoring the appearance of a foreign matter according to a fourth embodiment of the present invention. [Figure 20] FIG. 10 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a fifth embodiment of the present invention. [Figure 21] FIG. 10 is a block diagram of a processing device for monitoring the appearance of a foreign object according to a sixth embodiment of the present invention. [Figure 22] 13A and 13B are diagrams showing the operation of detecting a type of foreign matter by a processing device for monitoring the appearance of a foreign matter according to a sixth embodiment of the present invention. [Figure 23] FIG. 13 is a diagram showing an example of image data and color information stored in an image storage unit of a foreign substance appearance monitoring processing device according to a sixth embodiment of the present invention. [Figure 24] FIG. 2 is a block diagram of an alarm generating unit in the processing device for monitoring the appearance of a foreign object according to each embodiment of the present invention. [Figure 24A] FIG. 2 is a functional block diagram of an embodiment of a foreign object appearance monitoring processing device having an alarm generating unit. [Figure 25] FIG. 2 is a block diagram of an alarm generating unit that generates an alarm through communication in the foreign object appearance monitoring processing device according to each embodiment of the present invention. [Figure 26] 10A and 10B are explanatory diagrams illustrating an alarm generating operation in which an alarm is generated in the processing device for monitoring the appearance of a foreign object according to each embodiment of the present invention. [Figure 27] FIG. 10 is a second explanatory diagram of the alarm generating operation in the processing device for monitoring the appearance of a foreign object according to each embodiment of the present invention. [Figure 27A]FIG. 10 is a third explanatory diagram of the alarm generating operation in the processing device for monitoring the appearance of a foreign object according to each embodiment of the present invention. [Figure 27B] FIG. 4 is a fourth explanatory diagram of the alarm generating operation for generating an alarm in the processing device for monitoring the appearance of a foreign object according to each embodiment of the present invention. [Figure 27C] FIG. 5 is a fifth explanatory diagram of the alarm generating operation for generating an alarm in the processing device for monitoring the appearance of a foreign object according to each embodiment of the present invention. [Figure 27D] FIG. 6 is a sixth explanatory diagram of the alarm generating operation for generating an alarm in the processing device for monitoring the appearance of a foreign object according to each embodiment of the present invention. [Figure 27E] FIG. 7 is a seventh explanatory diagram of the alarm generating operation for generating an alarm in the processing device for monitoring the appearance of a foreign object according to each embodiment of the present invention. [Figure 27F] FIG. 8 is an eighth explanatory diagram of the alarm generating operation for generating an alarm in the processing device for monitoring the appearance of a foreign object according to each embodiment of the present invention. [Figure 27G] FIG. 9 is a ninth explanatory diagram of the alarm generating operation for generating an alarm in the processing device for monitoring the appearance of a foreign object according to each embodiment of the present invention. [Figure 27H] FIG. 10 is a tenth explanatory diagram of the alarm generating operation in the foreign object appearance monitoring processing device according to each embodiment of the present invention. [Figure 27I] FIG. 11 is an eleventh explanatory diagram of the alarm generating operation in the processing device for monitoring the appearance of a foreign object according to each embodiment of the present invention. [Figure 28] 13 is a flowchart showing the operation of a processing device for monitoring the appearance of a foreign object according to the seventh embodiment of the present invention. [Figure 29] 13A and 13B are diagrams showing the operation of detecting a type of foreign matter by a processing device for monitoring the appearance of a foreign matter according to a seventh embodiment of the present invention. [Figure 30] FIG. 13 is a diagram showing a foreign matter type detection operation in another example of the processing device for monitoring the appearance of a foreign matter according to the seventh embodiment of the present invention. [Figure 31] 13A and 13B are diagrams showing the operation of detecting a type of foreign object by capturing an image from a height direction using a handheld camera that monitors the appearance of a foreign object in a processing device for monitoring the appearance of a foreign object according to a seventh embodiment of the present invention. [Figure 32]13A and 13B are diagrams showing the operation of detecting a type of foreign object by capturing an image from a horizontal direction using a handheld camera that monitors the appearance of a foreign object in a processing device for monitoring the appearance of a foreign object according to a seventh embodiment of the present invention. [Figure 33] FIG. 13 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to an eighth embodiment of the present invention. [Figure 34A] FIG. 13 is an explanatory diagram of a case where the intensity of the irradiated light of the foreign matter appearance monitoring processing device according to the eighth embodiment of the present invention is weak. [Figure 34B] FIG. 13 is an explanatory diagram of a case where the intensity of the irradiated light of the foreign matter appearance monitoring processing device according to the eighth embodiment of the present invention is increased. [Figure 35] FIG. 13 is a block diagram of a foreign object appearance monitoring processing device according to a ninth embodiment of the present invention. [Figure 36] FIG. 13 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a ninth embodiment of the present invention. [Figure 37] FIG. 13 is a diagram showing a foreign matter type detection operation by a processing device for monitoring the appearance of a foreign matter according to a ninth embodiment of the present invention. [Figure 38] FIG. 23 is a diagram showing the operation of detecting a type of foreign matter by the processing device for monitoring the appearance of a foreign matter according to the tenth embodiment of the present invention. [Figure 39] FIG. 20 is a block diagram of a foreign object appearance monitoring processing device according to a tenth embodiment of the present invention. [Figure 40] FIG. 22 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a tenth embodiment of the present invention. [Figure 41] FIG. 20 is a block diagram of a foreign object appearance monitoring processing device according to a tenth embodiment of the present invention. [Figure 42] FIG. 22 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to an eleventh embodiment of the present invention. [Figure 43] FIG. 23 is a diagram showing the operation of detecting a type of foreign matter by the processing device for monitoring the appearance of a foreign matter according to the eleventh embodiment of the present invention. [Figure 44] FIG. 23 is a block diagram of a foreign object appearance monitoring processing device according to a twelfth embodiment of the present invention. [Figure 45] FIG. 22 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a twelfth embodiment of the present invention. [Figure 46]FIG. 23 is a diagram showing a first foreign matter type detection operation by a foreign matter appearance monitoring processing device according to a twelfth embodiment of the present invention. [Figure 47] FIG. 23 is a diagram showing a second foreign matter type detection operation by the foreign matter appearance monitoring processing device according to the twelfth embodiment of the present invention. [Figure 48] FIG. 23 is a diagram showing a third foreign matter type detection operation by the foreign matter appearance monitoring processing device according to the twelfth embodiment of the present invention. [Figure 49] FIG. 23 is a diagram showing a fifth foreign matter type detection operation by the foreign matter appearance monitoring processing device according to the twelfth embodiment of the present invention. [Figure 50] FIG. 23 is a diagram showing a sixth foreign matter type detection operation by the foreign matter appearance monitoring processing device according to the twelfth embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] The following describes a foreign object appearance monitoring device according to an embodiment of the present invention with reference to the accompanying drawings. In each figure, identical components are designated by the same reference numerals, and redundant description will be omitted. FIG. 2 shows a block diagram of a first embodiment of a foreign object appearance monitoring device 1 according to the present embodiment. The foreign object appearance monitoring device 1 can be configured using a cloud computer, a server computer, a personal computer, or other computers. The foreign object appearance monitoring device 1 according to this embodiment considers foreign objects to include not only water and oil but also solids. In other words, a foreign object refers to an object that should not be present in a certain location or that would not normally emerge from a device, and the type of foreign object being detected among those objects is referred to as a desired foreign object. For example, in the case of a semiconductor production device, various liquids that may leak from the device are considered desired foreign objects, and any other parts or components that fall and appear are referred to as objects other than the desired foreign object.

[0017] The foreign object appearance monitoring processing device 1 has a CPU 10 that performs processing based on programs and data in a main memory 11. An external storage device 23 is connected to the CPU 10 via a bus 12 and an external storage interface 13, and a program for foreign object appearance monitoring processing is stored in the external storage device 23. The CPU 10 reads the program for foreign object appearance monitoring processing from the external storage device 23 into the main memory 11 and executes this program, thereby functioning as the foreign object appearance monitoring processing device 1, and during this operation a foreign object appearance monitoring processing method is executed.

[0018] In addition to the external storage interface 13, an input interface 14, a display interface 15, and an IO interface 16 are connected to the bus 12. An input device 24 such as a keyboard or a touch panel and a pointing device 22 such as a mouse are connected to the input interface 14. A display device 25 for displaying data and images is connected to the display interface 15. The display device 25 may be equipped with a device that outputs alarms by voice or by turning on a light in addition to displaying information.

[0019] A camera 26 is connected to the IO interface 16. The camera 26 captures an image of a foreign object appearance area where a foreign object is predicted to appear, and obtains image data. As shown in FIG. 2A , the camera 26 includes a camera main body 26-0 having a main function of imaging, etc., which includes a light emitting unit 26-1 that emits light and a light receiving unit 26-2 that receives light reflected from a foreign object appearance area 27, such as a floor. The camera main body 26-0 photoelectrically converts the light obtained by the light receiving unit 26-2 to obtain planar image data using a matrix of cells. The image data may have RGB color levels or black and white levels, and may hereinafter be simply referred to as levels. 2B and 2C, when light emitting unit 26-1 is an infrared emitting unit, if there is a leak 130 as the desired foreign matter, the infrared light is absorbed by leak 130, resulting in weak or no reflection, and light receiving unit 26-2 receives reflected light in which the light from the part of leak 130 is weakened or absent. Therefore, it is possible to properly detect leak 130 as the desired foreign matter based on the difference in intensity of reflected light 135.

[0020] As shown in FIG. 3, the foreign object appearance monitoring processing device 1 of this embodiment includes an image storage unit 31, an image storage control means 32, a foreign object appearance / range detection means 33, and a foreign object type detection means 34 in the main memory 11. The image storage unit 31 stores pixel data of light received by the light receiving unit 26-2 and converted into multi-value levels according to the received light intensity. The image storage unit 31 may be provided in the external storage device 23. The image storage control means 32 acquires image data in chronological order from the camera 26 and stores the data in the image storage unit 31. The foreign object appearance / range detection means 33 detects the appearance and range of a foreign object based on a comparison of image data at a predetermined time point stored in the image storage unit 31 with previous image data or previously acquired image data when no foreign object is present. The foreign object type detection means 34 detects the type of foreign object present within the range of the foreign object detected by the foreign object appearance / range detection means 33 based on the image data stored in the image storage unit 31.

[0021] The foreign object appearance monitoring processing device 1 of this embodiment can be a flashlight-type device that is held in the hand of a person H and used to project light in a desired direction, as shown in Figure 4, or a mobile robot-type device as shown in Figure 5.

[0022] The foreign substance appearance monitoring processing device 1 of this embodiment operates according to the flowchart shown in Fig. 6. That is, image data is obtained in chronological order by the camera 26 and stored in the image storage unit 31 (S11). At this time, multiple frames are stored in the image storage unit 31 by storing them for a predetermined period of time. Next, the appearance of a foreign substance and the extent of the foreign substance are detected based on a comparison of the stored image data at a predetermined time point with previous image data or previously obtained image data without the presence of a foreign substance (S12).

[0023] For example, when there is an apparatus (e.g., semiconductor manufacturing apparatus) 100 as shown in Fig. 4 and the imaging direction of camera 26 is set to a predetermined direction, if leaking liquid 130 and fallen object 140 are captured, image data 110 as shown in Fig. 7(A) is obtained and stored in image storage unit 31. Although Fig. 7(A) shows one frame, multiple frames are stored in image storage unit 31.

[0024] 7(B) shows image data (one frame) in which no foreign matter is present, which is stored in advance in the image storage unit 31. For example, the presence of a foreign matter and the extent of the foreign matter are detected based on a comparison of the image data (one frame) shown in FIG. 7(B) with the image data shown in FIG. 7(A). The presence of a foreign matter is the detection of the presence or absence of a foreign matter, and the detection of the extent of the foreign matter is performed by defining one frame using xy coordinates, surrounding a detected foreign matter with, for example, a rectangular frame, and determining the coordinates of the range of this rectangular frame as the extent of the foreign matter.

[0025] Once the presence and extent of a foreign object are detected as described above, the foreign object appearance monitoring processing device 1 of this embodiment proceeds to step S13 of the flowchart shown in Figure 6, where the type of foreign object present in the range of the foreign object detected by the foreign object appearance / range detection means 33 is detected based on the stored image data (S13). To detect the type of foreign object, brightness data (average value, variance, etc.) of the pixels in the foreign object range, area data based on the coordinates of the rectangular frame range, or data on the change in size of the area data over time, etc., are collected in advance for each type of foreign object and stored in the external storage device 23. This data is compared with brightness data (average value, etc.) of the pixels in the foreign object range, area data based on the coordinates of the rectangular frame range, or data on the change in size of the area data over time, etc., obtained from the currently obtained image data, and any data that matches or falls within a predetermined range can be determined to be the type of foreign object. Regarding the leakage liquid 130 of the specified foreign object, it is determined whether it is water, machine oil, or a specific solution. In addition, if the fallen object 140 is a specific foreign object, it can be a piece of paper, a wallet, a mobile phone, a writing implement, etc. In this way, the type of foreign object can be identified and displayed on the display device 25.

[0026] Next, a description will be given of a foreign object appearance monitoring processing device 1 according to a second embodiment. The foreign object appearance monitoring processing device according to the second embodiment also has the configuration shown in the block diagram of Fig. 2, and can be a flashlight-type device that is held by a person H in his / her hand and used to project light in a desired direction, as shown in Fig. 4, or a mobile robot-type device as shown in Fig. 5.

[0027] 8, the foreign object appearance monitoring processing device 1 of this embodiment includes, in main memory 11, an image storage unit 31, image storage control means 32, foreign object appearance / range detection means 33, foreign object type detection means 34, specific direction image acquisition means 35, predetermined image detection means 36, and operation control means 37. The image storage unit 31, image storage control means 32, foreign object appearance / range detection means 33, and foreign object type detection means 34 are the same as those in the foreign object appearance monitoring processing device 1 of the first embodiment.

[0028] The specific direction image acquisition means 35 acquires image data in a specific direction relative to the imaging line of sight of the camera 26. Here, the specific direction relative to the imaging line of sight refers to the direction of a point within the range of the screen S captured by the camera 26, as shown in FIG. 9, and can be the direction of an area or location where foreign object appearance monitoring is to be performed. The predetermined image detection means 36 detects whether the image data acquired by the specific direction image acquisition means 35 includes image data of a predetermined image. The predetermined image can be a pattern image P as shown in FIG. 9. The predetermined image is not limited to a pattern, but can also be symbols such as "o" or "x," letters, or character strings. The predetermined image can also be an image in which a pattern or the like appears when illuminated with a predetermined light. When the predetermined image detection means 36 detects a predetermined image, the operation control means 37 controls operations such as capturing an image by the camera 26, storing the image data in the image storage unit 31 by the image storage control means 32, and performing detection by the foreign object appearance / range detection means 33 and the foreign object type detection means 34.

[0029] The foreign object appearance monitoring processing device 1 of this embodiment operates according to the flowchart shown in Fig. 10. That is, an image is taken by the camera 26 to obtain image data in a specific direction relative to the imaging line of sight of the camera 26 (S21). Next, it is determined whether or not the obtained image data includes a predetermined image (S22). If the result in step S22 is NO, the process returns to step S21 to continue processing. On the other hand, if the result in step S22 is YES, the process proceeds from step S11 to step S13 already described in Fig. 6, and "imaging by the camera 26, retention of the image data in the image processing unit 31 by the image retention control means 32, detection by the foreign object appearance / range detection means 33, and detection by the foreign object type detection means 34" become possible. That is, by capturing an image in the direction of device 100 using camera 26A shown in FIG. 11 and making it so that the predetermined image is included in the predetermined image P image data, camera 26A captures the image while facing the direction in which foreign object appearance monitoring should be performed, and therefore the image data captured by camera 26A is stored, foreign object appearance / range detection and foreign object type detection are performed, and foreign object appearance monitoring is performed by capturing an image in the direction of an appropriate area or location.

[0030] Conversely, if the process branches to NO in step S22, the process does not proceed from step S11 to step S13 already described in FIG. 6, and "imaging by the camera 26, retention of image data in the image processing unit 31 by the image retention control means 32, detection by the foreign object appearance / range detection means 33, and detection by the foreign object type detection means 34" are not performed. That is, when an image is captured in the direction of the device 100 by the camera 26B shown in FIG. 11 and the predetermined image P image data does not include a predetermined image, camera 26B is not facing the direction in which foreign object appearance monitoring should be performed, and therefore the image data captured by the camera 26B is not retained, and foreign object appearance / range detection and foreign object type detection are not performed. That is, foreign object appearance monitoring is not performed by capturing an image in the direction of an inappropriate area or location.

[0031] 12 is an explanatory diagram of processing by a foreign object appearance monitoring processing device 1A having a camera 26 according to a modification of the second embodiment. The foreign object appearance monitoring processing device 1A is provided with a wireless unit that, when a predetermined image is detected by the predetermined image detection means 36, receives information about the detection and transmits a predetermined image detection notification signal.

[0032] The door 28 is equipped with a locking / unlocking control mechanism 29, which has a wireless unit and controls unlocking when it receives the predetermined image detection notification signal, allowing the door 28 to be opened. When the door 28 is opened and then closed, the locking / unlocking control mechanism 29 locks the door. A predetermined image P is attached to the door 28, and the predetermined image P is captured by the camera 26 of the foreign object appearance monitoring processing device 1A to unlock the door, allowing the door 28 to be opened. This modification prevents unnecessary entry into the door 28, and allows only necessary persons to enter the door 28 and monitor for the appearance of a foreign object.

[0033] Next, a description will be given of a foreign object appearance monitoring processing device 1 according to a third embodiment. The foreign object appearance monitoring processing device 1 according to the third embodiment has the configuration shown in the block diagram of Fig. 13, and can be a flashlight-type device that is held by a person P in his / her hand to project light in a desired direction as shown in Fig. 4, or a mobile robot-type device as shown in Fig. 5.

[0034] As shown in Fig. 13, the foreign object appearance monitoring processing device 1 according to the third embodiment has almost the same configuration as that of the first embodiment shown in Fig. 2. That is, it has an IO interface 18 connected to the bus 12, and a distance sensor 19 serving as distance detection means connected to the IO interface 18. The distance sensor 19 serving as distance detection means detects the distance to an object to be imaged when an image is captured by the camera 26. The distance sensor 19 can be a type known as a TOF (Time of Flight) sensor, which measures distance based on the time it takes for a projected light pulse to be reflected and returned.

[0035] The functional block diagram of the foreign object appearance monitoring processing device 1 according to the third embodiment has the same configuration as that of the first embodiment shown in Fig. 3. The foreign object type detection means 34 in the third embodiment detects the type of foreign object based on the image data stored in the image storage unit 31 and the distance detected by the distance sensor 19, which is the distance detection means.

[0036] The foreign object appearance monitoring processing device 1 of this embodiment operates according to the flowchart shown in Fig. 14. That is, image data is obtained in time series by the camera 26, and distance data is obtained by the distance sensor 19, which is a distance detection means, and stored in the image storage unit 31 (S11A). At this time, by storing the image data for a predetermined time, multiple frames of data are stored in the image storage unit 31. Next, the appearance of a foreign object, its range, and its distance are detected based on a comparison of the stored image data at a predetermined time point with previous image data or previously obtained image data without the presence of a foreign object (S12A).

[0037] For example, suppose there is an apparatus (e.g., semiconductor manufacturing apparatus) 100 as shown in Fig. 15, and when an image is captured by camera 26, leaking liquid 130 and fallen object 140 are captured as shown in Fig. 16, data is detected in which the coordinates of star marks ST1, ST2, ST3, and ST4 are the center of the range of the foreign object. At this time, suppose that distance sensor 19, which is a distance detection means, measures distance data to the coordinates of star marks ST1, ST2, ST3, and ST4 as distance A, distance B, distance C, and distance D as shown in Fig. 17(a), and measures (reflected) intensity data at the coordinates of star marks ST1, ST2, ST3, and ST4 as intensities EA, EB, EC, and ED as shown in Fig. 17(b).

[0038] In this embodiment, the external storage device 23 is provided with a memory table that stores data on the type of foreign object corresponding to the distance to the image capture target and the intensity. Therefore, the foreign object appearance monitoring processing device 1 of this embodiment detects the type of foreign object present within the range of foreign objects detected by the foreign object appearance / range detection means 33 based on the stored image data, taking into account the distance and reflection intensity, and outputs any necessary alarms (S13A).

[0039] The foreign object appearance monitoring device 1 according to the fourth embodiment includes the same configuration as the foreign object appearance monitoring device according to the third embodiment, but also includes an image storage unit 31, an image storage control unit 32, a foreign object appearance / range detection unit 33, and a foreign object type detection unit 34 in the main memory 11, as shown in FIG. 18. The image storage control unit 38 changes the image capture direction of the camera 26C. In addition to the configuration of the camera 26, the camera 26C includes a swivel mechanism ND that changes the orientation (angle) of the head equipped with the image sensor in two dimensions or three dimensions, as shown in FIG. 19. The image capture direction can be changed by the control of the image capture direction change unit 38. This embodiment also includes a case where the image capture direction of the camera 26C is changed and fixed to a direction directly downward in the direction of gravity from the original position.

[0040] In the processing device 1 for monitoring the presence of a foreign object according to the fourth embodiment, the distance and angle can be automatically changed by turning on the control of the imaging direction changing means 38, and various data can be obtained to detect the type of foreign object. The processing device 1 for monitoring the presence of a foreign object according to the fourth embodiment operates according to the flowchart shown in FIG.

[0041] The foreign substance appearance monitoring processing device 1 according to the fifth embodiment has the same configuration as the foreign substance appearance monitoring processing device according to the third embodiment, but in addition to the image storage unit 31, image storage control means 32, foreign substance appearance / range detection means 33, and foreign substance type detection means 34 in the main memory 11, it also includes light intensity changing means 39 as shown in Fig. 20. The light intensity changing means 39 changes the intensity of light emitted by the light emitting unit 26-1.

[0042] In the foreign matter appearance monitoring processing device 1 according to the fifth embodiment, the light intensity can be automatically changed by turning on the control by the light intensity changing means 39, and data of various intensities can be obtained to detect the type of foreign matter. The foreign matter appearance monitoring processing device 1 according to the fifth embodiment operates according to the flowchart shown in FIG.

[0043] The foreign object appearance monitoring processing device 1 according to the sixth embodiment is equipped with color detection means for detecting color information of the imaged object when the image is captured by the camera 26. The foreign object appearance monitoring processing device 1 according to the sixth embodiment has a configuration shown in the block diagram of FIG. 21, which is substantially the same configuration as that of the first embodiment shown in FIG. 2. That is, it differs in that it is equipped with an IO interface 8 connected to a bus 12, and also equipped with a color sensor 9 serving as color detection means connected to the IO interface 8. The color sensor 9 is capable of detecting RGB data in multiple tones as color information, and may be an RGB camera.

[0044] The various means provided in the foreign object appearance monitoring processing device 1 according to the sixth embodiment are the same as those in the foreign object appearance monitoring processing device according to the first embodiment, as shown in Fig. 3. In the foreign object appearance monitoring processing device 1 according to this embodiment, the foreign object type detection means 34 is configured to detect the type of foreign object based on the image data stored in the image storage unit 31 and the color information detected by the color sensor 9, which is the color detection means. The color sensor 9 detects the color information of an object in the same direction as the camera 26.

[0045] As shown in Fig. 22, when there is an apparatus (for example, semiconductor manufacturing apparatus) 100 and the imaging direction of camera 26 is set to the direction of apparatus 100, if leak 130 and pattern line 150 on the floor are captured, image data 160 and color information as shown in Fig. 23(A) are obtained and stored in image storage unit 31. Although Fig. 23(A) shows one frame, multiple frames are stored in image storage unit 31.

[0046] As in the first embodiment, the appearance of a foreign object and the extent of the foreign object are detected. It is assumed that the light intensities obtained in the foreign object area related to the liquid leakage 130 and the foreign object area related to the floor pattern line 150 are "light intensity XX" and "light intensity XX," respectively, as shown in FIG. 23(A). It is assumed that the color information obtained in the foreign object area related to the liquid leakage 130 and the foreign object area related to the floor pattern line 150 are "color information aaa" and "color information ccc," respectively, as shown in FIG. 23(B).

[0047] In this embodiment, the external storage device 23 is provided with a memory table that stores data on foreign object types corresponding to the light intensity and color information of the imaged object. Therefore, the foreign object appearance monitoring processing device 1 of this embodiment detects the type of foreign object present in the range of foreign objects detected by the foreign object appearance / range detection means 33 based on the stored image data, taking into account the light intensity and color information, and outputs any necessary alarms.

[0048] In the foreign object appearance monitoring processing device 1 according to each embodiment described above, when an alarm is generated, it is basically possible to notify the user that a foreign object has been detected by displaying a message on the display device 25. However, when an alarm is generated, the person holding the foreign object appearance monitoring processing device 1 is not necessarily in a state where they can see the display.

[0049] The foreign object appearance monitoring processing device 1 according to each embodiment is equipped with alarm generating means for generating an alarm regarding the appearance of a foreign object. Fig. 24 shows an alarm generating device 7, which is alarm generating means, provided on the bus 12 of the foreign object appearance monitoring processing device 1 via the output interface 6. The alarm generating device 7 can generate an alarm by sound, smell, light, vibration, or a combination of these.

[0050] Fig. 24A shows a functional block diagram of this embodiment. In addition to an image storage unit 31, image storage control means 32, foreign object appearance / range detection means 33, and foreign object type detection means 34, main memory 11 is also provided with alarm generation control means 7A, as shown in Fig. 24A. Alarm generation control means 7A controls the generation of an alarm by alarm generation device 7, which is the alarm generation means, in accordance with the manner in which a foreign object appears detected by foreign object appearance / range detection means 33. For example, when a single frame of image as shown in Fig. 7(A) appears in multiple consecutive frames, this is a "normal manner" in which a foreign object appears in multiple frames, and therefore an alarm is generated by alarm generation device 7, which is the alarm generation means. The alarm generation basically continues unless a stop operation is performed by person H.

[0051] 25 shows a configuration in which a communication interface 5A is provided on the bus 12 of the foreign object appearance monitoring processing device 1, while a communication interface 5B and an alarm generating device 7 connected to the communication interface 5B are provided on a person H or a robot. This configuration is convenient when, for example, it is necessary to release the foreign object appearance monitoring processing device 1 including the camera 26 from the hand of the person H and place it on the floor below the device 100 when checking for the occurrence of a liquid leak 130 on the floor below the device 100, as shown in FIG. 26. Alternatively, it is convenient when, for example, it is necessary to hold the foreign object appearance monitoring processing device 1 including the camera 26 in the hand of the person H using a rod-shaped picking tool or the like and move it into the narrow gap between the walls 151A and 151B when checking for the occurrence of a liquid leak 130 on the floor below the device 100, as shown in FIG. 27.

[0052] The alarm generation control means 7A also generates an alarm in cases other than the above-described "normal cases." For example, as shown in FIG. 27A, assume that a camera 26 is attached to the ceiling 102 and captures images vertically downward from the ceiling 102. The area M of the floor 103 from the camera 26 is the capture range for one image. If an image frame is obtained in which the leaked liquid 130 is captured in area M, the foreign object appearance / range detection means 33 detects "foreign object present" for this frame of image. However, if, in the next frame, as shown in FIG. 27B, the leaked liquid 130 escapes from the area M of the floor 103 from the camera 26 and an image frame is obtained in which the leaked liquid 130 is not captured, the foreign object appearance / range detection means 33 detects "no foreign object present" for this frame of image. For example, if the floor 103 has a steep slope, the detection status may change from "foreign object present" to "no foreign object present" within one frame. Even in such a case, the alarm generation control means 7A causes the alarm generating device 7, which is the alarm generating means, to generate an alarm. That is, the alarm can notify the user that leaking water or the like is flowing forcefully and that the damage is expanding due to the movement of the leaking water.

[0053] FIG. 27C shows a case where the leaked liquid 130 is a highly volatile substance such as alcohol. As shown in FIG. 27D, if the leaked liquid 130 disappears from area M of floor 103 captured by camera 26 within a few frames, if not just one, and a single frame of image is obtained without the leaked liquid 130 appearing, the foreign substance appearance / range detection means 33 detects "no foreign substance" for these few frames. This example shows a change from "foreign substance present" to "no foreign substance present" within a few frames. Even in this case, the alarm generation control means 7A causes the alarm generation device 7, which is the alarm generation means, to generate an alarm. Although alcohol disappears, in the case of a leak caused by mist, it is possible to output a mist generation alarm by detecting "foreign substance present" and "no foreign substance present" every few frames, for example. It is also possible to narrow down or identify the type of foreign substance based on the frequency with which the "foreign substance present" and "no foreign substance present" states are repeated.

[0054] FIG. 27E shows a scene in which liquid is leaking from the ceiling 102, causing droplets 130S to fall vertically downward, capturing an image of the space between the ceiling 102 and the floor 103. This is a single frame of image capturing the droplets 130S. FIG. 27F shows a scene in which the fallen droplets 130S gather, forming a puddle 180 in the corner of the floor 103, and the droplets 130S are not captured in the space between the ceiling 102 and the floor 103. This is a single frame of image capturing no droplets 130S. In this manner, "foreign object presence" and "no foreign object presence" are repeatedly detected at intervals of multiple frames. Even in this case, the alarm generation control means 7A causes the alarm generation device 7, which is the alarm generation means, to generate an alarm. In this example, an alarm is appropriately generated even at the site of a foreign object presence monitoring process where "no foreign object presence" occurs for a certain period of time (over multiple frames) depending on the imaging direction of the camera 26.

[0055] FIG. 27G shows a case where water is being supplied to a water tank / reservoir 195 from a water pipe / water supply pipe 190 whose outlet is located higher than the water tank / reservoir 195. The supply water 130W between the outlet of the water pipe / water supply pipe 190 and the water tank / reservoir 195 is a "foreign object," and since it is captured by the camera 26, "foreign object has appeared." FIG. 27H shows a case where the supply water 130W has stopped, and "no foreign object has appeared." In this example, the water supply has stopped, and the state has changed from "foreign object has appeared" to "no foreign object has appeared." Even in such a case, the alarm generation control means 7A causes the alarm generation device 7, which is the alarm generation means, to generate an alarm. This allows the user to appropriately notify the user of an abnormal situation in which the water supply has stopped by using an alarm.

[0056] FIG. 27I shows a case where a moving object 130U, such as an insect or small animal, is detected as a foreign object. Multiple frames of images are obtained of the moving object randomly entering and exiting the imaging area of ​​camera 26. That is, as moving object 130U moves along movement line 196, multiple frames of images showing "foreign object present" and multiple frames of images showing "no foreign object present" are obtained alternately. Even in such a case, alarm generation control means 7A causes alarm generation device 7, the alarm generation means, to generate an alarm. This allows an alarm to be used to appropriately notify of an abnormal situation such as the intrusion of an insect or small animal.

[0057] Next, a foreign object appearance monitoring processing device 1 according to a seventh embodiment will be described. The foreign object appearance monitoring processing device 1 according to the seventh embodiment includes a distance sensor 19, which is a distance detection means similar to that of the foreign object appearance monitoring processing device 1 according to the third embodiment, and its functional block diagram has the same configuration as that of the first embodiment shown in Fig. 3. In this embodiment, image storage control means 32 causes the image storage unit 31 to store image data when the distance detected by the distance sensor 19, which is the distance detection means, is a predetermined distance.

[0058] The foreign object appearance monitoring processing device 1 of this embodiment operates according to the flowchart shown in Figure 28. Specifically, only image data obtained by the camera 26 at a predetermined distance is stored in the image storage unit 31 in chronological order (S11B). By storing the image data for a predetermined period of time, multiple frames are stored in the image storage unit 31. Next, the presence of a foreign object and the extent of the foreign object are detected based on a comparison of the stored image data at the predetermined time with previous image data or previously obtained image data without the presence of a foreign object (S12). Furthermore, the type of foreign object present within the range of the detected foreign object is detected by the foreign object appearance / extent detection means 33 (S13).

[0059] 29, assume that a camera 26 is attached to the ceiling 102 and takes images vertically downward from the ceiling 102, and that the distance from the camera 26 to the floor 103 is L1. An apparatus 104 is installed on the floor 103, and that the distance from the camera 26 to the top surface of the apparatus 104 is L2. In this embodiment, if the image storage unit 31 is set to store image data only when the distance detected by the distance sensor 19 is L2, image data related to a liquid leak 130 occurring on the top surface of the apparatus 104 is stored, but image data related to a liquid leak 130B occurring on the floor 103 is not stored. Therefore, it is possible to appropriately detect only foreign objects occurring on the top surface of the apparatus 104.

[0060] In FIG. 30, cameras 26 and 26K are attached to the ceiling 102 and are connected to a single foreign object appearance monitoring processing device 1 for unified control. Of the image data obtained by camera 26K, only image data at a predetermined distance is stored. In this way, using multiple cameras allows for a wide monitoring range and enables monitoring of foreign object appearances at positions at different distances from the cameras. In the case of FIG. 30, image data can be obtained and stored sequentially in a time-division manner, with one camera capturing an image followed by the next camera capturing an image. While this embodiment stores image data for a predetermined distance in the image storage unit 31, image data for all distances may be stored in the image storage unit 31, and the image data for a predetermined distance may be processed by the foreign object appearance / range detection means 33 and the foreign object type detection means 34.

[0061] 31 shows a configuration in which image capturing is performed from a height direction by a camera 26 of the type held by a person H for monitoring the appearance of a foreign object, and image data for a predetermined distance is stored in the image storage unit 31 and processed by the foreign object appearance / extent detection means 33 and the foreign object type detection means 34. Assume that the distance from the camera 26 to the leaking liquid 130B is L12. Assume that equipment 104 is installed on the floor 103, and the distance from the camera 26 to the top surface of the equipment 104 is L21. In this case, image data relating to the leaking liquid 130 occurring on the top surface of the equipment 104 is stored, and this image data is processed by the foreign object appearance / extent detection means 33 and the foreign object type detection means 34.

[0062] FIG. 32 shows a configuration in which a camera 26 held by a person H for monitoring the appearance of a foreign object is used to capture images from a horizontal direction. Image data for a predetermined distance is stored in the image storage unit 31 and processed by the foreign object appearance / range detection means 33 and the foreign object type detection means 34. Assume that the distance from the camera 26 to the leaked liquid 130 flowing downward in front of the device 104A is L1. The device 104B is installed on the floor 103, and the distance from the camera 26 to the top surface of the device 104B is L2. In this case, image data relating to the leaked liquid 130B occurring on the top surface of the device 104B is stored, and this image data is processed by the foreign object appearance / range detection means 33 and the foreign object type detection means 34. In this example, image data for a predetermined distance is stored in the image storage unit 31. However, image data for all distances may be stored in the image storage unit 31, and the image data for a predetermined distance may be processed by the foreign object appearance / range detection means 33 and the foreign object type detection means 34.

[0063] Next, a foreign object appearance monitoring processing device 1 according to an eighth embodiment will be described. As shown in FIG. 33, the functional block diagram of the foreign object appearance monitoring processing device 1 according to the eighth embodiment is substantially identical to that of the first embodiment shown in FIG. 3, but differs in that it includes an irradiated light intensity changing means 41. The irradiated light intensity changing means 41 changes the intensity of light emitted from the light irradiating unit 26-1. When capturing an image using the camera 26, the CPU 10 controls the light irradiating unit 26-1, for example, in several stages, to change the light intensity. If the CPU 10 operates as the foreign object appearance / range detecting means 33 and detects the appearance and range of a foreign object but does not detect it, this is likely because the irradiated light intensity is weak and the light reach distance W is short, as shown in FIG. 34A, and the light does not reach the leaking liquid 130.

[0064] Therefore, in this embodiment, the CPU 10 can be configured as the foreign object appearance / extent detection means 33 to operate, for example, at an initial low light intensity value after power-on. If the foreign object appearance and extent are not detected, the CPU 10 gradually increases the light intensity to a maximum value, fixes the intensity at the value at which the foreign object appearance and extent are detected, and then irradiates light at that intensity. By performing this operation, as the light intensity is gradually increased to a maximum value, the intensity of the irradiated light becomes stronger than in FIG. 34A, as shown in FIG. 34B, and the light reach distance W becomes longer than before, reaching the leak 130. At this point, the light intensity is fixed, and subsequent foreign object appearance and extent detection can be properly performed.

[0065] Next, a foreign object appearance monitoring processing device 1 according to a ninth embodiment will be described. As shown in the functional block diagram of FIG. 36, the foreign object appearance monitoring processing device 1 according to the ninth embodiment is provided with an imaging position / angle control means 44 in main memory 11 that stores a plurality of position information items at which the angle when imaging an object by a camera is a predetermined angle and past images when the angle at that position was realized, and that realizes the angle by the camera based on the past images when any of the plurality of positions is reached. The camera of this embodiment can be camera 26C equipped with a swivel mechanism ND shown in FIG. 18.

[0066] The foreign object appearance monitoring processing device 1 according to the ninth embodiment has the configuration shown in the block diagram of Fig. 35. The device according to the ninth embodiment differs from the foreign object appearance monitoring processing device 1 according to the first embodiment in that, in addition to being provided with a camera 26C, a driving unit 43 connected to a bus 12 moves a swivel mechanism ND under the control of an imaging position / angle control means 44 to achieve a predetermined angle.

[0067] As shown in Fig. 37, a plurality of cameras 26C are provided in the foreign matter appearance monitoring processing device 1 according to the ninth embodiment (four cameras in total, at the four corners in this embodiment) to eliminate blind spots in order to perform processing for monitoring the appearance of a liquid leak 130 in the device 100. The imaging position / angle control means 44 shown in Fig. 36, which realizes angles based on past images, is provided with past images captured at angles for performing processing for monitoring the appearance of a foreign matter so that there are no blind spots from each of the cameras 26C, and the imaging position / angle control means 44 realizes the angles of the cameras 26C based on these past images.

[0068] FIG. 38 shows a foreign object appearance monitoring processing system configured using a foreign object appearance monitoring processing device 1B according to the tenth embodiment. As shown in FIG. 39, the foreign object appearance monitoring processing device 1B includes an IO interface 45 and a communication unit 46, and functions as an information communication unit for communicating information with other devices. As shown in FIG. 40, the foreign object appearance monitoring processing device 1B includes an image capture position information acquisition means 47 in the main memory 11. The image capture position information acquisition means 47 acquires (stores) image capture position information captured by the camera 26C. The image capture position information may be a mixture of numbers and letters, or may be written in a way that identifies the installation location, such as "top right of device 2 in building A." The foreign object appearance monitoring processing device 1B transmits the image capture position information acquired by the image capture position information acquisition means 47, along with image data captured by the camera 26C, to a foreign object appearance monitoring processing center device 500 via the (information) communication unit 46.

[0069] 38, the foreign object appearance monitoring processing center device 500 has a communication unit 501 and a display unit 502, and performs foreign object appearance monitoring processing in the same manner as the foreign object appearance monitoring processing device 1 based on image data sent from the foreign object appearance monitoring processing device 1B, and displays the necessary information on the display unit 502. Therefore, the foreign object appearance monitoring processing center device 500 has components such as the foreign object appearance / range detection means 33 and foreign object type detection means, while the foreign object appearance monitoring processing device 1B does not require these components.

[0070] It is also possible to provide the foreign object appearance monitoring processing device 1 according to the first to eighth embodiments with the configuration of the foreign object appearance monitoring processing device 1B, and have the foreign object appearance monitoring processing center device 500 have configurations such as the foreign object appearance / range detection means 33 and the foreign object type detection means, and not have to provide these configurations in the foreign object appearance monitoring processing device 1. The (information) communication unit 46 may thin out the image data held in the image holding unit 31 at predetermined time intervals before transmitting. In this way, in a system in which a large number of foreign object appearance monitoring processing devices 1 and foreign object appearance monitoring processing devices 1B are provided, it is possible to prevent data loss due to communication congestion and reduce power consumption due to communication.

[0071] FIG. 41 shows the configuration of a foreign object appearance monitoring processing device 1 according to the eleventh embodiment. This embodiment has a configuration in which an optical rendering unit 52 connected to an IO interface 51 is added to the first embodiment. This embodiment has a configuration in which a future image prediction means 53 and a predicted image creation and display control means 54 are added to the main memory 11, as shown in FIG. 42, compared to the first embodiment. The future image prediction means 53 obtains image data by making predictions at a time point in the future relative to the time point at which the image data stored in the image storage unit 31 was acquired. The predicted image creation and display control means 54 creates and controls the display of a predicted image of a foreign object, with the imaging target as the screen, based on the image data obtained by the prediction.

[0072] Data on the predicted image of the foreign substance created by the predicted image creation and display control means 54 is sent from the CPU 10 via the IO interface 51 to the optical rendering unit 52, which then emits light corresponding to the predicted image of the foreign substance. The emitted light causes a predicted image 137 to be projected onto the floor 103, following the leaking liquid 130, which is the foreign substance that was actually detected, as shown in Figure 43.

[0073] 44 shows the configuration of a foreign object appearance monitoring processing device 1 according to the twelfth embodiment. This embodiment has a configuration in which an RGB camera 62 connected to an IO interface 61 is added to the first embodiment. This embodiment has a configuration in which a foreign object appearance area specifying means 63 is added to the main memory 11, as shown in FIG.

[0074] That is, the foreign object appearance monitoring processing device 1 according to the twelfth embodiment is provided with an RGB camera 62 that has the same line of sight as the camera 26, in addition to the camera 26. Foreign object appearance area specifying means 63 specifies the area where a foreign object appears using image data obtained by image capture by the RGB camera 62. Foreign object appearance / range detecting means 33 detects the presence or absence of a foreign object by referring to the range of the foreign object, and foreign object type detecting means 34 detects the type of foreign object by referring to the range of the foreign object.

[0075] 46, the foreign object appearance monitoring processing device 1 according to the twelfth embodiment detects a liquid leak that has occurred under the device 100 in the rain, as shown in FIG. 46, and it is expected that the position (appearance and range of foreign object) of the liquid leak 130 can be detected with higher accuracy in the image capture area C captured by the RGB camera 62 compared to when the image is captured by the camera 26. After the position (appearance and range of foreign object) of the liquid leak 130 has been identified, it is expected that the type of foreign object can be detected with higher accuracy by measuring the intensity of light reflection from the image captured by the camera 26.

[0076] 47, if a foreign object 130A such as oil flows out into the ocean, river, or aquarium 100A, and the position (appearance and range of the foreign object) of the foreign object 130A is detected in the image capture area C captured by the RGB camera 62 using the foreign object appearance monitoring processing device 1 according to the twelfth embodiment, it is expected that detection will be possible with higher accuracy than when the image is captured by the camera 26. After the position (appearance and range of the foreign object) of the foreign object 130A is identified, it is expected that the type of foreign object will be detected with higher accuracy by measuring the intensity of light reflection from the image captured by the camera 26.

[0077] It is also likely that the degree to which foreign matter (e.g., liquid leaks, chemicals, oil, etc.) is mixed into the liquid can be detected as the "type of foreign matter."Furthermore, it is expected that the technology will be able to accurately detect foreign matter (especially liquid leaks, chemicals, oil, etc.) in rooms and halls with black walls, ceilings, and floors that absorb infrared light, in places where there are multiple lights that interfere with the emitted light, and on mirrors, windows, and synthetic resins.

[0078] The foreign matter appearance monitoring processing device 1 of this 12th embodiment is expected to be very effective when there is moisture 130B as a foreign matter in the soil 100B as shown in Figure 48, when there is frost 130C as a foreign matter in the soil 100B as shown in Figure 49, or when there is a wet spot 130D as a foreign matter in the wood 100C as shown in Figure 50.

[0079] Although each embodiment has been described above as an independent configuration, it is also possible to create a plurality of embodiments by combining several embodiments and their modifications. [Explanation of symbols]

[0080] 5A Communication Interface 5B Communication Interface 6 Output interface 7 Alarm generator 7A Alarm generation control means 8 IO interface 9 color sensors 10 CPU 11 Main memory 12 Bus 13 External memory interface 14 Input interface 15 Display interface 16 IO interface 18 IO Interface 19 Distance Sensor 22 Pointing device 23 External storage device 24 Input device 25 Display device 26 Camera 26-0 Camera body 26-1 Light irradiation section 26-2 Light reception section 26A, 26B, 26C, 26K Camera 27 Required foreign object appearance area 28 Door 29 Locking / Unlocking Control Mechanism 31 Image storage unit 32 Image storage control means 33 Foreign object appearance / range detection means 34 Foreign object type detection means 35 Specific direction image acquisition means 36 Predetermined image detection means 37 Operation control means 38 Imaging direction changing means 39 Light intensity changing means 41 Irradiation light intensity changing means 43 Drive unit 44 Imaging position / angle control means 45 IO interface 46 Communication unit 47 Imaging position information acquisition means 51 IO interface 52 Optical drawing unit 53 Future image prediction means 54 Predicted image creation and display control means 61 IO interface 62 Camera 63 Foreign object appearance area identification means 100 Device 100A In the sea, rivers, and aquariums 100B Soil 100C Wood 102 Ceiling 103 Floor 104, 104B Equipment 110 Floor surface 120 Leak detection tape 130 Leak 130A, 130B Foreign matter 130C Frost 130D Wet Location 130S Droplets 130U Mobile 130W Water Supply 131 Leakage trace 135 Reflected light 137 Predicted image 140 Falling object 150 Pattern line 151A, 151B Wall 160 Image data 180 Droplets 190 Water pipes and water supply pipes 195 Water storage 196 Flow line 500 Foreign object appearance monitoring processing center device 501 communication unit 502 display unit

Claims

1. a camera including a light emitting unit for emitting light, and a camera body having a plurality of pixels for receiving the light reflected from the irradiated light at a light receiving unit and converting the received light into multi-value pixel data according to the intensity of the received light, the plurality of pixels providing a matrix-like one frame image; an image storage unit for storing image data; an image storage control means for obtaining image data in time series by the camera and storing the image data in the image storage unit; a foreign matter appearance / area detection means for detecting the appearance of a foreign matter and the extent of the foreign matter based on a comparison of image data at a predetermined time stored in the image storage unit with image data from before that time or image data obtained in advance when no foreign matter appears; a foreign matter type detection means for detecting the type of foreign matter present in the range of foreign matter detected by the foreign matter appearance / range detection means based on the image data stored in the image storage unit; A foreign object appearance monitoring processing device comprising:

2. a specific direction image acquisition means for acquiring image data in a specific direction relative to the imaging line of sight of the camera; a predetermined image detection means for detecting whether image data of a predetermined image is included in the image data obtained by the specific direction image acquisition means; an operation control means for controlling, when a predetermined image is detected by the predetermined image detection means, operations such that an image is taken by the camera, the image data is stored in an image processing unit by the image storage control means, and detection is performed by the foreign matter appearance / range detection means and the foreign matter type detection means; 2. The foreign matter appearance monitoring processing device according to claim 1, further comprising:

3. a distance detection means for detecting a distance to an object when an image is captured by the camera; 2. The foreign matter appearance monitoring processing device according to claim 1, wherein said foreign matter type detecting means detects the type of foreign matter based on the image data stored in said image storage unit and the distance detected by said distance detecting means.

4. an imaging direction changing means for changing the imaging direction of the camera; 4. The foreign object appearance monitoring processing device according to claim 3, wherein said image storage control means causes said image storage unit to store image data in chronological order together with image pickup direction data when said camera takes an image.

5. The light irradiating unit includes a light intensity changing means for changing the intensity of the light to be irradiated, 4. The foreign matter appearance monitoring processing device according to claim 3, wherein the image storage unit stores image data in chronological order together with light intensity data when the intensity of the irradiated light is changed.

6. 5. The foreign object appearance monitoring processing device according to claim 4, wherein said imaging direction changing means changes and fixes the imaging direction of said camera from the original position to a direction directly below in the direction of gravity.

7. a color detection means for detecting color information of an object when the object is photographed by the camera; 3. The foreign matter appearance monitoring processing device according to claim 2, wherein the foreign matter type detecting means detects the type of foreign matter based on the image data stored in the image storage unit and the color information detected by the color detecting means.

8. an alarm generating means for generating an alarm regarding the appearance of a foreign object; an alarm generation control means for controlling the generation of an alarm by the alarm generation means in accordance with the manner in which the foreign object appears, as detected by the foreign object appearance / range detection means; 2. The foreign matter appearance monitoring processing device according to claim 1, further comprising:

9. a distance detection means for detecting a distance to an object when an image is captured by the camera; 2. The foreign matter appearance monitoring processing device according to claim 1, wherein the image storage control means causes the image data when the distance detected by the distance detection means is a predetermined value to be stored in the image storage unit.

10. 2. The foreign object appearance monitoring processing device according to claim 1, further comprising an imaging position / angle control means for storing a plurality of positional information at which a predetermined angle is obtained when an image of an object is captured by a camera and past images of when the angle at that position was achieved, and for realizing the angle of the camera based on the past images when the object is captured at any of the plurality of positions.

11. 2. The foreign object appearance monitoring processing device according to claim 1, wherein cameras each having a predetermined angle for capturing an image of an object are arranged at a plurality of positions, and images are captured by the cameras at the plurality of positions.

12. an imaging position information acquisition means for acquiring imaging position information of a camera; an information communication unit that communicates information with other devices; Equipped with The foreign object appearance monitoring processing device according to claim 10, characterized in that the imaging position information acquired by the imaging position information acquisition means is transmitted via the information communication unit together with the image data captured by the camera to a foreign object appearance monitoring processing center device that performs foreign object appearance monitoring processing.

13. 13. The apparatus for monitoring and processing the appearance of a foreign object according to claim 12, wherein the information communication unit thins out the image data stored in the image storage unit at predetermined time intervals before transmitting the data.

14. a future image prediction means for obtaining image data by making a prediction at a time point in the future relative to the time point at which the image data stored in the image storage unit is acquired; a predicted image creation and display control means for creating and displaying a predicted image of the foreign object on the screen of the imaging target based on the image data obtained by the prediction; 2. The foreign matter appearance monitoring processing device according to claim 1, further comprising:

15. an RGB camera having the same line of sight as the camera, separate from the camera; a foreign substance appearance area specifying means for specifying an area where a foreign substance appears using image data obtained by imaging with the RGB camera; the foreign matter appearance / range detection means detects whether or not a foreign matter has appeared by referring to the different range; 2. The foreign matter appearance monitoring processing device according to claim 1, wherein said foreign matter type detecting means detects the type of foreign matter by referring to the range of the foreign matter.

16. 2. The foreign matter appearance monitoring processing device according to claim 1, further comprising an irradiation light intensity changing means for changing the intensity of the light irradiated from said light irradiating section.

17. a camera including a light emitting unit for emitting light, and a camera body having a plurality of pixels for receiving the light reflected from the irradiated light at a light receiving unit and converting the received light into multi-value pixel data according to the intensity of the received light, the plurality of pixels providing a matrix-like one frame image; an image storage unit for storing the image data; When performing a foreign object appearance monitoring process in a foreign object appearance monitoring processing device comprising: an image storage step of obtaining image data in time series by the camera and storing the image data in the image storage unit; a foreign substance appearance / extent detection step for detecting the appearance of a foreign substance and the extent of the foreign substance based on a comparison of the image data at a predetermined time point stored in the image storage unit with previous image data or image data obtained in advance when no foreign substance has appeared; a foreign matter type detection step of detecting the type of foreign matter present in the range of foreign matter detected in the foreign matter appearance / range detection step based on the image data stored in the image storage unit; A method for monitoring the appearance of a foreign object, comprising:

18. a specific direction image acquisition step of acquiring image data in a specific direction relative to the imaging line of sight of the camera; a predetermined image detection step of detecting whether image data of a predetermined image is included in the image data obtained by the specific direction image acquisition step; an operation control step of controlling, when a predetermined image is detected by the predetermined image detection step, operations to capture an image by the camera, store the image data in an image processing unit by the image storage step, detect by the foreign object appearance / range detection step, and detect by the foreign object type detection step; 18. The method for monitoring the appearance of a foreign object according to claim 17, wherein the method further comprises:

19. a distance detection step of detecting a distance to an object when an image is captured by the camera; 18. The foreign object appearance monitoring processing method according to claim 17, wherein the foreign object type detection step detects the type of foreign object based on the image data stored in the image storage unit and the distance detected in the distance detection step.

20. an imaging direction changing step of changing an imaging direction of the camera; 20. The method for monitoring the appearance of a foreign object according to claim 19, wherein said image storage step stores image data in chronological order in said image storage unit together with image capture direction data when said camera captures an image.

21. a light intensity changing step of changing the intensity of light to be irradiated in the light irradiating unit, 20. The foreign object appearance monitoring processing method according to claim 19, wherein the image storage unit stores image data in chronological order together with light intensity data when the intensity of the irradiated light is changed by the image storage step.

22. 21. The method for monitoring the appearance of a foreign object according to claim 20, wherein the imaging direction changing step changes and fixes the imaging direction of the camera from the direction of gravity to a direction directly below from the original position.

23. a color detection step of detecting color information of an object when the object is imaged by the camera; 19. The method for monitoring the appearance of a foreign object according to claim 18, wherein the foreign object type detection step detects the type of the foreign object based on the image data stored in the image storage unit and the color information detected in the color detection step.

24. 18. The foreign object appearance monitoring processing method according to claim 17, further comprising an alarm generation control step of controlling the generation of an alarm by an alarm generation means that generates an alarm regarding the appearance of a foreign object, depending on the manner in which the foreign object appears detected by the foreign object appearance / range detection step.

25. a distance detection step of detecting a distance to an object when an image is captured by the camera; 18. The method for monitoring the appearance of a foreign object according to claim 17, wherein the image storage control step stores image data in the image storage unit when the distance detected in the distance detection step is a predetermined distance.

26. 18. The foreign object appearance monitoring processing method according to claim 17, further comprising an imaging position / angle step for storing a plurality of positional information at which the angle when imaging the object with a camera is a predetermined angle and past images when the angle at that position was realized, and for realizing the angle by the camera based on the past images when any of the plurality of positions is reached.

27. 18. A method for monitoring the appearance of a foreign object according to claim 17, characterized in that cameras each having a predetermined angle for capturing an image of an object are arranged at a plurality of positions, and the cameras at the plurality of positions are used sequentially or simultaneously to capture images.

28. The foreign object appearance monitoring processing device is provided with an information communication unit for communicating information with other devices, an imaging position information acquisition step of acquiring imaging position information of a camera; A foreign object appearance monitoring processing method as described in claim 26, characterized in that when the imaging position information acquired by the imaging position information acquisition step is transmitted together with the image data captured by the camera to a foreign object appearance monitoring processing center device that performs foreign object appearance monitoring processing, the transmission is via the information communication unit.

29. 29. The method for monitoring the appearance of a foreign object according to claim 28, wherein, when communicating using said information communication section, the image data held in said image holding section is thinned out at predetermined time intervals and then transmitted.

30. a future image prediction step of obtaining image data by making predictions at a time point in the future relative to the time point at which the image data stored in the image storage unit is acquired; a predicted image creation and display control step of creating and displaying a predicted image of the foreign object on the screen of the imaging target based on the image data obtained by the prediction; 18. The method for monitoring the appearance of a foreign object according to claim 17, further comprising:

31. The foreign object appearance monitoring processing device is provided with an RGB camera having the same line of sight as the camera, in addition to the camera; a foreign substance appearance area specifying step of specifying an area where a foreign substance appears using image data obtained by imaging with the RGB camera; In the foreign object appearance / range detection step, the presence or absence of a foreign object is detected by referring to the range of the foreign object; 19. The method for monitoring the appearance of a foreign object according to claim 18, wherein the foreign object type detection step detects the type of the foreign object by referring to the range of the foreign object.

32. 18. The method for monitoring the appearance of a foreign object according to claim 17, further comprising an irradiation light intensity changing step of changing the intensity of the light irradiated from the light irradiating unit.

Images