Particle appearance monitoring apparatus and particle appearance monitoring method

Abstract

To provide a foreign matter appearance monitoring processor which is not greatly affected by the size, quantity and appearance position of foreign matter appearing.SOLUTION: This device is provided with a camera 26 for obtaining image data by photographing a required foreign matter appearance area where appearance of required foreign matter is predicted, an image holding means 31 for obtaining and holding image data in time series by the camera 26, and a processing means 40 for performing processing including image processing related to appearance of the required foreign matter based on the image data held by the image holding means 31 to detect presence / absence of appearance of the required foreign matter. The camera 26 is composed of a plurality of cameras each having a light irradiation part for irradiating light and a light receiving part for receiving light reflected from a required foreign matter appearing area. The image processing device has an interference area specifying means 82 for specifying an interference area based on image data obtained by each camera and an interference removing means 83 for performing interference removing processing by level correction to image data of the interference area in image data held by the image holding means 31.SELECTED DRAWING: Figure 37

Description

[Technical Field]

[0001] The present invention relates to a foreign matter appearance monitoring processing device and a foreign matter appearance monitoring processing method. [Background technology]

[0002] Conventionally, in devices that use liquid inside a housing, leak detection is performed using a liquid leak detection tape. For example, as shown in Fig. 1, a liquid leak detection tape 120 is attached to a desired position on a floor surface 110 on which a device 100 is placed to detect leaks.

[0003] However, the leakage 130 spreads from time t1 (Fig. 1(A)) to time t2 (Fig. 1(B)) to time t3 (Fig. 1(C)), and at time t3, the leakage is finally detected. Therefore, it takes time for the leakage 130 to reach the leakage detection tape 120 from time t1 when the leakage started, and the damage may become greater.

[0004] Furthermore, when attempting to detect leaks over a wide area simultaneously, the leak detection tape 120 requires the installation of many sensors because it detects leaks by point or line. It is not possible to easily determine the path of the leak in a continuous time series. When there are many types of leaks to be detected, methods that detect leaks that pass through points or lines are not sufficient. Once the leak has run out, it becomes difficult to identify the source of the leak. When the background is black, such as a floor, the background may be mistakenly detected as a leak. Water leak detection tape cannot detect oil. Water leak detection tape cannot detect leaks unless water reaches the installation location. Furthermore, physical changes to the water leak detection tape, such as peeling, cutting, or the attachment of foreign objects not intended for detection, can cause false detections. Another problem is that it is difficult to determine the amount of leaked liquid. False detections can occur due to influences from the surrounding environment, such as reflection or absorption of irradiated light. In addition, in Figure 1(C) at time t3, leakage traces 131 remain, but they are not necessarily clear, and the leakage traces may also disappear, making it difficult to identify the source of the leakage.

[0005] Patent Document 1 discloses a sheet for detecting water leaks. Patent Document 2 discloses a sheet-based water leak detection device that detects the occurrence of water leaks based on the pressing force caused by the expansion of an absorbing and expanding member. Patent Document 3 discloses a device that has a water collecting section that accumulates leaked water inside an electrical device, an optical fiber with a part disposed in the water collecting section, and detects water leaks by emitting light from the end of the optical fiber and receiving it with an optical fiber led from the water collecting section, and comparing the incident light with the received light. Patent Document 4 discloses a device that detects water leaks by providing a water collector and a condensation sensor in the water collector. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 7-92054 [Patent Document 2] Japanese Patent Application Laid-Open No. 2015-152313 [Patent Document 3] Japanese Patent Publication No. 56-142658 [Patent Document 4] Japanese Utility Model Application Publication No. 4-25248 Summary of the Invention [Problem to be solved by the invention]

[0007] Conventional liquid or water leakage detection devices have limitations in that they cannot detect leaks until a certain amount of leak has occurred, or that they cannot detect leaked liquid or water until it reaches the sensor portion.

[0008] An object of the embodiments of the present invention is to provide a foreign matter appearance monitoring processing device and a foreign matter appearance monitoring processing method that are not significantly affected by the size, amount, or location of the foreign matter that has appeared. [Means for solving the problem]

[0009] A foreign matter appearance monitoring processing device according to an embodiment of the present invention is a foreign matter appearance monitoring processing device comprising: a camera that captures an image of a foreign matter appearance region where a foreign matter is predicted to appear and obtains image data; an image storage means that obtains and stores image data in time series by the camera; and a processing means that performs processing including image processing related to the appearance of the foreign matter based on the image data stored by the image storage means to detect whether or not the foreign matter has appeared, The camera is composed of a plurality of cameras, each having a light emitting section that emits light and a light receiving section that receives light reflected from a desired foreign substance appearance area, and is characterized by having an interference area identifying means that identifies an interference area based on image data obtained by each camera, and an interference removal means that performs interference removal processing by level correction on image data of the interference area in the image data held by the image holding means, and the processing means performs processing on the image data from which interference has been removed.

[0010] A foreign matter appearance monitoring processing method according to an embodiment of the present invention includes an image holding step of obtaining and holding image data in time series by a camera that captures an image of a foreign matter appearance region where the appearance of a foreign matter is predicted, and a processing step of detecting the presence or absence of the appearance of the foreign matter by performing processing including image processing related to the appearance of the foreign matter based on the image data held in the image holding step, The camera is composed of a plurality of cameras, each having a light emitting section that emits light and a light receiving section that receives light reflected from a desired foreign substance appearance area, and the method comprises an interference area identifying step that identifies an interference area based on image data obtained by each camera, and an interference removal step that performs interference removal processing by level correction on image data of the interference area in the image data held by the image holding means, and is characterized in that in the processing step, processing is performed on the image data from which interference has been removed. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 10 is an explanatory diagram of conventional leak detection using a leak detection tape. [Figure 2] 1 is a block diagram of a foreign object appearance monitoring processing device according to a first embodiment of the present invention; [Figure 2A] 3A and 3B are diagrams showing illumination and reflection by a camera main body used in a processing device for monitoring the appearance of a foreign object according to an embodiment of the present invention; [Figure 2B] 3A and 3B are diagrams showing infrared light irradiation and reflection in a camera body used in a processing device for monitoring the appearance of a foreign object according to an embodiment of the present invention; [Figure 2C] 3A and 3B are diagrams showing infrared light irradiation and reflection in a camera body used in a processing device for monitoring the appearance of a foreign object according to an embodiment of the present invention; [Figure 3] 1 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a first embodiment of the present invention; [Figure 4] 4 is a flowchart showing the operation of the processing device for monitoring the appearance of a foreign object according to the first embodiment of the present invention. [Figure 5] FIG. 2 is a diagram showing the spread of leakage in the foreign matter appearance monitoring processing device according to the first embodiment of the present invention. [Figure 6] FIG. 10 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a second embodiment of the present invention. [Figure 7] 10 is a flowchart showing the operation of a processing device for monitoring the appearance of a foreign object according to a second embodiment of the present invention. [Figure 8] FIG. 10 is a diagram showing the spread of leakage in a foreign object appearance monitoring processing device according to a second embodiment of the present invention. [Figure 9] FIG. 10 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a third embodiment of the present invention. [Figure 10] 10 is a flowchart showing the operation of a processing device for monitoring the appearance of a foreign object according to a third embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing the spread of leakage in a foreign matter appearance monitoring processing device according to a third embodiment of the present invention. [Figure 12] FIG. 10 is a diagram showing the spread of leakage in a foreign matter appearance monitoring processing device according to a third embodiment of the present invention. [Figure 13] FIG. 10 is a diagram showing the spread of leakage in a foreign matter appearance monitoring processing device according to a third embodiment of the present invention. [Figure 14] FIG. 10 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a fourth embodiment of the present invention. [Figure 15] FIG. 10 is a diagram showing a first example of reference information created in a processing device for monitoring the appearance of a foreign substance according to a fourth embodiment of the present invention. [Figure 16] FIG. 10 is a diagram showing a second example of reference information created in the processing device for monitoring the appearance of a foreign substance according to the fourth embodiment of the present invention. [Figure 17] FIG. 11 is a diagram showing a third example of reference information created in the processing device for monitoring the appearance of a foreign substance according to the fourth embodiment of the present invention. [Figure 18] FIG. 10 is a diagram showing a fourth example of reference information created in the processing device for monitoring the appearance of a foreign substance according to the fourth embodiment of the present invention. [Figure 19] FIG. 10 is a diagram showing a fifth example of reference information created in the processing device for monitoring the appearance of a foreign substance according to the fourth embodiment of the present invention. [Figure 20] 10 is a flowchart showing the operation of a processing device for monitoring the appearance of a foreign object according to a fourth embodiment of the present invention. [Figure 21]FIG. 10 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a fifth embodiment of the present invention. [Figure 22] FIG. 13 is a diagram showing a first example of reference information created in a processing device for monitoring the appearance of a foreign substance according to the fifth embodiment of the present invention. [Figure 23] FIG. 13 is a diagram showing a second example of reference information created in the processing device for monitoring the appearance of a foreign substance according to the fifth embodiment of the present invention. [Figure 24] FIG. 13 is a diagram showing a third example of reference information created in the processing device for monitoring the appearance of a foreign substance according to the fifth embodiment of the present invention. [Figure 25] FIG. 13 is a diagram showing a fourth example of reference information created in the processing device for monitoring the appearance of a foreign substance according to the fifth embodiment of the present invention. [Figure 26] FIG. 11 is a diagram showing a fifth example of reference information created in the processing device for monitoring the appearance of a foreign substance according to the fifth embodiment of the present invention. [Figure 27] 10 is a flowchart showing the operation of a processing device for monitoring the appearance of a foreign object according to the fifth embodiment of the present invention. [Figure 28] FIG. 10 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a sixth embodiment of the present invention. [Figure 29] FIG. 13 is a diagram showing an example of explanatory variables used in machine learning by the main part of the processing device for monitoring the appearance of a foreign substance according to the sixth embodiment of the present invention. [Figure 30] 10 is a flowchart showing the operation of a processing device for monitoring the appearance of a foreign object according to a sixth embodiment of the present invention. [Figure 31] FIG. 13 is a block diagram of a foreign object appearance monitoring processing device according to a seventh embodiment of the present invention. [Figure 32] FIG. 13 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a seventh embodiment of the present invention. [Figure 32A] FIG. 13 is an explanatory diagram of height detection in a processing device for monitoring the appearance of a foreign object according to a seventh embodiment of the present invention. [Figure 33] 13 is a flowchart showing the operation of a processing device for monitoring the appearance of a foreign object according to the seventh embodiment of the present invention. [Figure 33A] FIG. 13 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a modified example of the seventh embodiment of the present invention. [Figure 33B]FIG. 13 is a diagram illustrating the distance detection process in a foreign object appearance monitoring processing device according to a modified example of the seventh embodiment of the present invention, in which a camera and a distance sensor are suspended from the ceiling to monitor from the side of the device. [Figure 33C] FIG. 13 is a diagram illustrating the distance detection process in a foreign object appearance monitoring processing device according to a modified example of the seventh embodiment of the present invention, in which a camera and a distance sensor are suspended from the ceiling to monitor from the side of the device. [Figure 34] FIG. 13 is an explanatory diagram of calibration performed in a processing device for monitoring the appearance of a foreign substance according to an eighth embodiment of the present invention. [Figure 35] FIG. 13 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to an eighth embodiment of the present invention. [Figure 36] FIG. 13 is an explanatory diagram relating to interference to be removed in a processing device for monitoring the appearance of a foreign object according to a ninth embodiment of the present invention. [Figure 37] FIG. 13 is a functional block diagram of a processing device for monitoring the appearance of a foreign object according to a ninth embodiment of the present invention. [Figure 38] FIG. 20 is an explanatory diagram of a first technique for identifying an interference occurrence region to be removed in a foreign object appearance monitoring processing device according to a ninth embodiment of the present invention. [Figure 39] FIG. 23 is an explanatory diagram of a second technique for identifying an interference occurrence region to be removed in the foreign object appearance monitoring processing device according to the ninth embodiment of the present invention. [Figure 40] FIG. 23 is an explanatory diagram of a third technique for identifying an interference occurrence region to be removed in the processing device for monitoring the appearance of a foreign object according to the ninth embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, a foreign object appearance monitoring device according to an embodiment of the present invention will be described with reference to the accompanying drawings. In each drawing, identical components are designated by the same reference numerals, and redundant description will be omitted. FIG. 2 shows a block diagram of a first embodiment of a foreign object appearance monitoring device 1 according to the embodiment. The foreign object appearance monitoring device 1 can be configured using a cloud computer, a server computer, a personal computer, or other computers. The foreign object appearance monitoring device 1 according to this embodiment considers foreign objects to include not only water and oil but also solids. In other words, foreign objects are objects that would not normally come out of a device, and the type of foreign object being detected among those objects is referred to as a desired foreign object. In other words, various liquids that may leak from a semiconductor production device are considered desired foreign objects, and if other parts or the like fall and appear, they are referred to as objects other than the desired foreign object.

[0013] The foreign object appearance monitoring processing device 1 has a CPU 10 that performs processing based on programs and data in a main memory 11. An external storage device 23 is connected to the CPU 10 via a bus 12 and an external storage interface 13, and a program for foreign object appearance monitoring processing is stored in the external storage device 23. The CPU 10 reads the program for foreign object appearance monitoring processing from the external storage device 23 into the main memory 11 and executes this program, thereby functioning as the foreign object appearance monitoring processing device 1, and during this operation a foreign object appearance monitoring processing method is executed.

[0014] In addition to the external storage interface 13, an input interface 14, a display interface 15, and an IO interface 16 are connected to the bus 12. An input device 24 such as a keyboard or a touch panel and a pointing device 22 such as a mouse are connected to the input interface 14. A display device 25 for displaying data and images is connected to the display interface 15. The display device 25 may also be equipped with a device for outputting audible alarms in addition to displaying information.

[0015] A camera 26 is connected to the IO interface 16. The camera 26 captures an image of a foreign object appearance area where a foreign object is predicted to appear, and obtains image data. As shown in FIG. 2A , the camera 26 includes a camera main body 26-0 having a main function of imaging, etc., which includes a light emitting unit 26-1 that emits light and a light receiving unit 26-2 that receives light reflected from a foreign object appearance area 27, such as a floor. The camera main body 26-0 photoelectrically converts the light obtained by the light receiving unit 26-2 to obtain planar image data using a matrix of cells. The image data may have RGB color levels or black and white levels, and may hereinafter be simply referred to as levels. 2B and 2C, when light emitting unit 26-1 is an infrared emitting unit, if there is a leak 130 as the desired foreign matter, the infrared light is absorbed by leak 130, resulting in weak or no reflection, and light receiving unit 26-2 receives reflected light in which the light from the part of leak 130 is weakened or absent. Therefore, it is possible to properly detect leak 130 as the desired foreign matter based on the difference in intensity of reflected light 135.

[0016] The foreign object appearance monitoring processing device of this embodiment has, in operation, image storage means 31 in the main memory 11 as shown in FIG. 3. The image storage means 31 acquires and stores image data in time series by the camera 26. The storage destination may be within the main memory 11 or the external storage device 23. In this embodiment, the storage destination is a storage unit 32 in the main memory 11.

[0017] Furthermore, in this embodiment, the main memory 11 includes a processing means 40. The processing means 40 performs processing, including image processing related to the appearance of the desired foreign matter, based on the image data stored in the image storage means 31 to detect the presence or absence of the desired foreign matter. In the simplest embodiment, the processing means 40 compares the image data stored in chronological order in the storage unit 32 with image data when no foreign matter is present in the image, and if a difference of a predetermined size (level) is obtained, it determines that a desired foreign matter has appeared and outputs an alarm or the like.

[0018] The processing flow is as shown in Figure 4. That is, image data is obtained in chronological order by a camera and stored (S11). Next, based on the stored image data, processing including image processing related to the appearance of a desired foreign substance is performed to detect the presence or absence of the desired foreign substance (S12). Next, the presence or absence of the desired foreign substance is determined (S13), and if the result is NO, the process returns to step S11 to continue processing, and if the result is YES, an alarm is output indicating the presence of a desired foreign substance (S14), and the process returns to step S11 to continue processing.

[0019] As a result of the above processing, the liquid leakage 130 spreads from time t1 (FIG. 5A) to time t2 (FIG. 5B) to time t3 (FIG. 5C). A threshold value (area (size) of the image data of the desired foreign object) for issuing an alarm is stored in advance. Therefore, the presence of a liquid leakage is detected at any of times t1, t2, and t3. Therefore, it is possible to detect the liquid leakage 130 from time t1, when the leakage begins, and take appropriate measures to prevent damage from worsening.

[0020] Next, a second embodiment of the foreign substance appearance monitoring processing device will be described. In the foreign substance appearance monitoring processing device of the second embodiment, as shown in FIG. 6, the main memory 11 further includes a future image prediction means 41 and a determination means 42. The future image prediction means 41 obtains image data by making a prediction at a time point in the future relative to the time point at which the image data to be stored in the accumulation unit 32 by the image storage means 31 was acquired. The future image prediction means 41 can use a machine learning technique using a neural network, such as those described in Japanese Patent Nos. 7092312 and 7216264. The determination means 42 detects the presence or absence of the desired foreign substance and determines whether the area of ​​the desired foreign substance has increased or decreased based on the image data obtained by the prediction.

[0021] The processing flow of the second embodiment is as shown in FIG. 7. Specifically, image data is acquired and stored in chronological order by a camera (S11). Next, prediction is performed for a future time point relative to the time point at which the stored image data was acquired, and image data is obtained (S22). Next, based on the image data obtained by the prediction, the presence or absence of the desired foreign matter is detected, and a determination is made as to whether the area of ​​the desired foreign matter has increased or decreased (S23). First, a determination is made as to whether the foreign matter has appeared (S24). If the result is NO, the process returns to step S11 to continue processing. If the result is YES, an alarm is output indicating that the desired foreign matter is expected to appear at the predicted future time point (S25). Then, the process proceeds to step S26, where a determination is made as to whether the area of ​​the desired foreign matter has increased or decreased (S26). If the result is an increase, an alarm is output indicating an increase in the amount of foreign matter appearing (S27), and the process returns to step S11 to continue processing. If the result is a decrease in step S26, an alarm is output indicating a decrease in the amount of foreign matter appearing (S28), and the process returns to step S11 to continue processing.

[0022] As a result of the above processing, in this embodiment, as in the first embodiment, the leakage 130 spreads from time t1 (FIG. 8(A)) to time t2 (FIG. 8(B)) to time t3 (FIG. 8(C)). By setting a desired threshold for issuing an alarm, damage can be prevented from becoming too severe. Furthermore, the foreign object appearance monitoring processing device according to this embodiment can obtain image data by making a prediction at time t1, for example, from a time point in the future relative to the time of acquisition of the stored image data, and determine whether or not a foreign object has appeared, thereby minimizing damage (FIG. 8(D)). Furthermore, it is possible to grasp trends in the amount of foreign objects and take appropriate measures. FIG. 8(D) shows the state of foreign object appearance over time, demonstrating that a prediction was made at time t1 and a prediction result 140 was obtained at time tX.

[0023] Next, a third embodiment of the foreign substance appearance monitoring device will be described. In the foreign substance appearance monitoring device of the third embodiment, as shown in FIG. 9 , the main memory 11 further includes a past image prediction means 43, a foreign substance appearance image detection means 44, and an appearance location detection means 45 in addition to the components of the first embodiment. The past image prediction means 43 obtains image data by making predictions at a time point in the past relative to the time point at which the image data stored in the storage unit 32 by the image storage means 31 was acquired. This past image prediction means 43 can use the neural network machine learning techniques described in Japanese Patent Nos. 7092312 and 7216264. While the machine learning and neural network techniques described in Japanese Patent Nos. 7092312 and 7216264 predict in the future, it is obvious to those skilled in the art that predictions can be made in the past. The foreign substance appearance image detection means 44 detects the presence or absence of the desired foreign substance and detects images in which the desired foreign substance size exceeds a predetermined value based on the image data obtained by the predictions. The foreign substance appearance image detecting means 44 determines the appearance position of the desired foreign substance based on the image data detected by the foreign substance appearance image detecting means 44.

[0024] The processing flow of this embodiment is as shown in Figure 10. Specifically, image data is acquired and stored in chronological order by a camera (S11). Next, prediction is performed for a time point in the past relative to the time point at which the stored image data was acquired, and image data is obtained (S32). Next, the presence or absence of the desired foreign matter is detected based on the image data obtained by the prediction (S33, S34). If the result in step S34 is NO, the process returns to step S11 to continue processing. If the result is YES, an alarm is output indicating that the desired foreign matter is predicted to appear at the predicted past time point (S35), and the process proceeds to step S36, where images in which the desired foreign matter is larger than a predetermined size are detected (S36). In the next step S37, it is detected whether the desired foreign matter is larger than a predetermined size (S37). If the result is YES, the process proceeds to step S38, where the location of the desired foreign matter is determined based on the detected image data (S38).

[0025] In the above, "the size of the required foreign matter is greater than or equal to a predetermined size" means that the required foreign matter is a liquid leak and that the leak is greater than or equal to a predetermined size. In the case of a detachment or fall, it means that the size of the detached or fallen object is greater than or equal to a predetermined size. When the "location of the appearance of the required foreign matter" is determined in step S38 above, and the determined position is displayed on the image (S39), a predetermined mark can be superimposed on the image obtained by camera 26 (the image before it is converted into planar image data composed of matrix-like cells) at the position obtained from the planar image data composed of matrix-like cells.

[0026] As a result of the above process, in this embodiment, as in the first embodiment, detection of the leak 130 is performed in the order shown in FIG. 11(A) at time t1, followed by FIG. 11(B) at time t2, and then FIG. 11(C) at time t3. However, if no foreign object is detected in FIG. 11(A) at time t1, and a large number of foreign objects suddenly appear in FIG. 11(B) at time t2 or FIG. 11(C) at time t3, or if a large number of foreign objects suddenly appear, or if their original location cannot be identified, past image prediction is performed. In this embodiment, when the location of the foreign object at time t1 is predicted and determined at time t3, the determined location is displayed on the image (FIG. 11(E)). At this time, a predetermined mark can be superimposed on the image obtained by the camera 26 at the location determined from the planar image data using a matrix of cells.

[0027] 12 shows an example in which a prediction into the future is made at time 3 based on information from times t1, t2, and t3, and only an image portion is displayed at time t3 of the time series state of the appearance of foreign matter, which reveals that the prediction result at future time tX has resulted in leakage 130 and leakage trace 131. In other words, in this embodiment, it is possible to predict the appearance of a desired foreign matter in the future at a desired time point, and the prediction can be used to respond to future problems. 13 shows only a time-series image portion of the state of foreign object appearance that makes it clear that a prediction into the past was made at time tX+3 based on information from times tX+3, tX+2, and tX+1, and the original appearance position was obtained from the prediction result at time 1. According to this embodiment, even if the original appearance position becomes unknown due to the loss of past image data, for example, it is possible to appropriately identify the original appearance position.

[0028] A fourth embodiment of the present invention is directed to a case where there are multiple types of foreign matter. In the foreign matter appearance monitoring processing device of the fourth embodiment, as shown in Fig. 14, in addition to the configuration of the first embodiment, main memory 11 further includes reference information creating means 46, target object candidate information creating means 47, and foreign matter type identifying means 48. The reference information creating means 46 creates at least one type of reference information based on image data obtained in the past: size information of the foreign matter, variance information of the size of the foreign matter, time-varying information of the size of the foreign matter, variance information of the time-varying information of the size of the foreign matter, and growth time, which is information about the time it takes for the foreign matter to reach a predetermined size.

[0029] The object candidate information creating means 47 creates at least one type of object candidate information including size information of the desired foreign substance candidates in the image data held by the image holding means 31, variance information of the sizes of the desired foreign substance candidates, information on changes in the sizes of the desired foreign substance candidates over time, variance information of the changes in the sizes of the desired foreign substance candidates over time, and growth time, which is information on the time it takes for the desired foreign substance candidates to reach a predetermined size. The foreign substance type identifying means 48 identifies the type of the desired foreign substance with respect to the object candidate, based on the at least one type of information created by the reference information creating means 46 and the at least one type of object candidate information created by the object candidate information creating means 47.

[0030] In this embodiment, a case where there are two types of desired foreign matter will be described. When there are two types of desired foreign matter, leakage A and leakage B, the reference information creation means 46 creates size information for the desired foreign matter as shown in FIG. 15 based on image data obtained in the past for each type. Image data of the area suspected to be a foreign matter is arranged in a rectangular matrix on the right side of FIG. 15, and image data cells exist in a matrix within this rectangle, with the level of each cell having a value represented by a numerical value. Here, size information refers to the average value of the numerical values ​​arranged in the matrix.

[0031] FIG. 16 is an explanatory diagram of size variance information. Size variance information refers to the variance of numerical values ​​arranged in a matrix. The time-varying information of the size of the desired foreign particle refers to the average value of each value that changes over time when the average value of the size information at the level of all cells changes over time as shown in FIG. 17. The variance value information of the time-varying size of the desired foreign particle refers to the variance of each value that changes over time when the average value of the size information at the level of all cells changes over time as shown in FIG. 18. As shown in FIG. 19, the time information from the state where the desired foreign particle is absent (FIG. 19(A)) to the state where the desired size (area) S is reached (FIG. 19(B)) is called growth time (naturally, this is measured as the spread when the same amount is dropped at the same speed from a hole of the same size). The above is reference information, and the reference information creation means 46 creates at least one type of reference information. This information is collected and stored in advance for each type of foreign particle before the device is put into operation.

[0032] The object candidate information creation means 47 is created from the image data stored by the image storage means 31. Therefore, it is obtained in the case of actual operation. The processing flow of the fourth embodiment of the present invention is as shown in Figure 20. Image data is obtained in chronological order by the camera 26 and stored (S11). Reference information collected in advance before the device is put into operation is called up and stored in the main memory, and the data is placed on standby (S42). Object candidate information of the same type as the reference information is created from the image data obtained and stored by the camera 26 (S43). The reference information is compared with the object candidate information of the same type to identify the type of the desired foreign matter (S44). The obtained type of the desired foreign matter is displayed (S45). When there are many pieces of reference information and object candidate information that match the same type, this type can be used as the type of foreign matter.

[0033] Next, a fifth embodiment will be described. This embodiment identifies any object that appears as an object other than a desired foreign object. In the foreign object appearance monitoring processing device of the fifth embodiment, as shown in FIG. 21, in addition to the configuration of the first embodiment, the main memory 11 further includes a reference information creating means 46, a target object candidate information creating means 47, and an object other than a desired foreign object identifying means 49.

[0034] The reference information creating means 46 creates at least one type of reference information based on previously obtained image data, including size information of the foreign matter, variance information of the size of the foreign matter, time-varying information of the size change of the foreign matter, variance information of the time-varying information of the size change of the foreign matter, and growth time, which is information about the time it takes for the foreign matter to reach a predetermined size, and is the same as that provided in the fourth embodiment. The object candidate information creating means 47 creates at least one type of foreign matter candidate information based on image data stored by the image storage means, including size information of the foreign matter candidate, variance information of the size change of the foreign matter candidate, time-varying information of the size change of the foreign matter candidate, and growth time, which is information about the time it takes for the foreign matter candidate to reach a predetermined size, and is the same as that provided in the fourth embodiment.

[0035] The non-desired foreign object identification means 49 identifies the type of desired foreign object for the target candidate based on at least one type of reference information created by the reference information creation means 46 and at least one type of target candidate information created by the target candidate information creation means 47. In this embodiment, in comparing the reference information with the target candidate information, the desired foreign object is not detected, but an object other than the desired foreign object is identified.

[0036] The reference information creation means 46 creates the size information of the above-mentioned required foreign matter based on the image data obtained in the past, as shown in Figure 22. It is assumed that the area suspected to be a foreign matter is a rectangle as shown in Figure 22, and that image data cells exist in a matrix within this rectangle, with the level of each cell having a value expressed numerically. Here, the size information refers to the average value of the numbers arranged in the matrix.

[0037] FIG. 23 is an explanatory diagram of size variance information. Size variance information refers to the variance of numerical values ​​arranged in a matrix. The time-dependent change information of the size of the required foreign particle refers to the average value of each value that changes over time when the average value of the size information at the level of all cells changes over time as shown in FIG. 24. The variance value information of the time-dependent change of the size of the required foreign particle refers to the variance of each value that changes over time when the average value of the size information at the level of all cells changes over time as shown in FIG. 25. As shown in FIG. 26, the time information from the state where the required foreign particle is absent (FIG. 26(A)) to the state where the required size (area) S is reached (FIG. 26(B)) is referred to as growth time (naturally, the spread when the same amount is dropped at the same speed from a hole of the same size is measured). The above is reference information, and at least one type of reference information is created. This information is collected and stored in advance for each type of foreign particle before the device is put into operation.

[0038] The processing flow of the fifth embodiment of the present invention is as shown in Figure 27. Image data is obtained in chronological order by the camera 26 and stored (S11). Reference information collected and stored beforehand before the device is put into operation is loaded into main memory and placed on standby (S42). Object candidate information of the same type as the reference information is created from the image data obtained and stored by the camera 26 (S43). The reference information is compared with the object candidate information of the same type to determine whether the object candidate information is significantly different from the reference information or whether it is within a "possible range" (S52), and a determination is made (S53). If the determination is YES, a message is displayed indicating that an object other than a desired foreign object has appeared (S54). Specifically, a message such as "A lost or discarded item has been found" can be displayed or output.

[0039] Next, a sixth embodiment will be described. The sixth embodiment of the present invention is a case where there are multiple types of foreign matter. In the foreign matter appearance monitoring processing device of the sixth embodiment, as shown in FIG. 28, the main memory 11 is provided with an explanatory variable waveform creation means 50 and a second foreign matter type identification means 51. The second foreign matter type identification means 51 is configured using a machine learning model. This machine learning model can be created by creating waveform information (a1), (b1), and (c1) as shown in FIG. 29(B) from time-varying information (a), (b), and (c) of image data level information of the foreign matter (object other than the specified foreign matter) to be detected as shown in FIG. 29(A), and then performing machine learning using the waveform information (a1), (b1), and (c1) as explanatory variables and the foreign matter name (liquid leakage A, liquid leakage B, object A) indicating the type of foreign matter as the objective variable. This machine learning model can be created from a large number of explanatory variables and objective variables before the foreign matter appearance monitoring processing device is put into operation. Furthermore, machine learning can be performed based on explanatory variables obtained during operation of the foreign matter appearance monitoring processing device and a target variable finally determined by an operator or the like, thereby improving the accuracy of the machine learning model. The machine learning means used in this case is not shown. The explanatory variable waveform creation means 50 creates explanatory variable waveforms from image data obtained during operation of the foreign matter appearance monitoring processing device.

[0040] The processing flow of the sixth embodiment of the present invention is as shown in Figure 30. Image data is obtained in time series by camera 26 and stored (S11). The stored image data is used to create and store waveforms, which are explanatory variables for machine learning (S62). Next, the second foreign matter type identification means 51 determines the type of foreign matter from the waveform, which is the explanatory variable (S63), and the obtained type of foreign matter is displayed (S64).

[0041] Next, a seventh embodiment will be described. The seventh embodiment of the invention is directed to detecting a desired foreign object height (thickness). In the foreign object appearance monitoring processing device of the seventh embodiment, as shown in FIG. 31, a camera 26 and a distance sensor 28 are connected to the IO interface 16 of the foreign object appearance monitoring processing device 1A. The distance sensor 28 may be a type known as a TOF (Time of Flight) sensor, which measures distance based on the time it takes for a projected light pulse to be reflected and returned. The camera 26 and the distance sensor 28 are used, for example, adjacent to each other as a set, with one set disposed in each desired foreign object detection area. When there are multiple desired foreign object detection areas, multiple sets are disposed correspondingly.

[0042] The main memory 11 of the foreign object appearance monitoring processing device according to the seventh embodiment is provided with area information acquisition means 52 and quantity information acquisition means 53, as shown in Fig. 32. The area information acquisition means 52 acquires area information of the foreign object candidate based on the image data held by the image holding means 31. The quantity information acquisition means 53 acquires quantity information of the foreign object candidate based on the height information and the area information obtained by the distance sensor 28. The processing means 40 of this embodiment performs image processing related to the appearance of the foreign object based on the image data held by the image holding means 31 to detect whether or not the foreign object has appeared, and also detects whether or not the foreign object has appeared using the quantity information acquired by the quantity information acquisition means 53.

[0043] Here, the height detection method will be described. Light is projected from the distance sensor 28 onto the desired foreign object detection area in advance, and the distance L0 from the distance sensor 28 to the desired foreign object detection area is obtained based on the reflection time. As shown in FIG. 32A, for example, light is projected from the distance sensor 28 onto the desired foreign object detection area in advance at time t11, and the distance L1 from the distance sensor 28 to the desired foreign object detection area is obtained based on the reflection time. The height h1 of the leaked liquid A is obtained by subtracting the distance L1 from the distance L0. Next, as shown in FIG. 32A, light is projected from the distance sensor 28 onto the desired foreign object detection area in advance at time t12, and the distance L2 from the distance sensor 28 to the desired foreign object detection area is obtained based on the reflection time. The height h2 of the leaked liquid A at time t12 is obtained by subtracting the distance L2 from the distance L0. Furthermore, as shown in FIG. 32A, light is projected from the distance sensor 28 onto the desired foreign object detection area in advance at time t13, and the distance L3 from the distance sensor 28 to the desired foreign object detection area is obtained based on the reflection time. The height h3 of the leaked liquid A at time t13 is obtained by subtracting the distance L3 from the distance L0. The height of the leaked liquid A can be obtained in the same manner.

[0044] The processing flow of the seventh embodiment of the present invention is as shown in Figure 33. Image data is acquired and stored in chronological order by the camera 26 (S11), and height information is acquired in chronological order by the distance sensor 28. The data acquired at the same time are stored as a set (S72). Area information of the foreign object candidate is acquired based on the stored image data (S73). Quantity information (area x height) of the foreign object candidate is acquired based on the image data and the stored height information (S74). Image processing regarding the appearance of the foreign object is performed based on the stored image data to detect its presence or absence (S75), and a presence / absence determination is made (S76). If the result is NO, the process returns to step S11 and continues. On the other hand, if the result is YES, the process proceeds to step S77, where the acquired quantity information is used to detect its presence or absence (S77), and the result is displayed according to the amount (S78). In other words, based on the quantity information, it is possible to notify whether the leakage is slight or large according to the characteristics of the foreign object, the amount of which changes from appearance to appearance.

[0045] In this embodiment, since the height is determined, the height itself may be displayed or output as information. Furthermore, since the area (width x depth) is determined, the depth can also be detected, and the depth may be displayed or output alone or together with other information. As shown in FIG. 33A, an embodiment including a depth information acquisition means 54 can be realized. In this embodiment, the depth information acquisition means 54 acquires depth information of the desired foreign object candidate based on the area information acquired by the area information acquisition means and the height information acquired by the distance sensor. The processing means 40 performs processing to provide depth information and / or height information of the desired foreign object in addition to detecting the presence or absence of the desired foreign object. In other words, the processing means 40 can also perform processing to provide distance information to the desired foreign object in addition to detecting the presence or absence of the desired foreign object.

[0046] In this embodiment, foreign object detection from image data is prioritized over height-based detection. However, detection by the distance sensor 28 may be prioritized, or both may be weighted equally, with re-detection processing performed for the method that fails to detect a foreign object (foreign object detection from image data or height-based detection). If no foreign object is detected by either method, priority is given to foreign object detection from image data or height-based detection. In this embodiment, one distance sensor 28 is provided corresponding to the camera 26, but multiple distance sensors may be provided. In this case, the corresponding range of each distance sensor 28 may be set to detect the range or area where a foreign object appears, and the height of each detected foreign object may be displayed. Furthermore, the quantity information detected in this embodiment may be used as reference information for use in the fourth and fifth embodiments.

[0047] In each of the previous embodiments, such as those shown in FIGS. 5, 8, and 11, camera 26 was positioned on the ceiling side and captured an image in the direction of floor surface 110 to obtain image data. In a modified example of the seventh embodiment shown in FIGS. 33B and 33C, camera 26 and distance sensor 28 are suspended from ceiling 115 in FIG. 33B, and camera 26 and distance sensor 28 are mounted on a support rod extending from floor surface 110 in FIG. 33C, and distance is detected in a foreign object appearance monitoring processing device configured to monitor from the side of device 100. In FIG. 33B, device 100 is placed on device stand 100A, and leaked liquid A flows from directly below device 100, along the top plate of device stand 100A, and downward along the side of device stand 100A. In FIG. 33C, leaked liquid A flows from the contact point between ceiling 115 and device 100, along the side of device 100, and downward. In these cases, the distance sensor 28 can detect the distance L to the surface of the leaking liquid A flowing downward along the side of the equipment stand 100A or the equipment 100. In such cases, as explained with reference to Fig. 32A, the height of the leaking liquid A flowing downward along the side of the equipment stand 100A or the equipment 100 can be determined and output. Naturally, the distance L can also be determined and output.

[0048] Next, an eighth embodiment will be described. The eighth embodiment of the invention is primarily concerned with the case where calibration is performed depending on the surrounding environment. In the foreign object appearance monitoring processing device of the eighth embodiment, position correction (calibration) is required when a foreign object, i.e., a leakage liquid A, appears in the center of the image as shown in FIG. 34(A) and when a foreign object, i.e., a leakage liquid A, appears in a corner or edge of the image as shown in FIG. 34(B). Furthermore, when the types of foreign objects differ as shown in FIG. 34(C) and FIG. 34(D), calibration is required because the absorbance, reflectance, and attenuation rate depending on distance are different. In this embodiment, we will describe the case where calibration is required for an appearance position and the case where the types of foreign objects are different. However, this is not limited to these. Calibration is performed using a method appropriate for each case when the temperature or humidity is different, the background (color, brightness, etc.) is different, the size of the foreign object that appears is a very small powder as shown in FIG. 34(E), or the foreign object that appears moves around like a small insect as shown in FIG. 34(F).

[0049] As shown in Fig. 35, the main memory 11 of the foreign matter appearance monitoring processing device according to the eighth embodiment is equipped with a case information storage means 62, a necessity determination means 63, a case-specific calibration means 64, and a calibration control means 65. The case information storage means 62 stores information on cases in which calibration is required when the appearance of a desired foreign matter is detected. The necessity determination means 63 searches the case information storage means 62 when the appearance of a desired foreign matter is detected to determine whether or not the case corresponds to that case. The case-specific calibration means 64 performs calibration for each case in which calibration is required.

[0050] When the appearance position (case) requires calibration, the case-specific calibration means 64 divides the image into m × n cells and performs calibration using a table in which a correction coefficient is set for the level of each cell. Instead of a table, a machine learning model created by performing machine learning to obtain a value obtained by multiplying each m-row × n-column cell by a correction coefficient (machine learning using the level before multiplication of each cell by the correction coefficient as the explanatory variable and the level (which may be the actual level value) after multiplication by each cell by the correction coefficient as the objective variable) can be used. Furthermore, when the types of foreign matter differ (cases), the case-specific calibration means 64 can perform calibration by preparing an m-row × n-column correction coefficient table according to the type. Alternatively, a machine learning model created by performing machine learning using the type as the objective variable and the level (which may be the actual level value) after multiplication by each cell by the correction coefficient as the objective variable) can be used. When the image data has RGB color levels, calibration can be performed on these color levels.

[0051] When the calibration necessity determination means 63 determines that calibration is necessary, the calibration control means 65 selects the case-specific calibration means 64 corresponding to the required case and causes calibration to be performed. In this embodiment, the processing means 40 performs processing using the result information calibrated by the selected case-specific calibration means 64. In other words, calibration means that the image data that is the premise for processing by the processing means 40 is calibrated to an appropriate value. The processing means 40 includes the various means of each embodiment, and as a result, it is possible to improve the accuracy of determining, predicting, and detecting the presence or absence of foreign matter.

[0052] Next, a ninth embodiment will be described. In the ninth embodiment of the invention, as described in Fig. 2A, camera 26 has light emitting unit 26-1 that emits light and light receiving unit 26-2 that receives light reflected from a desired foreign substance appearance area, and as shown in Fig. 36, a plurality of cameras are configured, causing interference and creating an interference area IFE.

[0053] As shown in Fig. 37, the main memory 11 of the foreign object appearance monitoring processing device according to the ninth embodiment is provided with an interference area identifying means 82 and an interference removal means 83. The interference area identifying means 82 identifies an interference area based on image data obtained by each camera. The interference removal means 83 performs interference removal processing by level correction on image data of the interference area in the image data held by the image holding means 31.

[0054] As shown in Fig. 38, the interference area identification means 82 obtains the cell levels during test imaging with each camera, compares them, and identifies the cell with the highest level as the cell in the interference area. Alternatively, as shown in Fig. 39, the variance σ of the cell levels is obtained during test imaging with each camera, and compares them and identifies the cell with the highest variance as the cell in the interference area. As shown in Fig. 40, the cell with the largest level difference between when two cameras are operated and there is interference and when one camera is operated and there is no interference is identified as the cell in the interference area. Although not shown, the cell levels are obtained in time series during test imaging with each camera, and the variance of this time series data is obtained and the cell with the largest variance is identified as the cell in the interference area. Furthermore, it is possible to use several of these to make a decision by majority vote.

[0055] The interference removal means 83 can be created by performing machine learning on the levels of actual image data without interference and the levels of images with interference obtained by imaging during operation or testing of the device, to obtain a machine learning model.

[0056] The processing means 40 processes the image data from which interference has been removed. The above-described embodiments can be realized as an apparatus employing some or all of them. [Explanation of symbols]

[0057] 1. 1A Foreign object appearance monitoring processing device 10 CPU 11 Main Memory 12 Bus 13 External memory interface 14 Input Interface 15 Display Interface 16 IO interfaces 22 Pointing Device 23 External storage device 24 Input Devices 25 Display device 26 Camera 26-0 Camera body 26-1 Light irradiation unit 26-2 Light receiving section 27 Required foreign material appearance area 28 Distance Sensor 31 Image retention means 32 Storage unit 40 Processing means 41 Future image prediction method 42 Judgment means 43 Past image prediction method 44 Foreign object appearance image detection means 45 Appearance location detection means 46 Reference Information Creation Methods 47 Object candidate information creation means 48 Foreign object type identification means 49 Required foreign object identification means 50. Means for creating explanatory variable waveforms 51 Second foreign material type identification means 52 Means of obtaining area information 53 Quantity information acquisition means 54 Depth information acquisition means 62 Case information storage means 63 Means for determining necessity / unnecessity 64 Case-specific calibration methods 65 Calibration Control Means 82 Interference area identification means 83 Interference Removal Means 100 devices 100A device stand 110 Floor 115 Ceiling 120 Leak detection tape 130 Leakage 131 Leakage trace 140 Prediction Results

Claims

1. a camera for capturing an image of a region where a desired foreign substance is expected to appear and obtaining image data; image storage means for obtaining and storing image data in time series by the camera; a processing means for detecting the presence or absence of the desired foreign matter by performing processing including image processing relating to the appearance of the desired foreign matter based on the image data stored by the image storage means; A foreign object appearance monitoring processing device comprising: the camera is composed of a plurality of cameras each having a light emitting unit for emitting light and a light receiving unit for receiving light reflected from a desired foreign substance appearance area, an interference area specifying means for specifying an interference area based on image data obtained by each camera; an interference removal means for performing an interference removal process by level correction on image data of the interference region in the image data stored by the image storage means; and The processing device for monitoring the appearance of a foreign substance is characterized in that the processing means processes image data from which interference has been removed.

2. 2. The foreign matter appearance monitoring processing device according to claim 1, wherein level and color information is used as image data for processing.

3. an image storage step of acquiring and storing image data in time series by a camera that captures an image of a region where a desired foreign substance is expected to appear, where the desired foreign substance is expected to appear; a processing step of detecting whether or not the desired foreign matter has appeared by performing processing including image processing related to the appearance of the desired foreign matter based on the image data stored in the image storing step; A method for monitoring the appearance of a foreign object, comprising: the camera is composed of a plurality of cameras each having a light emitting unit for emitting light and a light receiving unit for receiving light reflected from a desired foreign substance appearance area, an interference area specifying step of specifying an interference area based on image data obtained by each camera; an interference removal step of performing an interference removal process by level correction on image data of the interference region in the image data stored by the image storage means; and A method for monitoring the appearance of a foreign substance, wherein the processing step performs processing on image data from which interference has been removed.

4. 4. The method for monitoring the appearance of a foreign substance according to claim 3, wherein the processing is performed using level and color information as image data.

Images