Abnormality detection method and abnormality detection system

Abstract

The objective of this invention is to provide an anomaly detection method and system that can quickly and accurately determine abnormal noises and detect anomalies. A machine learning model 99 is constructed using machine learning based on image data captured from various parts installed within the substrate processing device and sound data from the sounds emitted by those parts during operation. As sound data, data on normal sounds emitted by each part during normal operation and data on abnormal sounds emitted by the part during abnormal operation are collected. During anomaly detection, eye-tracking is used to pinpoint the location viewed by the operator wearing smart glasses equipped with the machine learning model 99. An area including the pinpointed viewing location is captured, and sound is collected while the operator is viewing that location. The collected image and sound data are then input into the machine learning model 99.

Description

Technical Field

[0001] This invention relates to an anomaly detection method and system for detecting abnormal states in industrial machinery such as substrate processing apparatus that performs prescribed processing on substrates. The substrates processed by the substrate processing apparatus include, for example, semiconductor substrates, substrates for liquid crystal display devices, substrates for flat panel displays (FPDs), substrates for optical discs, substrates for magnetic disks, or substrates for solar cells. Prior Technology

[0002] As one of the inspection methods for drive components such as motors contained in industrial machinery such as substrate processing equipment, the industry has previously conducted inspections based on sound. Previously, inspectors would determine abnormalities (malfunctions) by listening for unusual noises from the parts, but this method is highly dependent on the inspector's skill level, and inexperienced inspectors sometimes cannot detect abnormalities.

[0003] Therefore, anomaly determination is performed by mechanically comparing the sound collected by a microphone or other means with a reference tone. For example, Patent Document 1 discloses a technique in which the sound generated within the field of vision of an operator wearing a head-mounted display equipped with a directional microphone is collected using the directional microphone as a measurement tone, and the measurement tone is compared with a reference tone to determine whether the measurement tone is an abnormal sound.

[0004] [Previous Technical Documents] [Patent Literature] [Patent Document 1] Japanese Patent Application Publication No. 2010-197361 Summary of the Invention

[0005] [The problem the invention aims to solve] However, generally speaking, sound-emitting components are densely packed in substrate processing devices, resulting in numerous sound-emitting components within the operator's field of vision. Consequently, the operator must select and specify the reference tone to be compared with the collected measurement sound one by one, a time-consuming process.

[0006] Furthermore, there are also cases where multiple nearby locations emit sounds simultaneously. Since the wavelength of the sound changes depending on the type of anomaly, there is considerable room for improvement in order to bring the accuracy of sound anomaly detection technology close to that of a skilled operator.

[0007] The present invention was made in view of the above-mentioned problems, and its purpose is to provide an anomaly detection method and anomaly detection system that can quickly and accurately determine abnormal noises and detect anomalies.

[0008] [Technical means to solve the problem] To address the aforementioned problems, the first-state anomaly detection method of the present invention detects abnormal states of industrial machinery and includes: a learning step, which generates a machine learning model by learning from image data of parts of the aforementioned industrial machinery captured by photography and sound data of the sounds emitted by those parts during operation; a specifying step, which causes an operator wearing a mobile terminal including a sound collector, a display unit, a camera unit, a communication unit, and the aforementioned look-tracking unit to specifically view the viewing location of the aforementioned industrial machinery; a collection step, which, while the operator is viewing the aforementioned viewing location, causes the sound collector to collect sound and causes the camera unit to capture images of the area including the aforementioned viewing location; and a determination step, which determines whether the object viewed by the operator is abnormal by inputting the sound data and image data collected in the aforementioned collection step into the aforementioned machine learning model.

[0009] Furthermore, the second state sample is an anomaly detection method similar to the first state sample, wherein in the aforementioned learning process, sound data of the sounds emitted by the aforementioned part during normal operation is learned.

[0010] Furthermore, the third state sample is an anomaly detection method similar to the second state sample, wherein in the aforementioned learning process, the sound data of the sound emitted when the aforementioned part performs an abnormal action is further learned.

[0011] Furthermore, the fourth state sample is an anomaly detection method like any of the first to third states sample, wherein in the aforementioned learning process, the learning is performed on the sound data of synthesized sounds emitted by multiple parts of the aforementioned industrial machine when they operate simultaneously.

[0012] Furthermore, the fifth state is an anomaly detection method as in any of the first to fourth states, wherein the aforementioned machine learning model includes: a first model, which outputs the parts contained in the image data after inputting the image data; and a second model, which determines whether the sound represented by the sound data is an abnormal noise after inputting the sound data; and the aforementioned determination process includes: a part identification process, which identifies the aforementioned object part by inputting the image data collected in the aforementioned collection process into the aforementioned first model; and an abnormal noise determination process, which determines whether the aforementioned object part is abnormal by inputting the sound data collected in the aforementioned collection process into the aforementioned second model.

[0013] Furthermore, the sixth state sample is an anomaly detection method like any of the first to fifth state samples, wherein the aforementioned industrial machine is a substrate processing device that performs prescribed processing on the substrate.

[0014] Furthermore, the seventh state sample is an anomaly detection method as used for any of the first to sixth states, wherein the aforementioned mobile terminal is a smart glasses.

[0015] Furthermore, the anomaly detection system of the eighth state detects abnormal states of industrial machinery and includes: a mobile terminal, which includes a sound collection unit, a display unit, a camera unit, a communication unit, and a gaze tracking unit; and a learner, which generates a machine learning model by learning from image data captured by the aforementioned parts of the industrial machinery and sound data emitted by those parts during operation; the aforementioned gaze tracking unit is specifically used by the operator wearing the aforementioned mobile terminal to view the aforementioned industrial machinery at the viewing position; while the operator is viewing the aforementioned viewing position, the aforementioned sound collection unit collects sound, and the aforementioned camera unit captures an image of the area including the aforementioned viewing position; the aforementioned mobile terminal further includes a determination unit, which determines whether the object viewed by the operator is abnormal by inputting the sound data collected by the aforementioned sound collection unit and the image data captured by the aforementioned camera unit into the aforementioned machine learning model.

[0016] Furthermore, the 9th state is an anomaly detection system similar to the 8th state, wherein the aforementioned learner learns sound data of the sounds emitted by the aforementioned parts during normal operation.

[0017] Furthermore, the 10th state is an anomaly detection system similar to the 9th state, wherein the aforementioned learner further learns the sound data of the sounds emitted when the aforementioned part performs abnormal actions.

[0018] Furthermore, the 11th state is an anomaly detection system like any of the 8th to 10th states, wherein the aforementioned learner learns the sound data of synthesized sounds emitted by multiple parts of the aforementioned industrial machine when they operate simultaneously.

[0019] Furthermore, the 12th state is an anomaly detection system like any of the 8th to 11th states, wherein the aforementioned machine learning model includes: a first model, which outputs the parts contained in the image data after inputting the image data; and a second model, which determines whether the sound represented by the sound data is abnormal after inputting the sound data; and the aforementioned determination unit, after identifying the aforementioned object part by inputting the image data captured by the aforementioned camera unit into the aforementioned first model, determines whether the aforementioned object part is abnormal by inputting the sound data collected by the aforementioned sound collecting unit into the aforementioned second model.

[0020] Furthermore, the 13th state sample is an anomaly detection system like any of the 8th to 12th states sample, wherein the aforementioned industrial machine is a substrate processing device that performs prescribed processing on the substrate.

[0021] Furthermore, the 14th state is an anomaly detection system like any of the 8th to 13th states, wherein the aforementioned mobile terminal is a smart glasses.

[0022] [Effects of the Invention] According to the anomaly detection method of states 1 to 7, a machine learning model is generated by learning from the image data of the parts of the industrial machine and the sound data of the sound emitted by the parts when they are in operation. The sound data collected by the sound collector and the image data collected by the camera are input into the machine learning model when the operator is looking at the viewed position to determine whether the object part seen by the operator is abnormal. Therefore, the operator can quickly and accurately detect anomalies by judging the abnormal sounds emitted by the object part without performing complicated operations.

[0023] In particular, according to the anomaly detection method of the third state sample, in the learning process, since the sound data of the sound emitted by the part during normal operation and the sound data of the sound emitted by the part during abnormal operation are learned, the accuracy of anomaly detection can be further improved.

[0024] In particular, according to the anomaly detection method of the fourth state, during the learning process, since the sound data of the synthesized sound emitted when multiple parts of the industrial machine operate simultaneously are learned, anomaly detection can also be performed when the industrial machine with multiple parts operating simultaneously is in operation.

[0025] Based on the anomaly detection system of states 8 to 14, a machine learning model is generated by learning from image data of parts of the industrial machine and sound data of the sounds emitted by those parts during operation. The machine learning model is then input into the sound data collected by the sound collector and the image data collected by the camera when the operator is looking at the viewed position. This determines whether the object part viewed by the operator is abnormal. Therefore, the operator can quickly and accurately detect anomalies by judging the abnormal sounds emitted by the object part without performing complicated operations.

[0026] In particular, according to the anomaly detection system of the 10th state, the learner can further improve the accuracy of anomaly detection by learning the sound data of the sound emitted by the part during normal operation and the sound data of the sound emitted by the part during abnormal operation.

[0027] In particular, according to the anomaly detection system of the 11th state, the learner can also perform anomaly detection when the industrial machine with multiple parts operating simultaneously is in operation because it learns the sound data of the synthesized sound emitted when multiple parts of the industrial machine operate simultaneously. Simple Explanation of the Diagram

[0028] Figure 1 is a schematic diagram showing the general structure of the anomaly detection system of the present invention. Figure 2 is a top view illustrating the internal layout of the substrate processing apparatus. Figure 3 is a top view showing the general structure of the processing unit. Figure 4 is a side view showing the general structure of the processing unit. Figure 5 is a three-dimensional view showing the appearance of the smart glasses. Figure 6 is a block diagram showing the functional configuration of the control unit of smart glasses, server, work support terminal and board processing device. Figure 7 is a flowchart showing the steps of constructing the model. Figure 8 shows an example of an image taken for the purpose of constructing a model. Figure 9 is a diagram used to conceptually illustrate model construction through machine learning. Figure 10 is a flowchart showing the steps of anomaly detection using a machine learning model. Figure 11 shows a specific example of visual location recognition achieved through eye tracking. Figure 12 is a diagram used to conceptually illustrate anomaly detection using a machine learning model. Figure 13 is a diagram used to conceptually illustrate the anomaly determination of the second implementation. Implementation

[0029] The following is a detailed description of embodiments of the present invention, with reference to the accompanying drawings. The expressions indicating relative or absolute positional relationships (e.g., "in a direction," "along a direction," "parallel," "orthogonal," "center," "concentric," "coaxial," etc.) unless otherwise specified, not only strictly indicate the positional relationship but also indicate a state of relative displacement of angles or distances within the tolerance range or to achieve the same level of functionality. Furthermore, expressions indicating equal states (e.g., "same," "equal," "homogeneous," etc.) unless otherwise specified, not only indicate a state of quantitatively strict equality but also indicate a state where there is a difference in tolerance or to achieve the same level of functionality. Furthermore, expressions indicating shapes (e.g., "circle," "rectangular shape," "cylindrical shape," etc.) unless otherwise specified, not only strictly represent the shape geometrically but also indicate the range of shapes that achieve the same level of effect; for example, they may have concave or convex shapes or chamfers. Furthermore, each expression of the constituent elements, such as "including," "providing," "possessing," "comprises," or "having," is not an exclusive expression excluding the existence of other constituent elements. Also, the expression "at least one of A, B, and C" includes "only A," "only B," "only C," "any two of A, B, and C," and "all of A, B, and C."

[0030] <First Implementation Form> Figure 1 is a schematic diagram showing the general configuration of the anomaly detection system of the present invention. The anomaly detection system of the present invention includes: a plurality of board processing devices 40, smart glasses 10, a server 70, and a work support terminal 80. The controllers of the smart glasses 10 and the board processing devices 40 are wirelessly connected to an information communication network 5 (e.g., the Internet). Furthermore, the work support terminal 80 and the server 70 are wiredly connected to the information communication network 5. Machines connected to the information communication network 5 can send and receive information, for example, information can be exchanged between the smart glasses 10 and the work support terminal 80. Moreover, the connection between each machine and the information communication network 5 is not limited to the above example; it can be configured in a suitable manner (e.g., the work support terminal 80 can be wirelessly connected to the information communication network 5).

[0031] Multiple substrate processing devices 40 are arranged, for example, in a cleanroom. A cleanroom is, for example, a room located within a semiconductor device manufacturing plant, where a certain level of air cleanliness and temperature and humidity are managed. Operators perform operations on the substrate processing devices 40 within the cleanroom.

[0032] Figure 2 is a top view illustrating the internal layout of the substrate processing apparatus 40. The substrate processing apparatus 40 is a single-piece substrate cleaning device that processes silicon substrates (substrate W) in the form of wafers, such as semiconductor wafers, one by one. The substrate processing apparatus 40 includes: a transfer unit 43, a plurality of processing units 50, a main transfer robot 48, and a control unit 45.

[0033] The transfer unit 43 has a plurality of wafer loading / unloading machines LP (three in this embodiment) and a transfer robot 41. Each wafer loading / unloading machine LP holds a carrier C that houses a plurality of substrates W processed by the processing unit 50. The carrier C can be a FOUP (front opening unified pod) that houses the substrates W in a closed space, or an SMIF (Standard Mechanical Interface) transfer box, or an OC (open cassette) that exposes the housed substrates W to the external atmosphere.

[0034] The transfer robot 41 transports the substrate W between the carrier C and the main transfer robot 48. The transfer robot 41 is, for example, a multi-joint robot, capable of receiving and transferring the substrate W from any carrier C placed on one of a plurality of wafer loading and unloading machines LP.

[0035] Furthermore, the main transfer robot 48 transports the substrate W between the transfer robot 41 and the processing unit 50. The main transfer robot 48 is configured to perform lifting, rotating, and forward / backward movements of its transfer arm. The main transfer robot 48 receives the unprocessed substrate W taken from the carrier C by the transfer robot 41 and moves it into the processing unit 50. Additionally, the transfer robot 41 receives the processed substrate W moved from the processing unit 50 by the main transfer robot 48 and stores it in the carrier C.

[0036] In the substrate processing apparatus 40, for example, three processing units 50 are stacked to form one stack (tower). Furthermore, for example, four stacks are arranged around the main transport robot 48. That is, one substrate processing apparatus 40 includes, for example, 12 (=3×4) processing units 50. Figure 2 schematically shows one segment of three overlapping processing units 50. Furthermore, the number of processing units 50 in the substrate processing apparatus 40 is not limited to 12 and can be appropriately varied.

[0037] The main transfer robot 48 is positioned at the center of the four laminates with processing units 50. The main transfer robot 48 moves the substrate W, which is to be processed and received from the transfer robot 41, into the inside of the cup 55 of any processing unit 50. Furthermore, the main transfer robot 48 removes the processed substrate W from each processing unit 50 and delivers it to the transfer robot 41.

[0038] Furthermore, the substrate processing apparatus 40 includes a control unit 45. The control unit 45 is a general-purpose computer that controls the operations of the aforementioned transfer robot 41, main transport robot 48, and mechanisms installed in each processing unit 50 within the apparatus. The control unit 45 has a touch panel serving as an input / output interface installed on the wall of the apparatus and a communication unit for communicating with the outside of the apparatus. Additionally, in FIG2, for ease of illustration, the control unit 45 is shown within the transfer unit 43, but this is not a limitation; the control unit 45 may be located at an appropriate position within the substrate processing apparatus 40.

[0039] The following description will focus on one of the twelve processing units 50 mounted on the substrate processing apparatus 40. However, the other processing units 50 have the same configuration except for the different arrangement of the nozzles.

[0040] The processing unit 50 sprays a processing solution onto one substrate W to perform a cleaning process. The processing solution is a term that includes various chemical solutions and pure water. As a chemical solution, it includes, for example, liquids used for etching or liquids used for removing particles. Specifically, it uses SC-1 solution (a mixed solution of ammonium hydroxide, hydrogen peroxide, and pure water), SC-2 solution (a mixed solution of hydrochloric acid, hydrogen peroxide, and pure water), or hydrofluoric acid, etc.

[0041] Figure 3 is a top view showing the schematic configuration of the processing unit 50. Figure 4 is a side view showing the schematic configuration of the processing unit 50. The processing unit 50 includes: a processing chamber 51, a rotating holding part 56, a processing liquid nozzle (first nozzle) 60, a spray nozzle (second nozzle) 65, and a cup 55. The processing chamber 51 is a hollow housing. The rotating holding part 56, the processing liquid nozzle 60, the spray nozzle 65, and the cup 55 are disposed inside the processing chamber 51.

[0042] A transfer inlet 52 is provided on the side wall of the processing chamber 51. The transfer inlet 52 is opened and closed by a stop door 53. With the stop door 53 opening the transfer inlet 52, the main transport robot 48 transfers the substrate W into and out of the processing chamber 51 through the transfer inlet 52. During the processing of the substrate W, the stop door 53 closes the transfer inlet 52. When the transfer inlet 52 is closed by the stop door 53, the processing chamber 51 becomes a semi-enclosed space.

[0043] An FFU (Fan Filter Unit) 54 is installed at the top of the processing chamber 51. The FFU 54 supplies clean air into the processing chamber 51 from the ceiling. This creates a downward flow of clean air within the processing chamber 51. The air supplied into the processing chamber 51 is discharged through an exhaust pipe 59 located at the bottom of the processing chamber 51.

[0044] The rotating holding part 56 includes a spin clamp 57 and a spin motor 58. The spin clamp 57 is a substrate holding part that holds the substrate W in a horizontal position (the normal of the main surface of the substrate W is along the vertical direction). The spin clamp 57 is, for example, a vacuum suction clamp. The spin clamp 57 suctions and holds the central part of the lower surface of the substrate W. In addition, the spin clamp 57 can be other types of clamps, such as a mechanical clamp that holds the edge of the substrate W.

[0045] The spin clamp 57 has a circular plate shape with a diameter smaller than that of the substrate W. When the lower surface of the substrate W is held in the spin clamp 57, the periphery of the substrate W is exposed to the outer side of the outer periphery of the spin clamp 57.

[0046] The spin clamp 57 is connected to the spin motor 58 via a motor shaft. That is, the upper end of the motor shaft of the spin motor 58 is connected to the center of the lower surface of the spin clamp 57. If the spin motor 58 rotates its motor shaft while the spin clamp 57 is holding the substrate W, the substrate W and the spin clamp 57 will rotate in the horizontal plane about a rotation axis along the vertical direction.

[0047] A cup 55 is arranged around the spin clamp 57. The cup 55 can be raised and lowered by a cup lifting mechanism 39, conceptually shown in FIG. 4. The cup 55 has a roughly cylindrical shape, and the upper part of the cup 55 is inclined upwards and closer to the spin clamp 57. However, the inner diameter of the upper part of the cup 55 is larger than the diameter of the substrate W. During the processing of the substrate W, the upper part of the cup 55 is located at a higher position than the substrate W held in the spin clamp 57. Therefore, during processing, the liquid scattered by centrifugal force from the substrate W rotating by the rotating holding part 56 is collected and recovered by the cup 55. The liquid recovered by the cup 55 is discharged from the drain pipe provided at the bottom of the cup 55 (not shown). In addition, the cup 55 can be a multi-segment structure with multiple recovery ports provided for different purposes.

[0048] The treatment fluid nozzle 60 includes a nozzle front end 61, a swing arm 62, and a nozzle drive unit 63. The treatment fluid nozzle 60 is, for example, a straight nozzle that sprays treatment fluid in a continuous flow. The nozzle front end 61 is mounted on the front end of the swing arm 62, which extends in a generally horizontal direction. The nozzle front end 61 receives treatment fluid from a treatment fluid supply source (not shown) and forms an outlet (not shown) from which the treatment fluid is sprayed. The swing arm 62 is moved up and down by the nozzle drive unit 63 and swings in the horizontal plane about a swing axis A1 along the vertical direction.

[0049] The nozzle drive unit 63 raises and lowers the swing arm 62 and swings it, allowing the nozzle tip 61 to move between a processing position held above the substrate W on the rotating holding unit 56 and a standby position on the outer side of the cup 55. When the nozzle tip 61 is in the processing position, the processing liquid nozzle 60 sprays a cleaning solution onto the substrate W held on the rotating holding unit 56 to perform, for example, cleaning the substrate W. Alternatively, the processing liquid nozzle 60 sprays pure water onto the substrate W to perform a pure water rinsing process.

[0050] On the other hand, the spray nozzle 65 includes a nozzle front end 66, a swing arm 67, and a nozzle drive unit 68. The spray nozzle 65 generates droplets by mixing a processing liquid with pressurized gas, for example, and sprays the mixture of droplets and gas onto the substrate W as a dual-fluid nozzle. The nozzle front end 66 is mounted on the front end of the swing arm 67, which extends in a generally horizontal direction. At the nozzle front end 66, processing liquid and pressurized gas are supplied from a processing liquid supply source and a gas supply source (not shown in the figure), and they are mixed inside or outside the nozzle front end 66 to form a mixed fluid. The swing arm 67 moves up and down by the nozzle drive unit 68 and swings in the horizontal plane about a swing axis A2 along the vertical direction.

[0051] The nozzle drive unit 68 causes the swing arm 67 to rise, fall, and swing, allowing the nozzle tip 66 to move between a processing position held above the substrate W in the rotating holding unit 56 and a standby position on the outer side of the cup 55. When the nozzle tip 66 is in the processing position, the spray nozzle 65 sprays a mixed fluid toward the substrate W held in the rotating holding unit 56 to perform, for example, cleaning of the substrate W.

[0052] As shown in Figure 3, there is a risk of interference between the rotation of the processing liquid nozzle 60 and the rotation of the spray nozzle 65. That is, if the spray nozzle 65 also moves upward toward the substrate W when the processing liquid nozzle 60 is in the processing position, there is a risk of collision between the two. Therefore, an interlocking device can be provided so that the other cannot operate when either the processing liquid nozzle 60 or the spray nozzle 65 is in the processing position.

[0053] Operators performing tasks such as operating the substrate processing device 40 wear smart glasses 10. Smart glasses 10 is a type of head-mounted display (HMD) wearable terminal. Smart glasses 10 is also a device used to realize AR (Augmented Reality) or MR (Mixed Reality). For example, Microsoft's "HoloLens" (registered trademark) can be used as smart glasses 10.

[0054] Figure 5 is a perspective view showing the appearance of the smart glasses 10. The smart glasses 10 includes goggles 11 and a headband 12. The user wears the smart glasses 10 by putting the headband 12 on their head. The user can adjust the length of the headband 12 to fit their head size. Furthermore, the headband 12 is equipped with a power button, a brightness button, and a volume button.

[0055] The goggles 11 include various sensors and a display. The display is a holographic lens. That is, the display can display a stereoscopic image in the operator's field of vision through a hologram, and allows light from real objects to pass through in the same way as ordinary eyeglass lenses. Therefore, the operator wearing the smart glasses 10 can also see real objects through the display and see the displayed stereoscopic image.

[0056] The sensors in the goggles 11 mainly include, for example, multiple visible light cameras that capture images of the front of the goggles 11, an infrared camera that tracks the worker's line of sight, a depth sensor that measures the distance to an object, and an inertial measurement unit (IMU). The infrared camera tracks the line of sight by measuring the movement of the wearer's eyes. The depth sensor measures the distance to an object, for example, using a Time-of-Flight (ToF) method. The IMU consists of an accelerometer, a gyroscope, a magnetometer, etc.

[0057] Furthermore, the smart glasses 10 have a built-in computer equipped with a CPU, memory, and a storage unit. The smart glasses 10 also have a wireless communication mechanism, which the computer uses to connect to the information communication network 5. Additionally, the smart glasses 10 also include a microphone, speaker, and battery.

[0058] Figure 6 is a block diagram showing the functional configuration of the smart glasses 10, server 70, work support terminal 80, and control unit 45 of the board processing device 40. The smart glasses 10 includes a camera unit 21, a communication unit 22, a display unit 23, a sound collection unit 24, a memory unit 29, and a gaze tracking unit 25. The camera unit 21 includes a visible light camera mounted on the aforementioned goggles 11. The camera unit 21 includes, for example, four visible light cameras positioned to capture images of the front and obliquely front, enabling it to capture the field of vision of the worker wearing the smart glasses 10.

[0059] The communication unit 22 includes the wireless communication mechanism of the aforementioned smart glasses 10. The communication unit 22 transmits and receives data with the operation support terminal 80 and the server 70 via the information communication network 5. Furthermore, the communication unit 22 can also directly transmit and receive data with the control unit 45 of the board processing device 40 at close range. That is, the communication unit 22 can send data and commands directly or via the information communication network 5 to the control unit 45 of the board processing device 40.

[0060] Display unit 23 includes a display for the aforementioned goggles 11. Display unit 23 has a holographic processing device that displays a stereoscopic image at a predetermined spatial position using holographic technology. Furthermore, the stereoscopic image displayed by display unit 23 is not limited to a three-dimensional shape and can be two-dimensional, such as a document.

[0061] The sound collection unit 24 includes the microphone of the smart glasses 10 described above. The sound collection unit 24 converts the sound arriving at the smart glasses 10 into electrical signals. Therefore, when an operator wearing the smart glasses 10 approaches the board processing device 40, the sound collection unit 24 can collect the sound emitted from the driving parts (rotation holding part 56, processing liquid nozzle 60, spray nozzle 65, cup 55, etc.) of the processing unit 50 and convert it into electrical signals. The signal output from the sound collection unit 24 can be stored in the memory unit of the smart glasses 10 (i.e., the collected sound can be recorded). The memory unit 29 is the memory of the computer system of the smart glasses 10.

[0062] The eye-tracking unit 25 includes two infrared cameras that measure the movement of the operator's eyes. The eye-tracking unit 25 has an eye-tracking function that tracks the operator's gaze using the two infrared cameras. Operators wearing smart glasses 10 can also perform operations based on their gaze using the eye-tracking function.

[0063] Furthermore, the smart glasses 10 includes an anomaly detection unit 31. This anomaly detection unit 31 is a functional processing unit implemented by the CPU of the smart glasses 10 executing a prescribed processing program. The processing content of the anomaly detection unit 31 will be further described later.

[0064] The control unit 45 of the substrate processing apparatus 40 controls the operation of mechanisms installed in the processing unit 50, such as the spin motor 58, the cup lifting mechanism 39, and the nozzle drive units 63 and 68. The control unit 45 of the substrate processing apparatus 40 can communicate with the communication unit 22 of the smart glasses 10, and can also control the operation of various mechanisms installed in the processing unit 50 according to the operation instructions sent from the smart glasses 10.

[0065] The operation support terminal 80 and server 70 are, for example, installed in the factory of the manufacturer that produces the substrate processing equipment 40 and is responsible for its maintenance and inspection. The operation support terminal 80 and server 70 can communicate with the smart glasses 10 via the information communication network 5. Furthermore, the operation support terminal 80 and server 70 can communicate with each other via the information communication network 5.

[0066] The operating support terminal 80 and server 70 are general computer systems. That is, the operating support terminal 80 and server 70 have: a CPU for performing various calculations and processing, a ROM for storing basic programs and reading them out, RAM for storing various information and reading them out, a memory unit (such as a disk or SSD) for pre-storing control software and data, and a communication unit for communicating with the information communication network 5.

[0067] The operation support terminal 80 is a computer used, for example, by an operation support operator on the manufacturer's side to support the work of operators in the cleanroom. The operation support operator can send various information from the operation support terminal 80 to the smart glasses 10 worn by the operator in the cleanroom.

[0068] In the anomaly detection system of the present invention, server 70 is a computer that performs prescribed processing according to requests from smart glasses 10 and work support terminal 80. Server 70 has a large-capacity memory unit 74. Large-sized data generated by smart glasses 10 and work support terminal 80 can be stored in memory unit 74. Furthermore, server 70 and work support terminal 80 are not essential components.

[0069] Next, an anomaly detection method using the anomaly detection system with the above-described configuration will be described. The anomaly detection method of the present invention comprises two stages: a model construction stage performed by machine learning as a preliminary preparation stage, and an anomaly detection stage using the constructed machine learning model. First, the model construction will be described.

[0070] Figure 7 is a flowchart showing the steps of the model building process. The model building process can be carried out at the appropriate time, for example, before the substrate processing device 40 is shipped, after the substrate processing device 40 is installed in a clean room, or during the development of the prototype of the substrate processing device 40.

[0071] First, image data is collected from each part of the substrate processing apparatus 40 (step S11). The imaging in step S11 can be performed using the camera unit 21 of the smart glasses 10, or using another camera. Figure 8 shows an example of an image captured for model construction. In the example shown in Figure 8, the image is captured inside the chamber of the processing unit 50.

[0072] In the image IM within the image processing unit 50 shown in Figure 8, various parts such as the processing fluid nozzle 60, the spin clamp 57, and the cup 55 are captured. For the portions of the image IM designated using rectangles or polygons, images IM1, IM2, and IM3 are labeled (step S12). Labeling is a process of attaching tags (labels) to image data to indicate the content of the object image. For example, for image IM1, which designates the processing fluid nozzle 60 as a region in the image IM, the label "nozzle arm rotation axis" is attached. Similarly, for image IM2, which designates the spin clamp 57 as a region in the image IM, the label "spin clamp rotation axis" is attached. Furthermore, for image IM3, which designates the cup 55 as a region in the image IM, the label "cup lifting axis" is attached. Alternatively, a portion of the image can be cropped from the image IM and labeled, instead of using rectangles or other region designations. Or, images that only capture specific parts (e.g., only the processing fluid nozzle 60) can be labeled. Alternatively, multiple images of a single body part taken from various angles can be prepared, and common markings can be applied to each image.

[0073] Next, the sounds emitted by each part (i.e., the marked object parts) of the projected images IM1, IM2, and IM3 during operation are recorded and the sound data is collected (step S13). This process can be performed using the sound collection unit 24 of the smart glasses 10, or using another microphone and recorder. However, since the sound data also depends on the characteristics of the microphone, it is preferable to use the sound collection unit 24 of the smart glasses 10 used for anomaly detection as described later.

[0074] In step S13, it is preferable to collect both the sound data emitted by each part during normal operation (normal sound data) and the sound data emitted by the part during abnormal operation (i.e., malfunction) (abnormal sound data). For example, the sound data of the spin clamp 57 rotating normally by the spin motor 58 is collected as normal sound data. On the other hand, the sound data of the spin clamp 57 rotating abnormally is collected as abnormal sound data. Normal sound data can be collected at any time sequence, but abnormal sound data is preferably collected and continuously collected whenever a malfunction occurs in the part. Furthermore, the sound data can be data in a time region (horizontal axis is time) or data in a frequency region (horizontal axis is frequency) obtained by Fourier transform.

[0075] Next, a machine learning model (AI model) is constructed (step S15) based on machine learning (step S14) using image data of each labeled part and sound data of the sound emitted by that part during operation. Figure 9 is a diagram used to conceptually illustrate the model construction performed by machine learning. The image data of a certain part labeled and the sound data of the sound emitted by that part during operation (including both normal sound data and abnormal sound data) are set as the dataset, and machine learning is performed using algorithms such as neural networks, decision trees, and support vector machines (SVM). The machine learning in step S14 is performed by a learner 85 (Figure 6) implemented, for example, in the work support terminal 80. The learner 85 constructs a machine learning model 99 by performing machine learning using image data and sound data of multiple parts such as the processing fluid nozzle 60, the spin clamp 57, and the cup 55 as sampling data.

[0076] When performing machine learning, the more sampled data learned, the more thoroughly trained and accurate the machine learning model 99 can be constructed. Therefore, as image data, it is preferable to use multiple images of various parts captured from various angles as sample data. Similarly, as sound data, it is preferable to collect sound data emitted by various parts under various conditions and use it as sample data. The machine learning model 99 constructed by the learner 85 is temporarily stored, for example, in the memory 74 of the server 70.

[0077] Next, the anomaly detection of the substrate processing apparatus 40 using the machine learning model 99 as described above will be explained. Figure 10 is a flowchart showing the steps of anomaly detection using the machine learning model 99. This anomaly detection can be performed, for example, during an inspection process before the substrate processing apparatus 40 is shipped.

[0078] When anomaly detection is performed, the operator wearing the smart glasses 10 observes the processing chamber 51 of any processing unit 50 of the substrate processing device 40 designated as the target of inspection. The aforementioned machine learning model 99 is downloaded and stored in the memory unit 29 of the smart glasses 10. Then, the eye-tracking function of the smart glasses 10 is used to determine the position recognized by the operator (step S21). Specifically, the gaze tracking unit 25 of the smart glasses 10 tracks the operator's gaze and determines the position in front of that gaze as the recognition position.

[0079] Figure 11 shows a specific example of visual recognition position achieved through eye tracking. In this example, a portion of the cup 55 of the worker's observation processing unit 50 wearing smart glasses 10 is observed. Thus, the eye tracking unit 25 tracks the worker's gaze, and the specific position indicated by the black dot in Figure 11 is taken as the visual recognition position.

[0080] Next, the camera unit 21 of the smart glasses 10 captures the area near the visual recognition position identified by eye tracking (step S22). Specifically, for example, the camera unit 21 captures a rectangular area PT centered on the visual recognition position and composed of a predetermined number of pixels in all directions.

[0081] Secondly, when the operator is viewing the viewed location, the sound-collecting unit 24 of the smart glasses 10 collects sound (step S23). At this time, the sound-collecting unit 24 collects sound when the object being viewed by the operator is in motion. Specifically, for example, the operator inputs a specified command to the control unit 45 of the substrate processing device 40 to make the object being viewed by the operator move. Then, when the object emits an action sound, the sound-collecting unit 24 collects sound. In the above example, when the cup lifting mechanism 39 responds to the command input from the operator and lifts the cup 55, the sound-collecting unit 24 collects sound. Furthermore, in the first embodiment, since it is a pre-shipment inspection cutoff, parts other than the object being viewed by the operator do not move, therefore, parts other than the object do not emit sound.

[0082] Next, the anomaly detection unit 31 of the smart glasses 10 inputs the image data obtained in step S22 and the sound data collected in step S23 into the machine learning model 99 (step S24). Figure 12 is a diagram used to conceptually illustrate the anomaly detection using the machine learning model 99. The anomaly detection unit 31 inputs the image data of the camera area PT captured by the camera unit 21 in step S22 and the sound data collected by the sound collection unit 24 in step S23 into the machine learning model 99 stored in the memory unit 29.

[0083] In the machine learning model 99, image data of each part within the processing unit 50 and sound data (including both normal and abnormal sound data) emitted by that part during operation are learned and correlated. The machine learning model 99 analyzes the input image and sound data, and determines whether the sound data is close to normal or abnormal sound data based on the correlation with the image data. Furthermore, when the input sound data is close to normal sound data, the machine learning model 99 outputs a normal judgment result, and when it is close to abnormal sound data, it outputs an abnormal judgment result. Also, when the input sound data is close to abnormal sound data, the machine learning model 99 can also output the type of abnormality. In this way, the operator determines whether the object part is abnormal (step S25).

[0084] In the first embodiment, a machine learning model 99 is generated by machine learning based on image data of various parts installed in the processing unit 50 and sound data of the sounds emitted by those parts when they are in motion. When anomaly detection is performed, the location is specifically identified by the operator wearing smart glasses 10 equipped with the machine learning model 99 using eye tracking. Furthermore, the camera unit 21 captures an area including the identified location, and the sound collection unit 24 collects sound while the operator is looking at the identified location. By inputting the collected image data and sound data into the machine learning model 99, it is determined whether the object being viewed by the operator is abnormal.

[0085] Thus, in the first embodiment, the machine learning model 99 can determine whether the object part is abnormal simply by inputting the image data captured by the camera unit 21 and the sound data collected by the sound collection unit 24 into the object part that the operator wearing smart glasses 10 recognizes and intends to determine the abnormality. Therefore, the operator can quickly and accurately detect abnormalities by judging the unusual sounds emitted by the object part without having to perform complicated operations.

[0086] <Second Implementation Form> Next, a second embodiment of the present invention will be described. The overall structure of the anomaly detection system in the second embodiment, as well as the structure of the smart glasses 10 and the substrate processing device 40, are the same as in the first embodiment. In the first embodiment, anomalies are detected by a machine learning model 99 based on image data and sound data. In contrast, in the second embodiment, the machine learning model is hierarchically separated.

[0087] Figure 13 is a diagram used to conceptually illustrate the anomaly determination of the second embodiment. In the second embodiment, during model construction, the following are generated hierarchically: a part determination model (first model), which outputs the parts contained in the image data after inputting image data; and an abnormal noise determination model (second model), which determines whether the sound represented by the sound data is an abnormal noise after inputting sound data. For example, a part determination model 191 is constructed by learning from image data of each part set in the processing unit 50 and the labels attached to the image data. Furthermore, an abnormal noise determination model 192 is constructed by supervised learning from normal sound data of the sound emitted by each part during normal operation and abnormal sound data of the sound emitted during abnormal operation. Regarding the abnormal noise determination model 192, it is preferable to pre-construct it for each part. For example, it is preferable to pre-construct one abnormal noise determination model 192 for each of the processing fluid nozzle 60, the spin clamp 57, and the cup 55.

[0088] Secondly, when performing anomaly detection, similar to the first embodiment, the operator wears smart glasses 10 equipped with a location determination model 191 and a plurality of abnormal sound determination models 192. The location the operator is looking at is identified through eye-tracking. Furthermore, the camera unit 21 captures the area including the identified location, and the sound collection unit 24 collects sound while the operator is looking at the identified location.

[0089] In the second embodiment, firstly, image data captured by camera unit 21 is input into part determination model 191, where the part contained in the specific image data is the object part perceived by the operator. Then, next, sound data collected by sound collection unit 24 is input into abnormal sound determination model 192 corresponding to the specific object part to determine whether the object part perceived by the operator is abnormal. That is, when the input sound data is close to normal sound data, the abnormal sound determination model 192 outputs a determination result that the object part is normal; when it is close to abnormal sound data, the abnormal sound determination model 192 outputs a determination result that the object part is abnormal.

[0090] Even in the second embodiment, the system can determine whether a part is abnormal simply by inputting the image data captured by the camera unit 21 into the part determination model 191 when the operator wearing smart glasses 10 recognizes the part intended for anomaly determination, and inputting the sound data collected by the sound collection unit 24 into the abnormal sound determination model 192. Therefore, the operator can quickly and accurately detect abnormalities by determining the abnormal sounds emitted by the part without performing complicated operations.

[0091] <Third Implementation Form> Next, a third embodiment of the present invention will be described. The overall structure of the anomaly detection system in the third embodiment, as well as the structure of the smart glasses 10 and the substrate processing apparatus 40, are the same as those in the first embodiment. In the first embodiment, anomaly detection was performed during the pre-shipment inspection stage, but in the third embodiment, anomaly detection is performed during the operation of the substrate processing apparatus 40.

[0092] During the pre-shipment inspection phase, parts other than the target area visually identified by the operator remain inactive. Conversely, during the operation of the substrate processing apparatus 40, multiple parts in the multiple processing units 50 operate simultaneously. For example, the spin clamp 57 rotates, and at the same time, the processing liquid nozzle 60 also moves. Thus, multiple parts emit sounds simultaneously, and during anomaly detection, the sound collection unit 24 collects the sounds emitted from parts other than the target area.

[0093] In the third embodiment, during model construction, sound data of synthesized sounds emitted when multiple parts that are known to operate simultaneously are collected, and machine learning is performed. It is possible to determine which multiple parts of the substrate processing apparatus 40 will operate simultaneously based on processing conditions. Processing conditions determine the processing sequence and conditions of the substrate processing apparatus 40, and the control unit 45 of the substrate processing apparatus 40 controls each part of the substrate processing apparatus 40 according to the processing conditions.

[0094] A machine learning model 99 is constructed by using machine learning based on image data of multiple parts that operate simultaneously according to the processing conditions, and sound data of synthesized sounds emitted by these multiple parts during simultaneous operation. For example, in a certain step of the processing conditions, when it is determined that the spin jig 57 and the processing fluid nozzle 60 operate simultaneously, a machine learning model 99 is generated by using machine learning based on image data of images projected onto the spin jig 57 and the processing fluid nozzle 60, and sound data of synthesized sounds emitted by them during simultaneous operation.

[0095] When anomaly detection is performed, eye-tracking is used to specifically target the location viewed by the operator wearing smart glasses 10 equipped with a machine learning model 99 that has learned synthesized sounds. Furthermore, the camera unit 21 captures an area including the targeted viewing location, and the sound collection unit 24 collects sound while the operator is viewing the targeted location. By inputting the collected image and sound data into the machine learning model 99, it is determined whether the object being viewed by the operator is abnormal. In the third embodiment, since machine learning is performed using sound data from synthesized sounds emitted when multiple parts move simultaneously, it is possible to determine whether the object is abnormal even in situations such as when multiple parts move simultaneously during the operation of the board processing device 40.

[0096] <Example of Change> The embodiments of the present invention have been described above. However, various modifications can be made to the present invention beyond the above description as long as they do not depart from its intent. For example, in the embodiments described above, both normal sound data and abnormal sound data are collected as sound data used for machine learning, but only normal sound data can be collected and used. In the case where only normal sound data is collected and used, when the machine learning model 99 performs anomaly detection, if the sound data collected by the sound collection unit 24 is close to the normal sound data, it outputs a normal judgment result; if it deviates from the normal sound data by a certain amount, it outputs an abnormal judgment result. However, as in the embodiments described above, if both normal sound data and abnormal sound data are collected and learned, it is also easy to identify the cause of the anomaly.

[0097] Furthermore, in the above embodiments, the eye-tracking function of the smart glasses 10 can be used to identify the location of the operator's gaze. Alternatively, the display unit 23 can display the parts that the operator has seen in the past 5 seconds in a list format in the stereoscopic image, and the operator can select a specific object part from there.

[0098] Furthermore, in the above embodiments, the smart glasses 10 is equipped with a machine learning model 99 and performs anomaly detection, but it is not limited to this. For example, the image and sound data collected by the smart glasses 10 can be sent to the operation support terminal 80, and the operation support terminal 80 uses the machine learning model 99 stored in the memory unit 74 to perform anomaly detection (Figure 6). Alternatively, the server 70 can perform anomaly detection.

[0099] Furthermore, the technology of the third embodiment can be applied to anomaly detection of the transport robot 41 or the main transport robot 48. In the transport robot 41 and the main transport robot 48, since multiple drive units operate simultaneously, anomalies of the transport robot 41 or the main transport robot 48 can be reliably detected by performing machine learning on the sound data of the synthesized sounds emitted by the multiple drive units when they operate simultaneously.

[0100] Furthermore, in the above embodiments, the operator uses smart glasses 10, but is not limited to this; a tablet terminal or smartphone or other mobile terminal can be used instead of smart glasses 10. That is, any mobile terminal with a camera and communication unit is acceptable. However, if a tablet terminal is used, it occupies the operator's hand, so it is preferable to use a wearable terminal such as smart glasses 10.

[0101] Furthermore, the substrate processing apparatus 40 is not limited to a substrate cleaning apparatus; it can be any apparatus that performs a prescribed treatment on the substrate, such as a heat treatment apparatus, an exposure apparatus, a coating and developing apparatus, a measuring apparatus, or an inspection apparatus. When the substrate processing apparatus 40 is a substrate cleaning apparatus, it can be a single-piece cleaning apparatus that cleans one substrate at a time, or it can be a batch cleaning apparatus that cleans multiple substrates in a batch.

[0102] Furthermore, the object of the anomaly detection technology of the present invention is not limited to substrate processing apparatus, but can be any industrial machine with an actuating part capable of performing certain actions. Examples of such industrial machines include printing processing apparatus, film forming apparatus, medical apparatus, and appearance inspection apparatus.

[0103] 5: Information and Communication Network 10: Smart Glasses 11: Goggles 12: Headband 21: Camera Department 22: Communications Department 23: Display Section 24: Collected Sounds Section 25: Eye Tracking Department 29: Memory Department 31: Anomaly Detection Department 39: Cup lifting mechanism 40: Substrate processing apparatus 41: Transport Robot 43: Transporter Section 45: Control Department 48: Main transport robot 50: Processing Unit 51: Processing Chamber 52:Moving out of the moving entrance 53: Blocking the door 54:FFU 55: Cup 56: Rotary retaining part 57: Spin clamp 58: Spin Motor 59: Exhaust pipe 60: Processing fluid nozzle (Nozzle 1) 61: Nozzle front end 62, 67: Swing arm 63, 68: Nozzle drive unit 65: Spray nozzle (second nozzle) 66: Nozzle front end 70: Server 74: Memory Department 80: Operation Support Terminal 85: Learning Device 99: Machine Learning Models 191: Location Determination Model 192: Abnormal Noise Judgment Model A1, A2: Swing axis C: Carrier IM: Image IM1, IM2, IM3: Partial images LP: Wafer Loading and Unloading Machine PT: Camera Area S11~S15, S21~S25: Steps W: substrate

Claims

1. An anomaly detection method for detecting abnormal states of industrial machinery, comprising: a learning step, which generates a machine learning model by learning from image data of parts of the industrial machinery captured by photography and sound data of sounds emitted by those parts during operation; a specifying step, which specifies the viewing position of the industrial machinery by an operator wearing a mobile terminal including a sound collector, a display unit, a camera unit, a communication unit, and the aforementioned look-tracking unit; a collecting step, which, while the operator is viewing the aforementioned viewing position, causes the sound collector to collect sound and the camera unit to capture images of an area including the aforementioned viewing position; and a determining step, which determines whether the object viewed by the operator is abnormal by inputting the sound data and image data collected in the aforementioned collecting step into the aforementioned machine learning model.

2. The anomaly detection method as described in claim 1, wherein in the aforementioned learning process, sound data of the sounds emitted by the aforementioned part during normal operation is learned.

3. The anomaly detection method as described in claim 2, wherein in the aforementioned learning process, the sound data of the sound emitted by the aforementioned part when performing an abnormal action is further learned.

4. The anomaly detection method as described in Request 1, wherein in the aforementioned learning process, the sound data of synthesized sounds emitted by multiple parts of the aforementioned industrial machine when they operate simultaneously are learned.

5. The anomaly detection method of claim 1, wherein the aforementioned machine learning model comprises: a first model, which outputs the parts contained in the image data after inputting image data; and a second model, which determines whether the sound represented by the sound data is an abnormal noise after inputting sound data; and the aforementioned determination process comprises: a part identification process, which identifies the aforementioned object part by inputting the image data collected in the aforementioned collection process into the aforementioned first model; and an abnormal noise determination process, which determines whether the aforementioned object part is abnormal by inputting the sound data collected in the aforementioned collection process into the aforementioned second model.

6. The anomaly detection method as described in claim 1, wherein the aforementioned industrial machine is a substrate processing apparatus that performs prescribed processing on the substrate.

7. The anomaly detection method as described in any of requests 1 to 6, wherein the aforementioned mobile terminal is a smart glasses.

8. An anomaly detection system for detecting abnormal states of industrial machinery, comprising: a mobile terminal including a sound collection unit, a display unit, a camera unit, a communication unit, and a gaze tracking unit; and a learner that generates a machine learning model by learning from image data captured by the machine and sound data emitted by the machine when the machine is in operation; wherein the gaze tracking unit is specifically used by an operator wearing the mobile terminal to view the machine at a designated viewing location; while the operator is viewing the machine at the designated viewing location, the sound collection unit collects sound, and the camera unit captures an image of an area including the designated viewing location; the mobile terminal further includes a determination unit that determines whether the viewed object is abnormal by inputting the sound data collected by the sound collection unit and the image data captured by the camera unit into the machine learning model.

9. The anomaly detection system as described in claim 8, wherein the aforementioned learner learns sound data of the sounds emitted by the aforementioned part during normal operation.

10. The anomaly detection system as described in claim 9, wherein the aforementioned learner further learns sound data of the sounds emitted by the aforementioned part when performing abnormal actions.

11. The anomaly detection system as described in claim 8, wherein the aforementioned learner learns sound data of synthesized sounds emitted when multiple parts of the aforementioned industrial machine operate simultaneously.

12. The anomaly detection system of claim 8, wherein the aforementioned machine learning model comprises: a first model that, after inputting image data, outputs the parts contained in the image data; and a second model that, after inputting sound data, determines whether the sound represented by the sound data is an abnormal noise; and the aforementioned determination unit, after identifying the aforementioned object part by inputting the image data captured by the aforementioned camera unit into the aforementioned first model, determines whether the aforementioned object part is abnormal by inputting the sound data collected by the aforementioned sound collecting unit into the aforementioned second model.

13. The anomaly detection system as described in claim 8, wherein the aforementioned industrial machine is a substrate processing apparatus that performs prescribed processing on the substrate.

14. An anomaly detection system as described in any of requests 8 to 13, wherein the aforementioned mobile terminal is a smart glasses.

Images