Abnormality detection method and abnormality detection system

Abstract

A machine learning model is constructed by machine learning based on image data that is obtained by imaging each portion provided in a substrate processing device and sound data of sounds emitted when the portions operate. As the sound data, both the data of a normal sound that is emitted when each portion operates normally and the data of an abnormal sound that is emitted when each portion is abnormally operated are collected. When abnormality detection is performed, an eye tracking function identifies a position that is being viewed by a worker wearing smart glasses in which a machine learning model is installed. An image of a region including the identified viewed position is captured, and a sound is collected in a state in which the worker is viewing the viewed position. The collected image data and sound data are input into the machine learning model.

Description

Anomaly detection method and anomaly detection system

[0001] The present invention relates to an abnormality detection method and an abnormality detection system for detecting an abnormal state in industrial equipment such as a substrate processing apparatus that performs predetermined processing on substrates. Substrates to be processed by the substrate processing apparatus include, for example, semiconductor substrates, substrates for liquid crystal display devices, substrates for flat panel displays (FPDs), substrates for optical disks, substrates for magnetic disks, and substrates for solar cells.

[0002] Sound detection is one of the methods used to inspect drive components such as motors in industrial equipment such as substrate processing equipment. Conventionally, inspectors would use their hearing to detect abnormal sounds from components, but this method is highly dependent on the skill of the inspector, and inexperienced inspectors may not be able to detect abnormalities.

[0003] For this reason, abnormality determination is performed by collecting sounds from the body parts using a microphone or the like and mechanically comparing the measurement sounds with a reference sound. For example, Patent Document 1 discloses a technology in which sounds generated within the visual field of a worker wearing a head-mounted display equipped with a directional microphone are collected as measurement sounds using the directional microphone, and the measurement sounds are compared with the reference sound to determine whether the measurement sounds are abnormal sounds.

[0004] JP 2010-197361 A

[0005] However, since sound-emitting parts are generally mounted densely in substrate processing equipment and many sound-emitting parts are present within the field of vision of the operator, the operator must select and specify the reference sound to be compared with the collected measurement sound one by one, which is a time-consuming task.

[0006] In addition, there are cases where multiple nearby parts are emitting sound at the same time, and the wavelength of the sound changes depending on the type of abnormality, so there was much room for improvement in order to bring the accuracy of sound-based abnormality detection technology closer to that of a skilled worker.

[0007] The present invention has been made in view of the above-mentioned problems, and has an object to provide an abnormality detection method and an abnormality detection system that can quickly and accurately determine abnormal sounds and detect abnormalities.

[0008] In order to solve the above problem, a first aspect of the present invention is an abnormality detection method for detecting an abnormal state of industrial equipment, comprising: a learning process for generating a machine learning model by learning based on image data of an image of a part of the industrial equipment and sound data of sounds emitted by the part when it is operating; an identification process for identifying a viewing position of the industrial equipment viewed by an operator wearing a mobile terminal equipped with a sound collection unit, a display unit, an imaging unit, a communication unit, and an eye tracking unit by the eye tracking unit; a collection process for collecting sound with the sound collection unit while the operator is viewing the viewing position and for the imaging unit to image an area including the viewing position; and a determination process for determining whether the target part viewed by the operator is abnormal by inputting the sound data and image data collected in the collection process into the machine learning model.

[0009] In a second aspect, in the abnormality detection method according to the first aspect, the learning step performs learning using sound data of sounds emitted when the part is operating normally.

[0010] In addition, in a third aspect, in the abnormality detection method according to the second aspect, the learning step further includes learning using sound data of sounds emitted when the part is performing an abnormal operation.

[0011] In addition, a fourth aspect is an anomaly detection method according to any one of the first to third aspects, wherein the learning step performs learning using sound data of a synthesized sound emitted when multiple parts of the industrial equipment are operating simultaneously.

[0012] Further, in a fifth aspect, in the anomaly detection method according to any one of the first to fourth aspects, the machine learning model includes a first model that, when image data is input, outputs a part included in the image data, and a second model that, when sound data is input, determines whether or not a sound represented by the sound data is an abnormal sound; and the determination step includes a part identification step of identifying the target part by inputting the image data collected in the collection step into the first model, and an abnormal sound determination step of determining whether or not the target part is abnormal by inputting the sound data collected in the collection step into the second model.

[0013] A sixth aspect is the abnormality detection method according to any one of the first to fifth aspects, wherein the industrial equipment is a substrate processing apparatus that performs a predetermined process on a substrate.

[0014] In addition, a seventh aspect is the anomaly detection method according to any one of the first to sixth aspects, wherein the mobile terminal is a pair of smart glasses.

[0015] In addition, an eighth aspect is an abnormality detection system for detecting an abnormal state of industrial equipment, comprising: a portable terminal having a sound collection unit, a display unit, an imaging unit, a communication unit, and an eye-gaze tracking unit; and a learning device that generates a machine learning model by learning based on image data of an image of a part of the industrial equipment and sound data of a sound emitted by the part when it is operating, wherein the eye-gaze tracking unit identifies a viewing position of the industrial equipment that is being viewed by a worker wearing the portable terminal, the sound collection unit collects sound while the worker is viewing the viewing position, and the imaging unit images an area including the viewing position, and the portable terminal further comprises a determination unit that determines whether the target part being viewed by the worker is abnormal by inputting the sound data collected by the sound collection unit and the image data captured by the imaging unit into the machine learning model.

[0016] In a ninth aspect, in the anomaly detection system according to the eighth aspect, the learning device performs learning using sound data of sounds emitted when the part is operating normally.

[0017] In addition, in a tenth aspect, in the anomaly detection system according to the ninth aspect, the learning device further performs learning using sound data of sounds emitted when the part is performing an abnormal operation.

[0018] In addition, an eleventh aspect is an anomaly detection system according to any one of the eighth to tenth aspects, wherein the learning device performs learning using sound data of a synthesized sound emitted when multiple parts of the industrial equipment are operating simultaneously.

[0019] Further, in a twelfth aspect, in an abnormality detection system according to any one of the eighth to eleventh aspects, the machine learning model includes a first model that, when image data is input, outputs a part included in the image data, and a second model that, when sound data is input, determines whether or not the sound represented by the sound data is an abnormal sound; and the determination unit identifies the target part by inputting image data captured by the imaging unit into the first model, and then determines whether or not the target part is abnormal by inputting sound data collected by the sound collection unit into the second model.

[0020] A thirteenth aspect is the abnormality detection system according to any one of the eighth to twelfth aspects, wherein the industrial equipment is a substrate processing apparatus that performs a predetermined process on a substrate.

[0021] In addition, a fourteenth aspect is the anomaly detection system according to any one of the eighth to thirteenth aspects, wherein the mobile terminal is a pair of smart glasses.

[0022] According to the anomaly detection methods of the first to seventh aspects, a machine learning model is generated by learning based on image data of an image of a part of industrial equipment and sound data of the sound emitted by the part when it is operating, and the sound data collected by the sound collection unit and the image data collected by the image capturing unit are input into the machine learning model while the worker is visually observing the viewing position, thereby determining whether the target part being viewed by the worker is abnormal or not.Therefore, it is possible to quickly and accurately determine the abnormal sound emitted by the target part and detect the abnormality without the worker having to perform complicated work.

[0023] In particular, according to the abnormality detection method of the third aspect, the learning process involves learning using sound data of sounds emitted when a part is operating normally and sound data of sounds emitted when a part is operating abnormally, thereby further improving the accuracy of abnormality detection.

[0024] In particular, according to the anomaly detection method of the fourth aspect, in the learning process, learning is performed using sound data of a synthesized sound emitted when multiple parts of the industrial equipment are operating simultaneously, so that anomalies can be detected even when the industrial equipment is in operation and multiple parts are operating simultaneously.

[0025] According to the anomaly detection systems of the eighth to fourteenth aspects, a machine learning model is generated by learning based on image data of an image of a part of industrial equipment and sound data of the sound emitted by the part when it is operating, and the sound data collected by the sound collection unit and the image data collected by the image capturing unit are input into the machine learning model while the worker is visually observing the viewing position, thereby determining whether the target part being viewed by the worker is abnormal or not.Therefore, it is possible to quickly and accurately determine the abnormal sound emitted by the target part and detect the abnormality without the worker having to perform complicated work.

[0026] In particular, according to the anomaly detection system of the tenth aspect, the learning device learns using sound data of sounds emitted when a part is operating normally and sound data of sounds emitted when a part is operating abnormally, thereby further improving the accuracy of anomaly detection.

[0027] In particular, according to the anomaly detection system of the eleventh aspect, the learning device performs learning using sound data of a synthesized sound emitted when multiple parts of an industrial device are operating simultaneously, and therefore, anomalies can be detected even when the industrial device is in operation and multiple parts are operating simultaneously.

[0028] FIG. 1 is a diagram schematically illustrating the configuration of an anomaly detection system according to the present invention. FIG. 2 is a plan view illustrating the internal layout of a substrate processing apparatus. FIG. 3 is a plan view illustrating the schematic configuration of a processing unit. FIG. 4 is a side view illustrating the schematic configuration of a processing unit. FIG. 5 is a perspective view illustrating the appearance of smart glasses. FIG. 6 is a block diagram illustrating the functional configuration of a control unit of the smart glasses, a server, a work support terminal, and a substrate processing apparatus. FIG. 7 is a flowchart illustrating a model construction procedure. FIG. 8 is a diagram illustrating an example of an image captured for model construction. FIG. 9 is a diagram conceptually illustrating model construction using machine learning. FIG. 10 is a flowchart illustrating a procedure for anomaly detection using a machine learning model. FIG. 11 is a diagram illustrating an example of identifying a gaze position using eye tracking. FIG. 12 is a diagram conceptually illustrating anomaly determination using a machine learning model. FIG. 13 is a diagram conceptually illustrating anomaly determination in a second embodiment.

[0029] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Hereinafter, expressions indicating relative or absolute positional relationships (e.g., "in one direction," "along one direction," "parallel," "orthogonal," "center," "concentric," "coaxial," etc.) not only strictly represent the positional relationship but also represent a state of relative angular or distance displacement within a tolerance or a range that provides equivalent functionality, unless otherwise specified. Furthermore, expressions indicating an equal state (e.g., "identical," "equal," "homogeneous," etc.) not only represent a state of strict quantitative equality but also represent a state of difference that provides a tolerance or equivalent functionality, unless otherwise specified. Furthermore, expressions indicating a shape (e.g., "circular," "square," "cylindrical," etc.) not only represent a geometrically strict shape but also represent a shape within a range that provides equivalent functionality, such as irregularities or chamfers, unless otherwise specified. Furthermore, expressions such as "comprise," "comprise," "include," "have," etc., regarding components, are not exclusive expressions that exclude the presence of other components. Furthermore, the expression "at least one of A, B, and C" includes "A only," "B only," "C only," "any two of A, B, and C," and "all of A, B, and C."

[0030] First Embodiment Fig. 1 is a diagram illustrating a schematic configuration of an anomaly detection system according to the present invention. The anomaly detection system according to the present invention includes a plurality of substrate processing apparatuses 40, smart glasses 10, a server 70, and a work support terminal 80. Controllers of the smart glasses 10 and the substrate processing apparatuses 40 are connected to an information and communication network 5 (e.g., the Internet) via wireless communication. The work support terminal 80 and the server 70 are connected to the information and communication network 5 via a wired connection. Information can be transmitted and received between devices connected to the information and communication network 5, for example, information can be exchanged between the smart glasses 10 and the work support terminal 80. Note that whether each device is connected to the information and communication network 5 wirelessly or via a wired connection is not limited to the above example and can be any suitable connection (e.g., the work support terminal 80 may be connected to the information and communication network 5 wirelessly).

[0031] The plurality of substrate processing apparatuses 40 are arranged side by side in, for example, a clean room. The clean room is provided in, for example, a semiconductor device manufacturing factory, and is a room where a certain level of air cleanliness is ensured and temperature and humidity are controlled. Workers perform work on the substrate processing apparatuses 40 in the clean room.

[0032] 2 is a plan view illustrating the internal layout of the substrate processing apparatus 40. The substrate processing apparatus 40 is a single-wafer type substrate cleaning apparatus that processes substrates W, which are disk-shaped silicon substrates such as semiconductor wafers, one by one. The substrate processing apparatus 40 includes an indexer unit 43, a plurality of processing units 50, a main transport robot 48, and a control unit 45.

[0033] The indexer section 43 has a plurality of load ports LP (three in this embodiment) and an indexer robot 41. Each load port LP is loaded with a carrier C that accommodates a plurality of substrates W to be processed in a processing unit 50. The carrier C may be in the form of a front opening unified pod (FOUP) that accommodates substrates W in an enclosed space, a standard mechanical interface (SMIF) pod, or an open cassette (OC) that exposes the accommodated substrates W to the outside air.

[0034] The indexer robot 41 transports substrates W between the carriers C and the main transport robot 48. The indexer robot 41 is, for example, an articulated robot, and can transfer substrates W to and from any of the carriers C placed on the multiple load ports LP.

[0035] The main transport robot 48 also transports substrates W between the indexer robot 41 and the processing units 50. The main transport robot 48 is configured to be capable of lifting and lowering, rotating, and moving its transport arm back and forth. The main transport robot 48 receives an unprocessed substrate W that the indexer robot 41 has removed from a carrier C and transports it into the processing unit 50. The indexer robot 41 also receives a processed substrate W that the main transport robot 48 has transported from the processing unit 50 and stores it in the carrier C.

[0036] In the substrate processing apparatus 40, for example, three processing units 50 are stacked to form one stack (tower). Then, for example, four stacks are arranged around the main transport robot 48. That is, one substrate processing apparatus 40 includes, for example, 12 (=3×4) processing units 50. In FIG. 2, one stage of three stacked processing units 50 is schematically shown. Note that the number of processing units 50 in the substrate processing apparatus 40 is not limited to 12 and may be changed as appropriate.

[0037] The main transport robot 48 is installed in the center of the stack of four stacked processing units 50. The main transport robot 48 transfers the substrate W to be processed received from the indexer robot 41 into the inside of the cup 55 of one of the processing units 50. The main transport robot 48 also transfers the processed substrate W from each processing unit 50 and hands it over to the indexer robot 41.

[0038] The substrate processing apparatus 40 also includes a control unit 45. The control unit 45 is a general-purpose computer that controls the operations of the indexer robot 41, the main transport robot 48, and mechanisms provided in each processing unit 50, all of which are provided within the apparatus. The control unit 45 has a touch panel serving as an input / output interface provided on a wall surface of the apparatus, and a communication unit that communicates with the outside of the apparatus. Note that, for convenience of illustration, the control unit 45 is shown in the indexer unit 43 in FIG. 2, but is not limited to this, and the control unit 45 may be provided in an appropriate position within the substrate processing apparatus 40.

[0039] Below, one of the twelve processing units 50 mounted on the substrate processing apparatus 40 will be described, but the other processing units 50 have the same configuration except for the relative arrangement of the nozzles.

[0040] The processing unit 50 discharges a processing liquid onto one substrate W to perform a cleaning process. The processing liquid is a term that encompasses various chemical liquids and pure water. Examples of chemical liquids include liquids for etching processes and liquids for removing particles. Specifically, SC-1 liquid (a mixed solution of ammonium hydroxide, hydrogen peroxide, and pure water), SC-2 liquid (a mixed solution of hydrochloric acid, hydrogen peroxide, and pure water), or hydrofluoric acid are used.

[0041] Fig. 3 is a plan view showing a schematic configuration of the processing unit 50. Fig. 4 is a side view showing a schematic configuration of the processing unit 50. The processing unit 50 includes a processing chamber 51, a rotating holder 56, a processing liquid nozzle (first nozzle) 60, a spray nozzle (second nozzle) 65, and a cup 55. The processing chamber 51 is a hollow housing. Inside the processing chamber 51, the rotating holder 56, the processing liquid nozzle 60, the spray nozzle 65, the cup 55, etc. are provided.

[0042] A loading / unloading port 52 is provided on a side wall of the processing chamber 51. The loading / unloading port 52 is opened and closed by a shutter 53. With the shutter 53 opening the loading / unloading port 52, the main transport robot 48 loads and unloads the substrate W into and from the processing chamber 51 through the loading / unloading port 52. The shutter 53 closes the loading / unloading port 52 while the substrate W is being processed. When the loading / unloading port 52 is closed by the shutter 53, the interior of the processing chamber 51 becomes a semi-enclosed space.

[0043] An FFU (fan filter unit) 54 is provided on the ceiling of processing chamber 51. FFU 54 supplies clean air from the ceiling of processing chamber 51 into processing chamber 51. This creates a downflow of clean air from above to below within processing chamber 51. The gas supplied into processing chamber 51 is exhausted through exhaust duct 59 provided at the bottom of processing chamber 51.

[0044] The rotation holder 56 includes a spin chuck 57 and a spin motor 58. The spin chuck 57 is a substrate holder that holds the substrate W in a horizontal position (a position in which the normal to the main surface of the substrate W is aligned vertically). The spin chuck 57 is, for example, a vacuum suction-type chuck. The spin chuck 57 suction-holds the central portion of the lower surface of the substrate W. Note that the spin chuck 57 may be another type of chuck, such as a clamping-type mechanical chuck that grips the edge portion of the substrate W.

[0045] The spin chuck 57 has a disk shape with a diameter smaller than that of the substrate W. When the lower surface of the substrate W is held by suction on the spin chuck 57, the peripheral edge of the substrate W protrudes outward beyond the outer circumferential edge of the spin chuck 57.

[0046] The spin chuck 57 is connected to the spin motor 58 via a motor shaft. That is, the upper end of the motor shaft of the spin motor 58 is connected to the center of the lower surface of the spin chuck 57. When the spin motor 58 rotates the motor shaft while the substrate W is held by suction on the spin chuck 57, the substrate W and the spin chuck 57 rotate in a horizontal plane around a rotation axis extending in the vertical direction.

[0047] A cup 55 is provided to surround the spin chuck 57. The cup 55 can be raised and lowered by a cup lifting mechanism 39 conceptually shown in FIG. 4. The cup 55 has a generally cylindrical shape, and the upper part of the cup 55 is inclined so that it approaches the spin chuck 57 as it goes up. However, the inner diameter of the upper end portion of the cup 55 is larger than the diameter of the substrate W. During processing of the substrate W, the upper end of the cup 55 is higher than the height of the substrate W held by the spin chuck 57. Therefore, liquid scattered by centrifugal force from the substrate W rotated by the spin holder 56 during processing is received and recovered by the cup 55. The liquid recovered by the cup 55 is discharged from a drain pipe (not shown) provided at the bottom of the cup 55. The cup 55 may have a multi-stage structure in which multiple recovery ports are provided for different purposes.

[0048] The processing liquid nozzle 60 includes a nozzle tip 61, a swing arm 62, and a nozzle driver 63. The processing liquid nozzle 60 is, for example, a straight nozzle that ejects the processing liquid in the form of a continuous flow. The nozzle tip 61 is attached to the tip of a swing arm 62 that extends in a substantially horizontal direction. The processing liquid is supplied to the nozzle tip 61 from a processing liquid supply source (not shown), and a discharge port (not shown) is formed in the nozzle tip 61, from which the processing liquid is ejected. The swing arm 62 is moved up and down by the nozzle driver 63, and is also swung in a horizontal plane around a swing axis A1 that extends in the vertical direction.

[0049] The nozzle drive unit 63 raises and lowers and swings the swing arm 62, thereby moving the nozzle tip 61 between a processing position above the substrate W held by the rotary holder 56 and a standby position outside the cup 55. When the nozzle tip 61 is located at the processing position, the processing liquid nozzle 60 discharges a chemical solution onto the substrate W held by the rotary holder 56, thereby proceeding with, for example, cleaning processing of the substrate W. Furthermore, the processing liquid nozzle 60 discharges pure water onto the substrate W, thereby proceeding with pure water rinsing processing of the substrate W.

[0050] On the other hand, the spray nozzle 65 includes a nozzle tip 66, a swing arm 67, and a nozzle driver 68. The spray nozzle 65 is a two-fluid nozzle that mixes, for example, a processing liquid with a pressurized gas to generate droplets and sprays the mixed fluid of the droplets and gas onto the substrate W. The nozzle tip 66 is attached to the tip of a swing arm 67 that extends in a substantially horizontal direction. A processing liquid and a pressurized gas are supplied to the nozzle tip 66 from a processing liquid supply source and a gas supply source (not shown), respectively, and are mixed inside or outside the nozzle tip 61 to form a mixed fluid. The swing arm 67 is moved up and down and swung in a horizontal plane around a swing axis A2 that extends vertically by the nozzle driver 68.

[0051] The nozzle driving unit 68 raises and lowers and swings the swing arm 67, thereby moving the nozzle tip 66 between a processing position above the substrate W held by the rotating holder 56 and a standby position outside the cup 55. When the nozzle tip 66 is located at the processing position, the spray nozzle 65 sprays the mixed fluid onto the substrate W held by the rotating holder 56, thereby proceeding with, for example, cleaning processing of the substrate W.

[0052] 3, there is a risk that the rotational movement of the processing liquid nozzle 60 and the rotational movement of the spray nozzle 65 may interfere with each other. That is, when the processing liquid nozzle 60 is located at the processing position, if the spray nozzle 65 also moves above the substrate W, there is a risk that the two may collide with each other. For this reason, an interlock is provided so that when either the processing liquid nozzle 60 or the spray nozzle 65 is located at the processing position, the other cannot operate.

[0053] An operator who performs operations or other tasks on the substrate processing apparatus 40 wears smart glasses 10. The smart glasses 10 are a type of wearable device that uses a head-mounted display (HMD). The smart glasses 10 are also devices that realize AR (Augmented Reality) or MR (Mixed Reality). For example, Microsoft's "HoloLens" (registered trademark) can be used as the smart glasses 10.

[0054] 5 is a perspective view showing the appearance of the smart glasses 10. The smart glasses 10 include a visor 11 and a headband 12. A worker wears the smart glasses 10 by placing the headband 12 on their head. The worker can adjust the length of the headband 12 to fit the size of their head. The headband 12 also includes a power button, a brightness button, a volume button, and the like.

[0055] The visor 11 includes various sensors and a display. The display is a see-through holographic lens. That is, the display can display a three-dimensional image in the worker's field of view using a hologram, and transmits light from real objects in the same way as regular eyeglass lenses. Therefore, a worker wearing the smart glasses 10 can view the displayed three-dimensional image while viewing real objects through the display.

[0056] The sensors of the visor 11 include, for example, multiple visible light cameras that mainly capture images in front of the visor 11, an infrared camera that tracks the worker's line of sight, a depth sensor that measures the distance to an object, and an inertial measurement sensor. The infrared camera tracks the line of sight by measuring the movement of the eyeballs of the wearer of the smart glasses 10. The depth sensor measures the distance to an object using, for example, a Time of Flight (ToF) method. The inertial measurement sensor is composed of an accelerometer, a gyroscope, a magnetometer, etc.

[0057] The smart glasses 10 also have a built-in computer including a CPU, memory, a storage unit, etc. The smart glasses 10 are also provided with a wireless communication mechanism, and the computer in the smart glasses 10 connects to the information communication network 5 using the wireless communication mechanism. The smart glasses 10 are also provided with a microphone, a speaker, a battery, etc.

[0058] 6 is a block diagram showing the functional configuration of the smart glasses 10, the server 70, the work support terminal 80, and the control unit 45 of the substrate processing apparatus 40. The smart glasses 10 include an imaging unit 21, a communication unit 22, a display unit 23, a sound collection unit 24, a memory unit 29, and an eye tracking unit 25. The imaging unit 21 includes a visible light camera provided on the visor 11 described above. The imaging unit 21 includes, for example, four visible light cameras that capture images of the front and diagonally forward, and can capture images of the field of view of the worker wearing the smart glasses 10.

[0059] The communication unit 22 includes the wireless communication mechanism of the smart glasses 10 described above. The communication unit 22 transmits and receives data to and from the work support terminal 80 and the server 70 via the information and communication network 5. The communication unit 22 can also transmit and receive data directly to and from the control unit 45 of the substrate processing apparatus 40 if the distance is short. That is, the communication unit 22 can transmit data and commands to the control unit 45 of the substrate processing apparatus 40 directly or via the information and communication network 5.

[0060] The display unit 23 includes the display of the visor 11. The display unit 23 has a holographic processing device and displays a 3D image at a predetermined spatial position using hologram technology. Note that the 3D image displayed by the display unit 23 is not limited to a 3D shape, and may be a 2D image such as a document.

[0061] The sound collection unit 24 includes the microphone of the smart glasses 10 described above. The sound collection unit 24 converts sounds that reach the smart glasses 10 into electrical signals. Therefore, when a worker wearing the smart glasses 10 approaches the substrate processing apparatus 40, the sound collection unit 24 can collect sounds emitted from the driving parts of the processing unit 50 (such as the rotational holder 56, the processing liquid nozzle 60, the spray nozzle 65, and the cup 55) and convert them into electrical signals. The electrical signals output from the sound collection unit 24 may be stored in a memory unit of the smart glasses 10 (i.e., the collected sounds may be recorded). The memory unit 29 is a memory of a computer system included in the smart glasses 10.

[0062] The gaze tracking unit 25 includes two infrared cameras that measure the movement of the worker's eyes. The gaze tracking unit 25 has an eye tracking function that tracks the worker's gaze using the two infrared cameras. A worker wearing the smart glasses 10 can perform operations based on their gaze using the eye tracking function.

[0063] The smart glasses 10 also include an abnormality determination unit 31. The abnormality determination unit 31 is a function processing unit that is realized by the CPU of the smart glasses 10 executing a predetermined processing program. The processing content of the abnormality determination unit 31 will be described in more detail later.

[0064] The control unit 45 of the substrate processing apparatus 40 controls the operations of mechanisms provided in the processing unit 50, such as the spin motor 58, the cup lifting mechanism 39, and the nozzle driving units 63 and 68. The control unit 45 of the substrate processing apparatus 40 can communicate with the communication unit 22 of the smart glasses 10, and can also control the operations of various mechanisms provided in the processing unit 50 in accordance with operation instruction commands transmitted from the smart glasses 10.

[0065] The work support terminal 80 and the server 70 are installed, for example, in a factory of a vendor that manufactures and maintains the substrate processing apparatus 40. The work support terminal 80 and the server 70 are capable of communicating with the smart glasses 10 via the information and communication network 5. The work support terminal 80 and the server 70 are also capable of communicating with each other via the information and communication network 5.

[0066] The work support terminal 80 and the server 70 are general computer systems. That is, the work support terminal 80 and the server 70 include a CPU which is a circuit that performs various arithmetic processing, a ROM which is a read-only memory that stores basic programs, a RAM which is a readable and writable memory that stores various information, a storage unit (for example, a magnetic disk or SSD) that stores control software and data, and a communication unit that communicates with the information communication network 5.

[0067] The work support terminal 80 is a computer that allows, for example, a work supporter on the vendor side to support the work of a worker in a clean room. The work supporter can send various information from the work support terminal 80 to the smart glasses 10 worn by the worker in the clean room.

[0068] In the anomaly detection system according to the present invention, the server 70 is a computer that executes predetermined processing in response to requests from the smart glasses 10 and the work support terminal 80. The server 70 includes a memory unit 74 with a relatively large capacity. Large-sized data created by the smart glasses 10 and the work support terminal 80 may be stored in the memory unit 74. Note that the server 70 and the work support terminal 80 are not essential elements.

[0069] Next, we will explain an anomaly detection method using the anomaly detection system having the above-mentioned configuration. The anomaly detection method according to the present invention includes two steps: a model construction step using machine learning as a preliminary step, and an anomaly detection step using the created machine learning model. First, we will explain the model construction.

[0070] 7 is a flowchart showing the procedure for constructing a model. The model construction may be performed at an appropriate timing, for example, before shipping the substrate processing apparatus 40, after installing the substrate processing apparatus 40 in a clean room, or when developing the first substrate processing apparatus 40.

[0071] First, each part of the substrate processing apparatus 40 is imaged to collect image data (step S11). The image capturing in step S11 may be performed using the imaging unit 21 of the smart glasses 10, or may be performed using a separate camera. Fig. 8 is a diagram showing an example of an image captured for model construction. In the example shown in Fig. 8, the inside of the chamber of the processing unit 50 is imaged.

[0072] The image IM captured inside the processing unit 50 shown in FIG. 8 includes various components, such as the processing liquid nozzle 60, the spin chuck 57, and the cup 55. Partial images IM1, IM2, and IM3, each of which has a designated area in the image IM using a rectangle or polygon, are labeled (step S12). Labeling is a process of tagging image data with a label indicating the content of the target image. For example, the partial image IM1, which designates the processing liquid nozzle 60 in the image IM, is labeled "nozzle arm rotation axis." The partial image IM2, which designates the spin chuck 57 in the image IM, is labeled "spin chuck rotation axis." The partial image IM3, which designates the cup 55 in the image IM, is labeled "cup lift axis." Instead of designating the area using a rectangle or the like, partial images may be cropped from the image IM and labeled. Alternatively, labeling may be performed on an image obtained by capturing only a specific portion (for example, only the processing liquid nozzle 60). Furthermore, multiple images may be prepared by capturing images of one portion from various angles, and common labeling may be performed on each of the images.

[0073] Next, the sound emitted by each part (i.e., the part to be labeled) reflected in the partial images IM1, IM2, and IM3 during operation is recorded to collect sound data (step S13). This process may be performed using the sound collection unit 24 of the smart glasses 10, or may be performed using a separate microphone and recorder. However, because sound data also depends on the characteristics of the microphone, it is preferable to collect sound using the sound collection unit 24 of the smart glasses 10 used in abnormality detection, which will be described later.

[0074] In step S13, it is preferable to collect both sound data (normal sound data) of sounds emitted when each part is operating normally and sound data (abnormal sound data) of sounds emitted when the part is operating abnormally (i.e., in the event of a malfunction). For example, sound data of sounds emitted when the spin chuck 57 is rotating normally by the spin motor 58 is collected as normal sound data. On the other hand, sound data of sounds emitted when the spin chuck 57 is rotating abnormally is collected as abnormal sound data. While normal sound data can be collected at any timing, it is preferable to collect and accumulate abnormal sound data each time a malfunction occurs in a part. Note that the sound data may be time domain data (horizontal axis is time) data or frequency domain data (horizontal axis is frequency) data obtained by Fourier transform.

[0075] Next, a machine learning model (AI model) is constructed by machine learning (step S14) based on labeled image data for each part and sound data of sounds emitted by the part when it is operating (step S15). FIG. 9 is a diagram for conceptually explaining model construction by machine learning. Using labeled image data for a certain part and sound data of sounds emitted by the part when it is operating (including both normal sound data and abnormal sound data), machine learning is performed using algorithms such as neural networks, decision trees, and support vector machines (SVMs). The machine learning in step S14 is performed, for example, by a learner 85 ( FIG. 6 ) implemented in the work support terminal 80. The learner 85 constructs a machine learning model 99 by performing machine learning using sampled data of image data and sound data for multiple parts, such as the processing liquid nozzle 60, the spin chuck 57, and the cup 55.

[0076] In machine learning, the more sampled data is learned, the more accurately a machine learning model 99 can be constructed that is sufficiently trained. Therefore, as image data, it is preferable to use data of multiple images of each body part taken from various angles as the sampled data. Furthermore, as sound data, it is preferable to collect data of sounds emitted by each body part in various situations and use the collected data as the sampled data. The machine learning model 99 constructed by the learner 85 is temporarily stored in, for example, the storage unit 74 of the server 70.

[0077] Next, an explanation will be given of anomaly detection for the substrate processing apparatus 40 using the machine learning model 99 created as described above. Fig. 10 is a flowchart showing the procedure for anomaly detection using the machine learning model 99. This anomaly detection may be performed, for example, in an inspection process before shipping the substrate processing apparatus 40.

[0078] When detecting an anomaly, an operator wearing the smart glasses 10 looks inside the processing chamber 51 of one of the processing units 50 in the substrate processing apparatus 40 to be inspected. The above-described machine learning model 99 is downloaded and stored in the memory unit 29 of the smart glasses 10. Then, the position where the operator is looking is identified by the eye tracking function of the smart glasses 10 (step S21). Specifically, the gaze tracking unit 25 of the smart glasses 10 tracks the operator's gaze and identifies the position at the end of the gaze as the viewing position.

[0079] 11 is a diagram showing an example of identifying a gaze position by eye tracking. In this example, a worker wearing smart glasses 10 is looking at a part of the cup 55 of the processing unit 50. Then, the gaze tracking unit 25 tracks the gaze of the worker and identifies the position indicated by the black circle in FIG. 11 as the gaze position.

[0080] Next, the imaging unit 21 of the smart glasses 10 captures an image of the area near the viewing position identified by eye tracking (step S22). Specifically, for example, the imaging unit 21 captures an image of a rectangular imaging area PT that is centered on the viewing position and is composed of a predetermined number of pixels on all four sides.

[0081] Next, the sound collection unit 24 of the smart glasses 10 collects sound while the worker is viewing the viewing position (step S23). At this time, the sound collection unit 24 collects sound while the target part being viewed by the worker is operating. Specifically, for example, the worker inputs a predetermined command to the control unit 45 of the substrate processing apparatus 40 to operate the target part being viewed by the worker. Then, the sound collection unit 24 collects sound while the target part is emitting operating sounds. In the above example, the sound collection unit 24 collects sound while the cup lifting mechanism 39 is lifting and lowering the cup 55 in response to a command input from the worker. Note that in the first embodiment, since this is the pre-shipment inspection stage, parts other than the target part being viewed by the worker are not operating, and therefore parts other than the target part are not emitting sounds.

[0082] Next, the abnormality determination unit 31 of the smart glasses 10 inputs the image data acquired in step S22 and the sound data collected in step S23 to the machine learning model 99 (step S24). Fig. 12 is a diagram for conceptually explaining abnormality determination using the machine learning model 99. The abnormality determination unit 31 inputs the image data of the imaging region PT captured by the imaging unit 21 in step S22 and the sound data collected by the sound collection unit 24 in step S23 to the machine learning model 99 stored in the storage unit 29.

[0083] In the machine learning model 99, image data for each part in the processing unit 50 and sound data (including both normal sound data and abnormal sound data) of sounds emitted by the part when it is operating are mutually associated through learning. The machine learning model 99 analyzes the input image data and sound data and determines whether the sound data is closer to normal sound data or abnormal sound data in relation to the image data. The machine learning model 99 then outputs a determination result that the input sound data is normal if it is closer to normal sound data, and outputs a determination result that the input sound data is abnormal if it is closer to abnormal sound data. Furthermore, if the input sound data is closer to abnormal sound data, the machine learning model 99 may also output the type of abnormality. In this way, it is determined whether the target part visually recognized by the operator is abnormal (step S25).

[0084] In the first embodiment, a machine learning model 99 is generated by machine learning based on image data of images of each part provided in the processing unit 50 and sound data of sounds emitted by the parts when they are activated. When detecting an abnormality, the eye tracking function is used to identify the position where a worker wearing smart glasses 10 equipped with the machine learning model 99 is looking. Then, the imaging unit 21 images an area including the identified looking position, and the sound collection unit 24 collects sound while the worker is looking at the looking position. The image data and sound data collected in this manner are input into the machine learning model 99, thereby determining whether the target part being looked at by the worker is abnormal.

[0085] As described above, in the first embodiment, the worker wearing the smart glasses 10 can determine whether the target area is abnormal by simply visually recognizing the target area for which an abnormality is to be determined, and inputting the image data captured by the imaging unit 21 and the sound data collected by the sound collection unit 24 into the machine learning model 99. Therefore, the worker does not need to perform complicated work, and can quickly and accurately determine the abnormal sound emitted by the target area to detect the abnormality.

[0086] Second Embodiment Next, a second embodiment of the present invention will be described. The overall configuration of the anomaly detection system in the second embodiment, as well as the configurations of the smart glasses 10 and the substrate processing apparatus 40, are the same as those in the first embodiment. In the first embodiment, an anomaly is detected from image data and sound data using a single machine learning model 99, whereas in the second embodiment, the machine learning models are separated into layers.

[0087] FIG. 13 is a diagram conceptually illustrating abnormality detection in the second embodiment. In the second embodiment, a part determination model (first model) that receives image data and outputs the part included in the image data, and an abnormal sound determination model (second model) that receives sound data and determines whether the sound represented by the sound data is an abnormal sound, are hierarchically generated during model construction. For example, a part determination model 191 is constructed by learning image data of each part provided in the processing unit 50 and labels attached to the image data. Furthermore, an abnormal sound determination model 192 is constructed by performing supervised learning on normal sound data representing sounds emitted by each part during normal operation and abnormal sound data representing sounds emitted by each part during abnormal operation. It is preferable to construct an abnormal sound determination model 192 for each part. For example, it is preferable to construct one abnormal sound determination model 192 for each of the processing liquid nozzle 60, the spin chuck 57, and the cup 55.

[0088] Next, when performing abnormality detection, as in the first embodiment, the worker wears the smart glasses 10 equipped with the part determination model 191 and the plurality of abnormal sound determination models 192, and the position at which the worker is looking is identified by the eye tracking function. Then, the imaging unit 21 captures an image of the area including the identified viewing position, and the sound collection unit 24 collects sound while the worker is looking at the viewing position.

[0089] In the second embodiment, first, the image data captured by the imaging unit 21 is input into the part determination model 191 to identify the part included in the image data, i.e., the target part visually recognized by the worker. Next, sound data collected by the sound collection unit 24 is input into the abnormal sound determination model 192 corresponding to the identified target part, thereby performing abnormal sound determination and determining whether the target part visually recognized by the worker is abnormal. That is, if the input sound data is close to normal sound data, the abnormal sound determination model 192 outputs a determination result that the target part is normal, and if the input sound data is close to abnormal sound data, it outputs a determination result that the target part is abnormal.

[0090] Even in the second embodiment, it is possible to determine whether or not the target part is abnormal simply by inputting image data captured by the imaging unit 21 into the part determination model 191 while the worker wearing the smart glasses 10 is visually recognizing the target part for which an abnormality determination is desired, and inputting sound data collected by the sound collection unit 24 into the abnormal sound determination model 192. Therefore, the worker does not need to perform complicated work, and can quickly and accurately determine abnormal sounds emitted by the target part to detect the abnormality.

[0091] Third Embodiment Next, a third embodiment of the present invention will be described. The overall configuration of the anomaly detection system in the third embodiment, as well as the configurations of the smart glasses 10 and the substrate processing apparatus 40, are the same as those in the first embodiment. In the first embodiment, anomaly detection is performed at the inspection stage before shipment, but in the third embodiment, anomaly detection is performed while the substrate processing apparatus 40 is in operation.

[0092] During the pre-shipment inspection stage, no parts other than the target part visually inspected by the operator are in operation, whereas during operation of the substrate processing apparatus 40, multiple parts are operating simultaneously in multiple processing units 50. For example, while the spin chuck 57 rotates, the processing liquid nozzle 60 also moves at the same time. This causes multiple parts to emit sounds simultaneously, and when an abnormality is detected, the sound collection unit 24 collects sounds emitted from parts other than the target part.

[0093] In the third embodiment, when constructing a model, sound data of synthesized sounds emitted when multiple components that are known to operate simultaneously are operated simultaneously is collected, and machine learning is performed. It is possible to determine which multiple components in the substrate processing apparatus 40 will operate simultaneously from the process recipe. The process recipe defines the processing procedure and processing conditions in the substrate processing apparatus 40, and the control unit 45 of the substrate processing apparatus 40 controls each component of the substrate processing apparatus 40 in accordance with the process recipe.

[0094] A machine learning model 99 is constructed by machine learning based on image data of multiple parts that operate simultaneously, which are identified from the processing recipe, and sound data of a composite sound that is emitted when the multiple parts operate simultaneously. For example, if it is determined that the spin chuck 57 and the processing liquid nozzle 60 operate simultaneously in a certain step of the processing recipe, the machine learning model 99 is generated by machine learning based on image data of an image that captures the spin chuck 57 and the processing liquid nozzle 60 and sound data of a composite sound that is emitted when they operate simultaneously.

[0095] When detecting an anomaly, the eye tracking function is used to identify the gaze position of a worker wearing smart glasses 10 equipped with a machine learning model 99 that has learned synthetic sounds. The imaging unit 21 then captures an image of an area including the identified gaze position, and the sound collection unit 24 collects sound while the worker is gazing at the gaze position. The image data and sound data collected in this manner are input into the machine learning model 99, thereby determining whether the target location being viewed by the worker is abnormal. In the third embodiment, machine learning is performed using sound data of synthetic sounds emitted when multiple locations operate simultaneously. This makes it possible to determine whether the target location is abnormal even in a situation where multiple locations operate simultaneously during operation of the substrate processing apparatus 40.

[0096] <Modifications> Although the embodiments of the present invention have been described above, various modifications other than those described above are possible without departing from the spirit of the present invention. For example, in the above embodiments, both normal sound data and abnormal sound data are collected as sound data used for machine learning. However, only normal sound data may be collected and used. When only normal sound data is collected and learned, the machine learning model 99 outputs a determination result that the sound data collected by the sound collection unit 24 is normal when detecting an abnormality if the data is close to the normal sound data, and outputs a determination result that the data is abnormal if the data deviates from the normal sound data by a certain amount or more. However, collecting and learning both normal sound data and abnormal sound data, as in the above embodiments, also makes it easier to identify the cause of an abnormality.

[0097] Furthermore, in each of the above embodiments, the position at which the worker is looking is identified by the eye tracking function of the smart glasses 10. However, instead, the display unit 23 may display the areas that the worker has been looking at, for example, in the last five seconds, in a 3D image in list format, and the worker may select and identify the target area from there.

[0098] In addition, in each of the above embodiments, the smart glasses 10 are equipped with the machine learning model 99 to perform abnormality determination, but this is not limited to this, and for example, the smart glasses 10 may transmit collected image data and sound data to the work support terminal 80, and the work support terminal 80 may perform abnormality determination using the machine learning model 99 stored in the storage unit 74 ( FIG. 6 ). Alternatively, the server 70 may perform abnormality determination.

[0099] The technology of the third embodiment may also be applied to detecting abnormalities in the indexer robot 41 or the main transport robot 48. In the indexer robot 41 and the main transport robot 48, multiple drive units operate simultaneously, and therefore, by performing machine learning using sound data of a synthesized sound that is emitted when the multiple drive units operate simultaneously, it is possible to accurately detect abnormalities in the indexer robot 41 or the main transport robot 48.

[0100] Furthermore, in each of the above embodiments, the worker uses smart glasses 10, but this is not limited thereto. Instead of the smart glasses 10, a portable terminal such as a tablet terminal or a smartphone may be used. That is, any portable terminal equipped with an imaging unit, a communication unit, etc. is sufficient. However, since using a tablet terminal or the like would occupy the worker's hands while holding it, it is preferable to use a wearable terminal such as the smart glasses 10.

[0101] Furthermore, substrate processing apparatus 40 is not limited to a substrate cleaning apparatus, but may be any apparatus that performs a predetermined process on a substrate, such as a heat treatment apparatus, an exposure apparatus, a coating and developing apparatus, a measurement apparatus, or an inspection apparatus. When substrate processing apparatus 40 is a substrate cleaning apparatus, it may be a single-wafer type cleaning apparatus that cleans substrates one by one, or a batch type cleaning apparatus that cleans multiple substrates at once.

[0102] Furthermore, the target of the anomaly detection technology according to the present invention is not limited to substrate processing apparatuses, but may be any industrial equipment having an operating part that performs some operation, such as a printing processing apparatus, a film forming apparatus, a medical apparatus, and a visual inspection apparatus.

[0103] 5 Information and communication network 10 Smart glasses 21 Imaging unit 22 Communication unit 23 Display unit 24 Sound collection unit 25 Gaze tracking unit 31 Abnormality determination unit 39 Cup lifting mechanism 40 Substrate processing apparatus 41 Indexer robot 45 Control unit 48 Main transport robot 50 Processing unit 55 Cup 56 Rotation holding unit 57 Spin chuck 58 Spin motor 60 Processing liquid nozzle 63, 68 Nozzle driving unit 65 Spray nozzle 70 Server 80 Work support terminal 85 Learning device 99 Machine learning model 191 Part determination model 192 Abnormal sound determination model W Substrate

Claims

1. An abnormality detection method for detecting an abnormal state of industrial equipment, comprising: a learning step of generating a machine learning model by learning based on image data of an image of a part of the industrial equipment and sound data of sounds emitted by the part when it is operating; an identification step of using an eye tracking unit to identify a viewing position of the industrial equipment viewed by an operator wearing a mobile terminal equipped with a sound collection unit, a display unit, an imaging unit, a communication unit, and an eye tracking unit; a collection step of using the sound collection unit to collect sound while the operator is viewing the viewing position, and the imaging unit to capture an image of an area including the viewing position; and a determination step of inputting the sound data and image data collected in the collection step into the machine learning model to determine whether the target part viewed by the operator is abnormal.

2. An abnormality detection method according to claim 1, wherein the learning step performs learning using sound data of sounds emitted when the part is operating normally.

3. An abnormality detection method according to claim 2, wherein the learning step further includes learning using sound data of sounds emitted when the part is performing an abnormal operation.

4. An abnormality detection method according to claim 1, wherein the learning step performs learning using sound data of a synthesized sound emitted when multiple parts of the industrial equipment are operating simultaneously.

5. An abnormality detection method according to claim 1, wherein the machine learning model includes a first model that, when image data is input, outputs a part included in the image data, and a second model that, when sound data is input, determines whether or not the sound represented by the sound data is an abnormal sound, and the determination step includes: a part identification step that identifies the target part by inputting the image data collected in the collection step into the first model; and an abnormal sound determination step that determines whether or not the target part is abnormal by inputting the sound data collected in the collection step into the second model.

6. An abnormality detection method according to claim 1, wherein the industrial equipment is a substrate processing apparatus that performs a predetermined process on a substrate.

7. The anomaly detection method according to any one of claims 1 to 6, wherein the mobile terminal is a pair of smart glasses.

8. An abnormality detection system for detecting an abnormal state of industrial equipment, comprising: a portable terminal equipped with a sound collection unit, a display unit, an imaging unit, a communication unit, and an eye-tracking unit; and a learning device that generates a machine learning model by learning based on image data of images of parts of the industrial equipment and sound data of sounds emitted by the parts when they are operating, wherein the eye-tracking unit identifies a viewing position of the industrial equipment that is being viewed by a worker wearing the portable terminal, the sound collection unit collects sound while the worker is viewing the viewing position, and the imaging unit captures an image of an area including the viewing position, and the portable terminal further comprises a determination unit that determines whether the target part being viewed by the worker is abnormal by inputting the sound data collected by the sound collection unit and the image data captured by the imaging unit into the machine learning model.

9. An anomaly detection system according to claim 8, wherein the learning device performs learning using sound data of sounds emitted when the part is operating normally.

10. An anomaly detection system according to claim 9, wherein the learning device further performs learning using sound data of sounds emitted when the part is performing an abnormal operation.

11. An anomaly detection system according to claim 8, wherein the learning device performs learning using sound data of a synthesized sound emitted when multiple parts of the industrial equipment are operating simultaneously.

12. An abnormality detection system according to claim 8, wherein the machine learning model includes a first model that, when image data is input, outputs a part included in the image data, and a second model that, when sound data is input, determines whether the sound represented by the sound data is an abnormal sound, and the determination unit inputs image data captured by the imaging unit into the first model to identify the target part, and then inputs sound data collected by the sound collection unit into the second model to determine whether the target part is abnormal.

13. An abnormality detection system according to claim 8, wherein the industrial equipment is a substrate processing apparatus that performs a predetermined process on a substrate.

14. An anomaly detection system according to any one of claims 8 to 13, wherein the mobile terminal is a pair of smart glasses.

Images