Abnormality detection method and abnormality detection system

Abstract

To provide an abnormality detection method and an abnormality detection system capable of detecting abnormality by quickly and highly accurately determining abnormal sound.SOLUTION: The machine learning model 99 is constructed by machine learning based on the image data obtained by imaging each part provided in the substrate processing apparatus and the sound data of the sound emitted when the part is operated. As the sound data, both data of a normal sound emitted when each part is operating normally and data of an abnormal sound emitted when the part is operating abnormally are collected. When performing abnormality detection, the position visually recognized by the worker wearing the smart glasses equipped with the machine learning model 99 is specified by the eye tracking function. An area including the specified visual recognition position is imaged, and the sound is collected in a state where the worker visually recognizes the visual recognition position. The collected image data and sound data are input to the machine learning model 99.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to an abnormality detection method and an abnormality detection system for detecting an abnormal state in industrial equipment such as a substrate processing apparatus that performs predetermined processing on substrates. Substrates to be processed by the substrate processing apparatus include, for example, semiconductor substrates, substrates for liquid crystal displays (LCDs), substrates for flat panel displays (FPDs), substrates for optical disks, substrates for magnetic disks, and substrates for solar cells. [Background technology]

[0002] Sound detection is one of the methods used to inspect drive parts such as motors in industrial equipment such as substrate processing equipment. Conventionally, inspectors would use their hearing to detect abnormal sounds from parts, but this method is highly dependent on the skill level of the inspector, and inexperienced inspectors may not be able to detect abnormalities.

[0003] For this reason, abnormality determination is performed by collecting sounds from the body parts using a microphone or the like and mechanically comparing the measurement sounds with a reference sound. For example, Patent Document 1 discloses a technology in which sounds generated within the visual field of a worker wearing a head-mounted display equipped with a directional microphone are collected as measurement sounds using the directional microphone, and the measurement sounds are compared with the reference sound to determine whether the measurement sounds are abnormal sounds. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-197361 Summary of the Invention [Problem to be solved by the invention]

[0005] However, since sound-emitting parts are generally mounted densely in substrate processing equipment and many sound-emitting parts are present within the field of vision of the operator, the operator must select and specify the reference sound to be compared with the collected measurement sound one by one, which is a time-consuming task.

[0006] In addition, there are cases where multiple nearby parts are emitting sound at the same time, and the wavelength of the sound changes depending on the type of abnormality, so there was much room for improvement in order to bring the accuracy of sound-based abnormality detection technology closer to that of a skilled worker.

[0007] The present invention has been made in view of the above-mentioned problems, and has an object to provide an abnormality detection method and an abnormality detection system that can quickly and accurately determine abnormal sounds and detect abnormalities. [Means for solving the problem]

[0008] In order to solve the above problem, a first aspect of the present invention is an abnormality detection method for detecting an abnormal state of industrial equipment, comprising: a learning process for generating a machine learning model by learning based on image data of an image of a part of the industrial equipment and sound data of sounds emitted by the part when it is operating; an identification process for identifying a viewing position of the industrial equipment viewed by an operator wearing a mobile terminal equipped with a sound collection unit, a display unit, an imaging unit, a communication unit, and an eye tracking unit using the mobile terminal; a collection process for collecting sound with the sound collection unit while the imaging unit images an area including the viewing position while the operator is viewing the viewing position; and a determination process for determining whether the target part viewed by the operator is abnormal by inputting the sound data and image data collected in the collection process into the machine learning model.

[0009] In a second aspect, in the abnormality detection method according to the first aspect, the learning step performs learning using sound data of a sound emitted when the part is operating normally.

[0010] In addition, a third aspect is the abnormality detection method according to the second aspect, wherein the learning step further includes learning using sound data of a sound emitted when the part is performing an abnormal operation.

[0011] In addition, a fourth aspect is an anomaly detection method according to any one of the first to third aspects, wherein the learning process involves learning using sound data of a synthesized sound emitted when multiple parts of the industrial equipment are operating simultaneously.

[0012] Furthermore, in a fifth aspect, in the anomaly detection method according to any one of the first to fourth aspects, the machine learning model includes a first model that, when image data is input, outputs a part included in the image data, and a second model that, when sound data is input, determines whether or not a sound represented by the sound data is an abnormal sound; and the determination step includes a part identification step of identifying the target part by inputting the image data collected in the collection step into the first model, and an abnormal sound determination step of inputting the sound data collected in the collection step into the second model to determine whether or not the target part is abnormal.

[0013] In addition, a sixth aspect is the abnormality detection method according to any one of the first to fifth aspects, wherein the industrial equipment is a substrate processing apparatus that performs a predetermined process on a substrate.

[0014] In addition, a seventh aspect is the anomaly detection method according to any one of the first to sixth aspects, wherein the mobile terminal is a pair of smart glasses.

[0015] In addition, an eighth aspect is an abnormality detection system for detecting an abnormal state of industrial equipment, comprising: a portable terminal having a sound collection unit, a display unit, an imaging unit, a communication unit, and an eye tracking unit; and a learning device that generates a machine learning model by learning based on image data of an image of a part of the industrial equipment and sound data of a sound emitted by the part when it is operating, wherein the eye tracking unit identifies a viewing position of the industrial equipment that is being viewed by a worker wearing the portable terminal, the sound collection unit collects sound while the worker is viewing the viewing position, and the imaging unit images an area including the viewing position, and the portable terminal further comprises a determination unit that determines whether the target part being viewed by the worker is abnormal by inputting the sound data collected by the sound collection unit and the image data captured by the imaging unit into the machine learning model.

[0016] In addition, a ninth aspect is the anomaly detection system according to the eighth aspect, wherein the learning device performs learning using sound data of a sound emitted when the part is operating normally.

[0017] In addition, a tenth aspect is the anomaly detection system according to the ninth aspect, wherein the learning device further performs learning using sound data of sounds emitted when the part is performing an abnormal operation.

[0018] In addition, an eleventh aspect is an anomaly detection system according to any one of the eighth to tenth aspects, wherein the learning device performs learning using sound data of a synthesized sound emitted when multiple parts of the industrial equipment are operating simultaneously.

[0019] In addition, a twelfth aspect is an abnormality detection system according to any one of the eighth to eleventh aspects, wherein the machine learning model includes a first model that, when image data is input, outputs a part included in the image data, and a second model that, when sound data is input, determines whether the sound represented by the sound data is an abnormal sound, and the determination unit identifies the target part by inputting image data captured by the imaging unit into the first model, and then determines whether the target part is abnormal by inputting sound data collected by the sound collection unit into the second model.

[0020] In addition, a thirteenth aspect is the abnormality detection system according to any one of the eighth to twelfth aspects, wherein the industrial equipment is a substrate processing apparatus that performs a predetermined process on a substrate.

[0021] In addition, a fourteenth aspect is the anomaly detection system according to any one of the eighth to thirteenth aspects, wherein the mobile terminal is a pair of smart glasses. [Effects of the Invention]

[0022] According to the anomaly detection methods of the first to seventh aspects, a machine learning model is generated by learning based on image data of an image of a part of industrial equipment and sound data of the sound emitted by the part when it is operating, and the sound data collected by the sound collection unit and the image data collected by the image capturing unit are input into the machine learning model while the worker is visually observing the viewing position, thereby determining whether the target part being viewed by the worker is abnormal or not.Therefore, it is possible to quickly and accurately determine the abnormal sound emitted by the target part and detect the abnormality without the worker having to perform complicated work.

[0023] In particular, according to the abnormality detection method of the third aspect, the learning process involves learning using sound data of sounds emitted when a part is operating normally and sound data of sounds emitted when a part is operating abnormally, thereby further improving the accuracy of abnormality detection.

[0024] In particular, according to the anomaly detection method of the fourth aspect, in the learning process, learning is performed using sound data of a synthesized sound emitted when multiple parts of the industrial equipment are operating simultaneously, so that anomaly detection can be performed even when the industrial equipment is in operation and multiple parts are operating simultaneously.

[0025] According to the anomaly detection system of the eighth to fourteenth aspects, a machine learning model is generated by learning based on image data of an image of a part of industrial equipment and sound data of the sound emitted by the part when it is operating, and the sound data collected by the sound collection unit and the image data collected by the image capturing unit are input into the machine learning model while the worker is visually observing the viewing position, thereby determining whether the target part being viewed by the worker is abnormal or not.Therefore, it is possible to quickly and accurately determine the abnormal sound emitted by the target part and detect the abnormality without the worker having to perform complicated work.

[0026] In particular, according to the anomaly detection system of the tenth aspect, the learning device learns using sound data of sounds emitted when a part is operating normally and sound data of sounds emitted when a part is operating abnormally, thereby further improving the accuracy of anomaly detection.

[0027] In particular, according to the anomaly detection system of the eleventh aspect, the learning device performs learning using sound data of a synthesized sound emitted when multiple parts of an industrial device are operating simultaneously, and therefore, anomalies can be detected even when the industrial device is in operation and multiple parts are operating simultaneously. [Brief explanation of the drawings]

[0028] [Figure 1] 1 is a diagram illustrating a schematic configuration of an anomaly detection system according to the present invention. [Figure 2] FIG. 2 is a plan view for explaining the internal layout of the substrate processing apparatus. [Figure 3] FIG. 2 is a plan view showing a schematic configuration of a processing unit. [Figure 4] FIG. 2 is a side view showing a schematic configuration of a processing unit. [Figure 5]FIG. 1 is a perspective view showing the appearance of smart glasses. [Figure 6] FIG. 2 is a block diagram showing functional configurations of a smart glass, a server, a work support terminal, and a control unit of a substrate processing apparatus. [Figure 7] 1 is a flowchart showing a procedure for constructing a model. [Figure 8] FIG. 10 is a diagram showing an example of an image captured for model construction. [Figure 9] FIG. 1 is a diagram for conceptually explaining model construction through machine learning. [Figure 10] 1 is a flowchart illustrating a procedure for anomaly detection using a machine learning model. [Figure 11] FIG. 10 is a diagram showing an example of identifying a gaze position by eye tracking. [Figure 12] FIG. 1 is a diagram for conceptually explaining anomaly determination using a machine learning model. [Figure 13] FIG. 10 is a diagram for conceptually explaining abnormality determination in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0029] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Hereinafter, expressions indicating relative or absolute positional relationships (e.g., "in one direction," "along one direction," "parallel," "orthogonal," "center," "concentric," "coaxial," etc.) not only strictly represent the positional relationship but also represent a state of relative angular or distance displacement within a tolerance or a range that provides equivalent functionality, unless otherwise specified. Furthermore, expressions indicating an equal state (e.g., "identical," "equal," "homogeneous," etc.) not only represent a state of strict quantitative equality but also represent a state of difference that provides a tolerance or equivalent functionality, unless otherwise specified. Furthermore, expressions indicating a shape (e.g., "circular," "square," "cylindrical," etc.) not only represent a geometrically strict shape but also represent a shape within a range that provides equivalent functionality, such as irregularities or chamfers, unless otherwise specified. Furthermore, expressions such as "comprise," "comprise," "include," "have," etc., regarding components, are not exclusive expressions that exclude the presence of other components. Furthermore, the expression "at least one of A, B, and C" includes "A only," "B only," "C only," "any two of A, B, and C," and "all of A, B, and C."

[0030] First Embodiment FIG. 1 is a diagram showing a schematic configuration of an anomaly detection system according to the present invention. The anomaly detection system according to the present invention includes a plurality of substrate processing apparatuses 40, smart glasses 10, a server 70, and a work support terminal 80. The controllers of the smart glasses 10 and the substrate processing apparatuses 40 are connected to an information and communication network 5 (e.g., the Internet) via wireless communication. The work support terminal 80 and the server 70 are connected to the information and communication network 5 via a wired connection. Information can be transmitted and received between devices connected to the information and communication network 5; for example, information can be exchanged between the smart glasses 10 and the work support terminal 80. Note that whether each device is connected to the information and communication network 5 wirelessly or via a wired connection is not limited to the above example, and can be any suitable form (for example, the work support terminal 80 may be connected to the information and communication network 5 wirelessly).

[0031] The plurality of substrate processing apparatuses 40 are arranged side by side, for example, in a clean room. The clean room is provided, for example, in a semiconductor device manufacturing factory, and is a room where a certain level of air cleanliness is ensured and temperature and humidity are controlled. Workers work on the substrate processing apparatuses 40 in the clean room.

[0032] 2 is a plan view illustrating the internal layout of the substrate processing apparatus 40. The substrate processing apparatus 40 is a single-wafer type substrate cleaning apparatus that processes substrates W, which are disk-shaped silicon substrates such as semiconductor wafers, one by one. The substrate processing apparatus 40 includes an indexer unit 43, a plurality of processing units 50, a main transport robot 48, and a control unit 45.

[0033] The indexer section 43 has a plurality of load ports LP (three in this embodiment) and an indexer robot 41. Each load port LP is loaded with a carrier C that accommodates a plurality of substrates W to be processed in a processing unit 50. The carrier C may take the form of a FOUP (front opening unified pod) that accommodates substrates W in an enclosed space, a SMIF (Standard Mechanical Interface) pod, or an OC (open cassette) that exposes the accommodated substrates W to the outside air.

[0034] The indexer robot 41 transports the substrate W between the carrier C and the main transport robot 48. The indexer robot 41 is, for example, an articulated robot, and can transfer the substrate W to and from any of the carriers C placed on the multiple load ports LP.

[0035] The main transport robot 48 also transports substrates W between the indexer robot 41 and the processing units 50. The main transport robot 48 is configured to be capable of lifting and lowering, rotating, and moving its transport arm back and forth. The main transport robot 48 receives an unprocessed substrate W that the indexer robot 41 has removed from a carrier C and transports it into the processing unit 50. The indexer robot 41 also receives a processed substrate W that the main transport robot 48 has transported from a processing unit 50 and stores it in the carrier C.

[0036] In the substrate processing apparatus 40, for example, three processing units 50 are stacked to form one stack (tower). Then, for example, four stacks are arranged around the main transport robot 48. That is, one substrate processing apparatus 40 includes, for example, 12 (=3×4) processing units 50. FIG. 2 shows one stage of three stacked processing units 50. Note that the number of processing units 50 in the substrate processing apparatus 40 is not limited to 12 and may be changed as appropriate.

[0037] The main transport robot 48 is installed in the center of the stack of four stacked processing units 50. The main transport robot 48 carries the substrate W to be processed, which it receives from the indexer robot 41, into the inside of the cup 55 of one of the processing units 50. The main transport robot 48 also carries out the processed substrate W from each processing unit 50 and hands it over to the indexer robot 41.

[0038] The substrate processing apparatus 40 also includes a control unit 45. The control unit 45 is a general-purpose computer that controls the operations of the indexer robot 41, the main transport robot 48, and mechanisms provided in each processing unit 50, all of which are provided within the apparatus. The control unit 45 has a touch panel serving as an input / output interface provided on a wall surface of the apparatus, and a communication unit that communicates with the outside of the apparatus. Note that, for convenience of illustration, the control unit 45 is shown in the indexer unit 43 in FIG. 2, but is not limited to this, and the control unit 45 may be provided in an appropriate position within the substrate processing apparatus 40.

[0039] The following describes one of the twelve processing units 50 mounted on the substrate processing apparatus 40, but the other processing units 50 have the same configuration except for the relative arrangement of the nozzles.

[0040] The processing unit 50 discharges a processing liquid onto one substrate W to perform a cleaning process. The processing liquid is a term that encompasses various chemical liquids and pure water. Examples of chemical liquids include liquids for etching or liquids for removing particles, and specific examples of such liquids include SC-1 liquid (a mixed solution of ammonium hydroxide, hydrogen peroxide, and pure water), SC-2 liquid (a mixed solution of hydrochloric acid, hydrogen peroxide, and pure water), and hydrofluoric acid.

[0041] Fig. 3 is a plan view showing a schematic configuration of the processing unit 50. Fig. 4 is a side view showing a schematic configuration of the processing unit 50. The processing unit 50 includes a processing chamber 51, a rotating holder 56, a processing liquid nozzle (first nozzle) 60, a spray nozzle (second nozzle) 65, and a cup 55. The processing chamber 51 is a hollow housing. Inside the processing chamber 51, the rotating holder 56, the processing liquid nozzle 60, the spray nozzle 65, the cup 55, etc. are provided.

[0042] A loading / unloading port 52 is provided on a side wall of the processing chamber 51. The loading / unloading port 52 is opened and closed by a shutter 53. With the shutter 53 opening the loading / unloading port 52, the main transport robot 48 loads and unloads the substrate W into and from the processing chamber 51 through the loading / unloading port 52. The shutter 53 closes the loading / unloading port 52 while the substrate W is being processed. When the loading / unloading port 52 is closed by the shutter 53, the interior of the processing chamber 51 becomes a semi-enclosed space.

[0043] An FFU (fan filter unit) 54 is provided on the ceiling of processing chamber 51. FFU 54 supplies clean air from the ceiling of processing chamber 51 into processing chamber 51. This creates a downflow of clean air from above to below within processing chamber 51. The gas supplied into processing chamber 51 is exhausted through exhaust duct 59 provided at the bottom of processing chamber 51.

[0044] The rotation holder 56 includes a spin chuck 57 and a spin motor 58. The spin chuck 57 is a substrate holder that holds the substrate W in a horizontal position (a position in which the normal to the main surface of the substrate W is aligned vertically). The spin chuck 57 is, for example, a vacuum suction-type chuck. The spin chuck 57 suction-holds the central portion of the lower surface of the substrate W. Note that the spin chuck 57 may be another type of chuck, such as a clamping-type mechanical chuck that grips the edge portion of the substrate W.

[0045] The spin chuck 57 has a disk shape with a diameter smaller than that of the substrate W. When the lower surface of the substrate W is held by suction on the spin chuck 57, the peripheral edge of the substrate W protrudes outward beyond the outer circumferential edge of the spin chuck 57.

[0046] The spin chuck 57 is connected to the spin motor 58 via a motor shaft. That is, the upper end of the motor shaft of the spin motor 58 is connected to the center of the lower surface of the spin chuck 57. When the spin motor 58 rotates the motor shaft while the substrate W is held by suction on the spin chuck 57, the substrate W and the spin chuck 57 rotate in a horizontal plane around a rotation axis extending in the vertical direction.

[0047] A cup 55 is provided to surround the spin chuck 57. The cup 55 can be raised and lowered by a cup lifting mechanism 39, conceptually shown in FIG. 4. The cup 55 has a roughly cylindrical shape, and the upper part of the cup 55 is inclined so that it approaches the spin chuck 57 as it goes up. However, the inner diameter of the upper end part of the cup 55 is larger than the diameter of the substrate W. During processing of the substrate W, the upper end of the cup 55 is higher than the height position of the substrate W held by the spin chuck 57. Therefore, liquid scattered by centrifugal force from the substrate W rotated by the spin holder 56 during processing is received and collected by the cup 55. The liquid collected by the cup 55 is discharged from a drain pipe (not shown) provided at the bottom of the cup 55. The cup 55 may have a multi-stage structure in which multiple recovery ports are provided for different purposes.

[0048] The processing liquid nozzle 60 includes a nozzle tip 61, a swing arm 62, and a nozzle driver 63. The processing liquid nozzle 60 is, for example, a straight nozzle that discharges the processing liquid in the form of a continuous flow. The nozzle tip 61 is attached to the tip of a swing arm 62 that extends in a substantially horizontal direction. The processing liquid is supplied to the nozzle tip 61 from a processing liquid supply source (not shown), and a discharge port (not shown) is formed in the nozzle tip 61, from which the processing liquid is discharged. The swing arm 62 is moved up and down by the nozzle driver 63, and is also swung in a horizontal plane around a swing axis A1 that extends in the vertical direction.

[0049] The nozzle drive unit 63 raises and lowers and swings the swing arm 62, thereby moving the nozzle tip 61 between a processing position above the substrate W held by the rotary holder 56 and a standby position outside the cup 55. When the nozzle tip 61 is located at the processing position, the processing liquid nozzle 60 discharges a chemical solution onto the substrate W held by the rotary holder 56, thereby progressing, for example, a cleaning process of the substrate W. Furthermore, the processing liquid nozzle 60 discharges pure water onto the substrate W, thereby progressing a pure water rinsing process of the substrate W.

[0050] On the other hand, the spray nozzle 65 includes a nozzle tip 66, a swing arm 67, and a nozzle driver 68. The spray nozzle 65 is a two-fluid nozzle that mixes, for example, a processing liquid with a pressurized gas to generate droplets and sprays the mixed fluid of the droplets and gas onto the substrate W. The nozzle tip 66 is attached to the tip of a swing arm 67 that extends in a substantially horizontal direction. A processing liquid and a pressurized gas are supplied to the nozzle tip 66 from a processing liquid supply source and a gas supply source (not shown), respectively, and are mixed inside or outside the nozzle tip 61 to form a mixed fluid. The swing arm 67 is moved up and down by the nozzle driver 68 and is also swung in a horizontal plane around a swing axis A2 that extends in the vertical direction.

[0051] The nozzle drive unit 68 raises and lowers and swings the swing arm 67, thereby moving the nozzle tip 66 between a processing position above the substrate W held by the rotary holder 56 and a standby position outside the cup 55. When the nozzle tip 66 is located at the processing position, the spray nozzle 65 discharges the fluid mixture onto the substrate W held by the rotary holder 56, thereby proceeding with, for example, a cleaning process of the substrate W.

[0052] 3, there is a risk that the rotational movement of the processing liquid nozzle 60 and the rotational movement of the spray nozzle 65 may interfere with each other. That is, when the processing liquid nozzle 60 is located at the processing position, if the spray nozzle 65 also moves above the substrate W, there is a risk that the two may collide with each other. For this reason, an interlock is provided so that when either the processing liquid nozzle 60 or the spray nozzle 65 is located at the processing position, the other cannot operate.

[0053] An operator who performs operations or other tasks on the substrate processing apparatus 40 wears smart glasses 10. The smart glasses 10 are a type of wearable terminal that uses a head-mounted display (HMD). The smart glasses 10 are also devices that realize AR (Augmented Reality) or MR (Mixed Reality). For example, Microsoft's "HoloLens" (registered trademark) can be used as the smart glasses 10.

[0054] FIG. 5 is a perspective view showing the appearance of the smart glasses 10. The smart glasses 10 include a visor 11 and a headband 12. A worker wears the smart glasses 10 by placing the headband 12 on their head. The worker can adjust the length of the headband 12 to fit the size of their head. The headband 12 also includes a power button, a brightness button, a volume button, and the like.

[0055] The visor 11 includes various sensors and a display. The display is a see-through holographic lens. That is, the display can display a three-dimensional image in the worker's field of view using a hologram, and transmits light from real objects in the same way as regular eyeglass lenses. Therefore, a worker wearing the smart glasses 10 can view the displayed three-dimensional image while viewing real objects through the display.

[0056] The sensors of the visor 11 include, for example, multiple visible light cameras that mainly capture images in front of the visor 11, an infrared camera that tracks the worker's line of sight, a depth sensor that measures the distance to an object, and an inertial measurement sensor. The infrared camera tracks the line of sight by measuring the movement of the eyeballs of the person wearing the smart glasses 10. The depth sensor measures the distance to an object using, for example, the ToF (Time of Flight) method. The inertial measurement sensor is composed of an accelerometer, a gyroscope, a magnetometer, etc.

[0057] The smart glasses 10 also have a built-in computer including a CPU, memory, a storage unit, etc. The smart glasses 10 are also provided with a wireless communication mechanism, and the computer in the smart glasses 10 connects to the information and communication network 5 using the wireless communication mechanism. The smart glasses 10 are also provided with a microphone, a speaker, a battery, etc.

[0058] 6 is a block diagram showing the functional configuration of the smart glasses 10, the server 70, the work support terminal 80, and the control unit 45 of the substrate processing apparatus 40. The smart glasses 10 include an imaging unit 21, a communication unit 22, a display unit 23, a sound collection unit 24, a memory unit 29, and an eye tracking unit 25. The imaging unit 21 includes a visible light camera provided in the visor 11 described above. The imaging unit 21 includes, for example, four visible light cameras that capture images of the front and diagonally forward, and can capture images of the field of view of the worker wearing the smart glasses 10.

[0059] The communication unit 22 includes the wireless communication mechanism of the smart glasses 10 described above. The communication unit 22 transmits and receives data to and from the work support terminal 80 and the server 70 via the information and communication network 5. The communication unit 22 can also transmit and receive data directly to and from the control unit 45 of the substrate processing apparatus 40 if the distance is short. That is, the communication unit 22 can transmit data and commands to the control unit 45 of the substrate processing apparatus 40 directly or via the information and communication network 5.

[0060] The display unit 23 includes the display of the visor 11. The display unit 23 has a holographic processing device and displays a 3D image at a predetermined spatial position using hologram technology. Note that the 3D image displayed by the display unit 23 is not limited to a 3D shape, and may be a 2D image such as a document.

[0061] The sound collection unit 24 includes the microphone of the smart glasses 10 described above. The sound collection unit 24 converts sounds that reach the smart glasses 10 into electrical signals. Therefore, when a worker wearing the smart glasses 10 approaches the substrate processing apparatus 40, the sound collection unit 24 can collect sounds emitted from the driving parts of the processing unit 50 (such as the rotating holder 56, processing liquid nozzle 60, spray nozzle 65, and cup 55) and convert them into electrical signals. The electrical signals output from the sound collection unit 24 may be stored in a memory unit of the smart glasses 10 (i.e., the collected sounds may be recorded). The memory unit 29 is a memory of a computer system included in the smart glasses 10.

[0062] The gaze tracking unit 25 includes two infrared cameras that measure the movement of the worker's eyes. The gaze tracking unit 25 has an eye tracking function that tracks the worker's gaze using the two infrared cameras. A worker wearing the smart glasses 10 can also perform operations based on their gaze using the eye tracking function.

[0063] The smart glasses 10 also include an abnormality determination unit 31. The abnormality determination unit 31 is a function processing unit that is realized by the CPU of the smart glasses 10 executing a predetermined processing program. The processing content of the abnormality determination unit 31 will be described in more detail later.

[0064] The control unit 45 of the substrate processing apparatus 40 controls the operation of mechanisms provided in the processing unit 50, such as the spin motor 58, the cup lifting mechanism 39, and the nozzle driving units 63 and 68. The control unit 45 of the substrate processing apparatus 40 can communicate with the communication unit 22 of the smart glasses 10, and can also control the operation of various mechanisms provided in the processing unit 50 in accordance with operation instruction commands transmitted from the smart glasses 10.

[0065] The work support terminal 80 and the server 70 are installed, for example, in a factory of a vendor that manufactures and maintains the substrate processing apparatus 40. The work support terminal 80 and the server 70 are capable of communicating with the smart glasses 10 via an information and communication network 5. The work support terminal 80 and the server 70 are also capable of communicating with each other via the information and communication network 5.

[0066] The work support terminal 80 and the server 70 are general computer systems. That is, the work support terminal 80 and the server 70 are equipped with a CPU which is a circuit that performs various arithmetic processing, a ROM which is a read-only memory that stores basic programs, a RAM which is a readable and writable memory that stores various information, a storage unit (for example, a magnetic disk or SSD) that stores control software and data, and a communication unit that communicates with the information communication network 5.

[0067] The work support terminal 80 is a computer that allows, for example, a work supporter on the vendor side to support the work of workers in a clean room. The work supporter can send various information from the work support terminal 80 to the smart glasses 10 worn by the workers in the clean room.

[0068] In the anomaly detection system according to the present invention, the server 70 is a computer that executes predetermined processing in response to requests from the smart glasses 10 and the work support terminal 80. The server 70 includes a memory unit 74 with a relatively large capacity. Large-sized data created by the smart glasses 10 and the work support terminal 80 may be stored in the memory unit 74. Note that the server 70 and the work support terminal 80 are not essential elements.

[0069] Next, we will explain an anomaly detection method using the anomaly detection system having the above-mentioned configuration. The anomaly detection method according to the present invention includes two steps: a model construction step using machine learning as a preliminary step, and an anomaly detection step using the created machine learning model. First, we will explain the model construction.

[0070] 7 is a flowchart showing the procedure for constructing a model. The model construction may be performed at an appropriate timing, for example, before shipping the substrate processing apparatus 40, after installing the substrate processing apparatus 40 in a clean room, or when developing the first substrate processing apparatus 40.

[0071] First, each part of the substrate processing apparatus 40 is imaged to collect image data (step S11). The image capturing in step S11 may be performed using the imaging unit 21 of the smart glasses 10, or may be performed using a separate camera. FIG. 8 is a diagram showing an example of an image captured for model construction. In the example shown in FIG. 8, the inside of the chamber of the processing unit 50 is imaged.

[0072] The image IM captured inside the processing unit 50 shown in FIG. 8 includes various components, such as the processing liquid nozzle 60, the spin chuck 57, and the cup 55. Partial images IM1, IM2, and IM3, each of which is designated as a region of the image IM using a rectangle or polygon, are labeled (step S12). Labeling is a process of tagging image data with labels indicating the content of the target image. For example, the partial image IM1, which designates the processing liquid nozzle 60 in the image IM, is labeled as "nozzle arm rotation axis." The partial image IM2, which designates the spin chuck 57 in the image IM, is labeled as "spin chuck rotation axis." The partial image IM3, which designates the cup 55 in the image IM, is labeled as "cup lift axis." Instead of designating the region using a rectangle or the like, partial images may be cropped from the image IM and labeled. Alternatively, labeling may be performed on an image capturing only a specific portion (e.g., only the processing liquid nozzle 60). Alternatively, a plurality of images of one region may be prepared by capturing the images from various angles, and each image may be labeled in the same way.

[0073] Next, the sound emitted by each part (i.e., the part to be labeled) reflected in the partial images IM1, IM2, and IM3 during operation is recorded to collect sound data (step S13). This process may be performed using the sound collection unit 24 of the smart glasses 10, or may be performed using a separate microphone and recorder. However, because sound data also depends on the characteristics of the microphone, it is preferable to collect sound using the sound collection unit 24 of the smart glasses 10 used in abnormality detection, which will be described later.

[0074] In step S13, it is preferable to collect both sound data (normal sound data) of sounds emitted when each part is operating normally and sound data (abnormal sound data) of sounds emitted when the part is operating abnormally (i.e., in the event of a malfunction). For example, sound data of sounds emitted when the spin chuck 57 is rotating normally by the spin motor 58 is collected as normal sound data. On the other hand, sound data of sounds emitted when the spin chuck 57 is rotating abnormally is collected as abnormal sound data. While normal sound data can be collected at any timing, it is preferable to collect and accumulate abnormal sound data each time a malfunction occurs in a part. Note that the sound data may be time domain data (horizontal axis is time) data or frequency domain data (horizontal axis is frequency) data obtained by Fourier transform.

[0075] Next, a machine learning model (AI model) is constructed (step S15) by machine learning (step S14) based on labeled image data for each part and sound data of sounds emitted by the part when it is operating. FIG. 9 is a diagram for conceptually explaining model construction by machine learning. Using labeled image data for a certain part and sound data of sounds emitted by the part when it is operating (including both normal sound data and abnormal sound data), machine learning is performed using algorithms such as neural networks, decision trees, and support vector machines (SVMs). The machine learning in step S14 is performed, for example, by a learner 85 ( FIG. 6 ) implemented in the work support terminal 80. The learner 85 constructs a machine learning model 99 by performing machine learning using sampled data of image data and sound data for multiple parts, such as the processing liquid nozzle 60, the spin chuck 57, and the cup 55.

[0076] In machine learning, the more sampled data is learned, the more accurately a machine learning model 99 can be constructed that is sufficiently trained. Therefore, as image data, it is preferable to use data of a plurality of images of each body part taken from various angles as the sampled data. Furthermore, as sound data, it is preferable to collect data of sounds emitted by each body part in various situations and use the collected data as the sampled data. The machine learning model 99 constructed by the learner 85 is temporarily stored in the storage unit 74 of the server 70, for example.

[0077] Next, an explanation will be given of anomaly detection for the substrate processing apparatus 40 using the machine learning model 99 created as described above. Fig. 10 is a flowchart showing the procedure for anomaly detection using the machine learning model 99. This anomaly detection may be performed, for example, in an inspection process before shipping the substrate processing apparatus 40.

[0078] When detecting an anomaly, an operator wearing the smart glasses 10 looks inside the processing chamber 51 of one of the processing units 50 in the substrate processing apparatus 40 to be inspected. The above-mentioned machine learning model 99 is downloaded and stored in the memory unit 29 of the smart glasses 10. Then, the position where the operator is looking is identified by the eye tracking function of the smart glasses 10 (step S21). Specifically, the gaze tracking unit 25 of the smart glasses 10 tracks the operator's gaze and identifies the position ahead of the gaze as the gaze position.

[0079] 11 is a diagram showing an example of identifying a gaze position by eye tracking. In this example, a worker wearing smart glasses 10 is looking at a part of a cup 55 in a processing unit 50. Then, the gaze tracking unit 25 tracks the gaze of the worker and identifies the position indicated by a black circle in FIG. 11 as the gaze position.

[0080] Next, the imaging unit 21 of the smart glasses 10 captures an image of an area near the viewing position identified by eye tracking (step S22). Specifically, for example, the imaging unit 21 captures an image of a rectangular imaging area PT that is centered on the viewing position and is composed of a predetermined number of pixels on all four sides.

[0081] Next, the sound collection unit 24 of the smart glasses 10 collects sound while the worker is viewing the viewing position (step S23). At this time, the sound collection unit 24 collects sound while the target part being viewed by the worker is operating. Specifically, for example, the worker inputs a predetermined command to the control unit 45 of the substrate processing apparatus 40 to operate the target part being viewed by the worker. Then, the sound collection unit 24 collects sound while the target part is emitting operating sounds. In the above example, the sound collection unit 24 collects sound while the cup lifting mechanism 39 is lifting and lowering the cup 55 in response to a command input from the worker. Note that in the first embodiment, since this is the pre-shipment inspection stage, parts other than the target part being viewed by the worker are not operating, and therefore parts other than the target part are not emitting sounds.

[0082] Next, the abnormality determination unit 31 of the smart glasses 10 inputs the image data acquired in step S22 and the sound data collected in step S23 to the machine learning model 99 (step S24). FIG. 12 is a diagram for conceptually explaining abnormality determination using the machine learning model 99. The abnormality determination unit 31 inputs the image data of the imaging region PT captured by the imaging unit 21 in step S22 and the sound data collected by the sound collection unit 24 in step S23 to the machine learning model 99 stored in the storage unit 29.

[0083] In the machine learning model 99, image data for each part in the processing unit 50 and sound data (including both normal sound data and abnormal sound data) of sounds emitted by the part when it is operating are mutually associated through learning. The machine learning model 99 analyzes the input image data and sound data and determines whether the sound data is closer to normal sound data or abnormal sound data in relation to the image data. The machine learning model 99 then outputs a determination result that the input sound data is normal if it is closer to normal sound data, and outputs a determination result that the input sound data is abnormal if it is closer to abnormal sound data. Furthermore, if the input sound data is closer to abnormal sound data, the machine learning model 99 may also output the type of abnormality. In this way, it is determined whether the target part visually recognized by the operator is abnormal (step S25).

[0084] In the first embodiment, a machine learning model 99 is generated by machine learning based on image data of images of each part provided in the processing unit 50 and sound data of sounds emitted by the parts when they are activated. When detecting an anomaly, the eye tracking function is used to identify the position where a worker wearing smart glasses 10 equipped with the machine learning model 99 is looking. Then, the imaging unit 21 images an area including the identified looking position, and the sound collecting unit 24 collects sound while the worker is looking at the looking position. The image data and sound data collected in this way are input into the machine learning model 99, thereby determining whether the target part being looked at by the worker is abnormal.

[0085] As described above, in the first embodiment, the worker wearing the smart glasses 10 can determine whether or not the target area is abnormal simply by visually recognizing the target area for which an abnormality determination is desired, and inputting the image data captured by the imaging unit 21 and the sound data collected by the sound collection unit 24 into the machine learning model 99. Therefore, the worker does not need to perform complicated work, and can quickly and accurately determine abnormal sounds emitted by the target area to detect the abnormality.

[0086] Second Embodiment Next, a second embodiment of the present invention will be described. The overall configuration of the anomaly detection system in the second embodiment, as well as the configurations of the smart glasses 10 and the substrate processing apparatus 40, are the same as those in the first embodiment. In the first embodiment, an anomaly is detected from image data and sound data using a single machine learning model 99, whereas in the second embodiment, the machine learning models are separated into layers.

[0087] FIG. 13 is a diagram conceptually illustrating abnormality determination in the second embodiment. In the second embodiment, a part determination model (first model) that receives image data and outputs parts included in the image data, and an abnormal sound determination model (second model) that receives sound data and determines whether the sound represented by the sound data is an abnormal sound, are hierarchically generated during model construction. For example, a part determination model 191 is constructed by learning image data of each part provided in the processing unit 50 and labels attached to the image data. Furthermore, an abnormal sound determination model 192 is constructed by performing supervised learning on normal sound data representing sounds emitted by each part during normal operation and abnormal sound data representing sounds emitted by each part during abnormal operation. It is preferable to construct an abnormal sound determination model 192 for each part. For example, it is preferable to construct one abnormal sound determination model 192 for each of the processing liquid nozzle 60, the spin chuck 57, and the cup 55.

[0088] Next, when performing abnormality detection, as in the first embodiment, the worker wears the smart glasses 10 equipped with the part determination model 191 and the plurality of abnormal sound determination models 192, and the position at which the worker is looking is identified by the eye tracking function. Then, the imaging unit 21 captures an image of the area including the identified viewing position, and the sound collection unit 24 collects sound while the worker is looking at the viewing position.

[0089] In the second embodiment, first, the image data captured by the imaging unit 21 is input into the part determination model 191 to identify the part included in the image data, i.e., the target part visually recognized by the worker. Next, sound data collected by the sound collection unit 24 is input into the abnormal sound determination model 192 corresponding to the identified target part, thereby performing an abnormal sound determination and determining whether the target part visually recognized by the worker is abnormal. That is, if the input sound data is close to normal sound data, the abnormal sound determination model 192 outputs a determination result that the target part is normal, and if the input sound data is close to abnormal sound data, it outputs a determination result that the target part is abnormal.

[0090] Even in the second embodiment, it is possible to determine whether or not the target part is abnormal by simply inputting image data captured by the imaging unit 21 into the part determination model 191 while the worker wearing the smart glasses 10 is visually recognizing the target part for which an abnormality determination is desired, and inputting sound data collected by the sound collection unit 24 into the abnormal sound determination model 192. Therefore, the worker does not need to perform complicated work, and can quickly and accurately determine abnormal sounds emitted by the target part to detect the abnormality.

[0091] <Third embodiment> Next, a third embodiment of the present invention will be described. The overall configuration of the anomaly detection system in the third embodiment, as well as the configurations of the smart glasses 10 and the substrate processing apparatus 40, are the same as those in the first embodiment. In the first embodiment, anomaly detection is performed at the inspection stage before shipment, but in the third embodiment, anomaly detection is performed while the substrate processing apparatus 40 is in operation.

[0092] During the pre-shipment inspection stage, no parts other than the target part visually inspected by the operator are in operation, whereas during operation of the substrate processing apparatus 40, multiple parts are operating simultaneously in multiple processing units 50. For example, while the spin chuck 57 is rotating, the processing liquid nozzle 60 is also moving at the same time. This causes multiple parts to emit sounds simultaneously, and when an abnormality is detected, the sound collection unit 24 collects sounds emitted from parts other than the target part.

[0093] In the third embodiment, when constructing a model, sound data of a synthesized sound emitted when multiple parts that are known to operate simultaneously are operated simultaneously is collected, and machine learning is performed. Which multiple parts of the substrate processing apparatus 40 will operate simultaneously can be determined from the process recipe. The process recipe defines the processing procedure and processing conditions in the substrate processing apparatus 40, and the control unit 45 of the substrate processing apparatus 40 controls each part of the substrate processing apparatus 40 in accordance with the process recipe.

[0094] A machine learning model 99 is constructed by machine learning based on image data of multiple parts that operate simultaneously, which are identified from the processing recipe, and sound data of a composite sound that is emitted when these multiple parts operate simultaneously. For example, if it is determined that the spin chuck 57 and the processing liquid nozzle 60 operate simultaneously in a certain step of the processing recipe, a machine learning model 99 is generated by machine learning based on image data of an image that captures the spin chuck 57 and the processing liquid nozzle 60, and sound data of a composite sound that is emitted when these parts operate simultaneously.

[0095] When detecting an anomaly, the eye tracking function is used to identify the gaze position of a worker wearing smart glasses 10 equipped with a machine learning model 99 that has learned synthetic sounds. Then, the imaging unit 21 captures an image of the area including the identified gaze position, and the sound collection unit 24 collects sound while the worker is gazing at the gaze position. By inputting the image data and sound data collected in this manner into the machine learning model 99, it is determined whether the target part being gazed at by the worker is abnormal. In the third embodiment, machine learning is performed using sound data of synthetic sounds emitted when multiple parts operate simultaneously, so it is possible to determine whether the target part is abnormal even in a situation where multiple parts are operating simultaneously during operation of the substrate processing apparatus 40.

[0096] <Modification> Although the embodiments of the present invention have been described above, various modifications can be made to the present invention without departing from the spirit thereof. For example, in the above embodiments, both normal sound data and abnormal sound data are collected as sound data used for machine learning. However, only normal sound data may be collected and used. When only normal sound data is collected and trained, the machine learning model 99 outputs a determination result that the sound data collected by the sound collection unit 24 is normal when detecting an abnormality if the data is close to the normal sound data, and outputs a determination result that the data is abnormal if the data deviates from the normal sound data by a certain amount or more. However, collecting and training both normal sound data and abnormal sound data, as in the above embodiments, also makes it easier to identify the cause of an abnormality.

[0097] Furthermore, in each of the above embodiments, the position at which the worker is looking is identified by the eye tracking function of the smart glasses 10. However, instead, the display unit 23 may display the areas that the worker has been looking at, for example, in the last 5 seconds, in a 3D image in list format, and the worker may select and identify the target area from there.

[0098] Furthermore, in each of the above embodiments, the smart glasses 10 are equipped with a machine learning model 99 to perform abnormality determination, but this is not limited to this. For example, the smart glasses 10 may transmit collected image data and sound data to the work support terminal 80, and the work support terminal 80 may perform abnormality determination using the machine learning model 99 stored in the storage unit 74 (FIG. 6). Alternatively, the server 70 may perform abnormality determination.

[0099] The technology of the third embodiment may also be applied to detecting abnormalities in the indexer robot 41 or the main transport robot 48. In the indexer robot 41 and the main transport robot 48, multiple drive units operate simultaneously, and therefore, by performing machine learning using sound data of a synthesized sound that is emitted when the multiple drive units operate simultaneously, it is possible to accurately detect abnormalities in the indexer robot 41 or the main transport robot 48.

[0100] Furthermore, in each of the above embodiments, the worker uses smart glasses 10, but this is not limited thereto, and a portable terminal such as a tablet terminal or a smartphone may be used instead of the smart glasses 10. That is, any portable terminal equipped with an imaging unit, a communication unit, etc. is sufficient. However, since using a tablet terminal or the like would occupy the worker's hands while holding it, it is preferable to use a wearable terminal such as the smart glasses 10.

[0101] Furthermore, substrate processing apparatus 40 is not limited to a substrate cleaning apparatus, but may be any apparatus that performs a predetermined process on a substrate, such as a heat treatment apparatus, an exposure apparatus, a coating and developing apparatus, a measurement apparatus, or an inspection apparatus. When substrate processing apparatus 40 is a substrate cleaning apparatus, it may be a single-wafer type cleaning apparatus that cleans substrates one by one, or a batch type cleaning apparatus that cleans multiple substrates at once.

[0102] Furthermore, the target of the anomaly detection technology according to the present invention is not limited to substrate processing apparatuses, but may be any industrial equipment having an operating part that performs some operation, such as a printing processing apparatus, a film forming apparatus, a medical apparatus, and a visual inspection apparatus. [Explanation of symbols]

[0103] 5. Information and Communications Network 10. Smart Glasses 21 Imaging unit 22 Communications Department 23 Display section 24 Sound collection section 25 Eye Tracking Unit 31 Abnormality determination section 39 Cup lifting mechanism 40 Substrate processing equipment 41 Indexer Robot 45 Control Unit 48 Main transport robot 50 processing units 55 cups 56 Rotation holding part 57 Spin Chuck 58 Spin Motor 60 Processing liquid nozzle 63,68 Nozzle drive unit 65 spray nozzle 70 servers 80 Work support terminal 85 Learning Units 99 Machine Learning Models 191 Body Part Identification Model 192 Abnormal noise detection model W substrate

Claims

1. An abnormality detection method for detecting an abnormal state of industrial equipment, comprising: a learning process for generating a machine learning model by learning based on image data of a part of the industrial equipment and sound data of sounds emitted by the part during operation; an identifying step in which a gaze tracking unit identifies a viewing position of the industrial equipment being viewed by a worker wearing a mobile terminal equipped with a sound collecting unit, a display unit, an imaging unit, a communication unit, and an gaze tracking unit; a collection step in which the sound collection unit collects sound while the worker is visually recognizing the visual recognition position, and the imaging unit captures an image of an area including the visual recognition position; a determination step of determining whether or not the target portion visually recognized by the operator is abnormal by inputting the sound data and image data collected in the collection step into the machine learning model; An anomaly detection method comprising:

2. 2. The anomaly detection method according to claim 1, The learning step is an abnormality detection method in which learning is performed using sound data of sounds emitted when the part is operating normally.

3. 3. The abnormality detection method according to claim 2, The learning step further includes learning using sound data of sounds emitted when the part is performing an abnormal operation.

4. 2. The anomaly detection method according to claim 1, The learning step is an anomaly detection method in which learning is performed using sound data of a synthesized sound emitted when multiple parts of the industrial equipment are operating simultaneously.

5. 2. The anomaly detection method according to claim 1, the machine learning model includes a first model that, when image data is input, outputs a part included in the image data, and a second model that, when sound data is input, determines whether a sound represented by the sound data is an allophone; The determination step includes: a region specifying step of specifying the target region by inputting the image data collected in the collecting step into the first model; an abnormal sound determination step of determining whether or not the target portion is abnormal by inputting the sound data collected in the collection step into the second model; An anomaly detection method comprising:

6. 2. The anomaly detection method according to claim 1, The abnormality detection method is such that the industrial equipment is a substrate processing apparatus that performs a predetermined process on a substrate.

7. 7. The abnormality detection method according to claim 1, The anomaly detection method, wherein the mobile terminal is a smart glass.

8. An abnormality detection system for detecting an abnormal state of industrial equipment, a mobile terminal including a sound collection unit, a display unit, an imaging unit, a communication unit, and an eye tracking unit; a learning device that generates a machine learning model by learning based on image data of a part of the industrial equipment and sound data of sounds emitted by the part when it is operating; Equipped with the gaze tracking unit identifies a viewing position of the industrial equipment that is being viewed by a worker wearing the mobile terminal; The sound collection unit collects sound while the worker is visually recognizing the visual recognition position, and the imaging unit captures an image of an area including the visual recognition position, The mobile terminal is an anomaly detection system further including a determination unit that determines whether or not the target area visually recognized by the worker is abnormal by inputting sound data collected by the sound collection unit and image data captured by the imaging unit into the machine learning model.

9. 9. The anomaly detection system according to claim 8, The learning device is an anomaly detection system that performs learning using sound data of sounds emitted when the part is operating normally.

10. 10. The anomaly detection system according to claim 9, The learning device further performs learning using sound data of sounds emitted when the part is performing abnormal operations.

11. 9. The anomaly detection system according to claim 8, The learning device is an anomaly detection system that performs learning using sound data of synthesized sounds emitted when multiple parts of the industrial equipment are operating simultaneously.

12. 9. The anomaly detection system according to claim 8, the machine learning model includes a first model that, when image data is input, outputs a part included in the image data, and a second model that, when sound data is input, determines whether a sound represented by the sound data is an allophone; The determination unit identifies the target area by inputting image data captured by the imaging unit into the first model, and then determines whether the target area is abnormal by inputting sound data collected by the sound collection unit into the second model.

13. 9. The anomaly detection system according to claim 8, The industrial equipment is a substrate processing apparatus that performs a predetermined process on a substrate.

14. The anomaly detection system according to any one of claims 8 to 13, The anomaly detection system, wherein the mobile terminal is a smart glass.

Images