Work assistance method and work assistance system
The work support method and system address the issue of incorrect voice recognition in industrial equipment maintenance by using a mobile terminal to analyze and confirm voice instructions, ensuring accurate command transmission and preventing malfunctions and safety risks through visual and gestural confirmation and specific language models.
Patent Information
- Application Number
- PCT/JP2024/017005
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-30
- Filing Date
- 2024-05-07
- Publication Date
- 2024-12-05
AI Technical Summary
In industrial equipment maintenance, voice recognition systems in smart glasses can misinterpret instructions, leading to potential malfunctions and safety risks due to incorrect recognition, especially when multiple workers are present and non-target voices are recognized.
A work support method and system that uses a mobile terminal with sound receiving, display, and communication sections to analyze voice instructions, confirm through visual or gestural confirmation, and transmit commands only when accurate, incorporating voiceprint matching and language models specific to industrial equipment operations to prevent erroneous executions.
Prevents malfunctions by ensuring accurate recognition and execution of instructions, enhancing safety by confirming the identity of the worker and the correctness of the operation through visual and gestural confirmation, reducing the risk of misinterpretation and unauthorized commands.
Smart Images

Figure JP2024017005_05122024_PF_FP_ABST
Abstract
Description
Work support method and work support system
[0001] The present invention relates to a method and system for supporting operations when performing predetermined operations such as maintenance operations on industrial equipment such as substrate processing apparatuses that perform predetermined operations on substrates. Substrates to be processed by the substrate processing apparatuses include, for example, semiconductor substrates, substrates for liquid crystal display devices, substrates for flat panel displays (FPDs), substrates for optical disks, substrates for magnetic disks, and substrates for solar cells.
[0002] Conventionally, in the manufacturing process of semiconductor devices, substrate processing apparatuses are used to perform various processes on substrates such as semiconductor substrates. Examples of substrate processing apparatuses include substrate cleaning apparatuses, heat treatment apparatuses, and inspection apparatuses. Typically, a large number of substrate processing apparatuses are arranged in an orderly fashion in a large clean room. Maintenance is performed on these substrate processing apparatuses at appropriate intervals. Patent Document 1 describes arranging a large number of substrate processing apparatuses at a relatively high density in a clean room and performing maintenance on the substrate processing apparatuses.
[0003] Furthermore, Patent Document 1 describes that when an operator issues a maintenance instruction to the substrate processing apparatus, the substrate processing apparatus performs a maintenance operation (e.g., ejecting a processing liquid from a processing liquid nozzle). In a typical conventional maintenance operation, an operator manually operates each component of the substrate processing apparatus while holding a remote control. However, in recent years, attempts have been made to have operators wear smart glasses to perform maintenance operations without holding a remote control. A configuration in which an operator operates the smart glasses using an operation panel displayed in the smart glasses or by voice recognition or eye tracking is being considered. Patent Document 2 describes that the operator's speech is recognized and displayed as text information on a mobile device.
[0004] JP 2020-4866 A JP 2021-83079 A
[0005] However, with voice recognition, smart glasses may misinterpret what a worker says. Furthermore, one substrate processing apparatus typically has multiple processing units, and multiple workers may simultaneously perform maintenance work on the same substrate processing apparatus. In such cases, there is a risk that the smart glasses may recognize the voice of someone other than the worker wearing the smart glasses. If the substrate processing apparatus operates based on such misinterpreted content, the worker may be put in danger.
[0006] The present invention has been made in view of the above-mentioned problems, and has an object to provide a task assistance method and a task assistance system that can prevent malfunctions due to erroneous speech recognition.
[0007] In order to solve the above problems, a first aspect of the present invention is a work assistance method for performing specified work on industrial equipment, comprising: a sound receiving step of receiving, by the sound receiving unit, a voice operation instruction issued by a worker wearing a mobile terminal equipped with a sound receiving unit, a display unit, an imaging unit, and a communication unit; a voice recognition step of analyzing the worker's voice received by the sound receiving unit and converting it into text of the operation instruction; a display step of displaying the text of the operation instruction by the display unit; a detection step of detecting a confirmation result indicated by a part of the body of the worker who has confirmed the displayed text of the operation instruction, and acquiring confirmation information; an action analysis step of analyzing the confirmation information and, if the confirmation result is positive, generating command information for the operation instruction; and a transmission step of transmitting, by the communication unit, the command information for the operation instruction to the industrial equipment.
[0008] In a second aspect, in the task support method according to the first aspect, in the detecting step, the imaging unit captures an image of the gesture of the worker.
[0009] In a third aspect, in the work support method according to the first or second aspect, the mobile terminal further includes an eye-gaze tracking unit that tracks the gaze of a worker wearing the mobile terminal, and in the detection step, the part of the industrial equipment that the worker is looking at is identified by the eye-gaze tracking unit, and in the motion analysis step, when the target part that is the subject of the text of the operation instruction matches the part that the worker is looking at, the confirmation result is determined to be acceptable.
[0010] In addition, a fourth aspect is a work assistance method according to any one of the first to third aspects, wherein the voice recognition step performs voice analysis only when the voiceprint of the voice received by the sound receiving unit matches a pre-registered voiceprint.
[0011] In a fifth aspect, in the work assistance method according to any one of the first to fourth aspects, the voice recognition process includes a phoneme identification process for identifying phonemes included in the voice of the worker received by the sound receiving unit using an acoustic model, and a word conversion process for converting the sequence of phonemes identified in the phoneme identification process into words using a language model, and only terms related to specific work performed on the industrial equipment are registered in the language model.
[0012] In a sixth aspect, in the work support method according to any one of the first to fifth aspects, the industrial equipment is a substrate processing apparatus that performs a predetermined process on a substrate.
[0013] In addition, a seventh aspect is the task assistance method according to any one of the first to sixth aspects, wherein the mobile terminal is a pair of smart glasses.
[0014] An eighth aspect is a work assistance system for performing predetermined work on industrial equipment using a mobile terminal, wherein the mobile terminal comprises: a sound receiving unit that receives spoken operation instructions uttered by a worker wearing the mobile terminal; a voice recognition unit that analyzes the worker's voice received by the sound receiving unit and converts it into text of the operation instructions; a display unit that displays the text of the operation instructions; a detection unit that detects a confirmation result indicated by a part of the body of the worker who has confirmed the displayed text of the operation instructions and acquires confirmation information; a motion analysis unit that analyzes the confirmation information and, if the confirmation result is positive, generates command information for the operation instructions; and a communication unit that transmits the command information of the operation instructions to the industrial equipment.
[0015] In a ninth aspect, in the work assistance system according to the eighth aspect, the detection unit includes an imaging unit that captures an image of the gesture of the worker.
[0016] In a tenth aspect, in the work support system according to the eighth or ninth aspect, the detection unit includes an eye-gaze tracking unit that tracks the eye gaze of a worker wearing the mobile terminal to identify the part of the industrial equipment that the worker is viewing, and the motion analysis unit determines that the confirmation result is acceptable when the target part that is the subject of the text of the operation instruction matches the part that the worker is viewing.
[0017] In addition, an eleventh aspect is a work assistance system according to any one of the eighth to tenth aspects, wherein the voice recognition unit performs voice analysis only when the voiceprint of the voice received by the sound receiving unit matches a pre-registered voiceprint.
[0018] In addition, a twelfth aspect is a work assistance system according to any one of the eighth to eleventh aspects, wherein the voice recognition unit identifies phonemes contained in the voice of the worker received by the sound receiving unit using an acoustic model, and converts the sequence of identified phonemes into words using a language model, and only terms related to specific work performed on the industrial equipment are registered in the language model.
[0019] A thirteenth aspect is the work support system according to any one of the eighth to twelfth aspects, wherein the industrial equipment is a substrate processing apparatus that performs a predetermined process on a substrate.
[0020] In addition, a fourteenth aspect is the work assistance system according to any one of the eighth to thirteenth aspects, wherein the mobile terminal is a pair of smart glasses.
[0021] According to the work support methods of the first to seventh aspects, the operation instructions given by the voice of the worker wearing the mobile terminal are received, converted into text, and displayed. If the worker confirms that the text is acceptable after checking it, command information for the operation instructions is sent to the industrial equipment. Therefore, if the voice recognition is incorrect, the operation instructions will not be given to the industrial equipment, and malfunctions due to incorrect voice recognition can be prevented.
[0022] In particular, according to the work assistance method of the fourth aspect, voice analysis is performed only when the voiceprint of the received voice matches a pre-registered voiceprint, thereby more reliably preventing operation instructions from being executed by someone other than the registered worker.
[0023] In particular, according to the work assistance method of the fifth aspect, only terms related to specific work on industrial equipment are registered in the language model, making it possible to more accurately and precisely convert spoken operation instructions given by the worker into text.
[0024] According to the work support system of the eighth to fourteenth aspects, the voice operation instructions given by the worker wearing the mobile terminal are received, converted into text, and displayed, and if the worker confirms that the text is acceptable after checking it, command information for the operation instructions is sent to the industrial equipment. Therefore, if the voice recognition is incorrect, the operation instructions will not be given to the industrial equipment, and malfunctions due to incorrect voice recognition can be prevented.
[0025] In particular, according to the work support system of the eleventh aspect, voice analysis is performed only when the voiceprint of the received voice matches a pre-registered voiceprint, thereby more reliably preventing operation instructions from being executed by someone other than the registered worker.
[0026] In particular, according to the work assistance system of the twelfth aspect, only terms related to specific work on industrial equipment are registered in the language model, making it possible to more accurately and precisely convert spoken operating instructions given by workers into text.
[0027] FIG. 1 is a diagram schematically showing a general configuration of a work support system according to the present invention. FIG. 2 is a plan view for explaining the internal layout of a substrate processing apparatus. FIG. 3 is a plan view showing a general configuration of a processing unit. FIG. 4 is a side view showing a general configuration of a processing unit. FIG. 5 is a perspective view showing an appearance of smart glasses. FIG. 6 is a block diagram showing functional configurations of a control unit of the smart glasses, a server, a work support terminal, and a substrate processing apparatus. FIG. 7 is a flowchart showing the procedure of a work support method according to the present invention. FIG. 8 is a diagram schematically showing a voice recognition process by a voice recognition unit. FIG. 9 is a diagram showing an example of text display by smart glasses. FIG. 10 is a diagram showing an example of a gesture of a worker captured by an imaging unit.
[0028] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Hereinafter, expressions indicating relative or absolute positional relationships (e.g., "in one direction," "along one direction," "parallel," "orthogonal," "center," "concentric," "coaxial," etc.) not only strictly represent the positional relationship but also represent a state of relative angular or distance displacement within a tolerance or a range that provides equivalent functionality, unless otherwise specified. Furthermore, expressions indicating an equal state (e.g., "identical," "equal," "homogeneous," etc.) not only represent a state of strict quantitative equality but also represent a state of difference that provides a tolerance or equivalent functionality, unless otherwise specified. Furthermore, expressions indicating a shape (e.g., "circular," "square," "cylindrical," etc.) not only represent a geometrically strict shape but also represent a shape within a range that provides equivalent functionality, such as irregularities or chamfers, unless otherwise specified. Furthermore, expressions such as "comprise," "comprise," "include," "have," etc., regarding components, are not exclusive expressions that exclude the presence of other components. Furthermore, the expression "at least one of A, B, and C" includes "A only," "B only," "C only," "any two of A, B, and C," and "all of A, B, and C."
[0029] First Embodiment Fig. 1 is a diagram illustrating a schematic configuration of a work support system according to the present invention. The work support system according to the present invention includes a plurality of substrate processing apparatuses 40, smart glasses 10, a server 70, and a work support terminal 80. Controllers of the smart glasses 10 and the substrate processing apparatuses 40 are connected to an information and communication network 5 (e.g., the Internet) via wireless communication. The work support terminal 80 and the server 70 are connected to the information and communication network 5 via a wired connection. Devices connected to the information and communication network 5 can transmit and receive information to and from each other. For example, information can be exchanged between the smart glasses 10 and the work support terminal 80. Note that whether each device is connected to the information and communication network 5 wirelessly or via a wired connection is not limited to the above example and can be any suitable connection (e.g., the work support terminal 80 may be connected to the information and communication network 5 wirelessly).
[0030] The plurality of substrate processing apparatuses 40 are arranged side by side in, for example, a clean room. The clean room is provided in, for example, a semiconductor device manufacturing factory, and is a room where a certain level of air cleanliness is ensured and temperature and humidity are controlled. Workers perform work on the substrate processing apparatuses 40 in the clean room.
[0031] 2 is a plan view illustrating the internal layout of the substrate processing apparatus 40. The substrate processing apparatus 40 is a single-wafer type substrate cleaning apparatus that processes substrates W, which are disk-shaped silicon substrates such as semiconductor wafers, one by one. The substrate processing apparatus 40 includes an indexer unit 43, a plurality of processing units 50, a main transport robot 48, and a control unit 45.
[0032] The indexer section 43 has a plurality of load ports LP (three in this embodiment) and an indexer robot 41. Each load port LP is loaded with a carrier C that accommodates a plurality of substrates W to be processed in a processing unit 50. The carrier C may be in the form of a front opening unified pod (FOUP) that accommodates substrates W in an enclosed space, a standard mechanical interface (SMIF) pod, or an open cassette (OC) that exposes the accommodated substrates W to the outside air.
[0033] The indexer robot 41 transports substrates W between the carriers C and the main transport robot 48. The indexer robot 41 is, for example, an articulated robot, and can transfer substrates W to and from any of the carriers C placed on the multiple load ports LP.
[0034] The main transport robot 48 also transports substrates W between the indexer robot 41 and the processing units 50. The main transport robot 48 is configured to be capable of lifting and lowering, rotating, and moving its transport arm back and forth. The main transport robot 48 receives an unprocessed substrate W that the indexer robot 41 has removed from a carrier C and transports it into the processing unit 50. The indexer robot 41 also receives a processed substrate W that the main transport robot 48 has transported from the processing unit 50 and stores it in the carrier C.
[0035] In the substrate processing apparatus 40, for example, three processing units 50 are stacked to form one stack (tower). Then, for example, four stacks are arranged around the main transport robot 48. That is, one substrate processing apparatus 40 includes, for example, 12 (=3×4) processing units 50. In FIG. 2, one stage of three stacked processing units 50 is schematically shown. Note that the number of processing units 50 in the substrate processing apparatus 40 is not limited to 12 and may be changed as appropriate.
[0036] The main transport robot 48 is installed in the center of the stack of four stacked processing units 50. The main transport robot 48 transfers the substrate W to be processed received from the indexer robot 41 into the inside of the cup 55 of one of the processing units 50. The main transport robot 48 also transfers the processed substrate W from each processing unit 50 and hands it over to the indexer robot 41.
[0037] The substrate processing apparatus 40 also includes a control unit 45. The control unit 45 is a general-purpose computer that controls the operations of the indexer robot 41, the main transport robot 48, and mechanisms provided in each processing unit 50, all of which are provided within the apparatus. The control unit 45 has a touch panel serving as an input / output interface provided on a wall surface of the apparatus, and a communication unit that communicates with the outside of the apparatus. Note that, for convenience of illustration, the control unit 45 is shown in the indexer unit 43 in FIG. 2, but is not limited to this, and the control unit 45 may be provided in an appropriate position within the substrate processing apparatus 40.
[0038] Below, one of the twelve processing units 50 mounted on the substrate processing apparatus 40 will be described, but the other processing units 50 have the same configuration except for the relative arrangement positions of the nozzles.
[0039] The processing unit 50 discharges a processing liquid onto one substrate W to perform a cleaning process. The processing liquid is a term that encompasses various chemical liquids and pure water. Examples of chemical liquids include liquids for etching processes and liquids for removing particles. Specifically, SC-1 liquid (a mixed solution of ammonium hydroxide, hydrogen peroxide, and pure water), SC-2 liquid (a mixed solution of hydrochloric acid, hydrogen peroxide, and pure water), or hydrofluoric acid are used.
[0040] Fig. 3 is a plan view showing a schematic configuration of the processing unit 50. Fig. 4 is a side view showing a schematic configuration of the processing unit 50. The processing unit 50 includes a processing chamber 51, a rotating holder 56, a processing liquid nozzle (first nozzle) 60, a spray nozzle (second nozzle) 65, and a cup 55. The processing chamber 51 is a hollow housing. Inside the processing chamber 51, the rotating holder 56, the processing liquid nozzle 60, the spray nozzle 65, the cup 55, etc. are provided.
[0041] A loading / unloading port 52 is provided on a side wall of the processing chamber 51. The loading / unloading port 52 is opened and closed by a shutter 53. With the shutter 53 opening the loading / unloading port 52, the main transport robot 48 loads and unloads the substrate W into and from the processing chamber 51 through the loading / unloading port 52. The shutter 53 closes the loading / unloading port 52 while the substrate W is being processed. When the loading / unloading port 52 is closed by the shutter 53, the interior of the processing chamber 51 becomes a semi-enclosed space.
[0042] An FFU (fan filter unit) 54 is provided on the ceiling of processing chamber 51. FFU 54 supplies clean air from the ceiling of processing chamber 51 into processing chamber 51. This creates a downflow of clean air from above to below within processing chamber 51. The gas supplied into processing chamber 51 is exhausted through exhaust duct 59 provided at the bottom of processing chamber 51.
[0043] The rotation holder 56 includes a spin chuck 57 and a spin motor 58. The spin chuck 57 is a substrate holder that holds the substrate W in a horizontal position (a position in which the normal to the main surface of the substrate W is aligned vertically). The spin chuck 57 is, for example, a vacuum suction-type chuck. The spin chuck 57 suction-holds the central portion of the lower surface of the substrate W. Note that the spin chuck 57 may be another type of chuck, such as a clamping-type mechanical chuck that grips the edge portion of the substrate W.
[0044] The spin chuck 57 has a disk shape with a diameter smaller than that of the substrate W. When the lower surface of the substrate W is held by suction on the spin chuck 57, the peripheral edge of the substrate W protrudes outward beyond the outer circumferential edge of the spin chuck 57.
[0045] The spin chuck 57 is connected to the spin motor 58 via a motor shaft. That is, the upper end of the motor shaft of the spin motor 58 is connected to the center of the lower surface of the spin chuck 57. When the spin motor 58 rotates the motor shaft while the substrate W is held by suction on the spin chuck 57, the substrate W and the spin chuck 57 rotate in a horizontal plane around a rotation axis extending in the vertical direction.
[0046] A cup 55 is provided to surround the spin chuck 57. The cup 55 can be raised and lowered by a lifting mechanism (not shown). The cup 55 has a generally cylindrical shape, and the upper part of the cup 55 is inclined so that it approaches the spin chuck 57 as it goes up. However, the inner diameter of the upper end part of the cup 55 is larger than the diameter of the substrate W. During processing of the substrate W, the upper end of the cup 55 is higher than the height position of the substrate W held by the spin chuck 57. Therefore, liquid scattered by centrifugal force from the substrate W rotated by the spin holder 56 during processing is received and recovered by the cup 55. The liquid recovered by the cup 55 is discharged from a drain pipe (not shown) provided at the bottom of the cup 55. The cup 55 may have a multi-stage structure in which multiple recovery ports are provided for different purposes.
[0047] The processing liquid nozzle 60 includes a nozzle tip 61, a swing arm 62, and a swing motor 63. The processing liquid nozzle 60 is, for example, a straight nozzle that ejects the processing liquid in a continuous flow state. The nozzle tip 61 is attached to the tip of a swing arm 62 that extends in a substantially horizontal direction. The processing liquid is supplied to the nozzle tip 61 from a processing liquid supply source (not shown), and a discharge port (not shown) is formed in the nozzle tip 61, from which the processing liquid is ejected. The swing arm 62 is swung in a horizontal plane around a swing axis A1 that extends in the vertical direction by the swing motor 63. The swing motor 63 is, for example, a pulse motor.
[0048] The swing motor 63 swings the swing arm 62, causing the nozzle tip 61 to move in an arc-shaped trajectory between a processing position above the substrate W held by the rotary holder 56 and a standby position outside the cup 55. When the nozzle tip 61 is located at the processing position, the processing liquid nozzle 60 discharges a chemical solution onto the substrate W held by the rotary holder 56, thereby, for example, cleaning the substrate W. Alternatively, the processing liquid nozzle 60 discharges pure water onto the substrate W, thereby performing a pure water rinsing process on the substrate W. An encoder for detecting the position of the nozzle tip 61 may be attached to the swing motor 63.
[0049] On the other hand, the spray nozzle 65 includes a nozzle tip 66, a swing arm 67, and a swing motor 68. The spray nozzle 65 is a two-fluid nozzle that mixes, for example, a processing liquid with a pressurized gas to generate droplets and sprays the mixed fluid of the droplets and gas onto the substrate W. The nozzle tip 66 is attached to the tip of a swing arm 67 that extends in a substantially horizontal direction. A processing liquid and a pressurized gas are supplied to the nozzle tip 66 from a processing liquid supply source and a gas supply source (not shown), respectively, and are mixed inside or outside the nozzle tip 61 to form a mixed fluid. The swing arm 67 is swung in a horizontal plane around a swing axis A2 that extends vertically by the swing motor 68. The swing motor 68 is, for example, a pulse motor.
[0050] The swing motor 68 swings the swing arm 67, causing the nozzle tip 66 to move in an arc-shaped trajectory between a processing position above the substrate W held on the rotary holder 56 and a standby position outside the cup 55. When the nozzle tip 66 is located at the processing position, the spray nozzle 65 sprays the fluid mixture onto the substrate W held on the rotary holder 56, thereby proceeding with, for example, cleaning of the substrate W. An encoder for detecting the position of the nozzle tip 66 may be attached to the swing motor 68.
[0051] 3, there is a risk that the rotational movement of the processing liquid nozzle 60 and the rotational movement of the spray nozzle 65 may interfere with each other. That is, when the processing liquid nozzle 60 is located at the processing position, if the spray nozzle 65 also moves above the substrate W, there is a risk that the two may collide with each other. For this reason, an interlock is provided so that when either the processing liquid nozzle 60 or the spray nozzle 65 is located at the processing position, the other cannot operate.
[0052] An operator who performs operations or other tasks on the substrate processing apparatus 40 wears smart glasses 10. The smart glasses 10 are a type of wearable device that uses a head-mounted display (HMD). The smart glasses 10 are also devices that realize AR (Augmented Reality) or MR (Mixed Reality). For example, Microsoft's "HoloLens" (registered trademark) can be used as the smart glasses 10.
[0053] 5 is a perspective view showing the appearance of the smart glasses 10. The smart glasses 10 include a visor 11 and a headband 12. A worker wears the smart glasses 10 by placing the headband 12 on their head. The worker can adjust the length of the headband 12 to fit the size of their head. The headband 12 also includes a power button, a brightness button, a volume button, and the like.
[0054] The visor 11 includes various sensors and a display. The display is a see-through holographic lens. That is, the display can display a three-dimensional image in the worker's field of view using a hologram, and transmits light from real objects in the same way as regular eyeglass lenses. Therefore, a worker wearing the smart glasses 10 can view the displayed three-dimensional image while viewing real objects through the display.
[0055] The sensors of the visor 11 include, for example, multiple visible light cameras that mainly capture images in front of the visor 11, an infrared camera that tracks the worker's line of sight, a depth sensor that measures the distance to an object, and an inertial measurement sensor. The infrared camera tracks the line of sight by measuring the movement of the eyeballs of the wearer of the smart glasses 10. The depth sensor measures the distance to an object using, for example, a Time of Flight (ToF) method. The inertial measurement sensor is composed of an accelerometer, a gyroscope, a magnetometer, etc.
[0056] The smart glasses 10 also have a built-in computer including a CPU, memory, a storage unit, etc. The smart glasses 10 are also provided with a wireless communication mechanism, and the computer in the smart glasses 10 connects to the information communication network 5 using the wireless communication mechanism. The smart glasses 10 are also provided with a microphone, a speaker, a battery, etc.
[0057] 6 is a block diagram showing the functional configuration of the smart glasses 10, the server 70, the work support terminal 80, and the control unit 45 of the substrate processing apparatus 40. The smart glasses 10 include an imaging unit 21, a communication unit 22, a display unit 23, a sound receiving unit 24, and an eye tracking unit 25. The imaging unit 21 includes a visible light camera provided on the visor 11 described above. The imaging unit 21 includes, for example, four visible light cameras that capture images of the front and diagonally forward, and can capture images of the field of view of the worker wearing the smart glasses 10.
[0058] The communication unit 22 includes the wireless communication mechanism of the smart glasses 10 described above. The communication unit 22 transmits and receives data to and from the work support terminal 80 and the server 70 via the information and communication network 5. The communication unit 22 can also transmit and receive data directly to and from the control unit 45 of the substrate processing apparatus 40 if the distance is short. That is, the communication unit 22 can transmit data and commands to the control unit 45 of the substrate processing apparatus 40 directly or via the information and communication network 5.
[0059] The display unit 23 includes the display of the visor 11. The display unit 23 has a holographic processing device and displays a 3D image at a predetermined spatial position using hologram technology. Note that the 3D image displayed by the display unit 23 is not limited to a 3D shape, and may be a 2D image such as a document.
[0060] The sound receiving unit 24 includes the microphone of the smart glasses 10 described above. The sound receiving unit 24 converts sounds that reach the smart glasses 10 into electrical signals. Therefore, the sound receiving unit 24 can collect sounds emitted by a worker wearing the smart glasses 10 and convert the collected sounds into electrical signals. The electrical signals output from the sound receiving unit 24 may be stored in a memory unit of the smart glasses 10 (i.e., the sound may be recorded).
[0061] The gaze tracking unit 25 includes two infrared cameras that measure the movement of the worker's eyes. The gaze tracking unit 25 has an eye tracking function that tracks the worker's gaze using the two infrared cameras. A worker wearing the smart glasses 10 can perform operations based on their gaze using the eye tracking function.
[0062] The smart glasses 10 also include a voice recognition unit 31 and a motion analysis unit 36. The voice recognition unit 31 and the motion analysis unit 36 are function processing units that are realized by the CPU of the smart glasses 10 executing a predetermined processing program. The processing contents of the voice recognition unit 31 and the motion analysis unit 36 will be described in more detail below.
[0063] The control unit 45 of the substrate processing apparatus 40 controls the operation of mechanisms provided in the processing unit 50, such as the swing motor 63. The control unit 45 of the substrate processing apparatus 40 can communicate with the communication unit 22 of the smart glasses 10, and can also control the operation of various mechanisms provided in the processing unit 50 in accordance with operation instruction commands transmitted from the smart glasses 10.
[0064] The work support terminal 80 and the server 70 are installed, for example, in a factory of a vendor that manufactures and maintains the substrate processing apparatus 40. The work support terminal 80 and the server 70 are capable of communicating with the smart glasses 10 via the information and communication network 5. The work support terminal 80 and the server 70 are also capable of communicating with each other via the information and communication network 5.
[0065] The work support terminal 80 and the server 70 are general computer systems. That is, the work support terminal 80 and the server 70 include a CPU which is a circuit that performs various arithmetic processing, a ROM which is a read-only memory that stores basic programs, a RAM which is a readable and writable memory that stores various information, a storage unit (for example, a magnetic disk or SSD) that stores control software and data, and a communication unit that communicates with the information communication network 5.
[0066] The work support terminal 80 is a computer that allows, for example, a work supporter on the vendor side to support the work of a worker in a clean room. The work supporter can send various information from the work support terminal 80 to the smart glasses 10 worn by the worker in the clean room.
[0067] In the work support system according to the present invention, the server 70 is a computer that executes predetermined processes in response to requests from the smart glasses 10 and the work support terminal 80. The server 70 includes a storage unit 74 with a relatively large capacity. Large-sized data created by the smart glasses 10 and the work support terminal 80 may be stored in the storage unit 74. Note that the server 70 and the work support terminal 80 are not essential elements.
[0068] Next, a work support method using the work support system having the above-described configuration will be described. FIG. 7 is a flowchart showing the procedure of the work support method according to the present invention. In this embodiment, an operator wearing smart glasses 10 performs maintenance work on the substrate processing apparatus 40. Specific examples of maintenance work items on the substrate processing apparatus 40 include teaching the processing position for the processing liquid nozzle 60, replacing the spin chuck 57, and teaching the arm of the main transport robot 48. The operator does not operate the substrate processing apparatus 40 by holding a remote control in his / her hand as in the past, but instead operates each part of the substrate processing apparatus 40 using the smart glasses 10. In particular, in this embodiment, the operator operates the apparatus by voice.
[0069] First, an operator wearing the smart glasses 10 issues an operation instruction by voice (step S1). For example, when teaching the processing liquid nozzle 60, the operator speaks something like "inching plus 8 pixels." Also, for example, when adjusting the spin chuck 57 after replacement, the operator speaks something like "chuck open."
[0070] The voice uttered by the worker is received by the sound receiving unit 24 of the smart glasses 10 (step S2). The received voice is converted into an electrical signal by the sound receiving unit 24.
[0071] Next, the voice recognition unit 31 converts the voice received by the sound receiving unit 24 into text through voice recognition (step S3). The voice recognition unit 31 analyzes the voice of the worker received by the sound receiving unit 24 and converts it into text of operation instructions.
[0072] 8 is a diagram showing the voice recognition process performed by the voice recognition unit 31. The voice uttered by the worker is received by the sound receiving unit 24 and converted into an electrical signal, and features are extracted from the voice signal and converted into voice data that is easy for a computer to recognize (acoustic analysis). Humans can hear voices that contain some noise, but computers may have difficulty recognizing voices as they are, so the voice data is processed to make it easier for the computer to recognize.
[0073] Next, the speech recognition unit 31 identifies phonemes from the speech data. A phoneme is the smallest unit of phonemes. The speech recognition unit 31 identifies phonemes contained in the speech of the worker received by the sound receiving unit 24 using an acoustic model. The acoustic model is a module that compares the characteristics of the speech signal extracted by the above acoustic analysis with data that has been trained in advance, and extracts the phoneme that is closest to the input speech. For example, as shown in FIG. 8, when the worker utters the word "open," the speech recognition unit 31 uses the acoustic model to identify the phoneme (phoneme sequence) contained in the worker's speech as "open."
[0074] Because the phoneme sequence identified using the acoustic model is merely a sequence of phonemes, the speech recognition unit 31 converts the identified phoneme sequence into meaningful words using a language model. The language model is a module that extracts and outputs words that are most likely to occur from the identified phonemes based on pre-stored word information. For example, when a phoneme sequence such as that shown in FIG. 8 is identified, the speech recognition unit 31 uses the language model to identify the speech uttered by the worker as the word "OPEN." Note that when converting a phoneme sequence into a word, a speech dictionary containing pre-stored word information may be used in addition to the language model.
[0075] The worker's voice is converted into text through the above-described voice recognition process. Returning to FIG. 7 , the display unit 23 of the smart glasses 10 displays the obtained text of the operation instruction (step S4). FIG. 9 is a diagram showing an example of text display by the smart glasses 10. The display unit 23 of the smart glasses 10 displays the text of the operation instruction obtained by voice recognition as a 3D image. In the example of FIG. 9 , "inching plus 8 pixels," obtained by analyzing the worker's voice, is displayed as a 3D image.
[0076] The worker wearing the smart glasses 10 visually checks the displayed text of the operation instructions. In the first embodiment, the worker expresses the result of the check, i.e., whether the displayed text is consistent with what was spoken, by, for example, a hand gesture (or a finger action). The gesture indicated by the worker is captured and detected by the imaging unit 21 of the smart glasses 10 (step S5). That is, in the first embodiment, the imaging unit 21 functions as a detection unit that detects the check result indicated by the worker and acquires the confirmation information (gesture). FIG. 10 is a diagram showing an example of the worker's gesture captured by the imaging unit 21.
[0077] Next, the motion analysis unit 36 of the smart glasses 10 analyzes the acquired confirmation information and determines whether the confirmation result is correct. In the first embodiment, the motion analysis unit 36 analyzes the gestures of the worker captured by the imaging unit 21 and determines whether the displayed text is correct. Specifically, for example, a model is constructed by learning data on images of a large number of hand gestures for each of the cases where the confirmation result is correct and the case where the confirmation result is incorrect, and the model is used to determine whether the confirmation result is correct.
[0078] For example, the gesture of making a circle with the thumb and index finger as shown in FIG. 10 is a gesture when the confirmation result is "yes." When the motion analysis unit 36 analyzes the gesture shown in FIG. 10 , it determines that the confirmation result is "yes," that is, that the operator has approved the displayed text. In this case, the process proceeds from step S6 to step S7, where the smart glasses 10 generate an operation instruction command in accordance with the operation instruction text obtained by voice recognition. Then, the communication unit 22 of the smart glasses 10 transmits the operation instruction command to the control unit 45 of the substrate processing apparatus 40. In the above example, the operation instruction command "inching plus 8 pixels" is transmitted from the smart glasses 10 to the control unit 45 of the substrate processing apparatus 40.
[0079] The control unit 45 executes an operation instruction command received from the smart glasses 10. For example, when the control unit 45 receives the command described above, the control unit 45 operates the swing motor 63 by eight pixels to move the treatment liquid nozzle 60 in an increment.
[0080] On the other hand, for example, a gesture of crossing the index fingers of both hands to form an X is a gesture indicating a negative confirmation result. When the motion analysis unit 36 analyzes such a gesture, it determines that the confirmation result is negative, that is, that the operator does not approve of the displayed text. In this case, the process proceeds from step S6 to step S8, and the smart glasses 10 cancel the operation. Specifically, the smart glasses 10 temporarily delete the text of the operation instruction obtained by voice recognition, and no command is sent from the smart glasses 10 to the substrate processing apparatus 40. Therefore, no part of the substrate processing apparatus 40 operates.
[0081] In the first embodiment, the smart glasses 10 receive operation instructions issued by a worker wearing the smart glasses 10, convert the audio into text using voice recognition, and display the text. The worker visually checks the displayed text, and if the content is correct, indicates that the confirmation result is acceptable using a hand gesture. The smart glasses 10 capture and analyze the worker's gestures, and transmit an operation instruction command to the substrate processing apparatus 40 only if the confirmation result is acceptable.
[0082] The smart glasses 10 convert the received voice into text through voice recognition and display the text, and the worker confirms the text, preventing operation instructions that are erroneously recognized through voice recognition from being given to the substrate processing apparatus 40. Therefore, even if the smart glasses 10 erroneously interpret an operation instruction given by a worker through voice, it is possible to prevent the substrate processing apparatus 40 from malfunctioning in accordance with the erroneous interpretation.
[0083] In addition, multiple workers may simultaneously perform maintenance work on multiple processing units 50 mounted on the substrate processing apparatus 40. In such cases, there is a risk that the smart glasses 10 may receive and convert into text voices from a person other than the worker wearing the smart glasses 10. In the first embodiment, the smart glasses 10 display the converted voices and allow the worker to confirm the converted voices. Therefore, even if the smart glasses 10 receive and convert into text voices from a person other than the worker, the worker can realize the error and stop the execution of the operation instructions from the person other than the worker. Therefore, it is possible to prevent malfunction of the substrate processing apparatus 40 due to voices from a person other than the worker. In other words, the first embodiment can prevent malfunction of the substrate processing apparatus 40 due to erroneous voice recognition, thereby ensuring the safety of the worker.
[0084] Second Embodiment Next, a second embodiment of the present invention will be described. The overall configuration of the work support system and the configuration of the substrate processing apparatus 40 in the second embodiment are the same as those in the first embodiment. The procedure of the work support method in the second embodiment is also generally the same as that in the first embodiment. In the first embodiment, the worker expresses the confirmation result of the converted text content by hand gestures, but in the second embodiment, the validity of the confirmation result is determined by eye tracking.
[0085] As in the first embodiment, the smart glasses 10 receive a voice instruction issued by the worker wearing the smart glasses 10, convert it into text using voice recognition, and display it. For example, if the worker utters "chuck open," the smart glasses 10 converts it into text and displays it as "chuck open." In the second embodiment, the worker visually checks the displayed text of the operation instruction, and if the content is correct, moves his / her gaze to the part to be operated. For example, if the content of the text displayed as "chuck open" is correct, the worker looks at the spin chuck 57, which is the part to be operated. The gaze tracking unit 25 of the smart glasses 10 tracks the worker's gaze and recognizes where the gaze is directed, and identifies the part of the substrate processing apparatus 40 that the worker is viewing. Note that the smart glasses 10 can identify the part by using, for example, a VPS (Visual Positioning System) function provided in the smart glasses 10 to register position information of each part of the substrate processing apparatus 40 in advance and using the position information. In the second embodiment, the gaze tracking unit 25 functions as a detection unit that detects the confirmation result indicated by the worker and acquires confirmation information (the direction of the gaze).
[0086] Next, in the second embodiment, the motion analysis unit 36 of the smart glasses 10 determines whether the confirmation result is acceptable based on the direction of the operator's line of sight. Specifically, the motion analysis unit 36 analyzes whether the target part of the text of the displayed operation instruction matches the part the operator is looking at, and if they match, determines that the confirmation result is acceptable (the text is correct). For example, if the target part of the text of the operation instruction "chuck open" is the spin chuck 57 and the part the operator is looking at is also the spin chuck 57, the two match, so the motion analysis unit 36 determines that the confirmation result is acceptable. In this case, the communication unit 22 of the smart glasses 10 transmits an operation instruction command to the control unit 45 of the substrate processing apparatus 40, and the control unit 45 executes the command.
[0087] On the other hand, if the target part of the displayed text of the operation instruction does not match the part that the operator is looking at, the motion analysis unit 36 determines that the confirmation result is negative (the text is incorrect). In this case, the smart glasses 10 cancel the operation, and the part of the substrate processing apparatus 40 does not operate.
[0088] In the second embodiment, the smart glasses 10 receive operation instructions issued by a worker wearing the smart glasses 10, convert the audio into text using voice recognition, and display the text. The worker visually checks the displayed text, and if the content is correct, indicates that the confirmation result is acceptable by the direction of the worker's gaze. The smart glasses 10 track and analyze the direction of the worker's gaze, and only if the confirmation result is acceptable, transmits an operation instruction command to the substrate processing apparatus 40.
[0089] As in the first embodiment, the smart glasses 10 convert the received voice into text through voice recognition and display the text, and the operator confirms the text, which prevents operation instructions that are erroneously recognized by voice from being given to the substrate processing apparatus 40. Therefore, even in the second embodiment, it is possible to prevent malfunction of the substrate processing apparatus 40 due to erroneous voice recognition.
[0090] Third Embodiment Next, a third embodiment of the present invention will be described. The overall configuration of the work support system and the configuration of the substrate processing apparatus 40 in the third embodiment are the same as those in the first embodiment. Furthermore, the procedure of the work support method in the third embodiment is also generally the same as that in the first embodiment.
[0091] In the third embodiment, when the smart glasses 10 receive a voice uttered by a worker and convert the voice into text through voice recognition, the voice recognition process is executed only if the voiceprint of the received voice matches the voiceprint of a pre-registered worker. A voiceprint is a sonagram representation of the result of frequency analysis of a voice. Specifically, for example, the worker's voiceprint is pre-registered in an acoustic model ( FIG. 8 ) used to identify phonemes in the voice recognition process. The voice recognition unit 31 compares the voiceprint obtained by frequency analysis of the received voice with the voiceprint pre-registered in the acoustic model. Then, only if the difference between the two is equal to or less than a certain threshold, the voice recognition unit 31 determines that the worker who uttered the voice is the same as the pre-registered worker and converts the received voice into text through voice recognition. The remaining details are the same as those in the first embodiment, except for the voiceprint check.
[0092] In the third embodiment, voice analysis is performed only when the voiceprint of the voice received by the sound receiving unit matches a pre-registered voiceprint. This reduces the possibility that the smart glasses 10 receive and convert into text a voice uttered by a person other than the operator, and more reliably prevents operation instructions from being executed by a person other than the operator. As a result, by following the third embodiment, it is possible to more reliably prevent malfunction of the substrate processing apparatus 40 due to erroneous voice recognition.
[0093] Fourth Embodiment Next, a fourth embodiment of the present invention will be described. The overall configuration of the work support system and the configuration of the substrate processing apparatus 40 in the fourth embodiment are the same as those in the first embodiment. Furthermore, the procedure of the work support method in the fourth embodiment is also generally the same as that in the first embodiment.
[0094] In the fourth embodiment, only terms related to maintenance work on the substrate processing apparatus 40 are registered in a language model ( FIG. 8 ) used when extracting words in the speech recognition process. For example, only terms such as "inching" related to teaching the processing liquid nozzle 60 and "chuck" related to replacing the spin chuck 57 are registered in the language model. By using this language model, the speech recognition unit 31 identifies only terms related to maintenance work on the substrate processing apparatus 40 as words. As a result, it is possible to convert speech uttered by an operator during maintenance of the substrate processing apparatus 40 into text with higher accuracy. The remaining details are the same as those in the first embodiment, except that only terms related to specific work are registered in the language model.
[0095] Typically, a general-purpose language model used for speech recognition is trained with a large number of words so that speech can be converted into sentences optimal for various situations. In the fourth embodiment, only terms related to specific tasks performed on the substrate processing apparatus 40 are registered in the language model. That is, while a language model would normally be registered with as many words as possible, the fourth embodiment deliberately narrows down the words registered in the language model to specific tasks. This enables more accurate and precise text conversion of spoken operational instructions issued by an operator under limited circumstances, such as during maintenance work on the substrate processing apparatus 40. As a result, malfunction of the substrate processing apparatus 40 due to incorrect speech recognition can be more reliably prevented. Furthermore, because the language model is compact, the processing efficiency of speech recognition is improved, allowing for shorter text conversion times.
[0096] <Modifications> Although the embodiments of the present invention have been described above, various modifications other than those described above can be made to the present invention without departing from the spirit of the present invention. For example, in the first embodiment, the worker's gestures are analyzed, and in the second embodiment, eye tracking is used. However, it is also possible to use both of these to determine whether the confirmation result of the textual content is acceptable. In other words, in addition to analyzing the worker's gestures, it is also possible to track the direction of the worker's gaze to determine whether the confirmation result is acceptable.
[0097] Furthermore, the worker may express the confirmation result of the displayed text using a method other than hand gestures or eye gaze. For example, the worker may indicate the confirmation result by moving his / her head while wearing the smart glasses 10. In other words, any method may be used in which the worker indicates the confirmation result using a part of his / her body.
[0098] Furthermore, in the fourth embodiment, only terms related to a specific task are registered in the language model. However, a language model specialized for each task may be prepared. For example, a first language model in which only terms related to teaching the processing liquid nozzle 60 are registered, a second language model in which only terms related to replacing the spin chuck 57 are registered, and a third language model in which only terms related to teaching the arm of the main transport robot 48 are registered may be prepared. Then, the operator selects the optimal language model depending on the maintenance task to be performed. For example, an operator who is replacing the spin chuck 57 selects the second language model. The language model may be selected, for example, through the smart glasses 10. In this way, the language model is specialized for the task to be performed by the operator, and thus, the spoken operation instructions issued by the operator can be converted into text with greater precision and accuracy.
[0099] Furthermore, in each of the above embodiments, the worker uses smart glasses 10, but this is not limited thereto. Instead of the smart glasses 10, a portable terminal such as a tablet terminal or a smartphone may be used. That is, any portable terminal equipped with an imaging unit, a communication unit, etc. is sufficient. However, since using a tablet terminal or the like would occupy the worker's hands while holding it, it is preferable to use a wearable terminal such as the smart glasses 10.
[0100] Furthermore, substrate processing apparatus 40 is not limited to a substrate cleaning apparatus, but may be any apparatus that performs a predetermined process on a substrate, such as a heat treatment apparatus, an exposure apparatus, a coating and developing apparatus, a measurement apparatus, or an inspection apparatus. When substrate processing apparatus 40 is a substrate cleaning apparatus, it may be a single-wafer type cleaning apparatus that cleans substrates one by one, or a batch type cleaning apparatus that cleans multiple substrates at once.
[0101] Furthermore, the work support technology according to the present invention is not limited to substrate processing apparatuses, but may be applied to any industrial equipment equipped with an operating unit that performs some operation, such as a printing processing apparatus, a film forming apparatus, a medical apparatus, and an appearance inspection apparatus.
[0102] 5 Information and communication network 10 Smart glasses 21 Imaging unit 22 Communication unit 23 Display unit 24 Sound receiving unit 25 Gaze tracking unit 31 Voice recognition unit 36 Motion analysis unit 40 Substrate processing apparatus 45 Control unit 48 Main transport robot 50 Processing unit 56 Rotation holding unit 57 Spin chuck 60 Processing liquid nozzle 63 Swing motor 65 Spray nozzle 70 Server 80 Work support terminal W Substrate
Claims
1. A work support method for performing specified work on industrial equipment, comprising: a sound receiving step of receiving, by the sound receiving unit, a voice operation instruction issued by a worker wearing a mobile terminal equipped with a sound receiving unit, a display unit, an imaging unit and a communication unit; a voice recognition step of analyzing the voice of the worker received by the sound receiving unit and converting it into text of the operation instruction; a display step of displaying the text of the operation instruction by the display unit; a detection step of detecting a confirmation result indicated by a part of the body of the worker who has confirmed the displayed text of the operation instruction, and acquiring confirmation information; a motion analysis step of analyzing the confirmation information and, if the confirmation result is positive, generating command information for the operation instruction; and a transmission step of transmitting the command information of the operation instruction to the industrial equipment by the communication unit.
2. A work support method according to claim 1, wherein in the detection step, the imaging unit captures an image of the gesture of the worker.
3. A work support method as described in claim 1, wherein the mobile terminal further comprises an eye-gaze tracking unit that tracks the gaze of a worker wearing the mobile terminal, and in the detection process, the part of the industrial equipment that the worker is viewing is identified by the eye-gaze tracking unit, and in the motion analysis process, when the target part covered by the text of the operation instructions matches the part that the worker is viewing, the confirmation result is determined to be acceptable.
4. A work support method according to claim 1, wherein in the voice recognition step, voice analysis is performed only when the voiceprint of the voice received by the sound receiving unit matches a preregistered voiceprint.
5. A work assistance method as claimed in claim 1, wherein the voice recognition process comprises a phoneme identification process for identifying phonemes contained in the voice of the worker received by the sound receiving unit using an acoustic model, and a word conversion process for converting the sequence of phonemes identified in the phoneme identification process into words using a language model, wherein only terms related to specific work performed on the industrial equipment are registered in the language model.
6. A work support method according to claim 1, wherein the industrial equipment is a substrate processing apparatus that performs a predetermined process on a substrate.
7. A work support method according to any one of claims 1 to 6, wherein the mobile terminal is a pair of smart glasses.
8. A work support system for performing specified work on industrial equipment using a mobile terminal, the mobile terminal comprising: a sound receiving unit that receives spoken operation instructions issued by a worker wearing the mobile terminal; a voice recognition unit that analyzes the voice of the worker received by the sound receiving unit and converts it into text of the operation instructions; a display unit that displays the text of the operation instructions; a detection unit that detects a confirmation result indicated by a part of the body of the worker who has confirmed the displayed text of the operation instructions and acquires confirmation information; a motion analysis unit that analyzes the confirmation information and, if the confirmation result is positive, generates command information for the operation instructions; and a communication unit that transmits the command information of the operation instructions to the industrial equipment.
9. A work support system according to claim 8, wherein the detection unit includes an imaging unit that captures an image of the gesture of the worker.
10. A work support system as described in claim 8, wherein the detection unit includes an eye-gaze tracking unit that tracks the gaze of a worker wearing the mobile terminal to identify the part of the industrial equipment that the worker is viewing, and the motion analysis unit determines that the confirmation result is acceptable when the target part covered by the text of the operation instructions matches the part that the worker is viewing.
11. A work support system according to claim 8, wherein the voice recognition unit performs voice analysis only when the voiceprint of the voice received by the sound receiving unit matches a preregistered voiceprint.
12. A work assistance system as described in claim 8, wherein the voice recognition unit uses an acoustic model to identify phonemes contained in the voice of the worker received by the sound receiving unit, and converts the identified sequence of phonemes into words using a language model, and only terms related to specific work performed on the industrial equipment are registered in the language model.
13. A work support system according to claim 8, wherein the industrial equipment is a substrate processing apparatus that performs a predetermined process on a substrate.
14. A work support system according to any one of claims 8 to 13, wherein the mobile terminal is a pair of smart glasses.
Citation Information
Patent Citations
Remote operation support system
JP2021083079A
Endoscope system
JP2002336184A
Method, unit, and program for control over equipment using speech recognition
JP2003255987A
Display method through head mounted device
JP2015043209A
Control method, control device, and program
JP2017117371A
