In-tool audio and visual behavior monitoring for process control.

Abstract

Embodiments of the disclosure provided herein include apparatus and methods for more efficient and effective in-tool monitoring of chemical mechanical polishing systems. The polishing apparatus includes a controller connected to at least one camera. The controller is configured to receive a media data stream from at least one camera, process the media data stream into processed media data, and detect anomalies in the polishing apparatus based on the processed media data. A method for monitoring a polishing apparatus includes receiving an image data stream or an audio data stream, processing the data stream, receiving a subsequent image or audio data stream, and comparing a second set of processed data with a first set of processed data.

Description

Technical Field

[0001]

[0001] Embodiments of the present invention generally relate to a chemical mechanical polishing (CMP) system used in the manufacture of semiconductor devices. Specifically, embodiments herein relate to an in-tool acoustic and visual behavior monitoring system for use in CMP.

Background Art

[0002]

[0002] Chemical mechanical polishing (CMP) tools are commonly used in the manufacture of semiconductor devices to planarize or polish substrates, which includes removing excess material and surface imperfections. During the CMP process, the substrate is held within a substrate carrier that presses the backside of the substrate against a rotating polishing pad in the presence of a polishing fluid. Material is removed across the surface of the material layer of the substrate in contact with the polishing pad by a combination of chemical and mechanical actions provided by the polishing fluid and the relative movement between the substrate and the polishing pad.

[0003]

[0003] Hardware and process problems can occur during the operation of these tools, and if not addressed in a timely manner, can ultimately lead to tool failure. Therefore, it is important to detect and analyze deviations from the expected operation of the CMP tool to ensure efficient and effective operation.

[0004]

[0004] Therefore, improved in-tool monitoring for process control of CMP tools is needed.

Summary of the Invention

[0005]

[0005] Embodiments described herein generally relate to systems and methods for monitoring a chemical mechanical polishing (CMP) system used in the manufacture of semiconductor devices. More specifically, embodiments herein provide processes and methods for in-tool monitoring to enable efficient and effective operation.

[0006]

[0006] In one embodiment, a polishing apparatus is provided. The polishing apparatus includes an arm rotatably fixed to a base of the polishing apparatus, a conditioning head connected to the arm and configured to rotate on a platen, at least one camera positioned within the polishing apparatus, and a controller connected to the at least one camera. The controller includes memory and a media analysis processor, and is configured to receive a media data stream from the at least one camera, process the media data stream into processed media data, and detect anomalies within the polishing apparatus based on the processed media data.

[0007]

[0007] In another embodiment, a method for monitoring a polishing apparatus is provided. This method includes receiving a first image data stream from a camera positioned on the polishing apparatus, processing the first image data stream using a video analysis processor on a controller connected to the polishing apparatus to generate a first set of processed image data, receiving a second image data stream from the camera, processing the second image data stream using a video analysis processor to generate a second set of processed image data, comparing the second set of processed image data with the first set of processed image data to determine whether a predetermined threshold has been exceeded, and, if it is determined that a predetermined threshold has been exceeded, sending a warning to a user interface connected to the controller.

[0008]

[0008] In yet another embodiment, a method for monitoring a polishing apparatus is provided. This method includes receiving a first audio data stream from a camera positioned on the polishing apparatus, processing the first audio data stream using an audio analysis processor on a controller connected to the polishing apparatus to generate a first set of processed audio data, receiving a second audio data stream from the camera, processing the second audio data stream using an audio analysis processor to generate a second set of processed audio data, comparing the second set of processed audio data with the first set of processed audio data to determine whether a predetermined threshold has been exceeded, and, if it is determined that a predetermined threshold has been exceeded, sending a warning to a user interface connected to the controller.

[0009]

[0009] To allow for a more detailed understanding of the above-described features of the Disclosure, a more specific description of the Disclosure outlined above can be obtained by referring to embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings show only exemplary embodiments of the Disclosure, as the Disclosure may also permit other equally valid embodiments, and therefore should not be considered to limit the scope of the Disclosure. [Brief explanation of the drawing]

[0010] [Figure 1A] This is a plan view of a polishing apparatus according to one or more embodiments. [Figure 1B] This is a schematic partial cross-sectional side view of the polishing apparatus shown in Figure 1A, according to a certain embodiment. [Figure 1C] This is a schematic partial cross-sectional side view of the polishing apparatus shown in Figure 1A, according to a specific embodiment. [Figure 2A] A top view of a polishing apparatus under a contouring process for an object, according to one embodiment, is shown. [Figure 2B] A top view of a polishing apparatus under image segmentation processing according to one embodiment is shown. [Figure 2C] A top view of a polishing apparatus under image segmentation processing according to one embodiment is shown. [Figure 3A-3C] This is an exemplary spectrogram generated as part of monitoring a polishing apparatus according to a certain embodiment. [Figure 4] This document describes a schematic real-time control method for monitoring a polishing apparatus according to certain embodiments. [Figure 5A] A method for monitoring a polishing apparatus according to certain embodiments is shown. [Figure 5B] A method for monitoring a polishing apparatus according to certain embodiments is shown. [Figures 5C1-5C2] A method for monitoring a polishing apparatus according to a specific embodiment is shown. [Modes for carrying out the invention]

[0011]

[0021] For ease of understanding, the same reference numerals were used to indicate identical elements common to the figures where possible. It is believed that elements and features of one embodiment can be usefully incorporated into other embodiments without further description.

[0012]

[0022] Embodiments of the present invention generally relate to chemical mechanical polishing (CMP) systems used in the manufacture of semiconductor devices. Specifically, embodiments herein relate to in-tool voice and visual behavior monitoring systems for use in CMP.

[0013]

[0023] Hardware and process problems in CMP systems are typically identified at the onset of a failure. Analyzing deviations from expected tool behavior is crucial to ensuring efficient and effective operation.

[0014]

[0024] One way to understand failure modes is to capture audio and visual data during tool operation and review the captured audio and visual data after a failure occurs. Current implementations of video cameras for capturing visual data within board processing systems are passive and primarily provide a visual representation of tool behavior. Therefore, user intervention is required to identify deviations from expected tool behavior or the environment.

[0015]

[0025] In addition, audio data analysis can be used to identify faults. Audio data is collected using localized microphones and then analyzed to identify unusual sounds that may indicate hardware process problems. However, the data collected is often limited and focused on specific hardware components.

[0016]

[0026] Therefore, the present disclosure provides a technical solution to these problems by providing a system and method for implementing audio and visual data captured from a CMP tool to assist in hardware health monitoring, tool misalignment, and process control. The monitoring system provides a camera system that relays video streams and audio streams to an integrated data server from various parts of the CMP system. From this server, the monitoring system extracts video and processes each image frame through trained algorithms. The monitoring system also collects and analyzes audio within the server. The collected video and audio data can be stored and retrieved for comparison with subsequent states of the system in order to monitor the health of the tool as part of a machine learning algorithm. Advanced algorithms can be used on the collected video data to detect object, misalignment component, or color changes, which can indicate abnormal state changes within the CMP system. During real-time substrate processing, the collected video and audio data may be supplied via an intermediate computer or processor, which can enable real-time video detection of audio to analyze video frame and audio changes and monitor substrate polishing behavior, as well as misalignment, anomalies, or endpoint behavior.

[0017]

[0027] The video data stream and the audio data stream can also be used to predict the occurrence of hardware and process problems, enabling predictive maintenance before tool failure. This process allows data collection on the CMP system to complement existing data streams mined from sensors and motors. Visual detection resulting from CMP processing and monitoring of tool status enable improved tool health maintenance, tool cleanliness, and foreign object detection within the CMP system.

[0018]

[0028] FIG. 1A is a plan view of a polishing apparatus 100, such as a chemical mechanical polishing (CMP) tool, for processing one or more substrates. The polishing apparatus 100 includes a polishing platform, or base 102, that at least partially supports and houses a plurality of polishing stations 124. For example, the illustrated polishing apparatus 100 includes four polishing stations 124a, 124b, 124c, and 124d. Each polishing station 124 is adapted to polish a substrate held within a carrier head 126.

[0019]

[0029] The polishing apparatus 100 also includes a plurality of carrier heads 126, each of the plurality of carrier heads 126 being configured to carry a substrate. The number of carrier heads 126 can be equal to or greater than the number of polishing stations 124, e.g., four carrier heads 126 or six carrier heads 126. For example, the number of carrier heads 126 may be two more than the number of polishing stations 124. This allows for the loading and unloading of substrates from two of the carrier heads 126 while polishing is being performed at the remaining portions of the polishing stations 124 with other carrier heads 126, thereby improving throughput.

[0020]

[0030] The polishing apparatus 100 also includes a transfer station 122 for loading substrates onto and unloading substrates from the carrier heads 126. The transfer station 122 can include a plurality of load cups 123, e.g., two load cups 123a and 123b, adapted to facilitate the transfer of substrates between the carrier heads 126 and a factory interface (not shown) or other device (not shown) by a transfer robot 110. The load cups 123 generally facilitate transfer between the robot 110 and each of the carrier heads 126.

[0021]

[0031] The stations of the polishing apparatus 100, including the transfer station 122 and the polishing station 124, can be positioned at substantially equal angular intervals around the center of the base 102. While not essential, this allows for a reduction in the footprint of the polishing apparatus 100.

[0022]

[0032] Each polishing station 124 includes a polishing pad 130 supported on a platen 120 (shown in Figure 1B). For polishing operations, one carrier head 126 is positioned at each polishing station 124. Two additional carrier heads are positioned within the transfer station 122 so that polished substrates can be exchanged for unpolished substrates while other substrates are being polished at the polishing station 124.

[0023]

[0033] The carrier head 126 is adapted to hold the substrate against the polishing surface of the polishing pad 130, while relative motion is provided between the carrier head 126 and the platen 120 for polishing the substrate. The relative motion can be rotational, lateral, or a combination thereof, and is provided by at least one of the carrier head 126 and the platen 120. Each carrier head 126 can individually control polishing parameters, such as pressure, associated with each substrate.

[0024]

[0034] Each carrier head 126 is held by a support structure that allows it to move along a path that sequentially passes through the first polishing station 124a, the second polishing station 124b, the third polishing station 124c, and the fourth polishing station 124d. This allows each carrier head to be selectively positioned on either the polishing station 124 or the load cup 123.

[0025]

[0035] In some implementations, each carrier head 126 is coupled to a carriage 108 mounted on an overhead track 128. By moving the carriage 108 along the overhead track 128, each carrier head 126 can be positioned over a selected polishing station 124 or load cup 123. As the carrier head 126 moves along the overhead track 128, it traverses a path that passes through each of the polishing stations 124.

[0026]

[0036] In the implementation shown in Figure 1A, the overhead track 128 has a circular configuration (shown by dashed lines) that allows the carriage 108 holding the carrier head 126 to selectively orbit above or outside the load cup 123 and polishing station 124. The overhead track 128 may have other configurations, including elliptical, linear, or other suitable orientations. Alternatively, in some embodiments (not shown), the carrier head 126 is suspended from a carousel, and the rotation of the carousel moves all of the carrier head 126 simultaneously along the circular path. Although the polishing apparatus shown herein is equipped with an overhead track, this disclosure can utilize any suitable polishing apparatus. In one embodiment, the polishing apparatus may have a robot that provides the same function as the overhead track.

[0027]

[0037] A controller 190, such as a programmable computer, is connected to the respective motors to independently control the rotational speeds of the platen 120 and the carrier head 126, as will be described in more detail below. For example, each motor may include an encoder that measures the angular position or rotational speed of its associated drive shaft. Similarly, the controller 190 is connected to actuators in each carriage 108 to independently control the lateral movement of each carrier head 126. For example, each actuator may include a linear encoder that measures the position of the carriage 108 along the overhead track 128.

[0028]

[0038] The controller 190 includes a programmable central processing unit (CPU) 192 that can operate together with memory 194 (e.g., non-volatile memory) and support circuitry 196. The support circuitry 196 is conventionally connected to the CPU 192 and includes a cache, clock circuit, input / output subsystem, power supply, etc. (and combinations thereof) connected to various components of the polishing device 100. A user interface 198 is connected to the controller 190 and is configured to send user input to the controller 190 and display information to the user.

[0029]

[0039] In some embodiments, the CPU 192 is one of any form of general-purpose computer processor used in an industrial setting (such as a programmable logic controller (PLC)) for controlling various monitoring system components and subprocessors. The memory 194 linked to the CPU 192 is non-transient and is typically one or more readily available memories such as random access memory (RAM), read-only memory (ROM), a floppy disk drive, a hard disk, or any other form of local or remote digital storage device.

[0030]

[0040] In this publication, memory 194 is in the form of a computer-readable storage medium (e.g., non-volatile memory) containing instructions that, when executed by the CPU 192, facilitate the operation of the polishing device 100. The instructions in memory 194 are in the form of a program product (e.g., a middleware application, an equipment software application, etc., a program that implements the method of this disclosure). The program code may be adapted to any one of several different programming languages. In one example, this disclosure may be implemented as a program product stored in a computer-readable storage medium for use in a computer system. The program of the program product defines the function of the embodiment (including the method of this specification).

[0031]

[0041] Exemplary computer-readable storage media include, but are not limited to, (i) non-writable storage media on which information is permanently stored (e.g., read-only memory devices in a computer, such as a CD-ROM disk readable by a CD-ROM drive, flash memory, ROM chip, or any type of solid-state non-volatile semiconductor memory), and (ii) writable storage media on which modifiable information is stored (e.g., floppy disks in a diskette drive or hard disk drive, or any type of solid-state random-access semiconductor memory). Such computer-readable storage media constitute embodiments of the present disclosure when they transmit computer-readable instructions that direct the functions of the methods described herein.

[0032]

[0042] Although shown as a single computer, the controller 190 may be a distributed system including, for example, multiple independently operating processors and memory. The computer architecture is adaptable to various polishing operations, controlling the order and timing in which the carrier head is positioned at the polishing station based on the programming of the controller 190.

[0033]

[0043] For example, the operating mode is such that the controller loads the circuit board into the carrier head 126 using one of the load cups 123, then sequentially positions the carrier head 126 at each polishing station 124a, 124b, 124c, and 124d, polishing the circuit board sequentially at each polishing station. After polishing at the last station, the carrier head 126 returns to one of the load cups 123, and the circuit board is unloaded from the carrier head 126.

[0034]

[0044] Figure 1B is a schematic partial cross-sectional side view of Figure 1A showing an exemplary spray bar 134 in combination with a polishing station 124. Each polishing station 124 of the polishing apparatus 100 includes a station cup 146 that radially surrounds a platen 120. The station cup 146 has an inner side wall surface 147 facing the platen 120. The inner side wall surface 147 extends above the polishing surface 131 of the polishing pad 130 positioned on the platen 120. Slurry 135 from the polishing pad 130 comes into contact with the inner side wall surface 147 and collects inside the station cup 146. A drain 148 through the bottom of the station cup 146 or the base 102 is used to discharge the slurry 135 collected inside the station cup 146.

[0035]

[0045] The polishing apparatus 100 includes multiple cameras 150 for monitoring a specific region of interest within the polishing apparatus 100. The multiple cameras 150 are configured to receive both video / image data streams and audio data streams. The multiple cameras 150 can transmit the video data streams and audio data streams to the controller 190 in real time.

[0036]

[0046] Each of the multiple cameras 150 may be mounted on a support (not shown) or directly or indirectly connected to each component of the polishing apparatus 100. In some embodiments, each of the multiple cameras 150 is fixed to the polishing apparatus 100. In some other embodiments, one or more of the multiple cameras 150 are movable relative to the polishing apparatus 100 to change the camera's position or orientation and thus change the camera's field of view. Furthermore, each of the multiple cameras 150 is connected to and controlled by a controller 190.

[0037]

[0047] Each of the multiple cameras 150 may include a light source and an image sensor. Alternatively, the light source may be separate from the camera. For example, the light source may be a separate component of the polishing apparatus 100. In one embodiment, the light source may consist of one or more light bars within the housing 101 of the polishing apparatus 100 to illuminate the processing area 105 of the polishing apparatus 100. Each of the multiple cameras 150 may use a lens capable of capturing an enlarged field of view, such as a wide-angle lens or a fisheye lens.

[0038] The polishing apparatus 100 may include a first camera 151 among a plurality of cameras 150 positioned above the carrier head 126. The first camera 151 may be coupled to at least one of the overhead track 128 or the carriage 108 (shown in Figure 1A). The position and orientation of the first camera 151 enable imaging of the surface of the carrier head 126 (for example, the surface of the housing 129 of the carrier head 126).

[0039] The polishing apparatus 100 may include a second camera 152 among a plurality of cameras 150 positioned above the polishing station 124. The second camera 152 may be connected to at least one of the upper wall 103 or side wall 104 of the polishing apparatus 100. Depending on the position and orientation of the second camera 152, it may be possible to image the surface of the carrier head 126 or the surface of one or more structures on the carrier head 126 (e.g., the overhead track 128, the drive system 106 for the carrier head 126, or the drive shaft 107 for the carrier head 126).

[0040]

[0050] The polishing apparatus 100 may include a third camera 153 of a plurality of cameras 150 positioned above the spray bar 134 of the polishing apparatus 100. The third camera 153 may be connected to at least one of the upper wall 103, the side wall 104, the overhead track 128, or the drive system 106. The position and orientation of the third camera 153 enable imaging of the surface of the spray bar 134 (for example, the surface of the arm 136 of the spray bar 134).

[0041] The polishing apparatus 100 may include a fourth camera 154 among a plurality of cameras 150, for example, positioned adjacent to the second camera 152 within the processing area 105. The fourth camera 154 may be connected to at least one of the upper wall 103, the side wall 104, the overhead track 128, or the drive system 106. The position and orientation of the fourth camera 154 enable imaging of the surface of the side wall 104.

[0042]

[0052] The polishing apparatus 100 may include a fifth camera 155 of a plurality of cameras 150 positioned above the station cup 146. The fifth camera 155 may be connected to at least one of the overhead track 128 or the drive system 106. The position and orientation of the fifth camera 155 enable imaging of the surface of the station cup 146, such as the inner side wall surface 147.

[0043]

[0053] Figure 1C is a schematic partial cross-sectional side view of Figure 1A showing an exemplary pad conditioning apparatus 112 in combination with a polishing station 124. Each pad conditioning apparatus 112 includes an arm 113 that supports a conditioner head 115 on its respective platen 120. The arm 113 is rotatably fixed to a base 102. The distal end of the arm 113 is connected to the housing 116 of the conditioner head 115. A motor 117 is connected to the distal end of the arm 113 and rotates the conditioner head 115 during pad conditioning. The proximal end of the arm 113 is connected to a base 118 that extends upward from the base 102 of the housing 101 of the polishing apparatus 100. The base 118 is rotatable to pivot the arm 113 and translate the conditioner head 115 laterally across the polishing surface 131.

[0044] The polishing device 100 may include a sixth camera 156 among a plurality of cameras 150 positioned above the pad conditioning device 112. The sixth camera 156 may be connected to at least one of the upper wall 103, the side wall 104, the overhead track 128, or the drive system 106. The position and orientation of the sixth camera 156 enable imaging of surfaces of the pad conditioning device 112, such as the surface of the arm 113, the conditioner head 115, or the motor 117.

[0045]

[0055] Figure 2A shows a top view of the polishing apparatus 100 under the object contouring process 200A. Object contouring involves detecting and recognizing the boundaries of the object in a digital image. During the contouring process 200A, the camera provides an image data stream to the media analysis processor 292 (such as the video analysis processor 292A) in the controller 190. The controller 190 is configured to reconstruct portions of the multiple contours 210 defined by various components of the polishing apparatus (e.g., arm 113) by performing edge grouping or as parts of the boundaries of the grouped areas.

[0046]

[0056] Controller 190 reconstructs contours using a parallel algorithm that pays particular attention to local image analysis and a generalized eigensolver used in the process. Alternatively, controller 190 can reconstruct contours by using an active contour model (ACM) for image segmentation and object tracking.

[0047]

[0057] ACM may involve evolving contours within an image toward the boundaries of objects. ACM can be based on curve evolution, Mumford-Shah functions for segmentation, and level-setting techniques. ACM can be beneficial due to its strong mathematical properties and efficient numerical programming based on level-setting methods. ACM may also include integrated region, boundary, or shape-based active contour models for overlapping resolution of multiple objects within tissue images.

[0048]

[0058] Figures 2B and 2C show top views of the polishing apparatus 100 under an image segmentation process. More specifically, Figure 2B shows a top view of the polishing apparatus 100 under image segmentation 200B for detecting abnormal environmental conditions such as the presence of steam. Figure 2C shows a top view of the polishing apparatus 100 under image segmentation 200C, identifying individual hardware components within the polishing apparatus.

[0049]

[0059] In Figure 2B, an image segmentation process 200B is used to detect abnormal environmental conditions within the polishing apparatus 100 using object tracking. Image segmentation for object tracking involves dividing an image received by at least one of the multiple cameras 150 into multiple segments or regions, each segment corresponding to a different object or part of an object.

[0050]

[0060] Image segmentation processing 200B may include color segmentation, which detects moving objects based on color difference. Alternatively, moving objects can be detected by tracking the movement of individually tracked image points using light flow and depth information based on received image data, and segmenting them into objects using a globally optimal graph cut algorithm. The controller 190 may use object tracking in image segmentation process 200B to identify abnormal environmental conditions 212 in the polishing apparatus 100, such as steam in the pad conditioning apparatus 112.

[0051]

[0061] As another alternative, the image segmentation process 200B may use cross-classification clustering to track interrelated objects in the image stack received by the sixth camera 156 (Figure 1A). Seeded watersheds can also be used to combine cell segmentation and tracking, and segmentation results from previous timeframes can be used as a seed for watershed segmentation in the current timeframe.

[0052]

[0062] In Figure 2C, the image segmentation process 200C can be used to detect obstacles or inaccurate movements based on tracking objects within the polishing apparatus 100. By detecting and tracking moving objects using image segmentation, obstacles and inaccurate movements can be detected and corrected. In addition, a line segment-based approach can be used for 3D motion estimation and tracking of multiple objects from monocular image sequences. Furthermore, the image segmentation and pattern matching implementation architecture on FPGA / ASIC can also be used for object tracking.

[0053]

[0063] The image segmentation process 200C enables the identification of objects in various components of the polishing apparatus, such as the overhead track 128, the drive system 106, the arm 113, and the pad conditioning device 112. Once the various components of the polishing apparatus 100 are identified, faults or inaccurate movements can be determined using continuous monitoring.

[0054]

[0064] Video / image data collected by at least one of the multiple cameras 150 can be used to detect improper hardware configurations that may not be readily apparent with existing sensors. This disclosure enables cameras in a polishing apparatus to be active components rather than passive components, reducing the user intervention required to identify deviations when a failure mode occurs in the system.

[0055]

[0065] Regarding the audio data processing within the media analysis processor 292 by the audio analysis processor 292B within the controller 190, Figures 3A to 3C show exemplary spectrograms from the audio data received from the polishing apparatus 100. More specifically, Figure 3A shows a normal spectrogram, Figure 3B shows a harmonic spectrogram, and Figure 3C shows a percussive spectrogram.

[0056]

[0066] Audio processing involves manipulating audio signals to accomplish a wide range of tasks, including filtering, data compression, speech processing, and noise suppression. The audio analysis processor 292B may be configured to perform Fast Fourier Transform (FFT) analysis, which is used to analyze the frequency content of the audio signal received by the sixth camera 156 (Figure 1A). FFT analysis can divide the Discrete Fourier Transform (DFT) into smaller DFTs, enabling more efficient computation. Using FFT analysis, the audio analysis processor 292B can also be used to extract features from input audio data for machine learning and deep learning applications.

[0057]

[0067] As shown in Figure 3A, the audio analysis processor 292B (Figure 2A) may be configured to generate a spectrogram 300 based on FFT analysis and analyze the frequency content of the audio signal 302 in a small overlapping time window 304. The resulting frequency content is then plotted as a function of time, with the amplitude 306 of each frequency represented by a color or grayscale value. The audio analysis processor 292B may then use the spectrogram 300 to determine an anomaly or abnormal condition 308 based on the amplitude 306 of the audio signal 302 across the time window 304.

[0058]

[0068] Figures 3B and 3C show exemplary harmonic spectrogram 310 and percussive spectrogram 320. The harmonic and percussive spectrograms are two types of spectrograms used in harmonic / percussive source separation (HPSS) of audio signals. During HPSS within the audio analysis processor 292B, the unique source-specific structure of the power spectrum is utilized to separate the harmonic and percussive components of the audio signal using a log-frequency spectrogram, enabling frequency shift invariance.

[0059]

[0069] The audio analysis processor 292B (Figure 2A) may perform HPSS to separate the harmonic and percussive components of an audio signal received from an audio source (e.g., at least one of the multiple cameras 150). The harmonic spectrogram 310 is used to extract the harmonic components 312 of the audio signal 302, characterized by its fundamental frequency and its harmonics. The audio analysis processor 292B can then determine anomaly conditions by comparing the vertical harmonic response 314 in the audio signal 302 with a spiked or anomalous harmonic response 316.

[0060]

[0070] Meanwhile, the percussive spectrometer 320 is used to extract percussive components 322 of the audio signal 302, characterized by transient audio data 324 and noisy audio data 326. The audio analysis processor 292B can then determine anomaly conditions based on the amplitude and frequency of the noisy audio data 326.

[0061]

[0071] Figure 4 shows an exemplary real-time control scheme 400 for use in the polishing apparatus 100. The real-time control scheme 400 includes a media stream 410, such as a video data stream 412, an audio data stream 414, or both, from a media source (e.g., at least one of a plurality of cameras 150 (Figure 1A)), a video analysis processor (e.g., video analysis processor 292A), and an audio analysis processor (e.g., audio analysis processor 292B), which is capable of processing video and image data, audio data, or a combination thereof. The real-time control scheme 400 includes a tool controller 420 within the polishing apparatus 100.

[0062]

[0072] The control system 400 can intercept image and audio streams for real-time processing. Using the video analysis processor 292A and the audio analysis processor 292B, the control system 400 can process image and audio data through separate computing layers. For example, the control system 400 may use a separate physical computer or a separate digital signal processor to enable faster processing of the image and audio data streams, respectively.

[0063]

[0073] The control system 400 may embed a trained artificial intelligence (AI) or machine learning (ML) inference model in the video analysis processor 292A, the audio analysis processor 292B, or both, to determine the health of the polishing device 100 during operation. For example, the audio analysis processor 292B may include an AI / ML-based classification method for classifying audio signals based on their content. The classification method may include audio tagging, acoustic scene classification, and sound event detection. The audio analysis processor 292B may further include a convolutional neural network (CNN) for extracting features from a spectrogram (e.g., spectrogram 300) for audio tagging.

[0064]

[0074] The control scheme 400 may further include a Long Short-Term Memory (LSTM) Regressive Neural Network (RNN) to selectively store or forget information over time using a gating mechanism to overcome the vanishing gradient problem that occurs in conventional RNNs. Using an LSTM network enables accurate processing even with very long time intervals and delays in the time series.

[0065]

[0075] Figures 5A to 5C2 illustrate video processing, audio processing, or a combination thereof for monitoring the polishing apparatus 100 for use in a real-time control system (e.g., real-time control system 400). More specifically, Figure 5A shows a video processing method for the polishing apparatus (e.g., polishing apparatus 100). Figure 5B shows an audio processing method for the polishing apparatus 100. Figures 5C1 and 5C2 show methods for joint video and audio processing of the polishing apparatus 100.

[0066]

[0076] In Figure 5A, method 500A begins in block 501, where a video analysis processor 292A in the controller 190 receives image data from at least one of the multiple cameras 150. In block 502, the controller 190 performs image processing techniques on the image data. The image processing techniques may include object contouring, image segmentation for object tracking, or a combination thereof, to generate a first set of processed image data. The processed image data may be object contours corresponding to various components of the CMP system, such as the arm (e.g., 113) of a polishing device (e.g., polishing device 112).

[0067]

[0077] Next, in block 503, the video analysis processor 292A may store the first set of processed image data in memory (e.g., memory 194). Then, in block 504, the video analysis processor 292A may receive a second image data stream before processing subsequent image data in the video analysis processor 292A in block 505 to generate a second set of processed image data. Then, in block 506, the video analysis processor 292A may detect anomalies by comparing the second set of processed image data with the first set of processed image data.

[0068]

[0078] In block 507, the video analysis processor 292A detects an anomaly when a second set of processed image data differs from a first set of processed image data by a predetermined threshold. The predetermined threshold may include a percentage difference, a value difference, or an acceptable range. For example, if the contour of the second set of processed image data differs from the contour of the first set of processed image data by more than 10%, more than 5%, or more than 3% in position, the video analysis processor 292A determines that the predetermined threshold has been exceeded. The predetermined threshold may correspond to excessive vibration of a component of the polishing apparatus 100 (e.g., arm 113), abnormal environmental conditions such as steam present in the polishing apparatus 100, or any other desired anomaly. In block 508, if the video analysis processor 292A determines that the predetermined threshold has been exceeded, the controller may send a warning to the user interface 198 connected to the controller 190.

[0069]

[0079] Alternatively, each of the first set of processed image data and subsequent sets of processed image data can be compared with baseline image data stored in memory 194. The baseline image data may include previous image data collected under normal operation of the CMP system.

[0070]

[0080] Method 500A then returns to block 504 for continuous system monitoring. In an LTSM application, Method 500A includes an additional block 509. In this block 509, a second set of processed image data passes through a gate mechanism, including an input gate, an output gate, and a forget gate, and is stored or forgotten according to the weights currently assigned to the LSTM network.

[0071]

[0081] Method 500A enables object contouring to identify various hardware components of the CMP system and detect anomalies in the CMP environment.

[0072]

[0082] In Figure 5B, method 500B begins in block 511 with the audio analysis processor 292B in the controller 190 receiving audio data from at least one of the multiple cameras 150. In block 512, the controller 190 performs an audio processing technique on the audio data. The audio processing technique may include FFT analysis, HPSS, or a combination thereof, and can generate a first set of processed audio data. The processed audio data may be a normal spectrogram, a harmonic spectrogram, a percussive spectrogram, or a combination thereof, correlating with the sound generated within the CMP system.

[0073]

[0083] Next, in block 513, the audio analysis processor 292B may store the first set of processed audio data in memory (e.g., memory 194). Then, in block 514, before processing subsequent audio data within the audio analysis processor 292B, the audio analysis processor 292B may receive subsequent audio data and generate a second set of processed audio data. Then, in block 515, the audio analysis processor 292B may detect anomalies by comparing the second set of processed audio data with the first set of processed audio data.

[0074]

[0084] In block 516, the audio analysis processor 292B detects an anomaly if a second set of processed audio data differs from a first set of processed audio data by a predetermined threshold. The predetermined threshold may include a percentage difference, a value difference, or an acceptable range. For example, if the contour of the second set of processed audio data differs from the contour of the first set of processed audio data by more than 10%, more than 5%, or more than 3%, the audio analysis processor 292B determines that the predetermined threshold has been exceeded. The predetermined threshold may correspond to the condition of consumables in the polishing device 100 (e.g., polishing pads), a failure of a tool in the polishing device 100, or any other desired anomaly. In block 517, if the audio analysis processor 292B determines that the predetermined threshold has been exceeded, the controller may send a warning to the user interface 198 connected to the controller 190.

[0075]

[0085] Alternatively, each of the first set of processed audio data and subsequent sets of processed audio data may be compared with baseline audio data stored in memory 194. The baseline audio data may include previous audio data collected under normal operation of the CMP system.

[0076]

[0086] Method 500B then returns to block 514 for continuous system monitoring. In an LTSM application, Method 500B includes an additional block 518. In this block 518, a second set of processed audio data passes through the input gate, output gate, and forget gate and is either stored or forgotten according to the weights currently assigned to the LSTM network.

[0077]

[0087] Method 500B allows monitoring the frequency pattern of a CMP system to detect anomalies in the CMP environment and determine the wear life status. For example, a polishing pad may generate sound at a certain frequency when it is nearing the beginning of its life. As the polishing pad wears down, the frequency it generates during the CMP process shifts. Method 500B can determine this exemplary frequency shift and send an alarm when the target frequency of the polishing pad nearing the end of its life is reached.

[0078]

[0088] In Figure 5C1, Method 500C begins in blocks 521A and 521B with receiving image data from at least one of the multiple cameras 150 to the video analysis processor 292A and audio analysis processor 292B in the controller 190, respectively. In block 522A, the controller 190 performs image processing techniques on the image data to generate a first set of processed image data, as described above in Method 500A. In block 522B, the controller 190 performs audio processing techniques on the audio data to generate a first set of processed audio data, as described in Method 500B.

[0079]

[0089] Next, in block 523A, the video analysis processor 292A may store a first set of processed image data in memory (e.g., memory 194). Then, before processing subsequent image data in the video analysis processor 292A in block 525A, the video analysis processor 292A may receive the subsequent image data in the video analysis processor 292A in block 524A and generate a second set of processed image data. Then, in block 526A, the video analysis processor 292A may detect anomalies by comparing the second set of processed image data with the first set of processed image data, as described in method 500A.

[0080]

[0090] The audio analysis processor 292B may store a first set of processed audio data in the memory of block 523B (e.g., memory 194). Then, before processing the subsequent audio data in the audio analysis processor 292B in block 525B, the audio analysis processor 292B may receive the subsequent audio data in block 524B and generate a second set of processed audio data. Then, in block 526B, the audio analysis processor 292B may detect anomalies by comparing the second set of processed audio data with the first set of processed audio data, as described in method 500B.

[0081]

[0091] As shown in Figure 5C2, method 500C involves the controller 190 subsequently comparing any anomalies detected by the video analysis processor 292A with any anomalies detected by the audio analysis processor 292B, and vice versa, in block 527. The controller 190 may then determine whether an anomaly detected by the audio analysis processor 292B verifies an anomaly detected by the video analysis processor 292A, or vice versa. For example, if the video analysis processor 292A detects a tool failure (e.g., arm 113), the controller 190 can verify this anomaly using spikes in percussive spectroscopic data provided by the audio analysis processor 292B. Alternatively, the controller 190 may determine that an anomaly exists if either the video analysis processor 292A or the audio analysis processor 292B detects an anomaly.

[0082]

[0092] Next, in block 528, the controller 190 sends a warning to the user interface 198 connected to the controller 190. The warning may correspond to a combination of anomalies or to individual anomalies.

[0083]

[0093] Method 500C then returns to blocks 524A and 524B for continuous system monitoring. In LTSM applications, Method 500C includes additional blocks 529A and 529B, in which a second set of processed image data and a second set of processed audio data pass through the system's respective input gates, output gates, and forget gates and are stored or forgotten according to the weights of the ongoing division of the LSTM network. Method 500C enables the use of real-time visual and audio information to determine and predict the processing state of the board in order to control the processing.

[0084]

[0094] This disclosure provides novel systems and methods for using video / image and audio data to complement existing data streams extracted from sensors and motors. Visual detection during the process and monitoring of tool conditions enable improved tool health maintenance, such as detecting tool cleanliness or foreign objects in the system. Captured environmental audio may provide additional insights into process conditions. This disclosure also enables the integration of sensors.

[0085]

[0095] When describing elements of this disclosure, or exemplary aspects or embodiments thereof, the articles "a," "an," "the," and "said" are intended to indicate that there is one or more elements.

[0086]

[0096] The expressions "comprising," "including," and "having" are intended to be comprehensive, meaning that there may be additional elements beyond those listed.

[0087]

[0097] The term “coupled” is used herein to refer to a direct or indirect connection between two objects. For example, even if object A is in physical contact with object B and object B is in contact with object C, objects A and C may still be considered coupled to each other, even if they are not in direct physical contact with each other. For example, an object can be coupled to an object second even if the first object never comes into direct physical contact with the second object.

[0088]

[0098] While the above description applies to embodiments of the present disclosure, other embodiments and further embodiments of the present disclosure can be devised without departing from the basic scope of the present disclosure, and the scope of the present disclosure is defined by the following claims.

Claims

1. A polishing device, the polishing device is An arm rotatably fixed to the base of the polishing device, A conditioning head connected to the aforementioned arm and configured to rotate on the platen, At least one camera positioned within the polishing apparatus, A controller connected to at least one of the cameras, and having memory and a media analysis processor. The controller is equipped with, The media data stream is received from at least one of the cameras. The media data stream is processed to obtain processed media data. Based on the processed media data, abnormalities within the polishing device are detected. A polishing device configured in such a way.

2. The polishing apparatus according to claim 1, wherein the media analysis processor is a video analysis processor, the media data stream is an image data stream, and the processed media data is processed image data.

3. The polishing apparatus according to claim 2, wherein the video analysis processor is configured to process the image data stream using the contour of an object.

4. The polishing apparatus according to claim 3, wherein the processed image data includes a first set of processed image data and a second set of processed image data, and the controller is configured to detect the abnormality by comparing the second set of processed image data with the first set of processed image data.

5. The polishing apparatus according to claim 1, wherein the media analysis processor is an audio analysis processor, the media data stream is an audio data stream, and the processed media data is processed audio media data.

6. The polishing apparatus according to claim 5, wherein the audio analysis processor is configured to process the audio data stream using fast Fourier transform analysis.

7. The polishing apparatus according to claim 6, wherein the processed audio data includes a first set of processed audio data and a second set of processed audio data, and the controller is configured to detect the anomaly by comparing the second set of processed audio data with the first set of processed audio data.

8. A method for monitoring a polishing device, Receiving a first image data stream from a camera positioned in the polishing apparatus, Using the video analysis processor of the controller connected to the polishing apparatus, the first image data stream is processed to generate a first set of processed image data. Receiving a second image data stream from the aforementioned camera, Using the video analysis processor, the second image data stream is processed to generate a second set of processed image data. In order to determine whether a predetermined threshold has been exceeded, the second set of processed image data and the first set of processed image data are compared, If it is determined that a predetermined threshold has been exceeded, a warning will be sent to the user interface connected to the controller. Methods that include...

9. The method according to claim 8, further comprising passing the first set of processed image data and the second set of processed image data to a gate mechanism of a long-short-term memory neural network.

10. The method according to claim 8, wherein processing the first image data stream includes performing object contour processing such that the reconstructed contour is defined by the components of the polishing apparatus.

11. The method according to claim 8, wherein processing the first image data stream includes performing image segmentation using a line segment-based approach.

12. The method according to claim 8, wherein the predetermined threshold corresponds to abnormal environmental conditions within the polishing apparatus.

13. The method according to claim 8, wherein the predetermined threshold corresponds to excessive vibration of the components of the polishing apparatus.

14. The method according to claim 8, wherein processing the first image data stream includes using cross-classification clustering to track related objects in the polishing apparatus.

15. A method for monitoring a polishing device, The polishing apparatus receives a first audio data stream from a camera positioned in the polishing apparatus, Using the audio analysis processor of the controller connected to the polishing apparatus, the first audio data stream is processed to generate a first set of processed audio data. Receiving a second audio data stream from the aforementioned camera, The audio analysis processor is used to process the second audio data stream to generate a second set of processed audio data, In order to determine whether a predetermined threshold has been exceeded, the second set of processed audio data is compared with the first set of processed audio data, If it is determined that a predetermined threshold has been exceeded, a warning will be sent to the user interface connected to the controller. Methods that include...

16. The method according to claim 15, further comprising passing the first set of processed audio data and the second set of processed audio data to a gate mechanism of a long-short-term memory neural network.

17. The method according to claim 15, wherein processing the first audio data stream includes performing a Fast Fourier Transform analysis.

18. The method according to claim 15, wherein processing the first audio data stream includes performing harmonic percussive source separation, and the first set of processed audio data includes harmonic spectrometer data, percussive spectrometer data, or both.

19. The method according to claim 15, wherein the first set of audio data corresponds to the state of consumables in the polishing apparatus.

20. The method according to claim 15, wherein the predetermined threshold corresponds to a failure of a tool in the polishing apparatus.

Images