Operating state monitoring method and device of die bonder and storage medium

Abstract

The invention discloses an operation state monitoring method and device of a die bonder and a storage medium. The method comprises the following steps: acquiring a bonding head image of each bonding head in the die bonder, a vibration signal when the bonding head works and pressure data; inputting the bonding head image, the vibration signal and the pressure data into a bonding head defect detection model, so that the bonding head defect detection model extracts image features of the bonding head according to the bonding head image, extracts vibration features of the bonding head during working according to the vibration signal, and extracts pressure features of the bonding head during working according to the pressure data; carrying out feature fusion on the image features, the vibration features and the pressure features, and determining the defect condition of the bonding head according to the fused features; if the defect head is monitored, early warning information is generated, so that a worker closes the corresponding head according to the early warning information; and when it is monitored that the bonding head in the closed state exists, feeding back state information that the bonding head is closed. According to the invention, the convenience of monitoring the operation state of the die bonder can be improved.

Description

Technical Field

[0001] The present invention relates to the field of die bonder status monitoring, and particularly to a method, device, and storage medium for monitoring the operating status of a die bonder. Background Art

[0002] During the production process of Mini LED displays, a die bonder has 6 bonding heads. When one or several of these bonding heads malfunction, manual shutdown or replacement is required, and at this time, the die bonder can still operate. In the existing daily maintenance process, it is necessary for staff to manually determine whether there are any faulty bonding heads. When a faulty bonding head is identified, the corresponding bonding head is shut down, and the number of bonding heads in the on state and off state of each die bonder is manually counted to determine the current operating status of each die bonder.

[0003] That is, in the prior art, the working status of the bonding heads in each die bonder needs to be checked and recorded one by one by the staff to master the working status of the entire die bonder, and the operation is cumbersome. Summary of the Invention

[0004] Embodiments of the present invention provide a method, device, and storage medium for monitoring the operating status of a die bonder, which can automatically identify defective bonding heads and provide feedback on the working status of each bonding head in the die bonder, improving the convenience of monitoring the operating status of the die bonder.

[0005] An embodiment of the present invention provides a method for monitoring the operating status of a die bonder, including:

[0006] Obtain the bonding head images, vibration signals during the operation of the bonding heads, and pressure data of each bonding head in the die bonder;

[0007] For each bonding head image, input the bonding head image, vibration signal, and pressure data into a preset bonding head defect detection model, so that the bonding head defect detection model extracts the image features of the bonding head according to the bonding head image, extracts the vibration features during the operation of the bonding head according to the vibration signal, extracts the pressure features during the operation of the bonding head according to the pressure data, performs feature fusion on the image features, vibration features, and pressure features to obtain fusion features, and determines the defect situation of the bonding head according to the fusion features;

[0008] If a defective bonding head is detected, generate a warning message so that the staff can shut down the corresponding bonding head according to the warning message;

[0009] When a bonding head that will be in the off state is detected, feedback the state information that the bonding head has been shut down.

[0010] Further, the bonding head defect detection model includes an image feature processing module: The image features include: geometric features, texture features, and color features of the bonding head;

[0011] The bonding head defect detection model extracts the image features of the bonding head from the bonding head image, including:

[0012] Through the image feature processing module, the bonding head image is grayscaled to obtain a grayscale image;

[0013] The grayscale image is binarized to obtain a binary image;

[0014] According to the binary image, the contour of the bonding head is extracted, and the perimeter and area of the bonding head are calculated to obtain the geometric features;

[0015] Calculate the LBP feature of the bonding head image to obtain the texture feature;

[0016] Calculate the color histogram of the bonding head image, and statistically analyze the pixel value distribution of each color channel according to the color histogram to obtain the color feature.

[0017] Further, the defect detection model further includes: a vibration feature extraction module; The vibration features include: frequency features, amplitude features, and acceleration features;

[0018] The extraction of the vibration features during the operation of the bonding head from the vibration signal includes:

[0019] Through the vibration feature extraction module, the vibration signal is subjected to a fast Fourier transform to obtain the frequency spectrum of the vibration signal;

[0020] According to the main frequency components in the frequency spectrum, the frequency features are obtained;

[0021] Extract the maximum value, minimum value, and average value of the amplitude from the vibration signal to obtain the amplitude feature;

[0022] Differentiate the vibration signal to obtain an acceleration signal;

[0023] Extract the peak value and root mean square value of the acceleration to obtain the acceleration feature.

[0024] Further, the bonding head defect detection model further includes: a pressure feature processing module;

[0025] The extraction of the pressure features during the operation of the bonding head from the pressure data includes:

[0026] Through the pressure feature processing module, the maximum value, minimum value, average value, and fluctuation range of the pressure are extracted to obtain the pressure feature.

[0027] Further, the bonder head defect detection model further includes: a feature fusion module;

[0028] The step of performing feature fusion on the image features, vibration features, and pressure features to obtain fused features includes:

[0029] Through the feature fusion module, the geometric features, texture features, color features, frequency features, amplitude features, acceleration features, and pressure features are fused to obtain the fused features.

[0030] Further, the training of the bonder head defect detection model includes:

[0031] Collect image samples of normal bonder heads and bonder heads with various defects in the die bonder under different lighting conditions and angles; collect vibration signals when various bonder heads work to obtain vibration signal samples; collect pressure data when various bonder heads work to obtain pressure data samples;

[0032] Label each image sample to obtain a label for characterizing the category of the bonder head in the image sample;

[0033] Associate the vibration signal samples and the pressure data samples with the corresponding image samples;

[0034] Group the corresponding image samples, vibration signal samples, and pressure data samples into the same group of samples;

[0035] Input each group of samples into the neural network model for iterative training until the neural network model converges to obtain the bonder head defect detection model; wherein, in each iterative training, taking a group of samples as input to obtain the corresponding predicted category, comparing the predicted category with the label corresponding to this group of samples, calculating the loss function value according to the comparison, and adjusting the parameters of the neural network model according to the loss function value.

[0036] Further, before inputting each group of samples into the neural network model for iterative training, it further includes:

[0037] Randomly rotate, flip, or scale the image samples to obtain processed image samples;

[0038] Add preset Gaussian noise to the vibration signal samples and pressure data samples to obtain processed vibration signal samples and processed pressure data samples.

[0039] Further, it further includes: when it is detected that a bonder head in the closed state is restarted, feeding back the status information that the corresponding bonder head has been turned on.

[0040] Based on the above method item embodiments, the present invention correspondingly provides device item embodiments;

[0041] An embodiment of the present invention provides a device for monitoring the operating state of a die bonder, including: a data acquisition module, a defect detection module, a warning generation module, and a bonding head working state feedback module;

[0042] The data acquisition module is configured to acquire the bonding head images of each bonding head in the die bonder, the vibration signals during the operation of the bonding head, and the pressure data;

[0043] The defect detection module is configured to input, for each bonding head image, the bonding head image, the vibration signal, and the pressure data into a preset bonding head defect detection model, so that the bonding head defect detection model extracts the image features of the bonding head according to the bonding head image, extracts the vibration features during the operation of the bonding head according to the vibration signal, extracts the pressure features during the operation of the bonding head according to the pressure data, fuses the image features, the vibration features, and the pressure features to obtain fused features, and determines the defect condition of the bonding head according to the fused features;

[0044] The warning generation module is configured to generate a warning message when a bonding head with defects is detected, so that the staff can turn off the corresponding bonding head according to the warning message;

[0045] The bonding head working state feedback module is configured to feedback the state information that the bonding head has been turned off when a bonding head that will be in the off state is detected.

[0046] Based on the above method item embodiments, the present invention correspondingly provides a storage medium item embodiment;

[0047] An embodiment of the present invention provides a storage medium, which includes a stored computer program, wherein when the computer program runs, it controls the device where the storage medium is located to execute the method for monitoring the operating state of the die bonder according to any one of the present invention.

[0048] By implementing the embodiments of the present invention, the following beneficial effects are achieved:

[0049] An embodiment of the present invention provides a method, device, and storage medium for monitoring the operating state of a die bonder. The method includes collecting die head images, vibration signals during die head operation, and pressure data of each die head in the die bonder; inputting the collected data into a trained die head defect detection model, so that the die head defect detection model extracts image features of the die head from the die head images, extracts vibration features during die head operation from the vibration signals, extracts pressure features during die head operation from the pressure data, fuses the image features, vibration features, and pressure features to obtain fused features, and determines the defect condition of the die head according to the fused features; when a defective die head is detected, generating a warning message so that the staff can close the corresponding die head according to the warning message; when a die head that will be in the closed state is detected, feedback the state information that the die head has been closed. By implementing the present invention, the defects of the die head can be automatically identified, and corresponding warning messages can be generated. In addition, when the die head is closed, the information that the corresponding die head has been closed can be automatically fed back, greatly improving the convenience of monitoring the operating state of the die bonder. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 FIG. is a schematic flowchart of a method for monitoring the operating state of a die bonder according to an embodiment of the present invention.

[0051] Figure 2 FIG. is a schematic structural diagram of a device for monitoring the operating state of a die bonder according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0053] As Figure 1 shown, an embodiment of the present invention provides a method for monitoring the operating state of a die bonder, which at least includes the following steps:

[0054] Step S1: Obtain die head images, vibration signals during die head operation, and pressure data of each die head in the die bonder.

[0055] Specifically, a high-definition industrial camera can be used to photograph the die head of the die bonder to ensure that the appearance features of the die head can be comprehensively captured, and die head images of each die head in the die bonder can be obtained.

[0056] Meanwhile, install a vibration sensor near the bonding head to collect the vibration signal during the operation of the bonding head. The sampling frequency should be reasonably set according to the working frequency of the bonding head and the possible vibration frequency range. Generally, it is recommended not to be lower than 10 kHz to ensure that the vibration details can be accurately captured.

[0057] In addition, install a pressure sensor at the part where the bonding head contacts the workpiece to monitor the pressure change during the operation of the bonding head in real time and obtain the above pressure data. The sampling frequency is adjusted according to the working cycle of the bonding head and the speed of pressure change, and is usually set to 1 kHz - 5 kHz.

[0058] In other optional embodiments, to avoid the problem that the installation of the sensor may change the contact stiffness between the bonding head and the workpiece, resulting in uneven pressure distribution and affecting the die bonding quality (such as loose wire bonding, chip offset), a high-precision laser displacement sensor can be used to monitor the pressure. Specifically, use the laser beam of the high-precision laser displacement sensor to vertically irradiate the center point of the contact end face of the bonding head, and record the minute deformation (typical deformation range 0 - 50 μm) of the bonding head due to pressure during the die bonding process. The sampling frequency is set to 100 kHz, and the complete waveform data of each die bonding action is stored. Use the Savitzky-Golay filter to remove high-frequency noise (cut-off frequency 1 kHz) and retain the dynamic response characteristics of the bonding head. Then, based on the deformation distribution of the bonding head simulated by finite element analysis (ANSYS) in advance under different pressures, the mathematical relationship between pressure and deformation amount is fitted as: P = a * δ 2 + b * δ + c; calculate the pressure, where P is the pressure, δ is the deformation amount, and a, b, and c are coefficients calibrated in advance.

[0059] Step S2: For each bonding head image, input the bonding head image, vibration signal, and pressure data into a preset bonding head defect detection model, so that the bonding head defect detection model extracts the image features of the bonding head according to the bonding head image, extracts the vibration features during the operation of the bonding head according to the vibration signal, extracts the pressure features during the operation of the bonding head according to the pressure data, performs feature fusion on the image features, vibration features, and pressure features to obtain fusion features, and determines the defect situation of the bonding head according to the fusion features.

[0060] In a preferred embodiment, the training of the bonding head defect detection model includes:

[0061] Collect image samples of normal bonding heads and bonding heads with various defects in the die bonder under different lighting conditions and angles; collect vibration signals during the operation of various bonding heads to obtain vibration signal samples; collect pressure data during the operation of various bonding heads to obtain pressure data samples;

[0062] Label each image sample to obtain a label for characterizing the category of the bonding head in the image sample;

[0063] Associate the vibration signal samples and the pressure data samples with the corresponding image samples;

[0064] Classify the corresponding image samples, vibration signal samples, and pressure data samples into the same group of samples;

[0065] Input each group of samples into a neural network model for iterative training until the neural network model converges to obtain the bonder head defect detection model; wherein, during each iterative training, using a group of samples as the input to obtain the corresponding predicted category, comparing the predicted category with the label corresponding to this group of samples, calculating the loss function value based on the comparison, and adjusting the parameters of the neural network model according to the loss function value.

[0066] In a preferred embodiment, before inputting each group of samples into the neural network model for iterative training, it further includes:

[0067] Randomly rotate, flip, or scale the image samples to obtain processed image samples;

[0068] Add preset Gaussian noise to the vibration signal samples and pressure data samples to obtain processed vibration signal samples and processed pressure data samples.

[0069] The following details the structure and training process of the defect detection model:

[0070] First, collect data samples: Use a high-definition industrial camera to photograph the bonder head of the die bonder under different lighting conditions and angles to ensure that the appearance features of the bonder head can be comprehensively captured. Collect images of normal bonder heads and bonder heads of various defect types (such as wear, deformation, cracks, etc.). The number of images of each type should be as balanced as possible to avoid bias in the model, and finally obtain the above-mentioned image samples.

[0071] Immediately afterwards, install vibration sensors near various bonder heads (normal bonder heads, bonder heads with wear, deformed bonder heads, bonder heads with cracks, etc.) to collect the vibration signals during the operation of various bonder heads to obtain vibrations. Install pressure sensors at the contact parts between various bonder heads and workpieces to monitor the pressure changes during the operation of the bonder heads in real time to obtain pressure data samples.

[0072] Immediately afterwards, call an image annotation tool (such as LabelImg) to annotate the category (normal or defective) of each image sample. For defective images, further annotate the type and location of the defect, and then associate the vibration data samples and pressure data samples with the corresponding image data samples to form each sample group.

[0073] Preferably, for the image sample data, operations such as randomly rotating the image (±30°), flipping (horizontally and vertically), or scaling (0.8 - 1.2 times) can be performed on the image to obtain the processed image sample, so as to increase the diversity of the image.

[0074] Secondly, an appropriate amount of Gaussian noise is added to the vibration signal sample and the pressure data sample to simulate the noise interference in the actual working environment and enhance the robustness of the model.

[0075] Finally, the obtained data set is used for model training. During training, the cross - entropy loss function can be adopted. The cross - entropy loss function can measure the difference between the model prediction result and the true label, and the parameters of the model are optimized by minimizing the loss function.

[0076] Regarding the structure of the model:

[0077] In the present invention, a multi - branch neural network structure is adopted. The first branch is the image feature processing module, which is used to extract the geometric features, texture features, and color features of the image; the second branch is the vibration feature extraction module, which is used to extract the frequency features, amplitude features, and acceleration features of the amplitude signal; the third branch is the pressure feature extraction module, which is used to extract the pressure features of the pressure data; finally, the output features of the three branches are fused through the feature fusion module to obtain the fused features, and the classification of the bonding head is determined based on the fused features.

[0078] In a preferred embodiment, the image features include: the geometric features, texture features, and color features of the bonding head;

[0079] The bonding head defect detection model extracts the image features of the bonding head according to the bonding head image, including:

[0080] Through the image feature processing module, the bonding head image is grayscale - processed to obtain a grayscale image;

[0081] The grayscale image is binarized to obtain a binary image;

[0082] According to the binary image, the contour of the bonding head is extracted, and the perimeter and area of the bonding head are calculated to obtain the geometric features;

[0083] The LBP features of the bonding head image are calculated to obtain the texture features;

[0084] The color histogram of the bonding head image is calculated, and the pixel value distribution of each color channel is statistically analyzed according to the color histogram to obtain the color features.

[0085] The basic geometric parameters of a normal bonding head, such as area and perimeter, are usually within a certain range. If there are defects such as wear, deformation or cracks on the bonding head, its perimeter and area may change. For example, wear at the tip of the bonding head will cause a decrease in its overall area, and the perimeter may also change accordingly. By calculating the perimeter and area, the dimensional information of the bonding head can be intuitively grasped as a whole, and compared with the geometric dimensions of a normal bonding head, so as to preliminarily judge whether there is any abnormality in the bonding head.

[0086] LBP is mainly used to describe the local texture information of an image. By comparing the gray values of the central pixel and the neighboring pixels, it generates a binary pattern, which can effectively reflect the local texture characteristics of the image. The defects on the surface of the bonding head are often local, such as tiny scratches and pockmarks. LBP can capture these changes in local texture and is very effective for detecting tiny defects. In addition, the color histogram statistically analyzes the pixel value distribution of each color channel in the image and can reflect the color characteristics of the bonding head. The color of the surface of the bonding head may change due to defects such as rust and oxidation. By analyzing the color histogram, these color changes can be discovered. For example, when the bonding head rusts, its surface color changes from the original metallic color to reddish-brown, and the pixel value distribution in the corresponding color channel of the color histogram will change significantly.

[0087] In a preferred embodiment, the vibration characteristics include: frequency characteristics, amplitude characteristics, and acceleration characteristics;

[0088] Extracting the vibration characteristics of the bonding head during operation according to the vibration signal includes:

[0089] Through the vibration characteristic extraction module, performing a fast Fourier transform on the vibration signal to obtain the frequency spectrum of the vibration signal;

[0090] According to the main frequency components in the frequency spectrum, obtaining the frequency characteristics;

[0091] Extracting the maximum value, minimum value, and average value of the amplitude from the vibration signal to obtain the amplitude characteristics;

[0092] Performing a differential process on the vibration signal to obtain an acceleration signal;

[0093] Extracting the peak value and root mean square value of the acceleration to obtain the acceleration characteristics.

[0094] Specifically, different types of defects may cause abnormal vibrations of the bonding head at specific frequencies. For example, internal looseness of the bonding head may result in abnormal vibration frequencies in the low-frequency band, while wear of certain components may lead to increased vibrations in the high-frequency band. By performing a fast Fourier transform (FFT) on the vibration signal, the spectrum of the signal is obtained, and the main frequency components are extracted to obtain frequency characteristics for subsequent defect identification. The magnitude of the amplitude reflects the intensity of the vibration. When there are defects in the bonding head, the vibration intensity often changes. For example, wear or looseness will cause the vibration of the bonding head during operation to intensify and the amplitude to increase. By directly extracting statistical quantities such as the maximum value, minimum value, and average value of the amplitude from the vibration signal, the vibration intensity of the bonding head can be intuitively understood to determine whether it is in a normal operating state. Acceleration is the rate of change of velocity, and it can more sensitively reflect changes in the dynamic characteristics of the bonding head. During the operation of the bonding head, changes in acceleration can reflect its force condition and changes in the motion state. For example, when the bonding head is suddenly impacted or undergoes abnormal vibrations, the acceleration will increase rapidly. By performing a differential operation on the vibration signal to obtain the acceleration signal and extracting features such as the peak value and root mean square value of the acceleration, the dynamic changes of the bonding head can be captured more accurately, and potential defects can be detected in a timely manner.

[0095] In a preferred embodiment, extracting the pressure characteristics of the bonding head during operation according to the pressure data includes:

[0096] Through the pressure characteristic processing module, the maximum value, minimum value, average value, and fluctuation range of the pressure are extracted to obtain the pressure characteristics.

[0097] The maximum and minimum values of the pressure can reflect the contact state between the bonding head and the workpiece. An abnormally high maximum pressure value may indicate that the contact force between the bonding head and the workpiece is too large, posing a risk of damage; an abnormally low minimum pressure value may indicate poor contact, which may affect the quality of die bonding. By monitoring the extreme pressure values, abnormalities in the contact state between the bonding head and the workpiece can be detected in a timely manner. The average pressure reflects the average pressure level of the bonding head during a working cycle. Abnormal changes in the average value may be related to wear or improper adjustment of the bonding head. For example, after long-term use of the bonding head, its wear will cause changes in the contact pressure with the workpiece, and the average pressure value may increase or decrease. By monitoring the average pressure value, the wear condition of the bonding head can be detected in a timely manner. The pressure fluctuation range reflects the change situation of the pressure. An excessively large pressure fluctuation range may indicate that the working state of the bonding head is unstable and there are internal defects. For example, looseness or wear inside the bonding head may cause large fluctuations in pressure during operation. By monitoring the pressure fluctuation range, abnormalities in the working state of the bonding head can be detected in a timely manner.

[0098] In a preferred embodiment, the bonding head defect detection model further includes: a feature fusion module;

[0099] Performing feature fusion on the image features, vibration features, and pressure features to obtain fused features, including:

[0100] Fusing the geometric features, texture features, color features, frequency features, amplitude features, acceleration features, and pressure features through a feature fusion module to obtain the fused features.

[0101] Specifically, after obtaining the above various types of features, they are fused into a unified feature vector through a feature fusion module, and finally identification is performed. Through feature fusion, the information complementarity can be enhanced. Geometric features (such as perimeter, area, etc.) reflect the morphological structure of the bonding head, texture features reflect the surface micro characteristics, color features describe the material properties, and vibration and pressure features reveal the dynamic behavior. After fusion, it can cover multi-dimensional information such as static morphology and dynamic behavior, macroscopic structure and microscopic details, avoiding the one-sidedness of a single feature. Finally, the bonding head is automatically identified whether there are defects and the corresponding defect types when there are defects, thereby improving the convenience of the entire die bonder operation status monitoring.

[0102] Step S3: If a bonding head with defects is detected, generate a warning message so that the staff can close the corresponding bonding head according to the warning message.

[0103] Specifically, if a bonding head with defects is monitored, the above warning message is generated according to the identification of the bonding head and the defect type, and then reported to the MES (Manufacturing Execution System, usually referring to a production information management system for the workshop execution layer of manufacturing enterprises). Through the warning message received by the staff through the MES, they can know which bonding head has a problem, and then close the corresponding bonding head. By implementing this embodiment, the staff does not need to open the instrument one by one to check the status of each bonding head, and can directly know the bonding head that needs to be closed, further improving the convenience of die bonder monitoring.

[0104] Step S4: When a bonding head that will be in the closed state is monitored, feedback the state information that the bonding head has been closed.

[0105] Specifically, after the staff closes the bonding head with defects, the above state information is generated by combining the identification of the closed bonding head and the current working state (closed state), and the state information is fed back to the MES.

[0106] In a preferred embodiment, when a bonding head in the closed state is detected and restarted, feedback the state information that the corresponding bonding head has been opened;

[0107] Specifically, if the subsequent staff repairs the defective bonding head and then restarts the bonding head, at this time, the status information is generated by combining the identifier of the restarted bonding head and the current working status (on state), and is fed back to the MES system.

[0108] In a preferred embodiment, for the bonding heads without defects, the status information of being turned on is generated by combining the identifiers of the bonding heads and fed back to the MES system.

[0109] By implementing the above embodiments of the present invention, the staff can view the working conditions of each bonding head in all die bonders through the MES system, without the need to check the bonding heads of each die bonder on site, improving the convenience of monitoring the operating status of the die bonder.

[0110] Based on the above method item embodiments, the present invention correspondingly provides device item embodiments;

[0111] An embodiment of the present invention provides an operating status monitoring device for a die bonder, including: a data acquisition module, a defect detection module, a warning generation module, and a bonding head working status feedback module;

[0112] The data acquisition module is configured to acquire the bonding head images, vibration signals during the operation of the bonding head, and pressure data of each bonding head in the die bonder;

[0113] The defect detection module is configured to, for each bonding head image, input the bonding head image, vibration signal, and pressure data into a preset bonding head defect detection model, so that the bonding head defect detection model extracts the image features of the bonding head according to the bonding head image, extracts the vibration features during the operation of the bonding head according to the vibration signal, extracts the pressure features during the operation of the bonding head according to the pressure data, performs feature fusion on the image features, vibration features, and pressure features to obtain fusion features, and determines the defect condition of the bonding head according to the fusion features;

[0114] The warning generation module is configured to generate a warning message when a defective bonding head is detected, so that the staff can turn off the corresponding bonding head according to the warning message;

[0115] The bonding head working status feedback module is configured to, when it is detected that a bonding head to be in a closed state, feedback the status information that the bonding head has been closed.

[0116] It should be noted that the device embodiments described above correspond to the method embodiments of the present invention, and can implement the method for monitoring the operating state of the die bonder described in any of the above method embodiments of the present invention. In addition, the devices described above in the present invention are merely illustrative, where the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Additionally, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationships between the modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those of ordinary skill in the art can understand and implement it without creative work.

[0117] Based on the above method embodiments, the present invention correspondingly provides a storage medium embodiment.

[0118] An embodiment of the present invention provides a storage medium. The storage medium includes a stored computer program, wherein when the computer program runs, it controls the device where the storage medium is located to execute the method for monitoring the operating state of the die bonder described in any one of the present invention.

[0119] The storage medium is a computer-readable medium, which may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0120] The above is the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can still be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A method for monitoring the operating state of a die bonder, characterized in that, Including: Obtaining the head images of each bonding head in the die bonder, the vibration signals during the operation of the bonding head, and the pressure data; For each bonding head image, inputting the bonding head image, vibration signal, and pressure data into a preset bonding head defect detection model, so that the bonding head defect detection model extracts the image features of the bonding head from the bonding head image, extracts the vibration features during the operation of the bonding head from the vibration signal, extracts the pressure features during the operation of the bonding head from the pressure data, performs feature fusion on the image features, vibration features, and pressure features to obtain fusion features, and determines the defect situation of the bonding head according to the fusion features; If a bonding head with defects is detected, generating a warning message so that the staff can close the corresponding bonding head according to the warning message; When a bonding head in the closed state is detected, feeding back the status information that the bonding head has been closed.

2. The method for monitoring the operating state of the die bonder according to claim 1, wherein, The bonding head defect detection model includes an image feature processing module: The image features include: geometric features, texture features, and color features of the bonding head; The bonding head defect detection model extracting the image features of the bonding head from the bonding head image includes: Through the image feature processing module, performing grayscale processing on the bonding head image to obtain a grayscale image; Performing image binarization on the grayscale image to obtain a binary image; Extracting the contour of the bonding head according to the binary image, and calculating the perimeter and area of the bonding head to obtain the geometric features; Calculating the LBP features of the bonding head image to obtain the texture features; Calculating the color histogram of the bonding head image, and statistically analyzing the pixel value distribution of each color channel according to the color histogram to obtain the color features.

3. The method for monitoring the operating state of the die bonder according to claim 2, wherein, The defect detection model further includes: a vibration feature extraction module; The vibration features include: frequency features, amplitude features, and acceleration features; The extracting the vibration features during the operation of the bonding head from the vibration signal includes: Through the vibration feature extraction module, performing a fast Fourier transform on the vibration signal to obtain the frequency spectrum of the vibration signal; Obtaining the frequency features according to the main frequency components in the frequency spectrum; Extracting the maximum value, minimum value, and average value of the amplitude from the vibration signal to obtain the amplitude features; Performing differential processing on the vibration signal to obtain an acceleration signal; Extracting the peak value and root mean square value of the acceleration to obtain the acceleration features.

4. The method for monitoring the operating state of a die bonder according to claim 3, characterized in that, The bonding head defect detection model further includes: a pressure feature processing module; The extracting the pressure features during the operation of the bonding head from the pressure data includes: Through the pressure feature processing module, extracting the maximum value, minimum value, average value, and fluctuation range of the pressure to obtain the pressure features.

5. The method for monitoring the operating state of the die bonder according to claim 4, wherein The bonding head defect detection model further includes: a feature fusion module; The performing feature fusion on the image features, vibration features, and pressure features to obtain fusion features includes: Through the feature fusion module, fusing the geometric features, texture features, color features, frequency features, amplitude features, acceleration features, and pressure features to obtain the fusion features.

6. The method for monitoring the operating state of the die bonder according to claim 5, characterized in that, The training of the bonding head defect detection model includes: Collect image samples of normal bond heads and bond heads with various defects in the die bonder under different lighting conditions and angles; collect vibration signals when various bond heads are working to obtain vibration signal samples; collect pressure data when various bond heads are working to obtain pressure data samples; Annotate each image sample to obtain a label for characterizing the category of the bond head in the image sample; Associate the vibration signal samples and the pressure data samples with the corresponding image samples; Divide the corresponding image samples, vibration signal samples, and pressure data samples into the same group of samples; Input each group of samples into a neural network model for iterative training until the neural network model converges to obtain the bond head defect detection model; wherein, during each iterative training, take a group of samples as input to obtain the corresponding predicted category, compare the predicted category with the label corresponding to this group of samples, calculate the loss function value according to the comparison, and adjust the parameters of the neural network model according to the loss function value.

7. The method for monitoring the operating state of the die bonder according to claim 6, characterized in that, Before inputting each group of samples into the neural network model for iterative training, it further includes: Randomly rotate, flip, or scale the image samples to obtain processed image samples; Add preset Gaussian noise to the vibration signal samples and the pressure data samples to obtain processed vibration signal samples and processed pressure data samples.

8. The method for monitoring the operating state of the die bonder according to claim 7, wherein, It further includes: When it is detected that a bond head in the closed state is restarted, feedback the state information that the corresponding bond head has been turned on.

9. An operating state monitoring device for a die bonder, characterized in that, It includes: A data acquisition module, a defect detection module, a warning generation module, and a bond head working state feedback module; The data acquisition module is used to acquire bond head images, vibration signals when the bond heads are working, and pressure data of each bond head in the die bonder; The defect detection module is used to, for each bond head image, input the bond head image, vibration signal, and pressure data into a preset bond head defect detection model, so that the bond head defect detection model extracts the image features of the bond head according to the bond head image, extracts the vibration features when the bond head is working according to the vibration signal, extracts the pressure features when the bond head is working according to the pressure data, fuse the image features, vibration features, and pressure features to obtain fused features, and determine the defect situation of the bond head according to the fused features; The warning generation module is used to generate a warning message when a defective bond head is detected, so that the staff can turn off the corresponding bond head according to the warning message; The bond head working state feedback module is used to feedback the state information that the bond head has been turned off when it is detected that a bond head in the closed state exists.

10. A storage medium, characterized in that, The storage medium includes a stored computer program, wherein, when the computer program runs, it controls the device where the storage medium is located to execute the method for monitoring the operating state of the die bonder according to any one of claims 1 to 8.

Images