Machine vision-based real-time monitoring and early warning method and system for semiconductor injection molding punch line

Abstract

The present application relates to the technical field of semiconductor, in particular to a real-time monitoring and early warning method and system for semiconductor injection molding line based on machine vision, comprising the following steps: S1, cavity image acquisition system building; S2, image preprocessing and ROI extraction; S3, feature extraction based on lightweight deep learning model: the lightweight convolutional neural network deployed on the edge computing device carries out real-time inference on the preprocessed image, and outputs the following key feature vectors: EMC front flow rate V, flow form feature F, gold line area stress distribution prediction S; S4, multi-level line risk assessment and grading early warning: based on the feature vector output by the deep learning model, a multi-level threshold judgment mechanism is constructed to realize grading response; S5, process parameter closed loop adjustment. Compared with the prior art, early warning, reduce the batch of waste products; real-time closed loop control, adaptive adjustment; edge computing, low delay; has the ability of continuous learning.

Description

Technical Field

[0001] This invention relates to the field of semiconductor technology, specifically to a method and system for real-time monitoring and early warning of semiconductor injection molding line runs based on machine vision. Background Technology

[0002] T-mold, or transfer molding, is one of the mainstream processes in semiconductor packaging. In this process, molten epoxy resin (EMC) is injected into the mold cavity through a runner driven by an injection molding rod, encapsulating the wire-bonded chip. However, during injection molding, the high-speed flowing resin exerts a fluid drag force on the gold wires, which are only 15-50 μm in diameter, causing the wires to shift or even break—a problem known as wire sweep. Traditional solutions to the wire sweep problem mainly focus on the following aspects: 1. Process parameter optimization: Multi-stage low-speed injection is adopted to reduce the flow rate of resin when it flows through the gold wire area; 2. Mold structure improvement: The gate position and shape are optimized to avoid the resin directly impacting the dense gold wire area; 3. Material modification: Special EMC materials with low viscosity, spherical filler and long gel time are selected.

[0003] However, the above solutions are all offline, open-loop control methods, which have the following inherent drawbacks: 1. Inability to detect thread breakage risk in real time: Traditional methods rely on experience-based settings of process parameters and cannot detect the real-time stress state of the gold thread during actual injection molding. 2. Response lag: Even if thread breakage occurs, it can only be detected during the inspection stage after packaging, resulting in batch scrap. 3. Lack of closed-loop control: When material batches fluctuate or mold temperatures change, the preset parameters cannot be adaptively adjusted. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, this invention provides a method and system for real-time monitoring and early warning of die-off in semiconductor injection molding based on machine vision. It can monitor the EMC flow pattern in real time during T-mold injection molding, provide early warning of die-off risks, and automatically adjust process parameters to solve the technical problems of die-off risks not being perceived in real time, delayed response, and lack of closed-loop control in the prior art.

[0005] To achieve the above objectives, a machine vision-based real-time monitoring and early warning method for semiconductor injection molding line punching is designed, including the following steps: S1, Construction of in-mold image acquisition system: A miniature high-speed camera and a high-temperature resistant optical window are integrated into the transfer molding mold to capture the flow pattern of EMC resin before it flows through the gold wire area in real time. S2, Image Preprocessing and ROI Extraction: Preprocess the acquired raw image, including: Denoising: Use median filtering or Gaussian filtering to eliminate image noise; ROI Extraction: Based on preset template matching or edge detection algorithms, automatically identify and crop out the gold line region and EMC flow front region. S3, Feature extraction based on a lightweight deep learning model: A lightweight convolutional neural network deployed on an edge computing device performs real-time inference on the preprocessed image and outputs the following key feature vectors: EMC front velocity V, flow morphology features F, and stress distribution prediction in the gold line region S. S4, Multi-level Finish Line Risk Assessment and Graded Early Warning: Based on the feature vector output by the deep learning model, a multi-level threshold determination mechanism is constructed to achieve graded response; S5, Closed-loop adjustment of process parameters: When a high-risk warning is triggered, an instruction is sent to the injection molding machine controller via the industrial bus to reduce the set speed of the current injection segment; the upper limit of the transfer pressure is reduced simultaneously; and finally, the abnormal event is stored in the database for incremental training or federated learning of subsequent models to continuously optimize the warning model.

[0006] In step S1, the miniature high-speed camera and the high-temperature resistant optical window are positioned inside the transfer molding mold at positions corresponding to the downstream of the gate and the upstream of the gold wire area, ensuring that the field of view covers the gold wire area.

[0007] In step S1, the miniature high-speed camera uses a miniaturized CMOS image sensor with a resolution of at least 640×480 pixels and a frame rate of at least 500fps; the high-temperature resistant optical window is made of sapphire glass and has a surface coating that can withstand temperatures of ≥200℃ and resist EMC adhesion.

[0008] In step S3, a lightweight backbone network is constructed using depthwise separable convolutions, including an input layer, a preprocessing layer, a Stem layer, three levels of depthwise separable convolutional blocks, a multi-scale feature fusion module, a multi-task prediction branch, and a feature concatenation output layer, wherein: The ROI image obtained in step S2 is input to the input layer and sent as input data to the downstream preprocessing layer. The preprocessing layer receives the ROI image output from the input layer, performs pixel normalization and Gaussian filtering noise reduction on the image, and outputs the preprocessed image to the Stem layer. The Stem layer receives the image output from the preprocessing layer, performs basic convolution operations using a 3×3 standard convolution kernel, and outputs an initial feature map with 32 channels. After the convolution operation, a batch normalization layer (BN) and a ReLU activation function layer are sequentially connected to complete the initial extraction of basic features at the bottom layer, and output the bottom feature map to the first depthwise separable convolution block. In the three-level depthwise separable convolutional block, the first depthwise separable convolutional block first performs a 3×3 depthwise convolution, then a 1×1 pointwise convolution, and connects the residual structure; the second depthwise separable convolutional block first performs a 3×3 depthwise convolution, then a 1×1 pointwise convolution, and connects the residual structure; the third depthwise separable convolutional block first performs a 3×3 depthwise convolution, then a 1×1 pointwise convolution, performs downsampling, and feeds the multi-scale basic features into the multi-scale feature fusion module. The multi-scale feature fusion module constructs a lightweight ASPP structure based on dilated convolution, performs dimensional alignment and fusion of features at different scales, outputs multi-scale fused features with a unified dimension, and branches them into three parallel task branches: the first task branch is the velocity regression branch, which completes the numerical regression prediction of fluid velocity and outputs the velocity feature vector; the second task is the flow state distribution prediction branch, which completes the prediction of fluid flow state distribution and outputs the flow state distribution feature vector; the third task is the stress prediction branch in the gold line region, which completes the stress distribution prediction and outputs the stress distribution feature vector. The output layer concatenates and fuses the three sets of feature vectors to generate a prediction result that includes the EMC front velocity V, flow morphology features F, and stress distribution prediction S in the gold line region.

[0009] The lightweight backbone network has fewer than 1M parameters to ensure real-time inference on edge devices with an inference latency of <10ms and ≥100 FPS; and the trained model is compressed through channel pruning and knowledge distillation.

[0010] In step S3, the EMC front velocity V is estimated by pixel-level optical flow, the flow morphology F is determined to be laminar or turbulent, and the stress distribution prediction S in the gold wire region is inferred based on image features.

[0011] In step S4, when the EMC leading-edge velocity V is close to the set lower threshold V_low and the flow pattern is laminar, it is determined to be a level three low risk and no intervention is taken; when the EMC leading-edge velocity V exceeds the set threshold midpoint V_mid, or turbulence precursors appear, it is determined to be a level two medium risk and the operator is prompted to pay attention; when the EMC leading-edge velocity V exceeds the set upper threshold V_high, or a gold line offset feature is detected, the parameter adjustment in step S5 is triggered and an alarm is triggered.

[0012] To achieve the above objectives, a machine vision-based real-time monitoring and early warning system for semiconductor injection molding line punching is designed to execute the above methods, including: Sensing layer: includes a miniature high-speed camera and a high-temperature resistant optical window integrated in the mold cavity of the transfer molding die, used to acquire images of EMC flow morphology in real time; Analysis layer: Includes an image preprocessing module and a lightweight deep learning model for extracting modulus flow features and risk assessment of the gold line region; Control layer: Based on the risk assessment structure, it triggers early warnings and automatically adjusts injection molding parameters.

[0013] Compared with the prior art, the present invention has the following advantages: 1. Early warning to reduce batch scrap: Early warning and intervention before or at the early stage of the defect can prevent the defect from spreading to the entire batch of products, which is expected to reduce the scrap rate caused by the defect by more than 60%.

[0014] 2. Real-time closed-loop control and adaptive adjustment: Abandoning the open-loop mode of empirical parameters, it realizes dynamic parameter adjustment based on the actual flow state.

[0015] 3. Edge computing, low latency: The lightweight model is deployed directly on the device, with inference latency of <10ms, meeting real-time control requirements.

[0016] 4. Continuous learning capability: Through the federated learning mechanism, multiple devices can collaboratively optimize the model and continuously adapt to new materials and processes. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the process of the present invention.

[0018] Figure 2 This is a schematic diagram showing the camera installation position inside the mold cavity of the transfer molding mold of the present invention.

[0019] Figure 3 This is a schematic diagram of the network structure of the lightweight deep learning model in step S3 of the present invention.

[0020] Figure 4 This is a flowchart illustrating the multi-level threshold determination and graded response in step S4 of the present invention.

[0021] Figure 5 This is a system architecture diagram of the present invention.

[0022] See Figures 1 to 5 Among them, 1 is the positioning screw, 2 is the injection port and flow channel, 3 is the miniature high-speed camera, 4 is the return pin, and 5 is the mold surface. Detailed Implementation

[0023] The present invention will now be further described with reference to the accompanying drawings.

[0024] like Figures 1 to 4 As shown in this embodiment, the real-time monitoring and early warning method for semiconductor injection molding line punching based on machine vision includes the following steps: S1, Construction of in-mold image acquisition system: A miniature high-speed camera and a high-temperature resistant optical window are integrated into the transfer molding mold to capture the flow pattern of EMC resin before it flows through the gold wire area in real time. S2, Image Preprocessing and ROI Extraction: Preprocess the acquired raw image, including: Denoising: Use median filtering or Gaussian filtering to eliminate image noise; ROI Extraction: Based on preset template matching or edge detection algorithms, automatically identify and crop out the gold line region and EMC flow front region. S3, Feature extraction based on a lightweight deep learning model: A lightweight convolutional neural network deployed on edge computing devices, such as industrial PCs or embedded AI modules, performs real-time inference on the preprocessed image and outputs the following key feature vectors: EMC front velocity V, flow morphology features F, and stress distribution prediction in the gold wire region S. S4, Multi-level Finish Line Risk Assessment and Graded Early Warning: Based on the feature vector output by the deep learning model, a multi-level threshold determination mechanism is constructed to achieve graded response; S5, Closed-loop adjustment of process parameters: When a high-risk warning is triggered, an instruction is sent to the injection molding machine controller via an industrial bus, such as EtherCAT, to reduce the set speed of the current injection section, for example, by 10%-30%; the upper limit of the transfer pressure is reduced simultaneously to prevent excessive pressure compensation due to speed reduction; finally, the abnormal event, including the image sequence, flow rate curve, and whether a run-through occurred, is stored in the database for incremental training or federated learning of subsequent models to continuously optimize the warning model.

[0025] In step S1, the miniature high-speed camera and the high-temperature resistant optical window are positioned inside the transfer molding mold at positions corresponding to the downstream of the gate and the upstream of the gold wire area, ensuring that the field of view covers the gold wire area, such as the long-span gold wire area and the high-density gold wire area.

[0026] In step S1, the miniature high-speed camera uses a miniaturized CMOS image sensor with a resolution of at least 640×480 pixels and a frame rate of at least 500fps to capture high-speed flow details at the EMC front; the high-temperature resistant optical window is made of sapphire glass and has a surface coating that can withstand temperatures of ≥200℃ and resist EMC adhesion.

[0027] In step S3, a lightweight backbone network is constructed using depthwise separable convolutions, including an input layer, a preprocessing layer, a Stem layer, three levels of depthwise separable convolutional blocks, a multi-scale feature fusion module, a multi-task prediction branch, and a feature concatenation output layer, wherein: The ROI image obtained in step S2 is input to the input layer and sent as input data to the downstream preprocessing layer. The preprocessing layer receives the ROI image output from the input layer, performs pixel normalization and Gaussian filtering noise reduction on the image, unifies the pixel value range by normalization, filters out random noise introduced during image acquisition by Gaussian filtering, eliminates irrelevant interference factors such as lighting and clutter, and outputs the preprocessed image to the Stem layer. The Stem layer receives the output image from the preprocessing layer, performs basic convolution operations using a 3×3 standard convolution kernel, and outputs an initial feature map with 32 channels. After the convolution operation, a batch normalization layer (BN) and a ReLU activation function layer are sequentially connected to complete the initial extraction of basic features such as edges and textures of the fluid image, and outputs the low-level feature map to the first depth-separable convolution block. In the three-level depthwise separable convolutional block, the first depthwise separable convolutional block first performs a 3×3 depthwise convolution, then a 1×1 pointwise convolution, and connects to a residual structure. The residual structure can retain the original feature information of the shallow layer, alleviate the gradient vanishing problem of the deep network, and output fine-grained local fluid features to the second depthwise separable convolutional block. The second depthwise separable convolutional block first performs a 3×3 depthwise convolution, then a 1×1 pointwise convolution, and connects to a residual structure to further deepen feature extraction and output mid-layer fused features to the third depthwise separable convolutional block. The third depthwise separable convolutional block first performs a 3×3 depthwise convolution, then a 1×1 pointwise convolution, and performs a downsampling operation. By downsampling, the spatial resolution of the feature map is compressed, the receptive field of the network is expanded, global coarse-grained flow features are extracted, and multi-scale basic features are sent to the multi-scale feature fusion module. The multi-scale feature fusion module constructs a lightweight ASPP structure based on dilated convolution, performs dimensional alignment and fusion of features at different scales, outputs multi-scale fused features with a unified dimension, and branches them into three parallel task branches: the first task branch is the velocity regression branch, which completes the numerical regression prediction of fluid velocity and outputs the velocity feature vector; the second task is the flow state distribution prediction branch, which completes the prediction of fluid flow state distribution and outputs the flow state distribution feature vector; the third task is the stress prediction branch in the gold line region, which completes the stress distribution prediction and outputs the stress distribution feature vector. The output layer concatenates and fuses the three sets of feature vectors to generate prediction results that include EMC front velocity V, flow morphology features F, and stress distribution prediction S in the gold line region.

[0028] A lightweight backbone network with fewer than 1M parameters ensures real-time inference on edge devices with an inference latency of <10ms and ≥100 FPS. The trained model is compressed through channel pruning and knowledge distillation to reduce computation while maintaining accuracy, achieving a finish line warning accuracy of ≥95%.

[0029] In step S3, the flow velocity V at the EMC front is estimated by pixel-level optical flow, the flow morphology F is determined to be laminar or turbulent, and the stress distribution prediction S in the gold wire region is inferred based on image features.

[0030] In step S4, when the EMC leading-edge velocity V is close to the set lower threshold V_low and the flow pattern is laminar, it is judged as a level three low risk and no intervention is taken; when the EMC leading-edge velocity V exceeds the set threshold midpoint V_mid, or turbulence precursors appear, it is judged as a level two medium risk and the operator is prompted to pay attention; when the EMC leading-edge velocity V exceeds the set upper threshold V_high, or the gold line deviation is detected, the parameter adjustment in step S5 is triggered and an alarm is triggered.

[0031] In practical use, three flow rate thresholds, V_low, V_mid, and V_high, are set based on historical lap data and actual production conditions.

[0032] like Figure 5 As shown, a machine vision-based real-time monitoring and early warning system for semiconductor injection molding line punching, used to perform the above method, includes: Sensing layer: includes a miniature high-speed camera and a high-temperature resistant optical window integrated in the mold cavity of the transfer molding die, used to acquire images of EMC flow morphology in real time; Analysis layer: Includes an image preprocessing module and a lightweight deep learning model for extracting modulus flow features and risk assessment of the gold line region; Control layer: Based on the risk assessment structure, it triggers early warnings and automatically adjusts injection molding parameters.

[0033] This invention provides real-time visual monitoring of the EMC flow pattern within the mold cavity of T-mold molds, achieving transparent perception of the mold flow pattern in the gold line area during T-mold injection molding, filling a technological gap. By deploying a lightweight deep learning model at the edge, it solves the real-time inference problem under limited computing power conditions, with model parameters <1M and inference speed ≥100FPS. A multi-level threshold-based early warning and response mechanism is established, realizing a tiered response from data recording → audible and visual warning → automatic parameter adjustment, balancing safety and production efficiency. Through closed-loop automatic adjustment of injection molding parameters, it overcomes the limitations of traditional open-loop control, achieving an intelligent closed loop of perception-decision-execution. Based on federated learning, the model is continuously optimized, with cross-device data collaborative training, continuously improving early warning accuracy while protecting data privacy.

[0034] This invention integrates a high-speed imaging system into a T-mold mold, uses a lightweight deep learning model to monitor and analyze the mold flow morphology of EMC flowing through the gold wire region in real time, extracts the characteristics of the thread breakage risk, and realizes graded early warning and closed-loop adjustment of process parameters based on a multi-level threshold mechanism.

Claims

1. A machine vision-based method for real-time monitoring and early warning of semiconductor injection molding line runs, characterized in that: Includes the following steps: S1, Construction of in-mold image acquisition system: A miniature high-speed camera and a high-temperature resistant optical window are integrated into the transfer molding mold to capture the flow pattern of EMC resin before it flows through the gold wire area in real time. S2, Image Preprocessing and ROI Extraction: Preprocess the acquired raw image, including: Denoising: Use median filtering or Gaussian filtering to eliminate image noise; ROI Extraction: Based on preset template matching or edge detection algorithms, automatically identify and crop out the gold line region and EMC flow front region. S3, Feature extraction based on a lightweight deep learning model: A lightweight convolutional neural network deployed on an edge computing device performs real-time inference on the preprocessed image and outputs the following key feature vectors: EMC front velocity V, flow morphology features F, and stress distribution prediction in the gold line region S. S4, Multi-level Finish Line Risk Assessment and Graded Early Warning: Based on the feature vector output by the deep learning model, a multi-level threshold determination mechanism is constructed to achieve graded response; S5, Closed-loop adjustment of process parameters: When a high-risk warning is triggered, an instruction is sent to the injection molding machine controller via the industrial bus to reduce the set speed of the current injection segment; the upper limit of the transfer pressure is reduced simultaneously; and finally, the abnormal event is stored in the database for incremental training or federated learning of subsequent models to continuously optimize the warning model.

2. The method for real-time monitoring and early warning of semiconductor injection molding line punching based on machine vision according to claim 1, characterized in that: In step S1, the miniature high-speed camera and the high-temperature resistant optical window are positioned inside the transfer molding mold at positions corresponding to the downstream of the gate and the upstream of the gold wire area, ensuring that the field of view covers the gold wire area.

3. The method for real-time monitoring and early warning of semiconductor injection molding line punching based on machine vision according to claim 1, characterized in that: In step S1, the miniature high-speed camera uses a miniaturized CMOS image sensor with a resolution of at least 640×480 pixels and a frame rate of at least 500fps; the high-temperature resistant optical window is made of sapphire glass and has a surface coating that can withstand temperatures of ≥200℃ and resist EMC adhesion.

4. The method for real-time monitoring and early warning of semiconductor injection molding line punching based on machine vision according to claim 1, characterized in that: In step S3, a lightweight backbone network is constructed using depthwise separable convolutions, including an input layer, a preprocessing layer, a Stem layer, three levels of depthwise separable convolutional blocks, a multi-scale feature fusion module, a multi-task prediction branch, and a feature concatenation output layer, wherein: The ROI image obtained in step S2 is input to the input layer and sent as input data to the downstream preprocessing layer. The preprocessing layer receives the ROI image output from the input layer, performs pixel normalization and Gaussian filtering noise reduction on the image, and outputs the preprocessed image to the Stem layer. The Stem layer receives the image output from the preprocessing layer, performs basic convolution operations using a 3×3 standard convolution kernel, and outputs an initial feature map with 32 channels. After the convolution operation, a batch normalization layer (BN) and a ReLU activation function layer are sequentially connected to complete the initial extraction of basic features at the bottom layer, and output the bottom feature map to the first depthwise separable convolution block. In the three-level depthwise separable convolutional block, the first depthwise separable convolutional block first performs a 3×3 depthwise convolution, then a 1×1 pointwise convolution, and connects the residual structure; the second depthwise separable convolutional block first performs a 3×3 depthwise convolution, then a 1×1 pointwise convolution, and connects the residual structure; the third depthwise separable convolutional block first performs a 3×3 depthwise convolution, then a 1×1 pointwise convolution, performs downsampling, and feeds the multi-scale basic features into the multi-scale feature fusion module. The multi-scale feature fusion module constructs a lightweight ASPP structure based on dilated convolution, performs dimensional alignment and fusion of features at different scales, outputs multi-scale fused features with a unified dimension, and branches them into three parallel task branches: the first task branch is the velocity regression branch, which completes the numerical regression prediction of fluid velocity and outputs the velocity feature vector; the second task is the flow state distribution prediction branch, which completes the prediction of fluid flow state distribution and outputs the flow state distribution feature vector; the third task is the stress prediction branch in the gold line region, which completes the stress distribution prediction and outputs the stress distribution feature vector. The output layer concatenates and fuses the three sets of feature vectors to generate a prediction result that includes the EMC front velocity V, flow morphology features F, and stress distribution prediction S in the gold line region.

5. The real-time monitoring and early warning method for semiconductor injection molding line punching based on machine vision according to claim 4, characterized in that: The lightweight backbone network has fewer than 1M parameters to ensure real-time inference on edge devices with an inference latency of <10ms and ≥100 FPS; and the trained model is compressed through channel pruning and knowledge distillation.

6. The real-time monitoring and early warning method for semiconductor injection molding line punching based on machine vision according to claim 1 or 4, characterized in that: In step S3, the EMC front velocity V is estimated by pixel-level optical flow, the flow morphology F is determined to be laminar or turbulent, and the stress distribution prediction S in the gold wire region is inferred based on image features.

7. The method for real-time monitoring and early warning of semiconductor injection molding line punching based on machine vision according to claim 1, characterized in that: In step S4, when the EMC leading-edge velocity V is close to the set lower threshold V_low and the flow pattern is laminar, it is determined to be a level three low risk and no intervention is taken; when the EMC leading-edge velocity V exceeds the set threshold midpoint V_mid, or turbulence precursors appear, it is determined to be a level two medium risk and the operator is prompted to pay attention; when the EMC leading-edge velocity V exceeds the set upper threshold V_high, or a gold line offset feature is detected, the parameter adjustment in step S5 is triggered and an alarm is triggered.

8. A machine vision-based real-time monitoring and early warning system for semiconductor injection molding thread breakage, used to execute the machine vision-based real-time monitoring and early warning method for semiconductor injection molding thread breakage as described in any one of claims 1-7, characterized in that: include: Sensing layer: includes a miniature high-speed camera and a high-temperature resistant optical window integrated in the mold cavity of the transfer molding die, used to acquire images of EMC flow morphology in real time; Analysis layer: Includes an image preprocessing module and a lightweight deep learning model for extracting modulus flow features and risk assessment of the gold line region; Control layer: Based on the risk assessment structure, it triggers early warnings and automatically adjusts injection molding parameters.

Images