A method, system, terminal, and storage medium for detecting and sorting defects in chip carrier tapes.

Abstract

This invention relates to the field of chip inspection, and discloses a method, system, terminal, and storage medium for chip carrier tape defect detection and sorting. The method includes: receiving voice commands, identifying chip models and loading corresponding detection parameters to control the carrier tape to advance at a constant speed; synchronously triggering a high-speed camera to acquire chip images based on carrier tape pitch pulses monitored by an encoder; generating a defect heatmap based on the image, fusing it with the image, and inputting it into a fine-tuned visual semantic understanding model to output defect semantic classification and confidence score; classifying and processing the chips according to the confidence score to obtain three categories of results: normal chips, high-confidence defective chips, and suspected defective chips; retesting the suspected defective chips and storing the results in a sample archive for incremental fine-tuning of the model; and generating a natural language detection report based on the semantic classification and confidence score. This invention improves the convenience and accuracy of chip carrier tape defect detection.

Description

Technical Field

[0001] This invention relates to the field of chip inspection technology, and in particular to a chip carrier defect detection and sorting method, system, terminal, and computer-readable storage medium. Background Technology

[0002] In the field of semiconductor packaging and testing, automated optical inspection equipment is a crucial step in ensuring the quality of chips leaving the factory. It typically acquires the appearance information of the chips carried on carrier tape through an image acquisition system and uses machine vision algorithms for comparison and analysis to achieve automatic defect identification and sorting. The level of intelligence and automation in this process directly affects the efficiency, reliability, and labor costs of the production line.

[0003] However, existing industrial chip AOI (Automated Optical Inspection) equipment still has significant limitations in practical applications. Firstly, the human-machine interaction methods are outdated. Operators in cleanroom environments must rely on touchscreens or barcode scanners to switch models and input commands, which is inconvenient and prone to contamination when operating with gloves. Secondly, the presentation of inspection results and the decision-making mechanism lack intelligence: the system often outputs simple abnormal area selections or heatmaps, lacking interpretable semantic descriptions such as "scratches" or "missing corners," requiring experienced employees to manually review the data, resulting in long training cycles for new employees. Furthermore, rejection decisions are mostly based on fixed thresholds, failing to differentiate judgments based on the specific nature and confidence level of the defect. Thirdly, the system lacks adaptive evolution capabilities; once deployed, its AI model struggles to utilize new data for real-time updates to identify new types of defects.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a chip carrier defect detection and sorting method, system, terminal, and computer-readable storage medium, aiming to solve the problems of uninterpretable detection results, lack of semantic description, unintelligent decision-making, and inability to self-evolve in traditional chip AOI equipment in the prior art, resulting in high reliance on manual labor, low detection accuracy, and poor adaptability to new defects.

[0006] To achieve the above objectives, the present invention provides a chip carrier tape defect detection and sorting method, which includes the following steps:

[0007] Upon receiving a voice command, the chip model is identified, corresponding detection parameters are loaded based on the chip model, and the carrier tape to be tested is controlled to move forward at a constant speed on the carrier tape transmission system based on the detection parameters.

[0008] The displacement of the carrier tape to be tested is monitored in real time by an encoder to obtain a trigger pulse, and based on the trigger pulse, a high-speed camera is synchronously controlled to perform exposure to obtain a chip image;

[0009] The chip image is input into a deep learning model, which outputs a heatmap of the defect-sensitive region. The chip image and the heatmap are then fused to obtain a fused image. The fused image is then input into a finely tuned visual semantic understanding model, which outputs a semantic classification and confidence score for the defect.

[0010] Based on the confidence score, the chips are graded and sorted to obtain sorting results. The sorting results include high-confidence defective chips that are physically marked and rejected, normal chips that are retained and passed, and suspected defective chips that are marked as needing retesting.

[0011] After the carrier tape to be tested is completed, the suspected defective chip is retested, the retest results are stored in the sample archive, and the visual semantic understanding model is incrementally fine-tuned based on the data in the sample archive.

[0012] The semantic classification is adjusted based on the retest results to obtain the target semantic classification, and a natural language detection report for the carrier to be detected is generated based on the target semantic classification.

[0013] Optionally, the chip carrier tape defect detection and sorting method, wherein the step of identifying the chip model upon receiving a voice command, loading corresponding detection parameters according to the chip model, and controlling the carrier tape to be inspected to move at a constant speed on the carrier tape transmission system according to the detection parameters, specifically includes:

[0014] The system receives the operator's voice commands via a high-sensitivity array microphone. Upon receiving the voice commands, it employs adaptive beamforming and deep learning noise reduction algorithms to denoise the voice commands.

[0015] A lightweight large language model is used to perform semantic understanding and intent mapping on the denoised voice commands, identify the chip model, and load the carrier pitch and traction speed corresponding to the chip model.

[0016] The servo traction roller is driven according to the loaded traction speed, which drives the carrier belt to be tested to slide at a constant speed on the low friction support platform. The high-resolution incremental encoder installed at the main shaft end of the servo traction roller provides real-time feedback of displacement information, and the rotation speed is dynamically adjusted in combination with the PID closed-loop algorithm to maintain a constant linear speed.

[0017] In the displacement information fed back by the high-resolution incremental encoder, a trigger pulse is generated for each movement of one carrier belt pitch.

[0018] Optionally, the chip carrier defect detection and sorting method, wherein the step of real-time monitoring of the displacement of the carrier to be inspected by an encoder to obtain a trigger pulse, and synchronously controlling a high-speed camera to perform exposure based on the trigger pulse to obtain a chip image, specifically includes:

[0019] The displacement of the carrier belt under test is monitored and fed back in real time by a high-resolution incremental encoder installed at the spindle end of the servo traction roller.

[0020] Whenever the cumulative displacement of the carrier tape to be detected reaches the carrier tape pitch, the high-resolution incremental encoder is controlled to generate a trigger pulse;

[0021] The trigger pulse is sent to the high-speed camera to control the high-speed camera to perform instantaneous exposure in the hardware external trigger mode to perform aerial photography and obtain chip images;

[0022] The flying shot is completed when the carrier belt to be tested slides at a constant speed across a low-friction support platform.

[0023] Optionally, in the chip carrier defect detection and sorting method, the step of inputting the chip image into the deep learning model further includes:

[0024] The chip image is sequentially denoised, white balance corrected, and local contrast enhanced to obtain an enhanced chip image;

[0025] A single chip region image is cropped from the enhanced chip image using template matching;

[0026] The single chip region image is normalized to a preset size to obtain a normalized chip image. The angle tilt of the normalized chip image is then corrected by affine transformation to obtain the target chip image for input into the deep learning model.

[0027] Optionally, the chip carrier defect detection and sorting method, wherein fusing the chip image with the heatmap to obtain a fused image, and inputting the fused image into a fine-tuned visual semantic understanding model to output a semantic classification and confidence score for the defect, specifically includes:

[0028] The chip image and the heatmap are fused together by channel splicing or weighted overlay to obtain a fused image containing the original visual information and the model's region of interest information;

[0029] The fused image is input into a large language vision model that has been fine-tuned with data from the chip manufacturing field. The large language vision model analyzes the fused image and outputs a semantic classification containing standardized defect names, as well as a confidence score corresponding to the semantic classification.

[0030] The semantic classification is used to describe the defect type of the chip in the natural language detection report, and the confidence score is used to perform hierarchical decision-making and processing.

[0031] Optionally, the chip carrier defect detection and sorting method, wherein the step of performing a classification decision and sorting process on the chips based on the confidence score to obtain a sorting result specifically includes:

[0032] If the confidence score is higher than the first threshold, the chip is determined to be a high-confidence defective chip. The laser drilling mechanism is controlled to physically mark the high-confidence defective chip, and the high-confidence defective chip is removed by the sorting mechanism.

[0033] If the confidence score is lower than the second threshold, the chip is determined to be a normal chip. The laser drilling mechanism and the sorting mechanism are not started. The normal chip is controlled to pass through the laser drilling station and the sorting station at a constant speed along with the carrier tape to be tested, and is wound up with the carrier tape to be tested.

[0034] If the confidence score is between the second threshold and the first threshold, the chip is determined to be a suspected defective chip, the location information of the suspected defective chip is recorded, and no further processing is performed.

[0035] Optionally, in the chip carrier defect detection and sorting method, the incremental fine-tuning of the visual semantic understanding model based on data in the sample archive specifically includes:

[0036] A preset fine-tuning trigger signal is established, and the presence of the fine-tuning trigger signal is monitored in real time. The fine-tuning trigger signal includes a sample archive signal, a detection statistics signal, and a manual review signal.

[0037] When at least one fine-tuning trigger signal is detected, the visual semantic understanding model is updated with parameters based on the data in the sample archive using low-rank adaptation technology to obtain the updated model.

[0038] The updated model is then subjected to performance verification, and once the performance verification is passed, the updated model is deployed as the current running model.

[0039] Furthermore, to achieve the above objectives, the present invention also provides a chip carrier defect detection and sorting system, wherein the chip carrier defect detection and sorting system includes:

[0040] The interactive control module is used to identify the chip model after receiving a voice command, load the corresponding detection parameters according to the chip model, and control the carrier tape to be tested to move forward at a constant speed on the carrier tape transmission system according to the detection parameters.

[0041] The image acquisition module is used to monitor the displacement of the carrier tape to be detected in real time through the encoder, obtain a trigger pulse, and synchronously control the high-speed camera to perform exposure based on the trigger pulse to obtain a chip image;

[0042] The defect identification module is used to input the chip image into a deep learning model, output a heat map of the defect-sensitive area, fuse the chip image and the heat map to obtain a fused image, input the fused image into a fine-tuned visual semantic understanding model, and output the semantic classification and confidence score of the defect.

[0043] The hierarchical decision module is used to perform hierarchical decision-making and sorting processing on the chips according to the confidence score, and obtain sorting results. The sorting results include high-confidence defective chips that are physically marked and rejected, normal chips that are retained and passed, and suspected defective chips marked as to be retested.

[0044] The retesting and optimization module is used to retest the suspected defective chip after the carrier tape to be tested is completed, store the retesting results in the sample archive, and perform incremental fine-tuning of the visual semantic understanding model based on the data in the sample archive.

[0045] The report generation module is used to adjust the semantic classification based on the retest results to obtain the target semantic classification, and generate a natural language detection report for the carrier to be detected based on the target semantic classification.

[0046] In addition, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a chip carrier defect detection and sorting program stored in the memory and executable on the processor, wherein when the chip carrier defect detection and sorting program is executed by the processor, it implements the steps of the chip carrier defect detection and sorting method as described above.

[0047] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a chip carrier defect detection and sorting program, and the chip carrier defect detection and sorting program, when executed by a processor, implements the steps of the chip carrier defect detection and sorting method as described above.

[0048] In this invention, voice commands are received, chip models are identified, and corresponding detection parameters are loaded to control the carrier tape to advance at a constant speed. Based on the carrier tape pitch pulses monitored by the encoder, a high-speed camera is synchronously triggered to acquire chip images. A defect heatmap is generated based on the images, and after being fused with the images, it is input into a fine-tuned visual semantic understanding model, which outputs a defect semantic classification and confidence score. Based on the confidence score, the chips are graded and processed to obtain three categories of results: normal chips, high-confidence defective chips, and suspected defective chips. The suspected defective chips are retested, and the results are stored in a sample archive for incremental fine-tuning of the model. Based on the semantic classification and confidence score, a natural language detection report is generated. This invention improves the convenience and accuracy of chip carrier tape defect detection, truly realizing a contactless, interpretable, and learning-enabled intelligent chip quality inspection closed loop. Attached Figure Description

[0049] Figure 1 This is a flowchart of a preferred embodiment of the chip carrier tape defect detection and sorting method of the present invention;

[0050] Figure 2 This is a schematic diagram illustrating the technical principle of the chip carrier defect detection and sorting method of the present invention.

[0051] Figure 3 This is a structural diagram of a preferred embodiment of the chip carrier tape defect detection and sorting system of the present invention;

[0052] Figure 4 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0053] This application provides a method, system, and terminal for detecting and sorting defects in chip carrier tapes. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0054] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0055] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0056] The chip carrier defect detection and sorting method described in the preferred embodiment of the present invention, such as... Figure 1 and Figure 2 As shown, the chip carrier defect detection and sorting method includes the following steps:

[0057] Step S10: Upon receiving a voice command, identify the chip model, load the corresponding detection parameters according to the chip model, and control the carrier tape to be tested to move forward at a constant speed on the carrier tape transmission system according to the detection parameters.

[0058] This embodiment aims to achieve contactless intelligent control throughout the entire process. The system captures natural language commands issued by the operator in a cleanroom environment through a high-sensitivity array microphone array, such as "Start testing chip model A". A dedicated voice processing module integrated into the equipment first uses adaptive beamforming technology to focus the human voice source and employs deep learning noise reduction algorithms to filter out ambient noise. Subsequently, the command is fed into a lightweight voice recognition and semantic understanding model finely tuned with semiconductor industry terminology, accurately parsing the operational intent and key parameters (such as chip model "A"). The system then automatically retrieves the corresponding carrier belt pitch, traction speed, optical parameters, and other complete testing configurations from a pre-stored parameter library based on the recognized model. Finally, the main controller drives the servo motor to start the carrier belt transmission system to run at a constant speed according to the loaded traction speed parameters, thus completing the seamless connection from voice command to physical execution and laying a stable motion foundation for subsequent high-precision testing.

[0059] Upon receiving a voice command, the process of identifying the chip model, loading corresponding detection parameters based on the chip model, and controlling the carrier tape to be tested to advance at a constant speed on the carrier tape transmission system based on the detection parameters specifically includes:

[0060] The system receives the operator's voice commands via a high-sensitivity array microphone. Upon receiving the voice commands, it employs adaptive beamforming and deep learning noise reduction algorithms to denoise the voice commands.

[0061] A lightweight large language model is used to perform semantic understanding and intent mapping on the denoised voice commands, identify the chip model, and load the carrier pitch and traction speed corresponding to the chip model.

[0062] The servo traction roller is driven according to the loaded traction speed, which drives the carrier belt to be tested to slide at a constant speed on the low friction support platform. The high-resolution incremental encoder installed at the main shaft end of the servo traction roller provides real-time feedback of displacement information, and the rotation speed is dynamically adjusted in combination with the PID closed-loop algorithm to maintain a constant linear speed.

[0063] In the displacement information fed back by the high-resolution incremental encoder, a trigger pulse is generated for each movement of one carrier belt pitch.

[0064] Understandably, this invention allows operators to perform contactless operations throughout the entire process, such as starting and stopping equipment, adjusting parameters, querying status, and handling anomalies, through natural language voice commands, significantly improving human-machine collaboration efficiency and adaptability to cleanroom environments.

[0065] When receiving voice commands, hardware configuration and noise reduction processing are required:

[0066] (1) The device integrates a high-sensitivity array microphone (signal-to-noise ratio ≥ 65 dB) and is deployed near the operation panel;

[0067] (2) Adaptive beamforming and deep learning noise reduction algorithms are used to effectively suppress background noise interference (≤85dB) in the workshop;

[0068] (3) Supports offline keyword wake-up (such as "start detection") to ensure real-time response and privacy security.

[0069] Furthermore, the denoised voice commands are adapted for multiple languages ​​and dialects:

[0070] (1) The speech recognition engine supports Mandarin, Cantonese and English commands, and the language library can be expanded through online updates;

[0071] (2) Fine-tune the acoustic model for semiconductor industry terms (such as “Pitch”, “carrier tape”, “punch hole”) to improve the accuracy of professional term recognition.

[0072] Furthermore, a lightweight large language model is used to perform semantic understanding and intent mapping on the denoised voice commands. Upon receiving the voice, the system parses the user's intent using a lightweight large language model (such as the Qwen-Max distillation version). For example, "Next roll" → triggers confirmation of unloading completion; "Pause" → executes the safety shutdown procedure. This embodiment also supports compound commands, such as "Skip this chip and drill a hole," which the system automatically decomposes into two atomic operations: "Skip the flyer + Trigger the laser." Finally, the execution result is announced in real time via synthesized voice, such as "125th chip skipped" or "Laser drilling completed." It should be noted that critical operations (such as forced shutdown) require secondary voice confirmation to prevent accidental triggering.

[0073] Furthermore, the servo traction rollers are initialized. Based on the material and thickness of the carrier tape to be inspected, the gap between the upper and lower servo traction rollers is adjusted to provide sufficient clamping force to prevent slippage and avoid damaging the carrier tape itself or the internal packaged chips. Upon receiving the "start inspection" command, the servo traction rollers first start at a low speed, smoothly pulling the head of the carrier tape from the starting position of the inspection station into the imaging area. After the carrier tape stabilizes, it automatically switches to the preset working speed, uniformly pulling the carrier tape forward. It should be noted that the carrier tape is actively pulled out by the traction rollers and does not require manual pre-threading to the take-up end. The system begins inspection from the first chip that enters the camera's field of view from the carrier tape, ensuring no missed inspections.

[0074] Step S20: The displacement of the carrier tape to be tested is monitored in real time by the encoder to obtain a trigger pulse, and based on the trigger pulse, the high-speed camera is synchronously controlled to perform exposure to obtain a chip image.

[0075] This embodiment achieves millisecond-level synchronous capture of chip images in motion, a crucial physical condition for ensuring consistent detection. During the uniform movement of the carrier belt, a high-resolution optical encoder, fixedly mounted on the servo traction roller spindle, continuously monitors the rotation angle and converts it into a high-precision displacement signal. The control system calculates based on the loaded "carrier belt pitch" parameter, ensuring that the encoder generates a steep-edge hardware trigger pulse whenever the carrier belt moves precisely by one pitch (i.e., the spacing between one chip). This pulse is directly sent to the hardware trigger port of the high-speed industrial camera, driving the camera to complete an exposure within a microsecond-level exposure time. At this moment, the chip moves precisely to the fixed imaging focal plane directly below the camera's field of view. Combined with a high-brightness stroboscopic flash source, this instantaneous motion is frozen, resulting in a clear, motion-free image of the chip surface. The entire process is directly driven by hardware signals, ensuring that each chip is captured at the same position, providing standardized input for subsequent analysis.

[0076] Specifically, a high-resolution incremental encoder installed on the spindle end of the servo traction roller monitors and provides feedback on the displacement of the carrier belt to be tested in real time; whenever the cumulative displacement of the carrier belt to be tested reaches the carrier belt pitch, the high-resolution incremental encoder is controlled to generate a trigger pulse; the trigger pulse is sent to a high-speed camera, and the high-speed camera is controlled to perform instantaneous exposure in the hardware external trigger mode to perform aerial photography to obtain a chip image; wherein, the aerial photography is completed when the carrier belt to be tested slides uniformly across a low-friction support platform.

[0077] Understandably, in this embodiment, a high-resolution incremental encoder is installed at the spindle end of the servo traction roller to ensure that its rotation is strictly synchronized with the traction roller, without slippage or lag. The encoder output signal is connected to the motion controller, and after calibration, a precise trigger pulse is output for each standard pitch of the carrier belt, used to synchronize the high-speed camera's shooting.

[0078] Furthermore, during the operation of the carrier belt, the encoder continuously sends position pulse signals to the central processing unit; the control system calculates the actual linear speed based on the pulse frequency per unit time and compares it with the set speed, and dynamically fine-tunes the servo traction roller speed through a PID (Proportional-Integral-Derivative) closed-loop algorithm to maintain a constant flow rate; at the same time, each pitch pulse serves as a hardware trigger signal to precisely control the high-speed camera to expose when the chip is located at the center of the imaging focal plane, ensuring that each chip is clearly captured in the same spatial position.

[0079] It should be noted that a low-friction, high-flatness carrier support platform, such as PTFE (Polytetrafluoroethylene) or a ceramic-coated surface, is provided below the testing station to prevent the carrier from sagging or vibrating and to ensure consistent imaging.

[0080] Furthermore, the carrier tape transmission system includes a servo take-up shaft; when the carrier tape to be tested is controlled to move forward at a constant speed, the servo take-up shaft is controlled to operate synchronously at a linear speed slightly higher than the traction speed, so as to establish a constant tension in the carrier tape.

[0081] Specifically, based on the total length and outer diameter of the carrier belt, a matching empty take-up spool is selected and securely installed on the right-side servo take-up shaft; the initial torque limit or tension mode parameter of the take-up shaft is set to prevent the carrier belt from stretching, deforming or breaking due to sudden load changes at startup.

[0082] When the carrier belt transport system is started, the servo take-up shaft operates synchronously at a ratio slightly higher than the linear speed of the traction roller (approximately 1.01–1.03 times), so that the carrier belt forms a constant micro-tension (approximately 1.5–2.5 N) between the traction section and the take-up section.

[0083] Once the tail of the carrier tape passes the inspection station, the system determines "one roll of carrier tape has been inspected" using any of the following methods:

[0084] (1) The encoder's cumulative pulse count reaches the preset total number of sections (corresponding to the total length of the carrier tape).

[0085] (2) If the vision module fails to detect a valid chip area for multiple consecutive frames (e.g., ≥5 frames), the image content indicates that the carrier tape has ended.

[0086] After confirming that the carrier tape is used up, the system automatically executes a graded deceleration and shutdown procedure: first, the speed of the servo traction roller is reduced to 10% of the rated value and maintained for 2 seconds to release residual tension; then, the servo traction roller and servo take-up shaft are stopped simultaneously to avoid carrier tape rebound, slack or breakage due to inertia; the full voice interaction module is triggered to announce: "Detection complete, please unload the right take-up reel".

[0087] Finally, manual unloading is performed. After hearing the voice prompt, the operator manually removes the inspected carrier tape reel from the right take-up shaft. If it is necessary to continue inspecting the next roll of carrier tape, repeat the "manual loading preparation" steps; otherwise, the system power can be turned off or the system can be put into standby mode.

[0088] Step S30: Input the chip image into the deep learning model, output a heat map of the defect-sensitive area, fuse the chip image and the heat map to obtain a fused image, input the fused image into the fine-tuned visual semantic understanding model, and output the semantic classification and confidence score of the defect.

[0089] This invention achieves a leap from pixel-level anomalies to interpretable defect descriptions. First, the acquired chip images, after preprocessing (e.g., denoising and normalization), are input into a pre-trained deep learning detection model based on the YOLO architecture (an object detection algorithm). This model not only outputs the bounding boxes of defects but also generates a highlighted "defect-sensitive region heatmap" through its internal activation features, visually highlighting the abnormal regions of interest to the model. Next, the original chip image and the generated heatmap are fused at the channel level to form an enhanced input containing both original visual information and model attention information. This fused image is fed into a large-scale visual-semantic understanding model (such as Qwen-VL, an open-source large-scale visual language model) fine-tuned with massive amounts of chip defect data. Leveraging its powerful multimodal understanding capabilities, this large model aligns visual features with the semantic space, ultimately outputting a structured result: on one hand, semantic classification labels for defects that humans can directly understand, such as "scratches," "missing corners," and "contamination"; on the other hand, a confidence score for the judgment, quantifying the degree of certainty of the judgment, thus providing intelligent judgments with both interpretability and quantitative basis for subsequent decision-making.

[0090] Specifically, the step of inputting the chip image into the deep learning model further includes:

[0091] The chip image is sequentially denoised, white balance corrected, and local contrast enhanced to obtain an enhanced chip image;

[0092] A single chip region image is cropped from the enhanced chip image using template matching;

[0093] The single chip region image is normalized to a preset size to obtain a normalized chip image. The angle tilt of the normalized chip image is then corrected by affine transformation to obtain the target chip image for input into the deep learning model.

[0094] In this embodiment, the image is first preprocessed. The original image is denoised, white balance corrected and local contrast enhanced. Then, a single chip region is cropped out as an independent analysis unit by template matching. The cropped image is uniformly normalized to 640×640 pixels and the angle tilt caused by the slight offset of the carrier is corrected by affine transformation to ensure input consistency.

[0095] Next, model training and deployment are carried out. A target detection model based on the YOLO architecture is adopted. This model is trained on a fully labeled dataset containing typical defects such as scratches and missing corners to ensure the ability to identify common defects. After training, the optimal weights are saved as a dedicated model file (e.g., best.pt) and deployed on an edge computing unit (such as NVIDIA Jetson AGX Orin or equivalent performance devices), supporting single-frame inference latency ≤30 ms.

[0096] Furthermore, the chip image is input into a deep learning model, which outputs a heatmap of the defect-sensitive region. After inputting a single chip image, the model outputs preliminary defect candidate regions (bounding boxes) and corresponding confidence scores, serving as the basis for heatmap generation and subsequent semantic discrimination.

[0097] In this embodiment, the process of generating a heatmap includes: generating a pixel-level heatmap based on the activation response of the intermediate feature layer of the YOLOv11 model using a gradient-based interpretability algorithm (such as the Grad-CAM series of methods); wherein, the heatmap intuitively reflects the model's attention to each region of the image with color intensity (red indicates high activation, blue indicates low activation), highlighting potential defect locations; the heatmap is then channel-stitched or weighted fused with the original chip image to form a multimodal input tensor, which is used to enhance the context-awareness of semantic discrimination in subsequent large models.

[0098] Furthermore, the process of fusing the chip image with the heatmap to obtain a fused image, and then inputting the fused image into a finely tuned visual semantic understanding model to output a semantic classification and confidence score for the defect, specifically includes:

[0099] The chip image and the heatmap are fused together by channel splicing or weighted overlay to obtain a fused image containing the original visual information and the model's region of interest information;

[0100] The fused image is input into a large language vision model that has been fine-tuned with data from the chip manufacturing field. The large language vision model analyzes the fused image and outputs a semantic classification containing standardized defect names, as well as a confidence score corresponding to the semantic classification.

[0101] The semantic classification is used to describe the defect type of the chip in the natural language detection report, and the confidence score is used to perform hierarchical decision-making and processing.

[0102] It is understood that this invention does not analyze the original chip image or the heatmap generated by the model separately, but rather combines the two into a "fused image" through "channel stitching" or "pixel-level weighted overlay" techniques. Channel stitching refers to merging the heatmap as an independent color channel (such as the red channel representing attention) with the RGB channels of the original image; weighted overlay dynamically enhances the contrast of corresponding areas in the original image based on the activation values ​​of the heatmap. It provides the subsequent model with both objective visual facts (the original image) and the model's attention prior (the heatmap), essentially giving the AI ​​a context for highlighting key points, greatly improving the efficiency and accuracy of subsequent semantic analysis.

[0103] Furthermore, the fused image is input into a visual semantic understanding model, which is a domain expert specifically fine-tuned from a general large-language visual model (such as Qwen-VL) using "chip manufacturing domain data." This fine-tuning enables it to deeply understand the microscopic visual features and semantic definitions of defects such as "scratches," "corner defects," and "particle contamination" in chip packaging processes. During the model's analysis, it parses the abnormal patterns in the fused image, combines it with its internally learned process knowledge base, maps abstract pixel anomalies into standardized natural language terms, and simultaneously evaluates the certainty of this judgment, outputting a confidence score from 0.0 to 1.0.

[0104] Furthermore, the core value of the aforementioned semantic tags lies in their readability and standardization. They enable defect types to be directly used to generate natural language reports that human engineers can quickly understand, facilitating defect distribution statistics and root cause analysis. Meanwhile, the confidence score provides the quantitative basis needed for decision-making; it is the sole and direct criterion for subsequent hierarchical sorting (e.g., direct removal of high-confidence samples and retesting of low-confidence samples). The combination of these two elements ensures that the system output is no longer isolated data points, but rather an intelligent judgment that is both interpretable and actionable. This addresses the core shortcomings of existing technologies, such as the need for manual review of red-boxed annotations and the lack of intelligence in fixed-threshold decision-making.

[0105] Step S40: Perform a classification decision and sorting process on the chips according to the confidence score to obtain the sorting results. The sorting results include high-confidence defective chips that are physically marked and rejected, normal chips that are retained and passed, and suspected defective chips marked as to be retested.

[0106] In this embodiment, based on the confidence score output in the previous step, a differentiated physical handling strategy is implemented to achieve precise sorting. The system presets two thresholds to classify chips into three categories: For high-confidence defective chips with a confidence score higher than the high threshold, the system immediately generates an instruction to precisely position and mark the chip surface with an ultraviolet laser as it passes through the downstream fixed laser station at a constant speed with the carrier tape. Subsequently, the dual-nozzle sorting mechanism accurately picks it up and rejects it into the waste bin. For normal chips with a confidence score lower than the low threshold, the system controls the laser and sorting mechanism to remain silent, allowing it to pass through all subsequent stations without damage and be normally wound up with the carrier tape. For suspected defective chips with a confidence score between the two thresholds, the system does not perform any physical intervention, but only records its precise position on the carrier tape (such as the pulse count value) and adds its image and preliminary judgment result to the retest queue. This classification mechanism ensures the rationality of the decision-making and avoids misjudgment and missed detection caused by the traditional fixed threshold method.

[0107] Specifically, if the confidence score is higher than the first threshold, the chip is determined to be a high-confidence defective chip, the laser drilling mechanism is controlled to physically mark the high-confidence defective chip, and the high-confidence defective chip is removed by the sorting mechanism.

[0108] If the confidence score is lower than the second threshold, the chip is determined to be a normal chip. The laser drilling mechanism and the sorting mechanism are not started. The normal chip is controlled to pass through the laser drilling station and the sorting station at a constant speed along with the carrier tape to be tested, and is wound up with the carrier tape to be tested.

[0109] If the confidence score is between the second threshold and the first threshold, the chip is determined to be a suspected defective chip, the location information of the suspected defective chip is recorded, and no further processing is performed.

[0110] Understandably, in this embodiment, a three-level processing strategy is implemented for the chips based on the confidence score output by the intelligent defect detection module: high-confidence defects are marked with laser drilling, low-confidence normal chips pass directly, and medium-confidence suspected defects enter the retesting path. Through the collaboration of a fixed-position laser and a dual-nozzle robotic arm, non-contact, high-precision, and zero-damage sorting is achieved.

[0111] Specifically, when the confidence score is higher than a first threshold (e.g., ≥0.8), the system clearly identifies the defect. Its handling process is highly efficient and automated: First, the system immediately generates a laser drilling command and, based on real-time position information fed back by the encoder, controls a picosecond ultraviolet laser to perform sub-millimeter precision physical marking on the chip's surface the instant the chip passes the fixed drilling station at a constant speed with the carrier tape, providing an indelible traceability mark for subsequent processes. Then, at the sorting station, a dual-nozzle sorting mechanism, based on synchronized position information, precisely picks up the marked chip from the carrier tape groove and transfers it to the waste bin, completing the "identification-marking-rejection" closed loop. This path ensures zero missed rejections and non-contact processing of clearly defective products.

[0112] Furthermore, when the confidence score falls below the second threshold (e.g., <0.4), the system determines the chip to be qualified. The system will explicitly prohibit all subsequent physical operations that may affect the chip: the laser drilling mechanism will not be triggered, and the sorting mechanism will not operate. Under the coordinated action of the traction and rewinding systems, the chip will slide smoothly and uniformly across the laser station and sorting station without any pauses or physical interference, and will finally be rewound with the entire roll of carrier tape, ensuring the absolute safety of good products and the maximum flow efficiency of the production line.

[0113] Furthermore, when the confidence score falls between two thresholds (e.g., 0.4-0.8), the system acknowledges the uncertainty in the current judgment. At this point, the system adopts a "delayed decision" strategy: instead of immediately initiating any costly or irreversible physical operations (such as punching or rejection), it only records the chip's precise position information on the carrier tape (e.g., absolute coordinates based on encoder pulses) and correlates its image with the initial judgment result, storing it in a separate retest queue. The chip continues to move with the carrier tape, its physical state fully preserved, creating the possibility for subsequent secondary judgments based on more information (such as multi-angle retests). This mechanism prevents misjudgments caused by the uncertainty of a single detection.

[0114] Step S50: After the carrier tape to be tested is completed, the suspected defective chip is retested, the retest results are stored in the sample archive, and the visual semantic understanding model is incrementally fine-tuned based on the data in the sample archive.

[0115] In this embodiment, after the initial inspection of an entire roll of carrier tape is completed, the servo take-up shaft and servo traction roller are controlled to rotate in reverse, causing all marked suspected defective chips to pass through the inspection station again in sequence. During the retest, the system can adjust the illumination angle and camera parameters to perform multi-angle imaging and obtain richer feature information. The results of the retest, along with its complete data link, are automatically stored in the sample archive.

[0116] Furthermore, this archive, serving as the system's learning memory, periodically triggers an incremental learning process: employing efficient fine-tuning techniques such as low-rank adaptation, it updates the large-scale visual-semantic understanding model using only these newly added, high-quality samples. After performance validation on the validation set, the new model seamlessly replaces the old version, enabling the system to continuously adapt to new defect patterns and achieve autonomous performance improvement through repeated use.

[0117] Specifically, the incremental fine-tuning of the visual semantic understanding model based on the data in the sample archive includes:

[0118] A preset fine-tuning trigger signal is established, and the presence of the fine-tuning trigger signal is monitored in real time. The fine-tuning trigger signal includes a sample archive signal, a detection statistics signal, and a manual review signal.

[0119] When at least one fine-tuning trigger signal is detected, the visual semantic understanding model is updated with parameters based on the data in the sample archive using low-rank adaptation technology to obtain the updated model.

[0120] The updated model is then subjected to performance verification, and once the performance verification is passed, the updated model is deployed as the current running model.

[0121] Understandably, all chip images, initial inspection results, retest results, and final manual confirmation labels entering the retesting path are automatically archived to the local / cloud sample library. Each sample contains metadata: defect type, confidence curve, light source conditions, carrier batch, operator ID, etc., supporting multi-dimensional retrieval and analysis. Simultaneously, the system regularly performs deduplication, cleaning, and hard case mining on the sample library, prioritizing the retention of boundary cases (such as low-contrast scratches and minor corner defects). Samples are divided into training and validation sets according to time windows to ensure continuous improvement in model generalization ability.

[0122] Furthermore, a low-rank adaptation technique is used to fine-tune the large language vision model. Only trainable low-rank decomposition matrices (total trainable parameters account for <1%) are injected into the side path of the original weights, which greatly reduces computing power consumption, GPU memory usage and storage overhead, and supports incremental learning on edge devices.

[0123] The visual semantic understanding model is updated using a low-rank adaptation technique, which is achieved by injecting a trainable low-rank decomposition matrix into the original weights of the model. The total number of updated parameters accounts for less than 1% of the original model parameters.

[0124] This invention establishes three intelligent trigger signals closely linked to actual production, ensuring that learning opportunities are both efficient and necessary: ​​The "sample archive signal" directly monitors the capacity of the system's knowledge base. When new samples with accurate labels (mainly from retesting and manual verification) accumulate to a certain threshold, it signifies the availability of data for effective learning. The "detection statistics signal" monitors the health status of the production line in real time. By analyzing real-time detection logs, when the detection rate of a certain type of defect (such as "scratches") experiences a statistically significant abnormal surge, the system can keenly perceive the possible emergence of new process variations or novel defect patterns, thus triggering targeted learning. The "manual review signal" introduces crucial quality supervision. When operators discover a high-confidence misjudgment during review (i.e., the system is very certain but actually misjudged), this strong feedback signal immediately triggers emergency fine-tuning, quickly correcting cognitive blind spots or biases in the model. This multi-source, proactive, and reactive triggering system endows the system with the ability to dynamically perceive environmental changes and its own defects.

[0125] After triggering the incremental fine-tuning, the system trains the visual semantic understanding model using newly added samples in the sample archive. After training, the fine-tuned model is validated using an independent validation set. When the validation results meet the following preset conditions, the fine-tuned model is updated to the currently running model:

[0126] The preset conditions are: the average precision improvement of the model in the main defect categories reaches or exceeds the preset performance improvement threshold, and the recall rate of the model in the historical key defect categories does not exceed the preset forgetting tolerance threshold.

[0127] Step S60: Adjust the semantic classification according to the retest results to obtain the target semantic classification, and generate a natural language detection report for the carrier to be detected based on the target semantic classification.

[0128] Specifically, the inspection data is transformed into knowledge that can directly guide production. After all inspections and retests are completed, the system invokes the report generation module. This module is based on the structured data accumulated in the preceding steps, including semantic classification labels, confidence scores, and handling results for each chip. Using natural language processing technology, the system automatically analyzes the distribution of defect types, calculates the yield rate, identifies high-frequency defects and their location patterns, and links them to the process knowledge base. Finally, a clearly structured and professionally worded natural language inspection report is generated.

[0129] Furthermore, the natural language inspection report includes: total number of chips, defect distribution heatmap, statistics of the top 3 defect types, typical defect examples and suggestions; the report uses natural language description, such as: "This volume inspected a total of 12,500 chips and found 87 defects, mainly lateral metal scratches (accounting for 62%), concentrated in the middle section of the carrier tape. It is recommended to check the upstream material supply organization."

[0130] The present invention has the following beneficial effects:

[0131] (1) Full voice control enhances operational convenience and safety: No manual scanning or touchscreen operation is required. All operations, such as chip model setting, detection start / stop, laser drilling switch, and report generation, can be completed via voice commands. This not only improves the operational convenience and efficiency of the cleanroom but also reduces the risk of contamination from contact with equipment, while enhancing the safety of the working environment.

[0132] (2) Precise physical marking enhances end-to-end traceability efficiency: Dynamic laser technology is used to precisely mark the location of defective chips, providing clear physical identification for subsequent processes (such as manual re-inspection, classification and archiving, or customer inspection). This helps to quickly distinguish between detected defective and qualified products, avoiding confusion and thus improving the traceability and management efficiency of the entire production process. Deep learning models reduce misjudgments and shorten training cycles: By using large-scale deep learning models to perform semantic analysis on heatmaps, various types of defects such as "scratches" and "missing corners" and their confidence levels can be accurately identified, replacing the traditional method of relying solely on red box markings in automated optical inspection (AOI) systems. This method reduces the need for manual re-inspection, lowers the misjudgment rate, and significantly shortens the learning curve and training time for new employees.

[0133] (3) Intelligent sorting mechanism ensures efficient and error-free processing: The dual-nozzle design realizes a highly efficient automated sorting process, in which high-confidence defective chips are directly rejected, while low-confidence suspicious samples are automatically sent back for secondary testing. This mechanism ensures that no potential problems are missed and good products are not overly rejected, while recording the retest results for continuous improvement of model performance.

[0134] (4) Flexible configuration to adapt to diverse process requirements: Operators can dynamically adjust the activation or deactivation of the laser drilling function via voice commands, and can select appropriate modes (such as "alarm only" or "drill rejection") according to the requirements of different production processes. This flexibility enables the system to better adapt to various production and quality control standards.

[0135] (5) Self-evolution engine enables continuous self-optimization: The built-in large-scale model self-evolution engine can automatically update the defect classification model periodically using the correction labels in the retest data, so that the accuracy of the system will continue to improve with the number of uses. This means that users do not need to wait for the manufacturer to provide remote upgrade services to enjoy the latest algorithm optimization results.

[0136] (6) Natural Language Reports Improve Data Analysis Efficiency: The system can automatically generate natural language format inspection reports that include yield rate statistics, defect distribution, and improvement suggestions, replacing the traditional table output method. This report format is more intuitive and easier to understand, making it easier for managers to quickly obtain key information and make corresponding decisions.

[0137] Furthermore, such as Figure 3 As shown, based on the above-described chip carrier defect detection and sorting method, the present invention also provides a chip carrier defect detection and sorting system, wherein the chip carrier defect detection and sorting system includes:

[0138] The interactive control module 51 is used to identify the chip model after receiving a voice command, load the corresponding detection parameters according to the chip model, and control the carrier tape to be tested to move forward at a constant speed on the carrier tape transmission system according to the detection parameters.

[0139] The image acquisition module 52 is used to monitor the displacement of the carrier tape to be detected in real time through the encoder, obtain a trigger pulse, and synchronously control the high-speed camera to perform exposure based on the trigger pulse to obtain a chip image;

[0140] The defect identification module 53 is used to input the chip image into a deep learning model, output a heat map of the defect-sensitive area, fuse the chip image and the heat map to obtain a fused image, input the fused image into a fine-tuned visual semantic understanding model, and output the semantic classification and confidence score of the defect.

[0141] The grading decision module 54 is used to perform grading decision and sorting processing on the chips according to the confidence score, and obtain sorting results. The sorting results include high-confidence defective chips that are physically marked and rejected, normal chips that are retained and passed, and suspected defective chips that are marked as to be retested.

[0142] The retesting and optimization module 55 is used to retest the suspected defective chip after the carrier tape to be tested is completed, store the retesting results in the sample archive, and perform incremental fine-tuning of the visual semantic understanding model based on the data in the sample archive.

[0143] The report generation module 56 is used to adjust the semantic classification according to the retest results to obtain the target semantic classification, and generate a natural language detection report of the carrier to be detected according to the target semantic classification.

[0144] Furthermore, such as Figure 4 As shown, based on the above-mentioned chip carrier defect detection and sorting method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 4 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0145] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a chip carrier defect detection and sorting program 40, which can be executed by the processor 10 to implement the chip carrier defect detection and sorting method of this application.

[0146] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the chip carrier defect detection and sorting method.

[0147] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0148] In one embodiment, when the processor 10 executes the chip carrier defect detection and sorting program 40 in the memory 20, the following steps are performed:

[0149] Upon receiving a voice command, the chip model is identified, corresponding detection parameters are loaded based on the chip model, and the carrier tape to be tested is controlled to move forward at a constant speed on the carrier tape transmission system based on the detection parameters.

[0150] The displacement of the carrier tape to be tested is monitored in real time by an encoder to obtain a trigger pulse, and based on the trigger pulse, a high-speed camera is synchronously controlled to perform exposure to obtain a chip image;

[0151] The chip image is input into a deep learning model, which outputs a heatmap of the defect-sensitive region. The chip image and the heatmap are then fused to obtain a fused image. The fused image is then input into a finely tuned visual semantic understanding model, which outputs a semantic classification and confidence score for the defect.

[0152] Based on the confidence score, the chips are graded and sorted to obtain sorting results. The sorting results include high-confidence defective chips that are physically marked and rejected, normal chips that are retained and passed, and suspected defective chips that are marked as needing retesting.

[0153] After the carrier tape to be tested is completed, the suspected defective chip is retested, the retest results are stored in the sample archive, and the visual semantic understanding model is incrementally fine-tuned based on the data in the sample archive.

[0154] The semantic classification is adjusted based on the retest results to obtain the target semantic classification, and a natural language detection report for the carrier to be detected is generated based on the target semantic classification.

[0155] The step of identifying the chip model upon receiving a voice command, loading corresponding detection parameters based on the chip model, and controlling the carrier tape to be tested to move at a constant speed on the carrier tape transmission system based on the detection parameters specifically includes:

[0156] The system receives the operator's voice commands via a high-sensitivity array microphone. Upon receiving the voice commands, it employs adaptive beamforming and deep learning noise reduction algorithms to denoise the voice commands.

[0157] A lightweight large language model is used to perform semantic understanding and intent mapping on the denoised voice commands, identify the chip model, and load the carrier pitch and traction speed corresponding to the chip model.

[0158] The servo traction roller is driven according to the loaded traction speed, which drives the carrier belt to be tested to slide at a constant speed on the low friction support platform. The high-resolution incremental encoder installed at the main shaft end of the servo traction roller provides real-time feedback of displacement information, and the rotation speed is dynamically adjusted in combination with the PID closed-loop algorithm to maintain a constant linear speed.

[0159] In the displacement information fed back by the high-resolution incremental encoder, a trigger pulse is generated for each movement of one carrier belt pitch.

[0160] Specifically, the step of monitoring the displacement of the carrier tape under test in real time through an encoder to obtain a trigger pulse, and then synchronously controlling a high-speed camera to perform exposure based on the trigger pulse to obtain a chip image, includes:

[0161] The displacement of the carrier belt under test is monitored and fed back in real time by a high-resolution incremental encoder installed at the spindle end of the servo traction roller.

[0162] Whenever the cumulative displacement of the carrier tape to be detected reaches the carrier tape pitch, the high-resolution incremental encoder is controlled to generate a trigger pulse;

[0163] The trigger pulse is sent to the high-speed camera to control the high-speed camera to perform instantaneous exposure in the hardware external trigger mode to perform aerial photography and obtain chip images;

[0164] The flying shot is completed when the carrier belt to be tested slides at a constant speed across a low-friction support platform.

[0165] The step of inputting the chip image into the deep learning model includes, prior to:

[0166] The chip image is sequentially denoised, white balance corrected, and local contrast enhanced to obtain an enhanced chip image;

[0167] A single chip region image is cropped from the enhanced chip image using template matching;

[0168] The single chip region image is normalized to a preset size to obtain a normalized chip image. The angle tilt of the normalized chip image is then corrected by affine transformation to obtain the target chip image for input into the deep learning model.

[0169] The process of fusing the chip image with the heatmap to obtain a fused image, and then inputting the fused image into a fine-tuned visual semantic understanding model to output a semantic classification and confidence score for the defect, specifically includes:

[0170] The chip image and the heatmap are fused together by channel splicing or weighted overlay to obtain a fused image containing the original visual information and the model's region of interest information;

[0171] The fused image is input into a large language vision model that has been fine-tuned with data from the chip manufacturing field. The large language vision model analyzes the fused image and outputs a semantic classification containing standardized defect names, as well as a confidence score corresponding to the semantic classification.

[0172] The semantic classification is used to describe the defect type of the chip in the natural language detection report, and the confidence score is used to perform hierarchical decision-making and processing.

[0173] Specifically, the step of performing grading decisions and sorting processes on the chips based on the confidence score to obtain sorting results includes:

[0174] If the confidence score is higher than the first threshold, the chip is determined to be a high-confidence defective chip. The laser drilling mechanism is controlled to physically mark the high-confidence defective chip, and the high-confidence defective chip is removed by the sorting mechanism.

[0175] If the confidence score is lower than the second threshold, the chip is determined to be a normal chip. The laser drilling mechanism and the sorting mechanism are not started. The normal chip is controlled to pass through the laser drilling station and the sorting station at a constant speed along with the carrier tape to be tested, and is wound up with the carrier tape to be tested.

[0176] If the confidence score is between the second threshold and the first threshold, the chip is determined to be a suspected defective chip, the location information of the suspected defective chip is recorded, and no further processing is performed.

[0177] Specifically, the incremental fine-tuning of the visual semantic understanding model based on the data in the sample archive includes:

[0178] A preset fine-tuning trigger signal is established, and the presence of the fine-tuning trigger signal is monitored in real time. The fine-tuning trigger signal includes a sample archive signal, a detection statistics signal, and a manual review signal.

[0179] When at least one fine-tuning trigger signal is detected, the visual semantic understanding model is updated with parameters based on the data in the sample archive using low-rank adaptation technology to obtain the updated model.

[0180] The updated model is then subjected to performance verification, and once the performance verification is passed, the updated model is deployed as the current running model.

[0181] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a chip carrier defect detection and sorting program, which, when executed by a processor, implements the steps of the chip carrier defect detection and sorting method as described above.

[0182] In summary, this invention provides a method, system, terminal, and storage medium for chip carrier tape defect detection and sorting. The method includes: receiving voice commands, identifying the chip model and loading corresponding detection parameters to control the carrier tape to advance at a constant speed; synchronously triggering a high-speed camera to acquire chip images based on carrier tape pitch pulses monitored by an encoder; generating a defect heatmap based on the image, fusing it with the image, and inputting it into a fine-tuned visual semantic understanding model to output defect semantic classification and confidence score; classifying and processing the chips according to the confidence score to obtain three categories of results: normal chips, high-confidence defective chips, and suspected defective chips; retesting the suspected defective chips and storing the results in a sample archive for incremental fine-tuning of the model; and generating a natural language detection report based on the semantic classification and confidence score. This invention improves the convenience and accuracy of chip carrier tape defect detection.

[0183] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0184] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0185] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for detecting and sorting defects in chip carrier tapes, characterized in that, The chip carrier defect detection and sorting method includes: Upon receiving a voice command, the chip model is identified, corresponding detection parameters are loaded based on the chip model, and the carrier tape to be tested is controlled to move forward at a constant speed on the carrier tape transmission system based on the detection parameters. The displacement of the carrier tape to be tested is monitored in real time by an encoder to obtain a trigger pulse, and based on the trigger pulse, a high-speed camera is synchronously controlled to perform exposure to obtain a chip image; The chip image is input into a deep learning model, which outputs a heatmap of the defect-sensitive region. The chip image and the heatmap are then fused to obtain a fused image. The fused image is then input into a finely tuned visual semantic understanding model, which outputs a semantic classification and confidence score for the defect. Based on the confidence score, the chips are graded and sorted to obtain sorting results. The sorting results include high-confidence defective chips that are physically marked and rejected, normal chips that are retained and passed, and suspected defective chips that are marked as needing retesting. After the carrier tape to be tested is completed, the suspected defective chip is retested, the retest results are stored in the sample archive, and the visual semantic understanding model is incrementally fine-tuned based on the data in the sample archive. The semantic classification is adjusted based on the retest results to obtain the target semantic classification, and a natural language detection report for the carrier to be detected is generated based on the target semantic classification.

2. The chip carrier defect detection and sorting method according to claim 1, characterized in that, Upon receiving a voice command, the process of identifying the chip model, loading corresponding detection parameters based on the chip model, and controlling the carrier tape to be tested to advance at a constant speed on the carrier tape transmission system based on the detection parameters specifically includes: The system receives the operator's voice commands via a high-sensitivity array microphone. Upon receiving the voice commands, it employs adaptive beamforming and deep learning noise reduction algorithms to denoise the voice commands. A lightweight large language model is used to perform semantic understanding and intent mapping on the denoised voice commands, identify the chip model, and load the carrier pitch and traction speed corresponding to the chip model. The servo traction roller is driven according to the loaded traction speed, which drives the carrier belt to be tested to slide at a constant speed on the low friction support platform. The high-resolution incremental encoder installed at the main shaft end of the servo traction roller provides real-time feedback of displacement information, and the rotation speed is dynamically adjusted in combination with the PID closed-loop algorithm to maintain a constant linear speed. In the displacement information fed back by the high-resolution incremental encoder, a trigger pulse is generated for each movement of one carrier belt pitch.

3. The chip carrier defect detection and sorting method according to claim 2, characterized in that, The process of real-time monitoring of the displacement of the carrier tape under test via an encoder to obtain a trigger pulse, and synchronously controlling a high-speed camera to perform exposure based on the trigger pulse to obtain a chip image, specifically includes: The displacement of the carrier belt under test is monitored and fed back in real time by a high-resolution incremental encoder installed at the spindle end of the servo traction roller. Whenever the cumulative displacement of the carrier tape to be detected reaches the carrier tape pitch, the high-resolution incremental encoder is controlled to generate a trigger pulse; The trigger pulse is sent to the high-speed camera to control the high-speed camera to perform instantaneous exposure in the hardware external trigger mode to perform aerial photography and obtain chip images; The flying shot is completed when the carrier belt to be tested slides at a constant speed across a low-friction support platform.

4. The chip carrier defect detection and sorting method according to claim 1, characterized in that, Before inputting the chip image into the deep learning model, the process further includes: The chip image is sequentially denoised, white balance corrected, and local contrast enhanced to obtain an enhanced chip image; A single chip region image is cropped from the enhanced chip image using template matching; The single chip region image is normalized to a preset size to obtain a normalized chip image. The angle tilt of the normalized chip image is then corrected by affine transformation to obtain the target chip image for input into the deep learning model.

5. The chip carrier defect detection and sorting method according to claim 1, characterized in that, The process of fusing the chip image with the heatmap to obtain a fused image, and then inputting the fused image into a finely tuned visual semantic understanding model to output a semantic classification and confidence score for the defect, specifically includes: The chip image and the heatmap are fused together by channel splicing or weighted overlay to obtain a fused image containing the original visual information and the model's region of interest information; The fused image is input into a large language vision model that has been fine-tuned with data from the chip manufacturing field. The large language vision model analyzes the fused image and outputs a semantic classification containing standardized defect names, as well as a confidence score corresponding to the semantic classification. The semantic classification is used to describe the defect type of the chip in the natural language detection report, and the confidence score is used to perform hierarchical decision-making and processing.

6. The chip carrier defect detection and sorting method according to claim 1, characterized in that, The step of performing a classification decision and sorting process on the chips based on the confidence score to obtain the sorting result specifically includes: If the confidence score is higher than the first threshold, the chip is determined to be a high-confidence defective chip. The laser drilling mechanism is controlled to physically mark the high-confidence defective chip, and the high-confidence defective chip is removed by the sorting mechanism. If the confidence score is lower than the second threshold, the chip is determined to be a normal chip. The laser drilling mechanism and the sorting mechanism are not started. The normal chip is controlled to pass through the laser drilling station and the sorting station at a constant speed along with the carrier tape to be tested, and is wound up with the carrier tape to be tested. If the confidence score is between the second threshold and the first threshold, the chip is determined to be a suspected defective chip, the location information of the suspected defective chip is recorded, and no further processing is performed.

7. The chip carrier defect detection and sorting method according to claim 1, characterized in that, The incremental fine-tuning of the visual semantic understanding model based on the data in the sample archive specifically includes: A preset fine-tuning trigger signal is established, and the presence of the fine-tuning trigger signal is monitored in real time. The fine-tuning trigger signal includes a sample archive signal, a detection statistics signal, and a manual review signal. When at least one fine-tuning trigger signal is detected, the visual semantic understanding model is updated with parameters based on the data in the sample archive using low-rank adaptation technology to obtain the updated model. The updated model is then subjected to performance verification, and once the performance verification is passed, the updated model is deployed as the current running model.

8. A chip carrier defect detection and sorting system, characterized in that, The chip carrier defect detection and sorting system includes: The interactive control module is used to identify the chip model after receiving a voice command, load the corresponding detection parameters according to the chip model, and control the carrier tape to be tested to move forward at a constant speed on the carrier tape transmission system according to the detection parameters. The image acquisition module is used to monitor the displacement of the carrier tape to be detected in real time through the encoder, obtain a trigger pulse, and synchronously control the high-speed camera to perform exposure based on the trigger pulse to obtain a chip image; The defect identification module is used to input the chip image into a deep learning model, output a heat map of the defect-sensitive area, fuse the chip image and the heat map to obtain a fused image, input the fused image into a fine-tuned visual semantic understanding model, and output the semantic classification and confidence score of the defect. The hierarchical decision module is used to perform hierarchical decision-making and sorting processing on the chips according to the confidence score, and obtain sorting results. The sorting results include high-confidence defective chips that are physically marked and rejected, normal chips that are retained and passed, and suspected defective chips marked as to be retested. The retesting and optimization module is used to retest the suspected defective chip after the carrier tape to be tested is completed, store the retesting results in the sample archive, and perform incremental fine-tuning of the visual semantic understanding model based on the data in the sample archive. The report generation module is used to adjust the semantic classification based on the retest results to obtain the target semantic classification, and generate a natural language detection report for the carrier to be detected based on the target semantic classification.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a chip carrier defect detection and sorting program stored in the memory and executable on the processor. When the chip carrier defect detection and sorting program is executed by the processor, it implements the steps of the chip carrier defect detection and sorting method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a chip carrier defect detection and sorting program, which, when executed by a processor, implements the steps of the chip carrier defect detection and sorting method as described in any one of claims 1-7.

Images