System and method to perform data augmentation for wire bond inspection

Abstract

The present invention provides a method and system (2000) to perform data augmentation for wire bond inspection. More specifically, the present invention provides an input module (2210), configured to perform a step of receiving captured point cloud data of one or more wire bonds of a semiconductor device from a data capturing unit (2100), a synthetic data generation module (2260), configured to perform the step of generating synthetic data for the captured point cloud data, a wire bond defect detection model generation module (2280), configured to perform the step of generating a wire bond defect detection model using the synthetic data, and a wire bond defect detection module (2290), configured to perform the step of detecting one or more wire bond defects within the captured point cloud data using the generated wire bond defect detection model. The synthetic data generation module (2260) generates synthetic data that pertains to any one or a combination wire bonds in a non-defect condition and wire bonds in a defect condition of a plurality of defect categories.

Description

[0001]

[0002] SYSTEM AND METHOD TO PERFORM DATA AUGMENTATION FOR WIRE BOND INSPECTION

[0003] FIELD OF INVENTION

[0004] The present invention relates to wire bond inspection. More specifically, a system and method to perform data augmentation for the implementation of machine learning in a wire bond inspection workflow.

[0005] BACKGROUND OF THE INVENTION

[0006] Wire bonding is the process of creating electrical interconnections between semiconductors (or other integrated circuits) and silicon chips using bonding wires. There are various types of defects could occur during the wire bonding process, such as lost or missing wire(s), curved wire(s), broken wire(s), etc.

[0007] Vision inspection is a popular technique used in the semiconductor industry to inspect these wire bonds. Vision inspection systems and methods conventionally involve the usage of a camerato capture two-dimensional images of the wire bond from a top-down position for its inspection. However, these conventional systems and methods are unable to capture the height information of wire bond. Therefore, they may fail to detect height-related wire bond defects, such as sagging wire and lifting wire. To overcome this, sensors that enable three-dimensional information of the wire bond have been incorporated within recent vision inspection systems and methods. Vision inspection may be further complemented with machine learning techniques to enable the detection of defects from complex and irregular data structure(s) in an automated manner.

[0008] However, there is currently a limited dataset available for implementing machine learning algorithm or techniques for vision inspection of the three-dimensional wire bond data. Machine learning algorithms or techniques require a large amount of data to train a model for it to achieve accurate output. Most data augmentation techniques solely focus on the usage of two-dimensional data and those techniques are unable to generate three-dimensional defect data for the wire bond, such as vertical sagging-wire defect and curved-wire defect.

[0009] There are several prior arts disclosing the implementation of machine learning for the wire bond inspection. Among them include the China Patent Application CN111862064B, which discloses a method to identify defects on silver wires by use of machine learning, more specifically, deep learning. The process involves collecting original images of silver wires, pre-processing the images, extracting statistical characteristics using a defect pre-detection module, and employing secondary defect detection using a convolutional neural network.

[0010] Yet another disclosed technology includes the China Patent Application CN116091404A, which discloses a system and method related to wire bond defect detection and pattern recognition in the wire bond during the manufacturing process for printed circuit boards (PCBs). The invention broadly involves acquiring a depth image of a PCB, judging the welding line contour using an oriented bounding box and meanabsolute-deviation algorithm, extracting depth data, converting it into a point cloud, performing point cloud pre-treatment and surface segmentation for accurate extraction of the wire bond welding line surface, constructing a defect classification model for detecting bonding wire defects, and identifying defect modes.

[0011] None of the above-cited prior arts discloses means to implement data augmentation to generate three-dimensional synthetic wire bond data using a limited dataset. Accordingly, it is desirable for data augmentation of a limited dataset to be implemented to complement machine learning algorithms or techniques for three-dimensional wire bonding inspection.

[0012] SUMMARY OF INVENTION

[0013] The main objective of the invention is to provide a method and system to perform data augmentation for wire bond inspection, especially for a limited dataset that has three- dimensional information on one or more wire bonds. To achieve this objective, the present invention provides a synthetic data generation module that performs data augmentation by generating synthetic data that pertains to any one or a combination of wire bonds in a non-defect condition and wire bonds in a defect condition of a plurality of defect categories based on a limited dataset, the limited dataset being captured three- dimensional point cloud data of one or more wire bonds of a semiconductor device. Advantageously, the present invention enables synthetic data relevant to wire bonds in a non-defect condition and wire bonds in a defect condition to be generated for this data to be used to train a wire bond defect detection model that identifies wire bond defects by taking into account three-dimensional information of an inspected wire bond.

[0014] The present invention intends to provide a method to perform data augmentation for wire bond inspection, characterised in that, the method comprises the steps of receiving, by an input module, captured point cloud data of one or more wire bonds of a semiconductor device, from a data capturing unit, generating, by a synthetic data generation module, synthetic data for the captured point cloud data, generating, by a wire bond defect detection model generation module, a wire bond defect detection model using the synthetic data, and detecting, by a wire bond defect detection module, one or more wire bond defects within the captured point cloud data using the generated wire bond defect detection model. The synthetic data generation module generates synthetic data that pertains to any one or a combination of wire bonds in a non-defect condition and wire bonds in a defect condition of a plurality of defect categories.

[0015] Preferably, the step of generating, by a synthetic data generation module, synthetic data for the received point cloud data, comprises the steps of converting, by a data conversion module, the captured point cloud data into a first set of depth maps, creating, by a data annotation module, an annotation map for one or more regions of each wire bond using the captured point cloud data, calculating, by a wire bond height calculation module, the height of the wire bonds in the annotated map and first set of depth maps, generating, by a synthetic map generation module, the synthetic data for the annotation map and each depth map for the first set of depth maps, based on the calculated height, and retrieving, by a retrieval and selection module, each wire bond from the annotation map and the first set of depth maps for selecting wire bonds that are of a non-defect condition to form reference data.

[0016] Preferably, the step of generating, by a synthetic data generation module, synthetic data for the received point cloud data, further comprises any one or a combination of steps that include generating, by a first synthetic data generation sub-module, synthetic data pertaining to wire bonds that are of a non-defect condition from the depth map, using the reference data, generating, by a second synthetic data generation sub-module, synthetic data pertaining to wire bonds that are of a first defect category where they have at least one sagging-wire defect, using the reference data, generating, by a third synthetic data generation sub-module, synthetic data pertaining to wire bonds that are of a second defect category where they have at least one curved-edge defect, using the reference data, generating, by a fourth synthetic data generation sub-module, synthetic data pertaining to wire bonds that are of a third defect category where they have at least one geometric-variant defect, using the reference data, generating, by a fifth synthetic data generation sub-module, synthetic data pertaining to wire bonds that are of a fourth defect category where they have at least one broken-wire defect, using the reference data, and generating, by a sixth synthetic data generation sub-module, synthetic data pertaining to wire bonds that are of a fifth defect category where they have least one missing-wire defect, using the reference data.

[0017] Preferably, the step of generating, by a synthetic data generation module, synthetic data for the received point cloud data, further comprises the steps of combining, by a synthetic data combiner module, the synthetic data generated by any one or a combination of the synthetic data generation sub-modules for generating a second set of depth maps, generating, by a synthetic point cloud generation module, synthetic point cloud data for the first set of depth maps and the second set of depth maps, and annotating, by the data annotation module, the synthetic point cloud data by generating bounding box data that relate to information of one or more three-dimensional bounding boxes within the synthetic point cloud data.

[0018] Preferably, the step of generating, by a wire bond defect detection model generation module, a wire bond defect detection model using the generated synthetic data comprises the step of training and validating, by a defect detection model training and validation module, the generated wire bond defect detection model using a combination of the captured point cloud data, the synthetic point cloud data, and corresponding bounding box data of the captured point cloud data and the synthetic point cloud data.

[0019] The present invention further intends to provide a system to perform data augmentation for wire bond inspection, characterised in that, the system comprises an input module, configured to receive captured point cloud data of one or more wire bonds of a semiconductor device from a data capturing unit, a synthetic data generation module, configured to generate synthetic data from the captured point cloud data provided by the input module, a wire bond defect detection model generation module, configured to generate a wire bond defect detection model using the synthetic data generated by the synthetic data generation module, and a wire bond defect detection module, configured to detect one or more wire bond defects within the captured point cloud data using the generated wire bond defect detection model generated by the wire bond defect detection model generation module. The synthetic data generation module generates synthetic data that pertains to any one or a combination of wire bonds in a non-defect condition and wire bonds in a defect condition of a plurality of defect categories.

[0020] Preferably, the system further comprises a data conversion module, configured to convert the captured point cloud data into a first set of depth maps, a data annotation module, configured to create an annotation map for one or more regions of each wire bond using the captured point cloud data, a wire bond height calculation module, for calculating the height of the wire bonds in the annotated map and first set of depth maps, a synthetic map generation module, configured to generate the synthetic data for the annotation map and each depth map for the first set of depth maps, based on the calculated height, and a retrieval and selection module, configured to retrieve each wire bond from the annotation map and the first set of depth maps for it to select wire bonds that are of a non-defect condition to form reference data.

[0021] Preferably, the system further comprises a first synthetic data generation sub-module, configured to generate synthetic data pertaining to wire bonds that are of a non-defect condition from the depth map, using the reference data, a second synthetic data generation sub-module, configured to generate synthetic data pertaining to wire bonds that are of a first defect category where they have at least one sagging-wire defect, using the reference data, a third synthetic data generation sub-module, configured to generate synthetic data pertaining to wire bonds that are of a second defect category where they have at least one curved-edge defect, using the reference data, a fourth synthetic data generation sub-module, configured to generate synthetic data pertaining to wire bonds that are of a third defect category where they have at least one geometric-variant defect, using the reference data, a fifth synthetic data generation sub-module, configured to generate synthetic data pertaining to wire bonds that are of a fourth defect category where they have at least one broken-wire defect, using the reference data, and a sixth synthetic data generation sub-module, configured to generate synthetic data pertaining to wire bonds that are of a fifth defect category where they have least one missing-wire defect, using the reference data.

[0022] Preferably, the system further comprises a synthetic data combiner module, configured to combine the synthetic data generated by any one or a combination of the synthetic data generation sub-modules for generating a second set of depth maps, a synthetic point cloud generation module, configured to generate synthetic point cloud data for the first set of depth maps and the second set of depth maps, with the data annotation module further configured to annotate the synthetic point cloud data by generating bounding box data that relate to information of one or more three-dimensional bounding boxes within the synthetic point cloud data.

[0023] Preferably, the system further comprises a defect detection model training and validation module configured to train and validate the generated wire bond defect detection model using a combination of the captured point cloud data, the synthetic point cloud data, and the corresponding bounding box data of the captured point cloud data and the synthetic point cloud data.

[0024] One skilled in the art will readily appreciate that the invention is well adapted to carry out the objects and obtain the ends and advantages mentioned, as well as those inherent therein. The embodiments described herein are not intended as limitations on the scope of the invention.

[0025] BRIEF DESCRIPTION OF DRAWINGS

[0026] These and other features, aspects, and advantages of the present invention will become better understood, when the following detailed description is read with reference to the accompanying drawings in which like characters represent like parts throughout the drawings, wherein:

[0027] FIG. 1 is a flowchart illustrating the process steps to perform defect data augmentation for point cloud-based wire bond inspection in accordance to an embodiment of the present invention.

[0028] FIG. 2 is a flowchart illustrating the process steps to generate synthetic data for the wire bond inspection in accordance to an embodiment of the present invention.

[0029] FIG. 3 is a flowchart illustrating the process steps to acquire the point cloud data in accordance to an embodiment of the present invention.

[0030] FIG. 4 is a flowchart illustrating the process steps to perform sampling on the point cloud data in accordance to an embodiment of the present invention.

[0031] FIG. 5 is a flowchart illustrating the process steps to perform filtering on the point cloud data in accordance to an embodiment of the present invention.

[0032] FIG. 6 is a flowchart illustrating the process steps to perform conversion of the point cloud data into a depth map in accordance to an embodiment of the present invention.

[0033] FIG. 7 is a flowchart illustrating the process steps to create an annotation map for the depth map in accordance to an embodiment of the present invention.

[0034] FIG. 8 is a flowchart illustrating the process steps to calculate a height of the wire bond in accordance to an embodiment of the present invention.

[0035] FIG. 9 is a flowchart illustrating the process steps to retrieve a sample generation number in accordance to an embodiment of the present invention.

[0036] FIG. 10 is a flowchart illustrating the process steps to select and retrieve a wire bond of a non-defect condition (i.e. of a good condition) in accordance to an embodiment of the present invention.

[0037] FIG. 11 is a flowchart illustrating the process steps to generate synthetic data of wire bond of a non-defect condition (i.e. of a good condition) in accordance to an embodiment of the present invention.

[0038] FIGS 12 - 13 are flowcharts that collectively illustrate the process steps to generate synthetic data pertaining to a wire bond having sagging-wire defect in accordance to an embodiment of the present invention.

[0039] FIGS 14 - 16 are flowcharts that collectively illustrate the process steps to generate synthetic data pertaining to a wire bond having curved-edge defect in accordance to an embodiment of the present invention.

[0040] FIGS 17 - 18 are flowcharts that collectively illustrate the process steps to generate synthetic data pertaining to a wire bond having geometric-variant defect in accordance to an embodiment of the present invention.

[0041] FIG. 19 is a flowchart illustrating the process steps to generate synthetic data pertaining to a wire bond having broken-wire defect in accordance to an embodiment of the present invention.

[0042] FIG. 20 is a flowchart illustrating the process steps to generate a missing-wire defect synthetic data in accordance to an embodiment of the present invention.

[0043] FIGS 21 - 22 are flowcharts that collectively illustrate the process steps to combine the generated synthetic data in accordance to an embodiment of the present invention.

[0044] FIG. 23 is a flowchart illustrating the process steps to generate point cloud data in accordance to an embodiment of the present invention.

[0045] FIG. 24 is a flowchart illustrating the process steps to annotate the generated point cloud data using a bounding box in accordance to an embodiment of the present invention.

[0046] FIG. 25 is a flowchart illustrating the process steps to store the synthetic data in a database in accordance to an embodiment of the present invention.

[0047] FIG. 26 is a flowchart illustrating the process steps to generate a wire bond defect detection model in accordance to an embodiment of the present invention.

[0048] FIG. 27 is a flowchart illustrating the process steps to acquire the point cloud data in accordance to an embodiment of the present invention.

[0049] FIG. 28 is a flowchart illustrating the process steps to perform sampling on the point cloud data in accordance to an embodiment of the present invention.

[0050] FIG. 29 is a flowchart illustrating the process steps to perform filtering on the point cloud data in accordance to an embodiment of the present invention.

[0051] FIG. 30 is a flowchart illustrating the process steps to annotate the point cloud data using a bounding box in accordance to an embodiment of the present invention.

[0052] FIG. 31 is a flowchart illustrating the process steps to retrieve the synthetic data in accordance to an embodiment of the present invention.

[0053] FIG. 32 is a flowchart illustrating the process steps to train the generated wire bond defect detection model in accordance to an embodiment of the present invention.

[0054] FIG. 33 is a flowchart illustrating the process steps to store the trained model in accordance to an embodiment of the present invention.

[0055] FIG. 34 is a flowchart illustrating the process steps to detect the wire bond defect using the wire bond defect detection model in accordance to an embodiment of the present invention.

[0056] FIG. 35 is a flowchart illustrating the process steps to retrieve the trained model from the database in accordance to an embodiment of the present invention.

[0057] FIG. 36 is a flowchart illustrating the process steps to acquire the point cloud data in accordance to an embodiment of the present invention.

[0058] FIG. 37 is a flowchart illustrating the process steps to perform sampling on the point cloud data in accordance to an embodiment of the present invention.

[0059] FIG. 38 is a flowchart illustrating the process steps to perform filtering on the point cloud data in accordance to an embodiment of the present invention.

[0060] FIG. 39 is a flowchart illustrating the process steps to execute defect detection and indicate the defect type in accordance to an embodiment of the present invention.

[0061] FIG. 40 is a flowchart illustrating the process steps to store and display the detection result in accordance to an embodiment of the present invention.

[0062] FIG. 41 is a block diagram illustrating a system to perform data augmentation for point cloud-based wire bond inspection in accordance to an embodiment of the present invention.

[0063] DETAILED DESCRIPTION OF THE INVENTION

[0064] The present invention discloses a system and method to perform data augmentation for wire bond inspection. In particular, captured data, which relates to three-dimensional information of one or more wire bonds, undergoes data augmentation for synthetic data to be generated. The synthetic data is to be provided to a machine-learning model that shall be trained to inspect condition of the wire bond within the captured data. The inspected condition of the wire bond may be of a non-defect condition or a defect condition. The defect condition may include a plurality of defect categories. The invention may also be presented in a number of different embodiments with common elements.

[0065] According to the concept of the invention, there is included a plurality of modules operated by a processor that receives captured point cloud data from a capturing unit for generating synthetic data for the synthetic data to be used by a wire bond defect detection model so that it is trained to identify wire bond defects within the captured point cloud data. In particular, the synthetic data that is generated pertains to any one or a combination of wire bonds in a non-defective condition and wire bonds in a defective condition of a plurality of defect categories. Preferably, the synthetic data generated through data augmentation may include five different categories of wire bond defects.

[0066] The invention may generally include three components. The first component involves synthetic data generation of wire bonds to generate multiple types of synthetic defect data that include any one of a combination of vertical sagging-wire defect, horizontal edge sagging-wire defect, missing-wire defect, broken-wire defect, and geometricvariant defect. The second component involves wire bond defect detection model creation via combining captured point cloud data and the synthetic data to train a wire bond defect detection model. The third component involves wire bond defect detection on the captured point cloud data by use of the trained wire bond defect detection model. The invention will now be described in greater detail, by way of example, with reference to the drawings.

[0067] From hereon, one or more flowcharts pertaining to the method of the present invention for training a machine learning model for image classification are to be described. It is noted that the steps described in these flowcharts are not to be interpreted as nonlimiting, and minor modifications to the steps (e.g. additions, omissions, or swaps) are permissible by a skilled person without substantial deviation from as described.

[0068] FIG. 1 illustrates a flowchart describing the process steps, Steps 100, 400, 1000, and 1600, to carry out data augmentation for wire bond inspection, according to an embodiment of the present invention.

[0069] The process steps of FIG. 1 may begin with Step 100, which involves receiving captured point cloud data for one or more wire bond connections on a semiconductor device from a data capturing unit 2100. Preferably, this step is performed by an input module 2210. Preferably, the captured point cloud data received by the input module 2210 relates to the three-dimensional spatial information of the wire bond connection.

[0070] Following Step 100 is Step 400. Step 400 involves generating synthetic data for the captured point cloud data that is provided by the input module 2210. Preferably, this step is performed by a synthetic data generation module 2260.

[0071] Following Step 400 is Step 1000. Step 1000 involves generating a wire bond defect detection model, using the synthetic data or the captured point cloud within the generated synthetic data. The synthetic data may be as previously generated in step 400. Preferably, this step is performed by a wire bond defect detection model generation module 2280.

[0072] Following Step 1OOO is Step 1600. Step 1600 involves detecting wire bond defect(s) within the captured point cloud data, using the wire bond defect detection model previously generated in Step 1000. Preferably, this step is performed by a wire bond defect detection module 2290.

[0073] Whilst not shown, following Step 1600 may be a step of informing defect detection results from the wire bond defect detection model. Preferably, this step is performed by an output module 2300.

[0074] FIG. 2 illustrates a flowchart describing the process steps, Steps 410 to 920, which are carried out to generate synthetic data for wire bond inspection, according to an embodiment of the present invention. The steps in FIG. 2 may be regarded as sub-steps of Step 400 of FIG. 1.

[0075] The steps of FIG. 2 may begin with Step 410, which involves acquiring captured point cloud data from the data capturing unit 2100. Preferably, this step is performed by the input module 2210.

[0076] Following Step 410 is Step 440. Step 440 involves performing a sampling process upon the captured point cloud data. Preferably, this step is performed by a data sampling module 2220.

[0077] Following Step 440 is Step 470. Step 470 involves performing a filtering process upon the captured point cloud data, which had been sampled in Step 440. Preferably, this step is performed by a data filtering module 2230.

[0078] Following Step 470 is Step 500. Step 500 involves converting the captured point cloud data, which had been sampled and filtered in the previous steps. Preferably, this step is performed by a data conversion module 2240.

[0079] Following Step 500 is Step 530. Step 530 involves creating an annotation map for the wire bond region using the captured point cloud data, which had been sampled and filtered in the previous steps. Preferably, this step is performed by a data annotation module 2250.

[0080] Following Step 530 is Step 560. Step 560 involves calculating the height of the wire bond in the annotated map and depth map. Preferably, this step is performed by a wire bond height calculation module 2261.

[0081] Following Step 560 is Step 590. Step 590 involves generating synthetic data for each depth map and the annotation map based on the calculated height. Preferably, this step is performed by a synthetic map generation module 2262.

[0082] Following Step 590 is Step 620. Step 620 involves retrieving wire bonds from the annotation map and the depth map for selecting one or more wire bonds that are of a non-defect condition (i.e. of a good condition) to form reference data. Preferably, this step is performed by a retrieval and selection module 2263.

[0083] Following Step 620 is any one or a combination of steps that include Step 650, Step 680, Step 710, Step 740, Step 770, and Step 800. It is to be noted that this combination of steps may be performed in a sequential manner or a simultaneous manner through appropriate hardware and software configurations.

[0084] In particular, Step 650 involves generating synthetic data pertaining to wire bonds that are of a non-defect condition (i.e. of a good condition) from the depth map, using the reference data. Preferably, this step is performed by a first synthetic data generation sub-module 2264.

[0085] In particular, Step 680 involves generating synthetic data pertaining to wire bonds that are of a first defect category where they have at least one sagging-wire defect, using the reference data. Preferably, this step is performed by a second synthetic data generation sub-module 2265.

[0086] In particular, Step 710 involves generating synthetic data pertaining to wire bonds that are of a second defect category where they have at least one curved-edge defect, using the reference data. Preferably, this step is performed by a third synthetic data generation sub-module 2266.

[0087] In particular, Step 740 involves generating synthetic data pertaining to wire bonds that are of a third defect category where they have at least one geometric-variant defect, using the reference data. Preferably, this step is performed by a fourth synthetic data generation sub-module 2267.

[0088] In particular, Step 770 involves generating synthetic data pertaining to wire bonds that are of a fourth defect category where they have at least one broken-wire defect, using the reference data. data. Preferably, this step is performed by a fifth synthetic data generation sub-module 2268.

[0089] In particular, Step 800 involves generating synthetic data pertaining to wire bonds that are of a fifth defect category where they have at least one missing-wire defect, using the reference data. Preferably, this step is performed by a sixth synthetic data generation sub-module 2269.

[0090] Following the combination of steps that include Step 650, Step 680, Step 710, Step 740, Step 770, and Step 800, is Step 830. Step 830 involves the synthetic data generated by any one of the synthetic data generation modules 2264 - 2269 for generating a second set of depth maps. Preferably, this step is performed by a synthetic data combiner module 2270. With this, synthetic data that relates to wire bonds of a non-defect condition (i.e. in a good condition), and synthetic data that relates to wire bonds in a defect condition that includes any one or a combination of defect categories such as sagging-wire defect, the curved-edge defect, geometric-variant defect, the broken-wire defect, and the missing-wire defect, are generated.

[0091] Following Step 830 is Step 850, which is a decision step whereby it is determined whether or not synthetic data have been generated by all of the synthetic data generation modules 2264 - 2269. Should this be the case, the Step 850 proceeds to Step 860. Else, Step 850 returns to Step 620 and repeats therefrom.

[0092] In particular, Step 860 involves generating the point cloud data for the depth map and a synthetic depth map. Preferably, this step is performed by a point cloud generation module 2271.

[0093] Following Step 860 is Step 890. Step 890 involves annotating the generated point cloud data using a bounding box. Preferably, this step is performed by the data annotation module 2250.

[0094] Finally, following Step 890 is Step 920. Step 920 involves storing the generated point cloud and its corresponding bounding box in a database 2400. Preferably, this step is performed by the synthetic data combination module 2270.

[0095] FIG. 3 illustrates a flowchart describing a process step, Step 411, which is carried out by the input module 2210 to acquire capture point cloud data. More specifically, Step 411 involves acquiring the captured point cloud data from the data capturing unit 2100. Step 411 may be regarded a as sub-step of Step 410 of FIG. 2.

[0096] FIG. 4 illustrates a flowchart describing the process steps, Steps 441 to 443, which are carried out by the data sampling module 2220 to perform a sampling process on the captured point cloud data, according to an embodiment of the present invention. The steps of FIG. 4 may be regarded as sub-steps of Step 440 of FIG. 2.

[0097] The steps of FIG. 4 may begin with Step 441, which involves receiving the captured point cloud data from the input module 2210.

[0098] Following Step 441 is Step 442. Step 442 involves performing the sampling process on the captured point cloud data based on user-defined settings.

[0099] Finally, following Step 442 is Step 443. Step 443 involves storing the sampled point cloud data in the database 2400. Its data file(s) may be labelled as [Sampled3DPointCloud],

[0100] FIG. 5 illustrates a flowchart describing the process steps, Steps 471 to 476, which are carried out to perform a filtering process on the sampled point cloud data, according to an embodiment of the present invention. The steps of FIG. 5 may be regarded as substeps of Step 470 of FIG. 2.

[0101] The steps of FIG. 5 may begin with Step 471, which involves prompting a user on a range of a region of interest in three-dimensions, more specifically, its range along an x-axis [Xmin, Xmax], a y-axis [ymin, ymax], and a z-axis [zmm, Zmax] . Preferably, this step is facilitated by the input module 2210.

[0102] Following Step 471 is Step 472. Step 472 involves evaluating a coordinate position of one point of the sampled point cloud data, in terms of its x-coordinate, y-coordinate, and z-coordinate, by comparing it against the user-defined range of the region of interest as provided by the user in Step 471.

[0103] Following Step 472 is Step 473. Step 473 is a decision step whereby it is determined if the point of the sampled point cloud data as evaluated in Step 472, falls within the user- defined range of the region of interest. Should this be the case, Step 473 returns to Step 472 and repeats therefrom with a different point in the sampled point cloud data being evaluated. Else, Step 473 proceeds to Step 474.

[0104] In Step 474, since it was determined that the point in the sampled point cloud data as evaluated in Step 472 does not fall within the user-defined range of the region of interest, the said point is removed from the sampled point cloud data.

[0105] Following Step 474 is Step 475. Step 475 is a decision step whereby it is determined whether all points in the sampled point cloud data have been checked. Should this be the case, Step 475 proceeds to Step 476. Else, Step 476 returns to Step 472 and repeats therefrom with a different point in the sampled point cloud data being evaluated.

[0106] Finally, in Step 476, since it was determined that all points in the sampled point cloud data have been checked, the step of storing the data as filtered point cloud data is performed. More specifically, the filtered point cloud data is stored in the database 2400. Its data file(s) may be labelled as [Filtered3DPointCloud] in the database 2400.

[0107] FIG. 6 illustrates a flowchart describing the process steps, Steps 501 to 504, which are carried out by the data conversion module 2240 to perform conversion of the filtered point cloud data into a depth map, according to an embodiment of the present invention. The steps of FIG. 6 may be regarded as sub-steps of Step 500 of FIG. 2.

[0108] The steps of FIG. 6 may begin with Step 501, which involves evaluating the format of the captured data loaded into the data conversion module 2240, which may be in the form of a depth map or a point cloud format. Accordingly, the captured data loaded into the data conversion module 2240 is the filtered point cloud data.

[0109] Following Step 501 is Step 502. Step 502 is a decision step whereby it is determined whether or not the captured data as loaded in Step 501 is a depth map. Should this be the case, Step 502 proceeds to Step 504. Else, Step 502 proceeds to Step 503.

[0110] In Step 503, since it was determined that the captured data as loaded in Step 501 is not a depth map, the step of converting the captured data is performed. More specifically, being in a point cloud format, the data has three-dimensional spatial information that includes x-coordinates, y-coordinates, and z-coordinates. This data is converted into a depth map with its x-coordinates corresponding to row indexes, its y-coordinates corresponding to column indexes, and its z-coordinates corresponding to intensity values.

[0111] Finally, Step 504 may follow from Step 502 (should it be determined that the data as acquired in Step 502 is a depth map), or Step 503 (in which a depth map is obtained from the conversion of the point cloud data). In Step 504, the step of storing the depth map in the database 2400 is performed. Its data file(s) may be labelled as [DepthMap] in the database 2400.

[0112] FIG. 7 illustrates a flowchart describing the process steps, Steps 531 to 533, to perform the creation of an annotation map for the wire bond region using the captured point cloud data as performed by the data annotation module 2250, according to an embodiment of the present invention. The steps of FIG. 7 may be regarded as sub-steps of Step 530 of FIG. 2.

[0113] The steps of FIG. 7 may begin with Step 531, which involves creating an annotation map, which is preferably dimensionally similar with the same size as the depth map.

[0114] Following Step 531 is Step 532. Step 532 involves prompting the user to add annotation information for the wire bond region on the annotation map by setting the wire bond region to have a unique colouration. By way of example, a wire bond with a non-defect condition (i.e. of a good condition) may be set to have its region coloured green, a wire bond with a defect condition of a first category may be set to have its region coloured red, a wire bond with a defect condition of a second category may be set to have its region coloured yellow, etc. Moreover, the background of the wire bond region may be coloured black.

[0115] Finally, following Step 532 is Step 533. Step 533 involves storing the depth map and the annotation map in the database 2400 in a manner such that they are both paired, substantially linked to each other, or substantially in a relationship with each other. Its data file(s) may be labelled as [AnnotationMap] in the database 2400.

[0116] FIG. 8 illustrates a flowchart describing the process steps, Steps 561 to 570, which are carried out by the wire bond height calculation module 2261 to perform a calculation of the height of the wire bond in the annotated map and the depth map, according to an embodiment of the present invention. The steps of FIG. 8 may be regarded as sub-steps of Step 560 of FIG. 2.

[0117] The steps of FIG. 8 may begin with Step 561, which involves loading all the stored depth maps and annotation maps from the database 2400.

[0118] Following Step 561 is Step 562. Step 562 involves finding or searching each coloured blob or region in each annotation map. Preferably, black blobs or regions are to be excluded from the search.

[0119] Following Step 562 is Step 563. Step 563 involves selecting one coloured blob or region within the annotation maps.

[0120] Following Step 563 is Step 564. Step 564 is a decision step whereby it is determined whether or not all coloured blobs or regions within the annotation maps have been checked. Should this be the case, Step 564 proceeds to Step 565. Else, Step 564 proceeds to Step 568.

[0121] In Step 568, since it was determined that not all coloured blobs or regions within the annotation maps have been checked, a further decision step is carried out whereby it is determined whether or not the selected blob or coloured region includes a wire bond that is of a non-defect condition (i.e. of a good condition). Should this be the case, Step 568 proceeds to Step 569. Else Step 568 returns to Step 563 for another coloured blob or region to be selected.

[0122] In Step 569, since it was determined that the selected blob or coloured region includes a wire bond that is of a non-defect condition (i.e. of a good condition), the step of extracting depth information from the blob or coloured region of the annotation map is performed.

[0123] Following Step 569 is Step 570. Step 570 involves storing the extracted depth information as a wire map in the database 2400. With this, Step 570 returns to Step 563 for another coloured blob or region within the annotation map to be selected.

[0124] In Step 565, since it was determined that all coloured blobs or regions within the annotation maps have been checked, the step of computing the maximum depth for each wire map and storing them separately as a wire loop height is performed.

[0125] Following Step 565 is Step 566. Step 566 involves computing an average of all wire loop heights to obtain an average wire loop height, and further computing a standard deviation of the wire loop height.

[0126] Finally, following Step 566 is Step 567. Step 567 involves storing the average wire loop height and the standard deviation of the wire loop heights in the database 2400.

[0127] FIG. 9 illustrates a flowchart describing the process steps, Steps 591 to 594, to perform the generation of synthetic data for each depth map and the annotation map based on the calculated height as performed by a synthetic map generation module 2262, according to an embodiment of the present invention. The steps of FIG. 9 may be regarded as sub-steps of Step 590 of FIG. 2.

[0128] The process steps of FIG. 9 may begin with Step 591, which involves loading all the stored depth maps and annotation maps from the database 2400.

[0129] Following Step 591 is Step 592. Step 592 involves prompting the user to input a sample generation number parameter that pertains to the number of synthetic data (in the form of depth maps) to be generated. This parameter may be labelled as [SampleGenerationNumber] .

[0130] Following Step 592 is Step 593. Step 593 involves retrieving each pair of depth map and annotation map from the database 2400.

[0131] Finally, following Step 593 is Step 594. Step 594 involves repeating the synthetic data generation process according to the sample generation number parameter.

[0132] FIG. 10 illustrates a flowchart describing the process steps, Steps 621 to 625, which are performed by the retrieval and selection module 2263 to check whether or not each wire bond connection from the annotation map and the depth map is of a non-defect condition (i.e. of a good condition), according to an embodiment of the present invention. The steps of FIG. 10 may be regarded as sub-steps of Step 620 of FIG. 2.

[0133] The process steps of FIG. 10 may begin with Step 621, which involves retrieving each pair of stored depth maps and annotation maps from the database 2400.

[0134] Following Step 621 is Step 622. Step 622 involves checking each wire bond from the annotation map.

[0135] Following Step 622 is Step 623. Step 623 involves checking the condition of the wire bond in the depth map to determine that it is of a non-defect condition (i.e. of a good condition).

[0136] Following Step 623 is Step 624. Step 624 is a decision step whereby it is determined whether or not the condition of the in depth map / annotation map is of a non-defect condition (i.e. of a good condition). Should this be the case, Step 624 proceeds to Step 625. Else, Step 624 returns to Step 623 and repeats therefrom.

[0137] Finally, following Step 624 is Step 625. Step 625 involves generating a modification type information to be part of the information pertaining to the wire bond in the depth map / annotation map. More specifically, a random integer value, preferably between 0 to 5, is computed to decide the modification type information, whereby by way of example, “0” represents a wire bond of anon-defect condition (i.e. in a good condition), “1” represents a wire bond in a defect condition being a sagging-wire defect, “2” represents a wire bond in a defect condition being a curved-edge defect, “3” represents a wire bond in a defect condition being a geometric-variant defect, “4” represents a wire bond in a defect condition being a broken wire defect, and “5” represents a wire bond in a defect condition being a missing wire defect.

[0138] FIG. 11 illustrates a flowchart describing the process steps, Steps 651 to 654, which are performed by the first synthetic data generation sub-module 2264 to generate synthetic data pertaining to wire bonds that are of a non-defect condition (i.e. of a good condition) from the depth map. The steps of FIG. 11 may be regarded as sub-steps of Step 650 of FIG. 2.

[0139] The process steps of FIG. 11 may begin with Step 651, which involves retrieving wire bonds that are of a non-defect condition (i.e. of a good condition) from the depth map.

[0140] Following Step 651 is Step 652. Step 652 involves computing a z-axis translation value by multiplying a random real number between -1 and 1 with the standard deviation of the wire loop heights.

[0141] Following Step 652 is Step 653. Step 653 involves translating the coordinates of the wire bond by the z-axis translation value.

[0142] Following Step 653 is Step 654. Step 654 involves storing the synthetically generated wire bond that has a non-defect condition (i.e. of a good condition) into a modification list.

[0143] FIGS 12 - 13, in combination, illustrate a flowchart describing the process steps, Steps 681 to 696, which are performed by the second synthetic data generation sub-module 2265 to generate synthetic data pertaining to wire bonds that are of a first defect category where they have a sagging-wire defect, according to an embodiment of the present invention. The steps of FIGS 12 - 13 may be regarded as sub-steps of Step 680 of FIG. 2

[0144] The steps of FIGS 12 - 13 may begin with Step 681, which involves retrieving wire bonds that are of a non-defect condition (i.e. of a good condition) from the depth map.

[0145] Following Step 681 is Step 682. Step 682 involves dividing the regions of the wire bonds having a non-defect condition (i.e. of a good condition) based on a user-defined range along the y-axis.

[0146] Following Step 682 is Step 683. Step 683 involves getting all of the y-axis coordinate values from the divided range, and storing them into a node list.

[0147] Following Step 683 is Step 684. Step 684 involves computing random height values through a product of three parameters that include (a) a value ranging from -1 to 1, (b) a user-defined reject ratio value, and (c) the standard deviation of the wire loop heights. Upon computation of the random height values, they are stored in the database 2400.

[0148] Following Step 684 is Step 685. Step 685 involves creating a curved line through a curve fitting function through the use of the y-axis coordinate values and the random height values.

[0149] Following Step 685 is Step 686. Step 686 involves performing a checking or an evaluation on the maximum or largest random height value, and the minimum or smallest random height value along the curved line, by comparing them against a user- defined limit value.

[0150] Following Step 686 is Step 687. Step 687 is a decision step whereby it is determined whether or not the height values that were checked in step 686 reach the user-defined limit value. Should this be the case, Step 687 proceeds to Step 689. Else, Step 687 proceeds to Step 688.

[0151] In Step 688, since it was determined that the height values that were checked in Step 686 did not reach the user-defined limit value, the curved line previously created in Step 685 is removed, and Step 688 returns to Step 684 and repeats therefrom.

[0152] In Step 688, since it was determined that the height values that were checked in Step 686 reach the user-defined limit value, the step of extracting the regions of the wire bond of a non-defect condition (i.e. of a good condition) from the depth map is performed.

[0153] Following Step 680 is Step 690. Step 690 involves computing a random value parameter, which is preferably between 0 to 1, to decide on a vertical augmentation or a horizontal augmentation. Preferably, should this random value parameter be below 0.5, then a vertical augmentation is to be performed. Else, should this random value parameter be above or equal to 0.5, then a horizontal augmentation is to be performed.

[0154] Following Step 690 is Step 691. Step 691 is a decision step whereby it is determined whether the random value parameter from Step 690 is below 0.5. Should this be the case, then Step 691 proceeds to Step 692. Else, Step 691 proceeds to Step 694.

[0155] In Step 692, since it was determined that the second random value parameter from Step 690 is smaller than 0.5, the step of expanding the curved line along the depth axis is performed. The expanded curved line is then stored in the database 2400.

[0156] Following Step 692 is Step 693. Step 693 involves performing an addition operation to impose the expanded curved line on regions of the wire bond that are of a non-defect condition (i.e. of a good condition). With this, a sagging- wire defect is created.

[0157] In Step 694, since it was determined that the second random value parameter from Step 690 is larger than or equal to 0.5, the step of drawing a curved line in a shifting map is performed. Preferably, the shifting map has the same size as the wire region.

[0158] Following Step 694 is Step 695. Step 695 involves shifting the wire region in the shifting map according to the curved line. With this, a sagging-wire defect is created.

[0159] Finally, following Step 693 or Step 695 is Step 696. Step 696 involves storing the sagging-wire defects created in Step 693 or Step 695 into the modification list, which is subsequently stored in the database 2400.

[0160] [

[0161] FIGS 14 - 16, in combination, illustrate a flowchart describing the process steps, Steps 711 to 733, which are carried out by the third synthetic data generation sub-module 2266 to generate synthetic data pertaining to wire bonds that are of a second defect category where they have at least one curved-edge defect. The steps of FIGS 14 - 16 may be regarded as sub-steps of Step 710 of FIG. 2.

[0162] The steps of FIGS 14 - 16 may begin with Step 711, which involves retrieving wire bonds that are of a non-defect condition (i.e. of a good condition) from the depth map.

[0163] Following Step 711 is Step 712. Step 712 involves creating a shifting map that has the same width size and height size as the depth map.

[0164] Following Step 712 is Step 713. Step 713 involves computing a random value parameter that is between 0 and 1. Preferably, this random value parameter will influence the curved-edge defect occurring on the upper edge or a lower edge of the wire bond.

[0165] Following Step 713 is Step 714. Step 714 is a decision step whereby it is determined whether or not the random value parameter computed in Step 713 is smaller than 0.5. Should this be the case, Step 714 proceeds to Step 715. Else, Step 714 proceeds to Step 716

[0166] In Step 715, since it was determined that the random value parameter computed in Step

[0167] 713 is smaller than 0.5, the step of computing an updated y-axis coordinate value is performed. More specifically, the computation involves a product of three parameters that include (a) a user-defined range, (b) a random value between 0 to 1 , and (c) a height value.

[0168] In Step 716, since it was determined that the random value parameter computed in Step 713 is larger than 0.5, the step of computing an updated y-axis coordinate value is performed. More specifically, the computation involves a difference of two parameters that include (a) the height value, and (b) a product of three sub-parameters that include (b(i)) a user-defined range, (b(ii)) a random value between 0 to 1, and (b(iii)) a height value.

[0169] Following Step 715 or Step 717 is Step 717. Step 717 involves computing a second random value parameter. Preferably, the second random value parameter will influence the curved-edge defect occurring on a left edge or a right edge of the wire bond.

[0170] Following Step 717 is Step 718. Step 718 is a decision step whereby it shall be determined whether or not the second random value parameter computed in Step 717 is below 0.5. Should this be the case, Step 718 proceed to Step 719. Else, should this not be the case, Step 718 proceeds to Step 721.

[0171] In Step 719, since it was determined that the second random value parameter is smaller than 0.5, the step of computing an updated x-axis coordinate value is performed. More specifically, the computation involves subtraction of two parameters that include (a) a half of a width value, and (b) a product of sub-parameters that include (b(i)) a user- defined range, (b(ii)) a random value that is between 0 to 1, and (b(iii)) the width value.

[0172] Following Step 719 is Step 720. In Step 720, the step of setting a shift direction parameter to the left is performed.

[0173] In Step 721, since it was determined that the second random value parameter is larger or equal to 0.5, the step of computing an updated x-axis coordinate value is performed. More specifically, the computation involves addition of two parameters that include (a) half of a width value, and (b) a product of sub-parameters that include (b(i)) a user- defined range, (b(ii)) a random value that is between 0 to 1, and (b(iii)) the width value.

[0174] Following Step 721 is Step 722. In Step 722, the step of setting a shift direction parameter to the right is performed.

[0175] Following Step 720 or Step 722 is Step 723. In Step 723, the step of drawing a triangle within a shifting map is performed. Preferably, the triangle is drawn to have its three vertices located within a two-dimensional coordinate space that includes a first coordinate position that is located at (half the width value, 0), a second coordinate position that is located at (half the width value, height), and a third coordinate position that is located at (updated x-axis coordinate value, updated y-axis coordinate value).

[0176] Following Step 723 is Step 724. Step 724 involves filling up the drawn triangle with white pixels.

[0177] Following Step 724 is Step 725. Step 725 involves computing a random value that is between 0 and 1. Preferably, should this random value be smaller than 0.5, then a vertical augmentation is to be performed. Else, should this random value be larger than or equal to 0.5, then a horizontal augmentation is to be performed.

[0178] Following Step 725 is Step 726. Step 726 is a decision step whereby it is determined whether or not the random value computed in Step 725 is smaller than 0.5. Should this be the case, Step 726 proceeds to Step 727. Else, Step 726 proceeds to Step 729.

[0179] In Step 726, since it was determined that the random value computed in Step 725 is smaller than 0.5, then the step of expanding the filled triangle along the depth axis is performed. The expanded filled triangle may be stored in the database 2400.

[0180] Following Step 727 is Step 728. In Step 728, the step of performing an addition operation to impose the expanded filled triangle created in step 726 upon the region of the wire bond having a non-defect condition (i.e. of a good condition) is performed. With this, a curved-edge defect is created.

[0181] In Step 729, since it was determined that the random value computed in Step 726 is not smaller than 0.5, then the step of parsing each row in the triangle map is performed. More specifically, one row of the triangle map is to be selected.

[0182] Following step 729 is Step 730. Step 730 involves summing the white pixels within the selected row to obtain a white pixel count.

[0183] Following Step 730 is Step 731. Step 731 involves shifting the columns of the wire map on the same row based on the white pixel count and shifting direction as determined previously in Step 720 or Step 722. With this, a curved-edge defect may be formed.

[0184] Following Step 731 is Step 732. Step 732 is a decision step whereby it is determined whether or not the final row of the triangle map has been reached. Should this be the case, Step 731 proceeds to Step 733. Else, Step 732 returns to Step 729 and repeats therefrom with a subsequent row in the triangle map being selected.

[0185] Finally, following Step 728 or Step 732 is Step 733. Step 733 involves storing the curved-edge defect created in Step 728 or Step 732 into the modification list.

[0186] FIGS 17 - 18, in combination, illustrate a flowchart describing the process steps, Steps

[0187] 741 to 749, which are carried out by the fourth synthetic data generation sub-module 2267 to generate synthetic data pertaining to wire bonds that are of a third defect category where they have at least one geometric-variant defect. The steps of FIGS 17 - 18 may be regarded as sub-steps of Step 740 of FIG. 2.

[0188] The steps of FIGS 17 - 18 may begin with Step 741, which involves retrieving wire bonds that are of a non-defect condition (i.e. of a good condition) from the depth map.

[0189] Following Step 741 is Step 742. Step 742 involves computing coordinates of a centre of the wire bond within the depth map along the x-axis and the y-axis. They are stored as centre coordinate values [Cx, Cy],

[0190] Following Step 742 is Step 743. Step 743 involves computing a defect angle through a product of parameters that include (a) a random real number between -1 and 1, and (b) an angle from a randomly selected from a user-defined range of angles.

[0191] Following Step 743 is Step 744. Step 744 involves computing a defect translation coordinates through a product of parameters that include (a) a random real number between -1 and 1, and (b) a set of coordinate values (dx,dy,dz) randomly selected from a user-defined range of values.

[0192] Following Step 744 is Step 745. Step 745 involves computing a defect scaling ratio through a product of parameters that include (a) a random real number between -1 and 1, and (b) a value randomly selected from a user-defined range of scaling values.

[0193] Following Step 745 is Step 746. Step 746 involves rotating the wire bond that is of a non-defect condition (i.e. of a good condition) per the defect angle that was previously computed in step 743 with respect to the centre location that was previously computed in step 742. This modification is further stored into geometric-variant defect.

[0194] Following Step 746 is Step 747. Step 747 involves further scaling the wire bond using the scaling ratio that was previously computed in Step 745. This modification is further stored into geometric-variant defect.

[0195] Following Step 747 is Step 748. Step 748 involves further translating the wire bond using the defect translational range previously computed in Step 744. This modification is further stored into geometric-variant defect.

[0196] Finally, following Step 748 is Step 749. Step 749 involves storing the geometric-variant defects into the modification list.

[0197] FIG. 19 illustrates a flowchart describing the process steps, Steps 771 to 776, which are carried out by the fifth synthetic data generation sub-module 2268 to generate synthetic data pertaining to wire bonds that are of a fourth defect category where they have at least one broken- wire defect. The steps of FIG 19 may be regarded as sub-steps of Step 770 of FIG. 2.

[0198] The steps of FIG. 19 may begin with Step 771, which involves retrieving wire bonds that are of a non-defect condition (i.e. of a good condition) from the depth map.

[0199] Following Step 771 is Step 772. Step 772 involves creating an empty rectangle mask that has the same width and height as the blob image.

[0200] Following Step 772 is Step 773. Step 773 involves computing a random y-axis coordinate position through multiplication of parameters that include a first parameter being a random value between 0 and 1, and a second parameter being the wire loop height.

[0201] Following Step 773 is Step 774. Step 774 involves computing broken range value through a product of parameters that include (a) a user-defined range, (b) a random real value between 0 to 0.5, and (c) the wire loop height of the wire in a non-defect condition (i.e. of a good condition).

[0202] Following Step 774 is Step 775. Step 775 involves using the computed y-axis coordinate position and the broken range value to remove the wire region.

[0203] Finally, following Step 775 is Step 776. Step 776 involves storing the broken-wire defect into the modification list.

[0204] FIG. 20 illustrates a flowchart describing the process steps, Steps 801 to 803, which are carried out by the sixth synthetic data generation sub-module 2269 to generate synthetic data pertaining to wire bonds that are of a fifth defect category where they have at least one missing-wire defect. The steps of FIG 20 may be regarded as substeps of Step 800 of FIG. 2.

[0205] The steps of FIG. 20 may begin with Step 801, which involves retrieving wire bonds that are of a non-defect condition (i.e. of a good condition) from the depth map.

[0206] Following Step 801 is Step 802. Step 802 involves inverting the wire bond with a non- defect condition (i.e. of a good condition) to generate a remove mask map. In particular, in this map, the regions of the wire bond are black, while the regions without the wire bond are white. Data files of the mask removal map may be labelled as [RemoveMask],

[0207] Finally, following Step 802 is Step 803. Step 803 involves storing the remove mask map as a missing wire defect into the modification list.

[0208] FIGS 21 - 22, in combination, illustrate a flowchart describing the process steps, Steps

[0209] 831 to 845, which are carried out by the synthetic data combination module 2270 to generate synthetic data for generating the synthetic depth map. The steps of FIGS 21 - 22 may be regarded as sub-steps of Step 830 of FIG. 2.

[0210] The steps of FIGS 21 - 22 may begin with Step 831, which involves cloning the depth map to a new map for it to be a synthetic depth map. In particular, the data file(s) of the synthetic depth map may be labelled as [SynDepthMap],

[0211] Following Step 831 is Step 832. Step 832 involves inverting the annotation map to create another remove mask map. This remove mask map is then combined with the synthetic depth map through an AND operation to remove all wire area in the synthetic depth map.

[0212] Following Step 832 is Step 833. Step 833 involves evaluating each wire bond in the modification lists. More specifically, one wire bond is to be selected for the subsequent processes.

[0213] Following Step 833 is Step 834. Step 834 is a decision step whereby it is determined whether or not the wire bond is passable, wherein the wire bond is in a non-defect condition (i.e. in a good condition). Should this be the case, Step 834 proceeds to Step 835. Else, Step 834 proceeds to Step 836.

[0214] In Step 836, since it was determined that the wire bond is not passable, i.e. the wire bond is in a defect condition, yet another decision step occurs whereby it is determined whether or not the wire bond has a broken-wire defect. Should this be the case, Step 836 proceeds to Step 835. Else, Step 836 proceeds to Step 837.

[0215] Step 835 may follow from either one of Step 834 or Step 835. Step 835 involves performing an addition operation to augment a modified wire map within the synthetic depth map. With this, Step 835 may proceed to Step 833 for another wire bond in the modification list to be selected.

[0216] In Step 837, since it was determined that the wire bond does not have a broken-wire defect, yet another decision step occurs whereby it is determined whether or not the wire bond has a missing-wire defect. Should this be the case, Step 837 proceeds to Step 833 for another wire bond in the modification list to be selected. Else, Step 837 proceeds to Step 838.

[0217] In Step 838, since it was determined that the wire bond does not have a missing-wire defect, yet another decision step occurs whereby it is determined whether or not the wire bond has a geometric-variant defect. Should this be the case, Step 838 proceeds to Step 841. Else, Step 838 proceeds to Step 839.

[0218] In Step 839, since it was determined that the wire bond does not have a geometric variant defect, yet another decision step occurs whereby it is determined whether or not the wire bond has a sagging-wire defect. Should this be the case, Step 839 proceeds to Step 841. Else, Step 839 proceeds to Step 840.

[0219] In Step 840, since it was determined that the wire bond does not have a sagging-wire defect, yet another decision step occurs whereby it is determined whether or not the wire bond has a curved-edge defect. Should this be the case, Step 840 proceeds to Step 841. Else, Step 839 proceeds to Step 844.

[0220] Step 841 may follow from any one of Step 838, Step 839, or Step 840 should the wire bond be deemed to have a geometric-variant defect, a sagging-wire defect, or a curved- edge defect. Step 841 involves creating a remove mask map by inverting the wire bond.

[0221] Following Step 841 is Step 842. Step 842 involves performing a bitwise AND operation with the remove mask map created previously in step 832 to remove the augmented region on the synthetic depth map.

[0222] Following Step 842 is Step 843. Step 843 involves performing an ADD operation to augment the modified wire map on the synthetic depth map. With this Step 843 may proceed to Step 844.

[0223] Step 844 may follow from any one of Step 840 or Step 843. Step 844 is yet another decision step whereby it is determined whether or not the checking of each wire in the modification list is finished. Should this be the case, Step 844 proceeds to Step 845. Else, Step 844 returns to Step 833 for yet another wire in the modification list to be selected for the subsequent processes.

[0224] The final step may be Step 845. In Step 845, since it was determined that the checking of each wire in the modification list is finished, the synthetic depth map is stored in the database 2400.

[0225] FIG. 23 illustrates a flowchart describing the process steps, Steps 861 to 862, which are carried out by the point cloud generation module 2271 to generate point cloud data for the depth map and the synthetic depth map. The steps of FIG 23 may be regarded as sub-steps of Step 860 of FIG. 2.

[0226] The steps of FIGS 23 may begin with Step 861. Step 861 involves loading the depth maps and the synthetic depth maps.

[0227] Finally, following Step 861 is Step 862. Step 862 involves generating coordinate points in the three-dimensional space for each value on the depth map. In particular, for each coordinate point, its x-coordinate corresponds to column of the depth map, its y- coordinate corresponds to the row of the depth map, and its z-coordinate corresponds to the height values of the depth map. With this, the generated point clouds are stored separately or as separate sets within the database 2400.

[0228] FIG. 24 illustrates a flowchart describing the process steps, Steps 891 to 892, which are carried out by the data annotation module 2250 to annotate the generated point cloud data using a three-dimensional bounding box. The steps of FIG 24 may be regarded as sub-steps of Step 890 of FIG. 2.

[0229] The steps of FIG 24 may begin with Step 891. Step 891 involves loading all of the stored point cloud data from the database 2400.

[0230] Finally, following Step 891 is Step 892. Step 892 involves prompting the user to manually label each wire bond through the use of three-dimensional bounding boxes. More specifically, each bounding box shall include (a) a centre in terms of x-coordinate, y-coordinate, and z-coordinate, (b) dimensions in terms of length, width and height, (c) rotation along the z-axis, and (d) the classification of its defect category.

[0231] FIG. 25 illustrates a flowchart describing the process step, Step 921, which is carried out by the synthetic data combination module 2270 to store generated point cloud data and its corresponding bounding box in a database 2400. This step of FIG 25 may be regarded as a sub-step of Step 920 of FIG. 2.

[0232] FIG. 26 illustrates a flowchart describing the process steps, Steps 1010 to 1190, which are carried out by the wire bond defect detection model generation module 2280 to generate the wire bond defect detection model. The steps of FIG. 26 may be regarded as sub-steps of Step 1000 of FIG. 1.

[0233] The steps of FIG 26 may begin with Step 1010. Step 1010 involves acquiring the captured point cloud data from the data capturing unit 2100 via the input module 2210.

[0234] Following Step 1010 is Step 1040. Step 1040 involves performing a sampling process upon the captured point cloud data. Preferably, this step is performed by the data sampling module 2220.

[0235] Following Step 1040 is Step 1070. Step 1070 involves performing a filtering process upon the sampled captured point cloud data. Preferably, this step is performed by the data filtering module 2230.

[0236] Following Step 1070 is Step 1090. Step 1090 involves converting the filtered point cloud data into a depth map. Preferably, this step is performed by the data conversion module 2240.

[0237] Following Step 1090 is Step 1100. Step 1100 involves annotating the depth map using the three-dimensional bounding box. Preferably, this step is performed by the data annotation module 2250.

[0238] Furthermore, there is Step 1130, which may be performed in a sequential or simultaneous manner. More specifically, Step 1130 involves retrieving the synthetic data from the database 2400. Preferably, this step is performed by a synthetic data retrieval module.

[0239] Step 1160 follows from both Step 1100 and Step 1130. More specifically, Step 1160 is performed after both Step 1100 and Step 1130 have been performed. Step 1160 involves training a wire bond defect detection model. Preferably, this step is performed by a defect detection model training and validation module 2281.

[0240] Finally, following Step 1160 is Step 1190. Step 1190 involves storing the trained model in the database 2400.

[0241] FIG. 27 illustrates a flowchart describing a process step, Step 1011, which is regarded to be a sub-step of Step 1010 of FIG. 26. In particular, Step 1011 involves acquiring the point cloud data from the data capturing unit 2100 via input module 2210.

[0242] FIG. 28 illustrates a flowchart describing the process steps, Steps 1041 to 1043, which are carried out by the data sampling module 2220 to perform a sampling process on the captured data. The steps of FIG. 28 may be regarded as sub-steps of Step 1040 of FIG.

[0243] 26.

[0244] The steps of FIG. 28 may begin with Step 1041. Step 1041 involves loading the inputted point cloud data into the data sampling module 2220.

[0245] Following Step 1041 is Step 1042. Step 1042 involves performing sampling on data points of the received point cloud data based on a user-defined setting.

[0246] Finally, following step 1042 is step 1043. Step 1043 involves storing the sampled point cloud data in the database 2400. Its data file(s) may be labelled as [Sampled3DPointCloud] in the database 2400.

[0247] FIG. 29 illustrates a flowchart describing the process steps, Steps 1071 to 1076, which are carried out by the data filtering module 2230 to perform a filtering process on the sampled data. Preferably, these steps are performed by the data filtering module 2230. The steps of FIG. 29 may be regarded as sub-steps of Step 1070 of FIG. 26. Furthermore, the steps of FIG. 29 may be similar to FIG. 5.

[0248] The steps of FIG. 29 may begin with Step 1071. Step 1071 involves prompting a user on a range of a region of interest in three-dimensions, more specifically, its range along an x-axis [xmin, Xmax], ay-axis [ymin, ymax], and a z-axis [zmin, Zmax]. Preferably, this step is facilitated by the input module 2210.

[0249] Following Step 1071 is Step 1072. Step 1072 involves evaluating each point in the point cloud data in terms of its x-coordinate, y-coordinate, and z-coordinate. More specifically, one point is to be selected for evaluation.

[0250] Following Step 1072 is Step 1072. Step 1073 is a decision step whereby it is determined whether or not the point in the point cloud is within the range of the region of interest as defined in Step 1071. Should this be the case, step 1073 returns to Step 1072 and repeats therefrom with a different point being selected. Else, Step 1073 proceeds to Step 1074

[0251] In Step 1074, since it was determined that the point was not within the range of the region of interest as defined in Step 1071, the step of removing the said point was performed.

[0252] Following Step 1074 is Step 1075. Step 1075 is yet another decision step whereby it is determined whether there are still points in the point cloud data that have not been checked. Should this be the case, Step 1075 returns to Step 1072 and repeats therefrom with a different point being selected. Else, Step 1075 proceeds to Step 1076.

[0253] Finally, in Step 1076, since it was determined that all points in the point cloud data have been checked, the step of storing the filtered data in the database 2400 is performed. Its data file(s) may be labelled as [Filtered3DPointCloud] in the database 2400.

[0254] FIG. 30 illustrates a flowchart describing the process steps, Steps 1101 to 1102, which are carried out by the data annotation module 2250 to annotate point cloud data using the bounding box. The steps of FIG. 30 may be regarded as sub-steps of Step 1100 of

[0255] FIG. 26

[0256] The steps of FIG. 30 may begin with Step 1101. Step 1101 involves loading all of the stored point cloud data into the data annotation module 2250.

[0257] Finally, following Step 1101 is Step 1102. Step 1102 involves prompting the user to label the three-dimensional bounding box in a manual manner for each wire bond. Preferably, each 3D bounding box includes (a) a centre parameter in terms of x- coordinate, y-coordinate, and z-coordinate, (b) a size parameter in terms of length, width, and height, (c) a rotation parameter in terms of rotation along the z-axis, and (iv) a classification category.

[0258] FIG. 31 illustrates a flowchart describing the process step, Step 1131, which is carried out by the synthetic data retrieval module to retrieve the synthetic data and its corresponding bounding box information from the database 2400. The step of FIG. 31 may be regarded as a sub-step of Step 1130 of FIG. 26.

[0259] FIG. 32 illustrates a flowchart describing the process steps, Steps 1161 to 1163, which are carried out by the defect detection model training and validation module 2281 to train the wire bond defect detection model. The steps of FIG. 32 may be regarded as sub-steps of Step 110 of FIG. 26.

[0260] The steps of FIG. 32 may begin with Step 1161. Step 1161 involves combining a current sensor data that is the point cloud data and the stored synthetic point cloud, together within their corresponding three-dimensional bounding box information.

[0261] Following Step 1161 is Step 1162. Step 1162 involves dividing the combined point cloud data and its three-dimensional bounding box data into a training dataset and a validation dataset based on a user-defined ratio.

[0262] Finally, following Step 1162 is Step 1163. Step 1163 involves training a defect detection model through the use of machine learning or deep learning techniques or algorithms, using the training dataset and the validation dataset.

[0263] FIG. 33 illustrates a flowchart describing the process step, Step 1191, which is carried out by the synthetic data retrieval module. The step of FIG. 33 may be regarded as a sub-step of Step 1190 of FIG. 26. In particular, Step 1191 involves storing the training model that has the highest validation accuracy in the database 2400.

[0264] FIG. 34 illustrates a flowchart describing the process steps, Steps 1610 to 1760, which are carried out by the wire bond defect detection module 2260 to detect wire bond defects using the generated wire bond detection model. The steps of FIG. 34 may be regarded as sub-steps of Step 1600 of FIG. 1.

[0265] The steps of FIG. 34 may begin with Step 1610. Step 1610 involves retrieving the trained defect detection model from the database 2400. Preferably, this step is performed by a training model retrieval module.

[0266] Following Step 1610 is Step 1640. Step 1640 involves acquiring the captured point cloud data from the data capturing unit 2100, via the input module 2210.

[0267] Following Step 1640 is Step 1670. Step 1670 involves performing the sampling process on the captured data. Preferably, this step is performed by the data sampling module 2220.

[0268] Following Step 1670 is Step 1700. Step 1700 involves performing the filtering process on the sampled data. Preferably, this step is performed by the data filtering 2230.

[0269] Following Step 1700 is Step 1730. Step 1730 involves performing the detection of the wire bond defect. Preferably, this step is performed by the trained defect detection model.

[0270] Finally, following Step 1730 is Step 1760. Step 1760 involves storing the detected result in the database 2400, and displaying the result on the output module 2300.

[0271] FIG. 35 illustrates a flowchart describing the process step, Step 1611. Preferably, this step is performed by the wire bond defect detection module 2290. The step of FIG. 35 may be regarded as a sub-step of Step 1610 of FIG. 26. In particular, Step 1191 involves retrieving the trained model from the database 2400, by the wire bond defect detection module 2290.

[0272] FIG. 36 illustrates a flowchart describing the process step, Step 1641. The step of FIG. 36 may be regarded as a sub-step of Step 1640 of FIG. 26. In particular, Step 1191 involves acquiring the point cloud data from the data capturing unit 2100 via the input module 2210.

[0273] FIG. 37 illustrates a flowchart describing the process steps, Steps 1671 to 1673, which are carried out by the data sampling module 2220 to perform sampling on the point cloud data, by the data using the data sampling module 2220. The steps of FIG. 37 may be regarded as sub-steps of Step 1670 of FIG. 1.

[0274] The steps of FIG. 37 may begin with Step 1671. Step 1671 involves retrieving captured point cloud data.

[0275] Following Step 1671 is Step 1672. Step 1672 involves performing a sampling process on the captured point cloud data based on user-defined settings.

[0276] Finally, following Step 1672 is Step 1673. Step 1673 involves storing the sampled point cloud data in the database 2400.

[0277] FIG. 38 illustrates a flowchart describing the process steps, Steps 1701 to 1706, to perform the filtering process on the sampled data. Preferably, these steps are performed by the data sampling module 2230. The steps of FIG. 38 may be regarded as sub-steps of Step 1700 of FIG. 1.

[0278] The steps of FIG. 38 may begin with Step 1701. Step 1701 involves prompting a user on a range of a region of interest in three dimensions, more specifically, its range along an x-axis [xmin, Xmax], ay-axis [ymin, ymax], and a z-axis [zmin, Zmax]. Preferably, this step is facilitated by the input module 2210.

[0279] Following Step 1701 is Step 1702. Step 1702 involves checking each point in the point cloud in terms of its x-coordinate, y-coordinate, and z-coordinate. More specifically, one point in the point cloud is to be selected for checking.

[0280] Following Step 1702 is Step 1703. Step 1703 is a decision step whereby it is determined whether or not the point selected in Step 1702 is within the range of the region of interest as defined in Step 1702. Should it be the case Step 1703 proceeds to Step 1702 and repeats therefrom with another point in the point cloud is to be selected for checking. Else, Step 1703 proceeds to Step 1704.

[0281] In Step 1704, since it was determined that the selected point was not within the range of the range of the region of interest as defined in Step 1702, the said point is removed.

[0282] Following Step 1704 is Step 1705. Step 17-5 is a decision step whereby it is determined whether or not are there points that have not been checked. Should it be the case Step 1705 proceeds to Step 1702 and repeats therefrom with another point in the point cloud is to be selected for checking. Else, Step 1705 proceeds to Step 1706.

[0283] Finally, in Step 1706, since it was determined that all points had been checked, the step of storing the filtered data in the database 2400. Its data file(s) may be labelled as [Filtered3DPointCloud] in the database 2400.

[0284] FIG. 39 illustrates a flowchart describing the process steps, Steps 1731 to 1733, which are carried out by a defect indication module 2291 to perform defect detection and indicate the defect category. The steps of FIG. 39 may be regarded as sub-steps of Step 1730 of FIG. 34

[0285] The steps of FIG. 39 may begin with Step 1731. Step 1731 involves loading the filtered point clouds from the database 2400.

[0286] Following Step 1731 is Step 1732. Step 1732 involves executing the defect detection on the point cloud using the wire bond defect detection model. With this, detection results are obtained.

[0287] Finally, following Step 1732 is Step 1733. Step 1733 involves indicating the detection result by drawing a detected bounding box on a resulting display image.

[0288] FIG. 40 illustrates a flowchart describing the process steps, Steps 1761 to 1762, which are carried out by the defect indication module 2291 to store and display the detection result. The steps of FIG. 40 may be regarded as sub-steps of Step 1760 of FIG. 34.

[0289] The steps of FIG. 40 may begin with Step 1761. Step 1761 involves storing the detected three-dimensional bounding box information into the database 2400.

[0290] Finally, following Step 1761 is Step 1762. Step 1762 involves displaying the detected three-dimensional bounding box information on the output module 2300.

[0291] FIG. 41 is a block diagram of a system 2000 to perform defect data augmentation for point cloud-based wire bond inspection. As shown, the system 2000 comprises the data capturing unit 2100, the processor 2200, the output module 2300, and the database 2400

[0292] Preferably, the data capturing unit 2100 is preferably one or a set of sensors that capture spatial three-dimensional information of the wire bond connections on a semiconductor device arranged under it. To this end, the data capturing unit 2100, by way of example, may be stereo vision cameras, a LiDAR system, or the like.

[0293] Preferably, the processor 2200 is configured to process the three-dimensional data for the wire bond defect detection.

[0294] Preferably, the output module 2300 is configured to display the detection result of the wire bond defect. More specifically, the output module 2300 may be in communication with a human-machine interface for the defect detection results to be displayed thereon.

[0295] Preferably, the database 2400 is configured to store the defect detection model.

[0296] The processor 2200 operates the input module 2210, the data sampling module 2220, the data filtering module 2230, the data conversion module 2240, the data annotation module 2250, the synthetic data generation module 2260, the wire bond defect detection model generation module 2280, and the wire bond defect detection module 2290.

[0297] The input module 2210 is configured to receive the point cloud data from the data capturing unit 2100.

[0298] The synthetic data generation module 2260 is configured to generate the synthetic data for the received point cloud data.

[0299] The wire bond defect detection model generation module 2280 is configured to generate the wire bond defect detection model using the generated synthetic data.

[0300] The wire bond defect detection module 2290 is configured to detect the wire bond defect for the received point cloud data using the generated wire bond defect detection model.

[0301] Further modules include the data sampling module 2220, the data filtering module 2230, the data conversion module 2240, and the data annotation module 2250 are configured to perform data processing tasks with the synthetic data generation module 2260, the wire bond defect detection generation module 2280, and the wire bond defect detection module 2290.

[0302] The processing steps carried out by these modules 2220 - 2250 have been described in detail above with respect to the FIG. 4, FIG. 6, FIG. 7, FIG. 24, FIG. 25, FIG. 28, FIG. 29, FIG. 30, FIG. 37, and FIG. 38

[0303] The synthetic data generation module 2260 comprises the wire bond height calculation module 2261, the synthetic map generation module 2262, the retrieval and selection module 2263, the first synthetic data generation sub-module 2264, the second synthetic data generation sub-module 2265, the third synthetic data generation sub-module 2266, the fourth synthetic data generation sub-module 2267, the fifth synthetic data generation sub-module 2268, the sixth synthetic data generation sub-module 2269, the synthetic data combination module 2270, and synthetic the synthetic point cloud data generation module 2271.

[0304] The wire bond height calculation module 2261 is configured to calculate the height of the wire bond in the annotated map and depth map. The processing steps carried out by the wire bond height calculation module 2261 have been described above with respect to FIG. 8

[0305] The synthetic map generation module 2262 configured to generate the synthetic data for each depth map and the annotation map based on the calculated height. The processing steps carried out by the synthetic map generation module 2262 have been described above with respect to FIG. 9.

[0306] The retrieval and selection module 2263 is configured to check whether each wire bond connection from the annotation map and the depth map is in good condition and to select only the good wire bond connection. The processing steps carried out by the retrieval and selection module 2263 have been described above with respect to FIG. 10

[0307] The first synthetic data generation sub-module 2264 is configured to generate the synthetic data for the good wire bond connection using the selected good wire bond connection. The processing steps carried out by the first synthetic data generation submodule 2264 have been described above with respect to FIG. 11.

[0308] The second synthetic data generation sub-module 2265 is configured to generate the synthetic data for the wire bond with sagging-wire defect using the selected good wire bond connection. The processing steps carried out by the second synthetic data generation sub-module 2265 have been described above with respect to FIGS 12 - 13.

[0309] The third synthetic data generation sub-module 2266 is configured to generate the synthetic data for the wire bond with curved-edge defect using the selected good wire bond connection. The processing steps carried out by the third synthetic data generation sub-module 2266 have been described above with respect to FIGS 14 - 16.

[0310] The fourth synthetic data generation sub-module 2267 is configured to generate the synthetic data for the wire bond with the geometric-variant defect using the selected good wire bond connection. The processing steps carried out by the fourth synthetic data generation sub-module 2267 have been described above with respect to FIGS 17 - 18

[0311] The fifth synthetic data generation sub-module 2268 is configured to generate the synthetic data for the wire bond with broken-wire defect using the selected good wire bond connection. The processing steps carried out by the fifth synthetic data generation sub-module 2268 have been described above with respect to FIG. 19.

[0312] The sixth synthetic data generation sub-module 2269 is configured to generate the synthetic data for the missing wire defect from the depth map using the selected good wire bond connection. The processing steps carried out by the sixth synthetic data generation sub-module 2269 have been described above with respect to FIG. 20.

[0313] The synthetic data combination module 2270 is primarily configured to combine the generated synthetic data on the good wire, the sagging-wire defect, the curved-edge defect, geometric-variant defect, the broken-wire defect, and the missing wire defect for generating the synthetic depth map. The processing steps carried out by the synthetic data combination module 2270 have been described above with respect to FIGS 21 - 22 and FIG. 25.

[0314] The synthetic point cloud generation module 2271 configured to generate the point cloud data for the depth map and the synthetic depth map. The processing steps carried out by the synthetic point cloud generation module 2271 have been described above with respect to FIG. 23.

[0315] The wire bond defect detection model generation module 2280 comprises the defect detection model training and validation module 2281 configured to train and validate the generated wire bond defect detection model using the combination of the point cloud data, synthetic point cloud data and their corresponding bounding box data. The wire bond defect detection generation module 2280 further comprises the synthetic data retrieval module configured to retrieve the synthetic data from the database 2400. The processing steps carried out by the wire bond defect detection model generation module 2280 have been described above with respect to FIGS 26 - 33.

[0316] The wire bond defect detection module 2290 comprises the defect indication module 2291, configured to indicate the detected wire bond defect by using the bounding box. Further, the wire bond defect detection module 2290 comprises the training model retrieval module configured to retrieve the trained defect detection model from the database 2400. The processing steps carried out by the wire bond defect detection module 2290 have been described above with respect to FIGS 34 - 40.

[0317] It should be noted that while the aforementioned modules may be in a software embodiment, they may also be a hardware embodiment where they are directly connected to the processor. Alternatively, these modules may each be an independent computer system. Finally, the processor 2200 may be, for example, a conventional processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), or a combination thereof.

[0318] The present disclosure includes as contained in the appended claims, as well as that of the foregoing description. Although this invention has been described in its preferred form with a degree of particularity, it is understood that the present disclosure of the preferred form has been made only by way of example and that numerous changes in the details of construction and the combination and arrangements of parts may be resorted to without departing from the scope of the invention.

Claims

CLAIMS1. A method to perform data augmentation for wire bond inspection, characterised in that, the method comprises the steps of receiving, by an input module (2210), captured point cloud data of one or more wire bonds of a semiconductor device, from a data capturing unit (2100); generating, by a synthetic data generation module (2260), synthetic data for the captured point cloud data; generating, by a wire bond defect detection model generation module (2280), a wire bond defect detection model using the synthetic data; and detecting, by a wire bond defect detection module (2290), one or more wire bond defects within the captured point cloud data using the generated wire bond defect detection model; wherein the synthetic data generation module (2260) generates synthetic data that pertains to any one or a combination wire bonds in a non-defect condition and wire bonds in a defect condition of a plurality of defect categories.

2. The method according to claim 1, wherein the step of generating, by a synthetic data generation module (2260), synthetic data for the received point cloud data, comprises the steps of converting, by a data conversion module (2240), the captured point cloud data into a first set of depth maps; creating, by a data annotation module (2250), an annotation map for one or more regions of each wire bond using the captured point cloud data; calculating, by a wire bond height calculation module (2261), the height of the wire bonds in the annotated map and first set of depth maps; generating, by a synthetic map generation module (2262), the synthetic data for the annotation map and each depth map for the first set of depth maps, based on the calculated height; andretrieving, by a retrieval and selection module (2263), each wire bond from the annotation map and the first set of depth maps for selecting wire bonds that are of a non-defect condition to form reference data.

3. The method according to claim 2, wherein the step of generating, by a synthetic data generation module (2260), synthetic data for the received point cloud data, further comprises any one or a combination of steps that include generating, by a first synthetic data generation sub-module (2264), synthetic data pertaining to wire bonds that are of a non-defect condition from the depth map, using the reference data; generating, by a second synthetic data generation sub-module (2265), synthetic data pertaining to wire bonds that are of a first defect category where they have at least one sagging-wire defect, using the reference data; generating, by a third synthetic data generation sub-module (2266), synthetic data pertaining to wire bonds that are of a second defect category where they have at least one curved-edge defect, using the reference data; generating, by a fourth synthetic data generation sub-module (2267), synthetic data pertaining to wire bonds that are of a third defect category where they have at least one geometric-variant defect, using the reference data; generating, by a fifth synthetic data generation sub-module (2268), synthetic data pertaining to wire bonds that are of a fourth defect category where they have at least one broken-wire defect, using the reference data; and generating, by a sixth synthetic data generation sub-module (2269), synthetic data pertaining to wire bonds that are of a fifth defect category where they have least one missing-wire defect, using the reference data.

4. The method according to claim 3, wherein the step of generating, by a synthetic data generation module (2260), synthetic data for the received point cloud data, further comprises the steps ofcombining, by a synthetic data combiner module (2270), the synthetic data generated by any one or a combination of the synthetic data generation sub-modules (2264 - 2269) for generating a second set of depth maps; generating, by a synthetic point cloud generation module (2271), synthetic point cloud data for the first set of depth maps and the second set of depth maps; and annotating, by the data annotation module (2250), the synthetic point cloud data by generating bounding box data that relate to information of one or more three- dimensional bounding boxes within the synthetic point cloud data.

5. The method according to claim 4, wherein the step of generating, by a wire bond defect detection model generation module (2280), a wire bond defect detection model using the generated synthetic data comprises the step of training and validating, by a defect detection model training and validation module (2281), the generated wire bond defect detection model using a combination of the captured point cloud data, the synthetic point cloud data, and corresponding bounding box data of the captured point cloud data and the synthetic point cloud data.

6. A system (2000) to perform data augmentation for wire bond inspection, characterised in that, the system comprises an input module (2210), configured to receive captured point cloud data of one or more wire bonds of a semiconductor device from a data capturing unit (2100); a synthetic data generation module (2260), configured to generate synthetic data from the captured point cloud data provided by the input module (2210); a wire bond defect detection model generation module (2280), configured to generate a wire bond defect detection model using the synthetic data generated by the synthetic data generation module (2260); and a wire bond defect detection module (2290), configured to detect one or more wire bond defects within the captured point cloud data using the wire bond defect detection model generated by the wire bond defect detection model generation module(2280); wherein the synthetic data generation module (2260) generates synthetic data that pertains to any one or a combination wire bonds in a non-defect condition and wire bonds in a defect condition of a plurality of defect categories.

7. The system according to claim 6, further comprising a data conversion module (2240), configured to convert the captured point cloud data into a first set of depth maps; a data annotation module (2250), configured to create an annotation map for one or more regions of each wire bond using the captured point cloud data; a wire bond height calculation module (2261), for calculating the height of the wire bonds in the annotated map and first set of depth maps; a synthetic map generation module (2262), configured to generate the synthetic data for the annotation map and each depth map for the first set of depth maps, based on the calculated height; and a retrieval and selection module (2263), configured to retrieve each wire bond from the annotation map and the first set of depth maps for it to select wire bonds that are of a non-defect condition to form reference data.

8. The system according to claim 7, further comprising a first synthetic data generation sub-module (2264), configured to generate synthetic data pertaining to wire bonds that are of anon-defect condition from the depth map, using the reference data; a second synthetic data generation sub-module (2265), configured to generate synthetic data pertaining to wire bonds that are of a first defect category where they have at least one sagging-wire defect, using the reference data; a third synthetic data generation sub-module (2266), configured to generate synthetic data pertaining to wire bonds that are of a second defect category where they have at least one curved-edge defect, using the reference data;a fourth synthetic data generation sub-module (2267), configured to generate synthetic data pertaining to wire bonds that are of a third defect category where they have at least one geometric-variant defect, using the reference data; a fifth synthetic data generation sub-module (2268), configured to generate synthetic data pertaining to wire bonds that are of a fourth defect category where they have at least one broken-wire defect, using the reference data; and a sixth synthetic data generation sub-module (2269), configured to generate synthetic data pertaining to wire bonds that are of a fifth defect category where they have least one missing-wire defect, using the reference data.

9. The system according to claim 8, further comprising a synthetic data combiner module (2270), configured to combine the synthetic data generated by any one or a combination of the synthetic data generation submodules (2264 - 2269) for generating a second set of depth maps; a synthetic point cloud generation module (2271), configured to generate synthetic point cloud data for the first set of depth maps and the second set of depth maps; with the data annotation module (2250) further configured to annotate the synthetic point cloud data by generating bounding box data that relate to information of one or more three-dimensional bounding boxes within the synthetic point cloud data.

10. The system according to claim 9, further comprising, a defect detection model training and validation module (2281) configured to train and validate the generated wire bond defect detection model using a combination of the captured point cloud data, the synthetic point cloud data, and corresponding bounding box data of the captured point cloud data and the synthetic point cloud data.

Images