A method and system for generating a multi-modal defect sample of a BGA package substrate

By preprocessing BGA package substrate images and point clouds, using instance-normalized generative adversarial networks and skip-connected point cloud generation networks, multimodal defect samples are generated, which solves the problem of insufficient defect detection data in existing technologies and achieves efficient defect detection dataset generation.

CN117974621BActive Publication Date: 2025-10-17SOUTH CHINA UNIV OF TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410220903.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-28
Publication Date
2025-10-17
Estimated Expiration
2044-02-28

AI Technical Summary

Technical Problem

Existing technologies are unable to effectively generate multimodal defect samples for BGA package substrates, resulting in a lack of sufficient data for defect detection, especially when the number of defect categories for BGA package substrates is unbalanced and the acquisition cost is high.

Method used

Threshold segmentation and rotation transformation matrix correction are used to process BGA package substrate images and point clouds. Combined with instance-normalized generative adversarial networks and skip-connected point cloud generation networks, multimodal defect samples are generated. The solder ball positions are found through Hough circle transform and small defect images are replaced to achieve defect diversity and authenticity.

Benefits of technology

Rich multimodal defect samples of BGA package substrates were generated, which solved the problem of insufficient defect detection data, reduced collection and annotation costs, and improved the data richness and accuracy of defect detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117974621B_ABST
    Figure CN117974621B_ABST
Patent Text Reader

Abstract

The application discloses a kind of BGA package substrate multimodal defect sample generation method and system, first, BGA package substrate image and point cloud are acquired, and it is preprocessed using threshold segmentation, rotation correction method;Adversarial network is constructed to generate small defect image;Welding point position is determined using Hough circle conversion, and small defect image is used to generate defect BGA package substrate image at suitable position;Point cloud generation network is constructed to generate corresponding defect BGA package substrate point cloud.The application realizes the generation of BGA package substrate multimodal defect sample by the method that depth learning is combined with traditional image processing technology, and the category and position of generated defect can be controlled, the generated defect has diversity and authenticity, solves the problem that BGA package substrate multimodal defect sample of multiple defect morphology and position cannot be generated in prior art under the condition of few samples, and the generated defect sample enriches defect data set, solves the problem of insufficient training data of defect detection.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of defect sample generation, in particular to a BGA packaging substrate multi-modal defect sample generation method and system. BACKGROUND

[0002] Defects are inevitable in industrial production process, and product appearance defect detection is an important quality detection link in manufacturing industry. With the development of artificial intelligence, defect detection methods based on deep learning are widely used in industrial quality detection. However, deep learning defect detection methods need a large number of defect samples as training data, and in actual production process, most of the defect samples of products are difficult to collect, such as BGA packaging substrate multi-modal defect samples, which are few in number, unbalanced in defect category, troublesome to collect and label, and require a large collection cost. Since BGA packaging substrates are widely used in mobile phone CPUs and other chips, the precision of BGA packaging substrates is high, and small defects may affect the performance of the chip, therefore, designing a BGA packaging substrate multi-modal defect sample generation method is of great significance for defect detection and quality control of BGA packaging substrates and related products.

[0003] The commonly used defect generation methods in the prior art include traditional image processing methods and defect generation methods based on GAN. The traditional image processing methods, such as picture cropping, translation, flipping, filtering, noise adding, contrast adjustment and the like, cannot generate new and diverse defects; the method based on GAN is limited by few samples, and it is difficult to generate a whole BGA packaging substrate with specific category defects and specific position defects, and the above two methods cannot generate multi-modal defect samples, and cannot complete the task of generating BGA packaging substrate multi-modal defect samples.

[0004] The present application aims to overcome the shortcomings and deficiencies of the prior art, and provides a BGA packaging substrate multi-modal defect sample generation method and system, which aims to solve the technical problem that the prior art cannot generate BGA packaging substrate multi-modal defect samples, resulting in lack of sufficient data in defect detection.

[0005] The present application is realized by the following technical scheme:

[0006] A BGA packaging substrate multi-modal defect sample generation method, comprising the following steps:

[0007] Step S1, BGA packaging substrate multi-modal sample preprocessing based on threshold segmentation and rotation transformation matrix correction: obtaining the photographed BGA packaging substrate image and point cloud, performing threshold segmentation and rotation correction processing on the same, and obtaining a standardized BGA packaging substrate image and point cloud;

[0008] Step S2, small defect image generation based on instance normalization generative adversarial network: crop various defects in the BGA package substrate image, build and train an instance normalization generative adversarial network suitable for small defect image generation, input noise to the generative adversarial network to generate small defect images with rich morphologies;

[0009] Step S3, defect BGA package substrate image generation based on solder ball position and defect feature: perform Hough circle transformation on the entire BGA package substrate image to find all solder balls, and sort the solder balls, replace the small defect images generated in step S2 to the appropriate position according to their features, and obtain the defect BGA package substrate image;

[0010] Step S4, corresponding defect BGA package substrate point cloud generation based on skip connection point cloud generation network: build and train a skip connection point cloud generation network suitable for defect BGA package substrate point cloud generation, input the defect BGA package substrate image generated in step S3 into the point cloud generation network to generate the corresponding defect BGA package substrate point cloud.

[0011] Preferably, in step S1, the specific process of BGA package substrate multi-modal sample preprocessing based on threshold segmentation and rotation transformation matrix correction is as follows:

[0012] Step S11, using a fixed threshold method, the BGA package substrate gray scale image is binarized, the contour of the binarized image is extracted, and the BGA package substrate contour is obtained;

[0013] Step S12, according to the obtained contour, an outer convex rotating rectangle is fitted, a perspective transformation matrix is generated by the outer convex rotating rectangle, and then perspective transformation is performed to rotate and correct the BGA package substrate.

[0014] Preferably, in step S2, the specific process of small defect image generation based on instance normalization generative adversarial network is as follows:

[0015] Step S21, crop various defects in the BGA package substrate image as training data;

[0016] Step S22, build an instance normalization generative adversarial network with a generator and a discriminator;

[0017] Step S23, input noise to the generator to generate small defect images, then input the generated defects and real defects to the discriminator, calculate the discriminator loss, use the optimizer to optimize the discriminator loss, so that the discriminator can better identify real defects and generated defects, at the same time, calculate the generator loss, use the optimizer to optimize the generator loss, so that the generator generates more realistic defects to deceive the discriminator, until the generation effect approaches the real situation or the training iteration number reaches the maximum iteration number, stop iteration, that is, the trained network model can be obtained;

[0018] Step S24, randomly generate noise, input the trained generative adversarial network, and generate a small defect image;

[0019] Further, in step S22, the generator has four transpose convolution blocks and a transpose convolution layer, and the transpose convolution block includes a transpose convolution layer, a batch normalization layer and a ReLU activation function arranged from front to back.

[0020] The discriminator has four convolution blocks and a convolution layer, the first convolution block includes a convolution layer, a LeakyReLU activation function arranged from front to back, and the last three convolution blocks include a convolution layer, an instance normalization layer and a LeakyReLU activation function arranged from front to back.

[0021] Preferably, in step S3, the specific process of generating a defect BGA packaging substrate image based on the solder ball position and defect features is as follows:

[0022] Step S31, perform Hough circle transformation on the entire BGA packaging substrate to obtain the center coordinates of all solder balls;

[0023] Step S32, sort the center coordinates of all solder balls according to the distance from the image center, thereby determining the serial number of each solder ball and its position;

[0024] Step S33, replace the generated small defect image to the normal solder ball position or its surrounding position according to the category and features.

[0025] Preferably, in step S4, the specific process of generating a corresponding defect BGA packaging substrate point cloud based on the skip connection point cloud generation network is as follows:

[0026] Step S41, build a skip connection point cloud generation network with an encoder module and a decoder module;

[0027] Step S42, input the original defect BGA packaging substrate image into the built point cloud generation network to generate a corresponding point cloud, make L1 loss between the generated point cloud and the label, use an optimizer to optimize, update the point cloud generation network parameters, and until the training iteration number reaches the maximum iteration number, a trained network model can be obtained;

[0028] Step S43, input the defect BGA packaging substrate image generated in step S3 into the trained point cloud generation network to generate a corresponding defect BGA packaging substrate point cloud, and a pair of defect BGA packaging substrate image and point cloud can be obtained.

[0029] Further, in step S41, the point cloud generation network is composed of an encoder module and a decoder module connected through a bottleneck block;

[0030] The encoder module comprises 7 down-sampling blocks, the first down-sampling block comprises a convolution layer and a LeakyReLu activation function, and the middle six down-sampling blocks comprise, from front to back, a convolution layer, a batch normalization layer and a LeakyReLU activation function;

[0031] The decoder module comprises 7 up-sampling blocks, the first six up-sampling blocks comprise, from front to back, a transpose convolution layer, a batch normalization layer and a LeakyReLU activation function, and the seventh up-sampling block comprises a transpose convolution layer and a Tanh activation function;

[0032] The bottleneck block comprises a convolution layer and a LeakyReLu activation function;

[0033] The point cloud generation network adopts a skip connection between the down-sampling blocks and the up-sampling blocks, adds the feature map of the down-sampling block to the feature map of the corresponding up-sampling block as the input of the next up-sampling block.

[0034] A BGA packaging substrate multi-modal defect sample generation system, comprising:

[0035] An acquisition module is configured to acquire a BGA packaging substrate image and a point cloud;

[0036] A preprocessing module is configured to segment and correct the BGA packaging substrate image and the point cloud;

[0037] A small defect image generation module is configured to train an instance normalized generative adversarial network and generate a small defect image;

[0038] A defect image generation module is configured to generate a whole defect BGA packaging substrate image;

[0039] A defect point cloud generation module is configured to train a skip connection point cloud generation network and generate a corresponding whole defect BGA packaging substrate point cloud.

[0040] Compared with the prior art, the present application has the following advantages and effects:

[0041] The BGA packaging substrate multi-modal defect sample generation method of the application first adopts the method of fixed threshold, rotating rectangular fitting and generating perspective transformation matrix to segment and rotate the original BGA packaging substrate, so as to obtain a standardized BGA packaging substrate image and point cloud; then generates a small defect image by using an instance normalization generative adversarial network, the instance normalization performs a normalization operation on a single channel of a single picture, and the generated effect is good and less dependent on the number of samples; then the Hough circle transformation is performed on the whole BGA packaging substrate to find all the solder balls and sort them, and the generated small defect image is replaced to the appropriate position to obtain a defect BGA packaging substrate image; finally, a jump connection point cloud generation network is used to generate the defect BGA packaging substrate point cloud corresponding to the defect BGA packaging substrate image, and the jump connection effectively saves the detail features of the image.

[0042] The application realizes the generation of BGA packaging substrate multi-modal defect samples by combining deep learning with traditional image processing technology, realizes the corresponding generation of BGA packaging substrate defect images and defect point clouds, and can control the type and position of the generated defects, the generated defects have diversity and authenticity, solves the problem that the prior art cannot generate BGA packaging substrate multi-modal defect samples and it is difficult to generate a whole BGA packaging substrate defect image with multiple defect shapes and positions under the condition of few samples, the generated defect samples enrich the defect data set, reduce the defect sample collection and labeling cost, solve the problem of insufficient defect detection training data, and have good practicability. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 For the application Figure 1 is a method flowchart of the application.

[0044] Figure 2 is the segmentation and correction of the BGA packaging substrate image in the embodiment of the application; wherein Figure 2 (a) is the original image of the BGA packaging substrate, Figure 2 (b) is the binary BGA packaging substrate image, Figure 2 (c) is the segmented and corrected BGA packaging substrate image.

[0045] Figure 3 is the adversarial neural network structure diagram in the embodiment of the application.

[0046] Figure 4 is the small defect image generation diagram of the BGA packaging substrate in the embodiment of the application; wherein Figure 4 (a) is a small tin defect generation diagram, Figure 4 (b) is a solder ball missing defect generation diagram, Figure 4 (c) is a dirt defect generation diagram, Figure 4 (d) is an abnormal solder ball defect generation diagram.

[0047] Figure 5 This is a diagram showing the arrangement of solder balls on a BGA package substrate in an embodiment of the present invention.

[0048] Figure 6 In an embodiment of the present invention, different defect maps are generated at the same position on the BGA package substrate.

[0049] Figure 7 The defect maps are generated at different positions on the BGA package substrate in an embodiment of the present invention.

[0050] Figure 8 Schematic diagram of the point cloud generation network structure in an embodiment of the present invention.

[0051] Figure 9 This is a point cloud generation diagram of a defective BGA package substrate in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0053] Example 1:

[0054] This embodiment discloses a method for generating multi-modal defect samples of a BGA package substrate, which can be executed on a smart device such as a computer. Figure 1 The method comprises the following steps:

[0055] Step S1, BGA package substrate multimodal sample preprocessing based on threshold segmentation and rotation transformation matrix correction: obtain the captured BGA package substrate grayscale image H and point cloud image P, the BGA package substrate grayscale image H is as follows Figure 2 As shown in (a), segmentation and rotation correction preprocessing are performed to obtain the processed BGA package substrate grayscale image H' and point cloud image P'. The BGA package substrate grayscale image H' is as follows Figure 2 As shown in (c), the defects are marked and the small defect image X is cropped as follows:

[0056] Step S11: Use a fixed threshold method to perform binarization processing on the captured BGA package substrate grayscale image H, such as Figure 2 As shown in (b), the outline of the binary image is extracted to obtain the outline of the BGA package substrate;

[0057] Step S12: fitting a convex rotated rectangle according to the obtained outline, generating a perspective transformation matrix through the convex rotated rectangle, and then performing perspective transformation on the convex rotated rectangle to rotate and correct the BGA package substrate, thereby obtaining a cropped and corrected BGA package substrate grayscale image H' and point cloud image P';

[0058] Step S2, small defect image generation based on instance normalization generation adversarial network: training the instance normalization generation adversarial network built with the real small defect image X as a label, inputting noise to the generation adversarial network to generate a small defect image, and the specific process is as follows:

[0059] Step S21, using labelme to mark defects on the BGA package substrate gray image H', and cutting out a small defect image X.

[0060] Step S22, building a generation adversarial network with a generator and a discriminator, and the structure diagram of the generation adversarial network is as shown in Figure 3 The generator has four transposed convolution blocks and one transposed convolution layer, and the transposed convolution block includes a transposed convolution layer, a batch normalization layer and a ReLU activation function arranged from front to back; the discriminator has four convolution blocks and one convolution layer, and the first convolution block includes a convolution layer, a LeakyReLU activation function arranged from front to back, and the last three convolution blocks include a convolution layer, an instance normalization layer and a LeakyReLU activation function arranged from front to back.

[0061] Step S23, inputting noise into the generator to generate a small defect sample, inputting the generated defect G and the real defect X into the discriminator, calculating the discriminator loss, using the optimizer to optimize the discriminator loss, so that the discriminator can better identify the real defect and the generated defect, at the same time, calculating the generator loss, using the optimizer to optimize the generator loss, so that the generator generates more realistic defects to deceive the discriminator, until the training iteration reaches the maximum iteration, that is, the trained network model is obtained;

[0062] Step S24, randomly generating noise and inputting it into the trained generation adversarial network to generate a small defect sample G, and the generation effect is as shown in Figure 4 Figure 4 (a) is a small tin defect generation diagram, Figure 4 (b) is a solder ball missing defect generation diagram, Figure 4 (c) is a dirt defect generation diagram, Figure 4 (d) is an abnormal solder ball defect generation diagram.

[0063] In the embodiment, the above generator loss can be as follows:

[0064]

[0065] Wherein, The generator loss is represented; z represents the input noise; G(·) represents the generator calculation; D(·) represents the discriminator calculation. The function expectation value calculation is represented.

[0066] The above discriminator loss can be as follows:​

[0067]

[0068] wherein, represents the discriminator loss; x represents the real image; λ is the weight of the gradient penalty; represents the gradient; is the interpolation between the real sample and the generated sample; ‖·‖2 represents the L2 norm calculation.

[0069] Step S3, defect BGA packaging substrate image generation based on solder ball position and defect feature: perform Hough circle transformation on the whole BGA packaging substrate to find all the solder balls and sort them, replace the generated small defect image to the appropriate position to obtain the defect BGA packaging substrate image, as follows:

[0070] Step S31, perform Hough circle transformation on the whole BGA packaging substrate to obtain the center coordinates of all the solder balls;

[0071] Step S32, sort the center coordinates of all the solder balls according to the distance from the image center, thereby determining the serial number of each solder ball and its position, as shown in Figure 5 .

[0072] Step S33, replace the generated small defect image to the normal solder ball position or the surrounding position according to the category to generate a new defect BGA packaging substrate image H”.

[0073] In this embodiment, the little tin defect, the solder ball missing defect and the solder ball abnormal defect can be replaced to the normal solder ball position, as shown in Figure 6 , the first image on the left is without defect, the second image adds the little tin defect at the third solder ball position, the third image adds the solder ball missing defect, and the fourth image adds the solder ball abnormal defect.

[0074] In this embodiment, the same defect can also be replaced to different positions and different defects can be combined, as shown in Figure 7 , the first image on the left is without defect, the second image adds the little tin defect at the third solder ball position, the third image adds the defect at the fourth solder ball position, and the fourth image combines the little tin defect and the dirty defect.

[0075] Step S4, corresponding defect BGA packaging substrate point cloud generation based on the skip connection point cloud generation network: use the defect BGA packaging substrate image H’ and the point cloud image P’ to train the built point cloud generation network, input the new defect BGA packaging substrate image H” generated in step S3 into the trained point cloud generation network to generate the corresponding new defect BGA packaging substrate point cloud P”. The specific process is as follows:

[0076] Step S41, a point cloud generation network connected by a bottleneck block through an encoder module and a decoder module is built. The encoder module contains 7 down-sampling blocks, the first down-sampling block includes a convolution layer and a LeakyReLu activation function, and the middle six down-sampling blocks include convolution layers, batch normalization layers and LeakyReLU activation functions arranged from front to back;

[0077] The decoder module contains 7 up-sampling blocks, the first six up-sampling blocks include transpose convolution layers, batch normalization layers and LeakyReLU activation functions arranged from front to back, and the seventh up-sampling block includes a transpose convolution layer and a Tanh activation function;

[0078] The bottleneck block includes a convolution layer and a LeakyReLu activation function;

[0079] The point cloud generation network adopts a skip connection between the down-sampling blocks and the up-sampling blocks, adds the feature map of the down-sampling block to the feature map of the corresponding up-sampling block as the input of the next up-sampling block. The point cloud generation network structure diagram is as shown in Figure 8 ;

[0080] Step S42, input the defect BGA packaging substrate image H' into the built point cloud generation network to generate a corresponding point cloud, make L1 loss between the generated point cloud and the label, use an optimizer to optimize the discriminator loss, update the point cloud generation network parameters until the training iteration number reaches the maximum iteration number, and then a trained network model is obtained;

[0081] Step S43, input the new defect BGA packaging substrate image H'' generated in step S3 into the trained point cloud generation network to generate a corresponding new defect BGA packaging substrate point cloud P'', and then a pair of generated defect BGA packaging substrate image and point cloud are obtained. As shown in Figure 9 , the first image represents no defect, the second image adds a little tin defect to the fourth solder ball, and the third image adds a missing defect to the third position. It can be seen that the generated point cloud realizes the multi-modal expansion of the defect sample and restores the point cloud morphology of the defect.

[0082] Those skilled in the art can understand that all or part of the steps in the method of the embodiment can be completed by programs instructing related hardware, and the corresponding programs can be stored in a computer readable storage medium.

[0083] Embodiment 2

[0084] The embodiment discloses a BGA packaging substrate multi-modal defect sample generation system, which comprises:

[0085] An acquisition module is configured to acquire a BGA packaging substrate image and a point cloud;

[0086] A preprocessing module is configured to segment and correct the BGA package substrate image and the point cloud;

[0087] A small defect image generation module is configured to train an instance normalized generative adversarial network and generate a small defect image;

[0088] A defect image generation module is configured to generate a whole defect BGA package substrate image;

[0089] A defect point cloud generation module is configured to train a skip connection point cloud generation network and generate a corresponding whole defect BGA package substrate point cloud.

[0090] The specific implementation of each module in the embodiment can be referred to the embodiment 1, and will not be repeated here. It should be noted that the system provided in the embodiment is only exemplified by the division of the above functional modules. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure is divided into different functional modules to complete all or part of the functions described above.

[0091] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application are equivalent replacement methods and are included in the protection scope of the present application.

Claims

1. A method for generating multimodal defect samples of a BGA package substrate, characterized in that: The steps include: Step S1, BGA package substrate multimodal sample preprocessing based on threshold segmentation and rotation transformation matrix correction: obtaining the captured BGA package substrate image and point cloud, performing threshold segmentation and rotation correction processing on them, and obtaining a standardized BGA package substrate image and point cloud; Step S2, generating small defect images based on an instance-normalized generative adversarial network: cropping small defect images, building and training an instance-normalized generative adversarial network suitable for small defect image generation, and inputting noise into the generative adversarial network to generate small defect images with rich morphology, including a generated image of a missing solder ball defect; Step S3, generating a defective BGA package substrate image based on solder ball positions and defect characteristics: performing a Hough circle transform on the entire BGA package substrate image to find all solder balls, sorting the solder balls, and replacing the small defect image generated in step S2 to the appropriate position based on its characteristics to obtain a defective BGA package substrate image; Step S4, generating a point cloud of the corresponding defective BGA package substrate based on a skip connection point cloud generation network: building and training a skip connection point cloud generation network suitable for generating a point cloud of the defective BGA package substrate, and inputting the defective BGA package substrate image generated in step S3 into the point cloud generation network to generate a corresponding point cloud of the defective BGA package substrate; In step S3, the specific process of generating the defective BGA package substrate image based on the solder ball position and defect characteristics is as follows: Step S31, performing Hough circle transform on the entire BGA package substrate to obtain the center coordinates of all solder balls; Step S32, sorting the center coordinates of all solder balls according to their distance from the center of the image, thereby determining the sequence number of each solder ball and its position; In step S33 , the generated small defect image is replaced to the normal solder ball position or its surrounding position according to the category and features.

2. The method for generating multi-modal defect samples of a BGA package substrate according to claim 1, characterized in that: In step S1, the specific process of BGA package substrate multimodal sample preprocessing based on threshold segmentation and rotation transformation matrix correction is as follows: Step S11, using a fixed threshold method to perform binarization processing on the captured BGA package substrate image, extracting the outline of the binarized image, and obtaining the BGA package substrate outline; Step S12: fitting a convex rotated rectangle according to the obtained outline, generating a perspective transformation matrix through the convex rotated rectangle, and then performing perspective transformation based on the convex rotated rectangle to rotationally correct the BGA package substrate.

3. The method for generating multi-modal defect samples of a BGA package substrate according to claim 1, wherein: In step S2, the specific process of generating small defect images based on the instance-normalized generative adversarial network is as follows: Step S21, cropping various defects in the BGA package substrate image as training data; Step S22, building an instance normalized generative adversarial network with a generator and a discriminator; Step S23: Input noise into the generator to generate small defect images, then input the generated defects and real defects into the discriminator, calculate the discriminator loss, and use the optimizer to optimize the discriminator loss so that the discriminator can better distinguish real defects from generated defects. At the same time, calculate the generator loss and use the optimizer to optimize the generator loss so that the generator generates more realistic defects to deceive the discriminator. Until the generation effect is close to the real situation or the number of training iterations reaches the maximum number of iterations, stop the iteration and obtain the trained network model. In step S24, noise is randomly generated and input into the trained generative adversarial network to generate a small defect image.

4. The method for generating multi-modal defect samples of a BGA package substrate according to claim 3, wherein: In step S22, the generator has four transposed convolution blocks and one transposed convolution layer, and the transposed convolution block includes a transposed convolution layer, an instance normalization layer, and a ReLU activation function arranged in sequence from front to back; The discriminator has four convolution blocks and one convolution layer. The first convolution block includes a convolution layer and a LeakyReLU activation function arranged in sequence from front to back, and the last three convolution blocks include a convolution layer, an instance normalization layer, and a LeakyReLU activation function arranged in sequence from front to back.

5. The method for generating multi-modal defect samples of a BGA package substrate according to claim 1, wherein: In step S4, the specific process of generating the corresponding defective BGA package substrate point cloud based on the jump connection point cloud generation network is as follows: Step S41: Building a skip-connected point cloud generation network with an encoder module and a decoder module. The encoder module includes 7 downsampling blocks, and the decoder module includes 7 upsampling blocks. The point cloud generation network uses skip connections between the downsampling blocks and the upsampling blocks. Step S42: Input the original defective BGA package substrate image into the constructed point cloud generation network to generate the corresponding point cloud. The generated point cloud is compared with the original point cloud using an L1 loss. The optimizer is used to optimize and update the point cloud generation network parameters until the number of training iterations reaches the maximum number of iterations. The trained network model is then obtained. In step S43, the defective BGA package substrate image generated in step S3 is input into the trained point cloud generation network to generate a corresponding defective BGA package substrate point cloud, thereby obtaining a pair of defective BGA package substrate images and point clouds.

6. A BGA package substrate multimodal defect sample generation system used in the method according to any one of claims 1 to 5, characterized in that include: Acquisition module, used to acquire BGA package substrate images and point clouds; Pre-processing module for segmenting and rectifying BGA package substrate images and point clouds; Generate small defect images module, used to train instance-normalized generative adversarial networks and generate small defect images; Generate defect image module, used to generate the entire defective BGA package substrate image; The defect point cloud generation module is used to train the skip connection point cloud generation network and generate the corresponding entire defective BGA package substrate point cloud.

Citation Information

Patent Citations

  • Defect image sample generation method and system based on generative adversarial network, medium and equipment

    CN114820469A

  • Defect sample generation system and method based on generative adversarial network and storage medium

    CN115661134A