An electronic component defect recognition method and device based on a convolutional neural network

Abstract

The method is a kind of electronic component defect identification method and system based on convolutional neural network, comprising: collecting multi-modal fusion images of multiple electronic components including defects as a raw image set;Determine the defect type and defect position of each electronic component in the raw image set and label, divide the labeled data into a training set and a test set;Use the training set to train a three-modal fusion convolutional neural network model, and use the test set to test the trained model, when the index meets the preset requirement, obtain the trained model;The multi-modal fusion image of the electronic component to be identified is input into the trained three-modal fusion convolutional neural network model, and the defect type and defect position of the electronic component to be identified are obtained.The present application can identify a variety of internal and surface defects that may occur in electronic components.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, and in particular to an electronic component defect recognition method and device based on a convolutional neural network. BACKGROUND

[0002] Integrated circuit electronic components are increasingly applied in industrial production, and the quality of electronic components directly affects the quality of products. Identifying defects in integrated circuit electronic components is a key link in the development of chip-related industries, so quality detection of electronic components is particularly important. In the existing production process, manual experience comparison or machine vision detection is mainly relied on. For electronic components of different sizes and shapes, flexibility is poor and recognition accuracy is poor. SUMMARY

[0003] To solve the above problems, the present application provides an electronic component defect recognition method and device based on a convolutional neural network, which introduces multi-modal integrated circuit electronic component high-resolution image data fusion of optical, ultrasonic and infrared, making the data details input into the network more rich, and effectively improving the defect recognition accuracy.

[0004] In one aspect, an electronic component defect recognition method based on a convolutional neural network comprises:

[0005] S101, collecting multi-modal fusion images of a plurality of electronic components including defects as a set of original images;

[0006] S102, determining the defect type and defect position of each electronic component in the set of original images according to the electronic component defect definition standard and labeling, dividing the labeled data into a training set and a test set;

[0007] S103, training a three-modal fusion convolutional neural network model using the training set, and testing the trained three-modal fusion convolutional neural network model using the test set, and obtaining a trained three-modal fusion convolutional neural network model when the index meets the preset requirements;

[0008] S104, inputting the multi-modal fusion image of the electronic component to be identified into the trained three-modal fusion convolutional neural network model to obtain the defect type and defect position of the electronic component to be identified.

[0009] Preferably, the method for obtaining the multi-modal fusion image comprises:

[0010] For each single electronic component, a set of optical, ultrasonic and infrared images are taken at the same position, and the three images are fused into one multi-modal high-resolution image data without any changes to the original images, only associated fusion, that is, the three images taken at the same position are recorded as a set of data.

[0011] Preferably, the defect types of the electronic component include cracks, delamination, scratches, missing corners, oil stains, scratches, breakage, spots, holes and particle contamination.

[0012] The cracks, scratches and scratches are first determined by the length ratio method. When the following conditions are met, it is determined that the electronic component has a crack, scratch or scratch defect. Then the internal crack of the component is determined by ultrasonic image, and the scratch or scratch is determined by actual function test:

[0013]

[0014] Wherein, L ′ represents the length of the defect, L represents the length of the diagonal of the plane; the spot or oil stain is first determined by the area ratio method. When the following conditions are met, it is determined that the electronic component has a spot or oil stain defect. Then, under dark environment, the oil stain presents a specific gorgeous luster by using white light:

[0015]

[0016] Wherein, S ′ represents the area of the defect, S represents the area of the plane;

[0017] The missing corner or breakage is determined by the volume ratio method. When the following conditions are met, it is determined that the electronic component has a corresponding defect. When the defect ratio is greater than 0.1, the defect condition is directly identified as breakage:

[0018]

[0019] Wherein, V ′ represents the volume of the defect, V represents the volume of the electronic component; when is greater than 0.1, the defect type is determined as breakage;

[0020] For delamination, hole or particle contamination, the number of defects is determined by counting method, and the number of defects is n; when n≥1, the defect type is determined as delamination or hole, and the delamination and hole are distinguished by infrared image / ultrasonic image; when n≥2, the defect type is determined as particle contamination.

[0021] Preferably, when the defect type is determined, the smallest rectangular frame that can wrap the defect position in the multi-modal high-resolution image data is recorded as the defect position.

[0022] Preferably, the three-modal fusion convolutional neural network model comprises a feature extraction module, a feature fusion module and a prediction module; the feature extraction module comprises three independent channels, each channel having 10 convolutional layers, and is used for extracting optical, ultrasonic and infrared data respectively; the feature fusion module comprises a feature splicing operation and 8 convolutional layers, takes the outputs of the three independent channels as inputs, performs a Concat operation and then a convolution operation to perform feature fusion; the prediction module comprises a defect category prediction module and a defect position prediction module, and takes the output of the feature fusion module as input and transmits it to two fully connected layers.

[0023] Preferably, the feature extraction module performs feature extraction based on the following formula:

[0024]

[0025] wherein, represents optical data output by the lth convolutional layer of the first channel; represents infrared data output by the lth convolutional layer of the second channel; represents ultrasonic data output by the lth convolutional layer of the third channel; represents optical data output by the (l-1)th convolutional layer of the first channel; represents infrared data output by the (l-1)th convolutional layer of the second channel; represents ultrasonic data output by the (l-1)th convolutional layer of the third channel; represents a filter kernel function of the lth convolutional layer; represents a bias coefficient of the lth convolutional layer; x∈[1, Num], Num represents the maximum convolution kernel size; * represents convolution operation;

[0026] The feature fusion module performs feature fusion based on the following formula:

[0027]

[0028] wherein, represents a feature map output by the 0th convolutional layer after fusion; represents a feature map output by the lth convolutional layer after fusion; represents a feature map output by the (l-1)th convolutional layer after fusion; Concat represents a feature fusion splicing operation of three kinds of images; YLU represents an exponential linear unit activation function, which is used for nonlinear conversion operation between layers;

[0029] In the prediction module, the two fully connected layers are processed as follows:

[0030]

[0031] wherein, represents the lth layer of the class prediction task, i.e., the class prediction task mapping layer, outputting a predicted class; represents the lth layer of the defect location prediction task, i.e., the defect location prediction task mapping layer, outputting a predicted defect location; represents the (l-1)th layer feature map output by the class prediction task; represents the (l-1)th layer feature map output by the defect location prediction task; represents the (l-1)th layer filter kernel function in the class prediction task; represents the (l-1)th layer filter kernel function in the defect location prediction task; represents the (l-1)th layer bias coefficient in the class prediction task; represents the (l-1)th layer bias coefficient in the defect location prediction task.

[0032] Preferably, the classification regression loss Loss of the defect class prediction task of the electronic component DCP The calculation is as follows:

[0033]

[0034] The regression loss Loss of the defect location prediction of the electronic component DLp The calculation is as follows:

[0035]

[0036] Wherein, N represents the number of samples taken from the training set and used to train the prediction model; represents the predicted value of the defect type of the jth classifier of the ith sample based on the prediction model; represents the true value of the defect type of the jth classifier of the ith sample; represents the predicted value of the defect location of the ith sample; L i represents the true value of the defect location of the ith sample.

[0037] Preferably, based on the labeled original image set, 40% of the sample data corresponding to each type of defect image data is randomly taken as training data and placed in the normal training set, 40% of the sample data is randomly taken for incremental defect class labeling, and placed in the incremental training set, and 20% of the sample data is randomly taken as test data and placed in the test set.

[0038] Preferably, when the index meets the preset requirement, a trained three-modal fusion convolutional neural network model is obtained, comprising:

[0039] The test effect of the three-modal fusion convolutional neural network model is evaluated by four indexes. When the test effect of the prediction model meets the preset requirements of the four indexes, the three-modal fusion convolutional neural network model is determined as a trained three-modal fusion convolutional neural network model, and the specific process is as follows:

[0040] The values of the defect recall rate index Recall, the precision rate index Precision, the model processing speed index FPS, and the incremental performance index IOP are calculated according to the following calculation formulas respectively:

[0041]

[0042] Wherein, N TP represents the number of correct target judgments, the marked frame is the position frame of the defect in the test set marked, and the predicted frame is the position frame of the defect in the test set predicted by the model. When the predicted frame and the marked frame have an intersection-over-union greater than or equal to 0.3, and the category judgment is correct, it is determined that the target judgment is correct; N gt represents the number of marked frames manually labeled; N Predict-P represents the number of all predicted frames predicted as defects; t begin represents the time point at which the picture defect prediction starts; t end represents the time point at which the picture defect prediction ends; n incre-category represents the number of incremental defect categories when the model is trained using the ordinary training set and then trained using the incremental training set; n origin-category represents the number of defect categories of the original trained model before incremental training; n incre-images represents the number of incremental training pictures when the model is trained and then trained using the incremental training set; n origin-images represents the number of training pictures used by the original trained model before incremental training; Recall end represents the recall rate after incremental learning training is completed; Recall origin represents the recall rate after training before incremental learning; Precision end represents the precision rate after incremental learning training is completed; Precision origin represents the precision rate after training before incremental learning;

[0043] If the obtained defect recall rate index Recall is greater than or equal to 0.95, the obtained precision rate evaluation index Precision is greater than or equal to 0.95, the obtained model processing speed index FPS is greater than or equal to 30, and the obtained incremental performance index IOP is greater than or equal to 0.9, the model is determined as a trained three-modal fusion convolutional neural network model.

[0044] In another aspect, an electronic component defect recognition device based on a convolutional neural network comprises:

[0045] An original image set acquisition module is configured to acquire multi-modal fusion images of a plurality of electronic components including defects as an original image set;

[0046] A data labeling module is configured to determine and label defect types and defect positions of each electronic component in the original image set according to electronic component defect definition standards, and divide the labeled data into a training set and a test set;

[0047] A model training module is configured to train a three-modal fusion convolutional neural network model using the training set, test the trained three-modal fusion convolutional neural network model using the test set, and obtain a trained three-modal fusion convolutional neural network model when an index meets a preset requirement;

[0048] A defect recognition module is configured to input multi-modal fusion images of an electronic component to be recognized into the trained three-modal fusion convolutional neural network model to obtain defect types and defect positions of the electronic component to be recognized.

[0049] The present application has the following innovations and advantages over the prior art:

[0050] The present application uses multi-modal integrated circuit electronic component high-resolution image data fused by optical, ultrasonic and infrared to serve as training data based on a three-modal fusion convolutional neural network model, and through a feature extraction module, a feature fusion module and a prediction module, performs electronic component defect recognition and positioning, and at the same time, designs an incremental performance index IOP to evaluate the incremental prediction performance of the model. Through actual experimental tests, it is proved that the present application can accurately position and recognize electronic component defects, and is helpful for related technology breakthroughs and industrial development. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A flowchart of the electronic component defect recognition method based on a convolutional neural network of the present application;

[0052] Figure 2 A framework diagram of the three-modal fusion convolutional neural network model of the present application;

[0053] Figure 3 An ablation experiment performance graph of the present application;

[0054] Figure 4 A structure block diagram of the electronic component defect recognition system based on a convolutional neural network of the present application. DETAILED DESCRIPTION

[0055] The present application will be further described below in combination with the drawings and examples.

[0056] Reference is made toFigure 1 The application discloses a method for identifying defects of electronic components based on a convolutional neural network, which comprises the following steps:

[0057] S101, a multi-modal fusion image of a plurality of electronic components including defects is collected as an original image set;

[0058] S102, according to the defect definition standard of the electronic component, the defect type and defect position of each electronic component in the original image set are determined and labeled, and the labeled data is divided into a training set and a test set;

[0059] S103, the training set is used to train a three-modal fusion convolutional neural network model, and the test set is used to test the trained three-modal fusion convolutional neural network model, and when the index meets the preset requirement, a trained three-modal fusion convolutional neural network model is obtained;

[0060] S104, a multi-modal fusion image of an electronic component to be identified is input into the trained three-modal fusion convolutional neural network model, and the defect type and defect position of the electronic component to be identified are obtained.

[0061] Specifically, S101 is a data preparation stage, and the multi-modal fusion image is a multi-modal integrated circuit electronic component high-resolution image data fused by optical, ultrasonic and infrared images.

[0062] For each single electronic component, a group of optical, ultrasonic and infrared images are taken at the same position, and the three images are fused into one multi-modal high-resolution image data.

[0063] According to the technical data of the electronic component and the original image set data, the defect type, defect definition standard and defect position of the electronic component are determined.

[0064] Specifically, for cracks, scratches and scratches, first, the length ratio method is used to determine the defects, if the defect length is L ′ , the diagonal line length of the surface is L, when the following conditions are met, it is determined that the electronic component has crack, scratch or scratch defect, and then the ultrasonic image is used to determine the crack inside the component, and the actual function test is used to distinguish the scratch and the scratch.

[0065]

[0066] For spots, oil stains, first use the area ratio method to determine defects, if the defect area is S ′ , the area of the plane is S, when the following conditions are met, it is determined that the electronic component has spot or oil stain defects, and then the white light is used in the dark environment, and the oil stain will present a specific brilliant luster:

[0067]

[0068] For corner defects, damage, use the volume ratio method to determine defects, the defect volume is V ′ , the volume of the electronic component is V, when the following conditions are met, it is determined that the electronic component has the corresponding defect, wherein when the defect ratio is greater than 0.1, the defect condition is directly identified as damage:

[0069]

[0070] For delamination, hole or particle contamination, use the counting method to determine defects, the number of defects is n, for delamination and hole (delamination is distinguished by infrared image and ultrasonic image, and hole refers to the penetrating defect visible to the human eye), when n≥1, it is identified as the corresponding defect; for particle contamination, when n≥2, it is identified as the corresponding defect.

[0071] When the defect type is determined, the smallest rectangular box that can wrap the defect position in the multi-modal high-resolution image data is taken as the defect position.

[0072] Referring to Figure 2 , the three-modal fusion convolutional neural network model of the application has three independent channel feature extraction modules in the first stage, each channel includes 10 convolutional layers to extract features from optical images, ultrasonic images and infrared images respectively, as follows:

[0073]

[0074] Wherein, represents the optical data output by the lth convolutional layer of the first channel; represents the infrared data output by the lth convolutional layer of the second channel; represents the ultrasonic data output by the lth convolutional layer of the third channel; represents the optical data output by the l-1th convolutional layer of the first channel; represents the infrared data output by the l-1th convolutional layer of the second channel; represents the ultrasonic data output by the l-1th convolutional layer of the third channel; represents the filter kernel function of the lth convolutional layer; represents the bias coefficient of the lth convolutional layer; x∈[1, Num], Num represents the maximum convolution kernel size; * represents convolution operation;

[0075] The feature fusion module adopts the following formula for feature fusion:

[0076]

[0077] wherein, represents the feature map output by the 0th convolutional layer after fusion; represents the feature map output by the lth convolutional layer after fusion; represents the feature map output by the (l-1)th convolutional layer after fusion; Concat represents the fusion splicing operation of the features of the three images; YLU represents the exponential linear unit activation function, which is used for nonlinear conversion operation between layers;

[0078] In the prediction module, the two fully connected layers are processed as follows:

[0079]

[0080] wherein, represents the lth layer of the category prediction task, i.e., the category prediction task mapping layer, which outputs the predicted category; represents the lth layer of the defect position prediction task, i.e., the defect position prediction task mapping layer, which outputs the predicted defect position; represents the (l-1)th layer feature map output by the category prediction task; represents the (l-1)th layer feature map output by the defect position prediction task; represents the (l-1)th layer filter kernel function in the category prediction task; represents the (l-1)th layer filter kernel function in the defect position prediction task; represents the bias coefficient of the (l-1)th layer in the category prediction task; represents the bias coefficient of the (l-1)th layer in the defect position prediction task.

[0081] Further, the classification regression loss Loss of the defect category prediction task of the electronic component is DCP The calculation method is as follows:

[0082]

[0083] The regression loss Loss of the defect position prediction of the electronic component is DLP The calculation method is as follows:

[0084]

[0085] wherein, N represents the number of samples taken from the training set and used for training the prediction model; a predicted value of a defect type of the i-th sample based on the prediction model; a true value of a defect type of the i-th sample based on the prediction model; a predicted value of a defect position of the i-th sample; i a true value of a defect position of the i-th sample.

[0086] In the training and verification stage: based on each type of defect image data, 40% of sample data in the labeled multi-modal electronic component high-resolution image data is randomly taken as training data and placed in a normal training set, 40% of sample data is randomly taken for incremental defect category labeling, and placed in an incremental training set, and 20% of sample data is randomly taken as test data and placed in a test set.

[0087] First, the multi-modal electronic component integrated circuit high-resolution image data in the training set is taken as an input of the three-modal fusion convolutional neural network model for training.

[0088] After training, the three-modal fusion convolutional neural network model is tested using the test set, and four indexes are used to evaluate the training effect of the three-modal fusion convolutional neural network model. When the training effect of the model meets the preset requirements of the four indexes, the model is determined as a trained three-modal fusion convolutional neural network model.

[0089] Specifically, the values of the defect recall rate index Recall, the precision rate index Precision, the model processing speed index FPS, and the incremental performance index IOP are calculated according to the following calculation formulas, respectively:

[0090]

[0091] wherein, N TP indicates the number of correct judgments, the marked box is the position box of the defect marked in the test set, and the predicted box is the position box of the defect predicted by the model. When the intersection over union of the predicted box and the marked box is greater than or equal to 0.3, and the category judgment is correct, it is determined that the target judgment is correct; N gt indicates the number of marked boxes manually labeled; N Predict-P indicates the number of all predicted boxes predicted as defects; t begin indicates a time point at which picture defect prediction starts; t end indicates a time point at which picture defect prediction ends; N incre-category indicates the number of incremental defect categories when the model is trained using the normal training set and then trained using the incremental training set; N origin-category indicates the number of defect categories of the original trained model before the incremental training; N incre-imagesN represents the number of incremental training pictures when the model is trained and incremental training is performed; N origin-images N represents the number of training pictures used by the original trained model before incremental training; Recall end N represents the recall rate after incremental learning training is completed; Recall origin N represents the recall rate after training before incremental learning; Precision end N represents the precision rate after incremental learning training is completed; Precision origin N represents the precision rate after training before incremental learning.

[0092] The fusion multi-modal image data in the test set is input into the three-modal fusion convolutional neural network model, and if the obtained defect recall rate index Recall≥0.95, the obtained precision rate evaluation index Precisios≥0.85, and the obtained model processing speed index FPS≥30 are satisfied, then the new class incremental test is performed. The specific implementation of the class incremental test is that the incremental training set with 40% sample data is put into the three-modal fusion convolutional neural network model for training. When the obtained incremental performance index IOP≥0.9, it is determined that the prediction model is a usable model.

[0093] The trained model is subjected to class incremental test, and satisfies the incremental performance index IOP≥0.9, which embodies the "incrementality", and FPS≥30, which embodies the "onlinearity".

[0094] Experimental result analysis. Referring to Figure 3 As shown in the figure, in order to verify the superiority of multi-modal data, data ablation experiments are performed on the modal data used. The specific implementation of the data ablation experiment is that the modal of the multi-modal data is gradually reduced, the model is trained, and different multi-modal data combination model performances are obtained. The model based on optical, ultrasonic and infrared fusion multi-modal integrated circuit electronic component image data achieves the best performance in Recall, Precision and IOP indexes. With the increase of the number of modal types, the FPS index slightly decreases, but the existing FPS is sufficient to meet the model feasibility requirements, which proves the superiority of the method in defect recognition and positioning.

[0095] The model trained by fusing optical and infrared image data has a 7.0% increase in recall performance, a 23.5% increase in precision performance, a 27.9% increase in IOP performance, and a 5.2% decrease in FPS performance compared with the model trained by using only optical data; the model trained by fusing optical, infrared, and ultrasonic image data has a 5.4% increase in recall performance, a 9.7% increase in precision performance, a 13.0% increase in IOP performance, and a 6.1% decrease in FPS performance compared with the model trained by using only optical and infrared data.

[0096] Referring to Figure 4 As an implementation of the method shown in the above figures, the embodiment provides an embodiment of an electronic component defect recognition device based on a convolutional neural network. Figure 1 The device embodiment corresponds to the method embodiment shown in the above figures, and the device can be applied to various electronic devices.

[0097] Specifically, an electronic component defect recognition device based on a convolutional neural network comprises:

[0098] An original image set acquisition module 401 is configured to acquire multi-modal fusion images of a plurality of electronic components including defects as an original image set.

[0099] A data labeling module 402 is configured to determine and label defect types and defect positions of each electronic component in the original image set according to electronic component defect definition standards, and divide the labeled data into a training set and a test set.

[0100] A model training module 403 is configured to train a three-modal fusion convolutional neural network model using the training set, test the trained three-modal fusion convolutional neural network model using the test set, and obtain a trained three-modal fusion convolutional neural network model when an index meets a preset requirement.

[0101] A defect recognition module 404 is configured to input multi-modal fusion images of an electronic component to be recognized into the trained three-modal fusion convolutional neural network model, and obtain defect types and defect positions of the electronic component to be recognized.

[0102] A specific implementation of an electronic component defect recognition system based on a convolutional neural network is the same as an electronic component defect recognition method based on a convolutional neural network, and the embodiment will not be repeated.

[0103] The above examples are only for illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the spirit and essence of the present application shall be covered within the protection scope of the present application.

Claims

1. A method for identifying defects in electronic components based on convolutional neural networks, characterized in that, include: S101, Collect multimodal fusion images of multiple electronic components including defects as the original image set; S102, Based on the electronic component defect definition standard, determine the defect type and defect location of each electronic component in the original image set and label them, and divide the labeled data into training set and test set; S103, the training set is used to train the three-modal fusion convolutional neural network model, and the test set is used to test the trained three-modal fusion convolutional neural network model. When the indicators meet the preset requirements, the trained three-modal fusion convolutional neural network model is obtained. S104, input the multimodal fusion image of the electronic component to be identified into the trained three-modal fusion convolutional neural network model to obtain the defect type and defect location of the electronic component to be identified; The method for acquiring the multimodal fused image includes: For each individual electronic component, a set of optical, ultrasonic and infrared images are taken at the same location. These three images are then fused into a single multimodal high-resolution image data. No changes are made to the original images; only correlation fusion is performed. In other words, three images taken at the same location are recorded as a set of data. The trimodal fusion convolutional neural network model includes a feature extraction module, a feature fusion module, and a prediction module. The feature extraction module includes three independent channels, each with 10 convolutional layers, used to extract optical, ultrasonic, and infrared data, respectively. The feature fusion module includes a feature concatenation operation and eight convolutional layers, taking the outputs of the three independent channels as input, performing a concat operation followed by a convolution operation to achieve feature fusion. The prediction module includes a defect category prediction module and a defect location prediction module, taking the output of the feature fusion module as input to two fully connected layers.

2. The method for identifying defects in electronic components based on convolutional neural networks according to claim 1, characterized in that, Defect types in electronic components include cracks, delamination, scratches, missing corners, oil stains, scratches, breakage, spots, holes, and particulate contamination; The cracks, scratches, and abrasions are first determined using the length ratio method. When the following conditions are met, the electronic component is deemed to have cracks, scratches, or abrasion defects. Then, ultrasonic imaging is used to examine the internal structure of the component to determine cracks, and actual functional testing is used to distinguish between scratches and abrasions. Among them, L ′ The length of the defect is represented by L, which represents the length of the diagonal of the face. The spots or oil stains are first determined using the area ratio method. When the following conditions are met, the electronic component is judged to have spot or oil stain defects. Then, they are distinguished using white light in a dark environment; the oil stains exhibit a specific, brilliant luster. Among them, S ′ The area of ​​the defect is represented by S, where S represents the area of ​​the face containing the defect. The missing corner or breakage is determined using the volume ratio method. When the following conditions are met, the electronic component is deemed to have a corresponding defect; where the defect ratio is greater than 0.1, the defect is directly identified as breakage: Among them, V ′ V represents the defect volume, and V represents the volume of the electronic component; when If the value is greater than 0.1, the defect type is determined to be breakage; For delamination, pores, or particulate contamination, a counting method is used to determine the defects, with the number of defects being n. When n≥1, the defect type is determined to be delamination or pores, and delamination and pores are distinguished by infrared / ultrasonic images. When n≥2, the defect type is determined to be particulate contamination.

3. The method for identifying defects in electronic components based on convolutional neural networks according to claim 1, characterized in that, Once the defect type is determined, the smallest rectangular box in the multimodal high-resolution image data that can encompass the defect location is denoted as the defect location.

4. The method for identifying defects in electronic components based on convolutional neural networks according to claim 1, characterized in that, The feature extraction module performs feature extraction based on the following formula: in, This represents the optical data output by the l-th convolutional layer of the first channel; This represents the infrared data output by the l-th convolutional layer in the second channel; This represents the ultrasound data output from the l-th convolutional layer in the third channel; This represents the optical data output from the l-1 convolutional layer of the first channel; This represents the infrared data output by the l-1 convolutional layer of the second channel; This represents the ultrasound data output from the l-1 convolutional layer of the third channel; This represents the filter kernel function of the l-th convolutional layer; represents the bias coefficient of the l-th convolutional layer; x∈[1,Num], where Num represents the maximum kernel size; * represents the convolution operation; The feature fusion module performs feature fusion using the following formula: in, This represents the feature map output by the 0th convolutional layer after fusion. This represents the feature map output by the l-th convolutional layer after fusion. This represents the feature map output by the (l-1)th convolutional layer after fusion; Concat represents the fusion and concatenation operation of the features of the three images; YLU represents the exponential linear unit activation function, used for non-linear transformation operations between layers. In the prediction module, the two fully connected layers are processed as follows: in, This represents the l-th layer of the category prediction task, i.e., the category prediction task mapping layer, which outputs the predicted category. This represents the l-th layer of the defect location prediction task, i.e., the defect location prediction task mapping layer, which outputs the predicted defect location. This represents the (l-1)th layer feature map output by the category prediction task; This represents the (l-1)th layer feature map output by the defect location prediction task; This represents the kernel function of the (l-1)th layer filter in the category prediction task; This represents the kernel function of the (l-1)th layer filter in the defect location prediction task; This represents the bias coefficient of the (l-1)th layer in the category prediction task; This represents the bias coefficient of layer l-1 in the defect location prediction task.

5. The method for identifying defects in electronic components based on convolutional neural networks according to claim 4, characterized in that, Classification regression loss for defect category prediction task of electronic components DCP The calculation method is as follows: Regression loss for defect location prediction in electronic components DLP The calculation method is as follows: Where N represents the number of samples taken from the training set and used to train the prediction model; This represents the predicted value obtained by the prediction model for the defect type of the i-th sample and the j-th classifier. This represents the true value of the defect type of the i-th sample and the j-th classifier; L represents the predicted value of the defect location in the i-th sample; i This represents the true value of the defect location in the i-th sample.

6. The method for identifying defects in electronic components based on convolutional neural networks according to claim 1, characterized in that, Based on the labeled original image set, for each type of defect image data, 40% of the sample data is randomly selected as training data and placed in the normal training set, 40% of the sample data is randomly selected for incremental defect category labeling and placed as training data in the incremental training set, and 20% of the sample data is randomly selected as test data and placed in the test set.

7. The method for identifying defects in electronic components based on convolutional neural networks according to claim 1, characterized in that, When the metrics meet the preset requirements, a well-trained three-modal fusion convolutional neural network model is obtained, including: The performance of the three-modal fusion convolutional neural network model is evaluated using four metrics. When the performance of the prediction model meets the preset requirements of the four metrics, the three-modal fusion convolutional neural network model is determined to be a well-trained three-modal fusion convolutional neural network model, as detailed below: Calculate the values ​​of Recall, Precision, FPS (Functions Per Second), and IOP (Incremental Performance) using the following formulas: Where, N TP This indicates the number of correctly identified targets. The marked bounding box represents the location of the defect in the test set, and the predicted bounding box represents the location of the defect in the test set predicted by the model. When the intersection-union ratio (IU) of the predicted bounding box and the marked bounding box is greater than or equal to 0.3, and the category identification is correct, the target is considered correctly identified; N gt N represents the number of manually labeled boxes. Predict-P t represents the total number of prediction boxes that are predicted to be defects; begin Indicates the time point at which image defect prediction begins; t end Indicates the time point at which image defect prediction ends; N incre-category N represents the number of incremental defect categories when training the model on the incremental training set after training the model on the normal training set; origin-category N represents the number of defect categories in the original, already trained model before incremental training; incre-images N represents the number of incremental training images used during incremental training after the model has been fully trained. origin-images Recall represents the number of training images used by the originally trained model before incremental training. end Recall represents the recall rate after incremental learning training is completed. origin Precision represents the recall rate after training is completed before incremental learning. end Precision represents the accuracy after incremental learning training is completed. origin This represents the accuracy rate before incremental learning and after training. If the obtained defect recall index Recall≥0.95, the obtained precision index Precision≥0.95, the obtained model processing speed index FPS≥30, and the obtained incremental performance index IOP≥0.9, then the model is determined to be a trained trimodal fusion convolutional neural network model.

8. A defect identification device for electronic components based on convolutional neural networks, characterized in that, The method based on any one of claims 1 to 7 includes: The original image set acquisition module is used to acquire multimodal fusion images of multiple electronic components, including those with defects, as the original image set; The data annotation module is used to determine the defect type and location of each electronic component in the original image set according to the electronic component defect definition standard, and to annotate the annotated data into training set and test set. The model training module is used to train the trimodal fusion convolutional neural network model using the training set and to test the trained trimodal fusion convolutional neural network model using the test set. When the indicators meet the preset requirements, the trained trimodal fusion convolutional neural network model is obtained. The defect identification module is used to input the multimodal fusion image of the electronic component to be identified into a trained three-modal fusion convolutional neural network model to obtain the defect type and defect location of the electronic component to be identified.

Images