Electronic component defect detection method and device, equipment and storage medium
A cross-modal, multi-task deep learning model enhances defect detection in electronic components by extracting and enhancing features from image and point cloud data, addressing the limitations of existing technologies in data dependency and cost, thereby improving accuracy and versatility.
Patent Information
- Application Number
- CN202411210988.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2025-01-03
AI Technical Summary
Existing deep learning-based defect detection technologies for electronic components require significant manpower and financial resources for new data collection and model retraining, limiting their versatility and efficiency.
A method and device that utilize a cross-modal, multi-task integrated circuit defect detection model based on deep learning, incorporating image and point cloud data, with feature extraction, mixing, enhancement, and attention calculation to accurately detect defects without extensive field data collection.
Improves defect detection accuracy and versatility while reducing costs by leveraging a cross-modal model that extracts and enhances features, enabling efficient defect identification in electronic components.
Smart Images
Figure CN119251151A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for detecting defects in electronic components. Background Art
[0002] In the electronics manufacturing industry, defect detection of electronic components is an important part of ensuring that products meet high quality standards. Traditional defect detection mainly relies on manual work, which has problems such as low efficiency, high error rate and high cost. With the continuous development of computer technology, especially in the fields of image processing and deep learning, defect detection technology based on machine vision has begun to receive widespread attention from academia and industry.
[0003] At present, deep learning-based defect detection technology has significant advantages, such as achieving good results in complex defect detection and defect detection under unbalanced data. However, there are also some challenges. Most of the existing deep learning-based defect detection technologies usually rely on a large amount of customer site data for training, which requires a lot of manpower, material and financial resources for new data collection and model retraining, limiting the versatility of such equipment. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a method, device, equipment and storage medium for detecting defects in electronic components, so as to improve the accuracy and versatility of detecting defects in electronic components and reduce the cost of defect detection.
[0005] In order to solve the above technical problems, the present application provides a method for detecting defects of electronic components, including:
[0006] Collecting image data or sensor data of electronic components to obtain initial data, and preprocessing the initial data to generate basic data;
[0007] Inputting the basic data into a defect detection model, performing high-level feature extraction and feature mixing on the image data and point cloud data in the basic data to obtain a first mixed feature;
[0008] Aggregate the first mixed features and extract global features to obtain aggregated features and global features;
[0009] The aggregated features and the global features are mixed and then feature enhanced to obtain enhanced features;
[0010] Performing channel attention and spatial attention calculation on the enhanced features to generate attention features;
[0011] The attention features are subjected to feature transformation and feature decoding to obtain target detection results.
[0012] In order to solve the above technical problems, the present application provides an electronic component defect detection device, including:
[0013] A data acquisition unit, used to acquire image data or sensor data of electronic components to obtain initial data, and pre-process the initial data to generate basic data;
[0014] A feature mixing unit, used for inputting the basic data into a defect detection model, performing high-level feature extraction and feature mixing on the image data and point cloud data in the basic data, and obtaining a first mixed feature;
[0015] A feature extraction unit, used for performing feature aggregation and global feature extraction on the first mixed features to obtain aggregated features and global features;
[0016] A feature enhancement unit, used for enhancing the aggregated feature and the global feature after feature mixing to obtain an enhanced feature;
[0017] An attention calculation unit, used to perform channel attention and spatial attention calculation on the enhanced features to generate attention features;
[0018] The detection result generating unit is used to perform feature transformation and feature decoding on the attention feature to obtain a target detection result.
[0019] To solve the above technical problems, a technical solution adopted by the present invention is: to provide an electronic device, including one or more processors; a memory for storing one or more programs, so that the one or more processors can implement any one of the above-mentioned methods for detecting defects in electronic components.
[0020] In order to solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements any one of the above-mentioned methods for detecting defects in electronic components.
[0021] The embodiment of the present invention provides a method, device, equipment and storage medium for detecting defects in electronic components. The method includes: collecting image data or sensor data of electronic components to obtain initial data, and preprocessing the initial data to generate basic data; inputting the basic data into the defect detection model, performing high-level feature extraction and feature mixing on the image data and point cloud data in the basic data to obtain a first mixed feature; performing feature aggregation and global feature extraction on the first mixed feature to obtain aggregated features and global features; performing feature enhancement on the aggregated features and the global features after feature mixing to obtain enhanced features; performing channel attention and spatial attention calculation on the enhanced features to generate attention features; performing feature transformation and feature decoding on the attention features to obtain target detection results. The embodiment of the present invention extracts features of different levels of the collected electronic component data, and mixes, enhances and calculates the features of the features, so as to accurately identify the relevant features of the electronic components. At the same time, the present application does not require a large amount of field data for data collection and retraining, which is conducive to improving the detection accuracy and versatility of electronic component defects and reducing the cost of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0023] Figure 1 It is a flow chart of the implementation process of the method for detecting defects of electronic components provided in an embodiment of the present application;
[0024] Figure 2 It is a flowchart for implementing a sub-process in the method for detecting defects of electronic components provided in an embodiment of the present application;
[0025] Figure 3 It is a flowchart for implementing a sub-process in the method for detecting defects of electronic components provided in an embodiment of the present application;
[0026] Figure 4 It is a flowchart for implementing a sub-process in the method for detecting defects of electronic components provided in an embodiment of the present application;
[0027] Figure 5 It is a flowchart for implementing a sub-process in the method for detecting defects of electronic components provided in an embodiment of the present application;
[0028] Figure 6 It is a flowchart for implementing a sub-process in the method for detecting defects of electronic components provided in an embodiment of the present application;
[0029] Figure 7 is a flowchart for implementing a process in a method for detecting defects of electronic components provided in another embodiment of the present application;
[0030] Figure 8 is a schematic diagram of a device for detecting defects of electronic components provided in an embodiment of the present application;
[0031] Fig. 9 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0033] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0035] The present invention is described in detail below with reference to the accompanying drawings and embodiments.
[0036] It should be noted that the electronic component defect detection method provided in the embodiment of the present application is generally executed by a server, and accordingly, the electronic component defect detection device is generally configured in the server.
[0037] See also Figure 1 , Figure 1 A specific implementation of a method for detecting defects in electronic components is shown.
[0038] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 1 The process sequence shown is limited to the following steps:
[0039] S1: Collect image data or sensor data of electronic components to obtain initial data, and pre-process the initial data to generate basic data.
[0040] Specifically, image data or sensor data of electronic components are collected from the production line and used as initial data. Preprocessing operations such as denoising, standardization and enhancement are performed on the initial data to generate basic data to improve the accuracy of subsequent detection algorithms.
[0041] Furthermore, computer vision technology or signal processing methods are used to extract useful features from the basic data, such as texture, shape, color, etc., so that the extracted data can input the features in the data into the subsequent defect detection model to detect defects in electronic components.
[0042] S2: Input the basic data into a defect detection model, perform high-level feature extraction and feature mixing on the image data and point cloud data in the basic data, and obtain a first mixed feature.
[0043] Specifically, the defect detection model HYM-I CDET uses deep learning technology to realize the automatic detection of integrated circuit defects by integrating data from two different modalities, image and point cloud. The model combines multiple components such as image feature transformation module (IFTM), point cloud feature transformation module (PFTM), VIT module, high-level aggregation module, FPN module and CABM module to form a complete detection process. The defect detection model is a cross-modal, multi-task integrated circuit defect detection model based on a deep learning large model.
[0044] In an embodiment of the present application, the basic data is input into a defect detection model, and firstly, the image data and point cloud data in the basic data are subjected to high-level feature extraction and feature mixing through an image feature transformation module (IFTM) and a point cloud feature transformation module (PFTM) to obtain a first mixed feature.
[0045] See also Figure 2 , Figure 2 A specific implementation of step S2 is shown, which is described in detail as follows:
[0046] S21: Input the basic data into the defect detection model, remove noise and perform image enhancement processing on the image data in the basic data, and obtain high-level features of the image.
[0047] Specifically, the image data in the basic data is subjected to noise removal and image enhancement processing by an image feature transformation module (IFTM) to obtain high-level image features. The image feature transformation module removes noise from the image by applying filters or other image processing techniques to improve the accuracy of subsequent processing. The image in the basic data is enhanced to increase contrast, increase details or improve image quality. The enhancement methods used in the embodiments of the present application include histogram equalization, contrast stretching and sharpening. The image is standardized to eliminate differences in brightness, color and scale between different images. This helps to ensure the consistency of feature extraction and defect detection algorithms on different images.
[0048] S22: performing denoising and filtering processing on the point cloud data in the basic data to obtain filtered point cloud data, and performing sampling processing or downsampling processing on the filtered point cloud data to obtain point cloud high-level features.
[0049] Specifically, the point cloud data in the basic data is denoised and filtered by the point cloud feature transformation module (PFTM) to obtain filtered point cloud data, thereby removing invalid or abnormal points in the point cloud data. Then, the filtered point cloud data is sampled or downsampled to obtain high-level features of the point cloud, thereby reducing the amount of data or balancing the point cloud density, so as to more efficiently process and extract features.
[0050] S23: performing channel mixing and channel rearrangement on the image high-level features and the point cloud high-level features, so as to perform feature mixing on the image high-level features and the point cloud high-level features, and obtain the first mixed features.
[0051] Specifically, the image high-level features and the point cloud high-level features are channel mixed and channel rearranged to achieve mixing of different features and obtain a first mixed feature.
[0052] See also Figure 3 , Figure 3 A specific implementation method before step S2 is shown, which is described in detail as follows:
[0053] S2A: Collect image data or sensor data of electronic components from the production line to obtain initial training data.
[0054] S2B: Preprocessing and defect labeling the initial training data to obtain target training data.
[0055] S2C: Train the initial defect detection model based on the target training data to generate a multi-task output result, and calculate the model loss based on a preset loss function and the multi-task output result.
[0056] The multi-task output results include two-dimensional image segmentation results, two-dimensional target detection results, three-dimensional point cloud segmentation results and three-dimensional target detection results.
[0057] S2D: adjusting the parameters of the initial defect detection model according to the model loss, and re-iteratively training the initial defect detection model to obtain the defect detection model.
[0058] Specifically, the embodiment of the present application is based on the cross-modal and multi-task technology of the deep learning large model, and proposes a defect detection model for integrated circuits. Therefore, it is necessary to train the defect detection model before the model is deployed. In order to effectively train the large model and to cope with the imbalance of the data volume of the four tasks in the large model, the loss function of the model is modified in the present application example, and different weight ratios are assigned to the four tasks (two-dimensional image segmentation, two-dimensional target detection, three-dimensional point cloud segmentation, and three-dimensional target detection). Its specific training process: collect image data or sensor data of electronic components from the production line to obtain initial training data; pre-process and mark defects on the initial training data to obtain target training data; train the initial defect detection model based on the target training data, generate multi-task output results, and calculate the model loss based on the preset loss function and the multi-task output results, wherein the multi-task output results include two-dimensional image segmentation results, two-dimensional target detection results, three-dimensional point cloud segmentation results, and three-dimensional target detection results; adjust the parameters of the initial defect detection model through model loss, and iteratively train the initial defect detection model again to obtain a defect detection model.
[0059] Furthermore, the preset loss function provided in the embodiment of the present application is:
[0060] L(Y,P(Y|X))=-Alog(P(Y|X))-Blog(P(Y|X))-Clog(P(Y|X))-Dlog(P(Y|X))-E;
[0061] Among them, the loss function L(Y, P(Y|X)) expresses that when sample X is classified as Y, the probability P(Y|X) reaches the maximum value, P(Y|X) is the probability value, A is the weight ratio of the two-dimensional image segmentation result, B is the weight ratio of the two-dimensional target detection result, C is the weight ratio of the three-dimensional point cloud segmentation result, D is the weight ratio of the three-dimensional target detection result, and E is an adjustable parameter.
[0062] S3: Aggregate the first mixed features and extract global features to obtain aggregated features and global features.
[0063] Specifically, the first mixed features are sent to the high-efficiency layer aggregation module and the visual conversion module for feature aggregation and global feature extraction, respectively, to obtain aggregated features and global features.
[0064] See also Figure 4 , Figure 4 A specific implementation of step S3 is shown, which is described in detail as follows:
[0065] S31: Input the first mixed features into the efficient layer aggregation module and the visual conversion module respectively.
[0066] S32: In the high-efficiency layer aggregation module, the aggregation feature is obtained by performing dilated convolution processing and feature mixing on the first mixed feature.
[0067] Specifically, in the high-efficiency layer aggregation module, the first mixed feature is convolved twice through dilated convolution to obtain a first dilated convolution; the first dilated convolution is merged with another first mixed feature, and the merged feature is convolved twice through dilated convolution to obtain a second dilated feature; the second dilated feature is merged with another first mixed feature, and the merged feature is convolved three times through dilated convolution to obtain a third dilated feature. Finally, the first mixed feature, the first dilated convolution, the second dilated feature, and the third dilated feature are feature mixed, and the feature after feature mixing is further subjected to dilated convolution to obtain an aggregated feature.
[0068] S33: In the visual conversion module, the first mixed feature is divided into a plurality of vector sequences, and the correlation degree between the vector sequences is calculated to perform feature extraction on the first mixed feature to obtain the global feature.
[0069] Specifically, the stacked Transformer is the main feature extractor of the visual transformation module (VI T module). The core idea of the Transformer is to capture the dependencies in the input sequence through the self-attention mechanism without relying on traditional recursive or convolution operations. In the Transformer, the input sequence is divided into multiple vector representations, and then the self-attention mechanism is used to calculate the degree of association between each input vector and other input vectors. This degree of association is calculated by calculating the similarity between the input vectors, and then weighted summing is performed according to the similarity. This enables the encoder to pay attention to all positions in the input sequence at the same time and capture global dependencies. The advantages of the Transformer include strong parallel computing capabilities, the ability to process variable-length sequences, and strong ability to model long-distance dependencies.
[0070] In the embodiment of the present application, the first mixed feature is divided into multiple vector sequences, each of which is a feature representation of the image block. The similarity between each vector sequence and other vector sequences is calculated to calculate the degree of association between the vector sequences; then the similarities are weighted and summed to achieve feature extraction of the first mixed feature and obtain a global feature.
[0071] S4: The aggregated features and the global features are mixed and then enhanced to obtain enhanced features.
[0072] See also Figure 5 , Figure 5 A specific implementation of step S4 is shown, which is described in detail as follows:
[0073] S41: Mixing the aggregated feature and the global feature to obtain a second mixed feature.
[0074] S42: Fusing the upper and lower layer semantic information with the second mixed feature to enhance the second mixed feature to obtain the enhanced feature.
[0075] Specifically, FPN (Feature Pyramid Network) is a feature pyramid from top to bottom, which transmits the strong semantic features of the high level and enhances the entire pyramid. However, it only enhances the semantic information, but does not transmit the positioning information (or because the upward transmission path is too long and the transmission effect is not good). In the detection network, an image is input into the network, and after layers of convolution, the semantic information will become more and more obvious, but the relative position information will become weaker and weaker, because the higher the convolution, the larger the receptive field of the feature map mapped to the original image, so the local position information is felt poorly. Therefore, in the embodiment of the present application, a bottom-up pyramid is added behind FPN to supplement FPN and transmit the positioning features of the low level. The pyramid formed in this way combines semantic information and has positioning information, that is, a fused feature pyramid (Path Aggregation Network). That is, in the embodiment of the present application, the aggregated features and the global features are first mixed by fusion feature pyramid to obtain a second mixed feature, and then the FPN in the fusion feature pyramid is used to fuse the upper and lower semantic information of the second mixed feature to enhance the second mixed feature to obtain the enhanced feature.
[0076] S5: Perform channel attention and spatial attention calculation on the enhanced features to generate attention features.
[0077] Specifically, the embodiment of the present application is based on a convolutional network with a CBAM (Convolutional Block Attention Module) attention mechanism to extract the main spatial features and main channel features of the signal. The main purpose of the attention mechanism is to focus on the important features of the image and suppress unnecessary regional responses.
[0078] See also Figure 6 , Figure 6 A specific implementation of step S5 is shown, which is described in detail as follows:
[0079] S51: performing average pooling and maximum pooling processing on the enhanced features respectively to obtain a first average pooling feature and a first maximum pooling feature.
[0080] S52: Perform feature processing based on the first average pooling feature and the first maximum pooling feature through a multi-layer perceptron network to obtain a channel attention feature.
[0081] The embodiment of the present application utilizes the channel attention mechanism to extract the channel features of the enhanced features. Specifically, global average pooling (GAP) and global maximum pooling (GMP) are performed on the input enhanced features to obtain the first average pooling features and the first maximum pooling features. The first average pooling features and the first maximum pooling features represent the average features and the maximum features of the feature map in the global range, respectively. The first average pooling features and the first maximum pooling features are then sent to a multi-layer perceptron (MLP) with shared weights. This MLP is usually composed of two fully connected layers, the first fully connected layer is used for dimensionality reduction, and the second fully connected layer is used to restore the original dimension. The ReLU activation function is usually used between the two fully connected layers.
[0082] The first average pooling feature and the first maximum pooling feature after MLP processing are added element by element, and the channel attention weight is obtained through the sigmo id activation function. The sigmo id function maps the channel attention weight to the range of [0,1], indicating the importance of each channel. Finally, the channel attention weight is multiplied element by element with the enhanced feature to obtain the channel attention feature. The embodiment of the present application realizes the recalibration of the feature map channel and enhances the feature representation of important channels.
[0083] S53: Perform average pooling and maximum pooling processing on the channel attention features in the channel dimension to obtain second average pooling and second maximum pooling features.
[0084] S54: Concatenate the second average pooling features and the second maximum pooling features, and perform convolution processing on the concatenated features to obtain the attention features.
[0085] Specifically, the channel attention features are average pooled and max pooled in the channel dimension to obtain the second average pooled and second max pooled features. These two feature maps represent the average feature and the maximum feature of the feature map in space, respectively.
[0086] The second average pooling and the second maximum pooling features are concatenated to obtain a new feature. Then a standard convolution layer (such as a 7x7 convolution kernel) is used to perform a convolution operation on the concatenated feature to generate a spatial attention feature. The spatial attention feature is then Sigma id activated to obtain the spatial attention weight. The Sigma id function maps the weight value to the range of [0,1], indicating the importance of each position on the feature map. Finally, the spatial attention weight is element-by-element multiplied with the channel attention feature to obtain the final attention feature. The embodiment of the present application realizes the recalibration of the spatial position of the feature map and enhances the feature representation of important areas.
[0087] S6: Perform feature transformation and feature decoding on the attention feature to obtain a target detection result.
[0088] Specifically, the attention features are transformed and decoded through the decoder of the defect detection model to obtain the target detection result.
[0089] See also Figure 7 , Figure 7 A specific implementation of step S6 is shown, which is described in detail as follows:
[0090] S61: Perform feature transformation and feature decoding on the attention feature through stacked deconvolution layers and upsampling layers to obtain decoded features.
[0091] S62: Outputting multi-task output results based on the decoded features through a multi-output head to obtain the target detection result.
[0092] Specifically, the decoder of the defect detection model includes multiple output heads and stacked deconvolution layers and upsampling layers. In the embodiment of the present application, the attention features are transformed and decoded by stacked deconvolution layers and upsampling layers to obtain decoded features; the multi-task output results are output based on the decoded features by the multiple output heads to obtain target detection results. Among them, the multi-task output results include two-dimensional image segmentation results, two-dimensional target detection results, three-dimensional point cloud segmentation results and three-dimensional target detection results.
[0093] In an embodiment of the present application, image data or sensor data of an electronic component is collected to obtain initial data, and the initial data is preprocessed to generate basic data; the basic data is input into a defect detection model, and high-level feature extraction and feature mixing are performed on the image data and point cloud data in the basic data to obtain a first mixed feature; the first mixed feature is feature aggregated and global feature extracted to obtain aggregated features and global features; the aggregated features and the global features are feature enhanced after feature mixing to obtain enhanced features; channel attention and spatial attention calculations are performed on the enhanced features to generate attention features; feature transformation and feature decoding are performed on the attention features to obtain target detection results. The embodiment of the present invention extracts features of different levels of the collected electronic component data, and mixes, enhances and calculates the features of the features, so as to accurately identify the relevant features of the electronic component. At the same time, the present application does not require a large amount of field data for data collection and retraining, which is conducive to improving the detection accuracy and versatility of electronic component defects and reducing the cost of defect detection.
[0094] Please refer to Figure 8 , as a response to the above Figure 1 The present application provides an embodiment of a device for detecting defects of electronic components. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0095] like Figure 8 As shown, the electronic component defect detection device of this embodiment includes: a data acquisition unit 71, a feature mixing unit 72, a feature extraction unit 73, a feature enhancement unit 74, an attention calculation unit 75 and a detection result generation unit 76, wherein:
[0096] The data acquisition unit 71 is used to acquire image data or sensor data of electronic components to obtain initial data, and pre-process the initial data to generate basic data;
[0097] A feature mixing unit 72 is used to input the basic data into the defect detection model, perform high-level feature extraction and feature mixing on the image data and point cloud data in the basic data, and obtain a first mixed feature;
[0098] A feature extraction unit 73, configured to perform feature aggregation and global feature extraction on the first mixed features to obtain aggregated features and global features;
[0099] A feature enhancement unit 74 is used to enhance the aggregated feature and the global feature after feature mixing to obtain an enhanced feature;
[0100] An attention calculation unit 75, used to perform channel attention and spatial attention calculation on the enhanced features to generate attention features;
[0101] The detection result generating unit 76 is used to perform feature transformation and feature decoding on the attention feature to obtain a target detection result.
[0102] Furthermore, the feature mixing unit 72 includes:
[0103] An image high-level feature acquisition unit, used to input the basic data into a defect detection model, remove noise from the image data in the basic data and perform image enhancement processing to obtain image high-level features;
[0104] A point cloud high-level feature acquisition unit is used to perform denoising and filtering processing on the point cloud data in the basic data to obtain filtered point cloud data, and perform sampling processing or downsampling processing on the filtered point cloud data to obtain point cloud high-level features;
[0105] The first mixed feature generating unit is used to perform channel mixing and channel rearrangement on the image high-level features and the point cloud high-level features, so as to perform feature mixing on the image high-level features and the point cloud high-level features to obtain the first mixed features.
[0106] Furthermore, the feature extraction unit 73 includes:
[0107] A feature input unit, used to input the first mixed features into the efficient layer aggregation module and the visual conversion module respectively;
[0108] an aggregate feature generating unit, configured to obtain the aggregate feature by performing dilated convolution processing and feature mixing on the first mixed feature in the high-efficiency layer aggregation module;
[0109] The global feature generation unit is used to divide the first mixed feature into multiple vector sequences in the visual conversion module, and extract features from the first mixed feature by calculating the degree of association between the vector sequences to obtain the global feature.
[0110] Furthermore, the feature enhancement unit 74 includes:
[0111] A second mixed feature generating unit, used for performing feature mixing on the aggregated feature and the global feature to obtain a second mixed feature;
[0112] The enhanced feature generating unit is used to fuse the upper and lower layer semantic information with the second mixed feature to enhance the second mixed feature to obtain the enhanced feature.
[0113] Further, the attention calculation unit 75 includes:
[0114] A first pooling unit, used to perform average pooling and maximum pooling processing on the enhanced features respectively, to obtain a first average pooling feature and a first maximum pooling feature;
[0115] a channel attention feature generating unit, configured to perform feature processing based on the first average pooling feature and the first maximum pooling feature through a multi-layer perceptron network to obtain a channel attention feature;
[0116] A second pooling unit is used to perform average pooling and maximum pooling processing on the channel attention feature in the channel dimension to obtain second average pooling and second maximum pooling features;
[0117] An attention feature generating unit is used to concatenate the second average pooling features and the second maximum pooling features, and perform convolution processing on the concatenated features to obtain the attention feature.
[0118] Furthermore, the detection result generating unit 76 includes:
[0119] A feature transformation unit, used to perform feature transformation and feature decoding on the attention feature through stacked deconvolution layers and upsampling layers to obtain decoded features;
[0120] A result generating unit is used to output a multi-task output result based on the decoded features through a multi-output head to obtain the target detection result.
[0121] Furthermore, the feature mixing unit 72 further includes:
[0122] An initial training data acquisition unit, used to acquire image data or sensor data of electronic components from a production line to obtain initial training data;
[0123] A target training data generating unit, used for preprocessing and defect marking the initial training data to obtain target training data;
[0124] A model loss calculation unit, used to train the initial defect detection model based on the target training data, generate a multi-task output result, and calculate the model loss based on a preset loss function and the multi-task output result, wherein the multi-task output result includes a two-dimensional image segmentation result, a two-dimensional target detection result, a three-dimensional point cloud segmentation result, and a three-dimensional target detection result;
[0125] A defect detection model generating unit, configured to adjust the parameters of the initial defect detection model according to the model loss, and iteratively train the initial defect detection model again to obtain the defect detection model;
[0126] The preset loss function is:
[0127] L(Y,P(Y|X))=-Alog(P(Y|X))-Blog(P(Y|X))-Clog(P(Y|X))-Dlog(P(Y|X))-E;
[0128] Among them, the loss function L(Y, P(Y|X)) expresses that when sample X is classified as Y, the probability P(Y|X) reaches the maximum value, P(Y|X) is the probability value, A is the weight ratio of the two-dimensional image segmentation result, B is the weight ratio of the two-dimensional target detection result, C is the weight ratio of the three-dimensional point cloud segmentation result, D is the weight ratio of the three-dimensional target detection result, and E is an adjustable parameter.
[0129] In an embodiment of the present application, image data or sensor data of an electronic component is collected to obtain initial data, and the initial data is preprocessed to generate basic data; the basic data is input into a defect detection model, and high-level feature extraction and feature mixing are performed on the image data and point cloud data in the basic data to obtain a first mixed feature; the first mixed feature is feature aggregated and global feature extracted to obtain aggregated features and global features; the aggregated features and the global features are feature enhanced after feature mixing to obtain enhanced features; channel attention and spatial attention calculations are performed on the enhanced features to generate attention features; feature transformation and feature decoding are performed on the attention features to obtain target detection results. The embodiment of the present invention extracts features of different levels of the collected electronic component data, and mixes, enhances and calculates the features of the features, so as to accurately identify the relevant features of the electronic component. At the same time, the present application does not require a large amount of field data for data collection and retraining, which is conducive to improving the detection accuracy and versatility of electronic component defects and reducing the cost of defect detection.
[0130] To solve the above technical problems, the present application also provides an electronic device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the electronic device of this embodiment.
[0131] The electronic device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected through a system bus. It should be noted that the figure only shows an electronic device 8 having three components: a memory 81, a processor 82, and a network interface 83, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the electronic device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.
[0132] The memory 81 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 81 can be an internal storage unit of the electronic device 8, such as a hard disk or memory of the electronic device 8. In other embodiments, the memory 81 can also be an external storage device of the electronic device 8, such as a plug-in hard disk equipped on the electronic device 8, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 81 can also include both the internal storage unit of the electronic device 8 and its external storage device. In this embodiment, the memory 81 is generally used to store the operating system and various application software installed on the electronic device 8, such as the program code of the method for detecting defects of electronic components, etc. In addition, the memory 81 can also be used to temporarily store various types of data that have been output or are to be output.
[0133] The processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 82 is generally used to control the overall operation of the electronic device 8. In this embodiment, the processor 82 is used to run the program code stored in the memory 81 or process data, such as running the program code of the above-mentioned electronic component defect detection method to implement various embodiments of the electronic component defect detection method.
[0134] The network interface 83 may include a wireless network interface or a wired network interface, and the network interface 83 is generally used to establish a communication connection between the electronic device 8 and other electronic devices.
[0135] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores a computer program. The computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned electronic component defect detection method.
[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.
[0137] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is also within the scope of patent protection of this application.
Claims
1. A method for detecting defects in electronic components, characterized in that: include: Collecting image data or sensor data of electronic components to obtain initial data, and preprocessing the initial data to generate basic data; Inputting the basic data into a defect detection model, performing high-level feature extraction and feature mixing on the image data and point cloud data in the basic data to obtain a first mixed feature; Aggregate the first mixed features and extract global features to obtain aggregated features and global features; The aggregated features and the global features are mixed and then feature enhanced to obtain enhanced features; Performing channel attention and spatial attention calculation on the enhanced features to generate attention features; The attention features are subjected to feature transformation and feature decoding to obtain target detection results.
2. The method for detecting defects in electronic components according to claim 1, characterized in that: The step of inputting the basic data into the defect detection model, performing high-level feature extraction and feature mixing on the image data and point cloud data in the basic data to obtain a first mixed feature includes: Inputting the basic data into a defect detection model, removing noise from image data in the basic data and performing image enhancement processing to obtain high-level features of the image; De-noising and filtering the point cloud data in the basic data to obtain filtered point cloud data, and sampling or down-sampling the filtered point cloud data to obtain point cloud high-level features; Channel mixing and channel rearrangement are performed on the image high-level features and the point cloud high-level features to perform feature mixing on the image high-level features and the point cloud high-level features to obtain the first mixed features.
3. The method for detecting defects in electronic components according to claim 1, characterized in that: The step of aggregating the first mixed features and extracting global features to obtain aggregated features and global features includes: Inputting the first mixed features into an efficient layer aggregation module and a visual conversion module respectively; In the high-efficiency layer aggregation module, the aggregation feature is obtained by performing dilated convolution processing and feature mixing on the first mixed feature; In the visual conversion module, the first mixed feature is divided into a plurality of vector sequences, and the correlation degree between the vector sequences is calculated to perform feature extraction on the first mixed feature to obtain the global feature.
4. The method for detecting defects in electronic components according to claim 1, characterized in that: The step of mixing the aggregated features and the global features and performing feature enhancement to obtain enhanced features includes: Mixing the aggregated feature with the global feature to obtain a second mixed feature; The second mixed feature is fused with upper and lower layer semantic information to enhance the second mixed feature to obtain the enhanced feature.
5. The method for detecting defects in electronic components according to claim 1, characterized in that: The performing channel attention and spatial attention calculation on the enhanced features to generate attention features includes: Performing average pooling and maximum pooling processing on the enhanced features respectively to obtain a first average pooling feature and a first maximum pooling feature; Performing feature processing based on the first average pooling feature and the first maximum pooling feature through a multi-layer perceptron network to obtain a channel attention feature; Performing average pooling and maximum pooling processing on the channel attention feature in the channel dimension to obtain second average pooling and second maximum pooling features; The second average pooling and the second maximum pooling features are concatenated, and the concatenated features are convolved to obtain the attention features.
6. The method for detecting defects in electronic components according to any one of claims 1 to 5, characterized in that: The step of performing feature transformation and feature decoding on the attention feature to obtain a target detection result includes: Performing feature transformation and feature decoding on the attention feature through stacked deconvolution layers and upsampling layers to obtain decoded features; The target detection result is obtained by outputting multi-task output results based on the decoded features through a multi-output head.
7. The method for detecting defects in electronic components according to any one of claims 1 to 5, characterized in that: Before inputting the basic data into the defect detection model and performing high-level feature extraction and feature mixing on the image data and point cloud data in the basic data to obtain the first mixed feature, the method further includes: Collect image data or sensor data of electronic components from a production line to obtain initial training data; Preprocessing and defect-marking the initial training data to obtain target training data; Training the initial defect detection model based on the target training data to generate a multi-task output result, and calculating the model loss based on a preset loss function and the multi-task output result, wherein the multi-task output result includes a two-dimensional image segmentation result, a two-dimensional target detection result, a three-dimensional point cloud segmentation result, and a three-dimensional target detection result; Adjusting the parameters of the initial defect detection model according to the model loss, and re-iteratively training the initial defect detection model to obtain the defect detection model; The preset loss function is: L(Y,P(Y|X))=-Alog(P(Y|X))-Blog(P(Y|X))-Clog(P(Y|X))-Dlog(P(Y|X))-E; Among them, the loss function L(Y, P(Y|X)) expresses that when sample X is classified as Y, the probability P(Y|X) reaches the maximum value, P(Y|X) is the probability value, A is the weight ratio of the two-dimensional image segmentation result, B is the weight ratio of the two-dimensional target detection result, C is the weight ratio of the three-dimensional point cloud segmentation result, D is the weight ratio of the three-dimensional target detection result, and E is an adjustable parameter.
8. A device for detecting defects in electronic components, characterized in that: include: A data acquisition unit, used to acquire image data or sensor data of electronic components to obtain initial data, and pre-process the initial data to generate basic data; A feature mixing unit, used for inputting the basic data into a defect detection model, performing high-level feature extraction and feature mixing on the image data and point cloud data in the basic data, and obtaining a first mixed feature; A feature extraction unit, used for performing feature aggregation and global feature extraction on the first mixed features to obtain aggregated features and global features; A feature enhancement unit, used for enhancing the aggregated feature and the global feature after feature mixing to obtain an enhanced feature; An attention calculation unit, used to perform channel attention and spatial attention calculation on the enhanced features to generate attention features; The detection result generating unit is used to perform feature transformation and feature decoding on the attention feature to obtain a target detection result.
9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the method for detecting defects of electronic components as claimed in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for detecting defects of an electronic component according to any one of claims 1 to 7 is implemented.