Integrated model construction and detection method for detecting electronic component excess under multi-information carrier perspective

Abstract

The application discloses a multi-information carrier perspective integrated model construction and detection method for sealed electronic component foreign matter detection, and belongs to the technical field of sealed electronic component foreign matter detection. In order to solve the problem that the existing sealed electronic component foreign matter detection method only extracts features from a single perspective and performs detection, the application trains and selects data-classifiers with the best classification performance from the data perspective of sealed electronic component foreign matter detection signals; from the image perspective, time-domain image-neural networks and acoustic spectrum image-neural networks with the best classification performance are trained and selected on time-domain picture sets and acoustic spectrum picture sets respectively; from the audio perspective, an audio-neural network with the best classification performance is trained and selected; the weights of each optimal classifier or optimal neural network are obtained based on fuzzy rule decision-level fusion technology, and then the classifiers and neural networks are integrated, and the integrated model is used for detection in actual detection.

Description

Technical Field

[0001] This invention belongs to the field of foreign object detection technology for sealed electronic components, specifically relating to an integrated model construction method and detection method for foreign object detection in sealed electronic components. Background Technology

[0002] Sealed electronic components are core parts of aerospace equipment. During the manufacturing process, particles such as metal debris and welding residue may be left inside. These particles, whether introduced externally or generated internally, disrupt the stable state of matter and are called foreign matter. Aerospace equipment typically operates in extreme environments such as hypergravity and weightlessness. Foreign matter present inside sealed electronic components can be activated and undergo random motion, leading to component failure and posing a significant threat to the reliable operation of aerospace equipment. Therefore, foreign matter detection before sealed electronic components leave the factory for service is crucial. Particle collision noise detection is currently the internationally recognized method for foreign matter detection. It is based on the principle of vibration-generated sound to activate any foreign matter that may be present inside the device under test, and the presence of foreign matter is determined by capturing the resulting sound signals. Typically, the sound signal generated by the random motion of foreign matter is called the foreign matter signal.

[0003] Sealed electronic components have a high degree of integration and contain a large number of components, including some loose components. Thus, during foreign object detection, these loose components can be activated, causing periodic vibrations at fixed positions and generating sound signals, known as component signals. These component signals can interfere with foreign object detection, making accurate identification of the foreign object signal and component signal crucial for foreign object detection. Previously, some scholars have used signal detection methods in related research. For example, based on signal generation mechanisms, Zhang Hui et al., Gao Hongliang et al., and Chen Jinbao et al. analyzed the aperiodic and near-periodic characteristics, and non-stationary and near-stationary characteristics of foreign object signals and component signals in the time and frequency domains, respectively, attempting to quantify the differences in characteristics between the two signals using different indices or combinations of indices. However, the indices or combinations of indices used in these methods are not stable; often, indices or combinations of indices determined by a batch of detection signals are not applicable to a new batch of detection signals, even if these detection signals were collected under the same experimental conditions. Furthermore, while these methods perform well in identifying signals from unwanted objects and loose components, they struggle with mixed signals generated by the simultaneous movement of unwanted objects and loose components, often resulting in misjudgments. This reduces the practicality of these methods. More importantly, these methods rely on manual analysis and judgment, lacking automation, further limiting their application in real-world scenarios.

[0004] In machine learning, the attribute differences between different objects can be quantified as distribution differences between different data. By training a suitable classifier, data can be classified, thereby achieving the classification of different objects. Similarly, the characteristic differences between redundant object signals and component signals can be quantified to train a classifier. In recent years, scholars have been committed to conducting research on redundant object signal and component signal recognition based on machine learning. For example, Wang Guotao et al., Liu Haijiang et al., and Meng Si et al. selected multiple features that can stably and completely describe the characteristic differences between the two signals from the time domain, frequency domain, and time-frequency domain, and constructed datasets to train high-performance classifiers, such as support vector machines, random forests, and XGBoost, to automatically complete data classification and signal recognition. These methods effectively solve the problems of incomplete indicator use and low automation in signal detection methods, but they are also only applicable to redundant object signals and component signals, and fail to solve the problem of recognizing mixed signals or other complex signals. In addition, these methods only consider using feature engineering to convert signals into data as an information carrier, without considering converting them into other popular information carriers such as images and audio, and using advanced deep learning to complete signal recognition. Summary of the Invention

[0005] This invention addresses the problem of poor performance stability in existing methods for detecting foreign objects in sealed electronic components, which rely solely on feature extraction and detection from a single perspective.

[0006] An integrated model construction method for detecting foreign objects in sealed electronic components from the perspective of multiple information carriers includes:

[0007] For the foreign object detection signals of sealed electronic components acquired in real-world scenarios, the type of the foreign object detection signal is determined as a label, and the foreign object detection signal is processed as follows:

[0008] From a data perspective, features are selected from the time and frequency domains, different types of detection signals are transformed into feature vectors, and labels are added to each to construct datasets of different types of detection signals. Then, multiple classifiers are trained, and the classifier with the best classification performance is selected, which is called a data-classifier.

[0009] Based on the image perspective, time-domain waveforms of different types of detection signals are directly plotted and labeled to construct time-domain image sets for different types of detection signals. Simultaneously, spectrogram technology is used to convert different types of detection signals into different types of spectrograms, and labels are added to construct spectrogram image sets for different types of detection signals. Multi-neural network models are trained on the time-domain image sets and spectrogram image sets respectively, and the neural networks with the best classification performance are compared to obtain the results, which are called time-domain graph-neural network and spectrogram-neural network.

[0010] From an audio perspective, speech processing technology is used to convert different types of detection signals into audio of different types of detection signals, and labels are added to each type to construct audio sets of different types of detection signals. Then, multiple neural network models are trained, and the neural network with the best classification performance is compared and obtained. This is called an audio-neural network.

[0011] An ensemble model is constructed using a decision-level fusion technique based on fuzzy rules: In the initialization phase, the classification accuracy obtained by each optimal classifier or optimal neural network is divided into three fuzzy sets, and the corresponding membership functions are determined. Then, the membership degree of each optimal classifier or optimal neural network is calculated, and the weights of each optimal classifier or optimal neural network are obtained using a fuzzy inference process. The sum of all weights is the total weight. The ratio of the weights of the data-classifier, temporal graph-neural network, spectrogram-neural network, and audio-neural network to the total weight is used as the corresponding prediction weight. The prediction weights of the data-classifier, temporal graph-neural network, spectrogram-neural network, and audio-neural network are the prediction weights corresponding to their prediction results. Finally, the data-classifier, temporal graph-neural network, spectrogram-neural network, and audio-neural network are integrated to obtain an ensemble model.

[0012] Furthermore, the types of foreign object detection signals include: normal signals, foreign object signals, component signals, mixed signals, and excessively large component signals; the foreign object signal refers to a signal whose pulse sequence is a non-periodic sequence; the component signal refers to a signal whose pulse sequence is approximately a periodic sequence; the mixed signal refers to a signal whose pulse sequence exhibits both randomness and periodicity; the excessively large component signal refers to a signal whose pulse sequence has periodicity in time intervals but randomness in amplitude.

[0013] Furthermore, the data classifier employs a random forest model.

[0014] Furthermore, both the time-domain graph-neural network and the spectrogram-neural network adopt the improved VGG19 network model. The improvement method of the VGG19 network model is as follows:

[0015] The first convolutional kernel of VGG19 was changed from 3×3 to 7×7, and the PReLU activation function was used instead of the ReLU activation function. A multi-head attention mechanism based on the Transformer architecture was introduced into VGG19, which was set between the last convolutional block and the fully connected layer.

[0016] Furthermore, the audio-neural network is as follows:

[0017] For the input audio, features are first extracted using a Flatten layer; after the Flatten layer, at least three fully connected layers are used, each with a ReLU activation function; finally, a Softmax activation function is used as the output layer to output the probability distribution of each category.

[0018] or,

[0019] For the input audio, features are first extracted using 1D-CNN. 1D-CNN contains multiple convolutional units, each of which includes a convolutional layer and a pooling layer, and a Dropout layer is added after each pooling layer. After 1D-CNN, two fully connected layers are set, and then the Softmax activation function is used as the output layer to output the probability distribution of each category.

[0020] Furthermore, based on the classification accuracy obtained by each optimal classifier or optimal neural network, the data is divided into three fuzzy sets, and the corresponding membership functions are determined as follows:

[0021]

[0022] In the formula, μ Low (x), μ Medium (x), μ High (x) represents the membership function of the three fuzzy sets corresponding to low, medium and high, respectively; x represents 100 times the classification accuracy obtained by the optimal classifier or the optimal neural network.

[0023] Furthermore, the weights of each optimal classifier or optimal neural network are obtained using the fuzzy inference process as follows:

[0024] W(i)=μ High +α×μ Medium

[0025] Where W(i) represents the weight of each optimal classifier or optimal neural network, and α is the adjustment weight.

[0026] Preferably, the adjustment weight α is 0.5.

[0027] A method for detecting foreign matter in sealed electronic components from the perspective of multiple information carriers includes the following steps:

[0028] For the foreign object detection signals of sealed electronic components in real-world scenarios, an integrated model constructed using the integrated model construction method for foreign object detection of sealed electronic components from the perspective of multiple information carriers is used for detection. The prediction results of the data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network each correspond to a prediction weight. For the classification results determined by each classifier and neural network, the prediction weights corresponding to the same classification are summed, and finally the category with the highest sum of prediction weights is selected as the final predicted classification.

[0029] Beneficial effects:

[0030] This invention is the first to consider converting detection signals in the field of foreign object detection into three information carriers: data, images, and audio, for signal recognition. It extracts features from multiple perspectives, which can effectively improve the stability of detection results. Furthermore, based on features from multiple perspectives, this invention uses fuzzy processing to determine the corresponding membership functions and determines the weights based on the membership of each optimal classifier or optimal neural network. This takes into account the reliability of feature extraction from multiple perspectives, which not only greatly improves the stability of the integrated model, but also ensures the accuracy of detection on the basis of the model's good stability. That is, it uses an integrated model for foreign object detection of sealed electronic components to give accurate and stable detection results.

[0031] Furthermore, this invention analyzes the characteristics of four types of detection signals in the detection of foreign objects in sealed electronic components: foreign object signals, component signals, mixed signals, and excessively large component signals. Based on the characteristics of multiple signal types, an integrated model is constructed. Therefore, this invention can effectively identify foreign object signals, component signals, mixed signals, and excessively large component signals. This invention has a very high classification and recognition effect and very high stability. Attached Figure Description

[0032] Figure 1 The time-domain plots (partial) of the four types of detection signals are shown.

[0033] Figure 2 Time-frequency diagrams for four types of detection signals;

[0034] Figure 3 A flowchart illustrating the implementation of the integrated model building method;

[0035] Figure 4 This is a sample image of a sealed electronic component.

[0036] Figure 5 Diagram of the DZJC-III type automatic foreign matter detection system;

[0037] Figure 6 The network structure diagram of PP-MHA-VGG19;

[0038] Figure 7 The image shows the performance of PP-MHA-VGG19 on a time-domain image set.

[0039] Figure 8 The performance of PP-MHA-VGG19 in the spectrogram image set;

[0040] Figure 9 This is a structural diagram of a feedforward neural network;

[0041] Figure 10 Here is a structural diagram of an audio convolutional network;

[0042] Figure 11 This describes the performance of audio convolutional networks on audio sets.

[0043] Figure 12 The performance of AlexNet on temporal image sets;

[0044] Figure 13 The performance of ResNet18 on temporal image sets;

[0045] Figure 14 The performance of ResNet50 on temporal image sets;

[0046] Figure 15 The performance of VGG16 on temporal image sets;

[0047] Figure 16 The performance of VGG19 on temporal image sets;

[0048] Figure 17 The performance of MobileNet on temporal image sets;

[0049] Figure 18 The performance of GoogLeNet on temporal image sets;

[0050] Figure 19 The performance of AlexNet on spectrogram image sets;

[0051] Figure 20 The performance of ResNet18 on the spectrogram image set;

[0052] Figure 21 The performance of ResNet50 on the spectrogram image set;

[0053] Figure 22 The performance of VGG16 on the spectrogram image set;

[0054] Figure 23 The performance of VGG19 in the spectrogram image set;

[0055] Figure 24 The performance of MobileNet on the spectrogram image set;

[0056] Figure 25 The performance of GoogLeNet on spectrogram image sets;

[0057] Figure 26 This represents the performance of the feedforward neural network on the audio set. Detailed Implementation

[0058] Sealed electronic components are core parts of aerospace equipment. Foreign matter generated during their production can lead to failure and threaten the reliable operation of aerospace equipment. Particle collision noise detection methods attempt to activate foreign matter by generating sound through vibration to determine its presence; however, the presence of component signals reduces the accuracy and reliability of foreign matter detection. Therefore, the identification of detection signals becomes crucial in foreign matter detection. Existing signal recognition research based on signal detection and machine learning has shortcomings: it only considers foreign matter and component signals, only considers converting signals into data, and only considers training models with single structures, failing to meet the application requirements of real-world scenarios. Therefore, this invention proposes an integrated model construction method for foreign matter detection in sealed electronic components from a multi-information carrier perspective, effectively solving many problems in existing research.

[0059] First, this study carefully considers four types of detection signals present in real-world scenarios for detecting foreign objects in sealed electronic components: foreign object signals, component signals, mixed signals, and signals from excessively large components. It then attempts to convert these signals into three information carriers: data, images, and audio, constructing datasets, time-domain image sets, spectrogram image sets, and audio sets. This effectively addresses the shortcomings of existing research that only considers foreign object and component signals.

[0060] Secondly, multiple high-performing classifiers were trained on the dataset, and the Random Forest classifier was found to be the best performer. Similarly, multiple high-performing neural networks were trained on both temporal and spectrogram image datasets, and VGG19 was found to be the best performer across the board. Simultaneously, a newly designed feedforward neural network suitable for audio datasets was trained. Based on this, the parameters of the aforementioned classifiers and neural networks were optimized to obtain the optimal classifiers or neural networks for three different information carrier perspectives, termed data-classifier, temporal-graph-neural network, spectrogram-neural network, and audio-neural network, achieving the highest classification accuracies of 87.84%, 99.55%, 99.76%, and 72.81%, respectively. This effectively addresses the shortcomings of existing research that only considers converting signals into data.

[0061] Then, based on fuzzy rule-based decision-level fusion technology, applicable fuzzy rules and weight calculation formulas are proposed to construct an ensemble model, achieving a classification accuracy of 94.94%. This effectively addresses the shortcomings of existing research that only considers training models with a single structure.

[0062] Finally, extensive experiments demonstrated the feasibility and practicality of the ensemble model, as well as its superiority over individual classifiers and neural networks. Real-world applications showed that the ensemble model achieved the highest classification accuracy of 92.31%. This effectively demonstrates that the ensemble model possesses excellent generalization and stability, highlighting its advancements compared to existing research and meeting the application requirements for unwanted object detection.

[0063] The present invention will be described in detail below with reference to specific embodiments.

[0064] Specific Implementation Method 1: This implementation method is an integrated model construction method for detecting foreign matter in sealed electronic components from the perspective of multiple information carriers.

[0065] Before proceeding with the construction, the detection signal is first analyzed:

[0066] Figure 1 Time-domain plots of four types of detection signals in the latest real-world scenario of foreign object detection in sealed electronic components are presented, where (a) is the foreign object signal, (b) is the component signal, (c) is the mixed signal, and (d) is the oversized component signal. The pulse sequence in the foreign object signal exhibits obvious randomness; the pulse time interval and amplitude have no clear pattern, indicating a non-periodic sequence. The pulse sequence in the component signal exhibits obvious regularity; the pulse time interval and amplitude are approximately equal, indicating an approximately periodic sequence. The pulse sequence in the mixed signal presents a chaotic and dense state; on the one hand, the pulse sequence exhibits both randomness and periodicity; on the other hand, the number of pulses in the pulse sequence is relatively large. The pulse sequence in the oversized component signal presents a complex state; its time interval exhibits periodicity, but its amplitude exhibits randomness. Furthermore, comparing the component signal and the oversized component signal reveals that the oversized component signal has a more significant noise floor, comparable to that of the foreign object signal and the mixed signal, and the overall time-domain performance of the oversized component signal is similar to that of the mixed signal.

[0067] Fast Fourier Transform was performed on the four types of detection signals respectively to obtain the corresponding time-frequency graphs, such as... Figure 2As shown, (a) is the extraneous object signal, (b) is the component signal, (c) is the mixed signal, and (d) is the oversized component signal. The extraneous object signal contains a large number of irregularly shaped and unevenly distributed patches, reflecting its aperiodicity. The component signal contains a large number of regularly distributed horizontal or vertical line segments, reflecting its approximate periodicity. The mixed signal contains a large number of patches and line segments, with the line segments still regularly distributed horizontally or vertically, and the patches distributed along the extension direction of the line segments, reflecting that it simultaneously contains the aperiodicity of the extraneous object signal and the approximate periodicity of the component signal. The oversized component signal also contains a large number of regularly distributed horizontal or vertical line segments, but the line segments are darker in color and more concentrated in distribution.

[0068] This study focuses on four types of detection signals in the latest real-world detection of foreign objects in sealed electronic components. In addition to the foreign object and component signals addressed in existing research, mixed signals and signals from excessively large components are also included. Based on this, the four types of detection signals are transformed into three information carriers: data, images, and audio. Corresponding machine learning or deep learning techniques are then used to obtain classifiers and neural networks suitable for each information carrier. Finally, using a decision-level fusion technique based on fuzzy rules, applicable fuzzy rules and weight calculation formulas are designed to construct the final ensemble model. Figure 3 The implementation process of the integrated model construction method proposed in this invention is given.

[0069] From a data perspective, multiple widely validated and superior features are selected from the time and frequency domains. Feature engineering is then performed on the detected signals, transforming the four types of detection signals into feature vectors with four different distributions. Labels are then added to each vector, constructing a dataset containing the four labels. Multiple representative classifiers are trained on this dataset, and their parameters are optimized. The classifier with the best classification performance is then compared and named the data-classifier.

[0070] From an image perspective, on the one hand, time-domain waveforms of the four types of detection signals are directly plotted and labeled to construct a time-domain image set containing the four labels. On the other hand, spectrogramming technology is used to convert the four types of detection signals into four types of spectrograms, and labels are added to construct a spectrogram image set containing the four labels. Multiple representative neural networks are trained on both the time-domain image set and the spectrogram image set, and their parameters are optimized separately. The neural network with the best classification performance is then compared and named the time-domain graph-neural network and the spectrogram-neural network.

[0071] From an audio perspective, this study utilizes speech processing technology to convert four types of detection signals into four types of audio, and then adds labels to each, constructing an audio set containing the four labels. A novel neural network suitable for audio classification is proposed and trained on the audio set. Based on the classification performance of the neural network, targeted parameter optimization is performed to obtain the optimal neural network, which is called the audio-neural network.

[0072] Finally, by combining the perspectives of these three information carriers, fuzzy rules are used to integrate data-classifiers, temporal graph-neural networks, spectrogram-neural networks, and audio-neural networks in parallel to construct an ensemble model. In essence, a mathematical mapping is used to fuse the classification performance of the four classifiers or neural networks, ultimately outputting a reliable classification result.

[0073] 1. Signal acquisition and data construction:

[0074] In the application, for a sealed electronic component under test, a signal is obtained experimentally and input into the ensemble model. The ensemble model processes this signal into three information carriers, resulting in a data segment, two images, and an audio segment, referred to as the test data, the test time-domain plot, the test spectrogram, and the test audio, respectively. The ensemble model uses four independent processing threads to predict the labels of the data, images, and audio in parallel, and performs a weighted vote on the four predicted labels to output a single predicted label, which is the final data classification result and also the final signal recognition result.

[0075] Referring to the models of sealed electronic components used in real-world aerospace equipment, and the properties of foreign objects detected from these components, such as quantity, weight, shape, and material, twenty identical sealed electronic components were custom-made from the original manufacturer as samples. Figure 4 As shown, the samples were numbered. Samples 1 to 5, consisting of sealed electronic components, each contained only one foreign object. These five foreign objects were: 1.1 μg of shaving iron, 1.2 μg of filamentous copper, 1.8 μg of spherical aluminum, 2.1 μg of spherical tin, and 1.6 μg of wire segments. Samples 6 to 10 each contained only one small loose component, specifically a bonding wire. Samples 11 to 15 contained both one foreign object and one small loose component. The properties of these five foreign objects were consistent with those of the foreign objects in samples 1 to 5; the small loose component was a bonding wire. Samples 16 to 20 each contained only one large loose component, specifically a movable nut.

[0076] Figure 5This paper presents the DZJC-III type automatic foreign object detection system, based on particle collision noise detection, independently developed by our research group. It consists of three parts: a vibration table, a data acquisition device, and an embedded computer. A sealed electronic component sample is fixed on the vibration table. The embedded computer controls the vibration table to apply mechanical excitation, causing foreign objects or components inside the sealed electronic component sample to move or vibrate, generating a detection signal. An acoustic sensor located on the vibration table surface captures the detection signal and inputs it to the data acquisition device via a highly shielded transmission line. The data acquisition device processes the detection signal into an editable ".dat" format signal file and sends it to the embedded computer via a serial port. The embedded computer receives the signal file, stores it in its local memory, and thus acquires the detection signal. Simultaneously, the embedded computer reads the signal file and visualizes it.

[0077] Through the above steps, each sealed electronic component sample was placed on a vibration table, and a large number of detection signals were collected. Statistical analysis revealed 1671 signals of unwanted material, 1714 signals of components, 1728 mixed signals, and 1834 signals of oversized components, totaling 6947 detection signals.

[0078] (1) Dataset Construction: From a data perspective, sixteen features were selected from the time and frequency domains, including seven time-domain features and nine frequency-domain features, as shown in Table 1. It is worth noting that the feasibility and superiority of these features have been fully verified in previous studies.

[0079] Table 1 Sixteen Time-Domain and Frequency-Domain Characteristics

[0080]

[0081]

[0082] Feature engineering was performed on each of the 6947 detection signal segments. Specifically, the values ​​of sixteen features were calculated for each of the 6947 segments, and the sixteen values ​​for each segment were arranged in a row according to the feature numbers in Table 1 to construct a feature vector. Based on this, the following labels were added: "0" for 1671 feature vectors from signals originating from redundant objects; "1" for 1714 feature vectors from signals originating from components; "2" for 1728 feature vectors from signals originating from mixed signals; and "3" for 1834 feature vectors from signals originating from excessively large components. This resulted in a dataset containing four types of labels.

[0083] (2) Image Set Construction: From an image perspective, as mentioned earlier, the 6947 detection signals are actually 6947 signal files. On one hand, the signal files are directly read, and a time-domain plot of the detection signals is drawn, such as... Figure 1As shown. It's worth noting that in the time-domain plots, the duration of the detected signal is 5 seconds, the upper limit of amplitude is 5V, and the lower limit of amplitude is -5V. This results in 6947 time-domain plots. Based on this, label "0" is added to 1671 time-domain plots of signals originating from redundant objects, label "1" is added to 1714 time-domain plots of signals originating from components, label "2" is added to 1728 time-domain plots of signals originating from mixed signals, and label "3" is added to 1834 time-domain plots of signals originating from excessively large components. This constructs a time-domain image set containing four types of labels.

[0084] On the other hand, spectrogram technology is used to convert the detection signal into a spectrogram, and the conversion process is as follows.

[0085] First, a high-pass filter is used to pre-emphasize the detection signal, the principle of which is shown in formula (1):

[0086] S[n]=x[n]-μ×x[n-1] (1)

[0087] In the formula, x[n] represents the detection signal, and μ represents the pre-emphasis coefficient, which is set to μ = 0.97.

[0088] Secondly, the pre-emphasized detection signal is divided into frames, with each frame lasting 50ms. Considering... Figure 5 The sampling frequency of the acquisition device is 500kHz, and each frame signal contains twenty-five sampling points.

[0089] Then, the detected signal after framing is windowed to make it continuous, and the principle is shown in formula (2):

[0090] S w [n]=S[n]×w[n] (2) where S w [n] represents the windowed detection signal, and w[n] is the Hamming window, which is calculated as follows:

[0091]

[0092] In the formula, N represents the number of sampling points in the detected signal.

[0093] Finally, a Fast Fourier Transform (FFT) was performed on each windowed frame signal with a length of 1024, and the power spectrum of each frame signal was processed using a bandpass filter. Based on this, each frame signal was arranged in chronological order and visualized to obtain the spectrogram corresponding to the detected signal, as shown below. Figure 3 As shown in the image.

[0094] This resulted in 6947 spectrograms. Based on this, the following labels were added: "0" for 1671 spectrograms from sources of redundant material signals, "1" for 1714 spectrograms from sources of component signals, "2" for 1728 spectrograms from sources of mixed signals, and "3" for 1834 spectrograms from sources of excessively large components. This constructed a spectrogram image set containing all four labels.

[0095] (3) Audio Set Construction: From an audio perspective, the detection signals acquired by the DZJC-III type foreign object automatic detection system are essentially single-channel signals or single-channel audio, which cannot be processed by current mainstream neural networks. Therefore, firstly, the detection signals are converted into overall audio, specifically a "wav" format signal file. Secondly, the overall audio is amplified to increase the details in the detection signals. Then, the overall audio is filtered to remove irrelevant frequency components. Finally, the overall audio is normalized to meet the input requirements of the neural network. Thus, 6947 detection signals are converted into 6947 qualified audio segments. Based on this, the following labels are added: 1671 audio segments from foreign object signals are labeled "0", 1714 audio segments from component signals are labeled "1", 1728 audio segments from mixed signals are labeled "2", and 1834 audio segments from excessively large component signals are labeled "3". Thus, an audio set containing four labels is constructed.

[0096] 2. Model Training and Optimization:

[0097] (a) Data Classifiers: In machine learning, classifiers can be mainly categorized into two types: tree-based classifiers and mathematical model-based classifiers. The former includes decision trees, random forests, and XGBoost, while the latter includes k-nearest neighbors (kNN) and support vector machines (SVM). Furthermore, deep learning, as an important branch of machine learning, also utilizes some classic neural networks for data classification, such as one-dimensional convolutional neural networks (1D-CNN). In this invention, the dataset is divided into a training set and a validation set in a 7:3 ratio. The six classifiers mentioned above, with default parameter configurations, are trained on the training set, and their classification performance is tested on the validation set. The classification accuracy achieved is then statistically analyzed. Table 2 shows the parameter configurations of the six classifiers, and Table 3 shows the classification accuracy achieved by the six classifiers.

[0098] Table 2 Parameter configurations for the six classifiers

[0099]

[0100] Table 3 shows the classification accuracy achieved by the six classifiers.

[0101]

[0102] As shown in Table 3, Random Forest achieves the highest classification accuracy and demonstrates a significant advantage over other classifiers. This is because Random Forest is more adaptable to data; it uses random sampling to construct data subsets during the construction of its base classifiers, increasing the base classifiers' ability to explore different feature combinations. This allows the ensemble Random Forest to discover more effective classification rules, effectively improving its classification performance. Decision Tree achieves relatively significant classification accuracy, indicating that it also has superior classification performance as a base classifier for Random Forest. 1D-CNN achieves the lowest classification accuracy, demonstrating that classic neural networks used for data classification struggle when faced with high-dimensional feature vectors.

[0103] Furthermore, a grid search method was used to optimize the parameters of the best-performing random forest to further improve its classification performance. By setting different value ranges and step sizes for the four parameters, multiple iterations were performed to obtain the optimal combination of values ​​for the four parameters, as shown in Table 4, thus obtaining the parameter-optimized random forest.

[0104] Table 4 Optimal Parameter Combinations for Random Forest

[0105]

[0106] Based on this, the classification accuracy achieved by the parameter-optimized random forest was recalculated to be 87.84%. Compared to the 86.10% classification accuracy achieved by the original random forest, this represents an improvement of 1.74%, a significant increase. This fully demonstrates the feasibility and superiority of the parameter-optimized random forest. Therefore, the parameter-optimized random forest is considered a data classifier, and the highest classification accuracy achieved by a data classifier from a data perspective is deemed to be 87.84%.

[0107] (b) Image-Neural Networks: In this invention, seven currently popular and commonly used neural networks for image classification were selected: AlexNet, ResNet18, ResNet50, GoogLeNet, VGG16, VGG19, and MobileNet. These cover convolutional neural networks, residual neural networks, and lightweight neural networks of different network sizes and complexities, and are widely representative. The temporal image set and spectrogram image set were divided into training and validation image sets in a 7:3 ratio. The seven neural networks with the above default structures and configurations were trained on the training image set, and their classification performance on the two training image sets was initially evaluated. Classification accuracy and loss curves were plotted over 150 epochs. Subsequently, their classification performance was further tested on the two validation image sets. In addition to plotting their classification accuracy and loss curves over 150 epochs, their average classification accuracy over 150 epochs was also calculated. Table 5 shows the structure and configuration of the seven neural networks, and Table 6 shows the average classification accuracy of the seven neural networks on the two validation image sets. Figures 12 to 25 The classification accuracy curves and loss curves of seven neural networks on two training image sets and two validation image sets are presented. Figure 12 For AlexNet's performance on temporal image sets, Figure 13 To illustrate the performance of ResNet18 on temporal image datasets, Figure 14 To illustrate the performance of ResNet50 on temporal image datasets, Figure 15 To demonstrate the performance of VGG16 on temporal image sets, Figure 16 This describes the performance of VGG19 on temporal image sets. Figure 17 To demonstrate MobileNet's performance on temporal image datasets, Figure 18 To illustrate the performance of GoogLeNet on temporal image datasets, Figure 19 This shows the performance of AlexNet on spectrogram image sets. Figure 20 The performance of ResNet18 on spectrogram image sets, Figure 21 The performance of ResNet50 on the spectrogram image set, Figure 22 The performance of VGG16 in the spectrogram image set, Figure 23 The performance of VGG19 in the spectrogram image set. Figure 24 This shows the performance of MobileNet on spectrogram image sets. Figure 25 This shows the performance of GoogLeNet on spectrogram image sets. Figures 12 to 25 In the figure, (a) is the classification accuracy curve and (b) is the loss curve.

[0108] Table 5. Structure and configuration of the seven neural networks

[0109]

[0110] Table 6 shows the highest classification accuracy achieved by seven neural networks.

[0111]

[0112] As shown in Table 6, VGG19 achieved the highest average classification accuracy on both validation image sets, at 97.60% and 99.29% respectively, demonstrating a significant advantage over the average classification accuracy achieved by other neural networks. Furthermore, the average classification accuracy of the seven neural networks over 150 epochs on the two training image sets was also statistically analyzed, with VGG19 again showing the most outstanding performance. The classification accuracy and loss curves show similar trends across both the two training and two validation image sets. Specifically, as the number of epochs increases, the classification accuracy achieved by AlexNet gradually increases, reflecting its learning and adaptive capabilities. However, in the later stages, AlexNet's loss curve exhibited slight oscillations, indicating its instability in approaching the optimal solution. This is because AlexNet's network structure is relatively shallow, limiting its ability to handle complex datasets. While ResNet18 and ResNet50 demonstrated good classification performance on the two training image sets, their performance on the two validation image sets was extremely unstable, clearly showing overfitting. This indicates that while the deep network structure of residual neural networks allows them to capture more nuanced features, it also makes them more prone to overfitting training samples, thus affecting their generalization performance. MobileNet, due to its lightweight design, maintains relatively stable classification accuracy and loss throughout training. This suggests that MobileNet may be limited by its network structure when extracting complex features from temporal and spectrogram maps. GoogLeNet exhibits significant fluctuations in classification performance during training and validation, demonstrating its instability. While VGG16 also performs well, it still lags behind VGG19. This further confirms that the additional feature extraction and learning capabilities provided by VGG19's unique network structure are key factors in its superior classification performance.

[0113] Furthermore, VGG19 is optimized to further improve its classification performance. Specifically, the first convolutional kernel of VGG19 is changed from 3×3 to 7×7 to further enhance its image understanding and processing capabilities. A new PReLU activation function is introduced to replace the ReLU activation function, enabling it to effectively handle negative signal components in the detected signal. The improved VGG19 is called PReLU-VGG19-Plus. In this invention, a multi-head attention mechanism is introduced into VGG19, which originates from the Transformer architecture. Specifically, the multi-head attention mechanism is placed between the last convolutional block and the fully connected layer, such as... Figure 6 As shown. The further improved VGG19 is called PP-MHA-VGG19.

[0114] Based on this, PP-MHA-VGG19 was trained on two separate training image sets to initially evaluate its classification performance on both sets, and its classification accuracy and loss curves over 150 epochs were plotted. Subsequently, its classification performance was further tested on two validation image sets, and its classification accuracy and loss curves over 150 epochs were plotted again. Figure 7 and Figure 8 The classification accuracy curves and loss curves of PP-MHA-VGG19 on the time-domain image set and the spectrogram image set are presented respectively. Figure 7 and Figure 8 In the figure, (a) is the classification accuracy curve.

[0115] (b) shows the loss curve. On the other hand, the average classification accuracy they achieved over 150 epochs is shown in Table 7.

[0116] Table 7 shows the average classification accuracy achieved by VGG19 and PP-MHA-VGG19.

[0117]

[0118] As shown in Table 7, PP-MHA-VGG19 achieved average classification accuracies of 99.55% and 99.76% on the two image sets, respectively, representing improvements of 1.95% and 0.47% compared to VGG19, a significant increase. This is especially valuable given limited room for further improvement. Figure 7 and Figure 8As can be seen, PP-MHA-VGG19 achieves significant improvements in both classification accuracy and loss by modifying the first-layer convolutional kernel, introducing the PReLU activation function, and employing a multi-head attention mechanism. Specifically, compared to VGG19, PP-MHA-VGG19 achieves higher individual classification accuracy and average classification accuracy on both image sets. This indicates that a larger first-layer convolutional kernel enhances the model's ability to capture broader information from the input image, the PReLU activation function enhances the model's learning and generalization abilities, and the multi-head attention mechanism improves the model's understanding of global features. The combined effect of these three factors significantly improves the model's classification performance. Furthermore, the loss curve of PP-MHA-VGG19 is smoother, reflecting its superior generalization performance compared to VGG19. Therefore, PP-MHA-VGG19 trained on the temporal image set and the spectrogram image set are identified as temporal graph-neural network and spectrogram-neural network, respectively. The highest classification accuracy achieved by temporal graph-neural network and spectrogram-neural network from the image perspective is identified as 99.55% and 99.76%, respectively.

[0119] (c) Audio-Neural Network: First, a deep learning-based feedforward neural network is proposed for audio datasets. The feedforward neural network first uses a Flatten layer to extract features that contribute to the complexity and diversity of the input audio, including Mel-frequency cepstral coefficients, chroma features, and Mel-frequency spectra. Simultaneously, the Flatten layer ensures that the feedforward neural network is applicable to audio of varying durations. Next, after the Flatten layer, three fully connected layers are used. The ReLU activation function in these three fully connected layers enhances the non-linear processing capability, enabling it to learn more complex feature combinations. Finally, a Softmax activation function is used as the output layer to output the probability distribution of each category, thereby achieving accurate classification of the input audio. Figure 9 The structure diagram of the feedforward neural network is given.

[0120] Similarly, the audio set was divided into a training audio set and a validation audio set in a 7:3 ratio. A feedforward neural network was trained on the training audio set to initially evaluate its classification performance, and its classification accuracy and loss curves over 150 epochs were plotted. Subsequently, its classification performance was further tested on the validation audio set. In addition to plotting its classification accuracy and loss curves over 150 epochs, its average classification accuracy over 150 epochs was also calculated, which was 71.09%. Figure 26The table shows the performance of the feedforward neural network on the audio set, with (a) representing the classification accuracy curve and (b) representing the loss curve. The classification accuracy and loss curves of the feedforward neural network on the training and validation audio sets show that after 150 epochs, the classification accuracy achieved by the feedforward neural network on both the training and validation audio sets did not show a significant improvement, and the classification accuracy on the validation audio set exhibited large fluctuations. This fluctuation may be due to overfitting, i.e., the feedforward neural network is overly sensitive to details in the training samples, resulting in limited generalization performance on new samples. Meanwhile, although the loss curves of the feedforward neural network on the training and validation audio sets gradually converge and the two curves gradually approach each other, they are still higher than ideal. This indicates that the feedforward neural network may not have sufficient complexity to fully fit the complex patterns in the audio, causing its loss to stop decreasing in the later stages. In fact, audio is essentially time-series data with significant time dependencies. The feedforward neural network cannot capture these characteristics. Furthermore, the lack of a dropout layer in the feedforward neural network increases the risk of overfitting.

[0121] Furthermore, improvements were proposed. Specifically, the feature extraction part was retained, and a 1D-CNN was used for the classification network. Compared to feedforward neural networks, 1D-CNN, by introducing convolutional and pooling layers, is able to capture temporal dependencies and local features in audio more effectively. In addition, to reduce the risk of overfitting, a Dropout layer was added after each convolutional and pooling layer in the 1D-CNN. The improved model is called an audio convolutional network, and its structure is as follows... Figure 10 As shown.

[0122] Based on this, an audio convolutional network was trained on the training audio set to initially evaluate its classification performance, and its classification accuracy and loss curves over 150 epochs were plotted. Subsequently, its classification performance was further tested on the validation audio set, and its classification accuracy and loss curves over 150 epochs were plotted again. Figure 11 The classification accuracy and loss curves of the audio convolutional network on the audio set are presented, where (a) is the classification accuracy curve and (b) is the loss curve. Furthermore, its average classification accuracy over 150 epochs is calculated to be 72.81%.

[0123] Compared to the average classification accuracy achieved by the feedforward neural network on the validation audio set, the audio convolutional network achieved a 1.72% improvement in average classification accuracy, a significant increase. Figure 11As can be seen, the classification accuracy curve of the audio convolutional network on the validation audio set still fluctuates, but compared with the feedforward neural network, the classification accuracy curve of the audio convolutional network on the training audio set steadily improves. This is because the audio convolutional network can effectively capture local features and temporal dependencies in audio through convolutional layers. This characteristic allows the network to focus on key parts of the audio, thereby improving its adaptability to audio. In addition, the amplitude of the loss curve of the audio convolutional network on the validation audio set has decreased, and the loss curves on both the training and validation audio sets have improved. This is because the Dropout layer introduced in the audio convolutional network randomly discards a portion of neurons during training, reducing its overfitting to the training audio set and improving its generalization performance on unknown samples. Therefore, the audio convolutional network is identified as an audio-neural network, and the highest classification accuracy achieved by the audio-neural network from an audio perspective is considered to be 72.81%.

[0124] (d) Ensemble Model Construction: In the field of multimodal data processing, fusion technology is widely recognized as a key to improving the efficiency and accuracy of classification tasks. This technology enhances the depth of data understanding and the accuracy of decision-making by integrating information from different data sources. Fusion technology specifically includes feature-level fusion and decision-level fusion. Feature-level fusion focuses on the direct integration and interaction of information between different modalities, and is particularly suitable for scenarios where there is a high degree of correlation and complementarity between modalities. Conversely, decision-level fusion emphasizes maintaining the independence between modalities and focuses on the integration of the final decision, making it more suitable for scenarios that require maintaining the independent processing capabilities of each modality. Considering that this study has already constructed optimal classifiers or optimal neural networks from the perspectives of three information carriers, and that these networks process independently without interference, a fuzzy rule-based decision-level fusion technology is used to construct the ensemble model.

[0125] Typically, fuzzy rules consist of four stages: initialization, fuzzification, fuzzy inference, and defuzzification. The initialization stage primarily defines the fuzzy sets and membership functions of the input variables. The fuzzification stage mainly calculates the membership degrees of each classifier or neural network. The fuzzy inference stage mainly designs fuzzy rules and calculates weights. The defuzzification stage mainly calculates the weighted average classification accuracy of the ensemble model. Table 8 lists the classification accuracies achieved by the optimal classifiers or optimal neural networks constructed from the perspectives of the three information carriers. Based on this, applicable fuzzy rules are designed to construct the ensemble model.

[0126] Table 8. Classification accuracy achieved by the optimal classifier or optimal neural network.

[0127]

[0128] The first stage is the initialization phase. In this phase, appropriate trapezoidal and triangular membership functions are designed to divide the classification accuracy achieved by each optimal classifier or optimal neural network into three fuzzy sets: Low, Medium, and High.

[0129] The calculation formulas for the three fuzzy sets are shown in formulas (4) to (6).

[0130]

[0131] In the formula, x represents 100 times the classification accuracy achieved by the optimal classifier or the optimal neural network.

[0132] The next stage is the fuzzification stage. In this stage, the membership degree of each optimal classifier or optimal neural network is calculated according to formulas (4) to (6), as shown in Table 9. Taking the data-classifier as an example, the detailed calculation process is given: μ Low =0, μ Medium = (90-87.84) / 10 = 0.216, μ High = (87.84-80) / 10 = 0.784.

[0133] Table 9 Membership Degrees of Optimal Classifiers or Optimal Neural Networks

[0134]

[0135] Then comes the fuzzy inference stage. The weight formula is defined as follows:

[0136] W(i)=μ High +0.5×μ Medium (7)

[0137] Therefore, the weights of the data-classifier, time-domain graph-neural network, spectrograph-neural network, and audio-neural network are calculated to be 0.892, 1, 1, and 0.1405, respectively, and the total weight is calculated to be 3.0325.

[0138] The ratios of the weights of the data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network to the total weights were used as the corresponding prediction weights, resulting in prediction weights of 0.892 / 3.0325 = 0.294, 1 / 3.0325 = 0.330, 1 / 3.0325 = 0.330, and 0.1405 / 3.0325 = 0.046, respectively.

[0139] Finally, in the defuzzing stage, for the classification results determined by each classifier and neural network, the prediction weights corresponding to the same classification are summed, and the category with the highest sum of prediction weights is selected as the final predicted classification.

[0140] To facilitate the evaluation of the final prediction results, the formula for calculating the weighted average classification accuracy is defined as follows:

[0141]

[0142] In the formula, Acc(i) represents the classification accuracy achieved by each classifier or neural network, W(i) represents the weight of each classifier or neural network, and W represents the total weight.

[0143] Therefore, the weighted average classification accuracy of the data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network is calculated to be (87.84×0.892+99.55×1+99.76×1+72.81×0.1405) / 3.0325=94.94. That is, the ensemble model is complete, and its achieved classification accuracy is 94.94%.

[0144] Compared to the classification accuracies of 87.74%, 72.81%, 99.55%, and 99.76% achieved by the optimal classifier or optimal neural network, the classification accuracy achieved by the ensemble model is at a moderate level. It is significantly higher than the classification accuracy achieved by the data-classifier and audio-neural network, but slightly lower than the classification accuracy achieved by the time-domain graph-neural network and spectrogram-neural network. This fully demonstrates the feasibility of the ensemble model. It shows that the ensemble model effectively balances the contributions of different classifiers or neural networks through fuzzy rules, avoiding the drag from low-performance classifiers or neural networks while retaining the advantages of high-performance classifiers or neural networks. Furthermore, by comprehensively considering multiple classifiers and neural networks, its generalization performance and stability can be guaranteed. The practicality and superiority of the ensemble model need to be verified in real-world scenarios.

[0145] In a real-world scenario, find thirty similar... Figure 4 The same type of sealed electronic component under test, using Figure 5 The DZJC-III type automatic foreign object detection system shown is used for foreign object detection. Therefore, for each sealed electronic component under test, a detection signal is acquired, referred to as the test signal, for a total of thirty test signals. From the perspectives of data, image, and audio, the thirty test signals are respectively converted into thirty test feature vectors, thirty test time-domain plots, thirty test spectrograms, and thirty test audio segments.

[0146] Based on this, we first use a data-classifier, a time-domain graph-neural network, a spectrogram-neural network, and an audio-neural network to predict the labels for thirty feature vectors, thirty time-domain graphs, thirty spectrograms, and thirty audio segments to be tested, respectively. Then, we use an ensemble model to predict the thirty feature vectors, thirty time-domain graphs, thirty spectrograms, and thirty audio segments in parallel. Using a decision-level fusion technique based on fuzzy rules and referencing the weights of the optimal classifier or optimal neural network, we provide thirty predicted labels. Table 10 shows the prediction results of the above classifiers, neural networks, and ensemble models. It is worth noting that "—" in the table indicates that the sealed electronic component under test is qualified, that is, there are no foreign objects or loose components in the sealed electronic component under test.

[0147] Table 10 Prediction Results of Sealed Electronic Components Under Test

[0148]

[0149] Calculations showed that, excluding the four qualified sealed electronic components under test, the prediction accuracies (classification accuracies) achieved by the data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network for the remaining twenty-six sealed electronic components under test were 80.77%, 80.77%, 88.46%, and 69.23%, respectively. This represents a significant decrease compared to their previous classification accuracies of 87.84%, 99.55%, 99.76%, and 72.81%. The time-domain graph-neural network and spectrogram-neural network, in particular, experienced substantial drops in classification accuracy, decreasing by 18.78% and 11.30%, respectively. This is an unavoidable problem in current machine learning applications. Classifiers or neural networks that perform well on familiar samples will show decreased generalization performance when faced with new samples.

[0150] This problem is even more severe for foreign object detection in sealed electronic components. In machine learning, familiar and new samples often refer to interpretable data, such as feature vectors, images, and time series. Thus, the difference between familiar and new samples is a direct difference in the distribution characteristics of interpretable data. However, in foreign object detection, familiar and new samples refer to the detection signal. Machine learning cannot directly process the detection signal; it needs to be processed into interpretable data first. Therefore, the difference between familiar and new samples is not a direct difference in signal characteristics, but rather an indirect difference in the distribution characteristics of processed interpretable data. It is well known that information loss is inevitable during signal processing into interpretable data. For foreign object detection, the already differing old and new detection signals suffer some loss of information about these differences after processing into new and old interpretable data. This leads to the amplification or distortion of the differences in the processed interpretable data. Consequently, classifiers and neural networks experience a decline in generalization performance when faced with data, images, and audio processed from detection signals. This explains the performance of the data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network in Table 10. Furthermore, since the process of converting the detected signal into data and audio is simpler than converting it into an image, resulting in less information loss, the generalization performance of the data-classifier and audio-neural network deteriorates relatively less. Even for the time-domain graph-neural network and spectrogram-neural network, the latter shows a relatively smaller decline in generalization performance and is significantly better than the data-classifier and audio-neural network. This is because spectrograms, compared to time-domain graphs, more comprehensively integrate the time-domain and frequency-domain characteristics of the detected signal.

[0151] In machine learning, collecting as many samples as possible to construct a complete dataset is crucial for ensuring the generalization performance of classifiers or neural networks. For foreign object detection, regardless of the type of detection signal, it's impossible to adequately describe them. Foreign object signals are generated by the random movement of foreign objects, and component signals are generated by uncertain loose components. Therefore, extensive sample collection is not feasible. This invention proposes a novel approach: training suitable classifiers and neural networks separately from a multi-information carrier perspective, and finally constructing an ensemble model to ensure comprehensive and stable prediction results. This effectively demonstrates the necessity of this research. Calculations show that the ensemble model achieves a classification accuracy of 92.31%, significantly higher than the classification accuracy achieved by four classifiers or neural networks. The decrease compared to the achieved 94.94% classification accuracy is small and understandable. Thus, the feasibility and practicality of the ensemble model, as well as its superiority over individual models, are highlighted. The table shows that the ensemble model only mispredicts two sealed electronic components under test, while the other four classifiers or neural networks all make prediction errors.

[0152] Furthermore, existing research on redundant object and component signal identification based on machine learning was used to predict thirty test signals, and the prediction results are also listed in Table 10. Specifically, the existing research referenced XGBoost with parameter optimization from a data perspective proposed by Li Chaoran et al. Calculations showed that the classification accuracy achieved by the existing research was 61.54%, far lower than the classification accuracy achieved by four classifiers or neural networks, let alone the classification accuracy achieved by ensemble models. The existing research predicted all data with true labels "2" and "3" as "0" and "1," reflecting the inadequacy of considering only redundant object and component signals. Of the thirty sealed electronic components under test shown in Table 10, ten (one-third) had mixed signals or excessively large component signals. This fully demonstrates the necessity and correctness of considering four types of detection signals. Furthermore, for data with a true label "3," the existing research did not predict all of them as "1," some as "0." This indicates that the difference between oversized component signals and ordinary component signals is not simply a difference in amplitude, but also includes differences in time and frequency domains such as time intervals and spectral distribution. This actually explains why oversized component signals are treated as a separate type of detection signal. If it is predicted as a foreign object signal, it will affect the foreign object detection results. This is because if it is a foreign object signal, the sealed electronic component under test is unqualified, while if it is an oversized component signal, the sealed electronic component under test is qualified.

[0153] Thus, the necessity of conducting this research has been demonstrated, and the feasibility, practicality, and superiority of the method proposed in this invention have been fully proven.

[0154] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An integrated model construction method for detecting foreign matter in sealed electronic components from the perspective of multiple information carriers, characterized in that, include: For the foreign object detection signals of sealed electronic components acquired in real-world scenarios, the type of the foreign object detection signal is determined as a label, and the foreign object detection signal is processed as follows: From a data perspective, features are selected from the time and frequency domains, different types of detection signals are transformed into feature vectors, and labels are added to each to construct datasets of different types of detection signals. Then, multiple classifiers are trained, and the classifier with the best classification performance is selected, which is called a data-classifier. Based on the image perspective, time-domain waveforms of different types of detection signals are directly plotted and labeled to construct time-domain image sets for different types of detection signals. Simultaneously, spectrogram technology is used to convert different types of detection signals into different types of spectrograms, and labels are added to construct spectrogram image sets for different types of detection signals. Multi-neural network models are trained on the time-domain image sets and spectrogram image sets respectively, and the neural networks with the best classification performance are compared to obtain the results, which are called time-domain graph-neural network and spectrogram-neural network. From an audio perspective, speech processing technology is used to convert different types of detection signals into audio of different types of detection signals, and labels are added to each type to construct audio sets of different types of detection signals. Then, multiple neural network models are trained, and the neural network with the best classification performance is compared and obtained. This is called an audio-neural network. An ensemble model is constructed using a decision-level fusion technique based on fuzzy rules: In the initialization phase, the classification accuracy of each optimal classifier or optimal neural network is divided into three fuzzy sets, and the corresponding membership functions are determined. Then, the membership degree of each optimal classifier or optimal neural network is calculated, and the weights of each optimal classifier or optimal neural network are obtained using fuzzy inference. The sum of all weights is the total weight. The ratio of the weights of the data-classifier, temporal graph-neural network, spectrogram-neural network, and audio-neural network to the total weight is used as the corresponding prediction weight. The prediction weights of the data-classifier, temporal graph-neural network, spectrogram-neural network, and audio-neural network are the prediction weights corresponding to their prediction results. Finally, the data-classifier, temporal graph-neural network, spectrogram-neural network, and audio-neural network are integrated to obtain an ensemble model. The types of foreign object detection signals include: normal signal, foreign object signal, component signal, mixed signal, and excessive component signal; The redundant signal refers to a pulse sequence that is a non-periodic sequence; the component signal refers to a pulse sequence that is approximately periodic; the mixed signal refers to a pulse sequence that exhibits both randomness and periodicity; the excessive component signal refers to a pulse sequence in which the time interval exhibits periodicity, while the amplitude exhibits randomness.

2. The integrated model construction method for detecting unwanted objects in sealed electronic components from a multi-information carrier perspective according to claim 1, characterized in that, The data classifier uses a random forest model.

3. The integrated model construction method for detecting unwanted objects in sealed electronic components from a multi-information carrier perspective according to claim 1, characterized in that, Both the time-domain graph-neural network and the spectrogram-neural network adopt the improved VGG19 network model. The improvement method of the VGG19 network model is as follows: The first convolutional kernel of VGG19 is changed from... Modified to The PReLU activation function is used instead of the ReLU activation function; a multi-head attention mechanism of the Transformer architecture is introduced in VGG19, which is set between the last convolutional block and the fully connected layer.

4. The integrated model construction method for detecting foreign matter in sealed electronic components from a multi-information carrier perspective according to claim 1, characterized in that, The audio-neural network is as follows: For the input audio, features are first extracted using a Flatten layer; after the Flatten layer, at least three fully connected layers are used, each with a ReLU activation function; finally, a Softmax activation function is used as the output layer to output the probability distribution of each category.

5. The integrated model construction method for detecting unwanted objects in sealed electronic components from a multi-information carrier perspective according to claim 1, characterized in that, The audio-neural network is as follows: For the input audio, features are first extracted using 1D-CNN. 1D-CNN contains multiple convolutional units, each of which includes a convolutional layer and a pooling layer, and a Dropout layer is added after each pooling layer. After 1D-CNN, two fully connected layers are set, and then the Softmax activation function is used as the output layer to output the probability distribution of each category.

6. The integrated model construction method for detecting foreign matter in sealed electronic components from the perspective of multiple information carriers, as described in any one of claims 1 to 5, is characterized in that... Based on the classification accuracy obtained by each optimal classifier or optimal neural network, the data is divided into three fuzzy sets, and the corresponding membership functions are determined as follows: In the formula, , , These are the membership functions corresponding to the three fuzzy sets representing low, medium, and high membership levels, respectively. This represents 100 times the classification accuracy achieved by the optimal classifier or optimal neural network.

7. The integrated model construction method for detecting unwanted objects in sealed electronic components from a multi-information carrier perspective according to claim 6, characterized in that, The weights of each optimal classifier or optimal neural network are obtained using fuzzy inference as follows: in, This represents the weights of each optimal classifier or optimal neural network. To adjust the weights.

8. The integrated model construction method for detecting foreign matter in sealed electronic components from a multi-information carrier perspective according to claim 7, characterized in that, The adjustment weight Pick .

9. A method for detecting foreign matter in sealed electronic components from the perspective of multiple information carriers, characterized in that, Includes the following steps: For the foreign object detection signal of sealed electronic components in the real scene, the integrated model constructed by the integrated model construction method for foreign object detection of sealed electronic components from the perspective of multiple information carriers as described in any one of claims 1 to 8 is used for detection. The prediction results of the data-classifier, time-domain graph-neural network, spectrogram-neural network and audio-neural network are respectively associated with a prediction weight. For the classification results determined by each classifier and neural network, the prediction weights corresponding to the same classification are summed, and finally the category with the highest sum of prediction weights is selected as the final prediction classification.

Images