Integrated model construction and detection method for detecting redundancy of sealed electronic component under multi-information carrier view angle

Abstract

The invention relates to an integrated model construction and detection method for sealing electronic component redundancy detection under a multi-information carrier view angle, and belongs to the technical field of sealing electronic component redundancy detection. The objective of the invention is to solve the problem of poor detection performance stability caused by only performing feature extraction and detection from a single view angle in an existing sealed electronic component redundancy detection method. The method comprises the following steps of: training and selecting a data-classifier with optimal classification performance from a data perspective aiming at a redundant object detection signal of a sealed electronic component; from an image perspective, respectively training on the time domain picture set and the spectrogram picture set, and selecting a time domain picture-neural network and a spectrogram-neural network with optimal classification performance; from an audio perspective, training and selecting an audio-neural network with optimal classification performance; the weight of each optimal classifier or optimal neural network is obtained based on a decision-making level fusion technology of a fuzzy rule, then the classifiers and the neural networks are integrated, and an integrated model is adopted for detection during actual detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of detecting foreign matters in sealed electronic components, and particularly relates to a method for constructing an integrated model and a detection method for detecting foreign matters in sealed electronic components. Background Art

[0002] Sealed electronic components are the core components of aerospace equipment. During the production process of sealed electronic components, metal debris, welding residues and other particles may be left inside. These particles that disrupt the stable state of substances, introduced from the outside or generated inside, are called foreign matters. Aerospace equipment usually operates in extreme environments such as overweight and weightlessness. The foreign matters existing inside the sealed electronic components will be activated to move randomly, which may cause the sealed electronic components to fail and pose an important threat to the reliable operation of aerospace equipment. Therefore, it is crucial to detect foreign matters before the sealed electronic components leave the factory and enter service. The particle impact noise detection method is currently the internationally recognized method for detecting foreign matters. It is based on the principle of vibration sound generation to activate the possible foreign matters inside the device to be detected, and judges whether there are foreign matters by capturing the possible sound signals generated. Usually, the sound signals generated by the random movement of foreign matters are called foreign matter signals.

[0003] Sealed electronic components have a high degree of integration and contain a large number of components inside, including some loose components. In this way, during the detection of foreign matters, the loose components will also be activated to vibrate periodically at fixed positions, generating sound signals, called component signals. Component signals will interfere with the detection of foreign matters. Accurately identifying foreign matter signals and component signals has become the key to foreign matter detection. Previously, some scholars used signal detection methods to carry out related research. For example, based on the signal generation mechanism, Zhang Hui, Gao Hongliang, Chen Jinbao, etc. respectively analyzed the non-periodic and approximately periodic characteristics, non-stationary and approximately stationary characteristics of foreign matter signals and component signals from the time domain and frequency domain, and tried to use different indicators or combinations of multiple indicators to quantify the characteristic differences between the two signals. However, the indicators or combinations of indicators used in these methods are not stable. Often, the indicators or combinations of indicators determined by a certain batch of detection signals are not applicable to a new batch of detection signals, even if these detection signals are collected under the same experimental conditions. Moreover, these methods achieved good results only in the identification of foreign matter signals and component signals, but seemed helpless when facing the mixed signals generated by the simultaneous movement of foreign matters and loose components, and often misjudged. This reduces the practicality of these methods. More importantly, these methods rely on manual analysis and judgment, and the degree of automation is not high, and their application in real scenarios is further limited.

[0004] In machine learning, the attribute differences between different objects can be quantified as the distribution differences between different data. By training a suitable classifier for data classification, the classification of different objects can be achieved. Similarly, the characteristic differences between the foreign object signals and the component signals can be quantified to train a classifier. In recent years, scholars have been committed to conducting research on the identification of foreign object signals and component signals based on machine learning. For example, Wang Guotao, Liu Haijiang, Meng Si, etc. selected multiple features that can stably and completely describe the characteristic differences between the two signals from the time domain, frequency domain, and time-frequency domain, constructed a dataset to train well-performing classifiers such as support vector machines, random forests, XGBoost, etc., and automatically completed data classification and signal identification. These methods effectively solve the problems of incomplete use of indicators and low automation degree existing in signal detection methods, but they also only face foreign object signals and component signals and fail to solve the identification problems of mixed signals or other complex signals. In addition, these methods only consider using feature engineering to convert signals into data, a single information carrier, without considering converting them into other popular information carriers such as images and audio, and using advanced deep learning to complete signal identification. Summary of the Invention

[0005] The present invention aims to solve the problem of poor stability in the detection performance of the existing foreign object detection method for sealed electronic components, which only extracts features from a single perspective and conducts detection.

[0006] An integrated model construction method for foreign object detection of sealed electronic components from the perspective of multiple information carriers, comprising:

[0007] For the foreign object detection signals of sealed electronic components obtained in the real scene, determine the type of the foreign object detection signals as labels, and at the same time perform the following processing on the foreign object detection signals:

[0008] From the data perspective, select features from the time domain and frequency domain, convert different types of detection signals into feature vectors, and add labels respectively to construct a dataset of different types of detection signals. Then train multiple classifiers and select the classifier with the best classification performance, which is called the data-classifier;

[0009] From the image perspective, directly draw the time-domain waveform diagrams of different types of detection signals, and add labels respectively to construct a time-domain picture set of different types of detection signals. At the same time, use the spectrogram technology to convert different types of detection signals into different types of spectrograms respectively, and add labels respectively to construct a spectrogram picture set of different types of detection signals; train multiple neural network models on the time-domain picture set and the spectrogram picture set respectively, and compare them respectively to obtain the neural network with the best classification performance, which is called the time-domain graph-neural network and the spectrogram-graph neural network;

[0010] From the audio perspective, using speech processing technology, different types of detection signals are respectively converted into the audio of different types of detection signals, and tags are added respectively to construct an audio set of different types of detection signals. Then, multiple neural network models are trained, and the neural network with the best classification performance is obtained through comparison, which is called the audio-neural network;

[0011] Construct an integrated model based on the decision-level fusion technology of fuzzy rules: In the initialization stage, divide the classification accuracy obtained by each optimal classifier or optimal neural network into three fuzzy sets and determine the corresponding membership functions; then calculate the membership degrees of each optimal classifier or optimal neural network respectively, and use the fuzzy inference process to obtain the weights of each optimal classifier or optimal neural network. The sum of all weights is the total weight; take the ratio of the weights of the data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network to the total weight as the corresponding prediction weights; the prediction weights of the data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network are the prediction weights corresponding to their prediction results. Furthermore, integrate the data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network to obtain an integrated model.

[0012] Furthermore, the types of the foreign object detection signals include: normal signals, foreign object signals, component signals, mixed signals, and oversized component signals; the foreign object signal refers to a signal whose pulse sequence belongs to a non-periodic sequence; the component signal refers to a signal whose pulse sequence is approximately a periodic sequence; the mixed signal refers to a signal whose pulse sequence reflects both randomness and periodicity; the oversized component signal refers to a signal whose time interval of the pulse sequence reflects periodicity, while the amplitude reflects randomness.

[0013] Furthermore, the data-classifier adopts a random forest model.

[0014] Furthermore, both the time-domain graph-neural network and the spectrogram-neural network adopt an improved VGG19 network model. The improvement method of the improved VGG19 network model is as follows:

[0015] Modify the first-layer convolution kernel of VGG19 from 3×3 to 7×7, and use the PReLU activation function to replace the ReLU activation function; introduce the multi-head attention mechanism of the Transformer architecture into VGG19, and the multi-head attention mechanism is set between the last convolutional block and the fully connected layer.

[0016] Furthermore, the audio-neural network is as follows:

[0017] For the input audio, first extract features using a Flatten layer; after the Flatten layer, use at least three fully connected layers, each of which is set with a ReLU activation function; finally, use a Softmax activation function as the output layer to output the probability distribution of each category.

[0018] Or,

[0019] For the input audio, first extract features using 1D-CNN. The 1D-CNN contains multiple convolutional units, each convolutional unit includes a convolutional layer and a pooling layer, and a Dropout layer is added after each pooling layer; after the 1D-CNN, set 2 fully connected layers, and then use a Softmax activation function as the output layer to output the probability distribution of each category.

[0020] Furthermore, divide the classification accuracy obtained by each optimal classifier or optimal neural network into three fuzzy sets and determine the corresponding membership functions as follows:

[0021]

[0022] In the formula, μ Low (x), μ Medium (x), μ High (x) are the membership functions corresponding to the three fuzzy sets corresponding to low, medium, and high respectively; x represents 100 times the classification accuracy obtained by the optimal classifier or optimal neural network.

[0023] Furthermore, use the fuzzy inference process to obtain the weights of each optimal classifier or optimal neural network as follows:

[0024] W(i) = μ High + α × μ Medium

[0025] Among them, W(i) represents the weight of each optimal classifier or optimal neural network, and α is the adjustment weight.

[0026] Preferably, the adjustment weight α takes 0.5.

[0027] A method for detecting foreign matters in sealed electronic components from the perspective of multiple information carriers includes the following steps:

[0028] For the obtained detection signals of foreign matters in sealed electronic components in a real scenario, the integrated model constructed by using the integrated model construction method for detecting foreign matters in sealed electronic components from the perspective of multiple information carriers is used for detection. The prediction results of the data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network respectively correspond to a prediction weight. For the classification results determined by each classifier and neural network, the prediction weights corresponding to the same classification are added up, and finally the category with the highest sum of prediction weights is selected as the final prediction classification.

[0029] Beneficial effects:

[0030] For the first time, the present invention considers converting the detection signals in the field of foreign matter detection into three information carriers, namely data, images, and audio, to carry out signal recognition. It extracts features from multiple perspectives, which can effectively improve the stability of the detection effect. Furthermore, based on the features from multiple perspectives, the present invention uses the means of fuzzy processing to determine the corresponding membership function, and determines the weight based on the membership degree of each optimal classifier or optimal neural network, taking into account the reliability of feature extraction from multiple perspectives. This not only greatly improves the stability of the integrated model, but also ensures the accuracy of detection on the basis of the good stability of the model, that is, the integrated model for detecting foreign matters in sealed electronic components is used to give accurate and stable detection results.

[0031] In addition, the present invention analyzes the characteristics of four types of detection signals, namely foreign matter signals, component signals, mixed signals, and oversized component signals, in the detection of foreign matters in sealed electronic components, and constructs an integrated model based on the characteristics of multiple types of signals. Therefore, the present invention can effectively identify foreign matter signals, component signals, mixed signals, and oversized component signals. The present invention has a very high classification and recognition effect and very high stability. Description of the drawings

[0032] Figure 1 Is the time-domain graph (partial) of four types of detection signals;

[0033] Figure 2 Is the time-frequency graph of four types of detection signals;

[0034] Figure 3 Is the implementation flowchart of the integrated model construction method;

[0035] Figure 4 Is the physical diagram of the sealed electronic component sample;

[0036] Figure 5 Is the diagram of the DZJC-III type foreign matter automatic detection system;

[0037] Figure 6 Is the network structure diagram of PP-MHA-VGG19;

[0038] Figure 7 Performance diagram of PP-MHA-VGG19 on the time-domain picture set;

[0039] Figure 8 Performance of PP-MHA-VGG19 on the spectrogram picture set;

[0040] Figure 9 Structure diagram of the feedforward neural network;

[0041] Figure 10 Structure diagram of the audio convolutional network;

[0042] Figure 11 Performance of the audio convolutional network on the audio set;

[0043] Figure 12 Performance of AlexNet on the time-domain picture set;

[0044] Figure 13 Performance of ResNet18 on the time-domain picture set;

[0045] Figure 14 Performance of ResNet50 on the time-domain picture set;

[0046] Figure 15 Performance of VGG16 on the time-domain picture set;

[0047] Figure 16 Performance of VGG19 on the time-domain picture set;

[0048] Figure 17 Performance of MobileNet on the time-domain picture set;

[0049] Figure 18 Performance of GoogLeNet on the time-domain picture set;

[0050] Figure 19 Performance of AlexNet on the spectrogram picture set;

[0051] Figure 20 Performance of ResNet18 on the spectrogram picture set;

[0052] Figure 21 Performance of ResNet50 on the spectrogram picture set;

[0053] Figure 22 Performance of VGG16 on the spectrogram picture set;

[0054] Figure 23 Performance of VGG19 on the spectrogram picture set;

[0055] Figure 24 Performance of MobileNet on the spectrogram image set;

[0056] Figure 25 Performance of GoogLeNet on the spectrogram image set;

[0057] Figure 26 Performance of the feedforward neural network on the audio set. Detailed implementation manners

[0058] Sealed electronic components are the core components of aerospace equipment. The foreign matters generated during their production can cause their failure and threaten the reliable operation of aerospace equipment. The particle impact noise detection method attempts to activate foreign matters by vibrating to generate sound to judge the existence of foreign matters, but the presence of component signals reduces the accuracy and reliability of foreign matter detection. The identification of detection signals in foreign matter detection becomes the key. Existing research on signal identification based on signal detection and machine learning has the deficiencies of only considering foreign matter signals and component signals, only considering converting signals into data, and only considering training models with a single structure, which does not meet the application requirements of real scenarios. Based on this, the present invention proposes an integrated model construction method for foreign matter detection of sealed electronic components from the perspective of multiple information carriers, effectively solving many problems existing in existing research.

[0059] First, seriously consider four types of detection signals existing in the foreign matter detection of sealed electronic components in real scenarios, namely foreign matter signals, component signals, mixed signals, and over-large component signals, and attempt to convert the detection signals into three information carriers: data, images, and audio, and construct a data set, a time-domain image set, a spectrogram image set, and an audio set. This effectively solves the deficiency of existing research that only considers foreign matter signals and component signals.

[0060] Secondly, train multiple classifiers with good performance on the data set, and compare to obtain the outstanding random forest. Respectively train multiple neural networks with good performance on the time-domain image set and the spectrogram image set, and compare to obtain the simultaneously outstanding VGG19. At the same time, newly design a feedforward neural network suitable for the audio set and train it. On this basis, optimize the parameters of the above classifiers and neural networks respectively to obtain the optimal classifier or the optimal neural network from the perspective of three information carriers, namely data-classifier, time-domain image-neural network, spectrogram image-neural network, and audio-neural network. The highest classification accuracies obtained are 87.84%, 99.55%, 99.76%, and 72.81% respectively. This effectively solves the deficiency of existing research that only considers converting signals into data.

[0061] Then, based on the decision-level fusion technology of fuzzy rules, applicable fuzzy rules and weight calculation formulas are proposed, an integrated model is constructed, and a classification accuracy of 94.94% is achieved. This effectively solves the deficiency of existing research that only considers training models with a single structure.

[0062] Finally, a large number of experiments prove the feasibility and practicality of the integrated model, as well as its superiority over individual classifiers and neural networks. Applications in real scenarios show that the integrated model achieves the highest classification accuracy of 92.31%. This effectively demonstrates that the integrated model has good generalization performance and stable performance, and its advancement compared to existing research is highlighted, meeting the application requirements for foreign object detection.

[0063] The following specifically describes the present invention in combination with specific embodiments.

[0064] Specific Embodiment 1: This embodiment is a method for constructing an integrated model for detecting foreign objects in sealed electronic components from the perspective of multiple information carriers.

[0065] Before construction, first analyze the detection signals:

[0066] Figure 1 The time-domain diagrams of four types of detection signals in the detection of foreign objects in sealed electronic components in the latest real scenario are given. Among them, (a) is the foreign object signal, (b) is the component signal, (c) is the mixed signal, and (d) is the over-sized component signal. The pulse sequence in the foreign object signal shows obvious randomness, and there is no obvious rule to follow for the time interval and amplitude of the pulses, that is, the pulse sequence belongs to a non-periodic sequence. The pulse sequence in the component signal shows obvious regularity, and the time interval and amplitude of the pulses are approximately equal, that is, the pulse sequence belongs to an approximately periodic sequence. The pulse sequence in the mixed signal shows a chaotic and dense state. On the one hand, the pulse sequence reflects both randomness and periodicity, and on the other hand, there are a large number of pulses in the pulse sequence. The pulse sequence in the over-sized component signal shows a complex state. Its time interval reflects periodicity, but its amplitude reflects randomness. At the same time, by comparing the component signal and the over-sized component signal, it can be found that the background noise of the over-sized component signal is more obvious, comparable to the background noise of the foreign object signal and the mixed signal, and the over-sized component signal is overall close to the mixed signal in the time domain.

[0067] Perform fast Fourier transforms on the four types of detection signals respectively to obtain the corresponding time-frequency diagrams, as Figure 2As shown, where (a) is the foreign object signal, (b) is the component signal, (c) is the mixed signal, and (d) is the over-sized component signal. The foreign object signal contains a large number of patches with irregular areas and uneven distributions, reflecting its non-periodicity. The component signal contains a large number of regularly horizontally or vertically distributed line segments, reflecting its approximate periodicity. The mixed signal contains both a large number of patches and line segments, and the line segments are still regularly horizontally or vertically distributed, while the patches are distributed along the extension direction of the line segments, reflecting that it contains both the non-periodicity of the foreign object signal and the approximate periodicity of the component signal. The over-sized component signal also contains a large number of regularly horizontally or vertically distributed line segments, but the line segments are darker in color and more concentrated in distribution.

[0068] Taking four types of detection signals existing in the detection of foreign objects in sealed electronic components in the latest real scenarios as the research object, in addition to the foreign object signal and the component signal that existing research focuses on, the mixed signal and the over-sized component signal are supplemented. On this basis, considering converting the four types of detection signals into three information carriers, namely data, images, and audio, respectively, and using corresponding machine learning or deep learning means, classifiers and neural networks applicable to different information carriers are obtained. Finally, with the help of the decision-level fusion technology based on fuzzy rules, applicable fuzzy rules and weight calculation formulas are designed to construct the final integrated model. Figure 3 The implementation process of the integrated model construction method proposed in the present invention is given.

[0069] From the data perspective, multiple features that have been widely verified to be excellent are selected from the time domain and the frequency domain. In this way, feature engineering is carried out on the detection signals, converting the four types of detection signals into four distributed feature vectors, and adding labels respectively to construct a data set containing four labels. Multiple representative classifiers are trained on the data set and their parameters are optimized respectively, and the classifier with the best classification performance is obtained through comparison, which is called the data-classifier.

[0070] From the image perspective, on the one hand, the time-domain waveform diagrams of the four types of detection signals are directly drawn and labeled respectively to construct a time-domain picture set containing four labels. On the other hand, using the spectrogram technology, the four types of detection signals are respectively converted into four types of spectrograms and labeled respectively to construct a spectrogram picture set containing four labels. Multiple representative neural networks are trained on the time-domain picture set and the spectrogram picture set respectively and their parameters are optimized respectively, and the neural networks with the best classification performance are obtained through comparison respectively, which are called the time-domain picture-neural network and the spectrogram-picture neural network.

[0071] From the audio perspective, using speech processing technology, four types of detection signals are respectively converted into four types of audio, and tags are added respectively to construct an audio set containing four tags. A newly proposed neural network applicable to audio classification is trained on the audio set, and targeted parameter optimization is performed according to the classification performance of the neural network, thereby obtaining the neural network with the optimal performance, called the audio-neural network.

[0072] Finally, from the perspectives of three information carriers, fuzzy rules are used to integrate the data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network in parallel to construct an integrated model. In fact, a mathematical mapping is used to fuse the classification performances of the four classifiers or neural networks, and finally a credible classification result is output.

[0073] 1. Signal acquisition and data construction:

[0074] In the application process, for a certain sealed electronic component to be tested, an unknown signal is obtained through experiments and input into the integrated model. The integrated model processes this unknown signal into three information carriers respectively, obtaining a piece of data, two pictures, and a segment of audio, which are respectively called the unknown data, unknown time-domain graph, unknown spectrogram, and unknown audio. The integrated model uses four independent processing threads to predict the tags of the data, pictures, and audio in parallel, and performs weighted voting on the four predicted tags to output a predicted tag, that is, the final data classification result and also the final signal recognition result.

[0075] Referring to the models of the sealed electronic components used in aerospace equipment in the real scenario, as well as the attributes of the foreign objects detected in the sealed electronic components, such as quantity, weight, shape, and material, twenty sealed electronic components of the same model were customized from the original manufacturer as samples. As Figure 4 shown, the samples are numbered. The sealed electronic component samples numbered 1 to 5 each contain only one foreign object, and the five foreign objects are respectively a 1.1 μg iron chip, a 1.2 μg copper wire, a 1.8 μg aluminum sphere, a 2.1 μg tin sphere, and a 1.6 μg wire segment. The five sealed electronic component samples numbered 6 to 10 each contain only one small loose component, specifically a bonding wire. The five sealed electronic component samples numbered 11 to 15 contain both a foreign object and a small loose component at the same time. Among them, the attributes of the five foreign objects are the same as those of the foreign objects contained in the sealed electronic component samples numbered 1 to 5, and the small loose component is a bonding wire. The five sealed electronic component samples numbered 16 to 20 each contain only one large loose component, specifically a movable nut.

[0076] Figure 5The DZJC-III type foreign object automatic detection system independently developed by the research group is presented. It consists of three parts: a vibration table, a collection device, and an embedded computer. Fix the sealed electronic component samples on the vibration table, operate the embedded computer to control the operation of the vibration table to apply mechanical excitation, so that the foreign objects or components inside the sealed electronic component samples move or vibrate, generating detection signals. The sound sensor located on the tabletop of the vibration table captures the detection signals and inputs them into the collection device through a high-shielded transmission line. The collection device processes the detection signals into signal files in an editable "dat" format and sends them to the embedded computer through the serial port. The embedded computer receives the signal files and stores them in the local memory, thereby collecting the detection signals. At the same time, the embedded computer reads the signal files and performs visual display.

[0077] Through the above steps, each sealed electronic component sample is placed on the vibration table, and a large number of detection signals are collected respectively. After statistics, 1671 segments of foreign object signals, 1714 segments of component signals, 1728 segments of mixed signals, and 1834 segments of oversized component signals are collected, a total of 6947 segments of detection signals.

[0078] (1) Dataset construction: From a data perspective, sixteen features are selected from the time domain and frequency domain, including seven time-domain features and nine frequency-domain features, as shown in Table 1. It is worth noting that the feasibility and superiority of these features have been fully verified in previous studies.

[0079] Table 1 Sixteen time-domain and frequency-domain features

[0080]

[0081]

[0082] Feature engineering is carried out on 6947 segments of detection signals respectively, that is, the values of sixteen features are calculated on 6947 segments of detection signals respectively, and according to the serial numbers of the features in Table 1, the sixteen values of each segment of detection signal are arranged in a row to construct a feature vector. On this basis, the label "0" is added to the 1671 feature vectors of the source foreign object signals, the label "1" is added to the 1714 feature vectors of the source component signals, the label "2" is added to the 1728 feature vectors of the source mixed signals, and the label "3" is added to the 1834 feature vectors of the source oversized component signals. Thus, a dataset containing four labels is constructed.

[0083] (2) Image set construction: From an image perspective, as mentioned above, 6947 segments of detection signals are actually 6947 signal files. On the one hand, directly read the signal files and draw the time-domain diagrams of the detection signals, such as Figure 1As shown. It should be noted that in the time-domain diagram, the duration of the detection signal is 5s, the upper amplitude limit is 5V, and the lower amplitude limit is -5V. Thus, 6947 time-domain diagrams are obtained. On this basis, label "0" is added to 1671 time-domain diagrams of source foreign matter signals, label "1" is added to 1714 time-domain diagrams of source component signals, label "2" is added to 1728 time-domain diagrams of source mixed signals, and label "3" is added to 1834 time-domain diagrams of source oversized component signals. Thus, a time-domain picture set containing four labels is constructed.

[0084] On the other hand, the detection signal is converted into a spectrogram using spectrogram technology, and its conversion process is as follows.

[0085] First, a high-pass filter is used to pre-emphasize the detection signal, and its principle is shown in formula (1):

[0086] S[n] = x[n] - μ × x[n - 1] (1)

[0087] In the formula, x[n] represents the detection signal, μ represents the pre-emphasis coefficient, and is set to μ = 0.97.

[0088] Secondly, the pre-emphasized detection signal is framed, and each frame signal is set to 50ms. Considering that Figure 5 the sampling frequency of the acquisition device is 500kHz, each frame signal contains twenty-five sampling points.

[0089] Then, the framed detection signal is windowed to make it continuous, and its principle is shown in formula (2):

[0090] S w [n] = S[n] × w[n] (2) In the formula, S w [n] represents the windowed detection signal, w[n] is a Hamming window, and its calculation formula is as follows:

[0091]

[0092] In the formula, N represents the number of sampling points in the detection signal.

[0093] Finally, the fast Fourier transform is performed on each windowed frame signal respectively, with the length set to 1024, and the power spectrum of each frame signal is processed by a band-pass filter. On this basis, each frame signal is arranged in chronological order and visualized to obtain the spectrogram corresponding to the detection signal, as Figure 3 shown in.

[0094] Thus, 6947 spectrograms are obtained. On this basis, the label "0" is added to 1671 spectrograms of source foreign matter signals, the label "1" is added to 1714 spectrograms of source component signals, the label "2" is added to 1728 spectrograms of source mixed signals, and the label "3" is added to 1834 spectrograms of source oversized component signals. Thus, a spectrogram image set containing four labels is constructed.

[0095] (3) Audio set construction: From the audio perspective, the detection signals obtained by the DZJC-III type foreign matter automatic detection system are essentially single-channel signals, or single-channel audio, which cannot be processed by current mainstream neural networks. Therefore, first, the detection signals are converted into overall audio, specifically signal files in the "wav" format. Second, the overall audio is amplified to increase the details in the detection signals. Then, the overall audio is filtered to remove irrelevant frequency components. Finally, the overall audio is normalized to meet the input requirements of the neural network. Thus, 6947 detection signals are converted into 6947 qualified audio segments. On this basis, the label "0" is added to 1671 audio segments of source foreign matter signals, the label "1" is added to 1714 audio segments of source component signals, the label "2" is added to 1728 audio segments of source mixed signals, and the label "3" is added to 1834 audio segments of source oversized component signals. Thus, an audio set containing four labels is constructed.

[0096] 2. Model training and optimization:

[0097] (a) Data-classifier: In machine learning, classifiers can be mainly classified into two categories, tree model-based classifiers and mathematical model-based classifiers. The former includes decision trees, random forests, and XGBoost, and the latter includes k-nearest neighbors (kNN) and support vector machines (SVM). In addition, as an important branch of machine learning, some classic neural networks in deep learning can also be used for data classification, such as one-dimensional convolutional neural networks (1D-CNN). In the present invention, the data set is divided into a training set and a validation set according to a ratio of 7:3. The above six classifiers with default parameter configurations are trained on the training set, and the classification performance of the six classifiers is tested on the validation set, and their obtained classification accuracies are statistically calculated. Table 2 shows the parameter configurations of the six classifiers, and Table 3 shows the classification accuracies obtained by the six classifiers.

[0098] Table 2 Parameter configurations of six classifiers

[0099]

[0100] Table 3 Classification accuracies obtained by six classifiers

[0101]

[0102] As can be seen from Table 3, the random forest achieves the highest classification accuracy and has obvious advantages compared with other classifiers. This is because the random forest has stronger adaptability to data. During the process of constructing the base classifier, it uses random sampling to construct data subsets, which increases the ability of the base classifier to explore different feature combinations. As a result, the integrated random forest can discover more effective classification rules and effectively improve its classification performance. The decision tree achieves relatively significant classification accuracy, indicating that it also has excellent classification performance as the base classifier of the random forest. The 1D-CNN achieves the lowest classification accuracy, indicating that the classical neural network for data classification will be overwhelmed when facing high-dimensional feature vectors.

[0103] Furthermore, the grid search method is used to optimize the parameters of the most prominent random forest to further improve its classification performance. By setting different value ranges and step sizes for four parameters and obtaining the optimal value combination of the four parameters through multiple rounds of iteration, as shown in Table 4, the random forest with optimized parameters is obtained.

[0104] Table 4 Optimal parameter combination of the random forest

[0105]

[0106] On this basis, the classification accuracy achieved by the random forest with optimized parameters is recalculated, which is 87.84%. Compared with the classification accuracy of 86.10% achieved by the random forest, it has increased by 1.74%, and the increase is obvious. This fully demonstrates the feasibility and superiority of the random forest with optimized parameters. Therefore, the random forest with optimized parameters is identified as the data-classifier, and the highest classification accuracy achieved by the data-classifier from the data perspective is identified as 87.84%.

[0107] (b) Image - Neural Networks: In the present invention, seven neural networks that are currently popular and commonly used in image classification are selected: AlexNet, ResNet18, ResNet50, GoogLeNet, VGG16, VGG19, and MobileNet, covering convolutional neural networks, residual neural networks, and lightweight neural networks with different network scales and complexities, which are widely representative. The time - domain picture set and the spectrogram picture set are respectively divided into a training picture set and a validation picture set according to a ratio of 7:3. The above - mentioned seven neural networks with default structures and configurations are trained on the training picture set, and their classification performances on the two training picture sets are initially evaluated, and the classification accuracy curves and Loss curves obtained within 150 epochs are plotted. Then, their classification performances are further tested on the two validation picture sets. In addition to plotting the classification accuracy curves and Loss curves obtained within 150 epochs, the average classification accuracy obtained within 150 epochs is also statistically calculated. Table 5 shows the structures and configurations of the seven neural networks, and Table 6 shows the average classification accuracies obtained by the seven neural networks on the two validation picture sets respectively. Figures 12 to 25 Shows the classification accuracy curves and Loss curves obtained by the seven neural networks on the two training picture sets and the two validation picture sets. Figure 12 Shows the performance of AlexNet on the time - domain picture set, Figure 13 Shows the performance of ResNet18 on the time - domain picture set, Figure 14 Shows the performance of ResNet50 on the time - domain picture set, Figure 15 Shows the performance of VGG16 on the time - domain picture set, Figure 16 Shows the performance of VGG19 on the time - domain picture set, Figure 17 Shows the performance of MobileNet on the time - domain picture set, Figure 18 Shows the performance of GoogLeNet on the time - domain picture set, Figure 19 Shows the performance of AlexNet on the spectrogram picture set, Figure 20 Shows the performance of ResNet18 on the spectrogram picture set, Figure 21 Shows the performance of ResNet50 on the spectrogram picture set, Figure 22 Shows the performance of VGG16 on the spectrogram picture set, Figure 23 Shows the performance of VGG19 on the spectrogram picture set, Figure 24 Shows the performance of MobileNet on the spectrogram picture set, Figure 25 Shows the performance of GoogLeNet on the spectrogram picture set, Figures 12 to 25 In (a) is the classification accuracy curve, and (b) is the Loss curve.

[0108] Table 5 Structures and Configurations of Seven Neural Networks

[0109]

[0110] Table 6 The highest classification accuracies obtained by seven neural networks

[0111]

[0112] As can be seen from Table 6, VGG19 achieved the highest average classification accuracies on both validation image sets, which were 97.60% and 99.29% respectively, showing a significant advantage over the average classification accuracies obtained by other neural networks. At the same time, the average classification accuracies obtained by the seven neural networks on two training image sets within 150 epochs were also statistically analyzed, and VGG19 still performed the most prominently. Through the classification accuracy curve and the Loss curve, it can be seen that whether on the two training image sets or the two validation image sets, the seven neural networks showed similar changing trends. Specifically, as the number of epochs increased, the classification accuracy obtained by AlexNet showed a gradually increasing trend, reflecting its certain learning and adaptation abilities. However, in the later stage, the Loss curve of AlexNet showed slight oscillations, indicating its instability in approaching the optimal solution. This is because the network structure of AlexNet is relatively shallow, which has certain limitations when dealing with complex data sets. Although ResNet18 and ResNet50 demonstrated good classification performance on the two training image sets, their performances on the two validation image sets were extremely unstable, clearly showing the overfitting phenomenon. This indicates that although the deep network structure of the residual neural network enables it to capture more delicate features, it also makes it more likely to overfit the training samples, thus affecting its generalization performance. Due to its lightweight design, MobileNet had little change in the classification accuracy and Loss obtained throughout the training process. This implies that MobileNet may be restricted by its network structure when extracting complex features in time-domain graphs and spectrograms. The classification performance of GoogLeNet fluctuated significantly during the training and validation processes, indicating the instability of its classification performance. Although VGG16 also performed well, there was still a certain gap compared with VGG19. This further confirmed that the additional feature extraction and learning capabilities brought by the unique network structure of VGG19 are the key factors for its excellent classification performance.

[0113] Furthermore, the parameters of VGG19 are optimized to further improve its classification performance. Specifically, the first-layer convolutional kernel of VGG19 is modified from 3×3 to 7×7 to further enhance its ability to understand and process images. A PReLU activation function is newly introduced to replace the ReLU activation function, enabling it to effectively process the negative signal components contained in the detection signals. The improved VGG19 is called PReLU-VGG19-Plus. In the present invention, a multi-head attention mechanism is introduced into VGG19, and this mechanism is derived from the Transformer architecture. Specifically, the multi-head attention mechanism is placed between the last convolutional block and the fully connected layer, as Figure 6 shown. The VGG19 after this improvement is called PP-MHA-VGG19.

[0114] On this basis, PP-MHA-VGG19 is trained on two training image sets respectively, and their classification performances on the two training image sets are preliminarily evaluated, and the classification accuracy curves and Loss curves obtained within 150 epochs are plotted. Then, their classification performances are further tested on two validation image sets. On the one hand, the classification accuracy curves and Loss curves obtained within 150 epochs are plotted. Figure 7 and Figure 8 respectively give the classification accuracy curves and Loss curves of PP-MHA-VGG19 on the time-domain image set and the spectrogram image set. Figure 7 and Figure 8 (a) in is the classification accuracy curve,

[0115] (b) is the Loss curve. On the other hand, their average classification accuracies obtained within 150 epochs are statistically calculated, as shown in Table 7.

[0116] Table 7 Average Classification Accuracies Obtained by VGG19 and PP-MHA-VGG19

[0117]

[0118] As can be seen from Table 7, the average classification accuracies of PP-MHA-VGG19 on the two image sets are 99.55% and 99.76% respectively, which are 1.95% and 0.47% higher than those of VGG19 respectively, and the improvement amplitude is obvious. Especially it is particularly valuable when the room for improvement is limited. From Figure 7 and Figure 8As can be seen, by improving the first-layer convolutional kernel, introducing the PReLU activation function, and the multi-head attention mechanism, PP-MHA-VGG19 has significantly improved both in classification accuracy and Loss. Specifically, compared with VGG19, PP-MHA-VGG19 has higher single-classification accuracy and average classification accuracy on two image sets, indicating that a larger first-layer convolutional kernel enhances the model's ability to capture more extensive information in the input image, the PReLU activation function enhances the model's learning and generalization ability, and the multi-head attention mechanism improves the model's understanding of global features. The combined effect of the three significantly improves the classification performance of the model. In addition, the Loss curve of PP-MHA-VGG19 is more stable, reflecting its advanced generalization performance compared to VGG19. Therefore, PP-MHA-VGG19 trained on the time-domain image set and the spectrogram image set are respectively identified as the time-domain image-neural network and the spectrogram image-neural network, and the highest classification accuracies obtained from the perspective of images for the time-domain image-neural network and the spectrogram image-neural network are 99.55% and 99.76% respectively.

[0119] (c) Audio-neural network: First, a feed-forward neural network based on deep learning is proposed for the audio set. The feed-forward neural network first uses the Flatten layer to extract features that reflect the complexity and diversity of the input audio, including Mel-frequency cepstral coefficients, chromatic features, and Mel spectrograms. At the same time, the Flatten layer ensures that the feed-forward neural network is applicable to audio of different durations. Then, after the Flatten layer, three fully connected layers are used. Through the ReLU activation function of the three fully connected layers, the non-linear processing ability is enhanced, enabling it to learn more complex feature combinations. Finally, the Softmax activation function is used as the output layer to output the probability distribution of each category, thereby achieving accurate classification of the input audio. Figure 9 The structure diagram of the feed-forward neural network is given.

[0120] Similarly, the audio set is divided into a training audio set and a validation audio set in a ratio of 7:3. The feed-forward neural network is trained on the training audio set to initially evaluate its classification performance, and the classification accuracy curve and Loss curve obtained within 150 epochs are plotted. Then, its classification performance is further tested on the validation audio set. In addition to plotting the classification accuracy curve and Loss curve obtained within 150 epochs, the average classification accuracy obtained within 150 epochs is also calculated, which is 71.09%. Figure 26Performance of the feedforward neural network on the AudioSet, where (a) is the classification accuracy curve and (b) is the Loss curve. From the classification accuracy curves and Loss curves of the feedforward neural network on the training audio set and the validation audio set, it can be seen that after 150 epochs, the classification accuracies achieved by the feedforward neural network on the training audio set and the validation audio set did not show significant improvement, and the classification accuracy achieved on the validation audio set showed large fluctuations. The reason for this fluctuation may be overfitting, that is, the feedforward neural network is overly sensitive to the details in the training samples, resulting in limited generalization performance on new samples. At the same time, although the Loss curves of the feedforward neural network on the training audio set and the validation audio set gradually tend to converge, and the two curves gradually approach, they are still relatively high compared to the ideal situation. This indicates that the feedforward neural network may not be complex enough to fully fit the complex patterns in the audio, resulting in the Loss not decreasing in the later stage. In fact, the essence of audio is time-series data with significant temporal dependence. However, the feedforward neural network cannot capture these characteristics. In addition, there is no Dropout layer in the feedforward neural network, increasing the risk of overfitting.

[0121] Furthermore, improvement measures were proposed. Specifically, the feature extraction part was retained, and for the classification network, 1D-CNN was used. Compared with the feedforward neural network, 1D-CNN can more effectively capture the temporal dependence and local features in the audio by introducing convolutional layers and pooling layers. In addition, to reduce the risk of overfitting, a Dropout layer was added after each convolutional layer and pooling layer in 1D-CNN. The improved model is called the audio convolutional network, and its structure is as Figure 10 shown.

[0122] On this basis, the audio convolutional network was trained on the training audio set, and its classification performance was initially evaluated by plotting the classification accuracy curve and Loss curve obtained within 150 epochs. Then, its classification performance was further tested on the validation audio set. On the one hand, the classification accuracy curve and Loss curve obtained within 150 epochs were plotted. Figure 11 The classification accuracy curve and Loss curve of the audio convolutional network on the audio set are given respectively, where (a) is the classification accuracy curve and (b) is the Loss curve. On the other hand, the average classification accuracy obtained within 150 epochs was statistically calculated to be 72.81%.

[0123] Compared with the average classification accuracy achieved by the feedforward neural network on the validation audio set, the average classification accuracy achieved by the audio convolutional network increased by 1.72%, with an obvious improvement. From Figure 11It can be seen that the classification accuracy curve of the audio convolutional network on the validation audio set still fluctuates. However, compared with the feedforward neural network, the classification accuracy curve of the audio convolutional network on the training audio set has increased steadily. This is because the audio convolutional network can effectively capture the local features and temporal dependencies in the audio through the convolutional layer. This characteristic enables the network to focus on the key parts of the audio, thereby improving its adaptability to audio. In addition, the amplitude of vibration of the Loss curve of the audio convolutional network on the validation audio set has decreased, and the Loss curves on the training audio set and the validation audio set have increased. This is because the Dropout layer introduced in the audio convolutional network randomly discards some neurons during the training process, reducing its overfitting to the training audio set and improving its generalization performance on unknown samples. Therefore, the audio convolutional network is identified as an audio-neural network, and the highest classification accuracy achieved by the audio-neural network from the audio perspective is 72.81%.

[0124] (d) Integrated model construction: In the field of multimodal data processing, fusion technology is recognized as the key to improving the efficiency and accuracy of classification tasks. This technology enhances the depth of data understanding and the precision of decision-making by integrating information from different data sources. Fusion technology specifically includes feature-level fusion technology and decision-level fusion technology. Among them, feature-level fusion technology focuses on the direct integration and interaction of information between different modalities, especially suitable for scenarios where there is a high degree of correlation and complementarity between modalities. In contrast, decision-level fusion technology emphasizes maintaining the independence between modalities and focuses on the integration of final decisions, making it more suitable for scenarios that require maintaining the independent processing capabilities of each modality. Considering that in this study, the optimal classifiers or optimal neural networks have been constructed from the perspectives of three information carriers respectively, and they are independently processed and do not interfere with each other, therefore, the decision-level fusion technology based on fuzzy rules is used to construct the integrated model.

[0125] Generally, fuzzy rules include four stages: initialization, fuzzification, fuzzy inference, and defuzzification. Among them, the initialization stage mainly defines the fuzzy sets and membership functions of input variables. The fuzzification stage mainly calculates the membership degrees of each classifier or neural network. The fuzzy inference stage mainly designs fuzzy rules and weight calculations. The defuzzification stage mainly calculates the weighted average classification accuracy of the integrated model. Table 8 lists the classification accuracies achieved by the optimal classifiers or optimal neural networks constructed from the perspectives of three information carriers respectively. On this basis, applicable fuzzy rules are designed to construct the integrated model.

[0126] Table 8 Classification accuracies achieved by the optimal classifiers or optimal neural networks

[0127]

[0128] First is the initialization stage. In this stage, appropriate trapezoidal and triangular membership functions are designed, and the classification accuracy obtained by each optimal classifier or optimal neural network is divided into three fuzzy sets: Low, Medium, and High.

[0129] The calculation formulas for the three fuzzy sets are shown in Formulas (4) to (6).

[0130]

[0131] In the formula, x represents 100 times the classification accuracy obtained by the optimal classifier or optimal neural network.

[0132] Secondly is the fuzzification stage. In this stage, according to Formulas (4) to (6), the membership degrees of each optimal classifier or optimal neural network are calculated respectively, as shown in Table 9. Taking the data-classifier as an example, the detailed calculation process is given: μ Low = 0, μ Medium = (90 - 87.84) / 10 = 0.216, μ High = (87.84 - 80) / 10 = 0.784.

[0133] Table 9 Membership Degrees of the Optimal Classifier or Optimal Neural Network

[0134]

[0135] Then is the fuzzy inference stage. The weight formula is defined as follows:

[0136] W(i) = μ High + 0.5×μ Medium (7)

[0137] Thus, the weights of the data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network are calculated to be 0.892, 1, 1, and 0.1405 respectively. Further summing them up, the total weight is calculated to be 3.0325.

[0138] Taking the ratio of the weights of the data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network to the total weight as the corresponding prediction weights, the obtained prediction weights are 0.892 / 3.0325 = 0.294, 1 / 3.0325 = 0.330, 1 / 3.0325 = 0.330, and 0.1405 / 3.0325 = 0.046 respectively;

[0139] Finally is the defuzzification stage. For the classification results determined by each classifier and neural network, the prediction weights corresponding to the same classification are summed up, and finally, the class with the highest sum of prediction weights is selected as the final predicted classification.

[0140] For the convenience of evaluating the final prediction results, the calculation formula for the weighted average classification accuracy is defined as follows:

[0141]

[0142] In the formula, Acc(i) represents the classification accuracy obtained by each classifier or neural network, W(i) represents the weight of each classifier or neural network, and W represents the total weight.

[0143] Thus, the weighted average classification accuracies of data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network are calculated as (87.84×0.892 + 99.55×1 + 99.76×1 + 72.81×0.1405) / 3.0325 = 94.94. That is, the integrated model is constructed, and the classification accuracy it obtains is 94.94%.

[0144] Compared with the classification accuracies of 87.74%, 72.81%, 99.55%, and 99.76% obtained by the optimal classifier or optimal neural network, the classification accuracy of the integrated model is at a medium level, significantly higher than that of the data-classifier and audio-neural network, and slightly lower than that of the time-domain graph-neural network and spectrogram-neural network. This fully proves the feasibility of the integrated model. This indicates that the integrated model effectively balances the contributions of different classifiers or neural networks through fuzzy rules, while avoiding the drag of low-performance classifiers or neural networks and retaining the advantages of high-performance classifiers or neural networks. At the same time, it comprehensively considers multiple classifiers and neural networks, and the generalization performance and stability performance can be guaranteed. The practicality and superiority of the integrated model need to be verified in real scenarios.

[0145] In a real scenario, find thirty sealed electronic components to be tested of the Figure 4 same model, and with the help of the Figure 5 shown DZJC-III type foreign object automatic detection system, carry out foreign object detection. Thus, for each sealed electronic component to be tested, collect a segment of detection signal, called the signal to be tested, a total of thirty signals to be tested. Starting from the data, image, and audio perspectives respectively, convert the thirty signals to be tested into thirty feature vectors to be tested, thirty time-domain graphs to be tested, thirty spectrograms to be tested, and thirty audio segments to be tested.

[0146] On this basis, the data-classifier, time domain graph-neural network, spectrogram-neural network and audio-neural network are used to respectively predict the labels of thirty feature vectors to be tested, thirty time domain graphs to be tested, thirty spectrograms to be tested and thirty audio segments to be tested. Then, the integrated model is used to predict the thirty feature vectors to be tested, thirty time domain graphs to be tested, thirty spectrograms to be tested and thirty audio segments to be tested in parallel. With the help of the decision-level fusion technology based on fuzzy rules, thirty predicted labels are given with reference to the weights of the optimal classifier or the optimal neural network. Table 10 shows the prediction results of the above classifiers, neural networks and integrated models. It is worth noting that the “—” in the table indicates that the sealed electronic component to be tested is qualified, that is, there are no redundant objects or loose components in the sealed electronic component to be tested.

[0147] Table 10 Prediction results of the tested sealed electronic components

[0148]

[0149] After calculation, except for the four qualified sealed electronic components to be tested, for the remaining 26 sealed electronic components to be tested, the prediction accuracy, that is, the classification accuracy, of the data-classifier, time domain graph-neural network, spectrogram-neural network and audio-neural network were 80.77%, 80.77%, 88.46% and 69.23% respectively. Compared with the classification accuracy of 87.84%, 99.55%, 99.76% and 72.81% they had previously achieved, the decline was obvious. In particular, the classification accuracy of the time domain graph-neural network and the spectrogram-neural network decreased by 18.78% and 11.30% respectively, which was a large decline. This is actually an inevitable problem in current machine learning applications. Classifiers or neural networks that perform well on familiar samples will have reduced generalization performance when faced with new samples.

[0150] For the detection of foreign matters in sealed electronic components, this problem is even more severe. In machine learning, familiar samples and new samples usually refer to interpretable data, such as feature vectors, images, time series, etc. Thus, the difference between familiar samples and new samples is the difference in the distribution characteristics of directly interpretable data. In foreign matter detection, however, familiar samples and new samples refer to detection signals. Machine learning cannot directly process detection signals; they need to be first processed into interpretable data. Therefore, the difference between familiar samples and new samples is not the direct difference in signal characteristics, but the indirect difference in the distribution characteristics of processed interpretable data. As is well known, there must be information loss during the process of processing signals into interpretable data. For foreign matter detection, the original detection signals of new and old samples, which already have differences, will experience information loss regarding these differences after being processed into new and old interpretable data. This results in the amplification or distortion of the differences in the processed interpretable data. This leads to a decline in the generalization performance of classifiers and neural networks when dealing with data, images, and audio processed from detection signals. This well explains the performance of data-classifier, time-domain graph-neural network, spectrogram-neural network, and audio-neural network in Table 10. In addition, since the process of converting detection signals into data and audio is simpler than that into images and results in less information loss, the decline in the generalization performance of data-classifier and audio-neural network is relatively small. Even for the time-domain graph-neural network and spectrogram-neural network, the latter shows a relatively smaller decline in generalization performance and is significantly better than the data-classifier and audio-neural network because the spectrogram comprehensively integrates the time-domain and frequency-domain characteristics of detection signals more comprehensively than the time-domain graph.

[0151] In machine learning, collecting as many samples as possible to construct a complete dataset is an important means to ensure the generalization performance of classifiers or neural networks. For foreign matter detection, no matter what type of detection signals they are, they cannot be fully described. Foreign matter signals are generated by the random movement of foreign matters, and component signals are generated by uncertain loose components. Therefore, the method of widely collecting samples is not feasible. The present invention proposes a new processing idea. The means considered is to start from the perspective of multi-information carriers, train applicable classifiers and neural networks respectively, and finally construct an integrated model to ensure comprehensive and stable prediction results. This effectively explains the necessity of carrying out this research. After calculation, the classification accuracy of the integrated model is 92.31%, which is much higher than that of the four classifiers or neural networks. Compared with the obtained classification accuracy of 94.94%, the decline is relatively small and can be understood. Thus, the feasibility and practicality of the integrated model, as well as its superiority compared with individual models, are highlighted. It can be seen from the table that the integrated model only makes prediction errors on two sealed electronic components to be tested, while the other four classifiers or neural networks all make prediction errors.

[0152] In addition, using the existing research on the identification of foreign object signals and component signals based on machine learning, the thirty signals to be measured were predicted, and the prediction results are also listed in Table 10. Specifically, the existing research refers to the parameter-optimized XGBoost from the data perspective proposed by Li Chaoran et al. After calculation, the classification accuracy achieved by the existing research is 61.54%, which is much lower than the classification accuracy achieved by the four classifiers or neural networks, let alone the classification accuracy achieved by the integrated model. The existing research predicted all the labels of the data with true labels of "2" and "3" as "0" and "1", which reflects the deficiency of the existing research in only considering foreign object signals and component signals. For the thirty sealed electronic components to be measured shown in Table 10, there are ten sealed electronic components to be measured with detection signals being mixed signals or over-sized component signals, accounting for 1 / 3. This fully demonstrates the necessity and correctness of considering the four types of detection signals. In addition, for the data with true label of "3", the existing research did not predict all of their labels as "1", and there were also predictions as "0". This shows that the difference between over-sized component signals and ordinary component signals is not simply the amplitude difference, but also includes other differences in the time domain and frequency domain such as time interval and spectral distribution. This actually explains why over-sized component signals are regarded as a separate type of detection signal. If it is predicted as a foreign object signal, it will affect the result of foreign object detection. Because, if it is a foreign object signal, the sealed electronic component to be measured is unqualified, while if it is an over-sized component signal, the sealed electronic component to be measured is qualified.

[0153] Thus, the necessity of carrying out this research is demonstrated, and the feasibility, practicality, and superiority of the method proposed by the present invention are fully proven.

[0154] The above numerical examples of the present invention are only for illustrating in detail the calculation model and calculation process of the present invention, rather than limiting the implementation manner of the present invention. For those of ordinary skill in the art, other different forms of changes or variations can be made based on the above description. It is impossible to list all the implementation manners here. Any obvious changes or variations derived from the technical solution of the present invention still fall within the protection scope of the present invention.

Claims

1. An integrated model construction method for detecting redundant objects in sealed electronic components from the perspective of multiple information carriers, characterized in that: include: According to the obtained redundant object detection signal of sealed electronic components in the real scene, the type of the redundant object detection signal is determined as a label, and the redundant object detection signal is processed as follows: Based on the data perspective, features are selected from the time domain and frequency domain, different types of detection signals are converted into feature vectors, and labels are added respectively to build data sets of different types of detection signals. Then, multiple classifiers are trained and the classifier with the best classification performance is selected, which is called data-classifier. Based on the image perspective, the time domain waveforms of different types of detection signals are directly drawn, and labels are added respectively to construct time domain image sets of different types of detection signals. At the same time, the spectrogram technology is used to convert different types of detection signals into different types of spectrograms, and labels are added respectively to construct spectrogram image sets of different types of detection signals. Multiple neural network models are trained on the time domain image set and the spectrogram image set, and the neural networks with the best classification performance are compared and obtained, which are called time domain image-neural network and spectrogram-neural network. Based on the audio perspective, using speech processing technology, different types of detection signals are converted into different types of audio of detection signals, and labels are added respectively to construct audio sets of different types of detection signals. Then, multiple neural network models are trained and compared to obtain the neural network with the best classification performance, which is called audio-neural network. An integrated model is constructed based on the decision-level fusion technology of fuzzy rules: in the initialization stage, the classification accuracy achieved by each optimal classifier or optimal neural network is divided into three fuzzy sets and the corresponding membership function is determined; then the membership of each optimal classifier or optimal neural network is calculated respectively, and the weight of each optimal classifier or optimal neural network is obtained by using the fuzzy reasoning process, and the sum of all weights is the total weight; the ratio of the weight of each data-classifier, time domain graph-neural network, spectrogram-neural network, and audio-neural network to the total weight is taken as the corresponding prediction weight; the prediction weight of each data-classifier, time domain graph-neural network, spectrogram-neural network, and audio-neural network is the prediction weight corresponding to its prediction result, and then the data-classifier, time domain graph-neural network, spectrogram-neural network, and audio-neural network are integrated to obtain an integrated model.

2. The integrated model construction method for detecting redundant objects in sealed electronic components from the perspective of multiple information carriers according to claim 1 is characterized in that: The types of the redundant object detection signal include: normal signal, redundant object signal, component signal, mixed signal and excessive component signal; the redundant object signal refers to a signal whose pulse sequence belongs to a non-periodic sequence; the component signal refers to a signal whose pulse sequence is approximately a periodic sequence; the mixed signal refers to a signal whose pulse sequence reflects both randomness and periodicity; the excessive component signal refers to a signal whose pulse sequence time interval reflects periodicity and whose amplitude reflects randomness.

3. The integrated model construction method for detecting redundant objects in sealed electronic components from the perspective of multiple information carriers according to claim 1 is characterized in that: The data-classifier adopts a random forest model.

4. The integrated model construction method for detecting redundant objects in sealed electronic components from the perspective of multiple information carriers according to claim 1 is characterized in that: The time domain graph-neural network and the spectrogram-neural network both adopt an improved VGG19 network model, and the improved VGG19 network model is improved as follows: The first-layer convolution kernel of VGG19 is changed from 3×3 to 7×7, and the PReLU activation function is used instead of the ReLU activation function; the multi-head attention mechanism of the Transformer architecture is introduced in VGG19, and the multi-head attention mechanism is set between the last convolution block and the fully connected layer.

5. The integrated model construction method for detecting redundant objects in sealed electronic components from the perspective of multiple information carriers according to claim 1, characterized in that: The audio-neural network is as follows: For the input audio, the Flatten layer is first used to extract features from it; after the Flatten layer, at least three fully connected layers are used, each of which is set with a ReLU activation function; finally, the Softmax activation function is used as the output layer to output the probability distribution of each category.

6. The integrated model construction method for detecting redundant objects in sealed electronic components from the perspective of multiple information carriers according to claim 1, characterized in that: The audio-neural network is as follows: For the input audio, 1D-CNN is first used to extract features. 1D-CNN contains multiple convolutional units, each of which includes a convolutional layer and a pooling layer, and a Dropout layer is added after the pooling layer; 2 fully connected layers are set after 1D-CNN, and then the Softmax activation function is used as the output layer to output the probability distribution of each category.

7. The integrated model construction method for detecting redundant objects in sealed electronic components from the perspective of multiple information carriers according to any one of claims 1 to 6, characterized in that: The classification accuracy obtained by each optimal classifier or optimal neural network is divided into three fuzzy sets and the corresponding membership functions are determined as follows: In the formula, μ Low (x), μ Medium (x), μ High (x) are the membership functions corresponding to the three fuzzy sets of low, medium and high, respectively; x represents 100 times the classification accuracy achieved by the optimal classifier or the optimal neural network.

8. The integrated model construction method for detecting redundant objects in sealed electronic components from the perspective of multiple information carriers according to claim 7, characterized in that: The fuzzy inference process is used to obtain the weights of each optimal classifier or optimal neural network as follows: W(i)=μ High +a×m Medium Among them, W(i) represents the weight of each optimal classifier or optimal neural network, and α is the adjustment weight.

9. The integrated model construction method for detecting redundant objects in sealed electronic components from the perspective of multiple information carriers according to claim 8, characterized in that: The adjustment weight α is set to 0.

5.

10. A method for detecting excess matter in sealed electronic components from a multi-information carrier perspective, characterized in that: The following steps are involved: For the obtained sealed electronic component surplus object detection signal in the real scene, the integrated model constructed by the integrated model construction method for sealed electronic component surplus object detection from the perspective of multiple information carriers as described in any one of claims 1 to 9 is used for detection, and the prediction results of the data-classifier, time domain graph-neural network, spectrogram-neural network and audio-neural network respectively correspond to a prediction weight. For the classification results determined by each classifier and neural network, the prediction weights corresponding to the same classification are added, and finally the category with the highest prediction weight and the highest classification is selected as the final prediction classification.

Images