Method for Locating Redundant Materials and Identifying Material Types in a Sealed Electronic Device with Multi-Source Information Fusion, and Pulse Extraction and Matching Methods

Abstract

A method for locating and identifying the material of foreign objects in a sealed electronic device through multi-source information fusion, as well as a method for pulse extraction and matching, belongs to the technical field of foreign object detection in sealed electronic devices. In order to solve the problem that the recognition accuracy of locating and identifying foreign objects in existing sealed electronic devices in the prior art needs to be further improved. For the signals obtained by simultaneous measurement of multiple acoustic emission sensors, pulse extraction and pulse matching are performed on multi-channel foreign object signals to obtain multiple pulse groups; spectrograms are generated respectively for the pulses, and neural networks are used to obtain position and material identification labels, and the results are obtained through majority voting; gray-level co-occurrence matrix features are extracted from the spectrograms, and combined with the time-domain and frequency-domain features of the pulses in the same group to obtain an identification feature vector, and a classifier is used to obtain position / material identification labels, and the results are obtained through majority voting; the results are obtained through majority voting based on all predicted labels; the final result is obtained through majority voting based on all the results obtained by voting.

Description

Technical Field

[0001] The present invention belongs to the technical field of redundant object detection for sealed electronic devices, and particularly relates to a method for positioning and material identification of redundant objects in sealed electronic devices, a pulse extraction method, and a pulse matching method. Background Art

[0002] A sealed electronic device refers to an electronic device encapsulated with special materials and processes to prevent damage or interference from the external environment, such as water, dust, chemical substances, etc. Sealed electronic devices usually have characteristics such as waterproof, dustproof, and corrosion-resistant, and can operate stably in harsh environments. They are widely used in the aerospace and defense fields. Due to the limitations of the production process, during the manufacturing process of sealed electronic devices, substances such as metal debris, spot welding spatter, and thin wire segments are encapsulated inside to form redundant objects. Sealed electronic devices usually operate in environments of weightlessness and severe shock. After the internal redundant objects are activated, they will collide and damage internal components, or adhere to the circuit surface to cause short circuits or open circuits, or get stuck in moving parts to make some functions fail, etc., which has an important impact on the reliable operation of sealed electronic devices, leading to the failure of space launch missions, and even triggering major space accidents and casualties. Therefore, it is crucial to detect redundant objects before the sealed electronic devices leave the factory and enter service.

[0003] The Particle Impact Noise Detection (PIND) method is the most commonly used redundant object detection method. The redundant object detection process pays more attention to whether a redundant object signal is detected rather than in-depth research on the signal itself. In fact, the redundant object signal contains information that can reflect the properties of the redundant object itself, such as the material, position, weight, etc. of the redundant object. Especially in the research on redundant object detection of sealed electronic devices with large sizes and complex structures, after determining the existence of redundant objects, obtaining the position of the redundant objects can guide the cleaning work, and obtaining the material of the redundant objects can trace the source and improve the process standards. Therefore, the complete redundant object detection information for sealed electronic devices should not only include whether there are redundant objects, but also include information on the properties of the redundant objects themselves, especially the position information and material information.

[0004] In recent years, machine learning has been widely applied to the research of signal classification and recognition. It constructs feature data by quantifying the signal characteristic differences between signals of different classes, trains a classifier in a high-dimensional feature space to classify the feature data with different labels, and then realizes the recognition of signals of different classes. Similarly, the signal characteristic differences between debris signals generated by debris at different positions and of different materials can also be quantified to train a suitable classifier to respectively realize debris positioning and material identification. In previous studies, aiming at the problem of debris positioning in sealed electronic devices, a complete experimental system was built with debris signals as the research object, specific implementation steps were designed, and research was carried out in aspects such as pulse extraction and matching, feature extraction and optimization, and integrated classifier optimization, achieving a positioning accuracy of 90.91%. Unfortunately, in previous studies, the internal structure of the used sealed electronic devices was relatively simple and regular, such as aerospace power supplies, etc. Thus, the internal space of the device models made was divided into airtight spaces with approximately equal sizes, and the distribution characteristics between the constructed feature data with different labels were relatively regular, and the performance of the trained integrated classifier was relatively excellent. This was friendly for carrying out initial research. However, in real application scenarios, the internal structure of sealed electronic devices is more complex and irregular. Thus, the internal space of the device models made can only be divided into airtight spaces with different shapes and sizes, and the feasibility and generalization of the original debris positioning method need to be further verified.

[0005] It is worth noting that as an important branch of machine learning, in recent years, deep learning has been widely applied to fields such as acoustic emission detection, speech recognition, and fault diagnosis, and has achieved remarkable results. For example, Lei et al. carried out research on the health maintenance decision-making and remaining life prediction of mechanical equipment, Li et al. carried out research on the intelligent operation and maintenance and defect detection of equipment based on visual signals, Feng et al. carried out research on the digital twin and fatigue wear of complex systems, and He et al. carried out research on computational intelligence and fault diagnosis in industrial processes. These studies all took signal images as the research object. Even if the research object in the initial stage was digital format fault signals and sound signals, they were also converted into signal images that can be processed by deep learning. Generally speaking, debris signals belong to digital format acoustic emission signals, and spectrogram technology can convert them into signal images. In previous studies, due to the concern that the attribute information of debris signals themselves would be lost during the process of converting signals into images, no deep learning-based debris detection research was carried out with images as the research object. However, the results of a large number of studies carried out by scholars show that the classic neural network in deep learning can make up for the information loss caused by the signal conversion process with its excellent classification performance, and achieve a classification effect higher than that with signals as the research object. This provides a new idea for the existing research on debris positioning and material identification.

[0006] Therefore, regarding the problems existing in the positioning of foreign matters in existing sealed electronic devices and the identification of the materials of foreign matters in sealed electronic components, namely, the positioning and material identification capabilities need to be improved, the pulse extraction and matching algorithms need to be optimized, a combined classifier with a stable structure needs to be designed, and the accuracy of material identification of foreign matters in sealed electronic devices remains to be further improved. Summary of the Invention

[0007] The present invention aims to solve the problem that the identification accuracy of positioning and material identification of foreign matters in existing sealed electronic devices in the prior art remains to be further improved.

[0008] A method for positioning and material identification of foreign matters in a sealed electronic device based on multi-source information fusion, comprising:

[0009] For a sealed electronic device to be tested, signals obtained from a single measurement by multiple acoustic emission sensors are acquired simultaneously. The signal corresponding to each acoustic emission sensor is recorded as a foreign matter signal of one channel. The number of multi-channels is denoted as N.

[0010] Pulse extraction is performed on the N foreign matter signals corresponding to the N channels. A plurality of useful pulses are obtained from a segment of the foreign matter signal of any one channel. Pulse matching is performed on the plurality of useful pulses of the N channels. After pulse matching, a plurality of pulse groups are obtained from the N foreign matter signals corresponding to the N channels. Each pulse group contains one useful pulse of each channel. One pulse group is denoted as a useful pulse group.

[0011] Taking the useful pulse as the basic unit, a spectrogram is generated for each useful pulse, and gray-level co-occurrence matrix features are respectively extracted for each spectrogram. At the same time, time-domain and frequency-domain features are extracted for each useful pulse.

[0012] For each group of useful pulse groups, based on the time-domain features, frequency-domain features, and gray-level co-occurrence matrix features corresponding to a group of useful pulses, a feature vector for positioning is obtained, denoted as the first identification feature vector, and / or, based on the time-domain features, frequency-domain features, and gray-level co-occurrence matrix features corresponding to a group of useful pulses, a feature vector for material identification is obtained, denoted as the second identification feature vector.

[0013] Then, all the spectrograms are respectively input into a positioning recognition neural network and / or a material recognition neural network. Each spectrogram obtains a position prediction label of a foreign matter in the electronic device and / or a material prediction label of a foreign matter in the electronic device. A position prediction result is obtained through majority voting of all the position prediction labels and is denoted as the first position prediction result, and / or, a material prediction result is obtained through majority voting of all the material prediction labels and is denoted as the first material prediction result.

[0014] Meanwhile, input all the first recognition feature vectors into the positioning recognition classifier respectively. Each first recognition feature vector obtains a position prediction label of the foreign object in the electronic device. A position prediction result is obtained through majority voting of all the position prediction labels and is denoted as the second position prediction result; and / or, input all the second recognition feature vectors into the material recognition classifier. Each second recognition feature vector obtains a material prediction label of the foreign object in the electronic device. A material prediction result is obtained through majority voting of all the material prediction labels and is denoted as the second material prediction result;

[0015] Merge and conduct majority voting on all the position prediction labels corresponding to the positioning recognition neural network and all the position prediction labels corresponding to the positioning recognition classifier to obtain a position prediction result and denote it as the third position prediction result, and / or, merge and conduct majority voting on all the material prediction labels corresponding to the material recognition neural network and all the material prediction labels corresponding to the material recognition classifier to obtain a material prediction result and denote it as the third material prediction result;

[0016] Conduct majority voting using the first to third position prediction results to obtain the final position prediction result, and / or, conduct majority voting using the first to third material prediction results to obtain the final material prediction result.

[0017] Furthermore, when merging and conducting majority voting on all the position prediction labels corresponding to the positioning recognition neural network and all the position prediction labels corresponding to the positioning recognition classifier, the majority voting is conducted according to the weights for all the position prediction labels corresponding to the positioning recognition neural network and all the position prediction labels corresponding to the positioning recognition classifier;

[0018] and / or,

[0019] When merging and conducting majority voting on all the material prediction labels corresponding to the material recognition neural network and all the material prediction labels corresponding to the material recognition classifier, the majority voting is conducted according to the weights for all the material prediction labels corresponding to the material recognition neural network and all the material prediction labels corresponding to the material recognition classifier.

[0020] Furthermore, the majority voting is conducted according to the weights of 0.8 and 0.2 for all the position prediction labels corresponding to the positioning recognition neural network and all the position prediction labels corresponding to the positioning recognition classifier;

[0021] and / or,

[0022] The majority voting is conducted according to the weights of 0.8 and 0.2 for all the material prediction labels corresponding to the material recognition neural network and all the material prediction labels corresponding to the material recognition classifier.

[0023] Further, for the N redundant signal segments corresponding to the N channels, the process of pulse extraction adopts a two-stage adaptive energy threshold pulse extraction algorithm to extract pulses from each redundant signal segment. The specific process includes the following steps:

[0024] Step 1: Calculate the average energy E of the redundant signal a , and calculate the standard deviation σ of the signal energy E ; Set the peak energy threshold as E p = E a + 3σ E , and set the endpoint energy threshold as E be = E a + σ E ;

[0025] Step 2: Perform frame processing on the signal;

[0026] Step 3: Starting from the first frame signal, sequentially compare the energy of each frame signal with E p ; When the energy of a certain frame signal is greater than E p , record this frame signal; and starting from it, continue to compare the energy of each subsequent frame signal with E p , until the energy of a certain frame signal is less than E p ; Find the frame signal with the maximum energy among these frame signals, and identify it as the maximum frame signal of the first useful pulse; Obtain the time of the sampling point with the maximum amplitude in the maximum frame signal, which is called the peak time t p_1 of the first useful pulse;

[0027] Step 4: Taking the peak time t p_1 of the first useful pulse extracted in Step 3 as the starting point, compare the energy of each frame signal with E be respectively forward and backward, until a certain frame signal with energy less than E be is found in both directions; Identify the previous frame signal of these two frame signals as the first start frame signal and the first end frame signal of the first useful pulse respectively; Obtain the time of the first sampling point in the first start frame signal, which is called the first start time t fb_1 of the first useful pulse; Obtain the time of the last sampling point in the first end frame signal, which is called the first end time t fe_1 of the first useful pulse;

[0028] Step 5: Taking the next frame signal after the first end time t fe_1 of the first useful pulse extracted in Step 4 as the new starting point, repeat the same processing procedures in Step 3 and Step 4 to extract the remaining useful pulses in the signal, and then save all the useful pulses extracted from the signal; At the same time, save the peak times of all useful pulses The first starting moment and the first ending moment

[0029] Step 6: Calculate the average energy and the standard deviation of the energy of each useful pulse saved in Step 5, denoted as E i and σ E_i On this basis, set the exclusive endpoint energy threshold of each useful pulse as t be_i = E i + σ Z_i ;

[0030] Step 7: Starting from the first useful pulse saved in Step 5, with the peak moment t p_1 extracted in Step 3 as the starting point, compare the energy of each frame signal with the exclusive endpoint energy threshold E be_1 forward and backward respectively until it is found that the energy of a certain frame signal is less than E be_1 in both directions; respectively identify the previous frame signal of these two frame signals as the second starting frame signal and the second ending frame signal of the first useful pulse;

[0031] Obtain the time of the first sampling point in the second starting frame signal, which is called the second starting moment t sb_1 of the first useful pulse; obtain the time of the last sampling point in the second ending frame signal, which is called the second ending moment t se_1 of the first useful pulse;

[0032] Step 8: Starting from the second useful pulse saved in Step 5, repeat the same processing steps in Step 7 to respectively obtain the second starting moment and the second ending moment

[0033] Step 9: Calculate the first duration t fl_i = t fe_i - t fb_i , the second duration t sl_i = t se_i - t sb_i , and the duration ratio If the τ i of the useful pulse is greater than the duration threshold T, then identify its first starting moment t fb_i and the first ending moment t fe_i as the final starting moment t b_i and the final ending moment t e_i ; otherwise, identify its first starting moment t fb_i and the second ending moment t se_i as the final starting moment tb_i and the final end time t e_i ; thus obtaining the final start time t of each useful pulse saved in step five b_i and the final end time t e_i and record them.

[0034] Furthermore, the process of pulse matching for multiple useful pulses of N channels adopts an improved threshold determination search pulse matching algorithm for matching. Based on the results of pulse extraction for the corresponding N segments of redundant signals in N channels, the matching process includes the following steps:

[0035] Step 1: For multiple useful pulses corresponding to different channels respectively, obtain the useful pulse with the maximum energy among the multiple useful pulses. The useful pulse with the maximum energy corresponding to N channels is called N reference useful pulses; according to the start times of the N reference useful pulses, determine the redundant signal collected by the acoustic emission sensor farthest from the redundant object, which is called the reference signal; determine the redundant signal collected by the acoustic emission sensor closest to the redundant object, which is called the comparison signal; obtain the start times of the reference signal and the comparison signal, and calculate the difference between the two start times, which is called the start time threshold T;

[0036] Step 2: Represent the useful pulses included in the reference signal in step 1 as P i , i = 1, 2,..., n. The start times of these useful pulses are t i respectively; according to the start time threshold T, set the start time threshold interval for each useful pulse as [t i - T, t i ;

[0037] Step 3: Starting from P1, search for useful pulses whose start times are within [t1 - T, t1] in the N - 1 segments of redundant signals corresponding to the other N - 1 channels respectively; if N - 1 useful pulses can be found, form a pulse group with them and P1; otherwise, skip P1, indicating that its matching is unsuccessful;

[0038] Step 4: Perform pulse matching on all P i in the manner of step 3; thus completing the pulse matching of the redundant signals of N channels and obtaining multiple pulse groups.

[0039] Furthermore, the time domain features and frequency domain features in the first recognition feature vector include: time delay, pulse rise time, pulse symmetry, amplitude, energy, root mean square voltage, zero crossing rate, spectral centroid, spectral mean square deviation, root mean square probability, frequency standard deviation. The gray level co - occurrence matrix features in the first recognition feature vector include: entropy value, contrast, homogeneity, correlation;

[0040] The time-domain features and frequency-domain features in the second recognition feature vector include: pulse rise time, pulse symmetry, energy, root mean square voltage, zero crossing rate, spectral centroid, spectral mean square deviation, root mean square probability, frequency standard deviation. The gray level co-occurrence matrix features in the second recognition feature vector include: image energy, entropy value, contrast, homogeneity, correlation.

[0041] Further, the positioning recognition neural network and the material recognition neural network adopt the PReLU-VGG19-Plus model. The PReLU-VGG19-Plus model is a VGG19 model in which the first convolutional layer is changed to a 7×7 convolutional kernel and the activation function is changed to a PReLU activation function.

[0042] Further, the positioning recognition classifier and the material recognition classifier adopt the XGBoost model;

[0043] The parameters of the XGBoost model corresponding to the positioning recognition classifier are as follows:

[0044] n_estimatores is 180, max_depth is 15, min_child_weight is 4, learning_rate is 0.38, gamma is 0.03;

[0045] The parameters of the XGBoost model corresponding to the material recognition classifier are as follows:

[0046] n_estimatores is 150, max_depth is 15, min_child_weight is 4, learning_rate is 0.41, gamma is 0.05;

[0047] n_estimatores refers to the number of base classifier decision trees; max_depth refers to the maximum depth of the base classifier decision tree; min_child_weight refers to the minimum leaf node sample weight that determines the base classifier decision tree; learning_rate refers to the learning rate; gamma refers to the penalty term coefficient.

[0048] A two-stage adaptive energy threshold pulse extraction method. This method is used to extract pulses from the debris signals obtained by a single measurement through an acoustic emission sensor when detecting a to-be-tested sealed electronic device. The signal corresponding to each acoustic emission sensor is recorded as the debris signal of one channel. The process of extracting pulses from the N segments of debris signals corresponding to N channels adopts a two-stage adaptive energy threshold pulse extraction algorithm to extract each segment of debris signal. The specific process includes the following steps:

[0049] Step 1: Calculate the average energy E of the redundant signal a , and calculate the standard deviation σ of the signal energy E ; Set the peak energy threshold as E p = E a + 3σ E , and set the endpoint energy threshold as E be = E a + σ E ;

[0050] Step 2: Perform frame processing on the signal;

[0051] Step 3: Starting from the first frame signal, compare the energy of each frame signal with E p in turn; When the energy of a certain frame signal is greater than E p , record this frame signal; and starting from it, continue to compare the energy of each subsequent frame signal with E p until the energy of a certain frame signal is less than E p ; Find the frame signal with the maximum energy among these frame signals, and identify it as the maximum frame signal of the first useful pulse; Obtain the time of the sampling point with the maximum amplitude in the maximum frame signal, which is called the peak time t p_1 of the first useful pulse;

[0052] Step 4: Taking the peak time t p_1 of the first useful pulse extracted in Step 3 as the starting point, compare the energy of each frame signal with E be forward and backward respectively until it is found that the energy of a certain frame signal is less than E be in both directions; Identify the previous frame signal of these two frame signals as the first starting frame signal and the first ending frame signal of the first useful pulse respectively; Obtain the time of the first sampling point in the first starting frame signal, which is called the first starting time t fb_1 of the first useful pulse; Obtain the time of the last sampling point in the first ending frame signal, which is called the first ending time t fe_1 of the first useful pulse;

[0053] Step 5: Taking the next frame signal after the first ending time t fe_1 of the first useful pulse extracted in Step 4 as the new starting point, repeat the same processing procedures of Step 3 and Step 4 to extract the remaining useful pulses in the signal, and then save all the useful pulses extracted from the signal; At the same time, save the peak times first starting time and the first ending time

[0054] Step 6: Calculate the average energy and the standard deviation of the energy of each useful pulse saved in Step 5, which are respectively represented as Ei and σ E_i On this basis, the exclusive endpoint energy threshold of each useful pulse is set to E be_i =E i +σ Z_i ;

[0055] Step 7: Starting from the first useful pulse saved in step 5, take its peak time t extracted in step 3 p_1 As the starting point, compare the energy of each frame signal with the exclusive endpoint energy threshold E be_1 Until the energy of a frame signal is found to be less than E in both directions be_1 ; Respectively identify the previous frame signal of the two frame signals as the second start frame signal and the second end frame signal of the first useful pulse;

[0056] The time of obtaining the first sampling point in the second starting frame signal is called the second starting time t of the first useful pulse sb_1 ; Get the time of the last sampling point in the second end frame signal, which is called the second end time t of the first useful pulse se_1 ;

[0057] Step 8: Starting from the second useful pulse saved in step 5, repeat the same processing steps as step 7 to obtain the second starting time of the remaining useful pulses saved in step 5 respectively and the second ending moment

[0058] Step 9: Calculate the first duration t of each useful pulse saved in step 5 fl_i =t fe_i -t fb_i , the second duration t sl_i =t se_i -t sb_i , and duration ratio If the useful pulse τ i If it is greater than the duration threshold T, then its first starting time t fb_i With the first end time t fe_i Determined as the final starting time t b_i and the final ending time t e_i Otherwise, its first starting time t fb_i and the second end time t se_i Determined as the final starting time t b_i and the final ending time t e_i ; Thus, the final starting time t of each useful pulse saved in step five is obtained b_i and the final ending time t e_i And record it.

[0059] An improved threshold determination search pulse matching algorithm for matching method, the method is used to match the redundant signals of multiple channels after pulse extraction of a single measurement signal, and the signal corresponding to each acoustic emission sensor is recorded as the redundant signal of one channel; the number of multi-channels is denoted as N, and when pulse extraction is performed, pulse extraction is performed on the corresponding N segments of redundant signals in the N channels; based on the pulse extraction results of the corresponding N segments of redundant signals in the N channels, an improved threshold determination search pulse matching algorithm is used for pulse matching, and the matching process includes the following steps:

[0060] Step 1: For multiple useful pulses corresponding to different channels respectively, obtain the useful pulse with the largest energy among the multiple useful pulses. The useful pulses with the largest energy corresponding to the N channels are called N reference useful pulses; according to the start times of the N reference useful pulses, determine the redundant signal collected by the acoustic emission sensor farthest from the redundant object, which is called the reference signal; determine the redundant signal collected by the acoustic emission sensor closest to the redundant object, which is called the comparison signal; obtain the start times of the reference signal and the comparison signal, and calculate the difference between the two start times, which is called the start time threshold T.

[0061] Step 2: Represent the useful pulses included in the reference signal in Step 1 as P i , i = 1, 2,..., n, and the start times of these useful pulses are t i respectively; according to the start time threshold T, set the start time threshold interval of each useful pulse as [t i - T, t i .

[0062] Step 3: Starting from P1, search for useful pulses whose start times are within [t1 - T, t1] in the corresponding N - 1 segments of redundant signals of the other N - 1 channels respectively; if N - 1 useful pulses can be found, form a pulse group with them and P1; otherwise, skip P1, indicating that its matching is unsuccessful.

[0063] Step 4: Perform pulse matching on all P i in the manner of Step 3; thus complete the pulse matching of the redundant signals of the N channels and obtain multiple pulse groups.

[0064] Advantages of the present invention:

[0065] (1) The present invention converts the useful pulses in the redundant signal into a spectrogram. On the one hand, directly train a suitable neural network on the spectrogram, and on the other hand, extract the gray-level co-occurrence matrix features on the spectrogram, establish a joint feature library and then establish a data set, and thus train a suitable classifier, which can effectively ensure and improve the recognition effect of the classifier.

[0066] (2) The present invention establishes a positioning picture set, a material picture set, a positioning data set, and a material data set through a large number of foreign object signals that simultaneously contain position information and material information. Then, a positioning neural network, a material neural network, a positioning classifier, and a material classifier are trained. Based on the positioning neural network, the material neural network, the positioning classifier, and the material classifier, joint recognition is performed. At the same time, a majority voting is also carried out based on the overall classification label set. The present invention designs a triple majority voting process, and then obtains the final positioning result after another majority voting (a majority voting is carried out overall). Thus, a combined positioning framework is constructed. That is, the present invention combines the requirements of foreign object positioning and material identification in sealed electronic devices, and for the first time extracts the method of local majority voting + overall label set voting + final overall voting, which can effectively ensure and improve the overall recognition effect. It can achieve excellent prediction effects not only for positioning but also for material identification.

[0067] In addition, in view of the deficiencies of the existing pulse extraction algorithm, the present invention newly proposes a pulse extraction algorithm based on multiple energy thresholds. In view of the deficiencies of the existing threshold determination search pulse matching algorithm, the matching method of the reference useful pulse is improved. That is, the two-stage adaptive energy threshold pulse extraction method and the improved threshold determination search pulse matching method newly proposed by the present invention can improve the effects of pulse extraction and matching, and further ensure the recognition effects of positioning and materials. Description of the Drawings

[0068] Figure 1 It is a flowchart for the implementation of the foreign object positioning and material identification method;

[0069] Figure 2 It is a physical diagram of a sealed electronic device;

[0070] Figure 3 It is an effect diagram of the internal space division;

[0071] Figure 4 It is a physical diagram of a foreign object sample;

[0072] Figure 5 It is a physical diagram of a special device;

[0073] Figure 6 It is a physical diagram of the foreign object signal acquisition process;

[0074] Figure 7 It is a schematic diagram of the two-stage adaptive energy threshold extraction algorithm;

[0075] Figure 8 It is a schematic diagram of the improved threshold determination search pulse matching algorithm;

[0076] Figure 9 It is a spectrogram;

[0077] Figure 10 It is the classification effect diagram obtained by VGG19 on the positioning picture set;

[0078] Figure 11 It is the classification effect diagram obtained by VGG19 on the material picture set;

[0079] Figure 12 It is the structural schematic diagram of VGG19-Plus;

[0080] Figure 13 It is the image of the PreLU and ReLU activation functions;

[0081] Figure 14 It is the classification effect diagram obtained by PReLU-VGG19-Plus on the positioning picture set;

[0082] Figure 15 It is the classification effect obtained by PReLU-VGG19-Plus on the material picture set. Specific implementation manner

[0083] To solve the problems in the background art, in the present invention, considering the requirements of locating and identifying the materials of foreign objects in sealed electronic devices in real application scenarios, a method for locating and identifying foreign objects by fusing multi-source information is proposed. Specifically, first, according to the latest foreign object detection information in real application scenarios, real sealed electronic devices are used to make foreign object samples of different materials, and a large number of foreign object signals generated by foreign object samples of different materials placed at different positions are collected. Secondly, the newly proposed pulse extraction algorithm and the improved pulse matching algorithm are respectively used to process the foreign object signals to obtain a large number of useful pulses. On this basis, on the one hand, the useful pulses are all converted into spectrograms, and according to the position information and material information of the foreign object signals from which the useful pulses are derived, a positioning picture set and a material picture set are respectively established. On the other hand, traditional time-domain and frequency-domain features are extracted from the useful pulses, and picture features are extracted from the corresponding spectrograms to establish a joint feature library. Through feature selection, the features that can reflect the position information and material information are respectively retained to establish a joint positioning feature library and a joint material feature library, thereby establishing a positioning data set and a material data set. Then, on the one hand, multiple neural networks are trained on the positioning picture set and the material picture set respectively, and the optimal one is selected and optimized for hyperparameters to obtain the optimal positioning-neural network and the optimal material-neural network. On the other hand, multiple classifiers are trained on the positioning data set and the material data set respectively, and the optimal one is selected and optimized for parameters to obtain the optimal positioning-classifier and the optimal material-classifier. Finally, combining the optimal positioning-neural network, the optimal positioning-classifier and the triple majority voting process, and combining the optimal material-neural network, the optimal material-classifier and the triple majority voting process, a new combined positioning framework and a combined material framework are respectively constructed, and multiple physical tests are carried out in real application scenarios to verify the feasibility, practicability and stability of the proposed method.

[0084] The core of the present invention lies in the two migrations of the research object. For the sealed electronic device to be tested, the present invention finds a sealed electronic device of the same model as a model, called the device model. Therefore, the first migration of the research object is from the sealed electronic device to be tested to the device model. According to the true internal structure of the device model, which is actually the true internal structure of the sealed electronic device, the internal space of the device model is divided into multiple sealed spaces, and a plurality of thin plates are used to completely separate some originally connected sealed spaces. In this way, the sealed spaces are independent of each other, and there will be no movement of the redundant objects located in a certain sealed space into other sealed spaces. It should be noted that the material of the thin plate is the same as that of the inner wall of the device model, and the thin plate is fixed in the internal space and can be regarded as integrated with the original device model, without causing interference during the experiment. On this basis, redundant objects of different materials are placed in different sealed spaces in turn, and a large number of redundant object signals with different position information and material information are collected by means of a redundant object automatic detection device. Through steps such as pulse preprocessing, feature engineering, model training and optimization, an optimal positioning-neural network, an optimal material-neural network, an optimal positioning-classifier and an optimal material-classifier are obtained respectively.

[0085] The second migration of the research object is from the device model to the sealed electronic device to be tested. A section of redundant object signal to be tested is collected by means of a redundant object automatic detection device, that is, its position information and material information are unknown. Similarly, through steps such as pulse preprocessing and feature engineering, a set of test spectrogram images for both positioning and material identification, that is, a set of test images, a test positioning data set for positioning and a test material data set for material identification are obtained respectively. The optimal positioning-neural network is used to make predictions on the set of test images, and the optimal positioning-classifier is used to make predictions on the test positioning data set to obtain two sets of predicted labels. On this basis, a combined positioning framework is constructed using a triple majority voting process, and a positioning result is obtained. Similarly, the optimal material-neural network is used to make predictions on the set of test images, and the optimal material-classifier is used to make predictions on the test material data set to obtain two sets of predicted labels, thereby constructing a combined material framework and obtaining a material identification result. Finally, the redundant object positioning and material identification results of the sealed electronic device to be tested are obtained. The following will describe the present invention in detail in conjunction with specific embodiments.

[0086] Specific Embodiment 1: In combination with Figure 1 describe this embodiment,

[0087] Step 1: A sealed electronic device of the same model as the sealed electronic device to be tested, called the device model, is used to divide its internal space into multiple independent sealed spaces by using thin plates and number them. According to the redundant object detection information in the actual application scenario, redundant object samples of different materials are made and encoded.

[0088] Place multiple acoustic emission sensors at different positions on the surface of the device model, and the acoustic emission sensors are jointly connected to the foreign object automatic detection device.

[0089] Step 2: Open the device model. According to the space number and material code, place the foreign object samples of the first material into the enclosed space numbered 1, and then re-seal the device model. With the help of the foreign object automatic detection device, collect a group of multi-channel foreign object signals, that is, a group of multi-segment foreign object signals simultaneously collected by multiple acoustic emission sensors. And so on, place the foreign object samples of the first material into other numbered enclosed spaces in turn, and collect multiple groups of multi-channel foreign object signals.

[0090] Then place the foreign object samples of the second material into different numbered enclosed spaces in turn, and collect multiple groups of multi-channel foreign object signals; then place the foreign object samples of the third material into different numbered enclosed spaces in turn, and collect multiple groups of multi-channel foreign object signals; until the foreign object samples of the last material are placed into different numbered enclosed spaces in turn, and collect multiple groups of multi-channel foreign object signals. Obtain a large number of multi-channel foreign object signals and their space numbers and material codes.

[0091] Step 3: Use the two-stage adaptive energy threshold pulse extraction algorithm to extract pulses from each segment of foreign object signal respectively, and obtain multiple useful pulses for each segment of foreign object signal. Use the improved threshold determination search pulse matching algorithm to perform pulse matching on each group of multi-channel foreign object signals respectively, and obtain multiple useful pulse groups for each group of multi-channel foreign object signals. Obtain a large number of useful pulse groups, and multiple useful pulses in each useful pulse group.

[0092] It should be noted that: both pulse extraction and pulse matching are based on each measurement signal. Subsequent steps can be carried out for the results of pulse extraction and matching corresponding to one measurement, or for multiple measurements, pulse extraction and matching can be carried out respectively, and then subsequent steps can be carried out based on all the results.

[0093] Step 4: Taking the useful pulses as the basic unit, convert each useful pulse into a spectrogram, and obtain a spectrogram set for all useful pulses; copy to obtain two spectrogram sets.

[0094] For the first spectrogram set, according to the space number of the foreign object signal to which the useful pulse belongs, add the same space number to the spectrogram corresponding to the useful pulse to obtain a set of positioning pictures with space number labels.

[0095] For the second spectrogram set, according to the material code of the foreign object signal to which the useful pulse belongs, add the same material code to the spectrogram corresponding to the useful pulse to obtain a set of material pictures with material number labels.

[0096] Step 5: Taking the useful pulse groups as the basic units, extract time-domain and frequency-domain features respectively from the multiple useful pulses included in each useful pulse group, and extract gray-level co-occurrence matrix features respectively from the multiple spectrograms corresponding to the multiple useful pulses.

[0097] According to the channel number, that is, the acoustic emission sensor number, concatenate the time-domain features, frequency-domain features, and gray-level co-occurrence matrix features of the multiple useful pulses in sequence to form a feature vector.

[0098] A large number of useful pulse groups construct a large number of feature vectors, obtaining a feature vector set. Copy to get two feature vector sets.

[0099] For the first feature vector set, according to the spatial number of the multi-channel foreign object signal to which the useful pulse group belongs, add the same spatial number to the feature vector corresponding to the useful pulse group to obtain a preliminary positioning data set with spatial number labels.

[0100] For the second feature vector set, according to the material code of the multi-channel foreign object signal to which the useful pulse group belongs, add the same material code to the feature vector corresponding to the useful pulse group to obtain a preliminary material data set with material number labels.

[0101] Use the feature selection method based on channel weighting to process the preliminary positioning data set and the preliminary material data set, respectively screen the features that effectively feedback position information and material information, construct a joint positioning feature library and a joint material feature library, and thus establish a positioning data set and a material data set.

[0102] Step 6: Train multiple neural networks respectively on the positioning picture set and the material picture set, compare to obtain two outstanding neural networks, and optimize their hyperparameters to obtain two optimal neural networks, called the optimal positioning-neural network and the optimal material-neural network.

[0103] Train multiple classifiers respectively on the positioning data set and the material data set, compare to obtain two outstanding classifiers, and optimize their parameters to obtain two optimal classifiers, called the optimal positioning-classifier and the optimal material-classifier.

[0104] So far, the research task of the first research object transfer ends.

[0105] Step Seven: Place multiple acoustic emission sensors at different positions on the surface of the electronic device to be tested for sealing. The acoustic emission sensors are jointly connected to the foreign object automatic detection device. With the help of the foreign object automatic detection device, a set of multi-channel foreign object signals is collected, which is called the foreign object signal group to be tested. Use the two-stage adaptive energy threshold pulse extraction algorithm to extract pulses from each segment of the foreign object signal in the foreign object signal group to be tested, and obtain multiple useful pulses. Use the improved threshold judgment search pulse matching algorithm to perform pulse matching on the foreign object signal group to be tested, and obtain multiple useful pulse groups.

[0106] Step Eight: Convert multiple useful pulses into multiple spectrograms to establish a set of test pictures. Extract the time-domain and frequency-domain features in the joint positioning feature library from each of the multiple useful pulses included in each useful pulse group, and extract the gray-level co-occurrence matrix features in the joint positioning feature library from the multiple spectrograms corresponding to the multiple useful pulses. According to the channel number, concatenate the time-domain features, frequency-domain features, and gray-level co-occurrence matrix features of the multiple useful pulses in sequence to form a feature vector.

[0107] Multiple useful pulse groups construct multiple feature vectors to establish a test positioning data set. Similarly, by extracting the time-domain and frequency-domain features in the joint material feature library from each of the multiple useful pulses included in each useful pulse group, and extracting the gray-level co-occurrence matrix features in the joint material feature library from the multiple spectrograms corresponding to the multiple useful pulses, multiple useful pulse groups can construct multiple feature vectors, and thus a test material data set is established.

[0108] Step Nine: Use the optimal positioning-neural network and the optimal material-neural network to make predictions on the set of test pictures respectively, and obtain multiple predicted labels for positions and materials, which are called the spectrogram-position label set and the spectrogram-material label set.

[0109] Use the optimal positioning-classifier to make predictions on the test positioning data set, and obtain multiple predicted labels for positions, which are called the feature vector-position label set. Use the optimal material-classifier to make predictions on the test material data set, and obtain multiple predicted labels for materials, which are called the feature vector-material label set.

[0110] Step Ten: Conduct majority voting on the spectrogram-position label set and the feature vector-position label set respectively to obtain the common position label 1 and the common position label 2. Merge the spectrogram-position label set and the feature vector-position label set, and conduct majority voting again to obtain the common position label 3. Conduct majority voting on the common position label 1, the common position label 2, and the common position label 3 to obtain the final position label. That is, the combined positioning framework is constructed, and the foreign object positioning is completed.

[0111] Perform majority voting on the spectrogram - material label set and the feature vector - material label set respectively to obtain common material label 1 and common material label 2. Merge the spectrogram - material label set and the feature vector - material label set, and perform majority voting again to obtain common material label 3. Perform majority voting on common material label 1, common material label 2, and common material label 3 to obtain the final material label. That is, the combined material framework is constructed, and the identification of foreign material is completed.

[0112] So far, the research task of the second research object transfer has ended.

[0113] Embodiment

[0114] Sealed electronic device: Select the core component of a certain type of aerospace equipment as the sealed electronic device. Its physical diagram is as Figure 2 shown. Figure 2 In it, (a) is the picture of its fully encapsulated state, and (b) is the picture of its internal space after being opened.

[0115] As can be seen from the figure, the internal space of the sealed electronic device is connected, and there is no independent enclosed space. At the same time, there are three inner walls and seven standing columns in the internal space. Taking the seven columns as a reference and considering the existing structure of the internal space, seven thin plates are used to divide the internal space into seven independent enclosed spaces. Among them, considering that a part of the height of one of the original inner walls is not enough to completely isolate two enclosed spaces, so the present invention uses a thin plate to fill this part of the gap. In order to clearly show the division effect of the internal space of the sealed electronic device, the present invention uses Solidworks to draw a 3D model of the sealed electronic device, as Figure 3 shown in (a). At the same time, Figure 3 the actual effect of the internal space after adding the thin plate is given in (b). On this basis, the seven enclosed spaces are numbered in sequence, and the numbering information refers to Figure 3 in (b). Thus, the device model is obtained for carrying out subsequent experiments.

[0116] Foreign material samples: According to the latest foreign material detection information in the actual application scenario, the inspectors often detect foreign materials made of metal materials, such as iron, copper, tin, and aluminum, from this sealed electronic device. Their weights are all between 0.8mg and 1.0mg. In addition, silicone has also been detected. Although the probability is not high, it is regarded as a foreign material made of non - metal material, and its weight is about 10mg. Therefore, foreign material samples of five materials are made, namely aluminum, tin, iron, copper, and silicone, and their weights are 1.0mg, 1.0mg, 0.8mg, 1.0mg, and 10mg respectively. Figure 4 The physical diagram of the foreign material samples is given.

[0117] Special equipment: The specific name of the special equipment is the DZJC-III type automatic foreign object detection equipment, and its physical object is as shown in Figure 5 . It includes three parts: four acoustic emission sensors, a signal processing unit, and an embedded computer.

[0118] The four acoustic emission sensors are placed at different positions on the surface of the sealed electronic equipment to collect foreign object signals and transmit them to the signal processing unit. The specific model of the acoustic emission sensor is the PXR04 type resonant acoustic emission sensor. Its resonant frequency is 40 kHz, and its frequency bandwidth at a sensitivity of 60 dB is from 15 kHz to 165 kHz. The minimum amplitude of the foreign object signal is 12 mV, and the frequency bandwidth is from 20 kHz to 120 kHz. Therefore, the PXR04 type resonant acoustic emission sensor can sensitively collect foreign object signals.

[0119] The signal processing unit includes a signal conditioning circuit, a signal acquisition circuit, a signal transmission circuit, etc. The signal conditioning circuit is used to complete the signal amplification and signal filtering of the four-channel foreign object signals collected once. The signal acquisition circuit is used to complete the conversion of the four-channel foreign object signals from analog signals to digital signals. The signal transmission circuit is used to transmit the four-channel foreign object signals in digital format to the embedded computer.

[0120] The embedded computer is used to run and display the signal acquisition software. During the experiment, the tester uses this software to control the special equipment to collect the four-channel foreign object signals, save the four-channel foreign object signals sent by the signal processing unit in the local memory, and display a set of currently collected four-channel foreign object signals on the main interface.

[0121] Experimental conditions: In this embodiment, according to the requirements for carrying out equipment-level foreign object detection stipulated in the Chinese GJB65B standard, the corresponding experimental conditions are set, as shown in Table 1.

[0122] Table 1 Experimental conditions stipulated by GJB65B

[0123]

[0124] Signal acquisition: Referring to Step 2 of the specific implementation method, foreign object samples of five materials are successively placed into the seven sealed spaces of the equipment model. After each placement, the equipment model is re-sealed and fixed to the vibration table. Referring to Table 1, the experimental conditions of the vibration table are set to make it work to activate the foreign object signals inside the equipment model and generate foreign object signals. The four acoustic emission sensors of the special equipment are placed at different positions on the surface of the equipment model to collect the four-channel foreign object signals, and finally saved in the local memory of the embedded computer. Figure 6A physical diagram of the process of collecting debris signals is given. It should be noted that the positions of four acoustic emission sensors are shown in the figure, marked by the red circular frames. Such a sensor layout can ensure sufficient collection of debris signals generated in any enclosed space. In fact, a suitable sensor layout can be designed according to the actual experimental object.

[0125] After a large number of balanced experiments, 100 sets of debris samples representing each material are collected respectively, and the four-channel debris signals in each enclosed space are collected, a total of 100×5×7 = 3500 sets of four-channel debris signals. The position information and material information of the debris samples are marked for each set of four-channel debris signals.

[0126] Pulse preprocessing: Based on a large number of four-channel debris signals, pulse preprocessing is performed on them respectively, including pulse extraction and pulse matching. Among them, a newly proposed two-stage adaptive energy threshold pulse extraction algorithm is used to extract the useful pulses in each segment of debris signal, and the threshold determination search pulse matching algorithm is improved to match the useful pulses that are temporally corresponding and relevant in each set of four-channel debris signals.

[0127] S31. Two-stage adaptive energy threshold pulse extraction algorithm:

[0128] According to the 3σ distribution criterion of the normal distribution, assume that the mean of a batch of numerical values is μ and the standard deviation is σ. Then, the probability that the numerical values are distributed in [μ - σ, μ + σ] is 68.27%, and the probability that the numerical values are distributed in [μ - 3σ, μ + 3σ] is 99.73%. Scholars usually identify the numerical values lower than μ - 3σ or higher than μ + 3σ as outliers, where μ + 3σ is an abnormally large value, and identify the numerical values hovering around μ - σ and μ + σ as valid values. Similarly, assume that the average energy of a segment of signal is E a , and the standard deviation of the energy is σ E . Those signal components with energy higher than E a + 3σ E are the abnormally large values of the entire segment of signal, that is, the peak region of the useful pulse. Those signal components with energy hovering around E a + σ E are the valid values of the entire segment of signal, that is, the endpoint region of the useful pulse. It should be noted that in the present invention, E a + σ E is selected instead of E a - σ E following the idea of being less but better, that is, the signal components with energy hovering around E a + σ E belong to the useful pulse with a higher credibility, and the signal components with energy hovering around E a - σ E may contain some background noise. Figure 7The schematic diagram of the two - level adaptive energy threshold pulse extraction algorithm is given, and its specific implementation steps are as follows:

[0129] Step 1: Calculate the average energy of the signal, that is, the average of the sum of the squares of the amplitudes of all sampling points it contains, denoted as e a . Calculate the standard deviation of the energy of the signal, denoted as σ E . Set the peak energy threshold as E p = E a + 3σ E , and set the endpoint energy threshold as e be = e a + σE.

[0130] Step 2: Perform frame processing on the signal. Set the duration of each frame signal as Δt = 50 μs, and thus the signal is divided into multiple frame signals. Calculate the energy of each frame signal.

[0131] The sampling frequency of the special device used in the present invention is 500 kHz, and each frame signal contains twenty - five sampling points.

[0132] Step 3: Starting from the first frame signal, compare the energy of each frame signal with E p in turn. When the energy of a certain frame signal is greater than E p , it indicates that the maximum frame signal of the first useful pulse is about to appear. Record this frame signal; and starting from it, continue to compare the energy of each subsequent frame signal with E p , until the energy of a certain frame signal is less than E p . Find the frame signal with the maximum energy among these frame signals, and identify it as the maximum frame signal of the first useful pulse. Obtain the time of the sampling point with the maximum amplitude in the maximum frame signal, which is called the peak moment t p_1 of the first useful pulse.

[0133] Step 4: Taking the peak moment t p_1 of the first useful pulse extracted in Step 3 as the starting point, compare the energy of each frame signal with E be forward and backward respectively, until it is found that the energy of a certain frame signal is less than E be in both directions. Respectively identify the previous frame signal of these two frame signals as the first starting frame signal and the first ending frame signal of the first useful pulse. Obtain the time of the first sampling point in the first starting frame signal, which is called the first starting moment t fb_1 of the first useful pulse. Obtain the time of the last sampling point in the first ending frame signal, which is called the first ending moment t fe_1 of the first useful pulse.

[0134] Step 5: Taking the first ending moment tfe_1 Using the next frame signal as a new starting point, repeat the same processing procedures of Step 3 and Step 4 to extract the remaining useful pulses in the signal, and then save all the useful pulses extracted from the signal. Meanwhile, save the peak moments of all the useful pulses. The first starting moment and the first ending moment

[0135] Thus, the pulse extraction in the first stage is completed.

[0136] Step 6: Calculate the average energy and the standard deviation of the energy of each useful pulse saved in Step 5, denoted as E i (i = 1, 2, …, n) and σ E_i (i = 1, 2, …, n). On this basis, set the exclusive endpoint energy threshold for each useful pulse as E be_i = E i + σ z_i (i = 1, 2, …, n).

[0137] Step 7: Starting from the first useful pulse saved in Step 5, using its peak moment t p_1 extracted in Step 3 as the starting point, compare the energy of each frame signal with the exclusive endpoint energy threshold E be_1 respectively forward and backward until it is found that the energy of a certain frame signal is less than E be_1 in both directions. Identify the previous frame signals of these two frame signals as the second starting frame signal and the second ending frame signal of the first useful pulse respectively.

[0138] Obtain the time of the first sampling point in the second starting frame signal, which is called the second starting moment t sb_1 of the first useful pulse. Obtain the time of the last sampling point in the second ending frame signal, which is called the second ending moment t se_1 of the first useful pulse.

[0139] Step 8: Starting from the second useful pulse saved in Step 5, repeat the same processing steps of Step 7 to obtain the second starting moments and the second ending moments

[0140] of the remaining useful pulses saved in Step 5 respectively. fl_i = t fe_i - t fb_i (i = 1, 2, …, n), the second duration t sl_i = t se_i - t sb_i (i = 1, 2, …, n), and the duration ratio If the useful pulse τ i If it is greater than the duration threshold T, then its first starting time t fb_i With the first end time t fe_i Determined as the final starting time t b_i and the final ending time t e_i Otherwise, set its first starting time t fb_i and the second end time t se_i Determined as the final starting time t b_i and the final ending time t e_i Thus, the final starting time t of each useful pulse saved in step 5 is obtained. b_i (i=1,2,…,n) and the final end time t e_i (i=1,2,…,n) and record it.

[0141] This concludes the second phase of pulse extraction. Using a two-stage adaptive energy threshold pulse extraction algorithm, we processed each of the 3,500 four-channel redundant signals (4 × 3,500 = 14,000 redundant signals) to extract the useful pulses from each segment of the redundant signal, yielding a total of 1,386,784 useful pulses.

[0142] It should be noted that in the embodiment, the duration threshold T is set to 0.82. Referring to the 3σ distribution criterion of the normal distribution, the probability that the value is distributed within [μ-σ, μ+σ] is 68.27%, and the probability that the value hovers around μ+σ is about 84.13%. Similarly, for independent useful pulses, using the exclusive endpoint energy threshold E be_i The extracted signal component should also account for approximately 84.13% of the useful pulses. For a large pulse composed of multiple consecutive useful pulses, its distribution is a mixture of the normal distributions of multiple independent useful pulses. Therefore, the larger useful pulses dominate the distribution, resulting in a higher exclusive endpoint energy threshold than the one applicable to the smaller useful pulses. This effectively discards the smaller useful pulses in the continuous multi-pulse stack, retaining only the largest useful pulses, thereby resolving the continuous multi-pulse problem. Therefore, the duration threshold should be set slightly less than 84.13%, ensuring sufficient capacity to resolve the continuous multi-pulse problem while retaining a reasonable amount of flexibility to allow for some degradation in actual use.

[0143] S32, improved threshold judgment search pulse matching algorithm:

[0144] After the pulse extraction is completed, a pulse matching algorithm is needed to find four useful pulses that are temporally corresponding in the four-channel redundant signal. Every four useful pulses form a useful pulse group, and multiple useful pulse groups can be obtained from a group of four-channel redundant signals. This is the basis of feature engineering. In previous research, a threshold determination search pulse matching algorithm was proposed, which uses the start time of the first useful pulse in each of the four-channel redundant signals, that is, the four segments of redundant signals, as a reference. In actual use, the redundant signals collected by the acoustic emission sensors closer to the redundant object are richer, while the redundant signals collected by the acoustic emission sensors farther from the redundant object are relatively sparser. If the amplitude of the first useful pulse is small, then in the redundant signals collected by the acoustic emission sensors closer to the redundant object, the first useful pulse can still be distinguished from the background noise and can be extracted by the two-stage adaptive energy threshold pulse extraction algorithm. However, in the redundant signals collected by the acoustic emission sensors farther from the redundant object, the first useful pulse is almost submerged in the background noise and cannot be extracted by the two-stage adaptive energy threshold pulse extraction algorithm. This results in the first useful pulse extracted from the redundant signals collected by the acoustic emission sensors farther from the redundant object not being the true first useful pulse, thus leading to inaccurate pulse matching. In the present invention, the threshold determination search pulse matching algorithm is improved, and the useful pulses for reference are adjusted. Figure 8 The schematic diagram of the improved threshold determination search pulse matching algorithm is given, and the specific implementation steps are as follows.

[0145] Step 1: Respectively obtain the useful pulses with the maximum energy in the four-channel redundant signals, that is, the four segments of redundant signals, which are called four reference useful pulses. According to the start times of the four reference useful pulses, determine the redundant signal collected by the acoustic emission sensor farthest from the redundant object, which is called the reference signal. Determine the redundant signal collected by the acoustic emission sensor closest to the redundant object, which is called the comparison signal. Obtain the start times of the reference signal and the comparison signal, and calculate the difference between the two start times, which is called the start time threshold T.

[0146] Step 2: Represent the useful pulses included in the reference signal in Step 1 as P i (i = 1, 2,..., n), and the start times of these useful pulses are t i (i = 1, 2,..., n). According to the start time threshold T, set the start time threshold interval for each useful pulse as [t i -T, t i (i = 1, 2,..., n).

[0147] Step 3: Starting from P1, search for useful pulses in the other three segments of the redundant signal whose starting moments are within [t1 - T, t1]. If three useful pulses can be found, form a pulse group with them and P1. Otherwise, skip P1, indicating that the matching is unsuccessful.

[0148] Step 4: Referring to Step 3, perform pulse matching on P i (i = 2, 3, …, n). Thus, the pulse matching of the four-channel redundant signal is completed, and multiple pulse groups are obtained.

[0149] The present invention uses an improved threshold determination search pulse matching algorithm to process 3500 groups of four-channel redundant signals respectively, match the useful pulses that are temporally corresponding and relevant in each group of four-channel redundant signals, obtain the useful pulse groups in each group of four-channel redundant signals, and thus obtain 287692 useful pulse groups, a total of 1150768 useful pulses.

[0150] Feature engineering: Each useful pulse group contains four useful pulses. A large number of useful pulse groups means a larger number of useful pulses. The useful pulses can be transformed into spectrograms to construct a picture set, and then a neural network can be trained. The useful pulses can also extract classical time-domain and frequency-domain features to construct feature data, and then a classifier can be trained. At the same time, the spectrograms can also extract picture features to be merged with the time-domain and frequency-domain features to construct feature data. In addition, considering the correlation existing between the four-channel redundant signals, which is called the four-channel characteristic, it is necessary to extract the above features simultaneously on the four useful pulses of the useful pulse group to construct feature data. The 3500 groups of four-channel redundant signals are simultaneously labeled with the position information and material information of the redundant sample. In this way, a large number of useful pulses and useful pulse groups also carry position information and material information. The neural network can feedback to the feature extractor according to the output situation to adaptively extract or process the feature data from the pictures and construct a perfect network structure. Therefore, the spectrograms can be input into the neural network without adjustment, and the neural network can adaptively adjust its own network structure according to the positioning or material recognition task. However, the composition structure of the feature data has a great impact on the classifier. It is necessary to determine the feature data with appropriate composition structures respectively to ensure that the trained classifier can better complete the positioning or material recognition task. This requires carrying out feature selection to construct a positioning data set and a material data set dedicated to positioning and material recognition respectively.

[0151] Spectrogram: A spectrogram is a two-dimensional image converted from a one-dimensional signal to better display the time-domain and frequency-domain characteristics of the signal. When constructing a spectrogram, the signal is first pre-emphasized, then windowed, then the windowed signal is subjected to a fast Fourier transform (FFT), and finally the signal is converted into a spectrogram. Considering that in the present invention, useful pulses are converted into spectrograms, the length of the FFT is set to 1024, the window function is set to a Hamming window, and the window length is set to 50 ms. The construction details of the spectrogram are as follows.

[0152] First, a high-pass filter is used to pre-emphasize the useful pulse, and its principle is shown in formula (1):

[0153] F(x) = 1 - μx -1 (1)

[0154] Where μ is the pre-emphasis coefficient, and usually μ = 0.97 is set.

[0155] Secondly, the pre-emphasized useful pulse is windowed to make it continuous, and its principle is shown in formula (2):

[0156] S w (n) = S(n) × w(n) (2)

[0157] Where S(n) is the original useful pulse, w(n) is the window function, and S w (n) is the windowed useful pulse. The calculation formula for the Hamming window is:

[0158]

[0159] Then, the windowed useful pulse is subjected to FFT, and then the power spectrum is passed through a band-pass filter to obtain the spectrogram corresponding to the useful pulse.

[0160] Figure 9 The spectrogram constructed from a certain useful pulse is given. As shown in the figure, the spectrogram contains multiple block regions, and the color shades of each block region are different. And among most of the multiple continuous block regions in the figure, the color shades are not very different and are relatively close. There are also some abrupt block regions showing particularly dark or particularly light colors, which are all related to the time-domain and frequency-domain characteristics of the redundant signal and need to be further studied.

[0161] Previously, 287,692 useful pulse groups were obtained, with a total of 1,150,768 useful pulses. From this, the useful pulses were converted into spectrograms, and a total of 1,150,768 spectrograms were obtained to construct an image set. Two image sets were copied. On this basis, according to the position information of the useful pulses corresponding to the spectrograms in the foreign object signals, position information, that is, position labels, was added to the spectrograms in one of the image sets. Thus, the 1,150,768 spectrograms were divided into seven categories to construct a positioning image set. According to the material information of the useful pulses corresponding to the spectrograms in the foreign object signals, material information, that is, material labels, was added to the spectrograms in the other image set. Thus, the 1,150,768 spectrograms were divided into five categories to construct a material image set.

[0162] Combined Feature Library: In addition to converting useful pulses into spectrograms to construct an image set, features can also be extracted from the useful pulses to construct a data set. In previous studies, eleven time-domain and frequency-domain features were extracted, as shown in Table 2.

[0163] Table 2 Specific Descriptions of Eleven Time-Domain and Frequency-Domain Features

[0164]

[0165] In addition, the present invention also extracts image features from the spectrograms converted from useful pulses, mainly gray-level co-occurrence matrix features. The gray-level co-occurrence matrix is a feature extraction technique widely used in image texture analysis. By analyzing the frequency and spatial relationships between pixels of different gray levels in an image, it effectively captures the texture information of the image. Therefore, the gray-level co-occurrence matrix can effectively analyze the texture differences between block regions in the constructed spectrograms. The selected gray-level co-occurrence matrix features include energy, entropy, contrast, mean, homogeneity, and correlation, and their specific descriptions are shown in Table 3. In addition, the explanations of the variables shown in the table are given. P(i,j) is the element at (i,j) in the normalized co-occurrence matrix. N is the number of gray levels in the image. Among them, i and j represent different pixels respectively. μ x and μ y respectively represent the means of the pixel gray values i and j in the image. σ x and σ y respectively represent the standard deviations of the pixel gray values i and j in the image.

[0166] Table 3 Specific Descriptions of Gray-Level Co-Occurrence Matrix Features

[0167]

[0168] Therefore, a combined feature library is constructed, including eleven time-domain and frequency-domain features and six gray-level co-occurrence matrix features. Previously, 287,692 useful pulse groups were obtained, with a total of 1,150,768 useful pulses. 1,150,768 spectrograms corresponding to the useful pulses were obtained previously. Eleven time-domain and frequency-domain features are extracted from the four useful pulses included in each useful pulse group, and six gray-level co-occurrence matrix features are extracted from the corresponding four spectrograms, resulting in a total of (11 + 6) × 4 = 68 feature values. Considering the four-channel characteristics existing among the four-channel debris signals, these feature values are arranged in the order of "the time-domain features, frequency-domain features, and gray-level co-occurrence matrix features of the useful pulses in the first channel, the time-domain features, frequency-domain features, and gray-level co-occurrence matrix features of the useful pulses in the second channel, the time-domain features, frequency-domain features, and gray-level co-occurrence matrix features of the useful pulses in the third channel, and the time-domain features, frequency-domain features, and gray-level co-occurrence matrix features of the useful pulses in the fourth channel" to obtain a feature vector. The channel numbers mentioned here are the numbers of the four acoustic emission sensors of the special equipment, that is, the debris signal collected by the No. 1 acoustic emission sensor is the debris signal in the first channel of the four-channel debris signal, and the useful pulses it contains are the useful pulses in the first channel.

[0169] Through the same processing steps, 287,692 feature vectors are obtained based on 287,692 useful pulse groups and 1,150,768 spectrograms, and a dataset is constructed. Two datasets are obtained by replication. On this basis, according to the position information of the useful pulse groups in the debris signals, position information, that is, position labels, is added to the feature vectors in one of the datasets, and thus the 287,692 feature vectors are divided into seven categories to construct a preliminary positioning dataset. According to the material information of the useful pulse groups in the debris signals, material information, that is, material labels, is added to the feature vectors in the other dataset, and thus the 287,692 feature vectors are divided into five categories to construct a preliminary material dataset.

[0170] Feature Selection: In machine learning, some feature data in a dataset do not contribute much to improving the classification performance of a classifier and may even have a negative impact. As a result, the classification accuracy achieved by the classifier on the dataset containing all feature data is lower than that on the dataset containing only effective feature data. Feature selection is the process of screening out those feature data that contribute significantly to the classification performance of the classifier from the dataset. In the present invention, the feature data of the preliminary positioning dataset and the preliminary material dataset are the same, but the labels of the feature vectors are different, which leads to different classification tasks and different classification performances of the classifiers trained on the two datasets. Therefore, it is necessary to perform feature selection on the preliminary positioning dataset and the preliminary material dataset respectively, screen out those features that contribute significantly to the classification performance of the classifier, construct a joint positioning feature library dedicated to foreign object localization and a joint material feature library dedicated to foreign object material identification, and thus construct a positioning dataset and a material dataset.

[0171] Considering the four-channel characteristics among the four-channel foreign object signals, the 68 features cannot be selected as individuals, but should be regarded as 17 combinations of similar features for selection. In previous research, a feature selection method based on channel weighting was proposed, which is dedicated to the feature selection of four-channel foreign object signals. The specific implementation steps are as follows.

[0172] Step 1: Calculate the absolute value of the Pearson correlation coefficient between each feature and the label in the dataset using formula (4), and denote it as r ij (i = 1, 2, 3, 4, j = 1, 2, …, 17). i represents the channel to which the feature belongs, and j represents the serial number of the feature.

[0173]

[0174] Among them, and σX respectively represent the standard score, mean, and standard deviation of feature X i′ ; Y i′ refers to the label value of the sample, and σY respectively represent the standard score, mean, and standard deviation of label Y i′ .

[0175] Step 2: Take the channel as the division unit, that is, take i in r ij as the division unit, and accumulate the absolute values r ij of the Pearson correlation coefficients between the 17 features belonging to the same channel and the label, and obtain four summation results corresponding to the four channels, denoted as R i (i = 1, 2, 3, 4). The calculation formula is as follows.

[0176]

[0177] Among them, r ij represents the absolute value of the Pearson correlation coefficient between the feature with serial number j in channel i and the label.

[0178] Step 3: Calculate the sum of the absolute values of the Pearson correlation coefficients between 68 features and the label, denoted as R all . Calculate R i (i = 1, 2, 3, 4) accounting for the proportion of R all is called the channel weighting coefficient, denoted as W i p (i = ①, ②, ③, ④), and its calculation formula is as follows.

[0179]

[0180] Among them, R i represents the sum of the absolute values of the Pearson correlation coefficients between 17 features in channel i and the label.

[0181] Step 4: According to i in r ij (i = 1, 2, 3, 4, j = 1, 2, …, 17), that is, representing the channel to which it belongs, multiply it by the corresponding channel weighting coefficient W i p (i = 1, 2, 3, 4). For example, r 1j (j = 1, 2, …, 17) are 17 features belonging to the first channel, multiply them by the channel weighting coefficient W1 of the first channel p . In this way, 68 new weighted values are obtained, denoted as and its calculation formula is as follows.

[0182]

[0183] Among them, r ij represents the absolute value of the Pearson correlation coefficient between the feature with serial number j in channel i and the label, and W i p represents the channel weighting coefficient corresponding to channel i.

[0184] Step 5: Taking the combination of features of the same type as the division unit, divide 68 features into 17 combinations of features of the same type. On this basis, accumulate the new values of the four features in the combination of features of the same type to obtain the sum of the new values of 17 combinations of features of the same type, denoted as and its calculation formula is as follows.

[0185]

[0186] Among them, It should be noted that the content you provided contains some unclear or incorrect expressions (such as "①, ②, ③, ④" which should probably be "1, 2, 3, 4" in the correct context). The above translation is based on the existing text as accurately as possible.A new value representing the absolute value of the Pearson correlation coefficient between the feature with sequence number j in the weighted channel i and the label, W i p Represents the channel weighting coefficient corresponding to channel i.

[0187] Step Six: Take Arrange them in descending order to obtain The corresponding sorting number N j (j = 1, 2, …, 17).

[0188] Step Seven: Retain the top 8 to top 17 similar feature combinations in the sorting number N j respectively, and construct 10 new datasets that retain different features. Use a random forest with default parameter configurations to perform ten-fold cross-validation on the 10 new datasets respectively, and obtain 10 average classification accuracies. Compare to find out on which new dataset the random forest achieves the highest average classification accuracy. Then the features that construct this new dataset are the best, and the corresponding feature selection result is the optimal one.

[0189] Use the feature selection method based on channel weighting to process the preliminary location dataset and the preliminary material dataset respectively, and thus construct a joint location feature library and a joint material feature library, which contain 15 features and 14 features respectively. Their detailed descriptions are shown in Tables 4 and 5.

[0190] Table 4 Joint Location Feature Library

[0191]

[0192] Table 5 Joint Material Feature Library

[0193]

[0194] According to the joint location feature library, retain 15 similar feature combinations in the preliminary location dataset, that is, all the feature values of 60 features, and remove all the feature values of the redundant 8 features to construct a location dataset. Similarly, according to the joint material feature library, retain 14 similar feature combinations in the preliminary material dataset, that is, all the feature values of 56 features, and remove all the feature values of the redundant 12 features to construct a material dataset.

[0195] So far, construct a location picture set, a location dataset, a material picture set, and a material dataset. Their specific descriptions are shown in Tables 6 and 7 respectively.

[0196] Table 6 Specific Descriptions of the Location Picture Set and the Location Dataset

[0197]

[0198]

[0199] Specific Descriptions of the Material Picture Set and the Material Data Set in Table 7

[0200]

[0201] Neural Network Training: The present invention selects five currently popular neural networks that are widely used by scholars in image classification, namely AlexNet, ResNet50, GoogLeNet, VGG19, and MobileNet. They cover different network scales and complexities and are widely representative. Table 8 lists the network structures of the five neural networks.

[0202] Network Structures of the Neural Networks in Table 8

[0203]

[0204] Train the above five neural networks on the localization picture set and the material picture set respectively, set 150 epochs, and count their average classification accuracies, as shown in Table 9.

[0205] Average Classification Accuracies Obtained by the Neural Networks in Table 9

[0206]

[0207] It can be seen from the table that whether on the localization picture set or the material picture set, VGG19 achieves the highest average classification accuracies of 95.16% and 97.56% respectively, and has obvious advantages compared with other neural networks. In addition, AlexNet, ResNet50, and GoogLeNet also perform excellently, indicating that they do have advantages in the field of image classification. In contrast, MobileNet achieves the lowest average classification accuracy, which may be due to its lightweight structure that focuses on reducing its own network structure and computational complexity without excessive pursuit of classification performance. Figure 10 and Figure 11 respectively give the curves of the classification accuracy and Loss obtained by VGG19 on the localization picture set and the material picture set after 150 epochs. Figure 10 Shows the classification effect obtained by VGG19 on the localization picture set, where (a) is the classification accuracy curve and (b) is the Loss curve. Figure 11 Shows the classification effect obtained by VGG19 on the material picture set, where (a) is the classification accuracy curve and (b) is the Loss curve.

[0208] As can be seen from the figure, as the number of epochs increases, the gap between the classification accuracies achieved by VGG19 on the training set and the validation set of the image set is not obvious and almost overlaps, but the gap between the achieved Losses increases and is relatively obvious, and even shows an oscillation phenomenon. Similarly, as the number of epochs increases, the gaps between the classification accuracies and Losses achieved by VGG19 on the training set and the validation set of the material image set increase. Generally speaking, however, the gap between the classification accuracies is not large, and the gap between the Losses is relatively obvious. This proves that the classification performance and generalization ability of VGG19 still need to be further improved. Therefore, further in-depth analysis of the internal structure of VGG19 is carried out, and hyperparameter optimization is attempted to further improve its classification performance and generalization ability.

[0209] Hyperparameter Optimization: Through in-depth analysis of the internal structure of VGG19, it is found that its differences from other neural networks are mainly reflected in the repeated use of multiple small 3×3 convolutional kernels. At the same time, VGG19 uses the ReLU activation function to increase its non-linear processing ability. In addition, compared with the previous version VGG16, VGG19 has increased the number of convolutional layers, enabling it to capture more complex features. The present invention considers exploring from two perspectives how the internal structure of VGG19 provides classification advantages when processing spectrograms. First, the effect of repeatedly using multiple small 3×3 convolutional kernels versus directly using large convolutional kernels in enhancing VGG19's ability to capture details will be evaluated. Second, the role of the ReLU activation function will be explored in depth. Through the analysis from these two perspectives, the network structure that plays a positive role in the classification performance of VGG19 is retained, and the network structure with room for improvement is optimized.

[0210] In the image classification task, the configuration of the convolutional layer plays a key role in the learning ability and generalization ability of the neural network. VGG19 has achieved good results in processing visual tasks with its structure of deeply stacked convolutional layers by repeatedly using multiple small 3×3 convolutional kernels. In addition, other models such as AlexNet and ResNet adopted larger-sized convolutional kernels (11×11 and 7×7 respectively) at the initial stage of the network. Such a design can not only capture more extensive information in the input image but also achieve a larger receptive field with fewer layers, helping the neural network learn richer feature representations at an early stage. Based on this, the first part of the adjustment to VGG19 is attempted. The first-layer convolutional kernel of VGG19 is modified from 3×3 to 7×7, attempting to enable it to learn more features at an early stage and further improve its understanding and processing ability of spectrograms. The optimized VGG19 is called VGG19-Plus, and its internal structure is as Figure 12As shown in the figure, the newly designed PReLU-VGG19-Plus model of the present invention improves the classification performance and generalization ability of the traditional VGG19. The first-layer convolutional kernel of VGG19 is modified from 3×3 to 7×7, and the PReLU activation function is introduced to replace the original ReLU activation function, effectively improving the neural network's ability to understand and process spectrograms, as well as its learning ability of physical structures.

[0211] As Figure 10 and Figure 11 shown, there is an underfitting phenomenon when VGG19 classifies the localization picture set and the material picture set. Specifically, in the later stage of epoch iteration, the classification accuracy obtained by VGG19 on the validation set is continuously lower than that on the training set, and the Loss obtained on the training set and the validation set even shows an oscillating phenomenon. This problem partly stems from the fact that the four-channel foreign matter signals collected by dedicated equipment contain a large number of negative signals, and finally are submerged in the useful pulses. These negative signals also exist in the corresponding spectrograms. The ReLU activation function outputs zero when processing negative inputs, resulting in the permanent inactivation of some neurons during the training process, thus limiting the neural network's learning ability for new data and further affecting its generalization ability. To solve this problem, the PReLU activation function is newly introduced to replace the ReLU activation function. The re-optimized VGG19 is called PReLU-VGG19-Plus. Figure 13 The images of the PreLU activation function and the ReLU activation function are given. It can be seen from the figure that the PReLU activation function introduces a small, learnable gradient parameter α when processing negative inputs, maintaining the flow of information in the neural network, thus helping to improve its learning ability and generalization ability.

[0212] On this basis, PReLU-VGG19-Plus is trained again on the localization picture set and the material picture set respectively, with 150 epochs set as well, and the curves of the classification accuracy and Loss obtained are respectively counted, as Figure 14 and Figure 15 shown. Figure 14 Fig. shows the classification effect obtained by PReLU-VGG19-Plus on the localization picture set, where (a) is the classification accuracy curve and (b) is the Loss curve. Figure 15 Fig. shows the classification effect obtained by PReLU-VGG19-Plus on the material picture set, where (a) is the classification accuracy curve and (b) is the Loss curve.

[0213] Comparing Figure 14 and Figure 10 Comparing Figure 15 and Figure 11, it can be found that, compared with VGG19, as the number of epochs increases, the differences between the classification accuracy and Loss obtained by PReLU-VGG19-Plus on the training set and validation set of the localization image set are not obvious. In particular, the classification accuracy curves almost overlap, and the oscillation phenomenon of the Loss curve is suppressed, looking relatively stable overall. Similarly, as the number of epochs increases, the differences between the classification accuracy and Loss obtained by PReLU-VGG19-Plus on the training set and validation set of the material image set are not obvious. The classification accuracy curves show almost no difference, and the Loss curve looks relatively stable overall. Generally speaking, compared with VGG19, the classification performance and generalization ability of PReLU-VGG19-Plus are significantly improved. This proves the feasibility and effectiveness of hyperparameter optimization for VGG19. On this basis, two PReLU-VGG19-Plus models trained by the localization image set and the material image set are obtained respectively, called the optimal localization-neural network and the optimal material-neural network.

[0214] Classifier training: Four classifiers that performed well in previous studies were selected, including the radial basis function-based support vector machine (RBF-SVM), random forest, XGBoost, and one-dimensional convolutional neural network (1D-CNN). Ten-fold cross-validation was performed on the localization dataset and the material dataset respectively, and their average classification accuracies were statistically calculated, as shown in Table 10.

[0215] Table 10 Average classification accuracies obtained by classifiers

[0216]

[0217] It can be seen from the table that whether on the localization dataset or the material dataset, XGBoost achieved the highest average classification accuracy, which were 96.92% and 97.46% respectively. In addition, the random forest achieved a close average classification accuracy with a small gap. This shows the significant advantages of the ensemble classifier. In fact, the random forest and XGBoost also performed outstandingly in previous studies. RBF-SVM achieved good classification results. Although there is still a certain gap from XGBoost and the random forest, it is significantly better than 1D-CNN. 1D-CNN is a classic convolutional neural network for data classification. Its classification performance is related to the learning rate set in the initial stage and the gradient update strategy, and it is easy to fall into the trap of local optimum. Although it achieved the lowest average classification accuracy, it was still higher than 90%, both higher than the average classification accuracy of MobileNet in Table 9, showing its unique advantages in data classification.

[0218] Overall, XGBoost performs excellently on both the positioning dataset and the material dataset. Parameter optimization is carried out separately to further improve its classification performance.

[0219] Parameter optimization: The simulated annealing algorithm is used to optimize the parameters of XGBoost, and the values of each parameter when it performs optimally on the positioning dataset and the material dataset are obtained, which are called the optimal values of the parameters. Specifically, the parameters of XGBoost mainly include three parts: general parameters, model parameters, and learning objective parameters. Among them, the general parameters are not optimized and are set to their default configurations. For the model parameters, mainly n_estimatores, max_depth, and min_child_weight are optimized, and the remaining model parameters are set to their default configurations. For the learning objective parameters, mainly learning_rate and gamma are optimized, and the remaining learning objective parameters are set according to the actual situation. For example, the Obj objective function is set to multi, that is, it is applicable to multi-classification problems. On this basis, the simulated annealing algorithm is used to optimize the above 5 parameters on the positioning dataset and the material dataset respectively to find their optimal values. The parameter optimization results are shown in Table 11.

[0220] Table 11 Optimal values of the parameters of XGBoost

[0221]

[0222] On this basis, parameter-optimized XGBoost is applied again to perform ten-fold cross-validation on the positioning dataset and the material dataset respectively, and the average classification accuracies obtained are statistically analyzed and compared with the classification accuracies obtained by XGBoost before parameter optimization, as shown in Table 12.

[0223] Table 12 Average classification accuracies obtained by XGBoost before and after parameter optimization

[0224]

[0225] It can be seen from the table that the average classification accuracies obtained by parameter-optimized XGBoost on the positioning dataset and the material dataset have both increased, by 0.41% and 0.25% respectively. This proves the feasibility and effectiveness of carrying out parameter optimization on XGBoost. On this basis, two parameter-optimized XGBoosts trained on the positioning dataset and the material dataset are obtained respectively, which are called the optimal positioning-classifier and the optimal material-classifier.

[0226] Combined framework construction: In machine learning, neural networks are used to predict the labels of one or a batch of images, and classifiers are used to predict the labels of one or a batch of feature data. In the present invention, for a group of four-channel foreign object signals collected once, it is desired to obtain the position information and material information of the foreign objects from them, and then complete foreign object positioning and material identification. However, as mentioned above, for a group of four-channel foreign object signals collected once, a two-stage adaptive energy threshold pulse extraction algorithm is used to extract multiple useful pulses among them, and an improved threshold determination search pulse matching algorithm is used to match the useful pulses that are temporally corresponding and relevant, obtaining multiple groups of useful pulses. On this basis, multiple spectrograms are constructed, and the feature data of multiple source joint positioning feature libraries and joint material feature libraries are respectively constructed. The optimal positioning-neural network is used to predict multiple spectrograms, obtaining multiple predicted labels. The optimal positioning-classifier is used to predict the feature data of multiple source joint positioning feature libraries, obtaining multiple predicted labels. The optimal material-neural network is used to predict multiple spectrograms, obtaining multiple predicted labels. The optimal material-classifier is used to predict the feature data of multiple source joint material feature libraries, obtaining multiple predicted labels. It can be found that no matter which neural network or classifier it is, what they finally give are multiple predicted labels, rather than a single prediction result. These predicted labels are essentially data or image classification results, and are not the foreign object positioning or material identification results required by the actual application scenario. There is a lack of a data processing process between the two.

[0227] In the present invention, the optimal positioning-neural network and the optimal positioning-classifier are trained to predict spectrograms and feature data respectively, obtaining multiple predicted labels, which are called the spectrogram-position label set and the feature vector-position label set. Similarly, the optimal material-neural network and the optimal material-classifier are trained to predict spectrograms and feature data respectively, obtaining multiple predicted labels, which are called the spectrogram-material label set and the feature vector-material label set. If a majority vote is respectively carried out on the spectrogram-position label set, the feature vector-position label set, the spectrogram-material label set, and the feature vector-material label set, two common position labels and two common material labels will be respectively obtained. At this time, it is not suitable to use the majority vote anymore because the number of inputs for the majority vote is an even number.

[0228] Therefore, on the basis of obtaining two common location tags and two common material tags, the spectrogram-location tag set and the feature vector-location tag set are merged, and majority voting is carried out again. When using the tags for voting again, the spectrogram-location tag set and the feature vector-location tag set can vote according to weights of 0.8 and 0.2 to obtain a third common location tag. The spectrogram-material tag set and the feature vector-material tag set are merged, and majority voting is carried out again to obtain a third common material tag. On this basis, majority voting is carried out on the three common location tags and the three common material tags respectively to obtain the final location tag and the final material tag, thereby constructing a combined positioning framework and a combined material framework, and obtaining the foreign object location and material identification results. In this way, the conversion from the data or picture classification result to the foreign object location or material identification result is completed, and a triple majority voting process is adopted to ensure the stability of the final foreign object location and material identification results.

[0229] Generalization performance verification: Referring to the aforementioned processing, foreign object samples of five materials are sequentially placed into seven sealed spaces of the device model. After each placement is completed, the device model is re-sealed and fixed to the vibration table. The vibration table is started to activate the foreign objects inside the device model to generate foreign object signals. Four acoustic emission sensors of the special device are placed at different positions on the surface of the device model to collect four-channel foreign object signals, which are finally stored in the local memory of the embedded computer. Through a large number of balanced experiments, four-channel foreign object signals of foreign object samples representing each material are collected in five groups and placed in each sealed space, for a total of 5×5×7 = 175 groups of four-channel foreign object signals. The position information and material information of the foreign object samples are labeled for each group of four-channel foreign object signals.

[0230] Using the two-stage adaptive energy threshold pulse extraction algorithm, 65,304 useful pulses are extracted from the 175 groups of four-channel foreign object signals. Using the improved threshold determination search pulse matching algorithm, 14,525 useful pulse groups are matched, which contain a total of 58,100 useful pulses. On this basis, the useful pulses are converted into 58,100 spectrograms to construct a verification picture set. Two verification picture sets are copied. By adding location tags and material tags to the spectrograms in the two verification picture sets respectively, a verification positioning picture set and a verification material picture set are established. Features in the joint positioning feature library and the joint material feature library are extracted from the useful pulse groups and the corresponding spectrograms respectively, and 14,525 feature vectors are obtained respectively to construct a verification data set. Two verification data sets are copied. By adding location tags and material tags to the feature vectors in the two verification data sets respectively, a verification positioning data set and a verification material data set are established. The specific descriptions of the verification picture set and the verification data set are shown in Table 13 and Table 14 respectively.

[0231] Specific descriptions of the verification positioning image set and the verification positioning data set

[0232]

[0233] Table 14 Specific descriptions of the verification material image set and the verification material data set

[0234]

[0235]

[0236] Apply the optimal positioning - neural network to make predictions on the verification positioning image set, apply the optimal material - neural network to make predictions on the verification material image set, apply the optimal positioning - classifier to make predictions on the verification positioning data set, apply the optimal material - classifier to make predictions on the verification material data set, and respectively count the classification accuracies they achieved, as shown in Table 15.

[0237] Table 15 Classification accuracies achieved by the optimal neural network and the optimal classifier

[0238]

[0239] Compared with the average classification accuracies of 95.16%, 97.56%, 96.92% and 97.46% achieved by the optimal positioning - neural network, the optimal material - neural network, the optimal positioning - classifier and the optimal material - classifier previously, the decline ranges of their newly achieved classification accuracies are relatively low, which are 0.12%, 0.05%, 0.07% and 0.13% respectively, and are almost negligible. This fully shows that the trained optimal positioning - neural network, optimal material - neural network, optimal positioning - classifier and optimal material - classifier have good generalization performance.

[0240] The above - mentioned examples of the present invention are only for explaining in detail the calculation model and calculation process of the present invention, rather than limiting the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or variations can be made on the basis of the above description. It is impossible to list all the implementation manners here. Any obvious changes or variations derived from the technical solutions of the present invention still fall within the protection scope of the present invention.

Claims

1. A method for locating foreign objects and identifying materials in a sealed electronic device with multi-source information fusion, characterized in that Including: Obtaining signals of a single measurement obtained by multiple acoustic emission sensors simultaneously for a sealed electronic device to be measured, and the signal corresponding to each acoustic emission sensor is recorded as the debris signal of one channel; recording the number of multi-channels as N, Performing pulse extraction on the N segments of debris signals corresponding to the N channels, and obtaining multiple useful pulses from a segment of debris signal of any one channel; performing pulse matching on the multiple useful pulses of the N channels. After pulse matching, multiple pulse groups are obtained from the N segments of debris signals corresponding to the N channels, and one pulse group contains one useful pulse of each channel. One pulse group is recorded as a useful pulse group; Taking the useful pulse as the basic unit, generating a spectrogram for each useful pulse, and respectively extracting gray-level co-occurrence matrix features for each spectrogram; Simultaneously extracting time-domain and frequency-domain features for each useful pulse; For each group of useful pulse groups, based on the time-domain features, frequency-domain features, and gray-level co-occurrence matrix features corresponding to the useful pulses of one group, obtaining a feature vector for positioning, recorded as the first recognition feature vector, and / or, based on the time-domain features, frequency-domain features, and gray-level co-occurrence matrix features corresponding to the useful pulses of one group, obtaining a feature vector for material recognition, recorded as the second recognition feature vector; Then inputting all the spectrograms into the positioning recognition neural network and / or the material recognition neural network respectively, and obtaining a position prediction label of the debris in the electronic device and / or a material prediction label of the debris in the electronic device for each spectrogram; obtaining a position prediction result by majority voting through all the position prediction labels and recording it as the first position prediction result, and / or, obtaining a material prediction result by majority voting through all the material prediction labels and recording it as the first material prediction result; Simultaneously inputting all the first recognition feature vectors into the positioning recognition classifier, and obtaining a position prediction label of the debris in the electronic device for each first recognition feature vector, and obtaining a position prediction result by majority voting through all the position prediction labels and recording it as the second position prediction result; and / or, inputting all the second recognition feature vectors into the material recognition classifier, and obtaining a material prediction label of the debris in the electronic device for each second recognition feature vector, and obtaining a material prediction result by majority voting through all the material prediction labels and recording it as the second material prediction result; Combining and performing majority voting on all the position prediction labels corresponding to the positioning recognition neural network and all the position prediction labels corresponding to the positioning recognition classifier to obtain a position prediction result and recording it as the third position prediction result, and / or, combining and performing majority voting on all the material prediction labels corresponding to the material recognition neural network and all the material prediction labels corresponding to the material recognition classifier to obtain a material prediction result and recording it as the third material prediction result; Performing majority voting using the first position prediction result to the third position prediction result to obtain the final position prediction result, and / or, performing majority voting using the first material prediction result to the third material prediction result to obtain the final material prediction result.

2. The method for positioning and material identification of foreign matters in a sealed electronic device with multi-source information fusion according to claim 1, characterized in that When all the position prediction labels corresponding to the positioning recognition neural network and all the position prediction labels corresponding to the positioning recognition classifier are merged for majority voting, all the position prediction labels corresponding to the positioning recognition neural network and all the position prediction labels corresponding to the positioning recognition classifier are subject to majority voting according to their weights; and / or, When all the material prediction labels corresponding to the material recognition neural network and all the material prediction labels corresponding to the material recognition classifier are merged for majority voting, all the material prediction labels corresponding to the material recognition neural network and all the material prediction labels corresponding to the material recognition classifier are subject to majority voting according to their weights.

3. The method for locating and identifying the material of foreign objects in a sealed electronic device with multi-source information fusion according to claim 2, wherein, All the position prediction labels corresponding to the positioning recognition neural network and all the position prediction labels corresponding to the positioning recognition classifier are subject to majority voting with weights of 0.8 and 0.2; and / or, All the material prediction labels corresponding to the material recognition neural network and all the material prediction labels corresponding to the material recognition classifier are subject to majority voting with weights of 0.8 and 0.

2.

4. The method for locating and material identifying redundant objects of a sealed electronic device with multi-source information fusion according to claim 1, wherein In the process of pulse extraction for the N segments of inclusion signals corresponding to the N channels, the two-stage adaptive energy threshold pulse extraction algorithm is used to extract pulses for each segment of inclusion signal. The specific process includes the following steps: Step 1: Calculate the average energy E of the redundant signal a , and calculate the standard deviation σ of the energy of the signal E ; Set the peak energy threshold as E p = E a + 3σ E , and set the endpoint energy threshold as E be = E a + σ E ; Step 2: Frame the signal; Step 3: Starting from the first frame signal, compare the energy of each frame signal with E in turn p ; When the energy of a certain frame signal is greater than E p , record this frame signal; and starting from it, continue to compare the energy of each subsequent frame signal with E p , until the energy of a certain frame signal is less than E p ; Find the frame signal with the maximum energy among these frame signals, and identify it as the maximum frame signal of the first useful pulse; Obtain the time of the sampling point with the maximum amplitude in the maximum frame signal, which is called the peak time t of the first useful pulse p_1 ; Step 4: Taking the peak moment t of the first useful pulse extracted in Step 3 p_1 as the starting point, compare the energy of each frame signal with E be forward and backward respectively until the energy of a certain frame signal is found to be less than Ebe in both directions; respectively identify the previous frame signal of these two frame signals as the first starting frame signal and the first ending frame signal of the first useful pulse; obtain the time of the first sampling point in the first starting frame signal, which is called the first starting moment t fb_1 of the first useful pulse; obtain the time of the last sampling point in the first ending frame signal, which is called the first ending moment t fe_1 ; Step 5: Using the next frame signal after the first ending moment t of the first useful pulse extracted in Step 4 as a new starting point, repeat the same processing procedures of Step 3 and Step 4 to extract the remaining useful pulses in the signal, and then save all the useful pulses extracted from the signal; meanwhile, save the peak moments of all the useful pulses fe_1 the first starting moment and the first ending moment ​ Step 6: Calculate the average energy and the standard deviation of the energy of each useful pulse saved in Step 5, denoted as E i and σ E_i On this basis, set the exclusive endpoint energy threshold for each useful pulse to E be_i = E i + σ Z_i ; Step Seven: Starting from the first useful pulse saved in Step Five, using the peak moment t of it extracted in Step Three as the starting point, compare the energy of each frame signal with the exclusive endpoint energy threshold E respectively forward and backward until it is found that the energy of a certain frame signal is less than E in both directions; respectively identify the previous frame signal of these two frame signals as the second starting frame signal and the second ending frame signal of the first useful pulse; p_1 be_1 be_1 ​​​ Obtain the time of the first sampling point in the second start frame signal, which is called the second start time t of the first useful pulse sb_1 ; Obtain the time of the last sampling point in the second end frame signal, which is called the second end time t of the first useful pulse se_1 ; Step 8: Starting from the second useful pulse saved in Step 5, repeat the same processing steps in Step 7 to respectively obtain the second start time of the remaining useful pulses saved in Step 5 and the second end time Step Nine: Calculate the first duration t of each useful pulse saved in Step Five fl_i = t fe_i - t fb_i , the second duration t sl_i = t se_i - t sb_i , and the duration ratio If the τ of the useful pulse i is greater than the duration threshold T, then its first start time t fb_i and the first end time t fe_i are recognized as the final start time t b_i and the final end time t e_i ; otherwise, its first start time t fb_i and the second end time t se_i are recognized as the final start time t b_i and the final end time t e_i ; thus, the final start time t b_i and the final end time t e_i of each useful pulse saved in Step Five are obtained and recorded.

5. The method for locating and material identifying redundant substances of a sealed electronic device with multi-source information fusion according to claim 1 or 4, characterized in that, In the process of pulse matching for multiple useful pulses of the N channels, the improved threshold determination search pulse matching algorithm is used for matching. Based on the results of pulse extraction for the N segments of inclusion signals corresponding to the N channels, the matching process includes the following steps: Step 1: For the multiple useful pulses corresponding to different channels respectively, obtain the useful pulse with the maximum energy among the multiple useful pulses. The useful pulses with the maximum energy corresponding to the N channels are called N reference useful pulses; According to the starting times of the N reference useful pulses, determine the inclusion signal collected by the acoustic emission sensor farthest from the inclusion, which is called the reference signal; Determine the inclusion signal collected by the acoustic emission sensor closest to the inclusion, which is called the comparison signal; Obtain the starting moments of the reference signal and the comparison signal, and calculate the difference between the two starting moments, which is called the starting moment threshold T; Step 2: Represent the useful pulses contained in the reference signal in Step 1 as P i , where i = 1, 2,..., n, and the starting times of these useful pulses are t i respectively; according to the starting time threshold T, set the starting time threshold interval of each useful pulse as [t i - T, t i ; Step 3: Starting from P1, search for useful pulses whose starting moments are within [t1 - T, t1] in the N - 1 segments of inclusion signals corresponding to the other N - 1 channels respectively; If N - 1 useful pulses can be found, form a pulse group with them and P1; Otherwise, skip P1, indicating that the matching is unsuccessful; Step 4: Pulse matching is performed on all P in the manner of Step 3 i ; thereby completing the pulse matching of the N-channel foreign matter signals to obtain multiple pulse groups.

6. The method for locating and material identifying redundant substances of a sealed electronic device with multi-source information fusion according to claim 1, characterized in that, The time domain features and frequency domain features in the first recognition feature vector include: time delay, pulse rise time, pulse symmetry, amplitude, energy, root mean square voltage, zero crossing rate, spectral centroid, spectral mean square deviation, root mean square probability, frequency standard deviation. The gray level co-occurrence matrix features in the first recognition feature vector include: entropy value, contrast, homogeneity, correlation; The time-domain features and frequency-domain features in the second recognition feature vector include: pulse rise time, pulse symmetry, energy, root mean square voltage, zero crossing rate, spectral centroid, spectral mean square deviation, root mean square probability, frequency standard deviation. The gray level co-occurrence matrix features in the second recognition feature vector include: image energy, entropy value, contrast, homogeneity, correlation.

7. The method for positioning and material identification of foreign matters in a sealed electronic device with multi-source information fusion according to claim 1, characterized in that, The positioning recognition neural network and the material recognition neural network adopt the PReLU-VGG19-Plus model. The PReLU-VGG19-Plus model is a VGG19 model with the first convolutional layer replaced by a 7×7 convolutional kernel and the activation function replaced by the PReLU activation function.

8. The method for locating and identifying the material of foreign matters in a sealed electronic device with multi-source information fusion according to claim 1, characterized in that The positioning recognition classifier and the material recognition classifier adopt the XGBoost model; The parameters of the XGBoost model corresponding to the positioning recognition classifier are as follows: n_estimatores is 180, max_depth is 15, min_child_weight is 4, learning_rate is 0.38, gamma is 0.03; The parameters of the XGBoost model corresponding to the material recognition classifier are as follows: n_estimatores is 150, max_depth is 15, min_child_weight is 4, learning_rate is 0.41, gamma is 0.05; Nestimatores refers to the number of base classifier decision trees; max_depth refers to the maximum depth of the base classifier decision tree; min_child_weight refers to the minimum leaf node sample weight that determines the base classifier decision tree; learning_rate refers to the learning rate; gamma refers to the penalty term coefficient.

9. A two-stage adaptive energy threshold pulse extraction method, which is used to extract pulses from the debris signals obtained by a single measurement through an acoustic emission sensor when detecting a sealed electronic device to be tested, and is characterized in that, The signal corresponding to each acoustic emission sensor is recorded as a channel of debris signal. For the N segments of debris signals corresponding to the N channels, the two-stage adaptive energy threshold pulse extraction algorithm is used to extract pulses from each segment of debris signal. The specific process includes the following steps: Step 1: Calculate the average energy E of the redundant signal a , and calculate the standard deviation σ of the energy of the signal E ; Set the peak energy threshold to E p = E a + 3σ E , Set the endpoint energy threshold to E be = E a + σ E ; Step 2: Frame the signal; Step 3: Starting from the first frame signal, compare the energy of each frame signal with E in sequence p ; When the energy of a certain frame signal is greater than E p , record this frame signal; and starting from it, continue to compare the energy of each subsequent frame signal with E p , until the energy of a certain frame signal is less than E p ; Find the frame signal with the maximum energy among these frame signals, and identify it as the maximum frame signal of the first useful pulse; Obtain the time of the sampling point with the maximum amplitude in the maximum frame signal, which is called the peak time t of the first useful pulse p_1 ; Step 4: Using the peak moment t of the first useful pulse extracted in Step 3 p_1 as the starting point, compare the energy of each frame signal with E be forward and backward respectively until it is found that the energy of a certain frame signal is less than E be in both directions; respectively identify the previous frame signal of these two frame signals as the first starting frame signal and the first ending frame signal of the first useful pulse; obtain the time of the first sampling point in the first starting frame signal, which is called the first starting moment t fb_1 of the first useful pulse; obtain the time of the last sampling point in the first ending frame signal, which is called the first ending moment t fe_1 of the first useful pulse; Step Five: Using the next frame signal after the first ending moment t of the first useful pulse extracted in Step Four as a new starting point, repeat the same processing procedures of Step Three and Step Four to extract the remaining useful pulses in the signal, and then save all the useful pulses extracted from the signal; meanwhile, save the peak moments of all the useful pulses fe_1 The first starting moment and the first ending moment ​ Step 6: Calculate the average energy and the standard deviation of the energy of each useful pulse saved in Step 5, denoted as E i and σ E_i On this basis, set the exclusive endpoint energy threshold for each useful pulse as E be_i = E i + σ Z_i ; Step Seven: Starting from the first useful pulse saved in Step Five, using the peak moment t of it extracted in Step Three as the starting point, compare the energy of each frame signal with the exclusive endpoint energy threshold E respectively forward and backward p_1 until it is found that the energy of a certain frame signal is less than E in both directions be_1 ; respectively identify the previous frame signal of these two frame signals as the second starting frame signal and the second ending frame signal of the first useful pulse; be_1 ​ Obtain the time of the first sampling point in the second start frame signal, which is called the second start time t of the first useful pulse sb_1 ; Obtain the time of the last sampling point in the second end frame signal, which is called the second end time t of the first useful pulse se_1 ; Step Eight: Starting from the second useful pulse saved in Step Five, repeat the same processing steps as in Step Seven to obtain the second starting moment of the remaining useful pulses saved in Step Five respectively and the second ending moment Step Nine: Calculate the first duration t of each useful pulse saved in Step Five fl_i = t fe_i - t fb_i , the second duration t sl_i = t se_i - t sb_i , and the duration ratio If the τ of the useful pulse i is greater than the duration threshold T, then its first start time t fb_i and the first end time t fe_i are recognized as the final start time t b_i and the final end time t e_i ; otherwise, its first start time t fb_i and the second end time t se_i are recognized as the final start time t b_i and the final end time t e_i ; thus obtaining the final start time t b_i and the final end time t e_i of each useful pulse saved in Step Five and recording them.

10. An improved threshold determination search pulse matching algorithm for matching method, the method is used to match the redundant signals of multiple channels after pulse extraction of a single measurement signal, and the signal corresponding to each acoustic emission sensor is recorded as the redundant signal of one channel; the number of multi-channels is denoted as N, and pulse extraction is performed on the corresponding N segments of redundant signals in the N channels during pulse extraction; characterized in that, Based on the pulse extraction results of the N segments of debris signals corresponding to the N channels, the improved threshold determination search pulse matching algorithm is used for pulse matching. The matching process includes the following steps: Step 1: For the multiple useful pulses corresponding to different channels respectively, obtain the useful pulse with the maximum energy among the multiple useful pulses. The useful pulses with the maximum energy corresponding to the N channels are called N reference useful pulses; according to the start times of the N reference useful pulses, determine the debris signal collected by the acoustic emission sensor farthest from the debris, which is called the reference signal; determine the debris signal collected by the acoustic emission sensor closest to the debris, which is called the comparison signal; obtain the start times of the reference signal and the comparison signal, and calculate the difference between the two start times, which is called the start time threshold T; Step 2: Represent the useful pulses included in the reference signal in Step 1 as P i , where i = 1, 2,..., n, and the starting moments of these useful pulses are t i respectively; according to the starting moment threshold T, set the starting moment threshold interval of each useful pulse as [t i - T, t i ; Step 3: Starting from P1, search for useful pulses in the N-1 segments of redundant signal corresponding to the other N-1 channels, where the starting time of the useful pulses is within [t1 - T, t1]; if N-1 useful pulses can be found, form a pulse group with them and P1; otherwise, skip P1, indicating that the matching is unsuccessful. Step 4: Pulse matching is performed on all P in the same way as in Step 3 i ; thus, the pulse matching of the N-channel foreign matter signals is completed, and multiple pulse groups are obtained.

Images