Multi-source information fusion sealed electronic equipment redundancy positioning and material identification method and pulse extraction and matching method
Through multi-source information fusion and neural network voting, the accuracy of the positioning of excess materials and material recognition of sealed electronic devices is improved, and the problem of insufficient identification accuracy in the prior art is solved, which is suitable for sealed electronic devices with complex structures.
Patent Information
- Application Number
- CN202411021436.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-29
AI Technical Summary
The accuracy of the identification of the excess positioning and material recognition of existing sealed electronic equipment is insufficient. Especially under complex structures, the feasibility and generalization of the original method need to be further verified.
Using multi-source information fusion method, multi-channel signals are obtained through acoustic emission sensors, pulse extraction and matching, sound spectrum diagrams are generated, grayscale symbiosis matrix features and time and frequency domain features are extracted, and the majority is performed by combining neural networks and classifiers. Voting to improve the accuracy of positioning and material identification.
It significantly improves the positioning and material recognition accuracy of the excess of sealed electronic equipment, and is suitable for sealed electronic equipment with complex structures, enhancing the stability and generalization ability of the recognition effect.
Smart Images

Figure CN118965267A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of detecting excess matter in sealed electronic equipment, and in particular relates to a method for locating excess matter and identifying materials in sealed electronic equipment, as well as a pulse extraction method and a pulse matching method. Background Art
[0002] Sealed electronic equipment refers to electronic equipment that is encapsulated using special materials and processes to prevent damage or interference from the external environment, such as water, dust, chemicals, etc. Sealed electronic equipment usually has the characteristics of being waterproof, dustproof, and corrosion-resistant, and can operate stably in harsh environments. It is widely used in the field of aerospace and national defense. Due to the limitations of production processes, during the manufacturing process of sealed electronic equipment, materials such as metal debris, spot welding spatter, and thin wire segments will be encapsulated inside to form excess materials. Sealed electronic equipment usually operates in an environment of super-weightlessness and severe impact. After the internal excess materials are activated, they will collide and damage internal components, or adhere to the surface of the circuit to cause short circuits and open circuits, or get stuck in active parts to cause some functions to fail, etc., which will have an important impact on the reliable operation of sealed electronic equipment, leading to the failure of space launch missions, and even causing major space accidents and casualties. Therefore, it is very important to detect excess materials before sealed electronic equipment leaves the factory for service.
[0003] The particle impact noise detection PIND method is the most commonly used method for detecting excess material. The excess material detection process focuses more on whether the excess material signal is detected, rather than conducting in-depth research on the signal itself. In fact, the excess material signal contains information that can feedback the properties of the excess material itself, such as the material, location, weight, etc. of the excess material. Especially in the research on excess material detection of sealed electronic equipment with huge size and complex structure, after judging the existence of excess material, obtaining the location of the excess material can guide the cleaning work, and obtaining the material of the excess material can trace the source and improve the process standards. Therefore, complete sealed electronic equipment excess material detection information should not only include whether there is excess material, but also include information on the properties of the excess material itself, especially location information and material information.
[0004] In recent years, machine learning has been widely used in signal classification and recognition research. It constructs feature data by quantifying the differences in signal characteristics between signals of different categories, and trains classifiers in high-dimensional feature space to classify feature data of different labels, thereby realizing the recognition of signals of different categories. Similarly, the differences in signal characteristics between redundant signals generated by redundant objects of different positions and materials can also be quantified to train appropriate classifiers to realize redundant object positioning and material recognition, respectively. In previous studies, for the problem of redundant object positioning of sealed electronic equipment, the redundant object signal was used as the research object, a complete experimental system was built, specific implementation steps were designed, and research was carried out in pulse extraction and matching, feature extraction and optimization, and integrated classifier optimization, achieving a positioning accuracy of 90.91%. Unfortunately, in previous studies, the internal structure of the sealed electronic equipment used was relatively simple and regular, such as aerospace power supplies, and the internal space of the equipment model produced was divided into approximately equal-sized enclosed spaces. The distribution characteristics between the constructed feature data of different labels were relatively regular, and the performance of the trained integrated classifier was relatively excellent. This is friendly to the initial research. However, in real application scenarios, the internal structure of sealed electronic devices is more complex and irregular. The internal space of the device model made in this way can only be divided into enclosed spaces of different shapes and sizes. The feasibility and generalization of the original redundant object positioning method need to be further verified.
[0005] It is worth noting that as an important branch of machine learning, deep learning has been widely used in acoustic emission detection, speech recognition, fault diagnosis and other fields in recent years, and has achieved remarkable results. For example, Lei et al. conducted a study on the health maintenance decision and remaining life prediction of mechanical equipment, Li et al. conducted a study on intelligent operation and maintenance of equipment and defect detection based on visual signals, Feng et al. conducted a study on digital twins and fatigue wear of complex systems, and He et al. conducted a study on computational intelligence and fault diagnosis of industrial processes. These studies all take signal images as the research object. Even if the research objects in the initial stage are fault signals and sound signals in digital format, they are converted into signal images that can be processed by deep learning. In a broad sense, the redundant signal belongs to the acoustic emission signal in digital format, and the spectrogram technology can convert it into a signal image. In previous studies, there was a concern that the attribute information of the redundant signal itself would be lost in the process of converting the signal into an image, and no redundant detection research based on deep learning was carried out with images as the research object. However, a large number of research results conducted by scholars show that the classic neural network in deep learning can make up for the information loss caused by the signal conversion process with its excellent classification performance, and achieve a higher classification effect than the signal as the research object. This provides new ideas for existing research on redundant object positioning and material identification.
[0006] Therefore, the positioning and material recognition capabilities in the existing research on the positioning of excess objects in sealed electronic equipment and the identification of the materials of excess objects in sealed electronic components need to be improved, the pulse extraction and matching algorithms need to be optimized, and a combined classifier with a stable structure needs to be designed. The accuracy of identifying the materials of excess objects in sealed electronic equipment also needs to be further improved. Summary of the invention
[0007] The present invention aims to solve the problem that the accuracy of redundant object positioning and material identification of the existing sealed electronic equipment in the prior art needs to be further improved.
[0008] The method for locating redundant objects and identifying materials of sealed electronic equipment based on multi-source information fusion includes:
[0009] For the sealed electronic device to be tested, a signal of a measurement is obtained by simultaneously performing a measurement on multiple acoustic emission sensors. The signal corresponding to each acoustic emission sensor is recorded as a redundant signal of a channel. The number of multiple channels is recorded as N.
[0010] Pulse extraction is performed on N sections of the corresponding redundant signal in N channels, and a section of the redundant signal in any channel obtains multiple useful pulses; pulse matching is performed on multiple useful pulses in the N channels, and after pulse matching, the N sections of the redundant signal corresponding to the N channels obtain multiple pulse groups, and one pulse group contains one useful pulse of each channel, and one pulse group is recorded as a useful pulse group;
[0011] Taking the useful pulse as the basic unit, a spectrogram is generated for each useful pulse, and the gray-level co-occurrence matrix features are extracted for each spectrogram; at the same time, the time domain and frequency domain features are extracted for each useful pulse;
[0012] For each group of useful pulses, based on the time domain features, frequency domain features and gray-level co-occurrence matrix features corresponding to a group of useful pulses, a feature vector for positioning is obtained, recorded as a first identification feature vector, and / or, based on the time domain features, frequency domain features and gray-level co-occurrence matrix features corresponding to a group of useful pulses, a feature vector for material identification is obtained, recorded as a second identification feature vector;
[0013] Then all the spectrograms are input into the positioning recognition neural network and / or the material recognition neural network respectively, and each spectrogram obtains a position prediction label of the redundant object in the electronic device and / or a material prediction label of the redundant object in the electronic device; a position prediction result is obtained by majority voting of all the position prediction labels and recorded as the first position prediction result, and / or, a material prediction result is obtained by majority voting of all the material prediction labels and recorded as the first material prediction result;
[0014] At the same time, all the first identification feature vectors are respectively input into the positioning identification classifier, and each first identification feature vector obtains a position prediction label of the redundant object in the electronic device, and a position prediction result is obtained by majority voting through all the position prediction labels and recorded as the second position prediction result; and / or, all the second identification feature vectors are input into the material identification classifier, and each second identification feature vector obtains a material prediction label of the redundant object in the electronic device, and a material prediction result is obtained by majority voting through all the material prediction labels and recorded as the second material prediction result;
[0015] All position prediction labels corresponding to the positioning recognition neural network and all position prediction labels corresponding to the positioning recognition classifier are combined and voted by majority to obtain a position prediction result and recorded as a third position prediction result, and / or, all material prediction labels corresponding to the material recognition neural network and all material prediction labels corresponding to the material recognition classifier are combined and voted by majority to obtain a material prediction result and recorded as a third material prediction result;
[0016] The first position prediction result to the third position prediction result are used to perform majority voting to obtain a final position prediction result, and / or the first material prediction result to the third material prediction result are used to perform majority voting to obtain a final material prediction result.
[0017] Further, when all the position prediction labels corresponding to the positioning recognition neural network and all the position prediction labels corresponding to the positioning recognition classifier are combined for majority voting, all the position prediction labels corresponding to the positioning recognition neural network and all the position prediction labels corresponding to the positioning recognition classifier are voted by majority according to the weights;
[0018] and / or,
[0019] When all material prediction labels corresponding to the material recognition neural network and all material prediction labels corresponding to the material recognition classifier are combined for majority voting, all material prediction labels corresponding to the material recognition neural network and all material prediction labels corresponding to the material recognition classifier are voted by majority according to the weights.
[0020] Furthermore, all the location prediction labels corresponding to the location recognition neural network and all the location prediction labels corresponding to the location recognition classifier are voted by majority according to weights 0.8 and 0.2;
[0021] and / or,
[0022] All material prediction labels corresponding to the material recognition neural network and all material prediction labels corresponding to the material recognition classifier are voted by majority with weights of 0.8 and 0.2.
[0023] Furthermore, the process of pulse extraction for the N sections of the redundant signal corresponding to the N channels adopts a two-level adaptive energy threshold pulse extraction algorithm to extract the pulse of each section of the redundant signal. The specific process includes the following steps:
[0024] Step 1: Calculate the average energy E of the redundant signal a , and calculate the standard deviation σ of the signal energy E ; Set the peak energy threshold to E p =E a +3σ E , set the endpoint energy threshold to E be =E a +σ E ;
[0025] Step 2: Frame the signal;
[0026] Step 3: Starting from the first frame signal, compare the energy of each frame signal with E p ; When the energy of a frame signal is greater than E p When the frame signal is recorded, and starting from it, the energy of each subsequent frame signal is compared with E p , until the energy of a frame signal is less than E p ; Find the frame signal with the largest energy from these frame signals and identify it as the maximum frame signal of the first useful pulse; Get the time of the sampling point with the largest amplitude in the maximum frame signal, which is called the peak time t of the first useful pulse p_1 ;
[0027] Step 4: Take the peak time t of the first useful pulse extracted in step 3 p_1 As the starting point, compare the energy of each frame signal with E be Until the energy of a frame signal is found to be less than E in both directions be ; Respectively identify the previous frame signals of the two frame signals as the first starting frame signal and the first ending frame signal of the first useful pulse; obtain the time of the first sampling point in the first starting frame signal, which is called the first starting time t of the first useful pulse fb_1 ; Get the time of the last sampling point in the first end frame signal, which is called the first end time t of the first useful pulse fe_1 ;
[0028] Step 5: Take the first end time t of the first useful pulse extracted in step 4 fe_1 The next frame signal is taken as the new starting point, and the same processing process of step 3 and step 4 is repeated to extract the remaining useful pulses in the signal, and then all the useful pulses extracted from the signal are saved; at the same time, the peak moments of all useful pulses are saved. First starting moment and the first ending moment
[0029] Step 6: Calculate the average energy and standard deviation of each useful pulse saved in step 5, expressed as E i and σ E_i On this basis, the exclusive endpoint energy threshold of each useful pulse is set to t be_i =E i +σ Z_i ;
[0030] Step 7: Starting from the first useful pulse saved in step 5, take its peak time t extracted in step 3 p_1 As the starting point, the energy of each frame signal is compared with the exclusive endpoint energy threshold E be_1 Until the energy of a frame signal is found to be less than E in both directions be_1 ; Respectively identify the previous frame signal of the two frame signals as the second start frame signal and the second end frame signal of the first useful pulse;
[0031] The time of obtaining the first sampling point in the second starting frame signal is called the second starting time t of the first useful pulse. sb_1 ; Get the time of the last sampling point in the second end frame signal, which is called the second end time t of the first useful pulse se_1 ;
[0032] Step 8: Starting from the second useful pulse saved in step 5, repeat the same processing steps as step 7 to obtain the second starting time of the remaining useful pulses saved in step 5 respectively. and the second end moment
[0033] Step 9: Calculate the first duration t of each useful pulse saved in step 5 fl_i =t fe_i -t fb_i , the second duration t sl_i =t se_i -t sb_i , and duration ratio If the useful pulse τ i If it is greater than the duration threshold T, then its first starting time t fb_i With the first end time t fe_i The final starting time t b_i and the final ending time t e_i Otherwise, its first starting time t fb_i and the second end time t se_i The final starting time tb_i and the final ending time t e_i ; Thus, the final starting time t of each useful pulse saved in step 5 is obtained b_i and the final ending time t e_i And record it.
[0034] Furthermore, the process of pulse matching for multiple useful pulses of N channels adopts an improved threshold judgment search pulse matching algorithm for matching, and pulse matching is performed based on the results of pulse extraction for N segments of corresponding redundant signals in N channels. The matching process includes the following steps:
[0035] Step 1: For multiple useful pulses corresponding to different channels, respectively obtain the useful pulses with the largest energy among the multiple useful pulses, and the useful pulses with the largest energy corresponding to N channels are called N reference useful pulses; according to the starting time of the N reference useful pulses, determine the redundant object signal collected by the acoustic emission sensor farthest from the redundant object, which is called the reference signal; determine the redundant object signal collected by the acoustic emission sensor closest to the redundant object, which is called the comparison signal; obtain the starting time of the reference signal and the comparison signal, and calculate the difference between the two starting times, which is called the starting time threshold T;
[0036] Step 2: Denote the useful pulses contained in the reference signal in step 1 as P i , i = 1, 2, ..., n, the starting times of these useful pulses are t i ; According to the starting time threshold T, set the starting time threshold interval of each useful pulse to [t i -T,t i ];
[0037] Step 3: Starting from P1, search for useful pulses whose starting time is within [t1-T, t1] in the N-1 segments of redundant signals corresponding to the other N-1 channels; if N-1 useful pulses can be found, they are combined with P1 into a pulse group; otherwise, P1 is skipped, indicating that its matching is unsuccessful;
[0038] Step 4: Follow the steps in step 3 to calculate all P i Pulse matching is performed; thereby, pulse matching of N-channel redundant signals is completed to obtain multiple pulse groups.
[0039] Further, the time domain features and frequency domain features in the first identification feature vector include: time delay, pulse rise time, pulse symmetry, amplitude, energy, root mean square voltage, zero crossing rate, spectrum centroid, spectrum mean square error, root mean square probability, frequency standard deviation, and the gray level co-occurrence matrix features in the first identification feature vector include: entropy, contrast, homogeneity, and correlation;
[0040] The time domain features and frequency domain features in the second identification feature vector include: pulse rise time, pulse symmetry, energy, root mean square voltage, zero crossing rate, spectrum centroid, spectrum mean square error, root mean square probability, and frequency standard deviation. The gray level co-occurrence matrix features in the second identification feature vector include: graph energy, entropy value, contrast, homogeneity, and correlation.
[0041] Furthermore, the positioning recognition neural network and the material recognition network neural network adopt a PReLU-VGG19-Plus model, and the PReLU-VGG19-Plus model is a VGG19 model in which the first convolution layer is replaced with a 7×7 convolution kernel and the activation function is replaced with a PReLU activation function.
[0042] Furthermore, the positioning recognition classifier and the material recognition classifier adopt the XGBoost model;
[0043] The parameters of the XGBoost model corresponding to the positioning recognition classifier are as follows:
[0044] n_estimatores is 180, max_depth is 15, min_child_weight is 4, learning_rate is 0.38, gamma is 0.03;
[0045] The parameters of the XGBoost model corresponding to the material recognition classifier are as follows:
[0046] n_estimatores is 150, max_depth is 15, min_child_weight is 4, learning_rate is 0.41, gamma is 0.05;
[0047] n_estimatores refers to the number of base classifier decision trees; max_depth refers to the maximum depth of the base classifier decision tree; min_child_weight refers to the minimum leaf node sample weight that determines the base classifier decision tree; learning_rate refers to the learning rate; gamma refers to the penalty term coefficient.
[0048] A two-level adaptive energy threshold pulse extraction method is provided. The method is used to extract pulses from a redundant signal obtained by a measurement by an acoustic emission sensor when testing a sealed electronic device to be tested. The signal corresponding to each acoustic emission sensor is recorded as a redundant signal of a channel. The pulse extraction process for N sections of redundant signals corresponding to N channels uses a two-level adaptive energy threshold pulse extraction algorithm to extract pulses from each section of the redundant signal. The specific process includes the following steps:
[0049] Step 1: Calculate the average energy E of the redundant signal a , and calculate the standard deviation σ of the signal energy E ; Set the peak energy threshold to E p =E a +3σ E , set the endpoint energy threshold to E be =E a +σ E ;
[0050] Step 2: Frame the signal;
[0051] Step 3: Starting from the first frame signal, compare the energy of each frame signal with E p ; When the energy of a frame signal is greater than E p When the frame signal is recorded, and starting from it, the energy of each subsequent frame signal is compared with E p , until the energy of a frame signal is less than E p ; Find the frame signal with the largest energy from these frame signals and identify it as the maximum frame signal of the first useful pulse; Get the time of the sampling point with the largest amplitude in the maximum frame signal, which is called the peak time t of the first useful pulse p_1 ;
[0052] Step 4: Take the peak time t of the first useful pulse extracted in step 3 p_1 As the starting point, compare the energy of each frame signal with E be Until the energy of a frame signal is found to be less than E in both directions be ; Respectively identify the previous frame signals of the two frame signals as the first starting frame signal and the first ending frame signal of the first useful pulse; obtain the time of the first sampling point in the first starting frame signal, which is called the first starting time t of the first useful pulse fb_1 ; Get the time of the last sampling point in the first end frame signal, which is called the first end time t of the first useful pulse fe_1 ;
[0053] Step 5: Take the first end time t of the first useful pulse extracted in step 4 fe_1 The next frame signal is taken as the new starting point, and the same processing process of step 3 and step 4 is repeated to extract the remaining useful pulses in the signal, and then all the useful pulses extracted from the signal are saved; at the same time, the peak moments of all useful pulses are saved. First starting moment and the first ending moment
[0054] Step 6: Calculate the average energy and standard deviation of each useful pulse saved in step 5, expressed as Ei and σ E_i On this basis, the exclusive endpoint energy threshold of each useful pulse is set to E be_i =E i +σ Z_i ;
[0055] Step 7: Starting from the first useful pulse saved in step 5, take its peak time t extracted in step 3 p_1 As the starting point, the energy of each frame signal is compared with the exclusive endpoint energy threshold E be_1 Until the energy of a frame signal is found to be less than E in both directions be_1 ; Respectively identify the previous frame signal of the two frame signals as the second start frame signal and the second end frame signal of the first useful pulse;
[0056] The time of obtaining the first sampling point in the second starting frame signal is called the second starting time t of the first useful pulse. sb_1 ; Get the time of the last sampling point in the second end frame signal, which is called the second end time t of the first useful pulse se_1 ;
[0057] Step 8: Starting from the second useful pulse saved in step 5, repeat the same processing steps as step 7 to obtain the second starting time of the remaining useful pulses saved in step 5 respectively. and the second end moment
[0058] Step 9: Calculate the first duration t of each useful pulse saved in step 5 fl_i =t fe_i -t fb_i , the second duration t sl_i =t se_i -t sb_i , and duration ratio If the useful pulse τ i If it is greater than the duration threshold T, then its first starting time t fb_i With the first end time t fe_i The final starting time t b_i and the final ending time t e_i Otherwise, its first starting time t fb_i and the second end time t se_i The final starting time t b_i and the final ending time t e_i ; Thus, the final starting time t of each useful pulse saved in step 5 is obtained b_i and the final ending time t e_i And record it.
[0059] A matching method for an improved threshold judgment search pulse matching algorithm is disclosed. The method is used to match redundant signals of multiple channels after a measurement signal is pulse extracted. The signal corresponding to each acoustic emission sensor is recorded as a redundant signal of one channel. The number of multiple channels is recorded as N. During pulse extraction, pulse extraction is performed for N sections of redundant signals corresponding to the N channels. Based on the pulse extraction results of the N sections of redundant signals corresponding to the N channels, pulse matching is performed using an improved threshold judgment search pulse matching algorithm. The matching process includes the following steps:
[0060] Step 1: For multiple useful pulses corresponding to different channels, respectively obtain the useful pulses with the largest energy among the multiple useful pulses, and the useful pulses with the largest energy corresponding to N channels are called N reference useful pulses; according to the starting time of the N reference useful pulses, determine the redundant object signal collected by the acoustic emission sensor farthest from the redundant object, which is called the reference signal; determine the redundant object signal collected by the acoustic emission sensor closest to the redundant object, which is called the comparison signal; obtain the starting time of the reference signal and the comparison signal, and calculate the difference between the two starting times, which is called the starting time threshold T;
[0061] Step 2: Denote the useful pulses contained in the reference signal in step 1 as P i , i = 1, 2, ..., n, the starting times of these useful pulses are t i ; According to the starting time threshold T, set the starting time threshold interval of each useful pulse to [t i -T,t i ];
[0062] Step 3: Starting from P1, search for useful pulses whose starting time is within [t1-T, t1] in the N-1 segments of redundant signals corresponding to the other N-1 channels; if N-1 useful pulses can be found, they are combined with P1 into a pulse group; otherwise, P1 is skipped, indicating that its matching is unsuccessful;
[0063] Step 4: Follow the steps in step 3 to calculate all P i Pulse matching is performed; thereby, pulse matching of N-channel redundant signals is completed to obtain multiple pulse groups.
[0064] Beneficial effects of the present invention:
[0065] (1) The present invention converts useful pulses in the redundant signal into a spectrogram. On the one hand, a suitable neural network is directly trained on the spectrogram. On the other hand, gray-level co-occurrence matrix features are extracted from the spectrogram to establish a joint feature library and then a data set, thereby training a suitable classifier, which can effectively guarantee and improve the recognition effect of the classifier.
[0066] (2) The present invention uses a large number of redundant object signals that contain both position information and material information to establish a positioning picture set and a material picture set, a positioning data set and a material data set, and thereby trains a positioning-neural network, a material-neural network, a positioning-classifier, and a material-classifier, and then performs joint recognition based on the positioning-neural network, the material-neural network, the positioning-classifier, and the material-classifier. At the same time, a majority vote is performed on the basis of the overall classification label set. The present invention designs a triple majority voting process, and then obtains the final positioning result after another majority vote (overall majority voting), thereby constructing a combined positioning framework. That is, the present invention combines the requirements of redundant object positioning and material identification of sealed electronic equipment, and first extracts a method of local majority voting + overall label set voting + final overall voting, which can effectively guarantee and improve the overall recognition effect, and can achieve excellent prediction results not only for positioning, but also for material identification.
[0067] In addition, in response to the shortcomings of existing pulse extraction algorithms, the present invention proposes a new pulse extraction algorithm based on multiple energy thresholds, and in response to the shortcomings of existing threshold judgment search pulse matching algorithms, improves the matching method of reference useful pulses, namely, the newly proposed two-level adaptive energy threshold pulse extraction method and improved threshold judgment search pulse matching method in the present invention, which can improve the effects of pulse extraction and matching, and further ensure the positioning and material recognition effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] Figure 1 It is a flow chart for realizing the method of locating redundant objects and identifying materials;
[0069] Figure 2 This is a physical picture of sealed electronic equipment;
[0070] Figure 3 Create renderings for the interior space division;
[0071] Figure 4 This is a physical picture of the redundant sample;
[0072] Figure 5 This is a physical picture of the special equipment;
[0073] Figure 6 This is a physical diagram of the redundant signal acquisition process;
[0074] Figure 7 It is a schematic diagram of a two-stage adaptive energy threshold extraction algorithm;
[0075] Figure 8 Schematic diagram of pulse matching algorithm for searching for improved threshold decision;
[0076] Fig. 9 is a spectrogram;
[0077] Fig.10 This is the classification effect diagram obtained by VGG19 on the positioning picture set;
[0078] Fig.11 This is the classification effect diagram obtained by VGG19 on the texture image set;
[0079] Fig.12 Schematic diagram of the structure of VGG19-Plus;
[0080] Fig.13 Images of PreLU and ReLU activation functions;
[0081] Fig.14 This is the classification effect diagram of PReLU-VGG19-Plus on the positioning picture set;
[0082] Fig.15 This is the classification effect achieved by PReLU-VGG19-Plus on the texture image set. DETAILED DESCRIPTION
[0083] In order to solve the problems in the background technology, in the present invention, the requirements for positioning and identifying materials of redundant objects in sealed electronic equipment in real application scenarios are comprehensively considered, and a method for positioning and identifying materials of redundant objects with multi-source information fusion is proposed. Specifically, firstly, according to the latest redundant object detection information in real application scenarios, real sealed electronic equipment is used to make redundant object samples of different materials, and a large number of redundant object signals generated by redundant object samples of different materials placed in different positions are collected. Secondly, the redundant object signals are processed using the newly proposed pulse extraction algorithm and the improved pulse matching algorithm to obtain a large number of useful pulses. On this basis, on the one hand, all useful pulses are converted into spectrograms, and positioning picture sets and material picture sets are established respectively according to the position information and material information of the redundant object signals from the useful pulse sources. On the other hand, traditional time domain and frequency domain features are extracted from the useful pulses, and picture features are extracted from the corresponding spectrograms to establish a joint feature library. Through feature selection, the features that can feedback the position information and material information are retained respectively, and a joint positioning feature library and a joint material feature library are established, thereby establishing a positioning data set and a material data set. Then, on the one hand, multiple neural networks are trained on the positioning picture set and the material picture set respectively, and the best one is selected for hyperparameter optimization to obtain the optimal positioning-neural network and the optimal material-neural network. On the other hand, multiple classifiers are trained on the positioning data set and the material data set respectively, and the best one is selected for parameter optimization to obtain the optimal positioning-classifier and the optimal material-classifier. Finally, by combining the optimal positioning-neural network, the optimal positioning-classifier and the triple majority voting process, and combining the optimal material-neural network, the optimal material-classifier and the triple majority voting process, a new combined positioning framework and combined material framework are constructed respectively, and multiple physical tests are carried out in real application scenarios to verify the feasibility, practicality and stability of the proposed method.
[0084] The core of the present invention lies in the two migrations of the research object. For the sealed electronic device to be tested, the present invention finds a sealed electronic device of the same model as a model, which is called a device model. Therefore, the first migration of the research object is from the sealed electronic device to be tested to the device model. According to the real internal structure of the device model, which is actually the real internal structure of the sealed electronic device, the internal space of the device model is divided into multiple closed spaces, and multiple thin plates are used to completely separate some originally connected closed spaces. In this way, the closed spaces are independent of each other, and there will be no movement of redundant objects located in a certain closed space to other closed spaces. It is worth noting that the material of the thin plate is the same as the material of the inner wall of the device model. The thin plate is fixed in the internal space and can be regarded as integrated with the original device model, and no interference will be generated during the experiment. On this basis, redundant objects of different materials are placed in different closed spaces in turn, and a large number of redundant object signals with different position information and material information are collected by means of redundant object automatic detection equipment. Through the steps of pulse preprocessing, feature engineering, model training and optimization, the optimal positioning-neural network and the optimal material-neural network, the optimal positioning-classifier and the optimal material-classifier are obtained respectively.
[0085] The second migration of the research object is from the device model to the sealed electronic device to be tested. With the help of the redundant automatic detection equipment, a section of the redundant signal to be tested is collected, that is, its position information and material information are unknown. Similarly, through the steps of pulse preprocessing, feature engineering, etc., the sound spectrum set to be tested for positioning and material identification is obtained, that is, the picture set to be tested, the positioning data set to be tested for positioning, and the material data set to be tested for material identification. The optimal positioning-neural network is used to predict on the picture set to be tested, and the optimal positioning-classifier is used to predict on the positioning data set to be tested, and a set of two predicted labels is obtained. On this basis, a combined positioning framework is constructed using a triple majority voting process, and the positioning result is obtained. Similarly, the optimal material-neural network is used to predict on the picture set to be tested, and the optimal material-classifier is used to predict on the material data set to be tested, and a set of two predicted labels is obtained, thereby constructing a combined material framework, and obtaining a material identification result. Finally, the redundant positioning and material identification results of the sealed electronic device to be tested are obtained. The present invention is described in detail below in conjunction with specific embodiments.
[0086] Specific implementation method 1: Combination Figure 1 To explain this embodiment,
[0087] Step 1: Use thin plates to divide the internal space of a sealed electronic device of the same model as the sealed electronic device to be tested into multiple independent closed spaces, and number them. According to the redundant object detection information in the real application scenario, make redundant object samples of different materials and encode them.
[0088] A plurality of acoustic emission sensors are placed at different positions on the surface of the equipment model, and the acoustic emission sensors are commonly connected to the automatic detection device for excess objects.
[0089] Step 2: Open the equipment model, and place the first material's waste sample into the confined space numbered 1 according to the space number and material code, and re-seal the equipment model. Collect a set of multi-channel waste signals with the help of the waste automatic detection equipment, that is, a set of multi-segment waste signals collected by multiple acoustic emission sensors at the same time. Similarly, place the first material's waste samples into other numbered confined spaces in turn, and collect multiple sets of multi-channel waste signals.
[0090] Then, the second material's waste samples are placed in different numbered closed spaces in turn, and multiple groups of multi-channel waste signals are collected; the third material's waste samples are placed in different numbered closed spaces in turn, and multiple groups of multi-channel waste signals are collected; until the last material's waste samples are placed in different numbered closed spaces in turn, and multiple groups of multi-channel waste signals are collected. A large number of multi-channel waste signals and their space numbers and material codes are obtained.
[0091] Step 3: Use the two-level adaptive energy threshold pulse extraction algorithm to extract pulses from each section of the redundant signal, and obtain multiple useful pulses of each section of the redundant signal. Use the improved threshold judgment search pulse matching algorithm to perform pulse matching on each group of multi-channel redundant signals, and obtain multiple useful pulse groups of each group of multi-channel redundant signals. A large number of useful pulse groups and multiple useful pulses in each useful pulse group are obtained.
[0092] It should be noted that pulse extraction and pulse matching are performed based on each measurement signal. Subsequent steps can be performed based on the results of pulse extraction and matching corresponding to one measurement, or pulse extraction and matching can be performed for multiple measurements separately, and then subsequent steps can be performed based on all the results.
[0093] Step 4: Taking the useful pulse as the basic unit, convert each useful pulse into a spectrogram, and obtain a spectrogram set for all useful pulses; and copy to obtain two spectrogram sets.
[0094] For the first set of spectrograms, according to the spatial number of the redundant signal to which the useful pulse belongs, the same spatial number is added to the spectrogram corresponding to the useful pulse, so as to obtain a positioning picture set with spatial number labels.
[0095] For the second set of spectrograms, according to the material coding of the redundant signal to which the useful pulse belongs, the same material coding is added to the spectrogram corresponding to the useful pulse to obtain a set of material images with material number labels.
[0096] Step 5: Taking the useful pulse group as the basic unit, respectively extract the time domain and frequency domain features from the multiple useful pulses contained in each useful pulse group, and respectively extract the gray level co-occurrence matrix features from the multiple spectrograms corresponding to the multiple useful pulses.
[0097] According to the channel number, that is, the acoustic emission sensor number, the time domain features, frequency domain features and gray-level co-occurrence matrix features of multiple useful pulses are connected front and back and arranged in a row to construct a feature vector.
[0098] A large number of useful pulse groups construct a large number of feature vectors to obtain a feature vector set, which is then replicated to obtain two feature vector sets.
[0099] For the first feature vector set, according to the spatial number of the multi-channel redundant signal to which the useful pulse group belongs, the same spatial number is added to the feature vector corresponding to the useful pulse group, so as to obtain a preliminary positioning data set with spatial number labels.
[0100] For the second feature vector set, according to the material coding of the multi-channel redundant signal to which the useful pulse group belongs, the same material coding is added to the feature vector corresponding to the useful pulse group to obtain a preliminary material data set with material number labels.
[0101] The preliminary positioning dataset and the preliminary material dataset are processed using a channel-weighted feature selection method to screen the features that effectively feedback the position information and material information, respectively, to construct a joint positioning feature library and a joint material feature library, thereby establishing the positioning dataset and the material dataset.
[0102] Step 6: Train multiple neural networks on the positioning image set and the material image set respectively, compare and obtain two neural networks with outstanding performance, and optimize their hyperparameters to obtain two optimal neural networks, called the optimal positioning-neural network and the optimal material-neural network.
[0103] Multiple classifiers are trained on the positioning dataset and the material dataset respectively, and two outstanding classifiers are obtained by comparison. Their parameters are optimized to obtain two optimal classifiers, which are called the optimal positioning-classifier and the optimal material-classifier.
[0104] At this point, the research task of the first research object transfer was completed.
[0105] Step 7: Place multiple acoustic emission sensors at different positions on the surface of the sealed electronic device to be tested, and connect the acoustic emission sensors to the automatic detection device for unwanted objects. Collect a group of multi-channel unwanted object signals with the help of the automatic detection device for unwanted objects, which is called the unwanted object signal group to be tested. Use a two-level adaptive energy threshold pulse extraction algorithm to extract pulses from each segment of the unwanted object signal in the unwanted object signal group to be tested, and obtain multiple useful pulses. Use an improved threshold judgment search pulse matching algorithm to perform pulse matching on the unwanted object signal group to be tested, and obtain multiple useful pulse groups.
[0106] Step 8: Convert multiple useful pulses into multiple spectrograms and establish a set of images to be tested. Extract the time domain and frequency domain features in the joint positioning feature library from the multiple useful pulses contained in each useful pulse group, and extract the gray-level co-occurrence matrix features in the joint positioning feature library from the multiple spectrograms corresponding to the multiple useful pulses. According to the channel number, the time domain features, frequency domain features, and gray-level co-occurrence matrix features of the multiple useful pulses are connected front and back and arranged in a column to construct a feature vector.
[0107] Multiple useful pulse groups construct multiple feature vectors to establish a positioning data set to be measured. Similarly, by extracting the time domain and frequency domain features in the joint material feature library on the multiple useful pulses contained in each useful pulse group, and extracting the gray level co-occurrence matrix features in the joint material feature library on the multiple spectrograms corresponding to the multiple useful pulses, multiple useful pulse groups can construct multiple feature vectors, thereby establishing a material data set to be measured.
[0108] Step 9: Use the optimal positioning-neural network and the optimal material-neural network to make predictions on the image set to be tested, and obtain prediction labels for multiple positions and materials, which are called the spectrogram-position label set and the spectrogram-material label set.
[0109] The optimal positioning-classifier is used to predict on the positioning data set to be measured, and the predicted labels of multiple positions are obtained, which are called feature vector-position label sets. The optimal material-classifier is used to predict on the material data set to be measured, and the predicted labels of multiple materials are obtained, which are called feature vector-material label sets.
[0110] Step 10: Perform majority voting on the spectrogram-position label set and the feature vector-position label set respectively to obtain common position label 1 and common position label 2. Combine the spectrogram-position label set and the feature vector-position label set, and perform majority voting again to obtain common position label 3. Perform majority voting on common position label 1, common position label 2, and common position label 3 to obtain the final position label. That is, the construction of the combined positioning framework is completed, and the positioning of the redundant objects is completed.
[0111] A majority vote is performed on the spectrogram-material label set and the feature vector-material label set respectively to obtain common material label 1 and common material label 2. The spectrogram-material label set and the feature vector-material label set are combined and a majority vote is performed again to obtain common material label 3. A majority vote is performed on common material label 1, common material label 2, and common material label 3 to obtain the final material label. That is, the construction of the combined material framework is completed, and the identification of redundant materials is completed.
[0112] At this point, the second research task of transferring research subjects has been completed.
[0113] Example
[0114] Sealed electronic equipment: Select the core components of a certain type of aerospace equipment as sealed electronic equipment, the actual picture is as follows Figure 2 shown. Figure 2 In the figure, (a) is a picture of it being completely enclosed, and (b) is a picture of the inner space after it is opened.
[0115] As can be seen from the figure, the internal space of the sealed electronic device is interconnected, and there is no independent enclosed space. At the same time, there are three inner walls and seven standing columns in the internal space. Taking the seven columns as a reference and considering the existing structure of the internal space, seven thin plates are used to divide the internal space into seven independent enclosed spaces. Among them, considering that a part of the original inner wall is not high enough to completely isolate the two enclosed spaces, the present invention uses a thin plate to fill this missing part. In order to clearly show the division effect of the internal space of the sealed electronic device, the present invention uses Solidworks to draw a 3D model of the sealed electronic device, as shown in the figure. Figure 3 At the same time, Figure 3 (b) shows the actual effect of adding thin plates to the internal space. On this basis, the seven confined spaces are numbered in sequence. The numbering information is referenced Figure 3 In (b), the device model is obtained for subsequent experiments.
[0116] Excess material samples: According to the latest excess material detection information in real application scenarios, inspectors often detect excess metal materials such as iron, copper, tin and aluminum from sealed electronic equipment. Their weights are all between 0.8mg and 1.0mg. In addition, silicone has also been detected, although the probability is not high. It is regarded as non-metallic excess material with a weight of about 10mg. Therefore, five types of excess material samples were produced, namely aluminum, tin, iron, copper and silicone, and their weights are 1.0mg, 1.0mg, 0.8mg, 1.0mg and 10mg respectively. Figure 4 A physical picture of the excess sample is given.
[0117] Special equipment: The specific name of the special equipment is DZJC-III type redundant material automatic detection equipment. Figure 5 It consists of three parts: four acoustic emission sensors, a signal processing unit and an embedded computer.
[0118] Four acoustic emission sensors are placed at different positions on the surface of the sealed electronic device to collect the unwanted object signal and transmit it to the signal processing unit. The specific model of the acoustic emission sensor is the PXR04 resonant acoustic emission sensor. Its resonant frequency is 40kHz, and its frequency bandwidth at a sensitivity of 60dB is 15kHz to 165kHz. The minimum amplitude of the unwanted object signal is 12mV, and the frequency bandwidth is 20kHz to 120kHz. Therefore, the PXR04 resonant acoustic emission sensor can sensitively collect unwanted object signals.
[0119] The signal processing unit includes a signal conditioning circuit, a signal acquisition circuit, a signal transmission circuit, etc. The signal conditioning circuit is used to complete the signal amplification and signal filtering of the four-channel redundant signal acquired in a single time. The signal acquisition circuit is used to complete the conversion of the four-channel redundant signal from an analog signal to a digital signal. The signal transmission circuit is used to transmit the four-channel redundant signal in a digital format to an embedded computer.
[0120] The embedded computer is used to run and display the signal acquisition software. During the experiment, the tester uses the software to control the dedicated equipment to collect the four-channel redundant signal, save the four-channel redundant signal sent by the signal processing unit in the local memory, and display the currently collected set of four-channel redundant signals in the main interface.
[0121] Experimental conditions: In this embodiment, according to the requirements for carrying out equipment-level redundant object detection specified in the Chinese GJB65B standard, corresponding experimental conditions are set, as shown in Table 1.
[0122] Table 1 Experimental conditions specified in GJB65B
[0123]
[0124] Signal acquisition: Referring to step 2 of the specific implementation method, the waste samples of five materials are placed in the seven enclosed spaces of the device model in turn. After each placement, the device model is repackaged and fixed to the vibration table. Referring to Table 1, the experimental conditions of the vibration table are set to work to activate the waste signal inside the device model and generate waste signals. The four acoustic emission sensors of the dedicated equipment are placed at different positions on the surface of the device model to collect four-channel waste signals, which are finally saved in the local memory of the embedded computer. Figure 6A physical diagram of the process of collecting the unwanted object signal is given. It is worth noting that the figure shows the positions of the four acoustic emission sensors, as marked by the red circles. Such a sensor layout can ensure that the unwanted object signals generated in any confined space are fully collected. In fact, a suitable sensor layout can be designed according to the actual experimental object.
[0125] After a large number of balanced experiments, 100 groups of four-channel redundant signals representing each material were collected and placed in each confined space, totaling 100×5×7=3500 groups of four-channel redundant signals. Each group of four-channel redundant signals was labeled with the location information and material information of the redundant samples.
[0126] Pulse preprocessing: Based on a large number of four-channel redundant signals, pulse preprocessing is performed on them respectively, including pulse extraction and pulse matching. Among them, a new two-level adaptive energy threshold pulse extraction algorithm is proposed to extract useful pulses in each segment of redundant signals, and an improved threshold judgment search pulse matching algorithm is used to match useful pulses that are temporally corresponding in each group of four-channel redundant signals.
[0127] S31, two-level adaptive energy threshold pulse extraction algorithm:
[0128] According to the 3σ distribution criterion of the normal distribution, if the mean of a batch of values is μ and the standard deviation is σ, then the probability that the value is distributed in [μ-σ,μ+σ] is 68.27%, and the probability that the value is distributed in [μ-3σ,μ+3σ] is 99.73%. Scholars usually regard values lower than μ-3σ or higher than μ+3σ as abnormal values, among which μ+3σ is an abnormally large value, and regard values hovering around μ-σ and μ+σ as valid values. Similarly, assuming that the average energy of a signal is E a , the standard deviation of energy is σ E Those with energies higher than E a +3σ E The signal component of E is the abnormally large value of the entire signal, that is, the peak area of the useful pulse. a +σ E The signal component near is the effective value of the entire signal, that is, the end point area of the useful pulse. It is worth noting that the present invention selects E a +σ E Instead of e a -σ E It follows the idea of less but better, that is, energy hovers around E a +σ E The nearby signal components are more reliable as useful pulses, and their energy hovers around E a -σ E Nearby signal components may contain some background noise. Figure 7A schematic diagram of a two-stage adaptive energy threshold pulse extraction algorithm is given, and its specific implementation steps are as follows:
[0129] Step 1: Calculate the average energy of the signal, which is the average of the sum of the squares of the amplitudes of all sampling points it contains, expressed as e a The standard deviation of the energy of the signal is calculated and expressed as σ E . Set the peak energy threshold to E p =E a +3σ E , set the endpoint energy threshold to e be =e a +σE.
[0130] Step 2: Perform frame processing on the signal, set the duration of each frame signal Δt=50 μs, so that the signal is divided into multiple frame signals. Calculate the energy of each frame signal.
[0131] The sampling frequency of the dedicated device used in the present invention is 500 kHz, and each frame signal contains twenty-five sampling points.
[0132] Step 3: Starting from the first frame signal, compare the energy of each frame signal with E p When the energy of a frame signal is greater than E p When the maximum frame signal of the first useful pulse is about to appear, record this frame signal; and start from it, continue to compare the energy of each subsequent frame signal with E p , until the energy of a frame signal is less than E p Find the frame signal with the largest energy from these frame signals and identify it as the maximum frame signal of the first useful pulse. The time of obtaining the sampling point with the largest amplitude in the maximum frame signal is called the peak time t of the first useful pulse. p_1 .
[0133] Step 4: Take the peak time t of the first useful pulse extracted in step 3 p_1 As the starting point, compare the energy of each frame signal with E be Until the energy of a frame signal is found to be less than E in both directions be The previous frame signals of the two frame signals are respectively identified as the first start frame signal and the first end frame signal of the first useful pulse. The time of obtaining the first sampling point in the first start frame signal is called the first start time t of the first useful pulse. fb_1 The time of obtaining the last sampling point in the first end frame signal is called the first end time t of the first useful pulse. fe_1 .
[0134] Step 5: Take the first end time t of the first useful pulse extracted in step 4fe_1 The next frame signal is taken as the new starting point, and the same processing of step 3 and step 4 is repeated to extract the remaining useful pulses in the signal, and then all the useful pulses extracted from the signal are saved. At the same time, the peak moments of all useful pulses are saved. First starting moment and the first ending moment
[0135] At this point, the first stage of pulse extraction is completed.
[0136] Step 6: Calculate the average energy and standard deviation of each useful pulse saved in step 5, expressed as E i (i=1,2,…,n) and σ E_i (i=1,2,…,n). On this basis, set the exclusive endpoint energy threshold of each useful pulse as E be_i =E i +σ z_i (i=1,2,…,n).
[0137] Step 7: Starting from the first useful pulse saved in step 5, take its peak time t extracted in step 3 p_1 As the starting point, the energy of each frame signal is compared with the exclusive endpoint energy threshold E be_1 Until the energy of a frame signal is found to be less than E in both directions be_1 The previous frame signal of the two frame signals is respectively identified as the second start frame signal and the second end frame signal of the first useful pulse.
[0138] The time of obtaining the first sampling point in the second starting frame signal is called the second starting time t of the first useful pulse. sb_1 The time of obtaining the last sampling point in the second end frame signal is called the second end time t of the first useful pulse. se_1 .
[0139] Step 8: Starting from the second useful pulse saved in step 5, repeat the same processing steps as step 7 to obtain the second starting time of the remaining useful pulses saved in step 5 respectively. and the second end moment
[0140] Step 9: Calculate the first duration t of each useful pulse saved in step 5 fl_i =t fe_i -t fb_i (i=1,2,…,n), second duration t sl_i =t se_i -t sb_i (i=1,2,…,n), and duration ratio If the useful pulse τ i If it is greater than the duration threshold T, then its first starting time t fb_i With the first end time t fe_i The final starting time t b_i and the final ending time t e_i Otherwise, set its first starting time t fb_i and the second end time t se_i The final starting time t b_i and the final ending time t e_i Thus, the final starting time t of each useful pulse saved in step 5 is obtained. b_i (i=1,2,…,n) and the final end time t e_i (i=1,2,…,n) and record it.
[0141] At this point, the second stage of pulse extraction is completed. The two-level adaptive energy threshold pulse extraction algorithm is used to process 3500 groups of four-channel redundant signals, that is, 4×3500=14000 redundant signals, and the useful pulses in each section of the redundant signal are extracted, and a total of 1386784 useful pulses are obtained.
[0142] It should be noted that in the embodiment, the duration threshold T is set to 0.82. Referring to the 3σ distribution criterion of the normal distribution, the probability that the value is distributed in [μ-σ, μ+σ] is 68.27%, and the probability that the value hovers in μ+σ is about 84.13%. Similarly, for independent useful pulses, the exclusive endpoint energy threshold E be_i The ratio of the extracted signal components to the useful pulses should also be about 84.13%. For a large pulse composed of multiple useful pulses that appear continuously, its distribution is a mixture of the normal distributions of multiple independent useful pulses. In this way, the useful pulses with larger energy dominate the distribution characteristics, resulting in the set exclusive endpoint energy threshold being larger than the exclusive endpoint energy threshold applicable to the useful pulses with smaller energy. This actually discards the useful pulses with smaller amplitudes in the continuous multiple pulses and only retains the useful pulses with the largest energy, so as to achieve the purpose of solving the problem of continuous multiple pulses. Therefore, the duration threshold is set to be slightly less than 84.13%, which not only ensures that it has sufficient capacity to solve the problem of continuous multiple pulses, but also retains a moderate amount of flexibility to allow for partial degradation in actual use.
[0143] S32, improved threshold determination search pulse matching algorithm:
[0144] After the pulse extraction is completed, a pulse matching algorithm is needed to find four useful pulses that are temporally corresponding in the four-channel redundant signal. Every four useful pulses form a useful pulse group, and a group of four-channel redundant signals can obtain multiple useful pulse groups. This is the basis of feature engineering. In previous studies, a threshold judgment search pulse matching algorithm was proposed, which uses the start time of the first useful pulse in the four-channel redundant signal, that is, the four-segment redundant signal as a reference. In actual use, the redundant signal collected by the acoustic emission sensor close to the redundant object is richer, and the redundant signal collected by the acoustic emission sensor far from the redundant object is relatively lacking. If the amplitude of the first useful pulse is small, then in the redundant signal collected by the acoustic emission sensor at a close distance, the first useful pulse can still be distinguished from the background noise and can be extracted by the two-stage adaptive energy threshold pulse extraction algorithm. In the redundant signal collected by the acoustic emission sensor at a long distance, the first useful pulse is almost submerged in the background noise and cannot be extracted by the two-stage adaptive energy threshold pulse extraction algorithm. This results in the first useful pulse extracted from the redundant signal collected by the acoustic emission sensor at a long distance not being the real first useful pulse, thus causing inaccurate pulse matching. The present invention improves the threshold determination search pulse matching algorithm and adjusts the useful pulse used for reference. Figure 8 A schematic diagram of an improved threshold judgment search pulse matching algorithm is given, and its specific implementation steps are as follows.
[0145] Step 1: Obtain the useful pulses with the largest energy in the four-channel redundant signals, that is, in the four segments of redundant signals, respectively, which are called four reference useful pulses. According to the starting time of the four reference useful pulses, determine the redundant signal collected by the acoustic emission sensor farthest from the redundant object, which is called the reference signal. Determine the redundant signal collected by the acoustic emission sensor closest to the redundant object, which is called the comparison signal. Obtain the starting time of the reference signal and the comparison signal, and calculate the difference between the two starting times, which is called the starting time threshold T.
[0146] Step 2: Denote the useful pulses contained in the reference signal in step 1 as P i (i=1,2,…,n), the starting times of these useful pulses are t i (i=1,2,…,n). According to the starting time threshold T, the starting time threshold interval of each useful pulse is set to [t i -T,t i ](i=1,2,…,n).
[0147] Step 3: Starting from P1, search for useful pulses with starting time in [t1-T, t1] in the other three redundant signals. If three useful pulses can be found, combine them with P1 to form a pulse group. Otherwise, skip P1, indicating that its matching is unsuccessful.
[0148] Step 4: Refer to step 3 and i (i=2,3,…,n) to perform pulse matching. Thus, the pulse matching of the four-channel redundant signal is completed, and multiple pulse groups are obtained.
[0149] The present invention uses an improved threshold judgment search pulse matching algorithm to process 3500 groups of four-channel redundant signals respectively, matches the useful pulses that correspond in time in each group of four-channel redundant signals, and obtains the useful pulse group in each group of four-channel redundant signals, thereby obtaining 287692 useful pulse groups, totaling 1150768 useful pulses.
[0150] Feature engineering: Each useful pulse group contains four useful pulses. A large number of useful pulse groups means a larger number of useful pulses. Useful pulses can be converted into spectrograms to construct a picture set and then train the neural network. Useful pulses can also extract classic time domain and frequency domain features to construct feature data and then train the classifier. At the same time, the spectrogram can also extract picture features to merge with time domain and frequency domain features to construct feature data. In addition, considering the correlation between the four-channel redundant signals, called the four-channel characteristics, it is necessary to extract the above features on the four useful pulses of the useful pulse group at the same time to construct feature data. 3500 groups of four-channel redundant signals are simultaneously labeled with the location information and material information of the redundant samples, so that a large number of useful pulses and useful pulse groups also carry location information and material information. The neural network can feed back the feature extractor according to the output to adaptively extract or process feature data from the picture and build a complete network structure. Therefore, the spectrogram can be input into the neural network without adjustment, and it will adaptively adjust its own network structure according to the positioning or material recognition task. However, the composition structure of the feature data has a huge impact on the classifier. It is necessary to determine the feature data of appropriate composition structure to ensure that the trained classifier can complete the positioning or material recognition task well. This requires feature selection to construct positioning data sets and material data sets dedicated to positioning and material recognition respectively.
[0151] Spectrogram: A spectrogram is a two-dimensional image converted from a one-dimensional signal to better display the time domain and frequency domain characteristics of the signal. When constructing a spectrogram, the signal is first pre-emphasized, then windowed, and then the windowed signal is subjected to a fast Fourier transform (FFT), and finally the signal is converted into a spectrogram. Considering that the present invention converts useful pulses into a spectrogram, the length of the FFT is set to 1024, the window function is set to a Hamming window, and the window length is set to 50ms. The details of the construction of the spectrogram are as follows.
[0152] First, a high-pass filter is used to pre-emphasize the useful pulse. The principle is shown in formula (1):
[0153] F(x)=1-μx -1 (1)
[0154] Wherein, μ is the pre-emphasis coefficient, and μ=0.97 is usually set.
[0155] Secondly, the useful pulse after pre-emphasis is windowed to make it continuous. The principle is shown in formula (2):
[0156] S w (n) = S (n) × w (n) (2)
[0157] Among them, S(n) is the original useful pulse, w(n) is the window function, S w (n) is the useful pulse after windowing. The calculation formula of the Hamming window is:
[0158]
[0159] Then, the useful pulse after windowing is subjected to FFT, and the power spectrum is passed through a bandpass filter to obtain the spectrogram corresponding to the useful pulse.
[0160] Fig. 9 The spectrogram constructed by a useful pulse is given. As shown in the figure, the spectrogram contains multiple block areas, each block area has different color depths, and most of the multiple continuous block areas in the figure have little difference in color depth, which is relatively close. There are also some abrupt block areas with particularly dark or light colors. This is related to the time domain and frequency domain characteristics of the redundant signal and needs to be further studied.
[0161] In the previous section, we obtained 287,692 useful pulse groups, a total of 1,150,768 useful pulses. Therefore, the useful pulses are converted into spectrograms, a total of 1,150,768 spectrograms are obtained, and a picture set is constructed. Two picture sets are obtained by copying. On this basis, according to the position information of the redundant signal where the useful pulse is located in the spectrogram corresponding to the useful pulse, the position information, i.e., the position label, is added to the spectrogram in one of the picture sets, thereby dividing the 1,150,768 spectrograms into seven categories and constructing a positioning picture set. According to the material information of the redundant signal where the useful pulse is located in the spectrogram corresponding to the useful pulse, the material information, i.e., the material label, is added to the spectrogram in another picture set, thereby dividing the 1,150,768 spectrograms into five categories and constructing a material picture set.
[0162] Joint feature library: In addition to converting useful pulses into spectrograms to construct a picture set, features can also be extracted from useful pulses to construct a data set. In previous studies, eleven time domain and frequency domain features were extracted, as shown in Table 2.
[0163] Table 2 Detailed description of eleven time domain and frequency domain features
[0164]
[0165] In addition, the present invention also extracts image features from the spectrogram converted by useful pulses, mainly grayscale co-occurrence matrix features. Grayscale co-occurrence matrix is a feature extraction technology widely used in image texture analysis. It effectively captures the texture information of the image by analyzing the frequency and spatial relationship between pixels of different gray levels in the image. Therefore, the grayscale co-occurrence matrix can effectively analyze the texture differences between block areas in the constructed spectrogram. The selected grayscale co-occurrence matrix features include energy, entropy, contrast, mean, homogeneity and correlation, and their specific descriptions are shown in Table 3. In addition, a description of the variables shown in the table is given. P(i,j) is the element located at (i,j) in the normalized co-occurrence matrix. N is the number of gray levels of the image. Wherein, i and j represent different pixels, respectively. μ x and μ y Respectively represent the mean of pixel grayscale values i and j in the image. x and σ y Represent the standard deviation of pixel grayscale values i and j in the image respectively.
[0166] Table 3 Detailed description of gray level co-occurrence matrix features
[0167]
[0168] Thus, a joint feature library is constructed, including eleven time domain and frequency domain features and six gray-level co-occurrence matrix features. 287692 useful pulse groups were obtained above, with a total of 1150768 useful pulses. 1150768 spectrograms corresponding to useful pulses were obtained above. Eleven time domain and frequency domain features are extracted from the four useful pulses contained in each useful pulse group, and six gray-level co-occurrence matrix features are extracted from the corresponding four spectrograms, and a total of (11+6)×4=68 feature values are obtained. Considering the four-channel characteristics between the four-channel redundant signals, these feature values are arranged in the order of "time domain features, frequency domain features and gray-level co-occurrence matrix features of the first channel useful pulses, time domain features, frequency domain features and gray-level co-occurrence matrix features of the second channel useful pulses, time domain features, frequency domain features and gray-level co-occurrence matrix features of the third channel useful pulses, and time domain features, frequency domain features and gray-level co-occurrence matrix features of the fourth channel useful pulses" to obtain a feature vector. The channel numbers mentioned here are the numbers of the four acoustic emission sensors of the special equipment, that is, the unwanted object signal collected by acoustic emission sensor No. 1 is the unwanted object signal of the first channel among the four-channel unwanted object signals, and the useful pulses contained therein are the useful pulses of the first channel.
[0169] Through the same processing steps, 287,692 feature vectors are obtained based on 287,692 useful pulse groups and 1,150,768 spectrograms to construct a data set. Two data sets are obtained by copying. On this basis, according to the location information of the redundant signal where the useful pulse group is located, the location information, i.e., the location label, is added to the feature vector in one of the data sets, thereby dividing the 287,692 feature vectors into seven categories to construct a preliminary positioning data set. According to the material information of the redundant signal where the useful pulse group is located, the material information, i.e., the material label, is added to the feature vector in another data set, thereby dividing the 287,692 feature vectors into five categories to construct a preliminary material data set.
[0170] Feature selection: In machine learning, some feature data in a data set have little effect on improving the classification performance of a classifier, and may even have a counterproductive effect. In this way, the classification accuracy obtained by a classifier on a data set containing all feature data is lower than the classification accuracy obtained on a data set containing only valid feature data. Feature selection is the process of screening out from a data set those feature data that contribute greatly to the classification performance of the classifier. In the present invention, the feature data of the preliminary positioning data set and the preliminary material data set are the same, but the labels of the feature vectors are different, which leads to different classification tasks for the classifiers trained on the two data sets, and different classification performances of the classifiers. Therefore, it is necessary to perform feature selection on the preliminary positioning data set and the preliminary material data set respectively, screen out those features that contribute greatly to the classification performance of the classifier respectively, and construct a joint positioning feature library dedicated to redundant object positioning and a joint material feature library dedicated to redundant object material recognition, thereby constructing a positioning data set and a material data set.
[0171] Considering the four-channel characteristics between the four-channel redundant signals, the 68 features cannot be selected as individuals, but should be selected as 17 similar feature combinations. In previous studies, a feature selection method based on channel weighting was proposed, which is dedicated to the feature selection of four-channel redundant signals. Its specific implementation steps are as follows.
[0172] Step 1: Use formula (4) to calculate the absolute value of the Pearson correlation coefficient between each feature and label in the data set and express it as r ij (i=1,2,3,4,j=1,2,…,17). i represents the channel to which the feature belongs, and j represents the sequence number of the feature.
[0173]
[0174] in, and σX represent the feature X i′ Standard score, mean and standard deviation of Y i′ refers to the label value of the sample, and σY represent the label Y i′ The standard score, mean, and standard deviation of
[0175] Step 2: Use channels as the division unit, that is, r ij i is the division unit, and the absolute value of the Pearson correlation coefficient between the 17 features and the label belonging to the same channel is r ij Accumulate and get four summation results corresponding to the four channels, expressed as R i (i=1,2,3,4). The calculation formula is as follows.
[0176]
[0177] Among them, r ij Represents the absolute value of the Pearson correlation coefficient between the feature with sequence number j in channel i and the label.
[0178] Step 3: Calculate the absolute value of the Pearson correlation coefficient between the 68 features and the label, expressed as R all Calculate R i (i=1,2,3,4) accounts for R all The proportion of the channel is called the channel weighting coefficient, expressed as W i p (i=1,2,3,4), the calculation formula is as follows.
[0179]
[0180] in, R i Represents the sum of the absolute values of the Pearson correlation coefficients between the 17 features and the label in channel i.
[0181] Step 4: According to r ij The i in (i=1,2,3,4,j=1,2,…,17) represents the channel to which it belongs, and multiplies it by the corresponding channel weighting coefficient W i p (i=1,2,3,4). For example, r 1j (j=1,2,…,17) are the 17 features belonging to the first channel, which are multiplied by the channel weighting coefficient W1 of the first channel. p In this way, 68 new weighted values are obtained, expressed as The calculation formula is as follows.
[0182]
[0183] Among them, r ij represents the absolute value of the Pearson correlation coefficient between the feature with sequence number j and the label in channel i, W i p Represents the channel weighting coefficient corresponding to channel i.
[0184] Step 5: Divide the 68 features into 17 similar feature combinations based on the same feature combination. On this basis, add up the new values of the four features in the same feature combination. The new value and of 17 similar feature combinations are obtained, expressed as The calculation formula is as follows.
[0185]
[0186] in, The new value representing the absolute value of the Pearson correlation coefficient between the feature with sequence number j and the label in the weighted channel i, W i p Represents the channel weighting coefficient corresponding to channel i.
[0187] Step 6: Arrange in descending order, and get The corresponding sort number N j (j=1,2,…,17).
[0188] Step 7: Set the sort number N j The top 8 to 17 similar feature combinations are retained respectively, and 10 new data sets with different features are constructed. The random forest with default parameter configuration is used to perform 10-fold cross validation on the 10 new data sets to obtain 10 average classification accuracies. The new data set on which the random forest achieves the highest average classification accuracy is compared, and the features of the new data set are the best, and the corresponding feature selection result is the optimal one.
[0189] The preliminary positioning dataset and preliminary material dataset are processed by the feature selection method based on channel weighting, thereby constructing a joint positioning feature library and a joint material feature library, which contain 15 features and 14 features respectively. Their detailed descriptions are shown in Tables 4 and 5.
[0190] Table 4 Joint positioning feature library
[0191]
[0192] Table 5 Joint material feature library
[0193]
[0194] According to the joint positioning feature library, 15 similar feature combinations in the preliminary positioning data set, that is, all feature values of 60 features, are retained, and all feature values of 8 redundant features are removed to construct the positioning data set. Similarly, according to the joint material feature library, 14 similar feature combinations in the preliminary material data set, that is, all feature values of 56 features, are retained, and all feature values of 12 redundant features are removed to construct the material data set.
[0195] So far, the positioning picture set, positioning data set, material picture set and material data set are constructed. Their specific descriptions are shown in Table 6 and Table 7 respectively.
[0196] Table 6 Detailed description of positioning image set and positioning dataset
[0197]
[0198]
[0199] Table 7 Detailed description of the texture image set and texture dataset
[0200]
[0201] Neural network training: This paper selects five neural networks that are currently popular and widely used by scholars for image classification, namely AlexNet, ResNet50, GoogLeNet, VGG19 and MobileNet. They cover different network scales and complexities and are widely representative. Table 8 lists the network structures of the five neural networks.
[0202] Table 8 Network structure of neural network
[0203]
[0204] The above five neural networks are trained on the positioning image set and the material image set respectively, 150 epochs are set, and their average classification accuracy is statistically calculated, as shown in Table 9.
[0205] Table 9 Average classification accuracy achieved by neural network
[0206]
[0207] As can be seen from the table, whether in the positioning image set or the material image set, VGG19 achieved the highest average classification accuracy of 95.16% and 97.56% respectively, and has obvious advantages over other neural networks. In addition, AlexNet, ResNet50 and GoogLeNet also performed well, showing that they do have advantages in the field of image classification. In contrast, MobileNet achieved the lowest average classification accuracy, which may be due to its lightweight structure, focusing on reducing its own network structure and computational complexity, without excessive pursuit of classification performance. Fig.10 and Fig.11 The classification accuracy and loss curves of VGG19 after 150 epochs on the positioning picture set and the material picture set are given respectively. Fig.10 The classification results of VGG19 on the positioning picture set, where (a) is the classification accuracy curve and (b) is the Loss curve. Fig.11 The classification results of VGG19 on the texture image set, where (a) is the classification accuracy curve and (b) is the Loss curve.
[0208] As can be seen from the figure, as the epoch increases, the difference between the classification accuracy of VGG19 on the training set and the validation set of the positioning picture set is not obvious, and they almost overlap, but the difference between the Losses increased, and the difference is more obvious, and even oscillation occurs. Similarly, as the epoch increases, the difference between the classification accuracy and Loss of VGG19 on the training set and the validation set of the material picture set increases, but in general, the difference between the classification accuracy is not large, and the difference between the Losses is more obvious. This proves that the classification performance and generalization ability of VGG19 need to be further improved. Therefore, we further analyze the internal structure of VGG19 and try to carry out hyperparameter optimization to further improve its classification performance and generalization ability.
[0209] Hyperparameter optimization: An in-depth analysis of the internal structure of VGG19 revealed that its difference from other neural networks is mainly reflected in the repeated use of multiple 3×3 small convolution kernels. At the same time, VGG19 uses the ReLU activation function to increase nonlinear processing capabilities. In addition, compared with the previous version of VGG16, VGG19 has increased the number of convolution layers, enabling it to capture more complex features. The present invention considers exploring how the internal structure of VGG19 can provide classification advantages when processing spectrograms from two perspectives. First, the effect of repeatedly using multiple 3×3 small convolution kernels and directly using large convolution kernels in improving VGG19's ability to capture details will be evaluated. Secondly, the role of the ReLU activation function will be explored in depth. Through analysis from these two perspectives, the network structure that plays a positive role in the classification performance of VGG19 is retained, and the network structure with room for improvement is optimized.
[0210] In image classification tasks, the configuration of convolutional layers plays a key role in the learning and generalization capabilities of neural networks. VGG19 has achieved good results in processing visual tasks by repeatedly using multiple 3×3 small-sized convolutional kernels and relying on the structure of deep stacked convolutional layers. In addition, other models such as AlexNet and ResNet use larger convolutional kernels (11×11 and 7×7, respectively) in the initial stage of the network. This design can not only capture a wider range of information in the input image, but also achieve a larger receptive field with fewer layers, helping the neural network to learn richer feature representations in the early stages. Based on this, an attempt is made to make the first part of adjustments to VGG19. The first-layer convolution kernel of VGG19 is modified from 3×3 to 7×7, in an attempt to enable it to learn more features in the early stages and further improve its understanding and processing capabilities of spectrograms. The optimized VGG19 is called VGG19-Plus, and its internal structure is as follows: Fig.12The newly designed PReLU-VGG19-Plus model of the present invention improves the classification performance and generalization ability of the traditional VGG19. The first-layer convolution kernel of VGG19 is modified from 3×3 to 7×7, and the PReLU activation function is introduced to replace the original ReLU activation function, which effectively improves the neural network's ability to understand and process the spectrogram, as well as its ability to learn physical structures.
[0211] like Fig.10 and Fig.11 As shown in the figure, VGG19 has underfitting phenomenon when classifying the positioning image set and the material image set. Specifically, in the later stage of epoch iteration, the classification accuracy of VGG19 on the validation set is continuously lower than that on the training set, and the Loss obtained on the training set and validation set even oscillates. This problem is partly due to the fact that the four-channel redundant signal collected by the dedicated equipment contains a large number of negative signals, which are eventually submerged in the useful pulses. These negative signals also exist in the corresponding spectrogram. The ReLU activation function outputs zero when processing negative inputs, causing some neurons to be permanently inactivated during training, thereby limiting the neural network's ability to learn new data and affecting its generalization ability. To solve this problem, the PReLU activation function is newly introduced to replace the ReLU activation function. The optimized VGG19 is called PReLU-VGG19-Plus. Fig.13 The images of the PreLU activation function and the ReLU activation function are given. As can be seen from the figure, the PReLU activation function introduces a small, learnable gradient parameter α when processing negative inputs, which keeps the information flowing in the neural network, thus helping to improve its learning and generalization capabilities.
[0212] On this basis, PReLU-VGG19-Plus is trained again on the positioning image set and the material image set, and 150 epochs are set. The classification accuracy and loss curves obtained by them are respectively counted, as shown in the figure below: Fig.14 and Fig.15 shown. Fig.14 The classification effect of PReLU-VGG19-Plus on the positioning picture set, where (a) is the classification accuracy curve and (b) is the Loss curve. Fig.15 The classification results of PReLU-VGG19-Plus on the texture image set, where (a) is the classification accuracy curve and (b) is the Loss curve.
[0213] contrast Fig.14 and Fig.10 ,contrast Fig.15 and Fig.11, it can be found that compared with VGG19, with the increase of epochs, the difference between the classification accuracy and Loss of PReLU-VGG19-Plus on the training set and validation set of the positioning picture set is not obvious, especially the classification accuracy curves are almost overlapped, and the oscillation phenomenon of the Loss curve is suppressed, and it looks relatively stable overall. Similarly, with the increase of epochs, the difference between the classification accuracy and Loss of PReLU-VGG19-Plus on the training set and validation set of the material picture set is not obvious, the classification accuracy curve is almost invisible, and the Loss curve looks relatively stable overall. In general, compared with VGG19, the classification performance and generalization ability of PReLU-VGG19-Plus are significantly improved. It proves the feasibility and effectiveness of hyperparameter optimization for VGG19. On this basis, two PReLU-VGG19-Plus trained by the positioning picture set and the material picture set are obtained, called the optimal positioning-neural network and the optimal material-neural network.
[0214] Classifier training: Four classifiers that performed well in previous studies were selected, including radial basis function-based support vector machine (RBF-SVM), random forest, XGBoost, and one-dimensional convolutional neural network (1D-CNN). Ten-fold cross validation was performed on the positioning dataset and material dataset, respectively, and their average classification accuracy was statistically calculated, as shown in Table 10.
[0215] Table 10 Average classification accuracy achieved by the classifier
[0216]
[0217] As can be seen from the table, XGBoost achieves the highest average classification accuracy, 96.92% and 97.46% respectively, in both the positioning dataset and the material dataset. In addition, random forest achieves a similar average classification accuracy with a small gap. This shows the significant advantage of the integrated classifier. In fact, random forest and XGBoost also performed well in previous studies. RBF-SVM achieves good classification results. Although there is still a certain gap from XGBoost and random forest, it is significantly better than 1D-CNN. 1D-CNN is a classic convolutional neural network for data classification. Its classification performance is related to the learning rate and gradient update strategy set in the initial stage, and it is easy to fall into the trap of local optimality. Although it achieved the lowest average classification accuracy, it is also higher than 90%, which is higher than the average classification accuracy achieved by MobileNet in Table 9, showing that it still has unique advantages in data classification.
[0218] Overall, XGBoost performs well on both the positioning dataset and the material dataset, and parameter optimization is performed to further improve its classification performance.
[0219] Parameter optimization: The simulated annealing algorithm is used to optimize the parameters of XGBoost, and the values of each parameter when it performs best on the positioning data set and the material data set are obtained, which are called the optimal values of the parameters. Specifically, the parameters of XGBoost mainly include three parts: general parameters, model parameters and learning target parameters. Among them, the general parameters are not optimized and are set to the default configuration. For the model parameters, n_estimatores, max_depth and min_child_weight are mainly optimized, and the remaining model parameters are set to the default configuration. For the learning target parameters, learning_rate and gamma are mainly optimized, and the remaining learning target parameters are set according to the actual situation. For example, the Obj objective function is set to multi, which is applicable to multi-classification problems. On this basis, the simulated annealing algorithm is used to optimize the above five parameters on the positioning data set and the material data set respectively to find their optimal values. The parameter optimization results are shown in Table 11.
[0220] Table 11 Optimal values of XGBoost parameters
[0221]
[0222] On this basis, we applied parameter optimization XGBoost again to perform ten-fold cross validation on the positioning dataset and material dataset, and statistically analyzed their average classification accuracy, which was compared with the classification accuracy obtained by XGBoost before parameter optimization, as shown in Table 12.
[0223] Table 12 Average classification accuracy achieved by XGBoost before and after parameter optimization
[0224]
[0225] As can be seen from the table, the average classification accuracy of parameter-optimized XGBoost on the positioning dataset and material dataset has been improved by 0.41% and 0.25% respectively. This proves the feasibility and effectiveness of parameter optimization for XGBoost. On this basis, two parameter-optimized XGBoost trained on the positioning dataset and material dataset are obtained, called the optimal positioning-classifier and the optimal material-classifier.
[0226] Combined framework construction: In machine learning, neural networks are used to predict the labels of one or a batch of images, and classifiers are used to predict the labels of one or a batch of feature data. In the present invention, for a set of four-channel redundant signals collected in a single time, it is hoped to obtain the location information and material information of the redundant objects, thereby completing the redundant object positioning and material identification. However, as mentioned above, for a set of four-channel redundant signals collected in a single time, a two-level adaptive energy threshold pulse extraction algorithm is used to extract multiple useful pulses therein, and an improved threshold judgment search pulse matching algorithm is used to match useful pulses corresponding to each other in time to obtain multiple useful pulse groups. On this basis, multiple spectrograms are constructed, and feature data of multiple source joint positioning feature libraries and joint material feature libraries are constructed respectively. The optimal positioning-neural network is used to predict multiple spectrograms to obtain multiple predicted labels. The optimal positioning-classifier is used to predict the feature data of multiple source joint positioning feature libraries to obtain multiple predicted labels. The optimal material-neural network is used to predict multiple spectrograms to obtain multiple predicted labels. The optimal material-classifier is used to predict the feature data of multiple source joint material feature libraries to obtain multiple predicted labels. It can be found that no matter which neural network or classifier, they ultimately give multiple prediction labels instead of one prediction result. These prediction labels are essentially the results of data or image classification, not the redundant object positioning or material recognition results required in real application scenarios. There is a lack of data processing between the two.
[0227] In the present invention, the optimal positioning-neural network and the optimal positioning-classifier are trained to predict the spectrogram and feature data respectively, and multiple prediction labels are obtained respectively, which are called the spectrogram-position label set and the feature vector-position label set. Similarly, the optimal material-neural network and the optimal material-classifier are trained to predict the spectrogram and feature data respectively, and multiple prediction labels are obtained respectively, which are called the spectrogram-material label set and the feature vector-material label set. If a majority vote is performed on the spectrogram-position label set, the feature vector-position label set, the spectrogram-material label set and the feature vector-material label set respectively, two common position labels and two common material labels will be obtained respectively. At this time, it is not suitable to use majority voting because the number of inputs used for majority voting is an even number.
[0228] Therefore, on the basis of obtaining two common position labels and two common material labels, the spectrogram-position label set and the feature vector-position label set are merged, and a majority vote is performed again. When the labels are used for voting again, the spectrogram-position label set and the feature vector-position label set can be voted according to weights 0.8 and 0.2 to obtain a third common position label. The spectrogram-material label set and the feature vector-material label set are merged, and a majority vote is performed again to obtain a third common material label. On this basis, the three common position labels and the three common material labels are voted by majority again to obtain the final position label and the final material label, thereby constructing a combined positioning framework and a combined material framework, and obtaining the redundant object positioning and material recognition results. In this way, the transformation from the data or image classification results to the redundant object positioning or material recognition results is completed, and the triple majority voting process is used to ensure the stability of the final redundant object positioning and material recognition results.
[0229] Verification of generalization performance: Referring to the above-mentioned process, the redundant samples of five materials are placed in the seven confined spaces of the equipment model in turn. After each placement, the equipment model is repackaged and fixed to the vibration table. The vibration table is started to activate the redundant materials inside the equipment model to generate redundant signals. The four acoustic emission sensors of the special equipment are placed at different positions on the surface of the equipment model to collect four-channel redundant signals, and finally saved in the local memory of the embedded computer. After a large number of balanced experiments, 5 groups of four-channel redundant signals representing redundant samples of each material placed in each confined space are collected, for a total of 5×5×7=175 groups of four-channel redundant signals. Each group of four-channel redundant signals is labeled with the location information and material information of the redundant samples.
[0230] A two-level adaptive energy threshold pulse extraction algorithm was used to extract 65,304 useful pulses from 175 groups of four-channel redundant signals. An improved threshold judgment search pulse matching algorithm was used to match 14,525 useful pulse groups, which contained a total of 58,100 useful pulses. On this basis, the useful pulses were converted into 58,100 spectrograms to construct a verification picture set. Two verification picture sets were obtained by copying. By adding position labels and material labels to the spectrograms in the two verification picture sets, a verification positioning picture set and a verification material picture set were established. The features in the joint positioning feature library and the joint material feature library were extracted from the useful pulse group and the corresponding spectrogram, and 14,525 feature vectors were obtained to construct a verification data set. Two verification data sets were obtained by copying. By adding position labels and material labels to the feature vectors in the two verification data sets, a verification positioning data set and a verification material data set were established. The specific descriptions of the verification picture set and the verification data set are shown in Tables 13 and 14, respectively.
[0231] Table 13 Detailed description of the verification positioning image set and verification positioning data set
[0232]
[0233] Table 14 Detailed description of the verification material image set and verification material data set
[0234]
[0235]
[0236] The optimal positioning-neural network is applied to make predictions on the verification positioning picture set, the optimal material-neural network is applied to make predictions on the verification material picture set, the optimal positioning-classifier is applied to make predictions on the verification positioning data set, and the optimal material-classifier is applied to make predictions on the verification material data set, and their classification accuracies are statistically calculated, as shown in Table 15.
[0237] Table 15 Classification accuracy achieved by the optimal neural network and the optimal classifier
[0238]
[0239] Compared with the average classification accuracy of 95.16%, 97.56%, 96.92% and 97.46% obtained by the optimal positioning-neural network, optimal material-neural network, optimal positioning-classifier and optimal material-classifier, the decrease in their newly obtained classification accuracy is relatively low, which is 0.12%, 0.05%, 0.07% and 0.13% respectively, which is almost negligible. This fully demonstrates that the trained optimal positioning-neural network, optimal material-neural network, optimal positioning-classifier and optimal material-classifier have good generalization performance.
[0240] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.
Claims
1. A method for locating redundant objects and identifying materials of sealed electronic equipment based on multi-source information fusion, characterized in that: include: For the sealed electronic device to be tested, a signal of a measurement is obtained by simultaneously performing a measurement on multiple acoustic emission sensors. The signal corresponding to each acoustic emission sensor is recorded as a redundant signal of a channel. The number of multiple channels is recorded as N. Pulse extraction is performed on N sections of the corresponding redundant signal in N channels, and a section of the redundant signal in any channel obtains multiple useful pulses; pulse matching is performed on multiple useful pulses in the N channels, and after pulse matching, the N sections of the redundant signal corresponding to the N channels obtain multiple pulse groups, and one pulse group contains one useful pulse of each channel, and one pulse group is recorded as a useful pulse group; Taking the useful pulse as the basic unit, a spectrogram is generated for each useful pulse, and the gray-level co-occurrence matrix features are extracted for each spectrogram; At the same time, time domain and frequency domain features are extracted for each useful pulse; For each group of useful pulses, based on the time domain features, frequency domain features and gray-level co-occurrence matrix features corresponding to a group of useful pulses, a feature vector for positioning is obtained, recorded as a first identification feature vector, and / or, based on the time domain features, frequency domain features and gray-level co-occurrence matrix features corresponding to a group of useful pulses, a feature vector for material identification is obtained, recorded as a second identification feature vector; Then all the spectrograms are input into the positioning recognition neural network and / or the material recognition neural network respectively, and each spectrogram obtains a position prediction label of the redundant object in the electronic device and / or a material prediction label of the redundant object in the electronic device; a position prediction result is obtained by majority voting of all the position prediction labels and recorded as the first position prediction result, and / or, a material prediction result is obtained by majority voting of all the material prediction labels and recorded as the first material prediction result; At the same time, all the first identification feature vectors are respectively input into the positioning identification classifier, and each first identification feature vector obtains a position prediction label of the redundant object in the electronic device, and a position prediction result is obtained by majority voting through all the position prediction labels and recorded as the second position prediction result; and / or, all the second identification feature vectors are input into the material identification classifier, and each second identification feature vector obtains a material prediction label of the redundant object in the electronic device, and a material prediction result is obtained by majority voting through all the material prediction labels and recorded as the second material prediction result; All position prediction labels corresponding to the positioning recognition neural network and all position prediction labels corresponding to the positioning recognition classifier are combined and voted by majority to obtain a position prediction result and recorded as a third position prediction result, and / or, all material prediction labels corresponding to the material recognition neural network and all material prediction labels corresponding to the material recognition classifier are combined and voted by majority to obtain a material prediction result and recorded as a third material prediction result; The first position prediction result to the third position prediction result are used to perform majority voting to obtain a final position prediction result, and / or the first material prediction result to the third material prediction result are used to perform majority voting to obtain a final material prediction result.
2. The method for locating and identifying materials of redundant objects in sealed electronic equipment by multi-source information fusion according to claim 1 is characterized in that: When all the position prediction labels corresponding to the positioning recognition neural network and all the position prediction labels corresponding to the positioning recognition classifier are combined for majority voting, all the position prediction labels corresponding to the positioning recognition neural network and all the position prediction labels corresponding to the positioning recognition classifier are voted by majority according to the weights; and / or, When all material prediction labels corresponding to the material recognition neural network and all material prediction labels corresponding to the material recognition classifier are combined for majority voting, all material prediction labels corresponding to the material recognition neural network and all material prediction labels corresponding to the material recognition classifier are voted by majority according to the weights.
3. The method for locating and identifying materials of redundant objects in sealed electronic equipment by multi-source information fusion according to claim 2 is characterized in that: All the location prediction labels corresponding to the location recognition neural network and all the location prediction labels corresponding to the location recognition classifier are voted by majority according to the weights of 0.8 and 0.2; and / or, All material prediction labels corresponding to the material recognition neural network and all material prediction labels corresponding to the material recognition classifier are voted by majority with weights of 0.8 and 0.
2.
4. The method for locating and identifying materials of redundant objects in sealed electronic equipment by multi-source information fusion according to claim 1, characterized in that: The process of pulse extraction for N sections of redundant signals corresponding to N channels uses a two-level adaptive energy threshold pulse extraction algorithm to extract pulses from each section of redundant signals. The specific process includes the following steps: Step 1: Calculate the average energy E of the redundant signal a , and calculate the standard deviation σ of the signal energy E ; Set the peak energy threshold to E p =E a +3σ E , set the endpoint energy threshold to E be =E a +σ E ; Step 2: Frame the signal; Step 3: Starting from the first frame signal, compare the energy of each frame signal with E p ; When the energy of a frame signal is greater than E p When the frame signal is recorded, and starting from it, the energy of each subsequent frame signal is compared with E p , until the energy of a frame signal is less than E p ; Find the frame signal with the largest energy from these frame signals and identify it as the maximum frame signal of the first useful pulse; Get the time of the sampling point with the largest amplitude in the maximum frame signal, which is called the peak time t of the first useful pulse p_1 ; Step 4: Take the peak time t of the first useful pulse extracted in step 3 p_1 As the starting point, compare the energy of each frame signal with E be , until the energy of a frame signal is found to be less than Ebe in both directions; the previous frame signal of the two frame signals is respectively identified as the first starting frame signal and the first ending frame signal of the first useful pulse; the time of the first sampling point in the first starting frame signal is obtained, which is called the first starting time t of the first useful pulse fb_1 ; Get the time of the last sampling point in the first end frame signal, which is called the first end time t of the first useful pulse fe_1 ; Step 5: Take the first end time t of the first useful pulse extracted in step 4 fe_1 The next frame signal is taken as the new starting point, and the same processing process of step 3 and step 4 is repeated to extract the remaining useful pulses in the signal, and then all the useful pulses extracted from the signal are saved; at the same time, the peak moments of all useful pulses are saved. First starting moment and the first ending moment Step 6: Calculate the average energy and standard deviation of each useful pulse saved in step 5, expressed as E i and σ E_i On this basis, the exclusive endpoint energy threshold of each useful pulse is set to E be_i =E i +σ Z_i ; Step 7: Starting from the first useful pulse saved in step 5, take its peak time t extracted in step 3 p_1 As the starting point, the energy of each frame signal is compared with the exclusive endpoint energy threshold E be_1 Until the energy of a frame signal is found to be less than E in both directions be_1 ; Respectively identify the previous frame signal of the two frame signals as the second start frame signal and the second end frame signal of the first useful pulse; The time of obtaining the first sampling point in the second starting frame signal is called the second starting time t of the first useful pulse. sb_1 ; Get the time of the last sampling point in the second end frame signal, which is called the second end time t of the first useful pulse se_1 ; Step 8: Starting from the second useful pulse saved in step 5, repeat the same processing steps as step 7 to obtain the second starting time of the remaining useful pulses saved in step 5 respectively. and the second end moment Step 9: Calculate the first duration t of each useful pulse saved in step 5 fl_i =t fe_i -t fb_i , the second duration t sl_i =t se_i -t sb_i , and duration ratio If the useful pulse τ i If it is greater than the duration threshold T, then its first starting time t fb_i With the first end time t fe_i The final starting time t b_i and the final ending time t e_i Otherwise, its first starting time t fb_i and the second end time t se_i The final starting time t b_i and the final ending time t e_i ; Thus, the final starting time t of each useful pulse saved in step 5 is obtained b_i and the final ending time t e_i And record it.
5. The method for locating redundant objects and identifying materials of sealed electronic equipment by multi-source information fusion according to claim 1 or 4, characterized in that: The process of pulse matching for multiple useful pulses of N channels adopts an improved threshold judgment search pulse matching algorithm for matching, and pulse matching is performed based on the results of pulse extraction for N segments of corresponding redundant signals in N channels. The matching process includes the following steps: Step 1: For multiple useful pulses corresponding to different channels, respectively obtain the useful pulses with the largest energy among the multiple useful pulses, and the useful pulses with the largest energy corresponding to N channels are called N reference useful pulses; according to the starting time of the N reference useful pulses, determine the redundant object signal collected by the acoustic emission sensor farthest from the redundant object, which is called the reference signal; determine the redundant object signal collected by the acoustic emission sensor closest to the redundant object, which is called the comparison signal; obtain the starting time of the reference signal and the comparison signal, and calculate the difference between the two starting times, which is called the starting time threshold T; Step 2: Denote the useful pulses contained in the reference signal in step 1 as P i , i = 1, 2, ..., n, the starting times of these useful pulses are t i ; According to the starting time threshold T, set the starting time threshold interval of each useful pulse to [t i -T, t i ]; Step 3: Starting from P1, search for useful pulses whose starting time is within [t1-T, t1] in the N-1 segments of redundant signals corresponding to the other N-1 channels; if N-1 useful pulses can be found, they are combined with P1 into a pulse group; otherwise, P1 is skipped, indicating that its matching is unsuccessful; Step 4: Follow the steps in step 3 to calculate all P i Pulse matching is performed; thereby, pulse matching of N-channel redundant signals is completed to obtain multiple pulse groups.
6. The method for locating and identifying materials of redundant objects in sealed electronic equipment by multi-source information fusion according to claim 1, characterized in that: The time domain features and frequency domain features in the first identification feature vector include: time delay, pulse rise time, pulse symmetry, amplitude, energy, root mean square voltage, zero crossing rate, spectrum centroid, spectrum mean square error, root mean square probability, frequency standard deviation, and the gray level co-occurrence matrix features in the first identification feature vector include: entropy value, contrast, homogeneity, and correlation; The time domain features and frequency domain features in the second identification feature vector include: pulse rise time, pulse symmetry, energy, root mean square voltage, zero crossing rate, spectrum centroid, spectrum mean square error, root mean square probability, and frequency standard deviation. The gray level co-occurrence matrix features in the second identification feature vector include: graph energy, entropy value, contrast, homogeneity, and correlation.
7. The method for locating redundant objects and identifying materials of sealed electronic equipment by multi-source information fusion according to claim 1, characterized in that: The positioning recognition neural network and the material recognition network neural network adopt the PReLU-VGG19-Plus model. The PReLU-VGG19-Plus model is a VGG19 model in which the first convolution layer is replaced with a 7×7 convolution kernel and the activation function is replaced with a PReLU activation function.
8. The method for locating redundant objects and identifying materials of sealed electronic equipment by multi-source information fusion according to claim 1, characterized in that: The positioning recognition classifier and material recognition classifier adopt the XGBoost model; The parameters of the XGBoost model corresponding to the positioning recognition classifier are as follows: n_estimatores is 180, max_depth is 15, min_child_weight is 4, learning_rate is 0.38, gamma is 0.03; The parameters of the XGBoost model corresponding to the material recognition classifier are as follows: n_estimatores is 150, max_depth is 15, min_child_weight is 4, learning_rate is 0.41, gamma is 0.05; nestimatores refers to the number of base classifier decision trees; max_depth refers to the maximum depth of the base classifier decision tree; min_child_weight refers to the minimum leaf node sample weight that determines the base classifier decision tree; learning_rate refers to the learning rate; gamma refers to the penalty term coefficient.
9. A two-stage adaptive energy threshold pulse extraction method, the method is used to extract pulses from a redundant signal obtained by a single measurement by an acoustic emission sensor when testing a sealed electronic device to be tested, characterized in that: The signal corresponding to each acoustic emission sensor is recorded as a channel of redundant signal. The pulse extraction process of the N sections of redundant signals corresponding to the N channels adopts a two-level adaptive energy threshold pulse extraction algorithm to extract the pulses of each section of redundant signal. The specific process includes the following steps: Step 1: Calculate the average energy E of the redundant signal a , and calculate the standard deviation σ of the signal energy E ; Set the peak energy threshold to E p =E a +3σ E , set the endpoint energy threshold to E be =E a +σ E ; Step 2: Frame the signal; Step 3: Starting from the first frame signal, compare the energy of each frame signal with E p ; When the energy of a frame signal is greater than E p When the frame signal is recorded, and starting from it, the energy of each subsequent frame signal is compared with E p , until the energy of a frame signal is less than E p ; Find the frame signal with the largest energy from these frame signals and identify it as the maximum frame signal of the first useful pulse; Get the time of the sampling point with the largest amplitude in the maximum frame signal, which is called the peak time t of the first useful pulse p_1 ; Step 4: Take the peak time t of the first useful pulse extracted in step 3 p_1 As the starting point, compare the energy of each frame signal with E be Until the energy of a frame signal is found to be less than E in both directions be ; Respectively identify the previous frame signals of the two frame signals as the first starting frame signal and the first ending frame signal of the first useful pulse; obtain the time of the first sampling point in the first starting frame signal, which is called the first starting time t of the first useful pulse fb_1 ; Get the time of the last sampling point in the first end frame signal, which is called the first end time t of the first useful pulse fe_1 ; Step 5: Take the first end time t of the first useful pulse extracted in step 4 fe_1 The next frame signal is taken as the new starting point, and the same processing process of step 3 and step 4 is repeated to extract the remaining useful pulses in the signal, and then all the useful pulses extracted from the signal are saved; at the same time, the peak moments of all useful pulses are saved. First starting moment and the first ending moment Step 6: Calculate the average energy and standard deviation of each useful pulse saved in step 5, expressed as E i and σ E_i On this basis, the exclusive endpoint energy threshold of each useful pulse is set to E be_i =E i +σ Z_i ; Step 7: Starting from the first useful pulse saved in step 5, take its peak time t extracted in step 3 p_1 As the starting point, the energy of each frame signal is compared with the exclusive endpoint energy threshold E be_1 Until the energy of a frame signal is found to be less than E in both directions be_1 ; Respectively identify the previous frame signal of the two frame signals as the second start frame signal and the second end frame signal of the first useful pulse; The time of obtaining the first sampling point in the second starting frame signal is called the second starting time t of the first useful pulse. sb_1 ; Get the time of the last sampling point in the second end frame signal, which is called the second end time t of the first useful pulse se_1 ; Step 8: Starting from the second useful pulse saved in step 5, repeat the same processing steps as step 7 to obtain the second starting time of the remaining useful pulses saved in step 5 respectively. and the second end moment Step 9: Calculate the first duration t of each useful pulse saved in step 5 fl_i =t fe_i -t fb_i , the second duration t sl_i =t se_i -t sb_i , and duration ratio If the useful pulse τ i If it is greater than the duration threshold T, then its first starting time t fb_i With the first end time t fe_i The final starting time t b_i and the final ending time t e_i Otherwise, its first starting time t fb_i and the second end time t se_i The final starting time t b_i and the final ending time t e_i ; Thus, the final starting time t of each useful pulse saved in step 5 is obtained b_i and the final ending time t e_i And record it.
10. A matching method using an improved threshold judgment search pulse matching algorithm, the method is used to match redundant signals of multiple channels after pulse extraction of a measurement signal, the signal corresponding to each acoustic emission sensor is recorded as a redundant signal of a channel; the number of multiple channels is recorded as N, and pulse extraction is performed on N sections of redundant signals corresponding to N channels during pulse extraction; it is characterized in that Based on the pulse extraction results of the N segments of redundant signals corresponding to the N channels, pulse matching is performed using an improved threshold judgment search pulse matching algorithm. The matching process includes the following steps: Step 1: For multiple useful pulses corresponding to different channels, respectively obtain the useful pulses with the largest energy among the multiple useful pulses, and the useful pulses with the largest energy corresponding to N channels are called N reference useful pulses; according to the starting time of the N reference useful pulses, determine the redundant object signal collected by the acoustic emission sensor farthest from the redundant object, which is called the reference signal; determine the redundant object signal collected by the acoustic emission sensor closest to the redundant object, which is called the comparison signal; obtain the starting time of the reference signal and the comparison signal, and calculate the difference between the two starting times, which is called the starting time threshold T; Step 2: Denote the useful pulses contained in the reference signal in step 1 as P i , i = 1, 2, ..., n, the starting times of these useful pulses are t i ; According to the starting time threshold T, set the starting time threshold interval of each useful pulse to [t i -T, t i ]; Step 3: Starting from P1, search for useful pulses whose starting time is within [t1-T, t1] in the N-1 segments of redundant signals corresponding to the other N-1 channels; if N-1 useful pulses can be found, they are combined with P1 into a pulse group; otherwise, P1 is skipped, indicating that its matching is unsuccessful; Step 4: Follow the steps in step 3 to calculate all P i Pulse matching is performed; thereby, pulse matching of N-channel redundant signals is completed to obtain multiple pulse groups.
Citation Information
Patent Citations
High-frame-rate target tracking method based on deep pulse neural network
CN117876927A
Multi-channel signal-based integrated spectrogram construction method for redundant object positioning
CN118091668A
Processing and analyzing physiological signals to detect a health condition
US20080058659A1
System and methods of integrating visual features and textual features for image searching
US20130080426A1
Cited By
Method for detecting redundant deep information of sealed electronic equipment
CN119740133A
Integrated model construction and detection method for detecting redundancy of sealed electronic component under multi-information carrier view angle
CN120219307A
Integrated model construction and detection method for detecting electronic component excess under multi-information carrier perspective
CN120219307B
Cutting pick wear detection method and system for heading machine
CN120927828A
A pick wear detection method and system for a roadheader
CN120927828B
