Integrated spectrogram construction method based on multi-channel signals for redundant object localization

Abstract

The invention discloses a method for constructing an integrated spectrogram based on multi-channel signals for locating superfluous objects, and belongs to the technical field of superfluous object detection of sealed electronic equipment. The invention aims to solve the problem that there is no integrated spectrogram construction method based on multi-channel signals for detecting superfluous object locating signals. The invention is directed to a multi-channel signal for locating superfluous objects, and firstly converts the signal of each channel into a spectrogram corresponding to each channel signal, and takes each superfluous object signal or a single pulse extracted based on each superfluous object signal as a processing object, firstly performs a pre-emphasis processing on the processing object, and then performs a windowing processing, and performs a fast Fourier transform on the processing object after windowing, and then passes the power spectrum through a bandpass filter, and then converts the spectrogram corresponding to each channel; then the spectrograms corresponding to the multiple channels are spliced ​​to obtain an integrated spectrogram of the multi-channel signal.

Description

Technical Field

[0001] The invention belongs to the technical field of detecting excess matter in sealed electronic equipment, and in particular relates to a method for constructing an integrated spectrogram based on multi-channel signals for locating excess matter. Background Art

[0002] Sealed electronic equipment is a complex system composed of mechanical structures, integrated circuits and actuators, and is an important part of the aerospace system. Due to the special application field and the limitations of the production process, particles such as solder particles, aluminum scraps, and wire skin may be encapsulated in the production process of sealed electronic equipment. These substances that enter from the outside or are generated inside and affect the original stable physical state of the object are called excess. The existence of excess poses a major threat to the reliable operation of sealed electronic equipment, and excess detection is crucial. The particle impact noise detection (PIND) method is currently a commonly used excess detection method. It activates the excess that may exist inside the test piece by vibrating and sounding, causing it to collide or slide to generate a sound signal, which is called an excess signal. The acoustic emission sensor is placed on the surface of the test piece to capture the excess signal that may be generated. Whether there is excess is converted into whether there is excess signal generation, and finally converted into whether the acoustic emission sensor has a signal output, thereby realizing excess detection. The redundant object detection technology based on the PIND method is relatively mature, and corresponding PIND detection equipment has been manufactured, such as the 4511 equipment of SD Company in the United States and the DZJC-III equipment of Yugao Electronic Technology Co., Ltd. in China. They have designed signal processing circuits that can simultaneously collect and save the redundant object signals output by multiple acoustic emission sensors.

[0003] For sealed electronic devices with three-dimensional structures, their volume and weight are large and their internal structure is complex. After determining that there are redundant objects, it is difficult to find and clean them. Obtaining the location of the redundant objects can provide a reference, and the research on the positioning of redundant objects has attracted attention. In the early days, scholars used PIND detection equipment to place acoustic emission sensors at different positions on the surface of sealed electronic equipment to synchronously collect redundant object signals. By analyzing the time domain characteristics of the redundant object signals captured by acoustic emission sensors at different positions, such as energy, phase, arrival time, etc., the three-dimensional coordinates or regional positions of the redundant objects are given, such as the work carried out by Yang Jing et al., Sun Yongling et al., Ding Dandan et al. This is also the time difference positioning method and regional positioning method commonly used in the field of acoustic emission source positioning. However, the internal structure of sealed electronic equipment is complex and the composition materials are uneven, and the time domain characteristics cannot stably and comprehensively represent all the signal characteristics of the redundant object signals. This leads to inaccurate and unstable results in the positioning of redundant objects. Therefore, these methods have not been applied in actual scenarios. At present, scholars such as Sun Zhigang use machine learning methods for signal recognition and have achieved preliminary application results in the fields of acoustic emission and fault diagnosis. However, in previous studies, most of them took redundant object signals as the research object, extracted multiple acoustic features that can comprehensively reflect the characteristics of redundant object signals from multiple domains, constructed data sets, and trained traditional machine learning models, such as k-nearest neighbors, support vector machines, random forests, XGBoost, etc., with the highest classification accuracy exceeding 90%.

[0004] In order to further improve the recognition accuracy, other methods can be tried for classification and recognition. In fact, signals can be converted into images, and signal recognition research can be carried out with the help of image recognition technology, such as recognition methods represented by neural networks. In previous studies, considering that there is information loss in the process of converting signals into images, the signal recognition accuracy will be reduced to a certain extent, resulting in unreliable signal recognition results. Therefore, previous studies did not convert redundant object signals into images to carry out positioning research. In recent years, spectrogram conversion technology is a relatively mature image processing technology for realizing signal-to-image conversion. It can retain the information of the signal before conversion in the converted image as much as possible to ensure that the signal characteristics are fully described. In addition, represented by residual neural networks and convolutional neural networks, the classification performance of the current classic or latest deep learning models has exceeded that of traditional machine learning models, especially in the field of image recognition. Therefore, even if there is a small amount of information loss in the process of converting signals into spectrograms, the deep learning model can make up for this deficiency with its excellent classification performance, and even the combined recognition scheme of "spectrogram + deep learning model" has achieved a classification accuracy that exceeds the combined recognition scheme of "signal + traditional machine learning model". Many scholars have used this scheme to conduct signal recognition research. For example, Vo et al. used parallel one-dimensional convolutional neural networks and recurrent neural networks to extract spatial, temporal, long-term and short-term features from signals, respectively, to identify fault signals of different categories, and to achieve intelligent fault detection and diagnosis of induction motors. Tang et al. used Gram angular fields to convert signals into two-dimensional feature maps, and used deep belief networks to build a deep learning framework to effectively classify two-dimensional feature maps, thereby achieving bearing fault diagnosis. Huang et al. converted the long-term series data change features of pump leakage that are difficult to identify into image pixel-level texture features, and used the convolution residual block structure with attributes and spatial attention for oil well pump leakage fault diagnosis. Spandonidis et al. converted the signal obtained by the accelerometer installed on the pipeline wall into a spectrogram, and used a two-dimensional convolutional neural network to automatically extract features for image classification, thereby achieving leakage detection of metal pipelines. Guatelli et al. converted the speech of patients with four symptoms into spectrograms, and used convolutional neural networks to extract convolutional feature vectors for image classification, thereby achieving symptom detection of Parkinson's patients. Kumar et al. converted time domain signals into angular domain signals, and performed wavelet transform to generate time-frequency images. On this basis, they used convolutional neural networks to classify images and realize the defect recognition of bearings and rotors. Although these works can provide new ideas for the research of redundant object positioning, due to the characteristics and principle limitations of the signals generated when detecting redundant objects in sealed electronic equipment, these methods are not ideal for the recognition of redundant object signals.

[0005] Generally, for the detection of redundant objects in sealed electronic devices, two acoustic emission sensors are used to achieve linear positioning, three acoustic emission sensors are used to achieve plane positioning, and four acoustic emission sensors are used to achieve spatial positioning. For sealed electronic devices with three-dimensional structures, scholars including Sun Zhigang and others usually place four acoustic emission sensors on the surface of the sealed electronic devices to synchronously collect redundant object signals. In this way, a single experiment will collect and save four segments of redundant object signals at the same time. There is a strong correlation between these four segments of redundant object signals. Therefore, whether it is to construct a signal-based data set or a picture-based picture set, it is necessary to consider the four segments of redundant object signals as a whole, rather than as four independent individuals. This requires the use of multiple channels to synchronously input the four segments of redundant object signals. In the existing research on the recognition of multi-channel input signals or pictures, scholars usually use four methods to process. One is to use a multi-input deep learning model to extract features from multiple input pictures at the same time, and construct multiple independent feature vectors respectively and input them into the hidden layer in turn for subsequent processing. The second is to use a multi-input deep learning model to extract features from multiple input pictures at the same time, and construct an integrated feature tensor to input into the hidden layer for subsequent processing. The third is to use multiple deep learning models to extract features from multiple input images in parallel, and construct multiple independent feature vectors to input their respective hidden layers for subsequent processing. The fourth is to use multiple deep learning models to extract features from multiple input images in parallel, and construct an integrated feature tensor to input the hidden layers of multiple deep learning models for subsequent processing. Such as the work carried out by Choudhary et al. It can be seen that, in essence, these four processing methods still treat multiple input images as independent individuals when extracting features. Whether using a multi-input deep learning model or multiple deep learning models, it only increases the speed of feature extraction. Perhaps the second and fourth methods constructing multiple independent feature vectors into an integrated feature tensor can preserve the correlation between them to a certain extent, but this can only be regarded as a remedial measure. In fact, if multiple input images can be processed as a whole in the feature extraction stage, that is, the input stage, the correlation between them can be better and more comprehensively preserved, but these are not considered in existing studies.

[0006] When the PIND detection device is used for the detection of superfluous objects, the superfluous objects will be activated, causing them to collide or slide to generate superfluous object signals. From the perspective of the acoustic emission sensor, every time the superfluous objects collide or slide with the inner wall or internal components of the sealed electronic device, energy will be generated, a piezoelectric reaction will occur, and a current pulse will be generated. When the superfluous objects are in a fixed or suspended state, that is, the superfluous objects cannot contact the inner wall or internal components, no energy will be generated at this time, and no piezoelectric reaction will occur and no current pulse will be generated. In this way, for a section of superfluous object signal collected by the acoustic emission sensor within a fixed time, its interior is composed of pulses and background noise. Among them, the pulse is generated by the collision or sliding of the superfluous objects, which is related to the superfluous objects, and the background noise is generated by other factors other than the superfluous objects, such as mechanical noise, electromagnetic interference, etc., which has nothing to do with the superfluous objects. In fact, if it is an ideal state, that is, no other factors generate noise or interference, the background noise will appear as a zero signal without amplitude. From the above analysis, it can be seen that the superfluous object signal is the pulse that can truly feedback the signal characteristics. Therefore, in the process of converting the signal into an image, there are two options. One is to directly convert a section of the superfluous signal into a picture, which contains all the state information related to the superfluous within a fixed acquisition time. In this way, the four sections of the superfluous signal collected by the four acoustic emission sensors are converted into four pictures, and the four pictures need to be processed as a whole. The other is to first extract the pulses in a section of the superfluous signal, which contain state information directly related only to the superfluous within a fixed acquisition time, and then convert each pulse into a picture. In this way, a section of the superfluous signal collected by each acoustic emission sensor is converted into multiple pictures, which are regarded as a set. The four sections of the superfluous signal collected by the four acoustic emission sensors are respectively converted into four sets containing multiple pictures. The four corresponding pictures in each set need to be treated as a whole for processing. As mentioned above, even if the source of the four pictures is not considered, there is no research in the prior art that treats the four pictures as a whole for processing, let alone exploring which signal is more suitable for conversion into pictures on this basis. These are all problems that need to be solved urgently. Summary of the invention

[0007] The present invention aims to solve the problem that there is currently no integrated spectrogram construction method based on multi-channel signals for redundant object positioning signal detection.

[0008] The method for constructing an integrated spectrogram based on a multi-channel signal for locating redundant objects includes recording each segment of the redundant object signal in each channel as a diagnostic signal for the multi-channel signal for locating redundant objects, recording a single pulse extracted based on a diagnostic signal as a pulse signal, and the specific process of extracting a single pulse includes the following steps:

[0009] Step 1: Calculate the average energy and average zero-crossing rate of the entire sound signal, expressed as Ea and Z a , calculate the standard deviation of the energy and the standard deviation of the zero-crossing rate of the entire sound signal, expressed as σ E and σ Z ; Set the peak energy threshold to E p =E a +σ E , set the endpoint zero crossing rate threshold to Z be =Z a -σ Z ;

[0010] Step 2: Frame the sound signal and calculate the energy and zero-crossing rate of each frame signal;

[0011] Step 3: Starting from the first frame signal, compare the energy of each frame signal with E p ; When the energy of a frame signal is greater than E p When the frame signal is recorded, the energy of each subsequent frame signal is compared with E p , until the energy of a frame signal is less than E p ; Find the frame signal with the largest energy from these frame signals, and obtain the time of the sampling point with the largest amplitude, which is called the peak time t of the current useful pulse p ;

[0012] Step 4: Take the peak time t of the useful pulse extracted in step 3 p As the starting point, compare the zero-crossing rate of each frame signal with Z be , until the zero-crossing rate of a frame signal is found to be greater than Z in both directions be ; Respectively identify the previous frame signals of the two frame signals as the first start frame signal and the first end frame signal of the current useful pulse; obtain the time of the first sampling point in the first start frame signal, which is called the first start time t of the current useful pulse b_1 ; Get the time of the last sampling point in the first end frame signal, which is called the first end time of the current useful pulse t e_1 ;

[0013] Step 5: Take the first end time t of the useful pulse extracted in step 4 e_1 The next frame signal is taken as a new starting point, and steps 3 and 4 are repeated to extract the second useful pulse; and so on, until the last frame signal of the sound signal is searched; and the extracted multiple useful pulses are saved;

[0014] Step 6: Calculate the average zero-crossing rate and standard deviation of the zero-crossing rate of each useful pulse saved in step 5, expressed as Z i and σ Z_i; Where i represents the serial number of the saved useful pulse; on this basis, the exclusive endpoint zero-crossing rate threshold of each useful pulse is set to Z be_i =Z i -σ Z_i ;

[0015] Step 7: For the first useful pulse saved in step 5, take its peak time t p As the starting point, the zero-crossing rate of each frame signal is compared forward and backward with the exclusive endpoint zero-crossing rate threshold Z be_1 , until the zero-crossing rate of a frame signal is found to be greater than Z in both directions be_1 ; Respectively identify the previous frame signal of the two frame signals as the second starting frame signal and the second ending frame signal of the current useful pulse; obtain the time of the first sampling point in the second starting frame signal, which is called the second starting time t of the current useful pulse b_2 ; Get the time of the last sampling point in the second end frame signal, which is called the second end time t of the current useful pulse e_2 ;

[0016] Step 8: starting from the second useful pulse saved in step 5, repeat step 7 to obtain the second starting time and the second ending time of each pulse, until all the useful pulses saved in step 5 are processed;

[0017] Step 9: Calculate the first duration t of each useful pulse l_1 =t e_1 -t b_1 , the second duration t l_2 =t e_2 -t b_2 On this basis, the ratio of the second duration of each useful pulse to the first duration is calculated; if the ratio exceeds the duration threshold T, the first starting time t b_1 With the first end time t e_1 The final starting time t of the useful pulse is considered b and the final ending time t e Otherwise, the first starting time t b_1 and the second end time t e_2 The final starting time t of the useful pulse is considered b and the final ending time t e ;

[0018] Step 10: retain the useful pulses processed in step 9, and the useful pulse extraction of the entire sound signal is completed;

[0019] First, the redundant signal of each channel is converted into the spectrogram corresponding to each channel signal: the diagnostic signal or pulse signal is taken as the processing object, the processing object is first pre-emphasized, then windowed, and the windowed processing object is fast Fourier transformed, and then the power spectrum is passed through a bandpass filter, and then the spectrogram corresponding to each channel is converted;

[0020] Then the spectrograms corresponding to the multiple channels are spliced ​​to obtain the multi-channel signal integrated spectrogram.

[0021] Preferably, when the spectrograms corresponding to multiple channels are spliced, a vertical splicing method is adopted to splice them, so as to obtain a multi-channel signal integrated spectrogram.

[0022] Furthermore, when the diagnostic signal is used as the processing object, the window length is set to the time length of a redundant signal during the windowing process. When the pulse signal is used as the processing object, the window length is set to the time length of the pulse during the windowing process.

[0023] Furthermore, when the diagnostic signal is used as the processing object, the FFT length is set to 2 during the fast Fourier transform process. k , k needs to satisfy: 2 k ≥ the number of sampling points contained in a redundant signal, and 2 k-1 <The number of sampling points contained in a redundant signal. When the pulse signal is used as the processing object, the FFT length is set to 2 during the fast Fourier transform process. k’ , k' needs to satisfy: 2 k’ ≥The number of sampling points contained in a pulse signal, and 2 k’-1 <The number of sampling points contained in a pulse signal.

[0024] Preferably, the duration threshold T is 0.82s.

[0025] Preferably, in the process of performing frame processing on the sound signal in step 2, the duration of each frame signal is set to Δt=50 μs.

[0026] Furthermore, when the pulse signal is used as the processing object, after extracting a single pulse based on the diagnostic signal and before performing the pre-emphasis processing, it is also necessary to match the pulse extracted from each channel in the multi-channel signal. The matching process uses a zero-filling fusion threshold determination pulse matching algorithm for matching. The specific process includes the following steps:

[0027] S201: The number of channels is recorded as N', and based on the starting time of the first pulse in the redundant object signal of each channel, the distance relationship between the redundant object and the sensor that captures the signal, as well as the corresponding relationship between the redundant object signal and the sensor that captures the signal are determined; the starting time of the first pulse in each of the N' channels is obtained respectively, which is expressed as t1, t2, t3, ..., t N’ ;

[0028] S202: Calculate t N’ -t1, t N’ –t2, …, t N’ -t N’-1 , t N’ -t1, t N’ –t2, …, t N’ -t N’-1 They represent the time difference between the redundant object signal reaching the corresponding sensor and the farthest sensor;

[0029] Add a time of t before the start time of the redundant object signal captured by the sensor closest to the redundant object N’ -t1 background noise, add a time of t before the start time of the redundant object signal captured by the sensor closest to the redundant object N’ –t2 noise floor, add a duration of t before the start of the signal of the unwanted object captured by the sensor that is the third closest to the unwanted object N’ –t3 noise floor, and so on, add a time length of t before the start time of the redundant object signal captured by the sensor closest to the redundant object N'-1 N’ -t N’-1 The background noise;

[0030] On this basis, the time length t is discarded from the end time of the redundant object signal captured by the sensor closest to the redundant object N’ -t1 signal component, starting from the end time of the redundant object signal captured by the sensor closest to the redundant object, discard the time length t N’ –t2 signal component, starting from the end time of the redundant object signal captured by the sensor that is the third closest to the redundant object and discarding the time t N’ –t3 signal component, and so on, starting from the end time of the redundant object signal captured by the sensor closest to the redundant object N'-1, the time length is t N’ -t N’-1 signal components;

[0031] S203: Taking the signal of the unwanted object captured by the sensor farthest from the unwanted object as a reference, the pulses extracted therefrom are expressed in sequence as P i, i is the pulse number in the redundant signal captured by the sensor; the starting time of these pulses is recorded as T i ;

[0032] S204: Starting from P1, taking T1 time point as the reference, respectively search for pulses near the T1 reference in each segment of the redundant signal of the other N'-1 sensors; if the corresponding N'-1 segments of redundant signal can find pulses near the T1 reference, that is, N'-1 pulses are found, they are combined with P1 into a pulse group; otherwise, skip P1 and continue matching P2;

[0033] S205: Follow the method of S204 to treat all P i Pulse matching is performed until all pulse matching of the redundant signal is completed, and multiple pulse groups are obtained, each of which is the pulse matching result corresponding to N' channels.

[0034] The beneficial effects of the present invention are as follows:

[0035] The present invention proposes for the first time a method for converting a section of a superfluous object signal or a pulse into a spectrogram. Directly converting a section of a superfluous object signal into a signal spectrogram can retain all state information, or extracting multiple pulses in a section of a superfluous object signal and converting them into multiple pulse spectrograms only retains relevant state information. This can improve the accuracy of superfluous object positioning signal recognition. On the other hand, this is the first time that the focus of superfluous object positioning research has been shifted to image classification. A one-dimensional superfluous object signal can be converted into a two-dimensional spectrogram, on which the latest or classic deep learning models can be trained for image classification, thereby achieving superfluous object positioning based on a deep learning model.

[0036] In the method of converting pulses into spectrograms in the present invention, the designed limited pulse extraction algorithm can avoid the problem of continuous multiple pulses, not only ensuring that the pulses in the redundant signal are accurately and effectively extracted, but also making it possible to retain the effective pulses as completely as possible and control the interference information when the spectrogram is formed later, thereby improving the quality of the generated image, and further affecting the training of the deep learning model and improving the recognition accuracy of the deep learning model; the designed fixed pulse matching algorithm can ensure the alignment of the pulses in the redundant signal to form a corresponding pulse group, which can ensure the quality of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Test system for positioning excess objects. Figure 2 Schematic diagram of the redundant signal. Figure 3 Schematic diagram of the corresponding relationship between the movement process of redundant objects and the impact of signals. Figure 4 Two spectrograms constructed. Figure 5 The signal spectrogram is the signal converted from four sections of redundant signals collected in a single experiment. Figure 6 Integrate the spectrogram for signal level. Figure 7 Vertically integrate the spectrogram for the signal. Figure 8 Cross-integrate the spectrogram for the signal. Fig. 9 Overlay an integrated spectrogram on the signal. Fig.10 Schematic diagram of the two-stage variable multi-threshold pulse extraction algorithm. Fig.11 Schematic diagram of continuous multiple pulses. Fig.12 Schematic diagram of continuous multi-pulse solution. Fig.13 Schematic diagram of the pulse matching algorithm for threshold decision search. Fig.14 Pulse spectrogram converted to a pulse group. Fig.15 Integrate the spectrogram for the pulse level. Fig.16 Vertically integrated spectrogram for a pulse. Fig.17 The spectrogram is the pulse cross-integrated spectrogram. Fig.18 Integrate the spectrogram for pulse superposition. Fig.19 A basic infographic for SEEM. DETAILED DESCRIPTION

[0038] This embodiment is a method for constructing an integrated spectrogram based on multi-channel signals for locating redundant objects.

[0039] Based on the PIND method, with the help of the DZJC-III equipment of China Yugao Electronic Technology Co., Ltd., a Figure 1 The redundant object positioning test system shown in FIG. It consists of three parts: the left part is the hardware platform supporting the PIND method, the middle part is the DZJC-III device, and the right part is the algorithm processing completed on the host computer.

[0040] It should be noted that due to the particularity of the application field of sealed electronic equipment, it is impossible to obtain authorization to directly dismantle or use the sealed electronic equipment in service to carry out experiments. Therefore, the present invention customized a sealed electronic equipment model (SEEM) from the same manufacturer and used it to carry out the redundant object positioning research.

[0041] On this basis, the redundant object signal corresponding to the redundant object positioning is obtained, and the spectrogram is constructed:

[0042] Step 1: There are multiple confined spaces inside the sealed electronic device or SEEM, which are possible locations of unwanted objects. Number each confined space and place a sample of unwanted objects in the confined space numbered 1.

[0043] Step 2: Use a fixture to fix the SEEM containing the waste sample to the vibration table, and place four acoustic emission sensors at different positions on the surface of the SEEM. The upper computer controls the vibration table drive device to make the vibration table work, apply mechanical excitation to the SEEM, activate the waste sample, and make it collide or slide to generate waste signals.

[0044] Step 3: Four acoustic emission sensors simultaneously capture the unwanted object signals, convert them into electrical signals, and transmit them to the DZJC-III equipment for processing via a high-shielding data transmission line, including signal conditioning, signal filtering, synchronous acquisition, etc. After processing, the four unwanted object signals are saved as signal files, i.e. digital signals.

[0045] Step 4: Construct a spectrogram based on the redundant signal; and set the label of the constructed spectrogram to the same number as the number of the enclosed space where the redundant sample is currently placed, so as to obtain a labeled spectrogram. For example, if the redundant sample is placed in the enclosed space numbered 1, the label of the spectrogram is "1".

[0046] Step 5: Adjust the enclosed space where the excess samples are placed in step 1, that is, sequentially place the excess samples in enclosed spaces numbered 2, 3, ..., n. Repeat steps 2 to 4 to obtain multiple spectrograms labeled "2", "3", ..., "n". By repeating the above implementation steps, a large number of labeled spectrograms are obtained, and a collection of spectrograms representing excess samples located in different enclosed spaces is established, which is called a picture set.

[0047] Step 6: Train different deep learning models on the image set, called positioning models. The best performing one is obtained through comparison, and its parameters are optimized to achieve the best performance. The optimal positioning model is obtained and can be used for the redundant object positioning test of the sealed electronic device to be tested.

[0048] Step 7: Use a clamp to fix the sealed electronic device to be tested to the vibration table, and place four acoustic emission sensors at different positions on the surface of the sealed electronic device to be tested. The upper computer controls the vibration table drive device to make the vibration table work, apply mechanical excitation to the sealed electronic device to be tested, activate the excess material located at an unknown position inside it, and cause it to collide or slide to generate excess material signals. The four acoustic emission sensors capture the excess material signals at the same time, convert them into electrical signals, and transmit them to the DZJC-III type equipment through a high-shielding performance data transmission line for processing, and obtain four segments of processed digital signals. On this basis, a spectrogram is constructed, whose label is unknown, called the spectrogram to be tested. Use the optimal positioning model in step 6 to predict the label of the spectrogram, obtain the predicted label of the spectrogram, and then obtain the predicted label of the confined space. That is, the excess material positioning test of the sealed electronic device to be tested is completed.

[0049] From the above description, it can be seen that the core of the redundant object positioning method lies in the two transformations of the research object. The first research object is SEEM, which corresponds to steps one to six of the specific implementation steps, and the ultimate goal is to obtain the optimal positioning model. The second research object is the sealed electronic device to be tested, which corresponds to step seven of the specific implementation steps, and the ultimate goal is to obtain the redundant object positioning test results of the sealed electronic device to be tested. The high correlation between the properties of SEEM and the sealed electronic device to be tested ensures the stability of the redundant object positioning test results given by the optimal positioning model. From this, it can be seen that it seems that the stability of the redundant object positioning test results depends on the classification performance of the positioning model, that is, the classification performance of the deep learning model. The better the classification performance of the deep learning model, the more reliable the redundant object positioning test results. However, in deep machine learning, the factors that affect the classification performance of the deep learning model mainly include picture sets, model selection and parameter optimization. Among them, model selection depends on the problem being solved at the moment, and parameter optimization can only improve limited classification performance. The picture set is the source of training the deep learning model and plays a key role. The better the quality of the picture set, the clearer the reference for model selection, and the better the classification performance of the trained deep learning model. Therefore, the research focus of the present invention is on "constructing a spectrogram based on the redundant signal" in step four and step five.

[0050] 2. Analysis of the mechanism of generating redundant signals:

[0051] Figure 2 A section of the superfluous signal is shown, which consists of pulses and background noise. As mentioned earlier, pulses are generated by the collision or sliding of superfluous objects with the inner wall or internal components of the sealed electronic device, while background noise is generated by other factors when the superfluous objects are in a fixed or suspended state. It can be seen that the pulse is generated by the superfluous objects and is directly related to the superfluous objects, while the background noise is not generated by the superfluous objects and has nothing to do with the superfluous objects. In this way, it seems that only pulses can feedback the movement state of the superfluous objects, and the background noise is of no value.

[0052] In fact, the pulse only reflects part of the motion state of the excess material, or in other words, it only reflects the motion process in which the excess material contacts other media, causes deformation and generates energy, while the background noise reflects the remaining motion state of the excess material, that is, the motion process in which the excess material does not contact other media or contacts but does not cause deformation and ultimately does not generate energy, such as being fixed or suspended. Figure 3A schematic diagram of the movement process of the excess material is given. The figure shows the movement trajectory of the excess material in a confined space inside a sealed electronic device as it changes over time. Under the mechanical excitation provided by the vibration table, the sealed electronic device is activated, driving the excess material inside to move, causing it to jump up and down. Every time the excess material comes into contact with the inner wall or internal components of the sealed electronic device, it will be pushed to move in the opposite direction and lose contact until it comes into contact again under the action of gravity, is subjected to force, bounces off, and falls, and this cycle repeats until the mechanical excitation disappears. Figure 3 As can be seen in the figure, the solid red dots represent the contact between the extraneous matter and the inner wall or internal components of the sealed electronic device, thereby generating pulses, and the dotted black lines represent the contact between the extraneous matter and the inner wall or internal components of the sealed electronic device, thereby generating background noise. It should be noted that the process of the extraneous matter contacting the inner wall or internal components of the sealed electronic device is the aforementioned collision or sliding.

[0053] Therefore, a section of the redundant signal feedbacks the entire motion state of the redundant object, including all state information related to the redundant object, and each section of the redundant signal is a diagnostic signal; while the pulse feedbacks the partial motion state of the redundant object, including state information directly related to the redundant object, and each pulse extracted based on the diagnostic signal is a pulse signal. Therefore, whether it is a pulse spectrogram constructed by a single pulse or a signal spectrogram constructed by a whole section of the redundant signal, both can feedback the state information of the redundant object.

[0054] 3. Spectrogram:

[0055] A spectrogram is a two-dimensional image converted from a one-dimensional signal, and displays the signal characteristics in a visual way, represented by time domain characteristics and frequency domain characteristics. When constructing a spectrogram, the signal is first pre-emphasized, then windowed, and then the windowed signal is subjected to a fast Fourier transform (FFT), and finally the signal is converted into a spectrogram. It should be noted that in the present invention, the entire redundant signal and a single pulse need to be converted into a spectrogram respectively, so the setting of the frame signal length and the FFT length needs to refer to the specific application object, and the window function uniformly adopts the Hamming window.

[0056] For the convenience of description, the entire redundant signal or a single pulse is called a transformation object. The specific process of constructing a spectrogram is as follows:

[0057] First, a high-pass filter is used to pre-emphasize the conversion object, as shown in formula (1):

[0058] F(x)=1-μx -1 (1)

[0059] Wherein, x represents the processing object, μ is the pre-emphasis coefficient, and μ=0.97 is usually set.

[0060] The converted object after pre-emphasis is recorded as S(n), and S(n) is subjected to windowing processing to make it continuous, as shown in formula (2):

[0061] S w (n) = S (n) × w (n) (2)

[0062] Among them, w(n) is the window function, S w (n) is the transformed object after windowing.

[0063] In the present invention, the calculation formula of the Hamming window used is:

[0064]

[0065] Wherein, N represents the total number of sampling points contained in the signal, and n represents the number of sampling points contained in the window function (Hamming window);

[0066] Then, the windowed transformation object is subjected to FFT, and the power spectrum is passed through a bandpass filter to obtain the spectrogram corresponding to the transformation object.

[0067] Figure 4 Two types of spectrograms constructed from a segment of redundant signal are given. One is a spectrogram constructed from the entire segment of redundant signal, namely the signal spectrogram, and the other is a spectrogram constructed from a single pulse in the redundant signal, called the pulse spectrogram. As can be seen from the figure, the two spectrograms have the same aspect ratio, that is, the same size (the same size obtained by the same steps and window function conversion), but the image properties displayed by the two spectrograms are not the same. For example, the signal spectrogram looks more uniform overall, while the pulse spectrogram shows obvious light and dark color changes. In addition, there is a thin, darker, and more abrupt rectangle at the top of the signal spectrogram, while the pulse spectrogram does not exist. Therefore, the different image properties displayed by the two spectrograms correspond to different motion states of the redundant objects, and the mean values ​​need to be further studied and compared.

[0068] This embodiment describes the method of constructing a signal spectrogram from a whole segment of redundant signal and a method of constructing a pulse spectrogram from a single pulse. On this basis, the four transformation methods of horizontal merging (horizontal splicing), vertical merging (vertical splicing), cross merging (cross splicing) and linear superposition of four spectrograms describing the conversion of redundant signal groups or pulse groups are called signal group-spectrogram and pulse group-spectrogram methods, that is, the integrated spectrogram construction method proposed by the present invention.

[0069] The characteristics of the method for signal group-spectrogram and pulse group-spectrogram of the present invention are as follows: 1. The present invention analyzes the signal generation mechanism and provides the principle description of converting the entire signal to a spectrogram and a single pulse to a spectrogram. 2. After research, it was found that a single pulse can effectively avoid interference, so the present invention needs to extract pulses and match pulses. At the same time, after research, it was also found that the effect of the previous pulse extraction algorithm needs to be improved, and it cannot be well applied to the spectrogram method, so the present invention has made overall coordinated improvements. 3. The previous processing basically uses N models to process N objects. If the object description of the present invention is used, N spectrograms (N channels) are directly input into a model with N inputs, that is, one picture corresponds to one input. However, in order to extract the implicit mutual influence characteristics between each channel, the present invention first merges the N pictures into one picture to avoid the correlation between the split pictures, that is, the merging process of the present invention can effectively retain the correlation.

[0070] (I) Signal Group - Spectrogram

[0071] The basis of the signal group-spectrogram method is to first convert each segment of the redundant signal into a signal spectrogram. In this way, based on the specific process of constructing the spectrogram mentioned above, the present invention sets the window length of the Hamming window to the time length of the redundant signal, that is, the fixed acquisition time set by the DZJC-III device, and sets the FFT length to a certain power of 2 that can cover the number of sampling points in the redundant signal. In the present invention, the fixed acquisition time set by the DZJC-III device is 5s, and the sampling frequency is 500kHz. In this way, the sampling points in the redundant signal are spaced apart by A redundant signal contains Therefore, the window length of the Hamming window is set to 5s, and the FFT length is set to 2 to the power of 22. Figure 5 The following are four signal spectrograms of the conversion of four sections of redundant signals collected by the DZJC-III device in a certain experiment. Figure 5 The upper left, upper right, lower left, and lower right in the middle correspond to the signal spectrograms converted from the redundant object signals collected by acoustic emission sensors No. 1-4.

[0072] As mentioned above, there is a strong correlation between the four sections of redundant signals collected in a single experiment. Therefore, it is necessary to consider the four sections of redundant signals as a whole, that is, it is necessary to merge the four signal spectrograms converted from the redundant signal group into one spectrogram. In the present invention, the four signal spectrograms converted from the same redundant signal group are transformed into four types of signal integrated spectrograms, namely, horizontal merging, vertical merging, cross merging and linear superposition, to obtain four types of signal integrated spectrograms, which are named signal horizontal integrated spectrogram, signal vertical integrated spectrogram, signal cross integrated spectrogram and signal superposition integrated spectrogram.

[0073] Figures 6 to 9 Given by Figure 5 The four signal spectrograms shown are constructed from four signal integrated spectrograms, among which Figure 6 Integrated spectrogram for signal level, Figure 7 is the vertically integrated spectrogram of the signal, Figure 8 The cross-integrated spectrogram for the signal, Fig. 9 Overlay an integrated spectrogram on the signal.

[0074] (II) Pulse group-spectrogram

[0075] As mentioned above, the pulse group-spectrogram method is to first find out the pulses in each section of the redundant signal, and then find out the corresponding related pulse groups in the four sections of the redundant signal. In this way, each pulse is converted into a spectrogram, that is, a pulse spectrogram. On this basis, the four pulse spectrograms converted from the pulse group are transformed into four types of pulse integrated spectrograms by horizontal merging, vertical merging, cross merging and linear superposition, and four types of pulse integrated spectrograms are obtained. Among them, it is necessary to use a pulse extraction algorithm to find out the pulses, and it is necessary to use a pulse matching algorithm to find out the pulse group. Therefore, the newly proposed two-stage variable multi-threshold pulse extraction algorithm of the present invention, and the newly proposed zero-filling fusion threshold judgment pulse matching algorithm, and then construct the pulse group-spectrogram.

[0076] 1. Two-level variable multi-threshold pulse extraction algorithm:

[0077] According to the 3σ criterion of normal distribution, the probability of the value being distributed in [μ-σ,μ+σ] is 68.27%, and the probability of the value being distributed in [μ-3σ,μ+3σ] is 99.73%. In other words, the probability of the value exceeding μ-σ is about 84.13%, and the probability of the value exceeding μ+σ is about 15.87%. The former is a larger probability, and the latter is a smaller probability. Therefore, for a sound signal, when calculating its average energy E a , average zero-crossing rate Z a , the standard deviation of energy σ E and the standard deviation of the zero-crossing rate σ Z Then, we can a +σ E Find the maximum frame signal of the useful pulse in the frame signal, and set the peak energy threshold accordingly. a -σ Z The start frame signal and the end frame signal of the useful pulse are found in the nearby frame signals, and the endpoint zero-crossing rate threshold is set accordingly. Fig.10 A schematic diagram of a two-stage variable multi-threshold pulse extraction algorithm is given, and its specific implementation steps are as follows:

[0078] Step 1: Calculate the average energy and average zero-crossing rate of the entire sound signal, expressed as Ea and Z a , calculate the standard deviation of the energy and the standard deviation of the zero-crossing rate of the entire sound signal, expressed as σ E and σ Z On this basis, the peak energy threshold is set to E p =E a +σ E , set the endpoint zero crossing rate threshold to Z be =Z a -σ Z .

[0079] Step 2: Perform frame processing on the sound signal, set the duration of each frame signal to Δt=50 μs, and calculate the energy and zero-crossing rate of each frame signal. In the present invention, each frame signal contains 25 sampling points.

[0080] Step 3: Starting from the first frame signal, compare the energy of each frame signal with E p When the energy of a frame signal is greater than E p When the frame signal is recorded, the energy of each subsequent frame signal is compared with E p , until the energy of a frame signal is less than E p Find the frame signal with the largest energy from these frame signals, and obtain the time of the sampling point with the largest amplitude, which is called the peak time t of the current useful pulse. p .

[0081] Step 4: Take the peak time t of the useful pulse extracted in step 3 p As the starting point, compare the zero-crossing rate of each frame signal with Z be Until the zero-crossing rate of a frame signal is found to be greater than Z in both directions be The previous frame signals of the two frame signals are respectively identified as the first start frame signal and the first end frame signal of the current useful pulse. The time of obtaining the first sampling point in the first start frame signal is called the first start time t of the current useful pulse. b_1 The time of obtaining the last sampling point in the first end frame signal is called the first end time t of the current useful pulse. e_1 .

[0082] Step 5: Take the first end time t of the useful pulse extracted in step 4 e_1 The next frame signal is taken as a new starting point, and steps 3 and 4 are repeated to extract the second useful pulse. This process is repeated until the last frame signal of the sound signal is found. The extracted multiple useful pulses are saved.

[0083] At this point, the first stage of pulse extraction is completed.

[0084] Step 6: Calculate the average zero-crossing rate and standard deviation of the zero-crossing rate of each useful pulse saved in step 5, expressed as Z i and σ Z_i Where i represents the serial number of the saved useful pulse. On this basis, the exclusive endpoint zero-crossing rate threshold of each useful pulse is set to Z be_i =Z i -σ Z_i .

[0085] Step 7: Take the first useful pulse saved in step 5 as an example, and take its peak time t p As the starting point, the zero-crossing rate of each frame signal is compared forward and backward with the exclusive endpoint zero-crossing rate threshold Z be_1 , until the zero-crossing rate of a frame signal is found to be greater than Z in both directions be_1 The previous frame signal of the two frame signals is respectively identified as the second start frame signal and the second end frame signal of the current useful pulse. The time of obtaining the first sampling point in the second start frame signal is called the second start time t of the current useful pulse. b_2 The time of obtaining the last sampling point in the second end frame signal is called the second end time t of the current useful pulse. e_2 .

[0086] Step 8: Starting from the second useful pulse saved in step 5, repeat step 7 to obtain the second starting time and the second ending time of each pulse until all the useful pulses saved in step 5 are processed.

[0087] Step 9: Calculate the first duration t of each useful pulse l_1 =t e_1 -t b_1 , the second duration t l_2 =t e_2 -t b_2 On this basis, the ratio of the second duration of each useful pulse to the first duration is calculated. If the ratio exceeds the duration threshold T, the first starting time t b_1 With the first end time t e_1 The final starting time t of the useful pulse is considered b and the final ending time t e Otherwise, the first starting time t b_1 and the second end time t e_2 The final starting time t of the useful pulse is considered b and the final ending time t e .

[0088] Step 10: The useful pulses processed in step 9 are retained, and the useful pulse extraction of the entire sound signal is completed.

[0089] At this point, the second stage of processing is completed, that is, the second level pulse extraction is completed.

[0090] In order to fully illustrate the difference between the above process of the present invention and the prior art, the improvement of the present invention is specifically described here in combination with the existing pulse extraction algorithm:

[0091] The three-threshold pulse extraction algorithm first calculates the average energy of all sampling points contained in the sound signal, that is, the basic detection threshold, and sets the endpoint detection threshold and the peak detection threshold on this basis, with a total of three thresholds. It first uses the peak detection threshold to quickly find the highest point of the useful pulse in the sound signal, and then uses the highest point as a reference to find the starting point and the end point of the useful pulse forward and backward through the endpoint detection threshold. However, there is no fixed rule for setting the endpoint detection threshold and the peak detection threshold, which are often based on manual experience. This requires high professional skills and experience accumulation of the user. Even when facing the same sound signal, the pulse extraction results obtained by users with different personal qualities are different. This brings about the problem of unstable pulse extraction results. More importantly, in order to retain as many useful pulses as possible, scholars usually set the endpoint detection threshold to be equal to or slightly higher than the average energy of the noise pulse. This has a significant effect on the pulse extraction of independent useful pulses. However, for multiple useful pulses that appear continuously, especially when the amplitude of the previous useful pulse has not yet oscillated and decayed below the endpoint detection threshold, and the next useful pulse continues to rise, this may extract multiple useful pulses that appear continuously into one useful pulse, that is, the problem of continuous multiple pulses. Moreover, this method will also cause the overall spectrogram obtained later to contain more interference information, thus affecting the image processing process.

[0092] The double-threshold pulse extraction algorithm proposed by Gao Leizhen et al. calculates the average energy and average zero-crossing rate of the sampling points contained in the noise pulse of a fixed duration in the sound signal, and sets two thresholds, namely the endpoint energy threshold and the endpoint zero-crossing rate threshold. First, the starting point and the ending point of the first round of useful pulses are searched by the endpoint energy threshold, and then the starting point and the ending point of the second round of useful pulses are searched by the endpoint zero-crossing rate threshold. The final starting point and the final ending point of the useful pulse are determined according to the results of the two rounds of searches. Similarly, the setting of the endpoint energy threshold and the endpoint zero-crossing rate threshold still relies on manual experience judgment, the problem of continuous multiple pulses is not considered, and the problems existing in the three-threshold pulse extraction algorithm are not solved. Moreover, because the zero-crossing rate is more sensitive in finding the starting point and the ending point of the useful pulse, although the double-threshold pulse extraction algorithm can stably extract all pulses with amplitude in the sound signal, this includes noise pulses with weak amplitude. This will lead to inaccurate pulse extraction results. This method will also cause the overall spectrogram obtained later to have more interference information, thereby affecting the image processing-based process.

[0093] As mentioned above, both the three-threshold pulse extraction algorithm and the double-threshold pulse extraction algorithm do not take into account the problem of continuous multiple pulses. Fig.11 A schematic diagram of continuous multi-pulses is given for visual display. Figure 1 What is shown is a continuous analog signal, not a discrete digital signal. In fact, it can be regarded as a smooth curve connecting multiple sampling points. In the three-threshold pulse extraction algorithm, after the peak detection threshold is used to find the maximum frame signal of a useful pulse, the endpoint detection threshold E be The end frame signal of the useful pulse being sought may be the end frame signal of the next useful pulse, not the real end frame signal of the current useful pulse. Fig.11 The reason is that the oscillation attenuation between the two useful pulses is not lower than the endpoint detection threshold. Similarly, in the dual threshold pulse extraction algorithm, because the endpoint energy threshold E is set be The endpoint detection threshold E of the three-threshold pulse extraction algorithm be They play the same role and are not much different from each other. Therefore, when the endpoint energy threshold is used to extract useful pulses, it will also regard the two useful pulses in the red box as one useful pulse. In this way, when the useful pulse extraction results obtained using the endpoint energy threshold and the endpoint zero-crossing rate threshold are comprehensively analyzed, if the useful pulse extraction results obtained using the endpoint zero-crossing rate threshold play a leading role, the problem of continuous multiple pulses may be avoided. However, if the useful pulse extraction results obtained using the endpoint energy threshold play a leading role, the problem of continuous multiple pulses is inevitable.

[0094] The present invention proposes a two-stage variable multi-threshold pulse extraction algorithm, which sets a unified reference standard, provides clear threshold setting rules, focuses on the problem of continuous multi-pulses, and aims to extract useful pulses from redundant signals more stably and accurately. Specifically, the algorithm includes two stages of processing. In the first stage, it calculates the average energy and average zero-crossing rate of all sampling points contained in the sound signal, and sets two thresholds, the peak energy threshold and the endpoint zero-crossing rate threshold, according to the 3σ distribution criterion of the normal distribution. First, the peak energy threshold is used to quickly find the highest point of the useful pulse in the sound signal, and then the highest point is used as a reference to find the starting point and the end point of the useful pulse forward and backward respectively through the endpoint zero-crossing rate threshold. In the second stage, it sets the endpoint zero-crossing rate threshold exclusive to each useful pulse again according to the 3σ distribution criterion of the normal distribution. Similarly, the highest point of the useful pulse extracted in the first stage is used as a reference, and the starting point and the end point of the useful pulse are found again forward and backward respectively through the exclusive endpoint zero-crossing rate threshold. By calculating the ratio of the duration of the useful pulses extracted in the two stages, the problem of continuous multiple pulses is effectively solved, ensuring the extraction of independent useful pulses, and giving the final starting point and final ending point of the useful pulses.

[0095] In the present invention, the duration threshold T is set to 0.82. For independent useful pulses, a dedicated endpoint zero-crossing rate threshold Z is used. be_i The ratio of the extracted signal components to the total useful pulses should be approximately equal to 84.13%. For a large pulse composed of multiple useful pulses that appear continuously, its distribution is a combination of the normal distributions of multiple independent useful pulses. In this way, the useful pulses with larger amplitudes dominate the distribution characteristics, resulting in the setting of the exclusive endpoint zero-crossing rate threshold being too large, which is greater than the exclusive endpoint zero-crossing rate threshold set based on a single useful pulse with smaller amplitude. This actually discards the useful pulses with smaller amplitudes in the continuous multiple pulses. Therefore, the exclusive endpoint zero-crossing rate threshold Z is used. be_i The ratio of the extracted signal components to the total large pulses is less than 84.13%. Therefore, the present invention sets the time threshold T=0.82, which retains a moderate elastic space and allows the occurrence of partial degradation phenomena in actual situations.

[0096] As can be seen from the above description, in the first stage of the processing, the present invention first uses the peak energy threshold to extract those useful pulses with a high degree of credibility, which effectively discards the pulses with smaller amplitudes in the sound signal, and then uses the endpoint zero-crossing rate threshold to accurately find the starting point and the end point of the useful pulse, which effectively ensures that the main signal components in the useful pulse are extracted, and the components of the noise pulse are discarded as much as possible at both ends. Most importantly, the peak energy threshold and the endpoint zero-crossing rate threshold are adaptively set with reference to the overall distribution characteristics of the sound signal and the 3σ criterion of the normal distribution, with a credible mathematical theoretical basis, and get rid of manual experience judgment. In the second stage of the processing, the present invention first calculates the average energy, average zero-crossing rate, standard deviation of energy and standard deviation of zero-crossing rate of each useful pulse that is saved, which actually reflects the exclusive distribution characteristics of each useful pulse. On this basis, again referring to the 3σ criterion of the normal distribution, the present invention sets the exclusive endpoint zero-crossing rate threshold of each useful pulse. In this way, the starting point and the end point of each useful pulse processed in the first stage are searched again. If the useful pulse is indeed an independent individual, then referring to the 3σ criterion of normal distribution, the duration of the useful pulse after the second stage of processing should be 84.13% of that before the processing. If the useful pulse is actually a continuous multi-pulse, then the ratio of the duration of the useful pulse before and after the processing should be lower than 84.13%. This can effectively distinguish between independent useful pulses and continuous multi-pulses, indicating that the judgment method adopted by the present invention is feasible and has a credible mathematical theoretical basis. In fact, the processing result of continuous multi-pulses is to effectively extract the first useful pulse therein, such as Fig.12 shown.

[0097] In addition, if the effective pulse extraction is incomplete, it will directly affect the amount of effective information carried, which will directly affect the final recognition effect. If the pulse is converted into a spectrogram for processing, then in order to ensure the integrity of the extracted pulse, the specific value of the threshold can actually be appropriately relaxed. However, the ultimate goal of the present invention is to convert it into a spectrogram for processing. Once the amount of noise information in the spectrogram is not well controlled, it will have a greater impact on the recognition of the image. The pulse extraction method of the present invention sets a special threshold value, which is to ensure that the interference information in the image is effectively controlled when the effective pulse is converted into an image, and at the same time, it can also ensure the sufficiency of the effective information, so it can effectively improve the subsequent signal recognition effect.

[0098] 2. Zero-filling fusion threshold determination pulse matching algorithm:

[0099] Pulse matching is completed on the basis of pulse extraction. That is, firstly, a two-level variable multi-threshold pulse extraction algorithm is used to process the four segments of the redundant signal group respectively, and the extracted pulses are retained respectively, and then a zero-filling fusion threshold determination pulse matching algorithm is used to find the pulse group in the redundant signal group. Fig.13 A schematic diagram of the zero-filling fusion threshold determination pulse matching algorithm is given, and its specific implementation steps are as follows:

[0100] Step 1: Get the starting time of the first pulse in the four sections of the redundant signal respectively, thereby obtaining the redundant signal captured by the acoustic emission sensor closest to the redundant object, the redundant signal captured by the acoustic emission sensor second closest to the redundant object, the redundant signal captured by the acoustic emission sensor third closest to the redundant object, and the redundant signal captured by the acoustic emission sensor farthest from the redundant object. Get the starting time of the first pulse among them respectively, expressed as t1, t2, t3 and t4.

[0101] Step 2: Calculate t4-t1, t4-t2 and t4-t3. Add a background noise of duration t4-t1 before the start time of the superfluous object signal captured by the acoustic emission sensor closest to the superfluous object, add a background noise of duration t4-t2 before the start time of the superfluous object signal captured by the acoustic emission sensor that is second closest to the superfluous object, and add a background noise of duration t4-t3 before the start time of the superfluous object signal captured by the acoustic emission sensor that is third closest to the superfluous object. On this basis, discard the signal component of duration t4-t1 from the end time of the superfluous object signal captured by the acoustic emission sensor that is closest to the superfluous object, discard the signal component of duration t4-t2 from the end time of the superfluous object signal captured by the acoustic emission sensor that is second closest to the superfluous object, and discard the signal component of duration t4-t3 from the end time of the superfluous object signal captured by the acoustic emission sensor that is third closest to the superfluous object.

[0102] Step 3: Taking the signal of the unwanted object captured by the acoustic emission sensor farthest from the unwanted object as a reference, the pulses extracted from it are expressed in sequence as P i (i=1,2,…,n), the starting times of these pulses are T i (i=1,2,…,n).

[0103] Step 4: Starting from P1, search for pulses near T1 in the other three redundant signals. If three pulses can be found, combine them with P1 to form a pulse group. Otherwise, skip P1, indicating that the pulse matching is unsuccessful.

[0104] Step 5: Refer to step 4 and i (i=2,3,…,n) to perform pulse matching. At this point, the pulse matching of four sections of redundant signals is completed, and multiple pulse groups are obtained.

[0105] 3. Pulse integrated spectrogram construction:

[0106] Based on the specific process of constructing the spectrogram, the window length of the Hamming window is set to the time length of the pulse, and the FFT length is set to a power of 2 that can cover the number of sampling points in the pulse. In the present invention, the time length of each pulse is different, but the longest is 8ms and the shortest is 5ms. In this way, a pulse contains at most sampling points, including at least Therefore, the window length of the Hamming window is set appropriately according to the time length of the pulse, and the FFT length is set to a power of 2 to 12. Fig.14 Shows Figure 5 Four pulse spectrograms corresponding to the transformation of the first pulse group in the excess signal group. Fig.14 The upper left, upper right, lower left, and lower right in the middle correspond to the pulse spectrograms of the transformations of the first to fourth pulses in the pulse group.

[0107] As mentioned above, there is a strong correlation between the four pulses contained in the pulse group. Therefore, the four pulses in the pulse group need to be considered as a whole, that is, the four pulse spectrograms need to be merged into one spectrogram. In the present invention, the four pulse spectrograms are also transformed into four types of pulse integrated spectrograms, namely, horizontal merging, vertical merging, cross merging and linear superposition, to obtain four types of pulse integrated spectrograms, which are named pulse horizontal integrated spectrogram, pulse vertical integrated spectrogram, pulse cross integrated spectrogram and pulse superposition integrated spectrogram. Figure 15-18 Given by Fig.14 The four pulse spectrograms shown are constructed from four pulse integrated spectrograms, where Fig.15 is the pulse level integrated spectrogram, Fig.16 is the vertically integrated spectrogram of the pulse, Fig.17 is the pulse cross-integrated spectrogram, Fig.18 Integrate the spectrogram for pulse superposition.

[0108] In the present invention, a certain type of satellite-borne electronic unit is used as the sealed electronic device to be tested. SEEM is made by using the specific implementation steps of the aforementioned redundant object positioning method. According to the distribution of the confined spaces inside the sealed electronic device to be tested, they are almost distributed in eight vertex areas. Therefore, in order to facilitate preliminary research, the eight vertex angles are used as a reference, and three mutually orthogonal thin plates are abstractly used to divide the interior of SEEM into eight approximate confined spaces, and they are numbered. In addition, four acoustic emission sensors are placed at different positions on the surface of SEEM, and these acoustic emission sensors are connected to DZJC-III type equipment. The distribution of confined spaces and the spatial layout of acoustic emission sensors are shown in the figure. Fig.19 shown.

[0109] On this basis, the excess samples were placed in eight confined spaces of SEEM in turn. By repeating multiple rounds of experiments, multiple excess signal groups were collected, each of which contained four segments of excess signals. Finally, a large number of excess signal groups representing excesses in different confined spaces were saved for carrying out the present invention.

[0110] Construction of integrated image set: After statistics, 120 groups (480 segments in total) of redundant object signals generated in the confined space numbered 1 were collected, 121 groups (484 segments in total) of redundant object signals generated in the confined space numbered 2 were collected, 118 groups (472 segments in total) of redundant object signals generated in the confined space numbered 3 were collected, 120 groups (480 segments in total) of redundant object signals generated in the confined space numbered 4 were collected, 121 groups (484 segments in total) of redundant object signals generated in the confined space numbered 5 were collected, 125 groups (500 segments in total) of redundant object signals generated in the confined space numbered 6 were collected, 121 groups (484 segments in total) of redundant object signals generated in the confined space numbered 7 were collected, and 120 groups (480 segments in total) of redundant object signals generated in the confined space numbered 8 were collected, totaling 966 groups (3864 segments) of redundant object signals.

[0111] Using the specific implementation steps of the aforementioned redundant object positioning method and the signal group-spectrogram method, the 3864 collected redundant object signals are first converted into 3864 signal spectrograms, and then the four signal spectrograms converted from the same redundant object signal group are transformed into four kinds of transformations, namely horizontal merging, vertical merging, cross merging and linear superposition, to respectively construct 966 signal horizontal integrated spectrograms, 966 signal vertical integrated spectrograms, 966 signal cross integrated spectrograms and 966 signal superposition integrated spectrograms. On this basis, according to the number of the closed space from which the redundant object signal group corresponding to the signal integrated spectrogram comes, labels are added to the signal horizontal integrated spectrogram, the signal vertical integrated spectrogram, the signal cross integrated spectrogram and the signal superposition integrated spectrogram, respectively, to respectively construct the signal horizontal integrated picture set, the signal vertical integrated picture set, the signal cross integrated picture set and the signal superposition integrated picture set. Considering that the total number of signal-integrated spectrograms and the number of signal-integrated spectrograms for each label contained in the four signal-integrated picture sets are the same, the authors summarized them in Table 1 as “picture sets constructed from signal-integrated spectrograms”.

[0112] Using the specific implementation steps of the aforementioned redundant object positioning method and the pulse group-spectrogram method, the two-level variable multi-threshold pulse extraction algorithm is first used to extract pulses from 3864 redundant object signals. On this basis, the threshold judgment search pulse matching algorithm is used to match pulses of 966 redundant object signal groups. After statistics, a total of 37,811 pulse groups, i.e., 151,244 pulses, are obtained. Next, the 151,244 pulses are converted into 151,244 pulse spectrograms, and then the four pulse spectrograms converted from the same pulse group are transformed into four types of transformations: horizontal merging, vertical merging, cross merging, and linear superposition, to construct 37,811 pulse horizontal integrated spectrograms, 37,811 pulse vertical integrated spectrograms, 37,811 pulse cross integrated spectrograms, and 37,811 pulse superposition integrated spectrograms. On this basis, according to the number of the closed space where the redundant signal group corresponding to the pulse group of the pulse integrated spectrogram is located, the pulse horizontal integrated spectrogram, the pulse vertical integrated spectrogram, the pulse cross integrated spectrogram and the pulse superposition integrated spectrogram are labeled, and the pulse horizontal integrated picture set, the pulse vertical integrated picture set, the pulse cross integrated picture set and the pulse superposition integrated picture set are constructed respectively. Similarly, considering that the total number of pulse integrated spectrograms contained in the four pulse integrated picture sets and the number of pulse integrated spectrograms of each label are the same, they are summarized and described as "picture sets constructed by pulse integrated spectrograms" in Table 1.

[0113] Table 1 Detailed description of the two types of integrated image sets

[0114]

[0115] Performance evaluation of integrated spectrogram construction method:

[0116] The picture set is the source of training the deep learning model. The better the quality of the picture set, the better the classification performance of the trained deep learning model. Therefore, in order to evaluate the performance of the signal group-spectrogram and pulse group-spectrogram methods, the quality of the integrated picture set constructed by them can be evaluated and intuitively reflected with the help of the classification performance of the deep learning model. That is, the same deep learning model is trained on multiple integrated picture sets constructed by the two methods respectively, and the classification accuracy achieved by the deep learning model under the same circumstances is compared to feedback the quality of the integrated picture set, so as to achieve the purpose of evaluating the performance of the two methods.

[0117] In the present invention, five deep learning models with good performance are selected, including classic Alexnet, Resnet-18 and Resnet-50, and the latest lightweight Mobilenet and Effcientnet. The above five deep learning models are trained on the signal horizontal integration picture set, signal vertical integration picture set, signal cross integration picture set and signal superposition integration picture set, and the average classification accuracy obtained in 150 iterations, as well as the highest classification accuracy and corresponding iteration values ​​obtained in 150 iterations are counted, as shown in Table 2. Similarly, the above five deep learning models are trained on the pulse horizontal integration picture set, pulse vertical integration picture set, pulse cross integration picture set and pulse superposition integration picture set, and the average classification accuracy obtained in 150 iterations, as well as the highest classification accuracy and corresponding iteration values ​​obtained in 150 iterations are counted, as shown in Table 3.

[0118] Table 2 Classification results of five deep learning models on the signal integration image set

[0119]

[0120]

[0121] Table 3 Classification results of five deep learning models on the pulse ensemble image set

[0122]

[0123] First, analyze Table 2. When the four signal integration picture sets are used as the analysis object, it can be found that the five deep learning models all achieve the highest average classification accuracy on the signal vertical integration picture set, and the highest classification accuracy achieved is the most significant, and the Epoch that achieves the highest classification accuracy is the smallest. This shows that for the four signal spectrograms transformed from the redundant signal group, vertical merging is the most appropriate. In addition to vertical merging, horizontal merging and cross merging are also transformation methods that do not lose the original signal spectrogram. However, it can be seen from the table that the five deep learning models all achieve higher average classification accuracy than the signal horizontal integration picture set on the signal cross integration picture set, and achieve most of the more significant highest classification accuracy, except for Effcientnet, and the Epoch that achieves the highest classification accuracy is smaller. This comprehensive explanation shows that for the four signal spectrograms transformed from the redundant signal group, cross merging is better than horizontal merging, but the best is still vertical merging. Correspondingly, the classification results obtained by the five deep learning models on the signal superposition integration picture set show a trend different from their classification results on the other three signal integration picture sets. For example, the average classification accuracy and the highest classification accuracy of Alexnet, Mobilenet and Effcientnet on the signal superposition integrated picture set are significantly lower than those on the other three signal integrated picture sets, but the average classification accuracy and the highest classification accuracy of Resnet-18 and Resnet-50 on the signal superposition integrated picture set are higher than those on the signal level integrated picture set and the signal cross integrated picture set. At the same time, there is no obvious pattern to follow for the epochs in which the five deep learning models achieve the highest classification accuracy on the signal superposition integrated picture set.

[0124] Similarly, Table 2 is analyzed. When the five deep learning models are used as the analysis object, it can be found that no matter which signal integrated picture set is used, Resnet-50 achieves the highest average classification accuracy, and the highest classification accuracy achieved is the most significant, and the Epoch with the highest classification accuracy is the smallest. This shows that Resnet-50 is the best performing deep learning model and is the most suitable deep learning model among the five deep learning models to solve the classification problem of the present invention, that is, the optimal positioning model. It is worth noting that Resnet-18 also achieves significant classification results. It achieves higher average classification accuracy and highest classification accuracy than the other three deep learning models on the four signal integrated picture sets, and the Epoch with the highest classification accuracy is the smallest. This fully demonstrates the performance advantage of the residual neural network in solving the classification problem of the present invention. In addition, the average classification accuracy and highest classification accuracy achieved by Alexnet, Mobilenet and Effcientnet on the signal horizontal integrated picture set, signal vertical integrated picture set and signal cross integrated picture set are significantly higher than those on the signal superposition integrated picture set. In particular, the average classification accuracy achieved by Mobilenet and Effcientnet does not exceed 25%, which is unsatisfactory.

[0125] In general, no matter whether the four signal integrated picture sets or the five deep learning models are used as the analysis objects, abnormal phenomena appear in the signal superposition integrated picture sets. As mentioned above, horizontal merging, vertical merging and cross merging are transformation methods that do not lose the original signal spectrogram, but superposition merging loses the original signal spectrogram. It only realizes the merging of four independent signal spectrograms in terms of means, but fails to independently retain the original information of the four signal spectrograms, but instead presents a chaotic state of information. This may be the reason why most deep learning models achieve unsatisfactory classification results. In addition, Mobilenet and Effcientnet are lightweight deep learning models. They are trained with the purpose of calculating a smaller number of parameters and reducing the computational cost. Therefore, they actually put less effort into learning image features than classic deep learning models. On some pictures with obvious features, they may not perform very badly, and even achieve better classification results than classic deep learning models through their own structural advantages, but they obviously overturned on the chaotic signal superposition integrated picture set. More importantly, for most deep learning models, when they are trained on a chaotic signal superposition integrated image set, the problem of unclear image features will also lead to the problem of gradient vanishing, so it is difficult for them to achieve ideal classification results. This is also the reason why Alexnet, Mobilenet and Effcientnet achieve very low average classification accuracy and insignificant maximum classification accuracy. The structure of the residual block is designed in Resnet-18 and Resnet-50 to increase the network depth. In this way, the residual block avoids the problem of gradient vanishing by propagating across layers, thereby achieving stable classification results.

[0126] Therefore, when using the signal group-spectrogram method, we should avoid using the superposition merging transformation method and give priority to the vertical merging transformation method. In addition, when choosing a deep learning model, the latest residual neural network should be selected, such as Resnet-50.

[0127] Next, Table 3 is analyzed. When the four pulse integrated picture sets are used as the analysis object, it can also be found that the five deep learning models all achieve the highest average classification accuracy on the pulse vertical integrated picture set, and the highest classification accuracy achieved is the most significant, and the Epoch that achieves the highest classification accuracy is the smallest. This also shows that for the four pulse spectrograms converted from the pulse group, it is most appropriate to use vertical merging. In addition, as in Table 2, the five deep learning models achieve more stable classification results on the pulse cross-integrated picture set and the pulse horizontal integrated picture set, and Alexnet, Mobilenet and Effcientnet still achieve complex classification results on the pulse overlapping integrated picture set. When the five deep learning models are used as the analysis object, it can be found that no matter which pulse integrated picture set is used, Resnet-50 also achieves the highest average classification accuracy, and the highest classification accuracy achieved is the most significant, and the Epoch that achieves the highest classification accuracy is the smallest. In addition, as in Table 2, Resnet-18 still performs well, and the performance advantage of the residual neural network in solving the classification problem of the present invention is once again apparent. Alexnet, Mobilenet, and Effcientnet still achieve the worst classification results on the pulse overlap ensemble image set. The reasons for these problems in Table 3 are almost the same as those in Table 2, so they will not be repeated here. Therefore, when using the pulse group-spectrogram method, we should also avoid using the superposition merging transformation method and give priority to the vertical merging transformation method. In addition, when choosing a deep learning model, the latest residual neural network should be selected.

[0128] Finally, a comprehensive analysis is conducted on Tables 2 and 3. The average classification accuracy of the five deep learning models on the signal horizontal integrated picture set, the signal vertical integrated picture set, and the signal cross integrated picture set is higher than that obtained on the corresponding pulse horizontal integrated picture set, the pulse vertical integrated picture set, and the pulse cross integrated picture set. The highest classification accuracy obtained is more significant than that obtained on the corresponding pulse horizontal integrated picture set, the pulse vertical integrated picture set, and the pulse cross integrated picture set, but the Epoch with the highest classification accuracy is larger. This may be because the signal integrated spectrogram contains more information, that is, signal characteristics, because it is constructed by four segments of redundant signals, while the pulse integrated spectrogram is only constructed by four pulses. Therefore, the deep learning model needs to spend more time to learn more comprehensive features in order to achieve good classification results on the signal horizontal integrated picture set, the signal vertical integrated picture set, and the signal cross integrated picture set. It can also be seen from this that in the present invention, the signal group-spectrogram method is superior to the pulse group-spectrogram method. It can also be seen from the description of the two methods that the former is based on the redundant signal group and contains all the state information related to the redundant objects, while the latter is based on the pulse group and only contains the state information directly related to the redundant objects. This actually has important reference value for the existing research on redundant object detection, because most of the research currently carried out only focuses on the state information directly related to the redundant objects. Unexpectedly, the average classification accuracy of the five deep learning models on the signal superposition integrated picture set is lower than that on the pulse superposition integrated picture set, and the highest classification accuracy is not significantly higher than that on the pulse superposition integrated picture set, and the Epoch with the highest classification accuracy is larger. This is because the signal superposition integrated spectrogram integrates the information of four signal spectrograms, that is, it mixes all the state information of four segments of redundant signals, and the data volume is large and the degree of confusion is high. The pulse superposition integrated spectrogram only integrates the information of four pulse spectrograms, that is, it mixes the information of four pulses, and the data volume is small and the degree of confusion is low. Therefore, the deep learning model achieves more ideal classification results on the relatively clear pulse superposition integrated picture set.

[0129] In summary, when the redundant object positioning method is used, when the integrated spectrogram is constructed based on the redundant object signal group, it is preferred to use the signal group-spectrogram method, and use the vertical merging transformation method to construct a vertical integrated image set of the signal. In addition, residual neural networks, such as Resnet-50 or Resnet-18, are recommended deep learning models. Of course, the pulse group-spectrogram method is also worth considering and should not be abandoned. Similarly, even when using the pulse group-spectrogram method, we also recommend the use of vertical merging transformation methods and residual neural networks. Most importantly, whether using the signal group-spectrogram method or the pulse group-spectrogram method, the superposition and merging transformation method cannot be considered. Some lightweight deep learning models are not recommended for use at present. Perhaps more mature lightweight deep learning models can be considered in the future.

[0130] Comparative analysis of different methods:

[0131] As mentioned above, whether the signal group-spectrogram method or the pulse group-spectrogram method is used, the four spectrograms are first merged into an integrated spectrogram, regardless of whether the four spectrograms come from four redundant signals or four pulses. On this basis, labels are added to multiple integrated spectrograms, an integrated picture set is constructed, and a deep learning model is trained. As described in the background technology, this is done to consider the correlation between the four redundant signals collected in a single experiment. In order to verify the correctness of the conjecture, and to verify the rationality and necessity of carrying out the present invention, it is chosen not to merge the four spectrograms converted from the signal group or the pulse group into an integrated spectrogram, but to treat them as independent individuals, add labels separately, thereby constructing an independent picture set and training a deep learning model.

[0132] Referring to the above method, 3864 segments of redundant signals were collected, and 151244 pulses were obtained after pulse extraction and pulse matching. On this basis, 3864 signal spectrograms and 151244 pulse spectrograms were constructed respectively. Based on the verification idea, according to the number of the enclosed space where the redundant signal corresponding to the signal spectrogram comes from, and according to the number of the enclosed space where the redundant signal corresponding to the pulse spectrogram comes from, labels are added to the 3864 signal spectrograms and 151244 pulse spectrograms, respectively, to construct independent signal picture sets and independent pulse picture sets respectively. Similarly, Alexnet, Resnet-18, Resnet-50, Mobilenet and Effcientnet are trained on two independent picture sets, and their average classification accuracy achieved in 150 iterations is counted, as shown in Table 4.

[0133] Table 4 Average classification accuracy of five deep learning models on independent image sets (%)

[0134]

[0135] As can be seen from the table, comparing Table 2 and Table 3, whether on the independent signal picture set or the independent pulse picture set, the average classification accuracy obtained by the five deep learning models is lower than that obtained on the signal horizontal integration picture set, signal vertical integration picture set, signal cross integration picture set, pulse horizontal integration picture set, pulse vertical integration picture set and pulse cross integration picture set. Some deep learning models even obtain an average classification accuracy lower than that obtained on the signal superposition integration picture set and the pulse superposition integration picture set. This fully demonstrates the necessity of combining the redundant signal group and the pulse group collected in a single experiment, that is, it is necessary to merge the four converted spectrograms into an integrated spectrogram. This also fully illustrates the rationality and necessity of carrying out the present invention.

[0136] At the same time, it can be seen from Table 4 that the average classification accuracy achieved by Alexnet, Resnet-18 and Resnet-50 on the independent signal image set is significantly higher than that achieved by Mobilenet and Effcientnet, but this advantage is reversed on the independent pulse image set. The average classification accuracy achieved by Mobilenet and Effcientnet is significantly higher than that achieved by Alexnet, Resnet-18 and Resnet-50. For the independent signal image set, it contains 3864 signal spectrograms, which is not a large number, and the signal spectrogram contains all the state information related to the redundant objects. Therefore, Alexnet, Resnet-18 and Resnet-50 can achieve a relatively good average classification accuracy after sufficient learning. However, Mobilenet and Effcientnet are lightweight deep learning models. They aim to calculate a small number of parameters and reduce the computational cost, and their investment in learning is limited. Moreover, the number of signal spectrograms to be learned in the independent signal image set is not large. Therefore, the average classification accuracy achieved by Mobilenet and Effcientnet is low. However, for the independent pulse image set, it contains 151,244 pulse spectrograms, which is a large number, and the pulse spectrograms only contain state information directly related to the redundant objects, that is, there is not much learning content. At this time, the advantages of Mobilenet and Effcientnet are fully revealed. They can achieve relatively good classification results based on the rapid learning of basic image features.

[0137] In addition, in the existing research on the recognition of multi-channel input signals or pictures, scholars usually use four methods to deal with it. The first is to use a multi-input deep learning model to extract features from multiple input pictures at the same time, and construct multiple independent feature vectors to input into the hidden layer in sequence for subsequent processing. The second is to use a multi-input deep learning model to extract features from multiple input pictures at the same time, and construct an integrated feature tensor to input into the hidden layer for subsequent processing. The third is to use multiple deep learning models to extract features from multiple input pictures in parallel, and construct multiple independent feature vectors to input into their respective hidden layers for subsequent processing. The fourth is to use multiple deep learning models to extract features from multiple input pictures in parallel, and construct an integrated feature tensor to input into the hidden layers of multiple deep learning models for subsequent processing. These four methods are named: multi-input-independent vector, multi-input-integrated tensor, multi-model-independent vector, and multi-model-integrated tensor.

[0138] Therefore, the above four methods are used to train the corresponding deep learning models on the independent signal picture set and the independent pulse picture set, and the average classification accuracy they achieve is calculated. If these four methods are used to train the deep learning model on the independent signal picture set, the four signal spectrograms converted from the same signal group are used as the overall input. If these four methods are used to train the deep learning model on the independent pulse picture set, the four pulse spectrograms converted from the same pulse group are used as the overall input. It should be noted that from the perspective of easy implementation and existing applications, Alexnet, Resnet-18 and Resnet-50 are used as the basis to implement the four methods of multi-input-independent vector, multi-input-integrated tensor, multi-model-independent vector and multi-model-integrated tensor. On this basis, the average classification accuracy achieved is shown in Tables 5 to 7 respectively.

[0139] Table 5 Average classification accuracy of four methods based on Alexnet on independent image sets (%)

[0140]

[0141] Table 6 Average classification accuracy (%) of four methods based on Resnet-18 on independent image sets

[0142]

[0143] Table 7 Average classification accuracy (%) of four methods based on Resnet-50 on independent image sets

[0144]

[0145]

[0146] A comprehensive analysis of the three tables shows that no matter what deep learning model is used as the basis, or on what independent picture set, the multi-input-integrated tensor and multi-model-integrated tensor achieve significantly higher average classification accuracy than the multi-input-independent vector and multi-model-independent vector. This once again illustrates the necessity of combining the four segments of redundant signals collected in a single experiment, and indirectly illustrates the necessity of carrying out the present invention. At the same time, no matter what deep learning model is used as the basis, the average classification accuracy achieved by the four methods on the independent signal picture set is higher than that achieved on the independent pulse picture set. This shows that the redundant signal contains all state information related to the redundant object, which is more worthy of consideration, while the pulse only contains state information directly related to the redundant object. This is consistent with the above conclusion, verifying its correctness. In general, by comparing Table 2 and Table 3, it can be found that although some average classification accuracies (such as 91.91%) obtained by multi-input-integrated tensors and multi-model-integrated tensors are higher than those obtained by Alexnet, Resnet-18 and Resnet-50 on pulse horizontal integrated picture sets, pulse vertical integrated picture sets and pulse cross integrated picture sets, and even higher than those obtained by Alexnet on signal horizontal integrated picture sets and signal cross integrated picture sets, and Resnet-18 on signal horizontal integrated picture sets, they are significantly lower than those obtained by Alexnet, Resnet-18 and Resnet-50 on signal vertical integrated picture sets. This shows the superiority of the method proposed in the present invention, that is, using the signal group-spectrogram method and using the vertical merging transformation method to construct the signal vertical integrated picture set. In principle, multi-input-integrated tensors and multi-model-integrated tensors are already facing multiple spectrograms of independent input, and this reality cannot be changed. Therefore, even if its performance is superior, the available resources are relatively small, and the existing resources can only be used as much as possible. The method proposed in the present invention is to merge multiple corresponding spectrograms as a whole from the source into an integrated spectrogram, which is equivalent to leaving relatively rich resources waiting for the deep learning model to fully utilize.

[0147] In addition, by comparing Tables 5 to 7 with Table 4, it can be found that no matter what deep learning model is based on or what independent picture set is used, the average classification accuracy achieved by the multi-input-independent vector is slightly higher than that achieved by Alexnet, Resnet-18 and Resnet-50. This is because, although the multi-input-independent vector also trains multiple input spectrograms as individuals, the adjacent multiple spectrograms sent to the hidden layer are related, while Alexnet, Resnet-18 and Resnet-50 randomly select spectrograms from independent picture sets, and the correlation between spectrograms is further reduced. In addition, no matter what deep learning model is based on or what independent picture set is used, the multi-model-independent vector achieves the lowest average classification accuracy, because it completely separates the spectrograms of multiple inputs and independently trains the model. Taking the present invention as an example, the multi-model-independent vector actually separates the redundant signals collected by four acoustic emission sensors for consideration, and each model is only responsible for the redundant signals collected by one acoustic emission sensor. This is equivalent to completely removing the correlation between spectrograms and even signals. The average classification accuracy achieved by the multi-input-ensemble tensor and the multi-model-ensemble tensor is comparable, with the latter being slightly higher. The structures of the two are very similar, with the only difference being that one uses a multi-input deep learning model and the other uses multiple single-input deep learning models. The feature extraction and tensor construction are the same. In terms of processing speed, the multi-model-ensemble tensor is faster. Moreover, the slight advantage in classification accuracy achieved by the multi-model-ensemble tensor also illustrates the stability of the multi-model structure.

[0148] In summary, it is necessary to consider multiple input spectrograms as a whole from the source. The signal spectrogram constructed by the redundant signal contains all state information, and the average classification accuracy of the deep learning model trained by this method is the highest. This fully demonstrates the necessity of carrying out the present invention and fully verifies the superiority of the method proposed by the present invention.

[0149] Evaluation Metrics:

[0150] In the process of carrying out the present invention, it is necessary to use evaluation indicators to evaluate the classification performance of different deep learning models or methods, so as to indirectly evaluate the advantages and disadvantages of the spectrogram construction method or the multi-channel input signal or image recognition method. In the present invention, the classification accuracy is mainly used, which is defined as follows.

[0151] Assume that the image set is Ima={X1,X2,…,X n}, where Y i It's a sound spectrum Figure X i The true label, R(X i) is the predicted label given by the model or method R. The classification accuracy is expressed as the ratio of the number of spectrograms with correct label prediction to the total number of spectrograms in the image set. Its calculation formula is as follows.

[0152]

[0153] Where I is the indicator function. When R(X i )=Y i When I(R(X i )=Y i )=1.

[0154] The present invention uses a certain type of satellite-borne electronic stand-alone as the experimental object, collects a large number of redundant signal groups, obtains a large number of pulse groups, and thereby constructs four signal integrated picture sets and four pulse integrated picture sets. On this basis, the present invention trains five deep learning models, namely Alexnet, Resnet-18, Resnet-50, Mobilenet and Effcientnet. The experimental results show that in the signal group-spectrogram method, vertical merging is the optimal transformation method, and Resnet-50 achieves the highest average classification accuracy of 98.31% on the signal vertical integrated picture set. In the pulse group-spectrogram method, vertical merging is also the optimal transformation method, and Resnet-50 achieves the highest average classification accuracy of 91.61% on the pulse vertical integrated picture set. The comprehensive analysis results show that the signal group-spectrogram method using vertical merging is the optimal integrated spectrogram construction method, and Resnet-50 is a suitable deep learning model worthy of consideration.

Claims

1. A method for constructing an integrated spectrogram based on multi-channel signals for locating redundant objects, characterized in that: For the multi-channel signal used for positioning the redundant objects, each segment of the redundant object signal in each channel is recorded as a diagnostic signal, and a single pulse extracted based on a diagnostic signal is recorded as a pulse signal. The specific process of extracting the pulse signal includes the following steps: Step 1: Calculate the average energy and average zero-crossing rate of the entire sound signal, expressed as E a and Z a , calculate the standard deviation of the energy and the standard deviation of the zero-crossing rate of the entire sound signal, expressed as σ E and σ Z ; Set the peak energy threshold to E p =E a +σ E , set the endpoint zero crossing rate threshold to Z be =Z a -σ Z ; Step 2: Frame the sound signal and calculate the energy and zero-crossing rate of each frame signal; Step 3: Starting from the first frame signal, compare the energy of each frame signal with E p ; When the energy of a frame signal is greater than E p When the frame signal is recorded, the energy of each subsequent frame signal is compared with E p , until the energy of a frame signal is less than E p ; Find the frame signal with the largest energy from these frame signals, and obtain the time of the sampling point with the largest amplitude, which is called the peak time t of the current useful pulse p ; Step 4: Take the peak time t of the useful pulse extracted in step 3 p As the starting point, compare the zero-crossing rate of each frame signal with Z be Until the zero-crossing rate of a frame signal is found to be greater than Z in both directions be ; Respectively identify the previous frame signals of the two frame signals as the first start frame signal and the first end frame signal of the current useful pulse; obtain the time of the first sampling point in the first start frame signal, which is called the first start time t of the current useful pulse b_1 ; Get the time of the last sampling point in the first end frame signal, which is called the first end time of the current useful pulse t e_1 ; Step 5: Take the first end time t of the useful pulse extracted in step 4 e_1 The next frame signal is taken as a new starting point, and steps 3 and 4 are repeated to extract the second useful pulse; and so on, until the last frame signal of the sound signal is searched; and the extracted multiple useful pulses are saved; Step 6: Calculate the average zero-crossing rate and standard deviation of the zero-crossing rate of each useful pulse saved in step 5, expressed as Z i and σ Z_i ; Where i represents the serial number of the saved useful pulse; on this basis, the exclusive endpoint zero-crossing rate threshold of each useful pulse is set to Z be_i =Z i -σ Z_i ; Step 7: For the first useful pulse saved in step 5, take its peak time t p As the starting point, the zero-crossing rate of each frame signal is compared forward and backward with the exclusive endpoint zero-crossing rate threshold Z be_1 Until the zero-crossing rate of a frame signal is found to be greater than Z in both directions be_1 ; Respectively identify the previous frame signal of the two frame signals as the second starting frame signal and the second ending frame signal of the current useful pulse; obtain the time of the first sampling point in the second starting frame signal, which is called the second starting time t of the current useful pulse b_2 ; Get the time of the last sampling point in the second end frame signal, which is called the second end time t of the current useful pulse e_2 ; Step 8: starting from the second useful pulse saved in step 5, repeat step 7 to obtain the second starting time and the second ending time of each pulse, until all the useful pulses saved in step 5 are processed; Step 9: Calculate the first duration t of each useful pulse l_1 =t e_1 -t b_1 , the second duration t l_2 =t r_2 -t b_2 On this basis, the ratio of the second duration of each useful pulse to the first duration is calculated; if the ratio exceeds the duration threshold T, the first starting time t b_1 With the first end time t e_1 The final starting time t of the useful pulse is considered n and the final ending time t e Otherwise, the first starting time t b_1 and the second end time t e_2 The final starting time t of the useful pulse is considered b and the final ending time t e ; Step 10: retain the useful pulses processed in step 9, and the useful pulse extraction of the entire sound signal is completed; First, the redundant signal of each channel is converted into the spectrogram corresponding to each channel signal: the diagnostic signal or pulse signal is taken as the processing object, the processing object is first pre-emphasized, then windowed, and the windowed processing object is fast Fourier transformed, and then the power spectrum is passed through a bandpass filter, and then the spectrogram corresponding to each channel is converted; Then the spectrograms corresponding to the multiple channels are spliced ​​to obtain the multi-channel signal integrated spectrogram.

2. The method for constructing an integrated spectrogram based on multi-channel signals for locating redundant objects according to claim 1, characterized in that: When the spectrograms corresponding to multiple channels are spliced, a vertical splicing method is adopted to splice them, thereby obtaining a multi-channel signal integrated spectrogram.

3. The method for constructing an integrated spectrogram based on multi-channel signals for locating redundant objects according to any one of claims 1 to 2, characterized in that: When the diagnosis signal is taken as the processing object, during the windowing process, the window length is set to the time length of a section of the redundant signal.

4. The method for constructing an integrated spectrogram based on multi-channel signals for locating redundant objects according to any one of claims 1 to 2, characterized in that: When the diagnostic signal is used as the processing object, the FFT length is set to 2 during the fast Fourier transform process. k , k needs to satisfy: 2 k ≥ the number of sampling points contained in a redundant signal, and 2 k-1 <The number of sampling points contained in a redundant signal.

5. The method for constructing an integrated spectrogram based on multi-channel signals for locating redundant objects according to any one of claims 1 to 2, characterized in that: When a pulse signal is taken as a processing object, the window length is set to the time length of the pulse during the windowing process.

6. The method for constructing an integrated spectrogram based on multi-channel signals for locating redundant objects according to any one of claims 1 to 2, characterized in that: When the pulse signal is used as the processing object, the FFT length is set to 2 during the fast Fourier transform process. k’ , k' needs to satisfy: 2 k’ ≥The number of sampling points contained in a pulse signal, and 2 k’-1 <The number of sampling points contained in a pulse signal.

7. The method for constructing an integrated spectrogram based on multi-channel signals for locating redundant objects according to any one of claims 1 to 2, characterized in that: The duration threshold T is 0.82s.

8. The method for constructing an integrated spectrogram based on multi-channel signals for locating redundant objects according to claim 7, characterized in that: In the process of performing frame processing on the sound signal in step 2, the duration of each frame signal is set to Δt=50 μs.

9. The method for constructing an integrated spectrogram based on multi-channel signals for locating redundant objects according to claim 8, characterized in that: When the pulse signal is used as the processing object, after extracting a single pulse based on the diagnostic signal, before pre-emphasis processing, it is necessary to match the pulse extracted from each channel in the multi-channel signal. The matching process uses the zero-filling fusion threshold determination pulse matching algorithm for matching. The specific process includes the following steps: S201: The number of channels is recorded as N', and based on the starting time of the first pulse in the redundant object signal of each channel, the distance relationship between the redundant object and the sensor that captures the signal, as well as the corresponding relationship between the redundant object signal and the sensor that captures the signal are determined; the starting time of the first pulse in each of the N' channels is obtained respectively, which is expressed as t1, t2, t3, ..., t N’ ; S202: Calculate t N’ -t1, t N’ –t2, …, t N’ -t N’-1 , t N’ -t1, t N’ –t2, …, t N’ -t N’-1 They represent the time difference between the redundant object signal reaching the corresponding sensor and the farthest sensor; Add a time of t before the start time of the redundant object signal captured by the sensor closest to the redundant object N’ -t1 background noise, add a time of t before the start time of the redundant object signal captured by the sensor closest to the redundant object N’ –t2 noise floor, add a duration of t before the start of the signal of the unwanted object captured by the sensor that is the third closest to the unwanted object N’ –t3 noise floor, and so on, add a time length of t before the start time of the redundant object signal captured by the sensor closest to the redundant object N'-1 N’ -t N’-1 The background noise; On this basis, the time length t is discarded from the end time of the redundant object signal captured by the sensor closest to the redundant object N’ -t1 signal component, starting from the end time of the redundant object signal captured by the sensor closest to the redundant object, discard the time length t N’ –t2 signal component, starting from the end time of the redundant object signal captured by the sensor third closest to the redundant object, discard the time t N’ –t3 signal component, and so on, starting from the end time of the redundant object signal captured by the sensor closest to the redundant object N'-1, the time length is t N’ -t N’-1 signal components; S203: Taking the signal of the unwanted object captured by the sensor farthest from the unwanted object as a reference, the pulses extracted therefrom are expressed in sequence as P i , i is the pulse number in the redundant signal captured by the sensor; the starting time of these pulses is recorded as T i ; S204: Starting from P1, taking T1 time point as the reference, respectively search for pulses near the T1 reference in each segment of the redundant signal of the other N'-1 sensors; if the corresponding N'-1 segments of redundant signal can find pulses near the T1 reference, that is, N'-1 pulses are found, they are combined with P1 into a pulse group; otherwise, skip P1 and continue matching P2; S205: Follow the method of S204 to treat all P i Pulse matching is performed until all pulse matching of the redundant signal is completed, and multiple pulse groups are obtained, each of which is the pulse matching result corresponding to N' channels.

Images