An arbitrary sampling rate sound analysis method based on a pre-trained neural network

Abstract

The application discloses an arbitrary sampling rate sound analysis method based on a pre-training neural network and relates to the technical field of sound analysis. The method comprises the following steps: acquiring input audio, converting a spectrogram, and performing short-time Fourier transform on the input audio to obtain the spectrogram. The application solves the problem of difficulty in unified modeling of various sound signals (such as 8KHz-192KHz) with large sampling rate span differences. Different from a traditional resampling-based method, the application proposes sub-band segmentation of a fixed frequency band on a spectrogram in a time-frequency domain, models the sub-bands, can decompose the spectrogram of an arbitrary sampling rate into a combination of sub-band graphs, extracts high-dimensional representations of the sub-band graphs, performs vector splicing, and obtains a final signal representation. Vector distance calculation is performed at the high-dimensional representation level to determine the similarity between sound segments. The method is not only suitable for sound, but also suitable for digital signals such as high-frequency vibration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sound analysis, in particular to an arbitrary sampling rate sound analysis method based on a pre-trained neural network. BACKGROUND

[0002] There are a large number of different sampling rate sound signals in the real world, such as Internet audio and video, bandwidth limitations, industrial scenarios, or differences in acquisition equipment in civilian scenarios, which can result in different sampling rates of sound signals obtained. In voice communication, 8K-16KHz sampling is often used, 8KHz sampling rate is often used for narrowband voice in telephone communication, suitable for bandwidth limited scenarios, and 16KHz is often used for wideband voice communication and commercial voice recognition systems; 20KHz-48KHz is often used for high-quality music production, such as 22.05KHz for low-quality music or voice recording, which is the sampling rate of some early formats such as old MP3, 32KHz for FM radio, early television audio, 44.1KHz for CD audio digital music, and 48KHz is the professional audio and video industry standard sampling rate. Higher than 48KHz sampling rate such as 88.2KHz is used for high-resolution audio, 96KHz is used for professional recording, film audio and high-resolution audio sampling rate, 192KHz is used for high-end recording and experimental audio sampling rate, and higher than 384KHz is very rare and is only used for specific high-end audio equipment or experiments. Although the audible sound range of the human ear is 20Hz-20KHz, the ultrasonic frequency range beyond the audible sound segment is more common in the industrial field, and has a wide range of applications in detecting discharge and gas leakage.

[0003] The existing sound acquisition and analysis method mainly has the following two kinds: 1. In the frequency domain, the high frequency part of the 8KHz sampling rate is filled to reach 16KHz, so that the two sampling rate audios have the same input feature dimension in the frequency domain, and the 16KHz and 8KHz sampling rate signals share the same neural network model, which can realize the simultaneous processing of 16KHz and 8KHz sampling rate audios; 2. The resampling method selects a fixed sampling rate, and all signals are resampled to the frequency, and finally a neural network model is used to process the input signal of the specific sampling rate. The advantage of the method is that the processing procedure is simple, but the accurate information will not increase when the low sampling rate is upsampled to the high sampling rate, and the calculation amount will increase; when the high sampling rate is downsampled to the low sampling rate, the high sampling rate high frequency segment information will be lost, and the information will be wasted. In addition, this method is suitable for scenarios with relatively small frequency range changes, such as 8K-48KHz, and cannot realize the acquisition of 8KHz-192KHz sampling rate with large span and arbitrary frequency signals, so the high-frequency vibration and other high-frequency digital signal processing method is a technical problem to be solved by those skilled in the art. SUMMARY

[0004] In view of the deficiencies of the prior art, the application provides an arbitrary sampling rate sound analysis method based on a pre-trained neural network, which solves the problems in the background art.

[0005] To achieve the above object, the application is implemented by the following technical scheme: an arbitrary sampling rate sound analysis method based on a pre-trained neural network, comprising the following steps:

[0006] Step 1: Obtain a multi-dimensional audio signal, the multi-dimensional audio signal includes a one-to-many signal, a many-to-one signal and a many-to-many signal, and the multi-dimensional signal is preprocessed to obtain an input audio. Through preprocessing of the sampling signal, effective information can be extracted as the input audio in the sampling environment of multiple sampling ends and multiple sound sources, noise interference can be reduced, and the neural network training is facilitated.

[0007] Step 2: Convert the spectrogram. Since the audio is a high-frequency one-dimensional time sequence in the time domain, there are tens of thousands or even hundreds of thousands of sampling points per second, therefore, the conventional method of audio analysis is to perform short-time Fourier transform on the input audio to obtain a spectrogram. The horizontal axis of the spectrogram is time, and the vertical axis is the analysis frequency. A plurality of analysis windows are set, and the window length and window shift are defined. The short-time Fourier transform window length is usually 25 ms, and the window shift is 10 ms. Taking 16 KHz sampling rate sound as an example, the analysis window length is 400 sampling points per Fourier transform, and the interval between adjacent analysis windows is 160 points.

[0008] After the input audio is subjected to short-time Fourier transform, the high-frequency time sequence signal is transformed from the time domain to the time-frequency domain to obtain a spectrogram. For different sampling rate sounds, the time axis is the same, but the frequency axis differs greatly. The spectrogram is used as the representation of each input audio, and the height of the vertical axis of each spectrogram is completely different when inputting the neural network. The height information represents the frequency range. For the neural network, the size of the input spectrogram differs greatly, and the input matrix shape changes randomly, so it is difficult to use a neural network model for modeling. The number of signals with a sampling rate higher than 192 KHz is very small. For the data-driven method of the neural network, the sample size is too small, and the modeling effect is relatively poor.

[0009] Step 3: Due to the great difference in the vertical axis of the spectrogram obtained by short-time Fourier transform, in order to realize unified modeling, the modeling granularity is changed from the whole spectrogram to modeling of the subbands of the spectrogram. The spectrogram is divided into subbands, and the neural network learns the deep information of each subband. In order to ensure that the same model can process all sampling rate data, the subband width is set to a fixed width. In order to realize the processing capability of arbitrary sampling rate signals, fixed-width subbands are randomly selected from the whole spectrogram and input into the neural network for training during the training process.

[0010] Because the data labeling of the input audio is difficult and the labeling cost is high, the existing neural network learning audio model mainstream architecture is mainly unsupervised or self-supervised pre-training learning;

[0011] Compared with the method of using all spectrograms as input at a single sampling rate, to cope with modeling of any sampling rate, actually, the present application can use any pre-training method, the biggest change is that in neural network training, a random selected subband in the whole spectrogram is selected as input;

[0012] Step 4: self-supervised pre-training, the neural network training adopts an Encoder-Decoder structure, the Encoder outputs most of the information of the original subband spectrogram, therefore, in the inference stage, after short-time Fourier transform of the sound with an arbitrary sampling rate sr, the spectrogram frequency interval is ( 0 ， sr 2 ] The frequency interval is divided into subband combinations, and the subband less than one is truncated ( sr 2 − sr band ， sr 2 ] rounded to one subband, and the middle hidden layer representation is taken in each subband, the middle hidden layer representation is the feature representation output by the middle layer in the process of data processing of the neural network training model;

[0013] Step 5: vector splicing is performed on each subband, the high-dimensional representation of the sound with an arbitrary sampling rate can be extracted, and then judgment and verification are performed based on the high-dimensional representation, the high-dimensional representation is a data set formed by subband vector splicing, which is the feature information extracted in sound sampling, and the most typical application of the high-dimensional representation is voiceprint verification in speech, such as in the voiceprint verification process, a fixed dimension voiceprint vector is extracted for each speech, and the distance between voiceprint vectors of different speakers is calculated in the vector dimension, so that whether it is the same speaker can be judged, after the normal sound collected by the sensor is extracted as a voiceprint vector, the running state voiceprint vector is compared with the normal sound vector in real time, whether an abnormality occurs is judged by the voiceprint distance, and the commonly used vector distance is the pre-similarity, and the calculation formula is as follows:

[0014] ;

[0015] After the high-dimensional representation vectors of two signals of an arbitrary sampling rate signal are calculated, whether the two audios are similar can be judged by calculating the cosine distance of the high-dimensional representation vectors, so that tasks such as speaker verification and industrial abnormal sound detection can be completed.

[0016] Further, the one-to-many signal specifically refers to a sampling signal of multiple sound sources obtained by one sampling end, and the preprocessing of the one-to-many signal specifically includes the following steps:

[0017] Step 101: frequency filtering, setting filter and filtering range, the main frequency range of human voice is 80Hz-10kHz, the high-dimensional representation of the sampling rate sound is mainly applied to the verification analysis of voiceprint, so a filter is used to filter the one-to-many signal according to the filtering range to obtain a first signal, where H(f) is the first signal, f is the frequency of the sampling signal, and the transfer function of the filter is:

[0018] ;

[0019] wherein, is the lower cutoff frequency, is the upper cutoff frequency;

[0020] Step 102: harmonic analysis, performing harmonic analysis on the first signal to obtain a second signal X [ k ] ;

[0021] Step 103: loudness analysis, performing loudness analysis on the second signal X [ k ] to separate it into several input audios.

[0022] Further, the one-to-many signal specifically refers to obtaining a sampling signal of one sound source by multiple sampling ends, and the preprocessing of the one-to-many signal specifically includes the following steps:

[0023] Step 104: obtaining the number of sampling ends, when the number of sampling ends is equal to 2, two sampling ends are insufficient to obtain the specific position of the sound source by calculating the phase difference, so it is necessary to calculate the sound intensity to assist and cooperate, so as to determine the specific position of the sound source, and jump to step 106, otherwise, when the number of sampling ends is greater than 2, proceed to the next step 105;

[0024] Step 105: comparing and calculating the phase difference between each pair of sampling ends , the calculation formula of the phase difference is:

[0025] ;

[0026] wherein, is the distance between sampling end A and sampling end B, and λ is the wavelength, which is obtained by dividing the sound speed by the frequency of the sampling signal of sampling end A or sampling end B, and the position information of the sound source is obtained by comparing the phase difference between the sampling ends , the position information of the sound source is converted into the relative position of each sampling signal, the relative position includes the distance and angle of the sound source from the sampling end, and the relative position is bound with the sampling signal of the corresponding sampling end to obtain the input audio.

[0027] Further, the one-to-many signal specifically refers to obtaining a sampling signal of one sound source by multiple sampling ends, and the preprocessing of the one-to-many signal specifically includes the following steps:

[0028] Step 110: Phase subtracting the sampling signal of the sampling end with the sampling signal of another sampling end farthest away from it to obtain a third signal, which can highlight the sound source near the sampling end farthest away;

[0029] Step 111: Repeating step 110 to obtain the third signal corresponding to each sampling end, and inputting the third signal into the beam forming calculation formula to obtain the fourth signal of each sampling end The calculation formula of beam forming is as follows:

[0030] ;

[0031] Wherein, M is the number of sampling ends, is the weight coefficient of the mth sampling end, is the sampling signal received by the mth sampling end at time t, and the sampling signal here is the original signal collected initially, is the third signal obtained by phase subtracting the mth sampling end, and the fourth signal obtained by calculation is the preprocessed signal of the sound source near the sampling end farthest away;

[0032] Step 112: Calculate the phase difference between the fourth signal and the sampling signal of each sampling end, and the calculation formula of the phase difference is consistent with step 105. The distance between the sampling end and another sampling end farthest away is marked as the standard distance, and the phase difference is multiplied by the standard distance to obtain the fourth signal corresponding to the third component of the sound source;

[0033] Step 113: Calculate the ratio of the fourth signals of two adjacent sampling ends , and multiply the ratio by the distance between the adjacent sampling ends to obtain the fourth component;

[0034] Step 114: Add the third component and the fourth component to obtain the corresponding sound source position information, convert the position information of the sound source into the relative position of each sampling end, and the relative position includes the distance and angle of the sound source from the sampling end. Bind the relative position and the sampling signal of the corresponding sampling end to obtain the input audio.

[0035] Further, the calculation steps of the second component are as follows:

[0036] Convert the sampling signal into a frequency response diagram, calculate the average loudness value of the frequency response diagram, and mark it as P. P can be used as the sound source power value obtained by the sampling end. Since the actual sound source power value is proportional to the loudness value, the average loudness value is used instead of the sound source power value. Calculate the sound intensity I1 and I2 of two sampling ends respectively, and the calculation formula is:

[0037] ;

[0038] wherein r1 and r2 are respectively the first component distance between the two sampling ends and the sound source, and I1 and I2 are respectively the second component of the sound intensity of the two sampling ends.

[0039] Further, in the self-supervised pre-training, the random masking specifically comprises the following steps:

[0040] Step 401: dividing the sub-band into a plurality of rectangular patches with equal areas, the side length of the rectangular patch being g1 and g2 respectively, g1 being the frequency dimension and g2 being the time dimension;

[0041] Step 402: setting a masking matrix, the masking matrix complying with the following conditions:

[0042] Condition one, in the same time dimension, the masked rectangular patch retains at least 2 and the rectangular images are not adjacent;

[0043] Condition two, the masking range is greater than 70%, that is, more than 70% of the rectangular patches in the masked sub-band;

[0044] Step 403: recording the masking position of each rectangular patch in the sub-band, setting the masking weight coefficient to be bound with the masking position, and increasing the value of the masking weight coefficient corresponding to the masking position after each training, the better the reconstruction effect, the increase amplitude being 0.1%.

[0045] Further, the harmonic analysis on the first signal specifically comprises the following steps:

[0046] The filtered first signal is split into a discrete signal with a length N x [ n ] wherein n=0, 1, …, N-1, the first signal is subjected to harmonic calculation based on the discrete Fourier transform to obtain a second signal X [ k ] The harmonic calculation formula is:

[0047] X [ k ] = ∑ n = 0 N − 1 x [ n ] e − j 2 π N kn ;

[0048] wherein k=0, 1, …, N-1, X [ k ] is the discrete signal x [ n ] The second signal obtained after the harmonic calculation represents the discrete signal x [ n ] at different frequency points k, e is a natural constant, j is an imaginary unit, and the harmonic calculation formula is avoided to be unsolvable in the real number range, and from the physical meaning, X [ k ] the amplitude of | X [ k ] | the relative intensity of the harmonic component with a frequency of is a sampling frequency, X [ k ] the phase of the harmonic component with a frequency of ∠ X [ k ] is phase information of the harmonic component with the frequency.

[0049] Further, the loudness analysis specifically includes the following steps:

[0050] convert the second signal into a frequency response graph, record the loudness values of each peak and each trough of the waveform in the frequency response graph, count the loudness values of each peak and each trough, mark the frequency corresponding to the trough loudness value with the highest occurrence frequency as the first reference frequency k1, mark the frequency corresponding to the peak loudness value with the highest occurrence frequency as the second reference frequency k2, and convert the second signal X [ k ] into a power spectrum P [ k ] = | X [ k ] | 2 , the first reference frequency k1 and the second reference frequency k2 define a frequency range, and the root mean square loudness L RMS in the frequency range is calculated, and the calculation formula is:

[0051] L RMS = 1 K ∑ k = k 1 k 2 P [ k ] ;

[0052] wherein K is the number of frequency points in the frequency range, K satisfies the condition K=k2-k1+1, and the root mean square loudness L RMS separate the second signal into a plurality of input audios.

[0053] Further, when the number of sampling ends is equal to 2, the following steps are included:

[0054] Step 106: calculate the phase difference between the two sampling ends, and the calculation formula is consistent with step 105, the phase difference between the two sampling ends is multiplied by the distance to obtain the first component corresponding to the sampling end;

[0055] Step 107: analyze the sound intensity of the sampling signal, and convert the sound intensity into a second component;

[0056] Step 108: add the first component and the second component of the sampling end to obtain the corresponding sound source position information;

[0057] Step 109: convert the position information of the sound source into the relative position of each sampling signal, the relative position includes the distance and angle of the sound source from the sampling end, and bind the relative position with the sampling signal of the corresponding sampling end to obtain the input audio. ​

[0058] Further, when the subbands are vector spliced, it is necessary to verify that the intermediate hidden layer representations in each subband are in the same dimension, and if the dimensions are inconsistent, the number of occurrences of each dimension in the subbands is counted, the dimension with the most occurrences is marked as the reference dimension of the subband, and the subbands with inconsistent intermediate hidden layer representation dimensions are adjusted according to the reference dimension to keep the dimensions consistent.

[0059] The intermediate hidden layer representation is represented using the numpy library, and the np.concatenate function is used for vector splicing of the subbands.

[0060] The present application has the following beneficial effects:

[0061] 1. The present application proposes an arbitrary sampling rate sound analysis method based on a pre-trained neural network, mainly solving the problem of unified modeling of various sound signals with large sampling rate span (such as 8KHz-192KHz), unlike traditional resampling-based methods, the present application proposes subband segmentation of a fixed frequency band on a spectrogram in the time-frequency domain, models the subbands, can decompose the spectrogram of any sampling rate into a combination of subband graphs, extracts high-dimensional representations from the subband graphs, and performs vector splicing to obtain the final signal representation, and performs vector distance calculation at the high-dimensional representation level to determine the similarity between sound segments, this method is not only suitable for sound, but also suitable for high-frequency vibration and other digital signals;

[0062] 2. The present application not only solves the problem of unified modeling of arbitrary sampling rate sound, but also is very beneficial for alleviating and ultimately overcoming the problem of data scarcity at a certain sampling rate, since the modeling is the expression of the subband spectrogram of any frequency, the sampling rate signal with less samples can reuse the spectrogram data of different frequency bands of other sampling rates, thereby increasing the generalization and migration ability of the model;

[0063] 3. Through pre-processing of the sampling signal, effective information can be extracted as input audio in a sampling environment with multiple sampling ends and multiple sound sources, noise interference can be reduced, and the neural network can be trained;

[0064] Of course, any product implementing the present application does not necessarily need to achieve all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed for the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0066] Figure 1A flowchart of a pre-trained neural network-based arbitrary sampling rate sound analysis method of the present application;

[0067] Figure 2 A spectrogram and analysis window schematic diagram of the present application;

[0068] Figure 3 A spectrogram sub-band division schematic diagram of the present application;

[0069] Figure 4 A flowchart of an existing unsupervised pre-training model;

[0070] Figure 5 A flowchart of a self-supervised pre-training of the present application. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0072] Please refer to Figure 1 The present application provides a technical solution: a pre-trained neural network-based arbitrary sampling rate sound analysis method, comprising the following steps:

[0073] Step 1: Obtain a multi-dimensional audio signal, the multi-dimensional audio signal includes one-to-many signals, many-to-one signals and many-to-many signals, and pre-process the multi-dimensional signals to obtain an input audio;

[0074] Step 2: Convert a spectrogram. Since audio is a high-frequency one-dimensional time sequence in the time domain, with up to tens of thousands or even hundreds of thousands of sampling points per second, the conventional approach for audio analysis is to perform short-time Fourier transform on the input audio to obtain a spectrogram. The horizontal axis of the spectrogram is time, and the vertical axis is the analysis frequency. A number of analysis windows are set, usually with a short-time Fourier transform window length of 25 ms and a window shift of 10 ms. Taking a 16 KHz sampling rate sound as an example, the analysis window length for each Fourier transform is 400 sampling points, and the interval between adjacent analysis windows is 160 points, as shown in Figure 2 ;

[0075] The input audio is transformed into the time-frequency domain by short-time Fourier transform, and the high-frequency time sequence signal is transformed from the time domain to the time-frequency domain to obtain a spectrogram. For different sampling rates of sound, the time axis is the same, but the frequency axis differs greatly. For example, for 16 KHz sound, the frequency axis value interval is (0, 8000] Hz, for 48 KHz sound, the frequency axis value interval is (0, 24000] Hz, and for 192 KHz sound, the frequency axis value interval is (0, 96000] Hz. The spectrogram is used as the representation of each input audio, and the height of the vertical axis of each spectrogram is completely different when inputting the neural network. The height information represents the frequency range and has a clear physical meaning. For the neural network, the size of the input spectrogram differs greatly, and the input matrix shape changes randomly. Therefore, it is difficult to use a neural network model for modeling. In addition, the number of signals of different sampling rates is inconsistent. For example, the signal with a sampling rate higher than 192 KHz is very rare. For the neural network, which is a data-driven method, the sample size is too small, and the modeling effect is relatively poor.

[0076] Step 3: Due to the large difference in the vertical axis of the spectrogram obtained by short-time Fourier transform, in order to realize unified modeling, the modeling granularity is changed from the whole spectrogram to the modeling of the subband of the spectrogram. The spectrogram is divided into subbands, and the deep information of each subband is learned by the neural network. In order to ensure that the same model can process all sampling rate data, the subband width is set to a fixed width. For example, if the subband width is selected to be 1 KHz, the 8 KHz sampling rate signal is transformed into a spectrogram with a frequency range of (0, 4000] Hz, which can be divided into 4 non-overlapping subbands. In order to realize the processing capability of signals with arbitrary sampling rates, during the training process, a fixed-width subband is randomly selected from the whole spectrogram and input into the neural network for training, as shown in Figure 3 .

[0077] Due to the difficulty and high cost of data labeling of input audio, the mainstream architecture of existing neural network learning audio models is mainly unsupervised or self-supervised pre-training learning. For example, in the AudioMAE unsupervised pre-training model, after inputting the spectrogram, the spectrogram is randomly masked, the unmasked part is used as the Encoder input for encoding, and the masked part is replaced with the MSK symbol and input into the Decoder to reconstruct the original spectrogram block, as shown in Figure 4 . The modeling target of this model is the spectrogram MSE (Mean Square Error), and the calculation formula is as follows:

[0078] .

[0079] wherein, is the i-th row, j-th column element of the original spectrogram, The smaller the loss function is, the better the reconstruction effect is, and the better the pre-training model is for reconstructing the i-th row and j-th column element of the spectrogram;

[0080] Compared with the method of using all spectrograms as input in AudioMAE for a single sampling rate, in order to model any sampling rate, the sub-band spectrogram is used as the input of AudioMAE, and only the sub-band information is learned by the neural network. Actually, any pre-training method can be used, and the biggest change is that the sub-band randomly selected from the whole spectrogram is randomly selected as the input of AudioMAE in neural network training.

[0081] Step 4: Self-supervised pre-training, the neural network training adopts an Encoder-Decoder structure, the Encoder outputs most of the information of the original sub-band spectrogram, therefore, in the inference stage, after performing short-time Fourier transform on the sound with an arbitrary sampling rate, the frequency interval of the spectrogram is ( 0 ， sr 2 ] The frequency interval is divided into sub-band combinations, and the sub-band less than one is truncated ( sr 2 − sr band ， sr 2 ] to one sub-band, and the middle hidden layer representation is taken from each sub-band based on random masking, as shown in Figure 5 ;

[0082] Step 5: Vector splicing is performed on each sub-band, steps 1-4 describe the detailed process of the application, and the high-dimensional representation of the arbitrary sampling rate sound can be extracted in Figure 1 Step 5: Vector splicing is performed on each sub-band, steps 1-4 describe the detailed process of the application, and the high-dimensional representation of the arbitrary sampling rate sound can be extracted in

[0083] ;

[0084] In actual application, after calculating the high-dimensional representation vectors of two signals of an arbitrary sampling rate signal, the cosine distance of the high-dimensional representation vectors can be calculated to determine whether the two audio signals are similar, so that tasks such as speaker verification and industrial abnormal sound detection can be completed.

[0085] The one-to-many signal specifically refers to a sampling signal obtained by sampling multiple sound sources at one sampling end, and the preprocessing of the one-to-many signal specifically includes the following steps:

[0086] Step 101: frequency filtering, set filter and filter range 80Hz-10kHz, the main frequency range of human voice is 80Hz-10kHz, the high-dimensional representation of the sampling rate sound is mainly applied to the verification analysis of voiceprint, therefore, a one-to-many signal is filtered according to the filter range to obtain a first signal using a filter, and the transfer function of the filter is:

[0087] ;

[0088] wherein, is the lower cutoff frequency, is the upper cutoff frequency; otherwise means in addition.

[0089] Step 102: harmonic analysis, performing harmonic analysis on the first signal to obtain a second signal X [ k ] ;

[0090] Step 103: loudness analysis, performing loudness analysis on the second signal X [ k ] to separate into several input audios.

[0091] Wherein, the many-to-one signal specifically refers to obtaining a sampling signal of an audio source by a plurality of sampling ends, and the preprocessing of the many-to-one signal specifically includes the following steps:

[0092] Step 104: obtaining the number of sampling ends, when the number of sampling ends is equal to 2, two sampling ends are insufficient to obtain the specific position of the audio source by calculating the phase difference, therefore, the sound intensity needs to be calculated to assist and cooperate, so as to determine the specific position of the audio source, and jump to step 106, otherwise, when the number of sampling ends is greater than 2, the next step 105 is performed.

[0093] Step 105: comparing and calculating the phase difference between each pair of sampling ends , and the calculation formula of the phase difference is:

[0094] ;

[0095] wherein, is the distance between the sampling end A and the sampling end B, and λ is the wavelength, which is obtained by dividing the sound speed by the frequency of the sampling signal of the sampling end A or the sampling end B, and the position information of the audio source is obtained by comparing the phase difference between the sampling ends , the position information of the audio source is converted into the relative position of each sampling signal, the relative position includes the distance and angle of the audio source from the sampling end, and the relative position is bound with the sampling signal of the corresponding sampling end to obtain the input audio.

[0096] Wherein, the many-to-many signal specifically refers to obtaining sampling signals of a plurality of audio sources by a plurality of sampling ends, and the preprocessing of the many-to-many signal specifically includes the following steps:

[0097] Step 110: Phase subtracting the sampling signal of the sampling end with the sampling signal of another sampling end farthest away from it to obtain a third signal, which can highlight the sound source near the sampling end farthest away;

[0098] Step 111: Repeating step 110 to obtain the third signal corresponding to each sampling end, and inputting the third signal into the beam forming calculation formula to obtain the fourth signal of each sampling end The calculation formula of beam forming is as follows:

[0099] ;

[0100] Wherein, M is the number of sampling ends, is the weight coefficient of the mth sampling end, Generally, the value is 1, representing that the sampling signal received by each sampling end is an equivalent signal, is the sampling signal received by the mth sampling end at time t, wherein the sampling signal is the original signal collected initially, is the third signal obtained by phase subtracting the mth sampling end, The fourth signal obtained by calculation is the pretreatment signal of the sound source near the sampling end farthest away;

[0101] Step 112: Calculating the phase difference between the fourth signal and the sampling signal of each sampling end, and the calculation formula of the phase difference is consistent with step 105, marking the distance between the sampling end and another sampling end farthest away as the standard distance, and multiplying the phase difference by the standard distance to obtain the fourth signal corresponding to the third component of the sound source;

[0102] Step 113: Calculating the ratio of the fourth signals of two adjacent sampling ends , and multiplying the ratio by the distance between the adjacent sampling ends to obtain the fourth component;

[0103] Step 114: Adding the third component and the fourth component to obtain the position information of the corresponding sound source, converting the position information of the sound source into the relative position of each sampling end, the relative position including the distance and angle of the sound source from the sampling end, and binding the relative position and the sampling signal of the corresponding sampling end to obtain the input audio.

[0104] Wherein, the calculation steps of the second component are as follows:

[0105] Convert the sampling signal into a frequency response diagram, calculate the average loudness value of the frequency response diagram, mark it as P, P can be obtained as the sound source power value at the sampling end, because the actual sound source power value is proportional to the loudness value, so the average loudness value is used instead of the sound source power value, and the sound intensity I1 and I2 of the two sampling ends are calculated respectively, and the calculation formula is:

[0106] ;

[0107] Wherein, r1 and r2 are the first component distance between the two sampling ends and the sound source, and the sound intensity I1 and I2 are marked as the second component of the two sampling ends.

[0108] Among them, in the self-supervised pre-training, the random masking specifically includes the following steps:

[0109] Step 401: divide the sub-band into a plurality of rectangular blocks with equal area, and the side length of the rectangular block is g1 and g2, g1 is the frequency dimension, and g2 is the time dimension;

[0110] Step 402: set the masking matrix, and the masking matrix follows the following conditions:

[0111] Condition one, in the same time dimension, the masked rectangular block retains at least two and the rectangular image is not adjacent;

[0112] Condition two, the masking range is greater than 70%, that is, more than 70% of the rectangular blocks in the masked sub-band;

[0113] Step 403: record the masking position of each rectangular block in the sub-band, set the masking weight coefficient bound to the masking position, and the better the reconstruction effect after each training, the greater the value of the masking weight coefficient corresponding to the masking position, and the increase amplitude is 0.1%.

[0114] Wherein, the harmonic analysis on the first signal specifically includes the following steps:

[0115] The filtered first signal is split into a discrete signal of length N x [ n ] , wherein n=0, 1, …, N-1, the first signal is calculated based on the discrete Fourier transform to obtain the second signal X [ k ] , and the harmonic calculation formula is:

[0116] X [ k ] = ∑ n = 0 N − 1 x [ n ] e − j 2 π N kn ;

[0117] Wherein, k=0, 1, …, N-1, X [ k ] is a discrete signal x [ n ] The second signal obtained after the harmonic calculation represents the discrete signal x [ n ] The complex value at different frequency points k, e is a natural constant, e is approximately equal to 2.7183, j is an imaginary unit, and the harmonic calculation formula is avoided in the real number range, and in the physical sense, X [ k ] The amplitude of | X [ k ] | Indicates the relative strength of the harmonic component of the signal with a frequency of , The sampling frequency is X [ k ] The phase of ∠ X [ k ] The phase information of the harmonic component of the frequency.

[0118] The loudness analysis specifically includes the following steps:

[0119] The second signal is converted into a frequency response diagram, the loudness values of each peak and each trough of the waveform in the frequency response diagram are recorded, the loudness values of each peak and each trough are counted, the frequency corresponding to the most frequently occurring trough loudness value is marked as the first reference frequency k1, the frequency corresponding to the most frequently occurring peak loudness value is marked as the second reference frequency k2, and the second signal X [ k ] is converted into a power spectrum P [ k ] = | X [ k ] | 2 , the first reference frequency k1 and the second reference frequency k2 define a frequency range, and the root mean square loudness L RMS in the frequency range is calculated, and the calculation formula is:

[0120] L RMS = 1 K ∑ k = k 1 k 2 P [ k ] ;

[0121] Wherein, K is the number of frequency points in the frequency range, K satisfies the condition K=k2-k1+1, and the root mean square loudness L RMS The second signal is separated into several input audios.

[0122] When the number of sampling ends is equal to 2, the following steps are included:

[0123] Step 106: Calculate the phase difference between the two sampling ends , the calculation formula is consistent with step 105, and the phase difference between the two sampling ends is multiplied by the distance to obtain the first component corresponding to the sampling end;

[0124] Step 107: Analyze the sound intensity of the sampling signal, and convert the sound intensity into a second component;

[0125] Step 108: adding the first component and the second component of the sampling end to obtain corresponding sound source position information;

[0126] Step 109: converting the position information of the sound source into the relative position of each sampling signal, the relative position including the distance and angle of the sound source from the sampling end, and binding the relative position with the sampling signal of the corresponding sampling end to obtain the input audio.

[0127] Wherein, when the sub-band is vector spliced, it is necessary to verify that the intermediate hidden layer representation in each sub-band is in the same dimension, if the dimensions are inconsistent, the number of occurrences of each dimension in the sub-band is counted, the dimension with the most occurrences is marked as the reference dimension of the sub-band, and the sub-band with inconsistent intermediate hidden layer representation dimensions is adjusted according to the reference dimension to keep the dimensions consistent.

[0128] The intermediate hidden layer representation is output through the intermediate layer of the neural network, the np.concatenate function is used for vector splicing of the sub-band, and the numpy library and the np.concatenate function are both located in the Python development environment.

[0129] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can make equivalent replacement or change according to the technical scheme and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A pre-trained neural network based arbitrary sampling rate sound analysis method, characterized by: The method comprises the following steps: Step 1: obtaining input audio; Step 2: converting a spectrogram, performing short-time Fourier transform on the input audio to obtain a spectrogram, the horizontal axis of the spectrogram is time, the vertical axis is analysis frequency, a plurality of analysis windows are set, the window length and window shift are defined, the window length of the short-time Fourier transform is 25 ms, and the window shift is 10 ms; Step 3: changing the modeling granularity from the whole spectrogram to modeling the sub-band of the spectrogram, dividing the spectrogram into sub-bands, learning deep information of each sub-band through a neural network, and setting a fixed width for the sub-band width to ensure that the same model can process all sampling rate data, and in the training process, a sub-band with a fixed width is randomly selected from the spectrogram and input into the neural network for training; Step 4: Self-supervised pre-training, the neural network training adopts an Encoder-Decoder structure, the Encoder outputs modeling most of the information of the original sub-band spectrogram, in the inference stage, after the short-time Fourier transform of the sound with an arbitrary sampling rate sr, the frequency interval of the spectrogram is , the frequency interval is divided into sub-band combinations, and the insufficient one sub-band is truncated to one sub-band, and the middle hidden layer representation is taken from each sub-band based on random masking; Step 5: vector splicing each sub-band to extract a high-dimensional representation of the sound of any sampling rate, and then judging and verifying based on the high-dimensional representation; The one-to-many signal specifically refers to obtaining sampling signals of a plurality of sound sources by one sampling end, and the preprocessing of the one-to-many signal specifically comprises the following steps: Step 101: frequency filtering, setting a filter and a filtering range, the main frequency range of human voice is 80Hz-10kHz, and the high-dimensional representation of the sampling rate sound is mainly applied to voiceprint verification analysis, therefore, the one-to-many signal is filtered according to the filtering range to obtain a first signal, wherein H(f) is the first signal, f is the frequency of the sampling signal, and the transfer function of the filter is: ； wherein is the lower cut-off frequency, is the upper cut-off frequency; Step 102: harmonic analysis, performing harmonic analysis on the first signal to obtain a second signal ; Step 103: Loudness analysis of the second signal The loudness analysis separates the input audio into several input audio; Further, the many-to-one signal specifically refers to obtaining sampling signals of one sound source by a plurality of sampling ends, and the preprocessing of the many-to-one signal specifically comprises the following steps: Step 104: obtaining the number of sampling ends, when the number of sampling ends is equal to 2, the two sampling ends are insufficient to obtain the specific position of the sound source by calculating the phase difference, therefore, the sound intensity needs to be calculated to assist and cooperate, so as to determine the specific position of the sound source, and the step 106 is jumped to, otherwise, when the number of sampling ends is greater than 2, the step 105 is performed; Step 105: Calculate the phase difference between each pair of sampling ends The formula for calculating the phase difference is: ； wherein, is the distance between the sampling end A and the sampling end B, λ is the wavelength, which is obtained by dividing the sound speed by the frequency of the sampling signal of the sampling end A or the sampling end B, the position information of the sound source is obtained by comparing the phase difference between the sampling ends , the position information of the sound source is converted into the relative position of each sampling signal, the relative position includes the distance and the angle of the sound source from the sampling end, and the input audio is obtained by binding the relative position with the sampling signal of the corresponding sampling end; When the number of sampling ends is equal to 2, the following steps are included: Step 106: calculate the phase difference between the two sampling ends The calculation formula is consistent with step 105, and the phase difference between the two sampling ends The ratio is multiplied by the interval The first component corresponding to the sampling end is obtained; Step 107: analyzing the sound intensity of the sampling signal, and converting the sound intensity into a second component; Step 108: adding the first component and the second component of the sampling end to obtain corresponding sound source position information; Step 109: converting the position information of the sound source into the relative position of each sampling signal, the relative position includes the distance and angle of the sound source from the sampling end, and the relative position is bound with the sampling signal of the corresponding sampling end to obtain the input audio; The many-to-many signal specifically refers to obtaining sampling signals of a plurality of sound sources by a plurality of sampling ends, and the preprocessing of the many-to-many signal specifically comprises the following steps: Step 110: performing phase subtraction on the sampling signal of the sampling end and the sampling signal of another sampling end farthest away to obtain a third signal, and the third signal can highlight the sound source near the sampling end farthest away; Step 111: repeating step 110 to obtain a third signal corresponding to each sampling end, and inputting the third signal into a beam forming calculation formula to obtain a fourth signal of each sampling end The calculation formula of the beam forming is as follows: ； Wherein, M is the number of sampling ends, is the weight coefficient of the mth sampling end, is the sampling signal received by the mth sampling end at time t, wherein the sampling signal is the original signal collected initially, is the third signal obtained by phase subtraction of the mth sampling end, and the fourth signal obtained by calculation is the preprocessed signal of the sound source near the sampling end closest to the farthest distance. Step 112: calculating the fourth signal The phase difference of the sampling signal of each sampling end is calculated according to the same formula as step 105, the interval between the sampling end and another sampling end farthest from it is marked as the standard interval, and the fourth signal is obtained by multiplying the phase difference by the standard interval The third component corresponding to the sound source; Step 113: Calculate the fourth signal at two adjacent sampling points. The ratio is multiplied by the distance between adjacent sampling points to obtain the fourth component; Step 114: adding the third component and the fourth component to obtain corresponding sound source position information, converting the position information of the sound source into the relative position of each sampling end, the relative position including the distance and angle of the sound source from the sampling end, and binding the relative position with the sampling signal of the corresponding sampling end to obtain the input audio.

2. The arbitrary sampling rate sound analysis method based on a pre-trained neural network according to claim 1, characterized in that, In step 2, after the input audio is subjected to short-time Fourier transform, the high-frequency time sequence signal is transformed from the time domain to the time-frequency domain to obtain a spectrogram. 3.The arbitrary sampling rate sound analysis method based on pre-trained neural network according to claim 1, wherein, In step 3, the mainstream architecture of the audio model learned by the neural network is mainly based on unsupervised or self-supervised pre-training learning, specifically: the AudioMAE unsupervised pre-training model, after inputting the spectrogram, randomly masking the spectrogram, the unmasked part is input into the encoder for encoding, the masked part is input into the decoder after being replaced by the MSK symbol, and the original spectrogram block is reconstructed, the modeling target is the spectrogram MSE, and the calculation formula is as follows: ； wherein, is the i-th row, j-th column element of the original spectrogram, is the i-th row, j-th column element of the reconstructed spectrogram.

4. The arbitrary sampling rate sound analysis method based on a pre-trained neural network according to claim 1, characterized in that, In step 5, the application of high-dimensional representation is voiceprint verification in speech. In the voiceprint verification process, a fixed-dimensional voiceprint vector is extracted from each segment of speech, and the distance between voiceprint vectors of different speakers is calculated in this vector dimension, that is, whether it is the same speaker is judged. In industrial applications, after the normal sound collected by the sensor is extracted into a voiceprint vector, the running state voiceprint vector is compared with the normal sound vector in real time, whether an abnormality occurs is judged by the voiceprint distance, and the commonly used vector distance is the pre-similarity, and the calculation formula is as follows: ； After calculating the high-dimensional representation vectors of two signals of any sampling rate signals, the cosine distance between the high-dimensional representation vectors is calculated, that is, whether the two audio signals are similar is judged, so as to complete the speaker verification and industrial abnormal sound detection tasks.

Images