Any sampling rate sound analysis method based on pre-training neural network

Abstract

The invention discloses an arbitrary sampling rate sound analysis method based on a pre-trained neural network, and relates to the technical field of sound analysis. Comprising the following steps: acquiring an input audio, converting a spectrogram, and performing short-time Fourier transform on the input audio to obtain the spectrogram. According to the method, the problem that unified modeling of various sound signals (such as 8KHz-192KHz) with large sampling rate span difference is difficult is solved. Different from a traditional method based on resampling, the method provided by the invention carries out sub-band segmentation of a fixed frequency band on a spectrogram of a time-frequency domain, carries out modeling on sub-bands, can decompose the spectrogram of any sampling rate into a combination of sub-band graphs, carries out vector splicing after carrying out high-dimensional representation extraction on the sub-band graphs, and obtains final signal representation. And carrying out vector distance calculation on the high-dimensional representation level to judge the similarity between the sound segments. The method is not only suitable for sound, but also suitable for digital signals such as high-frequency vibration.

Description

Technical Field

[0001] The present invention relates to the technical field of sound analysis, and in particular to a sound analysis method at an arbitrary sampling rate based on a pre-trained neural network. Background Art

[0002] In the real world, there are a large number of sound signals with different sampling rates. For example, in Internet audio and video, due to bandwidth limitations, differences in acquisition equipment in industrial or civilian scenarios, the sampling rates of the obtained sound signals vary greatly. 8K-16KHz sampling is often used in voice communication. 8KHz sampling rate is often used for narrowband voice in telephone communication, which is suitable for bandwidth-limited scenarios. 16KHz is often used for broadband voice communication and commercial voice recognition systems; 20KHz-48KHz is often used for high-quality music production, such as 22.05KHz for low-quality music or voice recording, which is the sampling rate of some early formats such as old MP3, 32KHz for FM broadcasting, early TV audio, 44.1KHz for CD audio digital music, etc., and 48KHz is the standard sampling rate for professional audio and video industries. Sampling rates higher than 48KHz, such as 88.2KHz, are used for high-resolution audio, 96KHz is the sampling rate for professional recording, movie audio and high-resolution audio, 192KHz is the sampling rate for high-end recording and experimental audio, and higher sampling rates above 384KHz are extremely rare and are only used for specific high-end audio equipment or experiments. Although the human ear can hear sounds in the range of 20Hz-20KHz, and the frequency range beyond the audible sound band is the ultrasonic frequency band, the ultrasonic frequency band is a more common processing signal in the industrial field and is widely used in detecting discharges and gas leaks.

[0003] There are two main existing sound collection and analysis methods: 1. Fill the high frequency part of the 8KHz sampling rate in the frequency domain to achieve the same processing bandwidth of 16KHz, so that the input feature dimensions of the two sampling rate audios in the frequency domain are consistent, and the 16KHz and 8KHz sampling rate signals share the same neural network model, which can realize the simultaneous processing of 16KHz and 8KHz sampling rate audio; 2. The resampling method selects a fixed sampling rate, resamples all signals to this frequency, and finally uses a neural network model to process the input signal of the specific sampling rate. The advantage of this method is that the processing flow is simple, but when the low sampling rate is upsampled to a high sampling rate, the accurate information will not increase, and the amount of calculation will increase; when the high sampling rate is downsampled to a low sampling rate, part of the information in the high frequency band of the high sampling rate will be lost, wasting information. In addition, this method is suitable for scenarios where the frequency range changes relatively small, such as 8K-48KHz, and cannot realize the acquisition of 8KHz-192KHz sampling rate spans too large and arbitrary frequency signals. Therefore, it is suitable for high-frequency vibration and other high-frequency digital signal processing methods. It is a technical problem that technicians in this field need to solve. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a method for analyzing sound at an arbitrary sampling rate based on a pre-trained neural network, which solves the problems raised in the above-mentioned background technology.

[0005] To achieve the above objectives, the present invention is implemented by the following technical solution: a method for analyzing sound at any sampling rate based on a pre-trained neural network, comprising the following steps: Step 1: Obtain a multi-dimensional audio signal, which includes a one-to-many signal, a many-to-one signal, and a many-to-many signal. Preprocess the multi-dimensional signal to obtain input audio. By preprocessing the sampled signal, effective information can be extracted as input audio in a sampling environment of multiple sampling ends and multiple sound sources, which can reduce noise interference and is conducive to the training of the neural network. Step 2: Convert to spectrogram. Since audio is a high-frequency one-dimensional time series in the time domain, with tens of thousands or even hundreds of thousands of sampling points per second, the conventional approach to audio analysis is to perform short-time Fourier transform on the input audio to obtain a spectrogram. The horizontal axis of the spectrogram is time, and the vertical axis is the analysis frequency. Set a number of analysis windows, define the window length and window shift. Usually, the short-time Fourier transform window length is 25ms, and the window shift is 10ms. Taking 16KHz sampling rate sound as an example, the Fourier transform analysis window length is 400 sampling points each time, and the interval between adjacent analysis windows is 160 points. After the input audio is short-time Fourier transformed, the high-frequency time series signal is transformed from the time domain to the time-frequency domain to obtain a spectrogram. For sounds with different sampling rates, the time axis is the same, but the frequency axis is quite different. The spectrogram is used as the representation of each input audio. When input into the neural network, the vertical axis height of each spectrogram is completely different. The height information represents the frequency range. For the neural network, the size of the input spectrogram varies greatly, and the shape of the input matrix changes at any time, so it is difficult to use a neural network model for modeling. The number of signals with different sampling rates is inconsistent, and signals above 192KHz are very rare. For a data-driven method such as a neural network, there are too few samples and the modeling effect is relatively poor. Step 3: Since the spectrograms obtained by short-time Fourier transform have large differences on the vertical axis, in order to achieve unified modeling, the modeling granularity is changed from the entire spectrogram to the modeling of the spectrogram subbands. The spectrogram is divided into subbands, and the deep information of each subband is learned through the neural network. To ensure that the same model can process data of all sampling rates, the subband width is set to a fixed width. In order to achieve the ability to process signals of any sampling rate, during the training process, a subband with a fixed width is randomly selected from the entire spectrogram and input into the neural network training; Due to the difficulty and high cost of labeling input audio data, the mainstream architecture of existing audio models learned by neural networks is mainly unsupervised or self-supervised pre-training learning; Compared with the method of using the entire spectrogram as input for a single sampling rate, in order to cope with modeling at any sampling rate, in fact, the present invention can use any pre-training method. The biggest change is that a randomly selected subband in the entire spectrogram is randomly selected as input in the neural network training; Step 4: Self-supervised pre-training. The neural network training adopts the Encoder-Decoder structure. The Encoder output models most of the information of the original sub-band spectrogram. Therefore, in the inference stage, after the sound with any sampling rate sr is short-time Fourier transformed, the frequency interval of the spectrogram is ( 0 ， sr 2 ] , dividing the frequency range into Sub-band combination, less than one sub-band interception ( sr 2 − sr band ， sr 2 ] Round it into a subband, and take the intermediate hidden layer representation for each subband. The intermediate hidden layer representation is the feature representation output by the intermediate layer when the neural network training model processes data; Step 5: Vector concatenation of each sub-band can extract high-dimensional representation of sound at any sampling rate, and then judgment and verification are performed based on the high-dimensional representation. The high-dimensional representation is a data set formed by concatenating sub-band vectors, which is the feature information extracted from the sound sampling. The most typical application of this high-dimensional representation is voiceprint verification in speech. For example, in the process of voice voiceprint verification, a fixed-dimensional voiceprint vector is extracted from each segment of speech. The distance between the voiceprint vectors of different speakers is calculated in this vector dimension to determine whether they are the same speaker. After the normal sound collected by the sensor is extracted as a voiceprint vector, the running voiceprint vector is compared with the normal sound vector in real time, and the voiceprint distance is used to determine whether an abnormality occurs. The commonly used vector distance is the pre-similarity, and the calculation formula is as follows: ; After calculating the high-dimensional representation vectors for two segments of signals with arbitrary sampling rate, the cosine distance of the high-dimensional representation vectors can be used to determine whether the two audio segments are similar, thereby completing tasks such as speaker verification and industrial heterophonic detection.

[0006] Furthermore, the one-to-many signal is specifically a sampling end obtaining sampling signals of multiple sound sources, and the preprocessing of the one-to-many signal specifically includes the following steps: Step 101: frequency filtering, setting the filter and filtering range. The main frequency range of human voice is 80Hz-10kHz. The high-dimensional representation of the sampling rate sound is mainly used for voiceprint verification analysis. Therefore, the one-to-many signal is filtered according to the filtering range using a filter to obtain the first signal. Here, H (f) is the first signal, f is the frequency of the sampling signal, and the filter transfer function is: ; in, is the lower cutoff frequency, is the upper cutoff frequency; Step 102: Harmonic analysis: perform harmonic analysis on the first signal to obtain a second signal X [ k ] ; Step 103: Loudness analysis: the second signal X [ k ] Perform loudness analysis to separate the input audio into several parts.

[0007] Furthermore, the many-to-one signal is specifically a sampling signal of a sound source obtained by multiple sampling terminals, and the preprocessing of the many-to-one signal specifically includes the following steps: Step 104: Obtain the number of sampling ends. When the number of sampling ends is equal to 2, two sampling ends are not enough to obtain the specific location of the sound source by calculating the phase difference. Therefore, it is necessary to calculate the sound intensity to assist in determining the specific location of the sound source, and then jump to step 106. Otherwise, when the number of sampling ends is greater than 2, proceed to the next step 105. Step 105: Compare each sampling end in pairs and calculate the phase difference , the calculation formula of phase difference is: ; in, is the distance between sampling end A and sampling end B, λ is the wavelength, and the wavelength λ is obtained by dividing the sound speed by the frequency of the sampling signal at sampling end A or sampling end B. By comparing the phase difference between the sampling ends , obtain the location information of the sound source, convert the location information of the sound source into the relative position of each sampling signal, the relative position includes the distance and angle between the sound source and the sampling end, and bind the relative position with the sampling signal of the corresponding sampling end to obtain the input audio.

[0008] Furthermore, the many-to-many signal is specifically a sampling signal of multiple sound sources obtained by multiple sampling terminals, and the preprocessing of the many-to-many signal specifically includes the following steps: Step 110: performing phase subtraction between the sampling signal of the sampling end and the sampling signal of another sampling end that is farthest away, to obtain a third signal, wherein the third signal can highlight the sound source near the sampling end that is farthest away; Step 111: Repeat step 110 to obtain the third signal corresponding to each sampling end, and input the third signal into the beamforming calculation formula to obtain the fourth signal of each sampling end. , the beamforming calculation formula is as follows: ; Where M is the number of sampling terminals, is the weight coefficient of the mth sampling terminal, is the sampling signal received by the mth sampling terminal at time t. The sampling signal here is the original signal initially collected. The third signal obtained by phase subtraction of the mth sampling end is calculated, and the fourth signal obtained is That is, it is the preprocessed signal of the sound source near the farthest sampling end; Step 112: Calculate the fourth signal The phase difference between the sampling signal of each sampling end and the sampling signal of each sampling end is calculated by the formula consistent with step 105. The distance between the sampling end and the other sampling end farthest away is marked as the standard distance. The phase difference is multiplied by the standard distance to obtain the fourth signal The third component corresponding to the sound source; Step 113: Calculate the fourth signal of two adjacent sampling ends A ratio, and multiplying the ratio by the spacing between adjacent sampling ends to obtain a fourth component; Step 114: Add the third component and the fourth component to obtain the corresponding sound source position information, convert the sound source position information into a relative position with respect to each sampling end, wherein the relative position includes the distance and angle between the sound source and the sampling end, and bind the relative position with the sampling signal of the corresponding sampling end to obtain the input audio.

[0009] Furthermore, the calculation steps of the second component are as follows: The sampled signal is converted into a frequency response diagram, and the average loudness value of the frequency response diagram is calculated and marked as P. P can be used as the sound source power value obtained by the sampling end. Since the actual sound source power value is proportional to the loudness value, the average loudness value is used instead of the sound source power value to calculate the sound intensities I1 and I2 of the two sampling ends respectively. The calculation formula is: ; Among them, r1 and r2 are the first component distances between the two sampling ends and the sound source, respectively, and the sound intensities I1 and I2 are marked as the second components of the two sampling ends, respectively.

[0010] Furthermore, in self-supervised pre-training, random masking specifically includes the following steps: Step 401: Divide the subband into a number of rectangular blocks of equal area, where the side lengths of the rectangular blocks are g1 and g2 respectively, where g1 is the frequency dimension and g2 is the time dimension; Step 402: Set a masking matrix, which complies with the following conditions: Condition 1: In the same time dimension, at least two rectangular blocks are retained after masking and the rectangular images are not adjacent; Condition 2: The masking range is greater than 70%, that is, more than 70% of the rectangular blocks in the minimum masked subband; Step 403: record the masked position of each rectangular block in the sub-band, set the masking weight coefficient to be bound to the masking position, and increase the value of the masking weight coefficient corresponding to the masking position by 0.1% as the reconstruction effect is better after each training.

[0011] Further, performing harmonic analysis on the first signal specifically includes the following steps: Split the filtered first signal into discrete signals of length N x [ n ] , where n=0, 1, ..., N-1, the first signal is harmonically calculated based on discrete Fourier transform to obtain the second signal X [ k ] , the harmonic calculation formula is: X [ k ] = ∑ n = 0 N − 1 x [ n ] e − j 2 π N k ; Where k=0, 1, ..., N-1, X [ k ] It is a discrete signal x [ n ] The second signal obtained after harmonic calculation represents the discrete signal x [ n ] The complex value at different frequency points k, e is a natural constant, j is an imaginary unit, to avoid the harmonic calculation formula having no solution in the real number range. In physical terms, X [ k ] Amplitude | X [ k ] | The frequency of the signal is The relative strength of the harmonic components, is the sampling frequency, X [ k ] Phase ∠ X [ k ] is the phase information of the harmonic component of this frequency.

[0012] Furthermore, the loudness analysis specifically includes the following steps: The second signal is converted into a frequency response diagram, and the loudness values ​​of each vertex and each bottom point of the waveform in the frequency response diagram are recorded, and the loudness values ​​of each vertex and each bottom point are counted. The frequency corresponding to the bottom point loudness value that appears the most times is marked as the first reference frequency k1, and the frequency corresponding to the vertex loudness value that appears the most times is marked as the second reference frequency k2. X [ k ] Convert to power spectrum P [ k ] = | X [ k ] | 2 The first reference frequency k1 and the second reference frequency k2 define the frequency range and calculate the root mean square loudness L within the frequency range. RMS , the calculation formula is: L RMS = 1 K ∑ k = k 1 k 2 P [ k ] ; Among them, K is the number of frequency points in the frequency range, and K satisfies the condition K=k2-k1+1. According to the equal loudness curve of the human ear and the root mean square loudness L RMS The second signal is separated into a plurality of input audio frequencies.

[0013] Furthermore, when the number of sampling terminals is equal to 2, the following steps are included: Step 106: Calculate the phase difference between the two sampling ends The calculation formula is the same as step 105. The phase difference between the two sampling ends is Ratio times spacing Obtaining a first component corresponding to the sampling end; Step 107: Analyze the sound intensity of the sampled signal and convert the sound intensity into a second component; Step 108: Add the first component and the second component of the sampling end to obtain corresponding sound source position information; Step 109: Convert the location information of the sound source into the relative position of each sampling signal, where the relative position includes the distance and angle between the sound source and the sampling end, and bind the relative position with the sampling signal of the corresponding sampling end to obtain the input audio.

[0014] Furthermore, when the sub-bands are concatenated, it is necessary to verify that the intermediate hidden layer representations of each sub-band are in the same dimension. If the dimensions are inconsistent, the number of occurrences of each dimension in the sub-band is counted, and the dimension with the largest number of occurrences is marked as the base dimension of the sub-band. The sub-bands with inconsistent intermediate hidden layer representation dimensions are adjusted according to the base dimension to keep the dimensions consistent. The numpy library is used to represent the intermediate hidden layer representation, and the np.concatenate function is used to concatenate the sub-band vectors.

[0015] The present invention has the following beneficial effects: 1. The present invention proposes a sound analysis method for arbitrary sampling rate based on a pre-trained neural network, which mainly solves the problem of difficulty in unified modeling of various sound signals with large sampling rate span differences (such as 8KHz-192KHz). Different from the traditional method based on resampling, the present invention proposes to perform sub-band segmentation of fixed frequency bands on the spectrogram in the time-frequency domain and model the sub-bands. The spectrogram of arbitrary sampling rate can be decomposed into a combination of sub-band graphs, and high-dimensional representations are extracted from the sub-band graphs and then vector splicing is performed to obtain the final signal representation. Vector distance calculation is performed at the high-dimensional representation level to determine the similarity between sound fragments. This method is not only applicable to sound, but also to digital signals such as high-frequency vibrations. 2. In addition to solving the problem of unified modeling of sounds at any sampling rate, the present invention is also very useful for alleviating and ultimately overcoming the problem of data scarcity at a certain sampling rate. Since the model is based on the expression of the spectrogram of any frequency subband, for sampling rate signals with fewer samples, spectrogram data of different frequency bands at other sampling rates can be reused, thereby increasing the generalization and migration capabilities of the model. 3. By preprocessing the sampled signal, effective information can be extracted as input audio in the sampling environment of multiple sampling ends and multiple sound sources, which can reduce noise interference and is conducive to the training of neural networks; Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0017] Figure 1 This is a flow chart of a method for analyzing sound at any sampling rate based on a pre-trained neural network according to the present invention; Figure 2 Schematic diagram of the spectrogram and analysis window of the present invention; Figure 3 A schematic diagram of sub-band division of a spectrogram of the present invention; Figure 4 This is a flowchart of the existing unsupervised pre-training model; Figure 5 Schematic diagram of the process of self-supervised pre-training of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] See also Figure 1 The present invention provides a technical solution: a method for analyzing sound at any sampling rate based on a pre-trained neural network, comprising the following steps: Step 1: obtaining a multi-dimensional audio signal, the multi-dimensional audio signal includes a one-to-many signal, a many-to-one signal and a many-to-many signal, and pre-processing the multi-dimensional signal to obtain an input audio; Step 2: Convert the spectrogram. Since audio is a high-frequency one-dimensional time series in the time domain, with tens of thousands or even hundreds of thousands of sampling points per second, the conventional approach to audio analysis is to perform a short-time Fourier transform on the input audio to obtain a spectrogram. The horizontal axis of the spectrogram is time, and the vertical axis is the analysis frequency. Set a number of analysis windows. Usually, the short-time Fourier transform window length is 25ms, and the window shift is 10ms. Taking 16KHz sampling rate sound as an example, the length of each Fourier transform analysis window is 400 sampling points, and the interval between adjacent analysis windows is 160 points. For example, Figure 2 As shown; After the input audio is short-time Fourier transformed, the high-frequency time series signal is transformed from the time domain to the time-frequency domain to obtain a spectrogram. For sounds with different sampling rates, the time axis is the same, but the frequency axis is quite different. For example, for 16KHz sound, the frequency axis value range is (0, 8000] Hz, for 48KHz sound, the frequency axis value range is (0, 24000] Hz, and for 192KHz sound, the frequency axis value range is (0, 96000] Hz. The spectrogram is used as a representation of each input audio. When input into the neural network, the vertical axis height of each spectrogram is completely different. The height information represents the frequency range and has a clear physical meaning. For the neural network, the size of the input spectrogram varies greatly, and the shape of the input matrix changes at any time, so it is difficult to use a neural network model for modeling. In addition, due to the inconsistency of the number of signals with different sampling rates, as mentioned above, signals above 192KHz are very rare. For data-driven methods such as neural networks, there are too few samples and the modeling effect is relatively poor. Step 3: Since the spectrograms obtained by short-time Fourier transform have large differences on the vertical axis, in order to achieve unified modeling, the modeling granularity is changed from the entire spectrogram to the modeling of the spectrogram subbands. The spectrogram is divided into subbands, and the deep information of each subband is learned through the neural network. To ensure that the same model can process all sampling rate data, the subband width is set to a fixed width. For example, if the selected subband width is 1KHz, the 8KHz sampling rate signal is transformed to the spectrogram frequency range of (0, 4000] Hz, which can be divided into 4 non-overlapping subbands. In order to achieve the ability to process signals with arbitrary sampling rates, during the training process, a fixed-width subband is randomly selected from the entire spectrogram and input into the neural network training, such as Figure 3 As shown; Due to the difficulty and high cost of data annotation for input audio, the mainstream architecture of existing audio models for neural network learning is mainly based on unsupervised or self-supervised pre-training. Taking the AudioMAE unsupervised pre-training model as an example, after the spectrogram is input, the spectrogram is randomly masked, and the unmasked part is encoded as the encoder input. The masked part is replaced with MSK symbols and input into the decoder to reconstruct the original spectrogram block, as shown in Figure 4As shown in the figure, the modeling target of this model is the spectrogram MSE (MeanSquareError), and the calculation formula is as follows: ; in, is the element in the i-th row and j-th column of the original spectrogram, To reconstruct the element in the i-th row and j-th column of the spectrogram, the smaller the loss function, the better the reconstruction effect and the better the pre-training model; Compared with the method of using the entire spectrogram as input for a single sampling rate in AudioMAE, in order to cope with modeling of any sampling rate, the present invention uses the sub-band spectrogram as the AudioMAE input, and only allows the neural network to learn the sub-band information. In fact, the present invention can use any pre-training method. The biggest change is that the randomly selected sub-band in the entire spectrogram is randomly selected as the AudioMAE input during the neural network training; Step 4: Self-supervised pre-training. The neural network training adopts the Encoder-Decoder structure. The Encoder output models most of the information of the original sub-band spectrogram. Therefore, in the inference stage, after the sound with any sampling rate sr is short-time Fourier transformed, the frequency interval of the spectrogram is ( 0 ， sr 2 ] , dividing the frequency range into Sub-band combination, less than one sub-band interception ( sr 2 − sr band ， sr 2 ] Round it into a subband, and take the intermediate hidden layer representation from each subband based on random masking, such as Figure 5 As shown; Step 5: vector concatenate each sub-band. Steps 1 to 4 describe the detailed process of the present invention, which can be realized. Figure 1 A high-dimensional representation of sound with any sampling rate is extracted from the data, and judgment and verification are performed based on the high-dimensional representation. The most typical application of this high-dimensional representation is voiceprint verification in speech. For example, in the process of voiceprint verification, a fixed-dimensional voiceprint vector is extracted from each 10-second speech. The distance between the voiceprint vectors of different speakers is calculated in this vector dimension to determine whether they are the same speaker. In the industrial field, after the normal sound collected by the sensor is extracted as a voiceprint vector, the running voiceprint vector is compared with the normal sound vector in real time, and whether an abnormality occurs is determined by the voiceprint distance. The commonly used vector distance is the pre-similarity, and the calculation formula is as follows: ; In practical applications, after calculating the high-dimensional representation vectors for two segments of signals with arbitrary sampling rates, the cosine distance of the high-dimensional representation vectors can be used to determine whether the two audio segments are similar, thereby completing tasks such as speaker verification and industrial noise detection.

[0020] The one-to-many signal is specifically a sampling signal obtained by a sampling end from multiple sound sources, and the preprocessing of the one-to-many signal specifically includes the following steps: Step 101: frequency filtering, setting the filter and the filter range to 80Hz-10kHz. The main frequency range of human voice is 80Hz-10kHz. The high-dimensional representation of the sampling rate sound is mainly used for voiceprint verification analysis. Therefore, the one-to-many signal is filtered according to the filter range to obtain the first signal. The transfer function of the filter is: ; in, is the lower cutoff frequency, is the upper cutoff frequency; otherwise means except this.

[0021] Step 102: Harmonic analysis: perform harmonic analysis on the first signal to obtain a second signal X [ k ] ; Step 103: Loudness analysis: the second signal X [ k ] Perform loudness analysis to separate the input audio into several parts.

[0022] The many-to-one signal is specifically a sampling signal of a sound source obtained by multiple sampling terminals, and the preprocessing of the many-to-one signal specifically includes the following steps: Step 104: Obtain the number of sampling ends. When the number of sampling ends is equal to 2, two sampling ends are not enough to obtain the specific location of the sound source by calculating the phase difference. Therefore, it is necessary to calculate the sound intensity to assist in determining the specific location of the sound source, and then jump to step 106. Otherwise, when the number of sampling ends is greater than 2, proceed to the next step 105. Step 105: Compare each sampling end in pairs and calculate the phase difference , the calculation formula of phase difference is: ; in, is the distance between sampling end A and sampling end B, λ is the wavelength, and the wavelength λ is obtained by dividing the sound speed by the frequency of the sampling signal at sampling end A or sampling end B. By comparing the phase difference between the sampling ends , obtain the location information of the sound source, convert the location information of the sound source into the relative position of each sampling signal, the relative position includes the distance and angle between the sound source and the sampling end, and bind the relative position with the sampling signal of the corresponding sampling end to obtain the input audio.

[0023] The many-to-many signal is specifically a sampling signal of multiple sound sources obtained by multiple sampling terminals, and the preprocessing of the many-to-many signal specifically includes the following steps: Step 110: performing phase subtraction between the sampling signal of the sampling end and the sampling signal of another sampling end that is farthest away, to obtain a third signal, wherein the third signal can highlight the sound source near the sampling end that is farthest away; Step 111: Repeat step 110 to obtain the third signal corresponding to each sampling end, and input the third signal into the beamforming calculation formula to obtain the fourth signal of each sampling end. , the beamforming calculation formula is as follows: ; Where M is the number of sampling terminals, is the weight coefficient of the mth sampling terminal, The general value is 1, which means that the sampling signal received by each sampling end is an equivalent signal. is the sampling signal received by the mth sampling terminal at time t. The sampling signal here is the original signal initially collected. The third signal obtained by phase subtraction of the mth sampling end is calculated, and the fourth signal obtained is That is, it is the preprocessed signal of the sound source near the farthest sampling end; Step 112: Calculate the fourth signal The phase difference between the sampling signal of each sampling end and the sampling signal of each sampling end is calculated by the formula consistent with step 105. The distance between the sampling end and the other sampling end farthest away is marked as the standard distance. The phase difference is multiplied by the standard distance to obtain the fourth signal The third component corresponding to the sound source; Step 113: Calculate the fourth signal of two adjacent sampling ends A ratio, and multiplying the ratio by the spacing between adjacent sampling ends to obtain a fourth component; Step 114: Add the third component and the fourth component to obtain the corresponding sound source position information, convert the sound source position information into a relative position with respect to each sampling end, wherein the relative position includes the distance and angle between the sound source and the sampling end, and bind the relative position with the sampling signal of the corresponding sampling end to obtain the input audio.

[0024] The calculation steps of the second component are as follows: The sampled signal is converted into a frequency response diagram, and the average loudness value of the frequency response diagram is calculated and marked as P. P can be used as the sound source power value obtained by the sampling end. Since the actual sound source power value is proportional to the loudness value, the average loudness value is used instead of the sound source power value to calculate the sound intensities I1 and I2 of the two sampling ends respectively. The calculation formula is: ; Among them, r1 and r2 are the first component distances between the two sampling ends and the sound source, respectively, and the sound intensities I1 and I2 are marked as the second components of the two sampling ends, respectively.

[0025] Among them, in self-supervised pre-training, random masking specifically includes the following steps: Step 401: Divide the subband into a number of rectangular blocks of equal area, where the side lengths of the rectangular blocks are g1 and g2 respectively, where g1 is the frequency dimension and g2 is the time dimension; Step 402: Set a masking matrix, which complies with the following conditions: Condition 1: In the same time dimension, at least two rectangular blocks are retained after masking and the rectangular images are not adjacent; Condition 2: The masking range is greater than 70%, that is, more than 70% of the rectangular blocks in the minimum masked subband; Step 403: record the masked position of each rectangular block in the sub-band, set the masking weight coefficient to be bound to the masking position, and increase the value of the masking weight coefficient corresponding to the masking position by 0.1% as the reconstruction effect is better after each training.

[0026] The harmonic analysis of the first signal specifically includes the following steps: Split the filtered first signal into discrete signals of length N x [ n ] , where n=0, 1, ..., N-1, the first signal is harmonically calculated based on discrete Fourier transform to obtain the second signal X [ k ] , the harmonic calculation formula is: X [ k ] = ∑ n = 0 N − 1 x [ n ] e − j 2 π N k ; Where k=0, 1, ..., N-1, X [ k ] It is a discrete signal x [ n ] The second signal obtained after harmonic calculation represents the discrete signal x [ n ] The complex value at different frequency points k, e is a natural constant, e is approximately equal to 2.7183, j is an imaginary unit, to avoid the harmonic calculation formula having no solution in the real number range. In physical terms, X [ k ] Amplitude | X [ k ] | The frequency of the signal is The relative strength of the harmonic components, is the sampling frequency, X [ k ] Phase ∠ X [ k ] is the phase information of the harmonic component of this frequency.

[0027] The loudness analysis specifically includes the following steps: The second signal is converted into a frequency response diagram, and the loudness values ​​of each vertex and each bottom point of the waveform in the frequency response diagram are recorded, and the loudness values ​​of each vertex and each bottom point are counted. The frequency corresponding to the bottom point loudness value that appears the most times is marked as the first reference frequency k1, and the frequency corresponding to the vertex loudness value that appears the most times is marked as the second reference frequency k2. X [ k ] Convert to power spectrum P [ k ] = | X [ k ] | 2 The first reference frequency k1 and the second reference frequency k2 define the frequency range and calculate the root mean square loudness L within the frequency range. RMS , the calculation formula is: L RMS = 1 K ∑ k = k 1 k 2 P [ k ] ; Among them, K is the number of frequency points in the frequency range, and K satisfies the condition K=k2-k1+1. According to the equal loudness curve of the human ear and the root mean square loudness L RMS The second signal is separated into a plurality of input audio frequencies.

[0028] When the number of sampling terminals is equal to 2, the following steps are included: Step 106: Calculate the phase difference between the two sampling ends The calculation formula is the same as step 105. The phase difference between the two sampling ends is Ratio times spacing Obtaining a first component corresponding to the sampling end; Step 107: Analyze the sound intensity of the sampled signal and convert the sound intensity into a second component; Step 108: Add the first component and the second component of the sampling end to obtain corresponding sound source position information; Step 109: Convert the location information of the sound source into the relative position of each sampling signal, where the relative position includes the distance and angle between the sound source and the sampling end, and bind the relative position with the sampling signal of the corresponding sampling end to obtain the input audio.

[0029] When the sub-bands are concatenated, it is necessary to verify that the intermediate hidden layer representations of each sub-band are in the same dimension. If the dimensions are inconsistent, the number of occurrences of each dimension in the sub-band is counted, and the dimension with the largest number of occurrences is marked as the base dimension of the sub-band. The sub-bands with inconsistent intermediate hidden layer representation dimensions are adjusted according to the base dimension to keep the dimensions consistent. The hidden layer representation is output through the middle layer of the neural network, and the np.concatenate function is used to concatenate the vectors of the sub-bands. The numpy library and the np.concatenate function are both located in the Python development environment.

[0030] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for analyzing sound at any sampling rate based on a pre-trained neural network, characterized in that: The method comprises the following steps: Step 1: obtaining input audio; Step 2: converting the spectrogram, performing short-time Fourier transform on the input audio to obtain a spectrogram, the horizontal axis of the spectrogram is time, the vertical axis is analysis frequency, setting a number of analysis windows, defining the window length and window shift, the short-time Fourier transform window length is 25ms, and the window shift is 10ms; Step 3: Change the modeling granularity from the entire spectrogram to modeling the spectrogram subbands, divide the spectrogram into subbands, and learn the deep information of each subband through the neural network. To ensure that the same model can process all sampling rate data, the subband width is set to a fixed width. During the training process, a fixed-width subband is randomly selected from the spectrogram and input into the neural network training; Step 4: Self-supervised pre-training. The neural network training adopts the Encoder-Decoder structure. The Encoder output models most of the information of the original sub-band spectrogram. In the inference stage, after the sound with any sampling rate sr is short-time Fourier transformed, the frequency interval of the spectrogram is , dividing the frequency range into Sub-band combination, less than one sub-band interception Round it into a subband and take the intermediate hidden layer representation from each subband based on random masking; Step 5: Vectorize each sub-band, extract the high-dimensional representation of the sound at any sampling rate, and then make judgments and verifications based on the high-dimensional representation.

2. The method for analyzing sound at any sampling rate based on a pre-trained neural network according to claim 1, characterized in that: In step 2, after the input audio is short-time Fourier transformed, the high-frequency time series signal is transformed from the time domain to the time-frequency domain to obtain a spectrogram.

3. The method for analyzing sound at any sampling rate based on a pre-trained neural network according to claim 1, characterized in that: In step 3, the mainstream architecture of the audio model learned by the neural network is mainly unsupervised or self-supervised pre-training learning, specifically: AudioMAE unsupervised pre-training model, after the spectrogram is input, the spectrogram is randomly masked, the unmasked part is encoded as the encoder input, the masked part is replaced with MSK symbols and input into the decoder to reconstruct the original spectrogram block, the modeling target is the spectrogram MSE, and the calculation formula is as follows: ； in, is the element in the i-th row and j-th column of the original spectrogram, To reconstruct the element in the i-th row and j-th column of the spectrogram.

4. The method for analyzing sound at any sampling rate based on a pre-trained neural network according to claim 1, characterized in that: In step 5, the application of high-dimensional representation is voiceprint verification in speech. During the voiceprint verification process, a fixed-dimensional voiceprint vector is extracted from each speech segment, and the distance between the voiceprint vectors of different speakers is calculated in this vector dimension, that is, whether they are the same speaker is determined. In industrial applications, after the normal sound collected by the sensor is extracted as a voiceprint vector, the running voiceprint vector is compared with the normal sound vector in real time, and whether an abnormality occurs is determined by the voiceprint distance. The commonly used vector distance is the pre-similarity, and the calculation formula is as follows: ； After calculating the high-dimensional representation vectors for two segments of signals with arbitrary sampling rate, the cosine distance of the high-dimensional representation vectors is calculated to determine whether the two audio segments are similar, thereby completing the speaker verification and industrial heterophonic detection tasks.

Images