A method for identifying water supply pipe leakage audio based on deep learning

Abstract

The present invention discloses a method, system, and computer-readable storage medium for identifying water supply pipeline leak audio based on deep learning. The method comprises the following steps: S1, collecting audio data from the water supply pipeline; S2, preprocessing the audio data to obtain a mel-spectrogram of the audio data; S3, training an improved MobileNet TV3 model using the mel-spectrogram data to obtain a trained MobileNet TV3 model; S4, collecting and preprocessing the audio data from the water supply pipeline in real time, inputting the preprocessed real-time audio data into the trained MobileNet TV3 model, and outputting a recognition result. The present invention improves the efficiency and accuracy of pipeline leak detection.

Description

Technical Field

[0001] The present invention relates to the technical field of pipeline leak detection, and more specifically, to a method and system for recognizing water supply pipeline leakage audio based on deep learning, and a computer-readable storage medium. Background Art

[0002] With the development of cities, the laying of urban water supply pipelines is becoming more and more concentrated. Some water supply pipelines are inevitably prone to dripping or leaking because they were laid at an early time or due to other external forces.

[0003] Leaks in underground pipes produce distinct sounds due to water pressure. Sound detection is a common technique for proactively detecting and locating leaks in water supply pipes. This simple principle, coupled with a long history of engineering application, has led to widespread adoption within the water supply industry both domestically and internationally. Currently, leak inspections in most urban pipe networks in my country still rely on manual sound detection. Even with the use of relevant instruments and other equipment, manual sound detection is still required to narrow the location range before excavation. During leak detection investigations, workers use mechanical sounding rods or portable measuring equipment to detect leaks using sound.

[0004] Prior art discloses a neural network-based intelligent identification method for leaking water from pressurized water supply pipes. The method includes: collecting the sound of a target pressurized water supply pipe and ambient audio to generate an audio file; preprocessing the resulting audio file, including bandpass filtering and Berouti spectrum subtraction for noise reduction; extracting the Mel-scale spectrum from the filtered signal to generate corresponding digital feature data for the pipe sound; inputting this digital feature data into a preset hybrid classification prediction model to output a classification result. This solution uses a hybrid classification model for classification and identification, resulting in limited detection accuracy. Summary of the Invention

[0005] The present invention provides a method for recognizing water supply pipe leakage audio based on deep learning, which improves the efficiency and accuracy of pipeline leakage detection.

[0006] The primary purpose of the present invention is to solve the above technical problems, and the technical solutions of the present invention are as follows:

[0007] A first aspect of the present invention provides a method for recognizing water supply pipe leakage audio based on deep learning, comprising the following steps:

[0008] S1, collecting audio data in the water supply pipeline;

[0009] S2. Preprocess the audio data to obtain a Mel-spectrogram of the audio data;

[0010] S3. Use the Mel-language spectrogram data to train the improved MobileNetV3 model to obtain the trained MobileNetV3 model;

[0011] S4. Collect and preprocess the audio data of the water supply pipeline in real time, input the preprocessed real-time audio data into the trained mobilenetv3 model, and output the recognition results.

[0012] Furthermore, in step S1, the audio data in the water supply pipe is collected by an underwater robot equipped with an audio collection module.

[0013] Furthermore, the audio data is preprocessed to obtain a Mel-spectrogram of the audio data, and the specific process is as follows:

[0014] S201, dividing the acquired audio data in the water supply pipe into frames according to a preset time length to obtain data with the same time interval;

[0015] S202, performing data enhancement on the framed data by time shifting;

[0016] S203: Extract the Mel-spectrogram of the enhanced audio data.

[0017] Furthermore, the preset time length is 1 second.

[0018] Furthermore, the specific process of step S203 is: the enhanced data is sequentially framed, windowed, Fourier transformed, the power spectrum is obtained, a mel filter bank is constructed, and a dot product operation is performed with the power spectrum to obtain a mel spectrogram of the enhanced audio data.

[0019] Furthermore, the improved mobilenetv3 model replaces the SEnet attention mechanism unit of the Bneck structure in the existing mobilenetv3 model with a CBAM attention mechanism unit.

[0020] Furthermore, the improved mobilenetv3 model includes 11 Bneck structures, among which the 1st, 2nd and 3rd Bneck structures use 3x3 convolution, and the remaining Bneck structures use 5x5 convolution; the 1st, 4th, 5th, 6th, 7th, 8th and 9th Bneck use CBAM structure, and the rest do not use it; the 1st, 2nd and 3rd Bneck structures use RELU activation function, and the remaining Bneck structures use H_Swish activation function; the step size of the 1st, 2nd, 4th and 9th Bneck structures is 2, and the step size of the remaining Bneck structures is 1.

[0021] In one aspect, the present invention provides a system for identifying water supply pipe leakage audio based on deep learning, characterized in that the system includes: a memory and a processor, wherein the memory includes a method program for identifying water supply pipe leakage audio based on deep learning, and when the method program for identifying water supply pipe leakage audio based on deep learning is executed by the processor, the following steps are implemented:

[0022] S1, collecting audio data in the water supply pipeline;

[0023] S2. Preprocess the audio data to obtain a Mel-spectrogram of the audio data;

[0024] S3. Use the Mel-language spectrogram data to train the improved MobileNetV3 model to obtain the trained MobileNetV3 model;

[0025] S4. Collect and preprocess the audio data of the water supply pipeline in real time, input the preprocessed real-time audio data into the trained mobilenetv3 model, and output the recognition results.

[0026] Furthermore, in step S1, the audio data in the water supply pipe is collected by an underwater robot equipped with an audio collection module.

[0027] The third aspect of the present invention provides a computer-readable storage medium, which includes a method program for identifying water supply pipe leakage audio based on deep learning. When the method program for identifying water supply pipe leakage audio based on deep learning is executed by a processor, the steps of the method for identifying water supply pipe leakage audio based on deep learning are implemented.

[0028] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0029] The present invention improves the mobilenetv3 model, replaces the SE attention mechanism unit in the Bneck module, processes the audio data in the water supply pipe into a Mel-level spectrogram, and inputs it into the improved model for water leakage detection. The present invention can reduce the number of times the attention mechanism module is used while making the model's attention mechanism more focused on audio features, thereby improving the efficiency and detection accuracy of pipeline leakage detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of a method for recognizing audio of water supply pipe leakage based on deep learning in the present invention.

[0031] Figure 2 This is a waveform diagram of audio data in a water supply pipe according to an embodiment of the present invention.

[0032] Figure 3 Schematic diagram of the CBAM attention mechanism unit structure in an embodiment of the present invention.

[0033] Figure 4 This is a schematic diagram of the improved Mobilenetv3 model structure according to an embodiment of the present invention.

[0034] Figure 5 The present invention is an embodiment of the present invention, which is an audio waveform diagram of water leakage in a water supply pipe.

[0035] Figure 6 This is a Mel-gram spectrogram of the audio of a water supply pipe leak according to an embodiment of the present invention.

[0036] Figure 7 This is an audio waveform diagram showing that the water supply pipe is leak-proof according to an embodiment of the present invention.

[0037] Figure 8 This is a Mel-gram spectrogram of the audio of the water supply pipe not leaking according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0039] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0040] Example 1

[0041] like Figure 1 As shown, the first aspect of the present invention provides a method for recognizing water supply pipe leakage audio based on deep learning, comprising the following steps:

[0042] S1, collecting audio data in the water supply pipeline;

[0043] It should be noted that, in the present invention, the audio data in the water supply pipe can be obtained through the audio acquisition module. The audio acquisition module can be set on an underwater detector or an underwater robot. The present invention does not limit the specific underwater device set up for the audio acquisition module. It should be noted that the audio acquisition module also includes a communication unit. The collected audio data can be transmitted through the communication unit, for example, transmitted to the on-site host computer or the remote background, and then the host computer or the remote background processes and analyzes the data. In a specific embodiment, the audio acquisition module performs audio acquisition at an acquisition rate of 44100, that is, 44100 audio values ​​are collected in one second. The above-mentioned collected data is discrete. These values ​​are smoothed by zero-order hold to finally form a common sound waveform, as shown in Figure 2.

[0044] S2. Preprocess the audio data to obtain a Mel-spectrogram of the audio data;

[0045] It should be noted that the audio data preprocessing process is specifically as follows:

[0046] S201. Divide the acquired audio data in the water supply pipe into frames according to a preset time length to obtain data with equal time intervals; wherein the preset time length can be set to 1 second.

[0047] S202, performing data enhancement on the framed data by time shifting;

[0048] It should be noted that in the present invention, when time-shifting the framed data, random shifting can be performed within a certain ratio range instead of using a fixed time shift, and cyclic shifting is performed instead of gap zero filling.

[0049] S203: Extract the Mel-spectrogram of the enhanced audio data.

[0050] More specifically, the enhanced data is sequentially framed, windowed, Fourier transformed, the power spectrum is obtained, a mel filter bank is constructed, and a dot product operation is performed with the power spectrum to obtain the mel spectrogram of the enhanced audio data.

[0051] In a specific embodiment, framing is performed to divide the signal into short time frames, typically with a framing interval of 20-40 ms, and preferably a frame length of 25 ms. Windowing involves multiplying each frame by a window function to reduce spectral leakage. The Fourier transform converts the time domain signal into an energy distribution in the frequency domain. It should be noted that the Fourier transform in the present invention is a short-time Fourier transform. The spectrum obtained by the Fourier transform is then modulo-squared to obtain a power spectrum, and the power spectrum is then dot-producted with the mel filter bank to obtain a spectrogram.

[0052] S3. Use the Mel-language spectrogram data to train the improved MobileNetV3 model to obtain the trained MobileNetV3 model;

[0053] It should be noted that the improved mobilenetv3 model replaces the SEnet attention mechanism unit of the Bneck structure in the existing mobilenetv3 model with CBAM.

[0054] It should be noted that the Bneck structure of the Mobilenetv3 network model has the SEnet attention mechanism, but the performance of the SEnet attention mechanism is not as good as the CBAM attention mechanism, so I used the CBAM attention mechanism instead of its original SEnet attention mechanism. Not every Bneck structure in the Mobilenet network structure uses the SEnet attention mechanism because different recognition categories occupy different pixels in the original image. Unlike the original MobilenetV3, in order to make the attention mechanism better focus on the audio features, I did not use the CBAM attention mechanism in the 10th and 11th Bnecks. Because when the feature map is upsampled to 4x4, the audio features can no longer be found. Therefore, not using the CBAM attention mechanism can reduce the number of parameters without affecting the detection accuracy.

[0055] Furthermore, the improved mobilenetv3 model includes 11 Bneck structures, among which the 1st, 2nd and 3rd Bneck structures use 3x3 convolution, and the remaining Bneck structures use 5x5 convolution; the 1st, 4th, 5th, 6th, 7th, 8th and 9th Bneck structures use CBAM structure, and the rest do not use it; the 1st, 2nd and 3rd Bneck structures use RELU activation function, and the remaining Bneck structures use H_Swish activation function; the 1st, 2nd, 4th and 9th Bneck structures have a step size of 2, and the remaining Bneck structures have a step size of 1. Figure 3 Shown is a schematic diagram of the CBAM attention mechanism unit structure.

[0056] In a specific embodiment, Figure 4The figure shows the structure of the improved mobilenetv3 model. The Mel-level spectrogram data first passes through a 3x3 convolution with a stride of 2, and then passes through 11 Bneck structures: the improved mobilenetv3 model includes 11 Bneck structures, among which the first, second, and third Bneck structures use 3x3 convolutions, and the remaining Bneck structures use 5x5 convolutions; the first, fourth, fifth, sixth, seventh, eighth, and ninth Bneck structures use the CBAM structure, and the rest do not use it; the first, second, and third Bneck structures use the RELU activation function, and the remaining Bneck structures use the H_Swish activation function; the first, second, fourth, and ninth Bneck structures have a stride of 2, and the remaining Bneck structures have a stride of 1. After passing through the above 11 Bneck structures, the feature map is obtained, and then a 1x1 convolution is used to change the number of channels, followed by a maximum pooling, and then a 1x1 convolution to change the number of channels, and finally a 1x1 convolution is used to obtain the classification.

[0057] It's important to note that the audio data from the water supply pipe includes both leaking and non-leaking audio. During training, the dataset needs to be segmented into leaking and non-leaking audio. This doesn't require labeling, as in object detection. Simply place the leaking and non-leaking audio files in separate folders and write a program to read and segment them for training.

[0058] S4. Collect and preprocess the audio data of the water supply pipeline in real time, input the preprocessed real-time audio data into the trained mobilenetv3 model, and output the recognition results.

[0059] As shown in the figure, Figure 5 This is the audio waveform of water supply pipe leakage. Figure 6 Mel spectrogram of the audio of a water pipe leak, such as Figure 7 The audio waveform diagram shows that the water supply pipe is not leaking. Figure 8 Mel-spectrogram of the audio file "Water pipes are leaking."

[0060] A second aspect of the present invention provides a system for identifying water supply pipe leakage audio based on deep learning, characterized in that the system includes: a memory and a processor, wherein the memory includes a method program for identifying water supply pipe leakage audio based on deep learning, and when the method program for identifying water supply pipe leakage audio based on deep learning is executed by the processor, the following steps are implemented:

[0061] S1, collecting audio data in the water supply pipeline;

[0062] It should be noted that, in the present invention, the audio data in the water supply pipe can be obtained through the audio acquisition module. The audio acquisition module can be set on an underwater detector or an underwater robot. The present invention does not limit the specific underwater device set up for the audio acquisition module. It should be noted that the audio acquisition module also includes a communication unit. The collected audio data can be transmitted through the communication unit, for example, transmitted to the on-site host computer or the remote background, and then the host computer or the remote background processes and analyzes the data. In a specific embodiment, the audio acquisition module performs audio acquisition at an acquisition rate of 44100, that is, 44100 audio values ​​are collected in one second. The above-mentioned collected data is discrete. These values ​​are smoothed by zero-order hold to finally form a common sound waveform, as shown in Figure 2.

[0063] S2. Preprocess the audio data to obtain a Mel-spectrogram of the audio data;

[0064] It should be noted that the audio data preprocessing process is specifically as follows:

[0065] S201. Divide the acquired audio data in the water supply pipe into frames according to a preset time length to obtain data with equal time intervals; wherein the preset time length can be set to 1 second.

[0066] S202, performing data enhancement on the framed data by time shifting;

[0067] It should be noted that in the present invention, when time-shifting the framed data, random shifting can be performed within a certain ratio range instead of using a fixed time shift, and cyclic shifting is performed instead of gap zero filling.

[0068] S203: Extract the Mel-spectrogram of the enhanced audio data.

[0069] More specifically, the enhanced data is sequentially framed, windowed, Fourier transformed, the power spectrum is obtained, a mel filter bank is constructed, and a dot product operation is performed with the power spectrum to obtain the mel spectrogram of the enhanced audio data.

[0070] In a specific embodiment, framing is performed to divide the signal into short time frames, typically with a framing interval of 20-40 ms, and preferably a frame length of 25 ms. Windowing involves multiplying each frame by a window function to reduce spectral leakage. The Fourier transform converts the time domain signal into an energy distribution in the frequency domain. It should be noted that the Fourier transform in the present invention is a short-time Fourier transform. The spectrum obtained by the Fourier transform is then modulo-squared to obtain a power spectrum, and the power spectrum is then dot-producted with the mel filter bank to obtain a spectrogram.

[0071] S3. Use the Mel-language spectrogram data to train the improved MobileNetV3 model to obtain the trained MobileNetV3 model;

[0072] It should be noted that the improved mobilenetv3 model replaces the SEnet attention mechanism unit of the Bneck structure in the existing mobilenetv3 model with CBAM.

[0073] It should be noted that the Bneck structure of the Mobilenetv3 network model has the SEnet attention mechanism, but the performance of the SEnet attention mechanism is not as good as the CBAM attention mechanism, so I used the CBAM attention mechanism instead of its original SEnet attention mechanism. Not every Bneck structure in the Mobilenet network structure uses the SEnet attention mechanism because different recognition categories occupy different pixels in the original image. Unlike the original MobilenetV3, in order to make the attention mechanism better focus on the audio features, I did not use the CBAM attention mechanism in the 10th and 11th Bnecks. Because when the feature map is upsampled to 4x4, the audio features can no longer be found. Therefore, not using the CBAM attention mechanism can reduce the number of parameters without affecting the detection accuracy.

[0074] Furthermore, the improved mobilenetv3 model includes 11 Bneck structures, among which the 1st, 2nd and 3rd Bneck structures use 3x3 convolution, and the remaining Bneck structures use 5x5 convolution; the 1st, 4th, 5th, 6th, 7th, 8th and 9th Bneck structures use CBAM structure, and the rest do not use it; the 1st, 2nd and 3rd Bneck structures use RELU activation function, and the remaining Bneck structures use H_Swish activation function; the 1st, 2nd, 4th and 9th Bneck structures have a step size of 2, and the remaining Bneck structures have a step size of 1. Figure 3 Shown is a schematic diagram of the CBAM attention mechanism unit structure.

[0075] In a specific embodiment, Figure 4The figure shows the structure of the improved mobilenetv3 model. The Mel-level spectrogram data first passes through a 3x3 convolution with a stride of 2, and then passes through 11 Bneck structures: the improved mobilenetv3 model includes 11 Bneck structures, among which the first, second, and third Bneck structures use 3x3 convolutions, and the remaining Bneck structures use 5x5 convolutions; the first, fourth, fifth, sixth, seventh, eighth, and ninth Bneck structures use the CBAM structure, and the rest do not use it; the first, second, and third Bneck structures use the RELU activation function, and the remaining Bneck structures use the H_Swish activation function; the first, second, fourth, and ninth Bneck structures have a stride of 2, and the remaining Bneck structures have a stride of 1. After passing through the above 11 Bneck structures, the feature map is obtained, and then a 1x1 convolution is used to change the number of channels, followed by a maximum pooling, and then a 1x1 convolution to change the number of channels, and finally a 1x1 convolution is used to obtain the classification.

[0076] It's important to note that the audio data from the water supply pipe includes both leaking and non-leaking audio. During training, the dataset needs to be segmented into leaking and non-leaking audio. This doesn't require labeling, as in object detection. Simply place the leaking and non-leaking audio files in separate folders and write a program to read and segment them for training.

[0077] S4. Collect and preprocess the audio data of the water supply pipeline in real time, input the preprocessed real-time audio data into the trained mobilenetv3 model, and output the recognition results.

[0078] As shown in the figure, Figure 5 This is the audio waveform of water supply pipe leakage. Figure 6 Mel spectrogram of the audio of a water pipe leak, such as Figure 7 The audio waveform diagram shows that the water supply pipe is not leaking. Figure 8 Mel-spectrogram of the audio file "Water pipes are leaking."

[0079] The third aspect of the present invention provides a computer-readable storage medium, which includes a method program for identifying water supply pipe leakage audio based on deep learning. When the method program for identifying water supply pipe leakage audio based on deep learning is executed by a processor, the steps of the method for identifying water supply pipe leakage audio based on deep learning are implemented.

[0080] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A method for identifying water supply pipe leakage audio based on deep learning, characterized in that: The following steps are involved: S1, collecting audio data in the water supply pipeline; S2. Preprocess the audio data to obtain a Mel-spectrogram of the audio data; S3. Use the Mel-language spectrogram data to train the improved MobileNetV3 model to obtain the trained MobileNetV3 model; S4: Collect and pre-process the audio data from the water supply pipe in real time, input the pre-processed real-time audio data into the trained mobilenetv3 model, and output the recognition result; The improved mobilenetv3 model includes 11 Bneck structures, among which the 1st, 2nd and 3rd Bneck structures use 3x3 convolution, and the remaining Bneck structures use 5x5 convolution; the 1st, 4th, 5th, 6th, 7th, 8th and 9th Bneck use CBAM structure, and the rest do not use it; the 1st, 2nd and 3rd Bneck structures use RELU activation function, and the remaining Bneck structures use H_Swish activation function; the 1st, 2nd, 4th and 9th Bneck structures have a step size of 2, and the remaining Bneck structures have a step size of 1.

2. The method for recognizing water supply pipe leakage audio based on deep learning according to claim 1 is characterized in that: In step S1, audio data in the water supply pipe is collected by an underwater robot equipped with an audio collection module.

3. The method for recognizing water supply pipe leakage audio based on deep learning according to claim 1 is characterized in that: The audio data is preprocessed to obtain a Mel-gram spectrogram of the audio data. The specific process is as follows: S201, dividing the acquired audio data in the water supply pipe into frames according to a preset time length to obtain data with the same time interval; S202, performing data enhancement on the framed data by time shifting; S203: Extract the Mel-spectrogram of the enhanced audio data.

4. The method for recognizing water supply pipe leakage audio based on deep learning according to claim 3 is characterized in that: The preset time length is 1 second.

5. The method for recognizing water supply pipe leakage audio based on deep learning according to claim 3 is characterized in that: The specific process of step S203 is: the enhanced data is framed, windowed, Fourier transformed, the power spectrum is obtained, the mel filter bank is constructed, and the dot product operation is performed with the power spectrum to obtain the mel spectrogram of the enhanced audio data.

6. The method for recognizing water supply pipe leakage audio based on deep learning according to claim 1, characterized in that: The improved mobilenetv3 model replaces the SEnet attention mechanism unit of the Bneck structure in the existing mobilenetv3 model with the CBAM attention mechanism unit.

7. A water supply pipe leakage audio recognition system based on deep learning, characterized in that: The system includes: a memory and a processor. The memory includes a method program for identifying water supply pipe leakage audio based on deep learning. When the method program for identifying water supply pipe leakage audio based on deep learning is executed by the processor, the following steps are implemented: S1, collecting audio data in the water supply pipeline; S2. Preprocess the audio data to obtain a Mel-spectrogram of the audio data; S3. Use the Mel-language spectrogram data to train the improved MobileNetV3 model to obtain the trained MobileNetV3 model; S4: Collect and pre-process the audio data from the water supply pipe in real time, input the pre-processed real-time audio data into the trained mobilenetv3 model, and output the recognition result; The improved mobilenetv3 model includes 11 Bneck structures, among which the 1st, 2nd and 3rd Bneck structures use 3x3 convolution, and the remaining Bneck structures use 5x5 convolution; the 1st, 4th, 5th, 6th, 7th, 8th and 9th Bneck use CBAM structure, and the rest do not use it; the 1st, 2nd and 3rd Bneck structures use RELU activation function, and the remaining Bneck structures use H_Swish activation function; the 1st, 2nd, 4th and 9th Bneck structures have a step size of 2, and the remaining Bneck structures have a step size of 1.

8. The water supply pipe leakage audio recognition system based on deep learning according to claim 7 is characterized in that: In step S1, audio data in the water supply pipe is collected by an underwater robot equipped with an audio collection module.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a method program for identifying water supply pipe leakage audio based on deep learning. When the method program for identifying water supply pipe leakage audio based on deep learning is executed by a processor, the steps of a method for identifying water supply pipe leakage audio based on deep learning as described in any one of claims 1 to 6 are implemented.

Images