Abnormal audio detection method, device and equipment

Abstract

The invention provides an abnormal audio detection method, device and equipment, and the method comprises the steps: segmenting a to-be-detected audio file, and generating at least one to-be-detected audio clip; for any to-be-detected audio clip, generating a sample spectrogram corresponding to the to-be-detected audio clip; and according to the similarity between the sample spectrogram and the reference spectrogram, carrying out differential display on an abnormal region and a non-abnormal region in the sample spectrogram displayed in a visual interface. According to the method, the abnormal area in the to-be-detected audio file can be more intuitive, and technical support is provided for improving the accuracy and efficiency of abnormal annotation of the to-be-detected audio file.

Description

Technical Field

[0001] This application relates to the field of audio data processing technology, and in particular to an abnormal audio detection method, apparatus and device. Background Technology

[0002] With the rapid development of the Internet of Things and intelligent monitoring, audio signals, as an important information carrier, play a crucial role in many fields such as equipment status monitoring and environmental listening. By analyzing and processing audio signals in real time, abnormal events can be effectively identified, such as abnormal noises from industrial equipment or broken glass in security scenarios, thereby enabling fault warnings, security precautions, or timely intervention. Summary of the Invention

[0003] In view of this, this application provides an abnormal audio detection method, apparatus, device and system.

[0004] Specifically, this application is implemented through the following technical solution: According to a first aspect of the embodiments of this application, an abnormal audio detection method is provided, comprising: The audio file to be detected is segmented to generate at least one audio segment to be detected. For any of the audio segments to be detected, a sample spectrogram corresponding to the audio segment to be detected is generated; Based on the similarity between the sample spectrogram and the benchmark spectrogram, the abnormal and non-abnormal regions in the sample spectrogram displayed in the visualization interface are differentiated; wherein, the similarity between the pixels in the abnormal region of the sample spectrogram and the corresponding pixels in the benchmark spectrogram is less than the similarity between the pixels in the non-abnormal region of the sample spectrogram and the corresponding pixels in the benchmark spectrogram, and the benchmark spectrogram is the spectrogram corresponding to the benchmark audio segment.

[0005] According to a second aspect of the embodiments of this application, an abnormal audio detection device is provided, comprising: The segmentation unit is used to segment the audio file to be detected, generating at least one audio segment to be detected. The visualization unit is used to generate a sample spectrum diagram corresponding to any of the audio segments to be detected; The processing unit is used to differentiate between abnormal and non-abnormal regions in the sample spectrum diagram displayed in the visualization interface based on the similarity between the sample spectrum diagram and the reference spectrum diagram; wherein, the similarity between pixels in the abnormal region of the sample spectrum diagram and the corresponding pixels in the reference spectrum diagram is less than the similarity between pixels in the non-abnormal region of the sample spectrum diagram and the corresponding pixels in the reference spectrum diagram, and the reference spectrum diagram is the spectrum diagram corresponding to the reference audio segment.

[0006] According to a third aspect of the present application, an electronic device is provided, including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to implement the method provided in the first aspect.

[0007] The abnormal audio detection method of this application segmentes the audio file to be detected into at least one audio segment to be detected. For any audio segment to be detected, a sample spectrogram corresponding to the audio segment is generated. Based on the similarity between the sample spectrogram and the reference spectrogram, abnormal and non-abnormal regions in the sample spectrogram displayed in the visualization interface are differentiated. By converting the audio file into a spectrogram for visualization and differentiating the abnormal and non-abnormal regions in the sample spectrogram based on the similarity between the sample spectrogram and the reference spectrum, the abnormal regions in the audio file to be detected are made more intuitive, providing technical support for improving the accuracy and efficiency of abnormal labeling of the audio file to be detected. Attached Figure Description

[0008] Figure 1 This is a schematic flowchart illustrating an abnormal audio detection method as an exemplary embodiment of this application; Figure 2 This is a schematic diagram illustrating an implementation process for abnormal audio detection and annotation, as shown in an exemplary embodiment of this application. Figure 3 This is a schematic diagram of audio segmentation shown as an exemplary embodiment of this application; Figure 4 A schematic diagram illustrating a spectrum visualization process is shown as an exemplary embodiment of this application; Figure 5 A schematic diagram illustrating a spectrum visualization result as shown in an exemplary embodiment of this application; Figure 6 This is a schematic diagram illustrating a similarity distance calculation scenario as an exemplary embodiment of this application; Figure 7 This is a schematic diagram illustrating a spectrum overlay display result, as shown in an exemplary embodiment of this application. Figure 8 This is a schematic diagram illustrating another spectrum overlay display result as an exemplary embodiment of this application; Figure 9 This is a schematic diagram illustrating the structure of an abnormal audio detection device according to an exemplary embodiment of this application; Figure 10 This is a schematic diagram of the hardware structure of an electronic device as illustrated in an exemplary embodiment of this application. Detailed Implementation

[0009] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, some terms involved in the embodiments of this application will be briefly explained below.

[0010] 1. Spectrum diagram: A visual representation of how the frequency components of an audio signal change over time. It is a time-frequency analysis tool that converts a one-dimensional audio signal into a two-dimensional image, where the horizontal axis represents time, the vertical axis represents frequency, and the color or brightness represents the intensity of the frequency at that time point (usually expressed in decibels dB).

[0011] 2. Time Domain Graph: This is a direct representation of the audio signal in the time dimension, with the horizontal axis representing time and the vertical axis representing amplitude (or voltage). It shows how the sound waveform changes over time.

[0012] 3. Visualization Conversion: The process of converting audio clips into visual graphics. Time-domain plots (waveform plots) show the change of amplitude over time, while spectrum plots (usually acoustic spectrograms) show the change of frequency components over time.

[0013] 4. Similarity: The mathematical degree to which two objects (audio / image) are consistent.

[0014] 5. DFT (Discrete Fourier Transform): A mathematical tool for converting discrete-time signals from the time domain to the frequency domain. It transforms a finite-length sampled sequence to obtain the amplitude and phase information of the signal at different frequency components.

[0015] 6. STFT (Short-Time Fourier Transform): This is a time-frequency analysis method used to analyze non-stationary signals (i.e., signals whose frequency components change over time). It obtains the local characteristics of the signal's frequency content changing over time by dividing a long signal into multiple shorter time intervals and performing a Fourier transform on each time interval.

[0016] To make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions of the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0017] Please see Figure 1 This is a flowchart illustrating an abnormal audio detection method provided in an embodiment of this application. Figure 1 As shown, the abnormal audio detection method may include the following steps: Step S100: Segment the audio file to be detected to generate at least one audio segment to be detected.

[0018] For example, the audio file to be detected (also known as the original audio file) can be segmented according to the timeline or content features.

[0019] In one example, if the audio data period in the audio file to be detected (which can be simply referred to as the period of the audio file to be detected) is known, the audio file to be detected can be segmented according to the period of the audio file to be detected.

[0020] For example, the audio file to be detected can be obtained from the input information (i.e., the period of the audio data can be used as one of the input parameters).

[0021] For example, when the audio file to be detected is long and the audio data shows obvious periodic changes, the audio file to be detected can be used as an input parameter (such as being manually entered into the system by relevant personnel).

[0022] For example, for an audio file to be detected, it can be divided into at least one audio segment to be detected, using a single cycle as the segmentation unit.

[0023] In cases where the length of the audio file to be detected is not an integer multiple of a single period, the portion that is less than a single period remaining after being divided into segments using a single period as the segmentation unit can be considered as a separate audio segment to be detected.

[0024] In another example, segmenting the audio file to be detected can include: The period of the audio file to be detected is determined using an autocorrelation analysis algorithm; The audio file to be tested is segmented according to its periodicity.

[0025] For example, when the period of the audio file to be detected is unknown, an autocorrelation analysis algorithm can be used to determine the period of the audio file to be detected, and the audio file to be detected can be segmented according to the period of the audio file to be detected.

[0026] Step S110: For any audio segment to be detected, generate a sample spectrum corresponding to the audio segment to be detected.

[0027] In this embodiment of the application, in order to improve the intuitiveness of audio detection, the audio data can be visualized and converted into visualized image data.

[0028] For example, for any audio segment to be detected, the audio segment to be detected can be converted into a spectrogram (which may be called a sample spectrogram).

[0029] Step S120: Based on the similarity between the sample spectrum and the reference spectrum, differentiate the abnormal and non-abnormal regions in the sample spectrum displayed in the visualization interface; wherein, the similarity between the pixels in the abnormal region of the sample spectrum and the corresponding pixels in the reference spectrum is less than the similarity between the pixels in the non-abnormal region of the sample spectrum and the corresponding pixels in the reference spectrum, and the reference spectrum is the spectrum corresponding to the reference audio segment.

[0030] In this embodiment of the application, for any sample spectrogram, the similarity between the sample spectrogram and the reference spectrogram can be determined.

[0031] For example, the similarity between a sample spectrogram and a baseline spectrogram can be characterized by the similarity between the sample spectrogram and the baseline spectrogram at each pixel position.

[0032] This allows us to determine the similarity of each pixel position between the sample spectrogram and the baseline sample image.

[0033] In the spectrum, the pixel positions can be identified using two-dimensional coordinates of time and frequency.

[0034] In this embodiment of the application, in order to enable relevant personnel to more intuitively determine whether there is any abnormality in the audio file to be detected, for any sample spectrogram, the abnormal and non-abnormal regions in the sample spectrogram can be determined based on the similarity between the sample spectrogram and the benchmark spectrogram, and the abnormal and non-abnormal regions in the sample spectrogram displayed in the visualization interface are displayed differently.

[0035] For example, the similarity between pixels in the abnormal region of the sample spectrogram and the corresponding pixels in the reference spectrogram is less than the similarity between pixels in the non-abnormal region of the sample spectrogram and the corresponding pixels in the reference spectrogram.

[0036] It can be seen that, in Figure 1 The method described above involves segmenting the audio file to be detected to generate at least one audio segment. For any audio segment, a sample spectrogram corresponding to that segment is generated. Based on the similarity between the sample spectrogram and the baseline spectrogram, abnormal and non-abnormal regions in the sample spectrogram displayed in the visualization interface are differentiated. By converting the audio file into a spectrogram for visualization and differentiating between the sample spectrogram and the baseline spectrogram, the abnormal regions in the sample spectrogram become more intuitive, providing technical support for improving the accuracy and efficiency of anomaly labeling in the audio file to be detected.

[0037] In some embodiments, the above-mentioned differential display of abnormal and non-abnormal regions in the sample spectrum displayed in the visualization interface based on the similarity between the sample spectrum and the reference spectrum may include: Based on the similarity between the sample spectrum and the baseline spectrum, and the set similarity threshold, abnormal regions in the sample spectrum are determined. The abnormal regions in the spectrum of the sample displayed in the visualization interface are highlighted.

[0038] For example, for any sample spectrogram, abnormal regions in the sample spectrogram can be determined based on the similarity between the sample spectrogram and the baseline spectrogram, as well as a set similarity threshold.

[0039] For example, the similarity between a sample spectrogram and a baseline spectrogram can be characterized by the similarity distance between the sample spectrogram and the baseline spectrogram, and is negatively correlated with this similarity distance.

[0040] Abnormal regions in the sample spectrum can be identified based on the similarity distance between the sample spectrum and the baseline spectrum, as well as the set similarity threshold.

[0041] For example, assuming a similarity threshold of 95%, then based on this similarity threshold, and the maximum and minimum values ​​of the overall similarity distance (i.e., the maximum and minimum values ​​of the similarity distance between the sample spectrogram and the baseline spectrogram at each pixel position, which can be denoted as follows): and The similarity distance boundary between anomalous and non-anomalous regions is determined in the following way: . Among them, regions with a similarity distance greater than the aforementioned similarity distance boundary are identified as abnormal regions.

[0042] For example, to help relevant personnel more intuitively identify abnormal regions in the sample spectrum, the abnormal regions in the sample spectrum displayed in the visualization interface can be highlighted, such as by highlighting.

[0043] For example, abnormal and non-abnormal regions in the sample spectrum can be displayed in different colors, with abnormal regions highlighted.

[0044] In one example, the abnormal audio detection scheme provided in this application embodiment may further include: If a similarity threshold adjustment instruction is detected, the current similarity threshold is updated according to the similarity threshold adjustment instruction; Based on the updated similarity threshold, abnormal regions in the sample spectrum displayed in the visualization interface are identified.

[0045] For example, considering that in real-world application scenarios, some tasks may require accurately identifying small differences between the sample spectrogram and the baseline spectrogram, while other tasks may require special attention when the differences between the sample spectrogram and the baseline spectrogram are large, the similarity thresholds used to determine abnormal and non-abnormal regions in the sample spectrogram can be dynamically adjusted as needed to adapt to the requirements of different task scenarios.

[0046] Accordingly, if a similarity threshold adjustment instruction is detected, the current similarity threshold can be updated according to the similarity threshold adjustment instruction.

[0047] If the similarity threshold is updated in the manner described above, the abnormal regions in the sample spectrum displayed in the visualization interface (which may include the spectrum displayed before the threshold update or the spectrum displayed after the threshold update) can be determined based on the updated similarity threshold.

[0048] In some embodiments, the above-mentioned differential display of abnormal and non-abnormal regions in the sample spectrum displayed in the visualization interface based on the similarity between the sample spectrum and the reference spectrum may include: Align the sample spectrogram with the reference spectrogram; The aligned spectrum of the sample is overlaid with the baseline spectrum in the visualization interface; For the overlapping area between the sample spectrogram and the reference spectrogram, based on the similarity between the sample spectrogram and the reference spectrogram for the same pixel position, abnormal and non-abnormal areas in the overlapping area are determined, and the abnormal and non-abnormal areas are displayed differently.

[0049] For example, in order to more intuitively determine the difference between the sample spectrum and the reference spectrum, and to more intuitively display the abnormal regions in the sample spectrum, the aligned sample spectrum and the reference spectrum can be overlaid in the visualization interface.

[0050] In one example, aligning the sample spectrogram with the reference spectrogram as described above can include: Based on the detected drag operation command, the position of the sample spectrum and / or the reference spectrum in the visualization interface is adjusted to align the sample spectrum with the reference spectrum. And / or, An automatic matching algorithm was used to align the sample spectrogram with the reference spectrogram.

[0051] For example, the alignment of the sample spectrogram with the reference spectrogram can be achieved through manual alignment and / or automatic alignment.

[0052] In manual alignment, relevant personnel can drag and drop the sample spectrum and / or reference spectrum displayed in the visualization interface to adjust their positions and achieve alignment between the sample spectrum and the reference spectrum.

[0053] With automatic alignment, an automatic matching algorithm can be used to quickly locate the starting point of the segment that best matches the selected template in the audio library, thereby aligning the sample spectrogram with the reference spectrogram.

[0054] As an example, aligning the sample spectrogram with the reference spectrogram as described above may include: An automatic matching algorithm was used to initially align the sample spectrogram with the baseline spectrogram. Based on the detected drag operation command, the position of the sample spectrum and / or the reference spectrum in the visualization interface is fine-tuned to align the sample spectrum with the reference spectrum.

[0055] For example, in order to improve the alignment accuracy and efficiency between the sample spectrogram and the reference spectrogram, an automatic alignment plus manual fine-tuning method can be used to align the sample spectrogram and the reference spectrogram.

[0056] For example, an automatic matching algorithm can be used to initially align the sample spectrum with the reference spectrum, and the positions of the sample spectrum and / or the reference spectrum in the visualization interface can be fine-tuned based on the detected drag operation command, so as to finally align the sample spectrum with the reference spectrum.

[0057] In some embodiments, the similarity between the sample spectrogram and the reference spectrogram is determined in the following manner: For any pixel location in the sample spectrogram, determine the first similarity distance and the second similarity distance between the sample spectrogram and the reference spectrogram for that pixel location; Based on the first similarity distance and the second similarity distance, the similarity distance between the sample spectrogram and the reference spectrogram for the pixel position is determined; The first similarity distance is the pixel distance between the sample spectrogram and the reference spectrogram at that pixel position; the second similarity distance is the distance between the original audio data of the time frame corresponding to the sample spectrogram at that pixel position and the original audio data of the time frame corresponding to the reference spectrogram at that pixel position.

[0058] For example, the similarity between a sample spectrogram and a benchmark spectrogram can be characterized by the similarity distance between them.

[0059] For example, the similarity between the sample spectrum and the baseline spectrum is negatively correlated with the similarity distance between the sample spectrum and the baseline spectrum. That is, the greater the similarity distance between the sample spectrum and the baseline spectrum, the lower the similarity between the sample spectrum and the baseline spectrum; the smaller the similarity distance between the sample spectrum and the baseline spectrum, the higher the similarity between the sample spectrum and the baseline spectrum.

[0060] For example, for any pixel location in the sample spectrogram, the similarity distance between the sample spectrogram and the reference spectrogram at that pixel location can include at least two parts: the pixel distance between the sample spectrogram and the reference spectrogram at that pixel location (which can be called the first similarity distance), and the distance between the original audio data of the time frame corresponding to the sample spectrogram at that pixel location and the original audio data of the time frame corresponding to the reference spectrogram at that pixel location (which can be called the second similarity distance).

[0061] For example, for any pixel location in the sample spectrogram, the similarity distance between the sample spectrogram and the reference spectrogram for that pixel location can be a weighted sum of a first similarity distance and a second similarity distance.

[0062] For example, the aforementioned similarity distance may include, but is not limited to, Euclidean distance or cosine distance.

[0063] For example, the first similarity distance is Euclidean distance and the second similarity distance is cosine distance, or the first similarity distance is cosine distance and the second similarity distance is Euclidean distance, or both the first and second similarity distances are cosine distances (or both are Euclidean distances).

[0064] In some embodiments, generating a sample spectrogram corresponding to any audio segment to be detected may include: The audio segment to be detected is divided into frames to obtain multiple overlapping frames; Apply a window function to each overlapping frame and perform a short-time Fourier transform to obtain the complex matrix corresponding to the audio segment to be detected; Convert the complex matrix to a decibel scale to obtain the decibel scale matrix; The decibel scale matrix is ​​mapped to image data to obtain the sample spectrum corresponding to the audio segment to be detected.

[0065] For example, an overlapping frame refers to an audio frame that overlaps with an adjacent audio frame.

[0066] For any audio segment to be detected, the audio can be divided into multiple overlapping frames according to the set parameters, and a window function can be applied to each overlapping frame. For example, each frame can be multiplied by a window function.

[0067] For example, taking the Hamming window as the window function, the window function can be applied to each overlapping frame to reduce spectral leakage, smooth the edges of the frames, and avoid boundary effects in the DFT.

[0068] For example, an STFT transformation can be performed on the overlapping frames of the applied window function to obtain the complex matrix corresponding to the audio segment to be detected, and the complex matrix can be converted into a decibel scale to obtain a decibel scale matrix.

[0069] For example, the power spectrum can be obtained by taking the square of the amplitude of the complex matrix and then converted to a decibel scale to obtain a decibel scale matrix.

[0070] For example, the decibel scale matrix can be mapped to image data to obtain a sample spectrogram corresponding to the audio segment to be detected.

[0071] In some embodiments, the abnormal audio detection scheme provided in this application may further include: Based on the detected annotation operation instructions, determine the annotation area in the sample spectrum; Determine the first time range and the first frequency range corresponding to the labeled area; Audio data in the audio file to be detected that falls within the first time range and the first frequency range are identified as abnormal audio data.

[0072] For example, based on the sample spectrum displayed in the above visualization interface (also known as the annotation interface), relevant personnel can intuitively identify abnormal areas in the sample spectrum and annotate the sample spectrum as needed. For example, areas with a high density of abnormal pixels (highlighted pixels) can be annotated (such as by selecting a box).

[0073] Accordingly, the annotation area can be determined based on the detected annotation operation instructions, and the abnormal audio data corresponding to the annotation area can be identified.

[0074] For example, a sample spectrum can be understood as an image in a two-dimensional coordinate system with time on the horizontal axis and frequency on the vertical axis. The above-mentioned labeled area is a rectangular area in the image. Based on the labeled area, the corresponding time range and frequency range (which can be called the first time range and the first frequency range) can be determined. Audio data in the audio file to be labeled that belong to the first time range and the first frequency range are identified as abnormal audio data.

[0075] In some embodiments, the abnormal audio detection scheme provided in this application may further include: If a selection instruction for a region within a second time range and / or a second frequency range is detected, and a playback instruction for that region is detected, the audio data in the audio file to be detected that belongs to the second time range and / or the second frequency range is played in a loop.

[0076] For example, relevant personnel can also select a specific time range and / or a specific frequency range in the sample spectrogram according to actual needs, and trigger an audio playback command to control the playback of audio data in the audio file to be labeled that belongs to the specific time range and / or specific frequency range.

[0077] Accordingly, when a selection instruction for a region within a specific time range (which may be referred to as the second time range) and / or a specific frequency range (which may be referred to as the second frequency range) is detected, and a playback instruction for that region is detected, audio data in the audio file to be labeled that belongs to the second time range and / or the second frequency range can be played in a loop to help the user focus on the region of interest.

[0078] For example, if a selection instruction for the second time range is detected but no selection instruction for the second frequency range is detected (i.e., the relevant personnel only selected the time range in the visualization interface), the second frequency range can be defaulted to the full frequency range of the sample spectrum.

[0079] Similarly, if a selection instruction for the second frequency range is detected but a selection instruction for the second time range is not detected, the second time range can be assumed to be the full time range of the sample spectrum.

[0080] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below with reference to specific examples.

[0081] In this embodiment, an abnormal audio detection and annotation scheme is provided, such as... Figure 2 As shown, its implementation process may include the following steps: 1) The audio file to be labeled (i.e. the audio file to be detected, also known as the original audio file) is segmented to generate audio segments.

[0082] For example, the original audio file to be labeled can be segmented according to the timeline or content features to generate multiple audio segments.

[0083] 2) Visualize the audio clips.

[0084] For example, an audio segment can be converted into a spectrogram to visualize the audio segment.

[0085] 3) Calculate the similarity between the sample spectrogram and the baseline spectrogram.

[0086] For example, a similarity algorithm can be selected to calculate the similarity between the sample spectrogram (the spectrogram of the sample audio segment, i.e., the audio segment to be labeled) and the reference spectrogram (the spectrogram of the reference audio segment).

[0087] 4) Identify the abnormal regions in the sample spectrum and highlight them.

[0088] For example, based on the set similarity threshold and the similarity between the sample spectrogram and the baseline spectrogram, abnormal regions in the sample spectrogram can be identified and highlighted.

[0089] 5) Based on the detected annotation operation instructions, determine the annotation area and the abnormal audio data corresponding to the annotation area.

[0090] For example, based on the sample spectrum displayed in the above visualization interface (also known as the annotation interface), relevant personnel can intuitively identify abnormal areas in the sample spectrum and annotate the sample spectrum as needed. For example, areas with a high density of abnormal pixels (highlighted pixels) can be annotated (such as by selecting a box).

[0091] Accordingly, the annotation area can be determined based on the detected annotation operation instructions, and the abnormal audio data corresponding to the annotation area can be identified.

[0092] For example, a sample spectrum can be understood as an image in a two-dimensional coordinate system with time on the horizontal axis and frequency on the vertical axis. The above-mentioned labeled area is a rectangular area in the image. Based on the labeled area, the corresponding time range and frequency range (such as the first time range and the first frequency range mentioned above) can be determined. Audio data in the audio file to be labeled that belong to the time range and frequency range are identified as abnormal audio data.

[0093] The implementation details of the above process are explained below.

[0094] 1. Segment the audio file to be annotated to generate audio segments.

[0095] For example, when the audio file to be labeled is long and the audio data shows obvious periodic changes, the audio can be segmented and the segmented audio data can be labeled to improve the labeling efficiency.

[0096] For a piece of audio data , assuming the period is T (T < N, where N is the length of the audio data), when N / T is an integer, it can be divided into N / T sequences, and when N / T is a non-integer, it can be divided into ⌊N / T⌋ + 1 segments of audio sequences.

[0097] Taking the example of being divided into ⌊N / T⌋ + 1 segments of audio sequences, these ⌊N / T⌋ + 1 segments of audio sequences can be: Among them, , , ⌊ ⌋ represents the floor operation.

[0098] For example, suppose there is a piece of audio , with a period T of 300, then the original audio can be divided into 4 segments of audio . Among them, , , , .

[0099] Another example, suppose the original audio is X[160000], with a period T of 8000, the original audio can be divided into 20 segments of audio clips, and the time-domain graph of this audio data can be as Figure 3 shown. Among them, Figure 3 from top to bottom in

[0100] are the original audio and the first 3 segments of audio clips after segmentation in turn.

[0101] Exemplarily, the spectrogram is a time-frequency analysis tool used to display the change of the signal frequency components over time, and it is based on the short-time Fourier transform (STFT) at the core. STFT obtains the time-frequency representation by dividing the signal into short-time frames and applying the discrete Fourier transform (DFT) to each frame. Mathematically, STFT is defined as: Among them, is the discrete audio signal, function (such as the Hamming window), with a length of N (frame length), used to reduce spectral leakage, smooth the edges of the frame, and avoid boundary effects in the DFT. Among them, the Hamming window can be defined as: m is the frame index, H is the hop number, representing the number of sampling points between frames, and k is the frequency index.

[0102] is a complex matrix, representing the amplitude and phase of the m-th frame at the k-th frequency component.

[0103] Exemplarily, as Figure 4As shown, the spectrum visualization process may include the following steps: 1) Set parameters to divide the input sequence into frames. For any audio segment, divide it into multiple overlapping frames.

[0104] For example, the data of the m-th frame is Among them, the setting of frame length N affects frequency resolution, and the number of hops H is usually set to 50% of frame length, which affects time resolution.

[0105] 2) Apply window functions to overlapping frames.

[0106] For example, each overlapping frame can be processed separately. Window function .

[0107] 3) Calculate STFT.

[0108] For example, a complex matrix can be obtained by applying the DFT to each frame separately. .

[0109] 4) Convert the complex matrix to a decibel scale to obtain the decibel scale matrix.

[0110] For example, take The power spectrum is obtained by squared amplitude. And converted to a decibel scale. Here, ε is a small constant to avoid taking the logarithm of zero.

[0111] 5) Generate a spectrum based on the decibel scale matrix.

[0112] For example, a decibel scale matrix can be mapped to image data.

[0113] For example, mapping to grayscale values ​​of 0-255 can be calculated as follows: For example, a schematic diagram of the spectrogram generated in the manner described above can be shown as follows: Figure 5 As shown, the frame length is 3ms, the number of hops is 75% of the frame length, and the window function is a Hamming window.

[0114] Among them, for Figure 5 The spectrum diagram shown has time on the horizontal axis and frequency on the vertical axis. The color of any pixel position (which can be represented by time and frequency) can represent the signal strength at a specific time and frequency point.

[0115] III. Calculate the similarity between the sample spectrum and the baseline spectrum.

[0116] For example, the similarity between a sample spectrogram and a benchmark spectrogram can be characterized by the similarity distance between the sample spectrogram and the benchmark spectrogram, and the similarity between the sample spectrogram and the benchmark spectrogram is negatively correlated with this similarity distance.

[0117] For example, calculate the similarity distance between the sample spectrogram and the baseline spectrogram. The calculation algorithm consists of two parts: one part is the pixel distance on the image. The other part is the distance of that pixel between the corresponding time frames of the original audio (i.e., the original audio data). The aforementioned distances may include, but are not limited to, cosine distances or Euclidean distances. That is: in, The total distance between pixels (a, b) Pixel distance α represents the distance to the corresponding segment in the original audio, and α is the weighting coefficient.

[0118] in, It can be used to characterize the difference between the sample spectrogram and the baseline spectrogram at pixel points (a, b). The larger the value, the greater the difference between the sample spectrogram and the baseline spectrogram at pixel points (a, b).

[0119] Here, 'a' can represent time, corresponding to the start of a specific time frame in the original audio, and 'Δt' is the length of the time frame. This represents the distance between the first audio segment (such as the reference audio, i.e., normal audio) and the second audio segment (such as the audio to be labeled) at time a to a+Δt.

[0120] For example, assuming a is 2 and Δt is 0.1, then This represents the distance calculated from the original audio data within the range of [2s, 2.1s], assuming the first and second audio segments are aligned to the same starting point.

[0121] For example, taking Euclidean distance for pixel distance and cosine distance for original audio clip distance, assuming α is 0.5, It can be represented as: in, This refers to the original audio data of the time frame corresponding to column a in the audio spectrogram I to be labeled. This is the original audio data of the time frame corresponding to column 0 in the normal audio spectrogram. This is the mean of the original audio data in that frame.

[0122] like Figure 6As shown, to calculate the similarity distance at (1, 2) ,but: I(1,2)=8, I0(1,2)=4 4. Identify the abnormal regions in the sample spectrum and highlight them.

[0123] For example, sample spectrum and reference spectrum can be overlaid in the same area, and the schematic diagram can be as follows: Figure 7 As shown. Among them, Figure 7 The red portion represents the spectrum of the audio to be labeled (i.e., Figure 7 Spectrum in Figure 1 The blue portion represents the spectrum of normal audio (i.e., Figure 7 Spectrum in Figure 2 The purple area represents the overlapping part of the two audio segments, while the yellow highlighted area represents the part where the two audio segments differ significantly in spectral similarity.

[0124] For example, assuming a similarity threshold of 95%, then for any pixel location in the sample spectrogram and the baseline spectrogram, if the similarity distance between the sample spectrogram and the baseline spectrogram at that pixel location is greater than 95%, then... In such cases, it can be marked as highlighted.

[0125] Among them, the overlay display can intuitively identify the differences between the audio to be labeled and normal audio in a specific time period and frequency range.

[0126] For example, users can drag the spectrograms of two audio segments along the horizontal axis to see the comparison effect under different overlap states.

[0127] In addition, the similarity threshold can be manually adjusted to adapt to the needs of different annotation scenarios.

[0128] For example, Figure 8 The display shows the spectral superposition effect of two audio segments that completely overlap in time scale, with the red area being the highlighted part and the threshold set to 90%.

[0129] 5. Based on the detected annotation operation instructions, determine the annotation area and the corresponding abnormal audio data of the annotation area.

[0130] For example, based on the sample spectrum displayed in the visualization interface, relevant personnel can intuitively identify abnormal areas in the sample spectrum and annotate the sample spectrum as needed. For example, areas with a high density of abnormal pixels (highlighted pixels) can be annotated (e.g., selected by box).

[0131] Accordingly, the annotation area can be determined based on the detected annotation operation instructions, and the abnormal audio data corresponding to the annotation area can be identified.

[0132] For example, a sample spectrum can be understood as an image in a two-dimensional coordinate system with time on the horizontal axis and frequency on the vertical axis. The above-mentioned labeled area is a rectangular area in the image. Based on the labeled area, the corresponding time range and frequency range (such as the first time range and the first frequency range mentioned above) can be determined. Audio data in the audio file to be labeled that belong to the time range and frequency range are identified as abnormal audio data.

[0133] For example, relevant personnel can also select a specific time range (such as the second specific time range mentioned above) and / or a specific frequency range (such as the second frequency range mentioned above) in the sample spectrogram according to actual needs, and trigger an audio playback command to control the playback of audio data in the audio file to be labeled that belongs to the specific time range and / or specific frequency range.

[0134] Accordingly, when a selection instruction for a region within a specific time range and / or a specific frequency range is detected, and a playback instruction for that region is detected, audio data in the audio file to be labeled that belongs to that specific time range and / or specific frequency range can be played in a loop to help the user focus on the area of ​​interest.

[0135] In addition, the visualization interface can adopt a multi-window layout, supporting the simultaneous display and linkage control of the visualization views of the benchmark audio and multiple sample audios, thus improving operational efficiency.

[0136] For example, the visualization interface can display multiple windows, one of which is the baseline spectrum graph, and the other windows are the superimposed display results of different sample spectrum graphs and the baseline spectrum graph. While keeping the position of the sample spectrum graph unchanged: the baseline spectrum graph can be dragged left and right, and the other windows will synchronously display the superimposed display results at the current position of the baseline audio; and / or, the view can be zoomed in / out / panned, and the view in the other windows will also be zoomed in / out / panned synchronously.

[0137] It should be noted that during the visualization process of audio clips, the audio clips can also be converted into time-domain graphs and displayed in the visualization interface.

[0138] Once the abnormal regions in the sample spectrum have been identified in the manner described above, relevant personnel can check the differences between the sample time-frequency graph and the reference time-frequency graph within the time range corresponding to the abnormal regions in the sample spectrum, so that relevant personnel can confirm the abnormal regions.

[0139] For example, assuming that the time range corresponding to the abnormal region determined in the above manner is 10s-12s (assuming it is aligned with the same starting point) and 1000hz-1500hz, relevant personnel can confirm the difference between the sample time-frequency map and the reference time-frequency map within the time range of 10s-12s on the time-frequency map.

[0140] As can be seen, in this embodiment, audio annotation is converted into image annotation by using spectrum chart annotation, which makes the annotation more intuitive, avoids misjudgment caused by human ear subjectivity and fatigue, improves the consistency of annotation, and reduces the ability requirements of annotation personnel.

[0141] In addition, the annotation process uses "similarity threshold filtering + visual highlighting" to narrow the scope of manual detection. Only a small number of differences selected by the machine need to be confirmed and annotated, thus improving annotation efficiency.

[0142] Furthermore, the adjustable threshold allows it to adapt to various precision requirements, ranging from fine-grained speech annotation to coarse-grained sound event detection.

[0143] The method provided in this application has been described above. The apparatus provided in this application is described below: Please see Figure 9 This is a schematic diagram of the structure of an abnormal audio detection device provided in an embodiment of this application, as shown below. Figure 9 As shown, the abnormal audio detection device may include: The segmentation unit is used to segment the audio file to be detected, generating at least one audio segment to be detected. The visualization unit is used to generate a sample spectrum diagram corresponding to any of the audio segments to be detected; The processing unit is used to differentiate between abnormal and non-abnormal regions in the sample spectrum diagram displayed in the visualization interface based on the similarity between the sample spectrum diagram and the reference spectrum diagram; wherein, the similarity between pixels in the abnormal region of the sample spectrum diagram and the corresponding pixels in the reference spectrum diagram is less than the similarity between pixels in the non-abnormal region of the sample spectrum diagram and the corresponding pixels in the reference spectrum diagram, and the reference spectrum diagram is the spectrum diagram corresponding to the reference audio segment.

[0144] For example, the specific implementation process of abnormal audio detection by each functional unit in the abnormal audio detection device can be found in the relevant description in the above embodiments, and the embodiments of this application are not limited here.

[0145] Please see Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 1001 and a memory 1002 storing machine-executable instructions. The processor 1001 and the memory 1002 can communicate via a system bus 1003. Furthermore, by reading and executing the machine-executable instructions corresponding to the abnormal audio detection control logic in the memory 1002, the processor 1001 can execute the abnormal audio detection method described above.

[0146] The memory 1002 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0147] In some embodiments, a machine-readable storage medium, such as Figure 10 The memory 1002 in the device stores machine-executable instructions, which, when executed by a processor, implement the abnormal audio detection method described above. For example, the machine-executable storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

Claims

1. An abnormal audio detection method, characterized in that, include: The audio file to be detected is segmented to generate at least one audio segment to be detected. For any of the audio segments to be detected, a sample spectrogram corresponding to the audio segment to be detected is generated; Based on the similarity between the sample spectrogram and the benchmark spectrogram, the abnormal and non-abnormal regions in the sample spectrogram displayed in the visualization interface are differentiated; wherein, the similarity between the pixels in the abnormal region of the sample spectrogram and the corresponding pixels in the benchmark spectrogram is less than the similarity between the pixels in the non-abnormal region of the sample spectrogram and the corresponding pixels in the benchmark spectrogram, and the benchmark spectrogram is the spectrogram corresponding to the benchmark audio segment.

2. The method according to claim 1, characterized in that, The method of differentiating abnormal and non-abnormal regions in the sample spectrum displayed in the visualization interface based on the similarity between the sample spectrum and the benchmark spectrum includes: Based on the similarity between the sample spectrum and the baseline spectrum, and the set similarity threshold, abnormal regions in the sample spectrum are determined. The abnormal regions in the spectrum of the sample displayed in the visualization interface are highlighted.

3. The method according to claim 2, characterized in that, The method further includes: Upon detecting a similarity threshold adjustment instruction, the current similarity threshold is updated according to the similarity threshold adjustment instruction; Based on the updated similarity threshold, abnormal regions in the sample spectrum displayed in the visualization interface are identified.

4. The method according to claim 1, characterized in that, The method of differentiating abnormal and non-abnormal regions in the sample spectrum displayed in the visualization interface based on the similarity between the sample spectrum and the benchmark spectrum includes: Align the sample spectrogram with the reference spectrogram; The aligned sample spectrum is overlaid with the reference spectrum in the visualization interface. For the overlapping area between the sample spectrogram and the reference spectrogram, based on the similarity between the sample spectrogram and the reference spectrogram for the same pixel position, abnormal and non-abnormal areas in the overlapping area are determined, and the abnormal and non-abnormal areas are displayed differently.

5. The method according to claim 4, characterized in that, Aligning the sample spectrogram with the reference spectrogram includes: Based on the detected drag operation command, the position of the sample spectrum and / or the reference spectrum in the visualization interface is adjusted to align the sample spectrum with the reference spectrum. And / or, An automatic matching algorithm is used to align the sample spectrogram with the reference spectrogram.

6. The method according to claim 5, characterized in that, Aligning the sample spectrogram with the reference spectrogram includes: An automatic matching algorithm is used to initially align the sample spectrogram with the reference spectrogram. Based on the detected drag operation command, the position of the sample spectrum and / or the reference spectrum in the visualization interface is finely adjusted to align the sample spectrum with the reference spectrum.

7. The method according to claim 1, characterized in that, The similarity between the sample spectrogram and the baseline spectrogram is determined in the following way: For any pixel location in the sample spectrogram, determine the first similarity distance and the second similarity distance between the sample spectrogram and the reference spectrogram for that pixel location; Based on the first similarity distance and the second similarity distance, the similarity distance between the sample spectrogram and the reference spectrogram for that pixel position is determined; Wherein, the first similarity distance is the pixel distance between the sample spectrogram and the reference spectrogram at that pixel position; the second similarity distance is the distance between the original audio data of the time frame corresponding to the pixel position of the sample spectrogram and the original audio data of the time frame corresponding to the pixel position of the reference spectrogram.

8. The method according to claim 1, characterized in that, The segmentation of the audio file to be detected includes: The period of the audio file to be detected is determined using an autocorrelation analysis algorithm; The audio file to be tested is segmented according to its period.

9. The method according to claim 1, characterized in that, For any of the audio segments to be detected, generating a sample spectrogram corresponding to that audio segment includes: The audio segment to be detected is divided into frames to obtain multiple overlapping frames; Apply a window function to each overlapping frame and perform a short-time Fourier transform to obtain the complex matrix corresponding to the audio segment to be detected; The complex matrix is ​​converted to a decibel scale to obtain the decibel scale matrix; The decibel scale matrix is ​​mapped to image data to obtain the sample spectrum corresponding to the audio segment to be detected.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: Based on the detected annotation operation instructions, the annotation region in the sample spectrum is determined; Determine the first time range and the first frequency range corresponding to the labeled area; Audio data in the audio file to be detected that falls within the first time range and the first frequency range are identified as abnormal audio data.

11. The method according to any one of claims 1-9, characterized in that, The method further includes: Upon detecting a selection instruction for a region within a second time range and / or a second frequency range, and upon detecting a playback instruction for that region, the audio data in the audio file to be detected that belongs to the second time range and / or the second frequency range is played in a loop.

12. An abnormal audio detection device, characterized in that, include: The segmentation unit is used to segment the audio file to be detected, generating at least one audio segment to be detected. The visualization unit is used to generate a sample spectrum diagram corresponding to any of the audio segments to be detected; The processing unit is used to differentiate between abnormal and non-abnormal regions in the sample spectrum diagram displayed in the visualization interface based on the similarity between the sample spectrum diagram and the reference spectrum diagram; wherein, the similarity between pixels in the abnormal region of the sample spectrum diagram and the corresponding pixels in the reference spectrum diagram is less than the similarity between pixels in the non-abnormal region of the sample spectrum diagram and the corresponding pixels in the reference spectrum diagram, and the reference spectrum diagram is the spectrum diagram corresponding to the reference audio segment.

13. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method as described in any one of claims 1-11.

Images