Audio key frame extraction method for belt conveyor inspection robot based on frequency band entropy and pixel saliency

Abstract

The present application relates to a kind of band conveyor inspection robot audio key frame extraction method based on frequency band entropy and pixel saliency, belong to the field of band conveyor roller fault detection.The method includes: input audio signal collected by inspection robot, obtain the time-frequency matrix of input audio signal by short-time Fourier transform;Calculate the entropy value of each row of time-frequency matrix to obtain the information entropy of each frequency band, obtain frequency band entropy;Determine whether there is the case that frequency band entropy is lower than lower threshold value, if there is, the audio frame is extracted as abnormal key frame, otherwise, determine whether there is the case that n frequency band entropies are lower than upper threshold value in a period of time, if there is, the audio frame is extracted as abnormal key frame;Based on time-frequency matrix, calculate pixel saliency index, determine whether the saliency of audio signal in medium-high frequency exceeds the set limit threshold, if exceed, the audio frame is extracted as abnormal key frame, otherwise, determine that conveyor roller is normal.

Description

Technical Field

[0001] This invention belongs to the field of fault diagnosis of idler rollers in mining belt conveyors based on inspection robots, and relates to a method for extracting audio keyframes from belt conveyor inspection robots based on bandwidth entropy and pixel saliency. Background Technology

[0002] As the main transport equipment in coal mines, the safety and stability of belt conveyors are crucial to the production efficiency and safety of coal mines. Traditional manual inspection methods suffer from low efficiency, poor real-time performance, and potential safety hazards; fixed sensors are difficult to deploy, maintain, and costly. The emergence of belt conveyor inspection robots provides a new mode of inspection. Unlike the single data from fixed sensors with fixed modes, the data collected by inspection robots during movement is dynamic and time-varying. The direct application of traditional fixed sensor fault diagnosis methods to inspection robots often yields poor results, mainly due to the following reasons:

[0003] 1) Fixed sensors installed near a roller collect signals indicating health or malfunction, exhibiting two distinct patterns. Signal processing and pattern recognition can effectively detect malfunctions. However, because the inspection robot is constantly moving, the signals it collects are time-varying and dynamic, lacking a fixed pattern, making them difficult to handle using traditional malfunction detection methods.

[0004] 2) The inspection robot's path is subject to various non-stationary noise interferences, and the audio it collects may include sounds from nearby equipment, pedestrians, and environmental noise. Traditional signal processing methods struggle to distinguish and eliminate these various noises, and traditional fault detection methods easily identify this type of noise as a roller fault. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method for extracting audio keyframes of a belt conveyor inspection robot based on frequency band entropy and pixel saliency. Frequency band entropy can extract fault or abnormal noise frames with concentrated audio energy, and pixel saliency can extract fault or abnormal noise frames caused by slight continuous collisions. The extraction of audio keyframes provides basic data for subsequent idler roller fault location and fault identification.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for extracting audio keyframes from a belt conveyor inspection robot based on bandwidth entropy and pixel saliency, the method comprising:

[0008] S1. Input the audio signal collected by the inspection robot, and obtain the time-frequency matrix of the input audio signal through short-time Fourier transform; each row of the time-frequency matrix represents the change of signal amplitude within a certain frequency band within a time period, and each column represents the frequency distribution within the same time window.

[0009] S2. Calculate the entropy value for each row of the time-frequency matrix to obtain the information entropy of each frequency band, thus obtaining the frequency band entropy;

[0010] S3. Determine if the frequency band entropy is below the lower threshold. If it is, extract the audio frame as an abnormal key frame. Otherwise, proceed to step S4.

[0011] S4. Determine if there are n frequency bands with entropy lower than the upper threshold within a certain period of time. If so, extract the audio frame as an abnormal key frame; otherwise, proceed to step S5.

[0012] S5. Calculate the pixel saliency index based on the time-frequency matrix of the audio signal, and determine whether the sum of the saliency of the audio signal in the mid-to-high frequency range exceeds the set threshold. If it does, the audio frame is extracted as an abnormal key frame; otherwise, it is determined that the conveyor roller is not abnormal.

[0013] Furthermore, in step S2, the information entropy is calculated using the following formula:

[0014]

[0015] P i =P9x i _, i = 1, 2, ..., N

[0016]

[0017] In the formula, H(X) represents information entropy, x i Let P represent a discrete random variable. i x represents i The probability of occurrence; where, when P i When = 0, we define ln0 = 0.

[0018] Furthermore, in step S5, the method for calculating the pixel saliency index includes:

[0019] The pixel P in the i-th row and j-th column of the time-frequency matrix i,j For the target pixel, a local window of length 2n is selected centered on the target pixel as follows:

[0020]

[0021] In the formula, win i,j Indicated by pixel P i,j A local window centered on the user;

[0022] Calculate the local distance matrix:

[0023] D i,j =P i,j -win i,j

[0024] In the formula, "-" indicates that pixel P is used. i,j Values ​​of window win i,j Subtract each value from the matrix D. i,j It represents the degree of difference between the target pixel and its surrounding pixels, indicating the drastic changes in the local image.

[0025] Calculate the saliency of the target pixel:

[0026]

[0027] In the formula, N represents the number of discrete audio signals; the significance index S i,j Measure target pixel P i,j The pixel changes of its neighboring local window.

[0028] After calculating the pixel saliency index, the 99.5% upper quantile of the saliency index of all pixels is set as the threshold, and pixels with values ​​greater than the threshold are retained as saliency pixels.

[0029] Furthermore, in step S5, the threshold for exceeding the limit is the sum of the saliency of pixels with frequencies above 2kHz, and this sum reaches one-third of the total sum of pixel saliency.

[0030] The beneficial effects of this invention are as follows:

[0031] (1) This invention breaks through the limitation that traditional fault detection algorithms cannot be directly applied to inspection robots, and can extract key frames that may contain roller faults from the audio signals collected by the inspection robot.

[0032] (2) This invention combines the advantages of signal processing and image analysis algorithms to extract audio frames near faulty idlers such as broken, worn, stalled, and bearing damaged rollers.

[0033] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0034] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0035] Figure 1 This is a schematic diagram of the audio keyframe extraction method for a belt conveyor inspection robot based on bandwidth entropy and pixel saliency proposed in an embodiment of the present invention.

[0036] Figure 2 This represents the frequency band entropy corresponding to the audio signal under different conditions of the idler roller. Figure 2 (a) is the normal state. Figure 2 (b) shows the bearing damage condition. Figure 2 (c) represents the worn state. Figure 2 (d) indicates a stalled state;

[0037] Figure 3 This is a graph showing the statistical results of pixel significance in different frequency bands for normal and broken idler rollers. Figure 3 (a) is a normal idler roller. Figure 3 (b) is a broken idler roller;

[0038] Figure 4 This is a schematic diagram illustrating the extraction effect of continuous audio frames for idler roller stall faults. Detailed Implementation

[0039] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0040] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0041] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0042] Given the complexity of the audio signals collected by the inspection robot, this invention addresses the detection of idler roller faults using the audio signals from the inspection robot. It extracts abnormal audio frames that can reflect idler roller faults from the audio collected along the inspection robot's path using frequency band entropy and pixel saliency indicators, thus providing a foundation for fault detection based on inspection robots.

[0043] The method for extracting abnormal audio frames from inspection robots based on frequency band entropy includes: obtaining the time-frequency diagram of the signal through short-time Fourier transform, where the short-time Fourier transform is expressed as:

[0044]

[0045] Where x(n) represents the input signal and w(n) represents the window function. The short-time Fourier transform can be used to obtain the time-varying spectral distribution of the signal, thus providing more refined time and frequency characteristics of the audio signal.

[0046] Let there be a set of discrete random variables {X} = {x1, x2, ..., xn}. N} represents the given audio signal, and each x i The probability of occurrence is P i =P(x i ), i = 1, 2, ..., N, The information entropy of an audio signal can then be expressed as:

[0047]

[0048] In this invention, ln0 = 0 is defined. The information entropy of the audio signal is calculated using the amplitude of the audio signal as a random variable. Assume the time-frequency matrix obtained from the short-time Fourier transform of signal x(n) is:

[0049]

[0050] Where M represents the number of frequency bands, and C is the number of windows in the short-time Fourier transform. Each row represents the change in signal amplitude within a certain frequency band over a given time period, and each column represents the frequency distribution within the same time window. By calculating the entropy value for each row of the time-frequency matrix, the information entropy value for each frequency band can be obtained, which is called the frequency band entropy.

[0051] When the inspection robot patrols along the belt conveyor, since most of the idlers are normal and healthy, the frequency band entropy obtained through time-frequency transformation fluctuates at a relatively large value. When the inspection robot reaches the vicinity of a faulty idler, it can collect the sound of the faulty idler. At this time, the observed value of the frequency band entropy will drop significantly at the fault frequency. By monitoring the change in frequency band entropy, the abnormal audio frame region where the fault energy is concentrated can be accurately identified.

[0052] Anomaly audio frame extraction for inspection robots based on pixel saliency includes:

[0053] The time-frequency feature matrix of an audio signal can be viewed as a single-channel two-dimensional image. The significant pixels of an image defined in this invention are those whose differences from surrounding points are relatively obvious. First, the target pixel P is selected. i,j For the pixel in row i and column j, a local window of length 2n is selected centered on the target pixel as follows:

[0054]

[0055] Among them, win i,j Indicated by pixel P i,j A local window centered on the target pixel. By calculating the difference between the target pixel and all values ​​within the local window, the local image changes around the target pixel can be obtained. The local distance matrix is ​​defined as:

[0056] D i,j =P i,j -win i,j

[0057] The minus sign indicates the use of pixel P. i,j Values ​​of the window win i,j Subtract each value individually from the matrix D. i,j It represents the degree of difference between the target pixel and its surrounding pixels, indicating the drastic changes in the local image.

[0058] The salience of the target pixel can then be defined as:

[0059]

[0060] Where, ∑D i,j 2 Represents the matrix D i,j Sum of the squares of all elements. Significance index Si,j It can measure the target pixel P i,j The pixel changes of its neighboring local window.

[0061] After calculating the saliency index of all pixels in the time-frequency image, the most salient pixels in the image are extracted by setting a threshold. A fixed threshold is used, set at the 99.5% upper quantile of the saliency index of all pixels; only pixels exceeding this threshold are retained as salient pixels. The salient pixels saved for each frequency band are summed. Since the audio signal value of a normal idler roller is concentrated in the low frequency range, its salient pixels are concentrated in the low frequency range. When the sum of the saliency in the mid-to-high frequency range exceeds the set threshold, an idler roller fault is identified, and the audio frame at this time is extracted as an abnormal frequency frame. The pixel saliency index can supplement the frequency band entropy, extracting abnormal audio of idler roller faults where the energy is not highly concentrated but has abnormal energy distribution in the high-frequency range.

[0062] In summary, this invention proposes an effective method for identifying two types of abnormal situations by using a lower and upper threshold for frequency band entropy. When the frequency band entropy is lower than the lower threshold, it indicates a very obvious energy concentration fault, and this audio frame can be directly extracted as an abnormal key frame. When no frequency band entropy is lower than the lower threshold, but n frequency band entropies are lower than the upper threshold, this frame can also be extracted as an abnormal frame. Then, the ratio of high frequencies in pixel saliency is used to determine whether this frame is a continuous collision roller fault, thus completing the entire abnormal frame extraction process.

[0063] like Figure 1 As shown, the method specifically includes:

[0064] 1. Input the audio signal collected by the inspection robot;

[0065] 2. Obtain the time-frequency matrix of the input audio signal through short-time Fourier transform;

[0066] 3. Calculate the frequency band entropy based on the time-frequency matrix;

[0067] 4. Determine if the frequency band entropy is below the lower threshold. If it is, extract the audio frame as an abnormal keyframe; otherwise, proceed to the next step.

[0068] 5. Determine if there are n frequency bands with entropy lower than the upper threshold. If so, extract the audio frame as an abnormal keyframe; otherwise, proceed to the next step.

[0069] 6. Calculate the significance index based on the time-frequency feature matrix of the audio signal, and determine whether the sum of the significance of the audio signal in the mid-to-high frequency range exceeds the set threshold. If it does, the audio frame is extracted as an abnormal key frame; otherwise, the conveyor roller is determined to be normal.

[0070] Figure 2 The figure shows the frequency band entropy corresponding to the four idler roller audio frequencies. From Figure 2 As shown in (a), the entropy value of the normal idler roller audio remains consistently high, fluctuating around 5.14 in most frequency bands, except for a very low frequency range where the entropy value is around 5.04. Idler rollers with damaged bearings exhibit very obvious energy concentration at certain frequencies, such as... Figure 2 As shown in (b), its bandwidth entropy decreases significantly in these bandwidths, even dropping below 4.6 near 5kHz. Roller wear exhibits a high-frequency concentrated energy distribution across a wide bandwidth; however, due to its high frequency, the energy is not particularly concentrated, such as... Figure 2 As shown in (c), its entropy value only decreases to around 4.7–4.9. Due to the wide bandwidth, the entropy value of a large portion of the frequency band is below 4.9. Stalled idlers will generate periodic friction with the belt, producing a strong sound in the friction frequency range, with a noticeable concentrated energy sound around 6kHz, and its corresponding entropy value is significantly reduced, such as… Figure 2 As shown in (d), the frequency band entropy can effectively extract abnormal audio frames related to roller damage, wear, and stall.

[0071] like Figure 3 The image shows the statistical results of pixel saliency in different frequency bands for normal and broken idlers. The comparison reveals that the image saliency of the normal idler only appears in the low-frequency region, such as... Figure 3 As shown in (a), no significant points appear at medium to high frequencies; however, many significant points at medium to high frequencies appear on the broken idler roller, such as... Figure 3 As shown in (b), this invention uses a threshold where the sum of the saliency values ​​of frequencies above 2kHz reaches one-third of the total saliency. When this threshold is exceeded, the idler roller is considered to have malfunctioned, and the corresponding audio frame is extracted as an abnormal keyframe. It is evident that the pixel saliency index can effectively extract abnormal audio frames of broken idler rollers.

[0072] Figure 4 The image shows the extraction results of continuous audio frames related to idler roller stall faults. Due to the large number of time-frequency graphs of the audio frames, only the keyframe extraction results for the beginning, middle, and end stages of keyframe detection are displayed; all frames in between can be extracted. Figure 4 As can be seen, when the robot begins to collect the audio from the stalled roller, the energy begins to concentrate; as the robot passes the stalled roller, the energy is most concentrated, and the frequency band entropy decreases significantly; as the robot leaves the stalled roller, the time-frequency diagram begins to slowly return to normal.

[0073] In summary, this invention can effectively extract continuous audio keyframes during the process of an inspection robot passing through a stall fault. In addition, roller wear and bearing damage can also be effectively extracted, which will not be shown here.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for extracting audio keyframes from a belt conveyor inspection robot based on bandwidth entropy and pixel saliency, characterized in that: The method includes the following steps: S1. Input the audio signal collected by the inspection robot, and obtain the time-frequency matrix of the input audio signal through short-time Fourier transform; each row of the time-frequency matrix represents the change of signal amplitude within a certain frequency band within a time period, and each column represents the frequency distribution within the same time window. S2. Calculate the entropy value for each row of the time-frequency matrix to obtain the information entropy of each frequency band, thus obtaining the frequency band entropy; S3. Determine if the frequency band entropy is below the lower threshold. If it is, extract the corresponding audio frame as an abnormal key frame. Otherwise, proceed to step S4. S4. Determine if there are n frequency bands with entropy lower than the upper threshold within a certain period of time. If so, extract the corresponding audio frame as an abnormal key frame. Otherwise, proceed to step S5. S5. Calculate the pixel saliency index based on the time-frequency matrix of the audio signal, and determine whether the sum of the saliency of the audio signal in the mid-to-high frequency range exceeds the set threshold. If it does, the corresponding audio frame is extracted as an abnormal key frame; otherwise, it is determined that the conveyor roller is not abnormal.

2. The audio keyframe extraction method according to claim 1, characterized in that: In step S2, the information entropy is calculated using the following formula: P i =P(x i ),i=1,2,…,N In the formula, H(X) represents information entropy, x i Let P represent a discrete random variable. i x represents i The probability of occurrence; where, when P i When = 0, we define ln0 = 0.

3. The audio keyframe extraction method according to claim 1, characterized in that: In step S5, the method for calculating the pixel saliency index includes: The pixel P in the i-th row and j-th column of the time-frequency matrix i,j For the target pixel, a local window of length 2n is selected centered on the target pixel as follows: In the formula, win i,j Indicated by pixel P i,j A local window centered on the user; Calculate the local distance matrix: D i,j =P i,j -win i,j In the formula, "-" indicates that pixel P is used. i,j Values ​​of the window win i,j Subtract each value from the matrix D. i,j It represents the degree of difference between the target pixel and its surrounding pixels, indicating the drastic changes in the local image; Calculate the saliency of the target pixel: In the formula, N represents the number of discrete audio signals; the significance index S i,j Measure target pixel P i,j The pixel changes of its neighboring local window.

4. The audio keyframe extraction method according to claim 3, characterized in that: After calculating the pixel saliency index, the 99.5% upper quantile of the saliency index of all pixels is set as the threshold, and pixels with values ​​greater than the threshold are retained as saliency pixels.

5. The audio keyframe extraction method according to claim 1, characterized in that: In step S5, the threshold value is the sum of the saliency of pixels with frequencies above 2KHz, and the sum reaches one-third of the total saliency of pixels.

Images