Visual and auditory integrated perception coal mine overhead rail type inspection robot

Abstract

The invention relates to a coal mine overhead rail type inspection robot with visual sense and auditory sense integrated perception. The inspection robot slides on an I-shaped steel overhead rail. A base is mounted at the bottom of the inspection robot, and a primary sensing microphone array is mounted on the side wall of the base around the periphery; the two ends of the bottom surface of the base are rotationally connected with turntables; two audio-visual integration perception devices are symmetrically installed at the front end and the rear end of the bottom face of the rotating disc. Through traction motors arranged in multiple directions, a traction line is utilized to drive an eyeball shell to move in the pitching, rolling and yawing directions, so that a camera and a secondary sensing microphone array arranged on the eyeball shell can adaptively adjust the probing angle; meanwhile, a visual and auditory system arranged in the robot can obtain underground voiceprint information through a carried voiceprint feature extraction method, and the underground fault type can be judged; therefore, the invention plays a role in improving the sensing sensitivity and the sensing accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of coal mine inspection, and particularly to a monorail type inspection robot for coal mines with integrated visual and auditory perception. Background Art

[0002] There are risks such as equipment failures, gas accumulation, and roof collapses in coal mines, and regular inspections are required to ensure the normal production of coal mines and the safety of workers. However, traditional manual inspection methods have many deficiencies, such as low efficiency and a relatively high risk of exposing personnel to dangerous environments. With the rapid development of artificial intelligence, robotics, and sensing technologies, the intelligentization of coal mines has gradually become an important trend in the industry's development. Developing intelligent robot inspection technology for coal mines can improve the work efficiency of coal mine inspections and the operational safety of miners.

[0003] In the prior art, a Chinese patent with the publication number CN209682199U discloses a monorail type inspection robot for coal mine monitoring, which is used to monitor the state of underground belt conveyors. The robot disclosed in this patent replaces manual work to complete coal mine inspection, reducing the labor intensity of workers and the safety risks faced during manual inspections. However, it can only perform acoustic localization on the abnormal sound location of the belt conveyor by installing commercially available sound collection devices. In addition, the robot cannot resist strong background noise interference in coal mines for the collected sounds, and thus cannot extract characteristic information, making it difficult to identify abnormal sounds through sound.

[0004] At the same time, a Chinese invention patent with the application number 202310391332.6 also discloses a monorail type inspection robot. This patent uses a temperature sensor and a camera installed on a turntable to inspect for abnormalities in coal mines. However, it cannot identify internal mechanical failures of underground equipment without external deformation.

[0005] Moreover, due to the complex environment in coal mines, identifying corresponding acoustic signals is an important feature for judging equipment failures and environmental abnormalities. Such as abnormal sounds in the early stage of mine equipment failures (bearing abnormal sounds, belt tearing sounds, motor coil loosening sounds, etc.), underground environmental disaster sounds (gas / gas leakage sounds, water flow abnormal sounds in the precursor of a water inrush accident, roof pressure release sounds, etc.), and miners' calls for help.

[0006] The above-mentioned publicly available inspection robots mainly rely on vision and temperature sensors and cannot identify the above-mentioned sound scenarios. However, simply adding a microphone to the inspection robot to collect acoustic signals can detect abnormal sounds, but cannot achieve acoustic localization, and it is even more difficult to identify the cause of the abnormal sounds.

[0007] In addition, the complex downhole environmental noise seriously interferes with the analysis of the concerned voiceprint features, increasing the monitoring difficulty of acoustic events during downhole inspection tours.

[0008] To sum up, the existing coal mine inspection robots have the following three deficiencies in the abnormal inspection of downhole equipment:

[0009] First, the acquisition and analysis of images and sounds on existing inspection robots are either not independent of each other or only include the acquisition and analysis of images. And those equipped with sound acquisition only use ordinary microphones. For inspection robots without sound acquisition function, that is, mainly relying on image acquisition to complete the inspection, they cannot identify early external faults of mine equipment with small deformations and internal faults of mine equipment without external deformations, nor can they identify equipment faults with blocked vision. For inspection robots with a single additional microphone, although in theory they can detect abnormal sounds emitted by internal equipment faults, they cannot locate the position of the abnormal sound source; there are many downhole equipment, so fault identification cannot be achieved.

[0010] Second, the image acquisition of existing inspection robots is all completed through cameras installed on the pan-tilt. That is, by controlling the rotation of the pan-tilt, the field of view of the camera is controlled to detect equipment at different positions in the mine. However, each time the pan-tilt rotates, it cannot quickly respond and turn to the position direction where the coal mine equipment is in an abnormal state, resulting in missed visual inspections. In addition, the cameras on the pan-tilt point in the same direction at the same time, with a small detection angle, further increasing the possibility of missed detections of sudden equipment abnormalities.

[0011] Third, the acoustic environment in the coal mine is complex, and the sounds emitted by early faults of downhole equipment are easily interfered by environmental noise. In the process of sound acquisition by existing coal mine inspection robots, the problem of this strong background noise is not considered, and thus the voiceprint features of early faults of mine equipment cannot be effectively extracted. Summary of the Invention

[0012] A visual and auditory integrated perception coal mine suspended rail type inspection robot provided by the present invention drives the eyeball shell to move in the pitch, roll, and yaw directions by using a traction line through traction motors arranged in multiple directions, so that the cameras and secondary perception microphone arrays arranged on the eyeball shell can adaptively adjust the detection angles; at the same time, the visual and auditory system arranged inside the robot can obtain the voiceprint information in the mine through the carried voiceprint feature extraction method and can judge the types of faults in the mine; thereby, the present invention plays a role in improving the perception sensitivity and perception accuracy.

[0013] A coal mine overhead rail inspection robot with visual and auditory integration perception, which slides on an I-beam overhead rail; a base is installed at the bottom of the inspection robot, and a primary perception microphone array is installed around the side wall of the base; at both ends of the bottom surface of the base, turntables are rotatably connected; at the front and rear ends of the bottom surface of the turntable, two audio-visual integration perception devices are symmetrically installed;

[0014] Among them, the audio-visual integration perception device includes an eyeball-shaped perception component, an orbital-shaped fixing component, and a driving component;

[0015] The eyeball-shaped perception component includes an eyeball shell, a camera vertically installed in the middle of the eyeball shell, and a secondary perception microphone array arranged on the surface of the eyeball shell;

[0016] The orbital-shaped fixing component includes a U-shaped frame vertically fixed at a corner of the bottom surface of the turntable, an external pitching ring rotatably connected inside the U-shaped frame, and a rear support ring arranged in a semi-circular shape on the back of the eyeball shell. Moreover, both ends of the rear support ring are fixedly connected to the external pitching ring, and the eyeball shell is arranged inside the spherical space formed by the external pitching ring and the rear support ring;

[0017] Among them, the eyeball shell is connected to the external pitching ring and the rear support ring through a rolling support; the rolling support includes a central frame, a rolling shaft, a pitching shaft, and a yaw shaft; the central frame is U-shaped and is arranged at the center of the eyeball shell; the rolling shaft penetrates the eyeball shell along the X-axis direction, and its extending end is fixedly connected to the inner wall of the rear support ring, and the other end is rotatably connected to the central frame; the pitching shaft penetrates the eyeball shell along the Y-axis direction, and its extending end is fixedly connected to the inner wall of the external pitching ring; the yaw shaft is arranged inside the eyeball shell along the Z-axis direction, one end of which is connected to the inner wall of the eyeball shell, and the other end is rotatably connected to the central frame;

[0018] The driving component includes a traction motor and a wire winding wheel; three groups of the traction motors are installed at an angle of 90 degrees at the other three corners of the bottom surface of the turntable except for the U-shaped frame; the output end of the traction motor is coaxially fixed with the wire winding wheel;

[0019] Pitching traction wires penetrate through the top and bottom of the external pitching ring, rolling traction wires are arranged at the top and bottom of the eyeball shell, and yaw traction wires are arranged on both sides of the eyeball shell; the pitching traction wires, rolling traction wires, and yaw traction wires are respectively wound on different wire winding wheels.

[0020] Preferably, the power system of the inspection robot includes a power box and a friction wheel; among them, the power box is located at the top of the inspection robot, and the friction wheel is fixed to the output shaft above it. The friction wheel is used to drive the inspection robot to move on the I-shaped steel rail; a dust cover is installed on the outside of the inspection robot.

[0021] Preferably, a plurality of L-shaped brackets are annularly arranged around the turntable at the base, and rollers horizontally placed are rotatably connected to the inner walls of the L-shaped brackets; an annular groove is formed in the middle of the turntable, and the short sides of the L-shaped brackets are inserted into the annular groove for hanging and installing the turntable.

[0022] A rotating shaft is disposed through the central axis of the turntable, and a large gear is connected to the end thereof passing through the turntable; a displacement motor is installed on the bottom wall of the turntable, and the output shaft of the displacement motor penetrates the turntable and a small gear is connected to the end thereof; the large gear and the small gear are meshed.

[0023] Preferably, rollers are installed at the corners of the top of the turntable, and an annular guiding chute is provided at the bottom of the base, and the rollers are used for sliding in the guiding chute.

[0024] Preferably, the secondary sensing microphone array provided on the surface of the eyeball shell adopts any one of the layout modes of circular, spiral or spherical layout.

[0025] Preferably, the primary sensing microphone array adopts a combination of linear array arrangement and circular arrangement.

[0026] Preferably, the inspection robot is provided with a vision and hearing system, and the vision and hearing system is equipped with a voiceprint feature extraction method based on machine hearing, which includes the following steps:

[0027] S1: Extract the operation sounds of the underground equipment through the primary sensing microphone array and the secondary sensing microphone array in the audio-visual integration sensing module, and preprocess the extracted sound signals;

[0028] S2: Use a machine hearing model to extract the auditory voiceprint features of the preprocessed sound signals, and summarize to obtain an auditory spectrum;

[0029] S3: Use an auditory saliency model to extract the time correlation features and frequency correlation features from the auditory spectrum, perform saliency calculation and superposition through the center-surround difference algorithm to obtain a final auditory feature map.

[0030] Preferably, the above step S1 is specifically:

[0031] The signal X = {x1, x2,..., x i ,..., x N} is collected by the microphone array, where n represents the nth data point, N represents the Nth group of data, and i represents the sound signals collected by the i channels of the microphone array;

[0032] Among them, each group of sound signals needs to be preprocessed; the preprocessing includes the following steps:

[0033] a. Sample and clip the original sound signal imported by the microphone array to obtain n data points;

[0034] b. Perform band-pass filtering analysis on each group of processed sound signals to obtain N groups of preprocessed sound data.

[0035] Preferably, step S2 specifically includes the following steps:

[0036] S2.1: Input the preprocessed sound signal into the middle ear model to finally obtain the sound pressure signal; among them, the middle ear model is implemented by a first-order band-pass filter that conforms to the characteristics of the human middle ear (0.6 kHz - 4 kHz);

[0037] S2.2: The transmission line cochlea model discretizes the cochlear basilar membrane into n segments. The series admittance and parallel admittance of each segment of the basilar membrane can be expressed by the following formula:

[0038]

[0039] In the formula, converting the complex variable s can reduce one variable, s = jω / ω n ;

[0040] In the formula, j represents the imaginary unit, ω represents the angular frequency of the input sound pressure signal, ω n represents the natural frequency of the nth segment of the basilar membrane, and are constants; δ n (t) is the damping parameter. When δ n (t) < 0, the cochlea is an active model; ρ n (t) is the feedback strength, and ψ n (t) is the dimensionless delay;

[0041] Let the sound pressure at the input end be P in , and the sound pressure on the nth segment of the basilar membrane be P n . In the Laplace transform domain, the sound pressure response input to each segment of the basilar membrane can be obtained by the following formula:

[0042] P0(s) = P in (s)

[0043]

[0044] The displacement Y n (s) of each segment of the basilar membrane can be expressed by the following formula:

[0045] Y n (s) = H n (s)P n (s)

[0046] Among them, H n (s) is the cochlear transfer function, which is jointly determined by the series admittance and the parallel admittance :

[0047]

[0048] Then the vibration velocity v n (t) can be obtained by the inverse Laplace transform:

[0049] v n (t) = L -1 {sY n (s) - y(0)}

[0050] where y(0) is the initial displacement;

[0051] S2.3: Through the inner hair cell transduction model, the vibration velocity of the basilar membrane can be calculated by the following formula to obtain the membrane potential during the sinusoidal deflection of the ciliary bundle:

[0052]

[0053] Among them, C m is the capacitance of the inner hair cell, V m is the membrane potential of the inner hair cell, I MET is the mechanoelectrical transduction current (MET), and are the fast and slow currents of the outward K + , G is a constant gain value, v(t) is the vibration velocity on the segment of the basilar membrane, x0 = 35nm, s = 16nm, the maximum transduction gain G MET,max = 30nS, the endocochlear potential EP = 90mV, V 0.5 = -31mV, a k is a constant, the maximum transduction gain G K,f / s = 230nS, V K,f = -71mv, V K,s = -78mV;

[0054] S2.4: Through the nerve synapse exocytosis model, the electrical signal of the inner hair cell derived from the inner hair cell transduction model is converted into a nerve signal. The calculated exocytosis rate k(t) of the inner hair cell ribbon synapse per unit time can be calculated by the following formula:

[0055] I Ca = g Ca (V m - E ca )

[0056] k(t) = zmax(I Ca-I Ca,th ,0)

[0057] where I Ca is the whole-cell Ca 2+ channel current, g Ca is the Ca 2+ channel voltage-dependent conductivity, V m is the membrane potential of the inner hair cell, E Ca is the reversal potential, I Ca,th is the linear offset related to the single synaptic exocytosis rate, and z is a constant;

[0058] S2.5: Substitute the inner hair cell ribbon synapse exocytosis rate k(t) calculated in S2.4 into the three-layer diffusion model, and the firing probability P of the nerve fiber is obtained from the following formula Firing :

[0059]

[0060] where λ max is the maximum firing rate, k0 is the half-saturation constant, and Δt is the time window length; according to the nerve fiber firing probability, the firing rate τ Firing (t) of a single nerve fiber can be finally obtained, that is, calculated as follows:

[0061]

[0062] where P Rel is a constant representing the probability that at least one vesicle is released, and t a is the time constant;

[0063] By summing the responses of nerve fibers with different firing rates at each characteristic frequency, the auditory brainstem response spectrum, that is, the auditory voiceprint feature, can be obtained.

[0064] Preferably, the above step S3 specifically includes the following steps:

[0065] A. Filter the auditory spectrum output obtained in the above step S2.5 multiple times with different scales using a 2D Gaussian filter to obtain sound intensity features;

[0066] B. Then filter the obtained sound intensity features multiple times with different scales using a 2D Gabor filter to extract time correlation features and frequency correlation features, where the Gabor filter used for the frequency correlation features is set with a phase θ = 0°, and the Gabor filter used for the time correlation features has a phase θ = 90°;

[0067] C. Superimpose the finally obtained multi-scale features of sound intensity, time correlation, and frequency correlation to obtain the final auditory feature map, and judge the category of underground environmental disasters based on the auditory feature map.

[0068] Advantages of the present invention compared with the prior art:

[0069] First, the inspection robot proposed by the present invention realizes the preliminary positioning of the abnormal sound position of the underground device through the microphone array arranged around the base; controls the eyeball to quickly point to the abnormal sound position, and then accurately locates the abnormal sound position again through the sound field distribution measured by the microphone array in the audiovisual integration perception module on the eyeball, and combines the camera in the audiovisual integration perception module to identify specific underground equipment. Therefore, the inspection robot of the present invention can identify equipment failures without external shape changes underground and equipment failures with blocked vision through accurate sound positioning of abnormal sounds and visual recognition integration perception.

[0070] Second, the designed audiovisual integration perception module of the present invention is placed in the eyeball towed by multiple wires, can swing to the specified orientation in one step, has a fast response speed, and reduces the possibility of missing detection of sudden abnormal sounds of equipment. In addition, the front and rear audiovisual integration perception modules of the designed inspection robot are independent of each other, one is responsible for the first half and the other is responsible for the second half, increasing the detection angle of the inspection and further reducing the possibility of missing detection of sudden abnormal sounds of equipment.

[0071] Third, the audiovisual integration perception module of the inspection robot proposed by the present invention introduces an auditory calculation module that simulates the human ear auditory system, which more effectively improves the sound feature recognition ability of the inspection robot for early equipment failures under strong background noise underground. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a schematic structural diagram of the present invention;

[0073] Figure 2 is a structural diagram of the robot power system of the present invention;

[0074] Figure 3 is a bottom structural diagram of the base of the present invention;

[0075] Figure 4 is a connection structural diagram of the turntable of the present invention;

[0076] Figure 5 is an assembly structural diagram of the eyeball and the eye socket of the present invention;

[0077] Figure 6 is a rear view structural diagram of the eyeball of the present invention;

[0078] Figure 7 is a partial internal structural diagram of the eyeball shell of the present invention;

[0079] Figure 8 is a structural diagram of the short L-shaped bracket and the roller of the present invention;

[0080] Figure 9Structural diagram of the long L-shaped bracket and roller of the present invention;

[0081] Figure 10 Schematic diagram of the circumferential arrangement of the microphone array;

[0082] Figure 11 Schematic diagram of the spiral arrangement of the microphone array;

[0083] Figure 12 Schematic diagram of the spherical arrangement of the microphone array;

[0084] Figure 13 Flowchart of the voiceprint feature extraction method based on machine hearing carried by the intelligent auditory system used in the present invention.

[0085] In the figure: 1, I-beam rail; 2, inspection robot; 201, dust cover; 202, power box; 203, friction wheel; 3, base; 4, eyeball-shaped sensing component; 5, orbital-shaped fixing component; 6, primary sensing microphone array; 7, slewing motor; 8, wire winding wheel; 9, traction motor; 10, long L-shaped bracket; 11, short L-shaped bracket; 12, turntable; 13, camera; 14, secondary sensing microphone array; 15, rotating shaft; 16, U-shaped frame; 17, large gear; 18, small gear; 19, toothed disc; 20, roller; 21, forward mounting limit block; 22, reverse mounting limit block; 23, external pitch ring; 24, rear support ring; 25, pitch traction wire; 26, yaw traction wire; 27, rolling traction wire; 28, through hole; 29, pitch axis; 30, avoidance groove; 31, rolling axis; 32, center frame; 33, yaw axis; 34, eyeball shell; 35, roller; 36, circumferential microphone array; 37, spiral microphone array; 38, spherical microphone array. Specific embodiments

[0086] Combine the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0087] Refer to Figures 1 to 12, this application provides a coal mine overhead rail inspection robot 2 with visual and auditory integrated perception, which slides on the I-beam overhead rail; a base 3 is installed at the bottom of the inspection robot 2, and the top of the base 3 is connected to the bottom end of the power box 202; a primary perception microphone array 6 is installed around the side wall of the base 3 for primary positioning and monitoring of omnidirectional sound waves underground; at both ends of the bottom surface of the base 3, turntables 12 are rotatably connected, and a plurality of small through holes 28 are opened on the turntables 12 for fixing the eyeball-shaped perception component 4 and other parts; at the front and rear ends of the bottom surface of the turntable 12, two audio-visual integration perception devices are symmetrically installed. The two audio-visual integration modules are arranged symmetrically front and back. The symmetrical distribution can promote the dynamic balance during the rotational movement of the mechanism and reduce the impact of unbalanced centripetal force on the mechanism; among them, the audio-visual integration perception device includes an eyeball-shaped perception component 4, an orbital-shaped fixing component 5, and a driving component.

[0088] Referring to Figures 3 to 7 , the eyeball-shaped perception component 4 includes an eyeball shell 34 that can swing up and down, a camera 13 vertically installed in the middle of the eyeball shell 34, and a secondary perception microphone array 14 arranged on the surface of the eyeball shell 34; the eye-frame-shaped fixing component 5 includes a U-shaped frame 16 vertically fixed at a corner of the bottom surface of the turntable 12, an external pitch ring 23 rotatably connected inside the U-shaped frame 16, and a rear support ring 24 arranged in a semi-circular shape on the back of the eyeball shell 34. Moreover, both ends of the rear support ring 24 are fixed to the external pitch ring 23, and the eyeball shell 34 is arranged inside the spherical space formed by the external pitch ring 23 and the rear support ring 24; among them, the U-shaped frame 16 is composed of two vertical brackets, and a vertical bracket is also installed on the back side of the eyeball shell 34 where the camera 13 is located. These three vertical brackets are used to fix the traction motor 9 on the turntable 12.

[0089] Among them, the eyeball shell 34 is connected to the external pitch ring 23 and the rear support ring 24 through rolling brackets; the rolling brackets include a central frame 32, a rolling shaft 31, a pitch shaft, and a yaw shaft 33; the central frame 32 is U-shaped and is arranged at the center of the eyeball shell 34; the rolling shaft 31 penetrates the eyeball shell 34 along the X-axis direction, and its extended end is fixedly connected to the inner wall of the rear support ring 24, and the other end is rotatably connected to the central frame 32; the pitch shaft penetrates the eyeball shell 34 along the Y-axis direction, and its extended end is fixedly connected to the inner wall of the external pitch ring 23; the yaw shaft 33 is arranged inside the eyeball shell 34 along the Z-axis direction, one end of which is connected to the inner wall of the eyeball shell 34, and the other end is rotatably connected to the central frame 32. In addition, an avoidance groove 30 is provided on the back of the eyeball shell 34, and the avoidance groove 30 matches the rolling shaft 31 to increase the rotational flexibility of the eyeball shell 34.

[0090] Referring to Figure 3, the driving assembly includes a traction motor 9 and a wire winding wheel 8; at the other three corners of the bottom surface of the turntable 12 except for the U-shaped frame 16, three groups of the traction motors 9 are installed at an angle of 90 degrees; each group of traction motors 9 is a pair of stepper motors for controlling the rotation of the eyeball shell 34 in the same direction; the output end of the traction motor 9 is coaxially fixed with the wire winding wheel 8; the top and bottom of the external pitch ring 23 are both provided with pitch traction wires 25, the top and bottom of the eyeball shell 34 are both provided with rolling traction wires 27, and yaw traction wires 26 are provided on both sides of the eyeball shell 34; the pitch traction wires 25, the rolling traction wires 27 and the yaw traction wires 26 are respectively wound on different wire winding wheels 8.

[0091] Specifically, referring to Figure 5 , the pitch traction wires 25 are arranged at the through holes 28 symmetrically located at the upper and lower ends of the external pitch ring 23 and the rear support ring 24 for pitch movement, and these are used as pitch traction points. When the traction motor 9 rotates, by simulating the tendon movement mode of a biological eyeball, the traction wires on the upper and lower sides are respectively tightened and relaxed, and thus the pitch movement can be completed. The pitch axis 29 is located on the outermost side of the entire eyeball mechanism, so it will drive all parts to move together;

[0092] The rolling traction wires 27 are arranged at the through holes 28 reserved at the upper and lower ends of the eyeball shell 34, and these are used as rolling traction points. When controlling the rotation of the traction motor 9, the upper and lower traction wires are respectively contracted and relaxed. The traction wires pass through the side plates of the eye socket, which determines the traction direction. The eyeball shell 34 realizes rolling movement under the action of the traction wires. When the eyeball shell 34 rolls, the moving parts include the central frame 32, the eyeball shell 34 and the camera 13.

[0093] The yaw traction wires 26 are arranged at the through holes 28 reserved on both sides of the eyeball shell 34, and these are used as yaw traction points. When implementing the yaw movement, the wire winding wheel 8 is driven by the traction motor 9, simulating the tendon traction of a biological eyeball, respectively tightening one of the traction wires on the left and right sides and relaxing the other, and completing the yaw movement of the eyeball shell 34. When performing the yaw movement, the moving parts include the eyeball shell 34 and the central frame 32.

[0094] Referring to Figure 2 , the power system of the inspection robot 2 includes a power box 202 and a friction wheel 203; among them, the power box 202 is located at the top of the inspection robot 2, and the inspection robot 2 provides forward power and electricity through the power box 202; the friction wheel 203 is fixed to the output shaft above it, and the friction wheel 203 is used to drive the inspection robot 2 to move on the I-shaped steel rail 1; a dust cover 201 is installed on the outside of the inspection robot 2 to protect the transmission device from being damaged by falling stones or coal blocks above.

[0095] Referring to Figure 3 、 8At positions 3 and 9, multiple L-shaped brackets are annularly arranged around the turntable 12 on the base 3 to prevent the turntable 12 from falling. According to different installation positions, the L-shaped long brackets 10 are installed on the side wall of the base 3, and the L-shaped short brackets 11 are installed on the bottom wall of the base 3. Both are fixed with bolts. A horizontally placed roller 35 is rotatably connected to the inner wall of the L-shaped bracket. An annular groove is formed in the middle of the turntable 12, and the short side of the L-shaped bracket is inserted into the annular groove to realize the hanging installation of the turntable 12.

[0096] Refer to Figure 3 and 4 , a rotating shaft 15 is penetrated through the central axis of the turntable 12, and a large gear 17 is connected to the end of the rotating shaft 15 passing through the turntable 12. A displacement motor 7 is installed on the bottom wall of the turntable 12. The output shaft of the displacement motor 7 penetrates the turntable 12 and a small gear 18 is connected to the end. The large gear 17 and the small gear 18 are meshed. Specifically, at the position of the turntable 12 corresponding directly below the displacement motor 7 fixed on the back of the eyeball perception component, a relatively large round hole is opened to facilitate the installation of the small gear 18 for meshing with the large gear 17 on the bottom gear disk 19 and realizing the rotation of the turntable 12.

[0097] Refer to Figure 4 , rollers 20 are installed at the corners of the top of the turntable 12, and an annular guiding chute is provided at the bottom of the base 3. The rollers 20 are used to slide in the guiding chute. Among them, the rollers 20 serve as supports to bear a relatively large force in the vertical direction and can reduce the sliding friction of the turntable 12.

[0098] Refer to Figures 10 to 12 , the secondary perception microphone array 14 provided on the surface of the eyeball shell 34 adopts any one of the layout methods of circular layout, spiral layout or spherical layout, namely Figure 10 the circular microphone array 36 in Figure 11 the spiral microphone array 37 in Figure 12 the spherical microphone array 38 in

[0099] Refer to Figure 3 and 4 , the primary perception microphone array 6 adopts a combination of linear array arrangement and circular arrangement.

[0100] Refer to Figure 3 and 4 , a positive installation limit block 21 and a reverse installation limit block 22 are installed on the upper part of the turntable, which are used to cooperate with the base to complete the positive fixation of the turntable.

[0101] Refer to Figure 13, the inspection robot provided by this application is equipped with a vision and hearing system, and the vision and hearing system is equipped with a voiceprint feature extraction method based on machine hearing, which includes the following steps:

[0102] S1: Extract the operating sounds of underground equipment through the primary perception microphone array and the secondary perception microphone array in the audio-visual integration perception module, and preprocess the extracted sound signals;

[0103] S2: Use the machine hearing model to extract the auditory voiceprint features of the preprocessed sound signals, and summarize them to obtain an auditory spectrum;

[0104] S3: Use the auditory saliency model to extract the time correlation features and frequency correlation features from the auditory spectrum, perform saliency calculation and superposition through the center-surround difference algorithm to obtain the final auditory feature map.

[0105] Preferably, the above step S1 is specifically as follows:

[0106] The signal X = {x1, x2,..., x i ,..., x N} is collected by the microphone array, where n represents the nth data point, N represents the Nth group of data, and i represents the sound signals collected by the i channels of the microphone array;

[0107] Among them, each group of sound signals needs to be preprocessed; the preprocessing includes the following steps:

[0108] a. Sample and clip the original sound signals imported by the microphone array to obtain n data points;

[0109] b. Perform band-pass filtering analysis on each group of processed sound signals to obtain N groups of preprocessed sound data.

[0110] Preferably, the above step S2 specifically includes the following steps:

[0111] S2.1: Input the preprocessed sound signals into the middle ear model to finally obtain sound pressure signals; among them, the middle ear model is implemented by a first-order band-pass filter that conforms to the characteristics of the human middle ear (0.6 kHz - 4 kHz);

[0112] S2.2: The transmission line cochlea model discretizes the cochlear basilar membrane into n segments, and the series admittance and parallel admittance of each segment of the basilar membrane can be expressed according to the following formula:

[0113]

[0114] In the formula, converting the complex variable s can reduce one variable, s = jω / ω n ;

[0115] where \(j\) represents the imaginary unit, \(\omega\) represents the angular frequency of the input sound pressure signal, \(\omega\) n represents the natural frequency of the \(n\)th segment of the basilar membrane, and are constants; \(\delta\) n (t) is the damping parameter. When \(\delta\) n (t) < 0, the cochlea is an active model; \(\rho\) n (t) is the feedback strength, and \(\psi\) n (t) is the dimensionless delay;

[0116] Let the sound pressure at the input end be \(P\) in , and the sound pressure on the \(n\)th segment of the basilar membrane be \(P\) n . In the Laplace transform domain, the sound pressure response input to each segment of the basilar membrane can be obtained from the following formula:

[0117] \(P_0(s)=P\) in (s)

[0118]

[0119] The displacement \(Y\) n (s) on each segment of the basilar membrane can be expressed by the following formula:

[0120] \(Y\) n (s)=H n (s)P n (s)

[0121] where \(H\) n (s) is the cochlear transfer function, which is jointly determined by the series admittance and the parallel admittance :

[0122]

[0123] Then the vibration velocity \(v\) n (t) on the basilar membrane can be obtained by the inverse Laplace transform:

[0124] \(v\) n (t)=L -1 \{sY n (s)-y(0)\}

[0125] where \(y(0)\) is the initial displacement;

[0126] S2.3: Through the inner hair cell transduction model, the vibration velocity of the basilar membrane can be calculated by the following formula to obtain the membrane potential during the sinusoidal deflection of the ciliary bundle:

[0127]

[0128] where \(C\) mis the capacitance of the inner hair cell, V m is the membrane potential of the inner hair cell, I MET is the mechanoelectrical transduction current (MET), and is the fast and slow current of the outward K + with G being a constant gain value, v(t) being the vibration velocity on the segment of the basilar membrane, x0 = 35 nm, s = 16 nm, the maximum transduction gain G MET,max = 30 nS, the endocochlear potential EP = 90 mV, V 0.5 = -31 mV, a k is a constant, the maximum transduction gain G K,f / s = 230 nS, V K,f = -71 mv, V K,s = -78 mV;

[0129] S2.4: Convert the inner hair cell electrical signal derived from the inner hair cell transduction model into a neural signal through the synaptic exocytosis model. The exocytosis rate k(t) of the inner hair cell ribbon synapse per unit time calculated can be calculated by the following formula:

[0130] I Ca = g Ca (V m - E ca )

[0131] k(t) = zmax(I Ca - I Ca,th , 0)

[0132] where I Ca is the whole-cell Ca 2+ channel current, g Ca is the Ca 2+ channel voltage-related conductivity, V m is the membrane potential of the inner hair cell, E Ca is the reversal potential, I Ca,th is the linear offset related to the single synaptic exocytosis rate, and z is a constant;

[0133] S2.5: Substitute the exocytosis rate k(t) of the inner hair cell ribbon synapse calculated in S2.4 into the three-layer diffusion model to obtain the firing probability P Firing of the nerve fiber as follows:

[0134]

[0135] where λ max is the maximum firing rate, k0 is the half-saturation constant, and Δt is the time window length; According to the firing probability of the nerve fiber, the firing rate τ Firing (t) of a single nerve fiber can be finally obtained, that is, calculated by the following formula:

[0136]

[0137] Among them, P Rel is a constant representing the probability that at least one vesicle is released, and t a is the time constant;

[0138] The auditory brainstem response spectrum, i.e., the auditory voiceprint feature, can be obtained by summing the responses of nerve fibers with different emission rates at each characteristic frequency.

[0139] Preferably, the above step S3 specifically includes the following steps:

[0140] A. The auditory spectrum output obtained in the above step S2.5 is filtered multiple times at different scales using a 2D Gaussian filter to obtain sound intensity features;

[0141] B. Then, the obtained sound intensity features are filtered multiple times at different scales using a 2D Gabor filter to extract time correlation features and frequency correlation features. For the frequency correlation features, the Gabor filter is set with a phase θ = 0°, and for the time correlation features, the Gabor filter has a phase θ = 90°;

[0142] C. The multi-scale features of sound intensity, time correlation, and frequency correlation finally obtained are superimposed to obtain the final auditory feature map, and the type of underground environmental disaster is judged based on the auditory feature map.

[0143] Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A coal mine overhead rail inspection robot (2) for visual and auditory integrated perception, which slides on an I-beam overhead rail; characterized in that: A base (3) is installed at the bottom of the inspection robot (2), and a primary sensing microphone array (6) is installed around the side wall of the base (3); two turntables (12) are rotatably connected to both ends of the bottom surface of the base (3); two audio-visual integration sensing devices are symmetrically installed at the front and rear ends of the bottom surface of the turntable (12). Among them, the audio-visual integration sensing device includes an eyeball-shaped sensing component (4), an orbital-shaped fixing component (5) and a driving component. The eyeball-shaped sensing component (4) includes an eyeball shell (34), a camera (13) vertically installed in the middle of the eyeball shell (34), and a secondary sensing microphone array (14) arranged on the surface of the eyeball shell (34). The orbital-shaped fixing component (5) includes a U-shaped frame (16) vertically fixed at a corner of the bottom surface of the turntable (12), an external pitching ring (23) rotatably connected inside the U-shaped frame (16), and a rear support ring (24) arranged in a semi-circular shape on the back of the eyeball shell (34). Moreover, both ends of the rear support ring (24) are fixedly connected to the external pitching ring (23), and the eyeball shell (34) is arranged inside the spherical space formed by the external pitching ring (23) and the rear support ring (24). Among them, the eyeball shell (34) is connected to the external pitching ring (23) and the rear support ring (24) through rolling brackets; the rolling brackets include a central frame (32), a rolling shaft (31), a pitching shaft and a yaw shaft (33); the central frame (32) is U-shaped and is arranged at the center of the eyeball shell (34); the rolling shaft (31) penetrates the eyeball shell (34) along the X-axis direction, and its extended end is fixedly connected to the inner wall of the rear support ring (24), and the other end is rotatably connected to the central frame (32); the pitching shaft penetrates the eyeball shell (34) along the Y-axis direction, and its extended end is fixedly connected to the inner wall of the external pitching ring (23); the yaw shaft (33) is arranged inside the eyeball shell (34) along the Z-axis direction, one end of which is connected to the inner wall of the eyeball shell (34), and the other end is rotatably connected to the central frame (32). The driving component includes a traction motor (9) and a wire winding wheel (8); three groups of the traction motors (9) are installed at an angle of 90 degrees at the other three corners of the bottom surface of the turntable (12) except the U-shaped frame (16); the output end of the traction motor (9) is coaxially fixed with the wire winding wheel (8). Pitching traction wires (25) penetrate through the top and bottom of the external pitching ring (23), rolling traction wires (27) are arranged at the top and bottom of the eyeball shell (34), and yaw traction wires (26) are arranged on both sides of the eyeball shell (34); the pitching traction wires (25), the rolling traction wires (27) and the yaw traction wires (26) are respectively wound on different wire winding wheels (8).

2. The coal mine suspension rail type inspection robot (2) with visual and auditory integration sensing according to claim 1, characterized in that: The power system of the inspection robot (2) includes a power box (202) and a friction wheel (203); wherein, the power box (202) is located at the top of the inspection robot (2), and the friction wheel (203) is fixed to the output shaft above it. The friction wheel (203) is used to drive the inspection robot (2) to move on the I-shaped steel rail (1); a dust cover (201) is installed on the outer side of the inspection robot (2).

3. The coal mine suspended rail type inspection robot (2) with visual and auditory integrated perception according to claim 1, characterized in that: A plurality of L-shaped brackets are annularly arranged around the turntable (12) at the base (3) with the turntable (12) as the central axis. A horizontally placed roller (35) is rotatably connected to the inner wall of the L-shaped bracket; an annular groove is formed in the middle of the turntable (12), and the short side of the L-shaped bracket is inserted into the annular groove for realizing the hanging installation of the turntable (12); A rotating shaft (15) is penetrated through the central axis of the turntable (12), and a large gear (17) is connected to the end thereof passing through the turntable (12); a displacement motor (7) is installed on the bottom wall of the turntable (12), and the output shaft of the displacement motor (7) penetrates through the turntable (12) and is connected to a small gear (18) at the end; the large gear (17) and the small gear (18) are meshed with each other.

4. The coal mine suspended rail type inspection robot (2) with visual and auditory integrated perception according to claim 3, characterized in that: Rollers (20) are installed at the corners of the top of the turntable (12), and an annular guiding chute is provided at the bottom of the base (3). The rollers (20) are used to slide in the guiding chute.

5. The coal mine suspended rail type inspection robot (2) with visual and auditory integrated perception according to claim 1, characterized in that: The secondary perception microphone array (14) provided on the surface of the eyeball shell (34) adopts any one of the layout modes of circular layout, spiral layout or spherical layout.

6. The coal mine suspended rail type inspection robot (2) with visual and auditory integrated perception according to claim 1, characterized in that: The primary perception microphone array (6) adopts a combination of linear array arrangement and circular arrangement.

7. The coal mine suspended rail type inspection robot with visual and auditory integrated perception according to claim 1, characterized in that: The inspection robot is provided with a visual and auditory system, and the visual and auditory system is equipped with a voiceprint feature extraction method based on machine audition, which includes the following steps: S1: Extract the running sounds of underground equipment through the primary perception microphone array and the secondary perception microphone array in the audio-visual integrated perception module, and preprocess the extracted sound signals; S2: Use a machine audition model to extract the auditory voiceprint features of the preprocessed sound signals, and summarize to obtain an auditory spectrum; S3: Use an auditory saliency model to extract the time correlation features and frequency correlation features from the auditory spectrum, and perform saliency calculation and superposition through the center-surround difference algorithm to obtain a final auditory feature map.

8. The visual and auditory integrated perception coal mine overhead rail inspection robot according to claim 7, characterized in that, The above step S1 is specifically: The signal X = {x1, x2,..., x i ,..., x N} is obtained by microphone array acquisition, where n represents the nth data point, N represents the Nth group of data, and i represents the sound signal collected by the ith channel of the microphone array; Wherein, each group of sound signals needs to be preprocessed; the preprocessing includes the following steps: a. Sample and clip the original sound signals imported by the microphone array to obtain n data points; b. Perform band-pass filtering analysis on each group of processed sound signals to obtain N groups of preprocessed sound data.

9. The visual and auditory integrated perception coal mine overhead rail inspection robot according to claim 8, characterized in that, The above step S2 specifically includes the following steps: S2.1: Input the preprocessed sound signals into the middle ear model to finally obtain sound pressure signals; among them, the middle ear model is implemented by a first-order band-pass filter that conforms to the characteristics of the human middle ear (0.6 kHz to 4 kHz); S2.2: The transmission line cochlea model discretizes the cochlear basilar membrane into n segments. The series admittance and parallel admittance of each segment of the basilar membrane can be expressed by the following formula: In the formula, converting the complex variable s can reduce one variable, where s = jω / ω n ; Where \(j\) represents the imaginary unit, \(\omega\) represents the angular frequency of the input sound pressure signal, \(\omega\) n represents the natural frequency of the \(n\)-th segment of the basilar membrane, and are constants; \(\delta\) n (t) is the damping parameter. When \(\delta\) n (t) < 0, the cochlea is an active model; \(\rho\) n (t) is the feedback strength, \(\psi\) n (t) is the dimensionless delay; Let the sound pressure at the input end be P in , and the sound pressure on the nth segment of the basilar membrane be P n . In the Laplace transform domain, the sound pressure response input to each segment of the basilar membrane can be obtained by the following formula: P0(s) = P in (s) The displacement Y on each basement membrane n (s) can be expressed by the following formula: Y n (s) = H n (s)P n (s) Among them, H n (s) is the cochlear transfer function, which is jointly determined by the series admittance and the parallel admittance as follows: Then the vibration velocity v on the basement membrane n (t) can be obtained according to the inverse Laplace transform: v n v(t) = L -1 {sY n (s) - y(0)} where y(0) is the initial displacement; S2.3: Through the inner hair cell transduction model, the vibration velocity of the basilar membrane can be calculated by the following formula to obtain the membrane potential when the ciliary bundle deflects sinusoidally: Among them, C m is the capacitance of the inner hair cell, V m is the membrane potential of the inner hair cell, I MET is the mechanoelectrical transduction current (MET), and are the fast and slow currents of the outward K + , G is a constant gain value, v(t) is the vibration velocity on the segment of the basilar membrane, x0 = 35 nm, s = 16 nm, the maximum conduction gain G MET,max = 30 nS, the endocochlear voltage EP = 90 mV, V 0.5 = -31 mV, a k is a constant, the maximum conduction gain G K,f / s = 230 nS, V K,f = -71 mv, V K,s = -78 mV; S2.4: Through the nerve synapse exocytosis model, convert the inner hair cell electrical signals exported by the inner hair cell transduction model into nerve signals. The exocytosis rate k(t) of the inner hair cell ribbon synapses per unit time calculated can be calculated by the following formula: I Ca = g Ca (V m - E ca ) k(t) = zmax(I Ca -I Ca,th , 0) where I Ca is the whole-cell Ca 2+ channel current, g Ca is the Ca 2+ channel voltage-dependent conductivity, V m is the membrane potential of the inner hair cell, E Ca is the reversal potential, I Ca,th is the linear offset related to the single synaptic exocytosis rate, and z is a constant; S2.5: Substitute the ribbon synapse exocytosis rate k(t) of inner hair cells calculated in S2.4 into the three-layer diffusion model, and obtain the firing probability P of nerve fibers from the following formula Firing : where λ max is the maximum firing rate, k0 is the half-saturation constant, and Δt is the time window length; according to the neural fiber firing probability, the firing rate τ Firing (t) of a single neural fiber can finally be obtained, that is, calculated as follows: where P Rel is a constant representing the probability that at least one vesicle is released, and t a is the time constant; The auditory brainstem response spectrum, that is, the auditory voiceprint feature, can be obtained by summing the nerve fiber responses with different emission rates at each characteristic frequency.

10. The visual and auditory integrated perception coal mine overhead rail type inspection robot according to claim 9, characterized in that The above step S3 specifically includes the following steps: A. Output the auditory spectrum obtained in the above step S2.5 and perform multi-scale filtering using a 2D Gaussian filter to obtain sound intensity features; B. Then use a 2D Gabor filter to perform multi-scale filtering on the obtained sound intensity features to extract time correlation features and frequency correlation features. Among them, the Gabor filter used for the frequency correlation features sets the phase θ = 0°, and the Gabor filter used for the time correlation features sets the phase θ = 90°; C. Superimpose the multi-scale features of the finally obtained sound intensity, time correlation, and frequency correlation to obtain the final auditory feature map, and judge the category of underground environmental disasters based on the auditory feature map.

Images