A belt conveyor roller fault positioning method and system based on acoustic-visual bimodal fusion

Abstract

The application discloses a kind of based on acoustic vision bimodal fusion belt conveyor roller fault positioning method and system, it is related to coal mine belt conveyor intelligent inspection and fault positioning technical field, solve the problems that existing roller inspection method is difficult to unifiedly realize long-distance abnormal search, constrained approach, near-field credible confirmation, acoustic vision handover decision and risk constraint termination etc., the method utilizes microphone array to obtain sound source direction, two-dimensional rough positioning center and acoustic uncertainty, generates acoustic credible candidate area by posterior scale calibration, and guides robot limited approach;After entering near field, acoustic positioning information is projected to thermal infrared image, and thermal anomaly center and visual uncertainty are extracted, and are converted to working plane coordinate system.Acoustic vision fusion stage combines perception result and uncertainty, and is realized using adaptive probability decision strategy, and after meeting the conditions such as risk convergence and position stability, the roller fault position is output.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection and fault location technology for belt conveyors in coal mines, specifically a method and system for fault location of belt conveyor idlers based on audio-visual dual-modal fusion. Background Technology

[0002] Belt conveyors are widely used in industrial settings such as coal mines, mining, and bulk material transport, serving as crucial equipment in continuous material transport systems. Idler rollers, as key load-bearing components of belt conveyors, are numerous and widely distributed. They operate under complex conditions of high load, high dust levels, strong noise, and continuous vibration, making them prone to malfunctions such as bearing wear, lubrication failure, jamming, and abnormal overheating. If idler roller failures are not detected and accurately located in a timely manner, they may further lead to problems such as conveyor belt misalignment, tearing, and downtime, affecting the safe and stable operation of the conveying system.

[0003] Existing idler inspection methods mainly include manual inspection, vibration monitoring, acoustic detection, and visible light or thermal infrared visual inspection. Manual inspection suffers from low efficiency, strong subjectivity, and insufficient continuous coverage. Vibration monitoring usually requires the installation of contact sensors on the equipment or adjacent structures, resulting in high installation and maintenance costs. Acoustic detection is non-contact and insensitive to lighting conditions, making it suitable for long-distance anomaly detection. However, the stability and reliability of single acoustic positioning results are still limited under strong noise, reverberation, and complex mechanical background interference. Visible light or thermal infrared visual inspection can characterize the surface condition or local thermal anomalies of the idler under close-range conditions, but it is affected by line of sight, obstruction, viewing angle, and background interference, making it difficult to complete the complete inspection process of long-distance search, approach guidance, and near-field confirmation independently.

[0004] Therefore, existing technologies urgently need a roller fault location method that can unify long-distance anomaly search, constrained approach, near-field reliable confirmation, audio-visual handover decision-making, and risk-constrained inspection termination determination. Summary of the Invention

[0005] To overcome the difficulties of existing idler roller inspection methods in uniformly achieving long-distance anomaly search, constrained approach, near-field reliable confirmation, audio-visual handover decision-making, and risk constraint termination, this invention solves the above technical problems through the following technical solution: Option 1: This invention proposes a method for fault location of belt conveyor idler rollers based on audio-visual dual-modal fusion, the method comprising the following steps: Step 1: Set up a microphone array, a thermal infrared camera and a pose measurement unit on the mobile inspection robot to enable the mobile inspection robot to run along the inspection path of the belt conveyor and establish a unified working plane coordinate system within the working area of ​​the belt conveyor. Step 2: Use the microphone array set in Step 1 to collect multi-channel acoustic signals during the operation of the idler roller. Extract time delay correlation features, inter-channel phase difference features, and observation quality features from the multi-channel acoustic signals. Based on the extracted time delay correlation features, inter-channel phase difference features, and observation quality features, obtain the acoustic coarse localization result. The acoustic coarse localization result includes the direction of the fault sound source, the two-dimensional coarse localization center, and its acoustic uncertainty characterization. Step 3: Perform posterior scale calibration on the original position uncertainty covariance representation in the acoustic coarse localization results obtained in Step 2 to obtain an acoustic deployment covariance that is adapted to the actual deployment scenario. And based on the acoustic deployment covariance Determine acoustically reliable candidate regions ; Step 4: Once the mobile inspection robot meets the near-field confirmation conditions, based on the pose information of the mobile inspection robot, the extrinsic parameters of the thermal infrared camera, and the calibration relationship between the coordinate system of the conveyor working plane and the coordinate system of the thermal infrared image, the acoustically reliable candidate region mentioned in Step 3 is selected. The image is projected onto a thermal infrared image, and a local region of interest is determined within the thermal infrared image. Thermal anomaly features within the local region of interest are extracted, and the thermal anomaly center and its image domain uncertainty characterization are obtained. Step 5: Based on the calibration relationship between the thermal infrared image coordinate system and the conveyor working plane coordinate system, map the thermal anomaly center from the image coordinate system to the unified working plane coordinate system to obtain the visual positioning position; at the same time, propagate the image domain uncertainty according to the calibration relationship to obtain the visual positioning uncertainty in the working plane coordinate system. Step 6: Based on the mobile inspection robot and the acoustically reliable candidate region The relative positions between them, acoustic observation quality, visual observation quality, visual positioning uncertainty, and acoustic-visual position differences determine the current inspection stage; and under a unified working plane coordinate system, a probabilistic decision fusion with stage perception and uncertainty perception is performed. Based on the current inspection stage, acoustic-visual observation consistency and uncertainty level, a precision domain probabilistic fusion, covariance cross fusion, or high-risk conservative backoff strategy is selected to generate the current positioning output. Step 7: Based on the uncertainty level of the current positioning output, the position change during continuous observation, and the preset output source eligibility conditions, determine whether the inspection termination conditions are met. If the current positioning output does not meet the risk convergence conditions, position stability conditions, or preset output source eligibility conditions, the mobile inspection robot continues to execute constrained approach, thermal infrared supplementary observation, local verification, or audio-visual handover decision-making until the inspection termination conditions are met and the final idler roller fault location is output.

[0006] Furthermore, a preferred embodiment is provided, wherein the method for acquiring multi-channel acoustic signals during the operation of the idler roller in step 2, extracting time delay correlation features, inter-channel phase difference features, and observation quality features from the multi-channel acoustic signals, and obtaining acoustic coarse positioning results based on the extracted time delay correlation features, inter-channel phase difference features, and observation quality features is as follows: Set time The array multi-channel observations are as follows:

[0007] in, The number of microphone channels; the acoustic branch output is represented as:

[0008] in, For the direction estimation of the fault sound source relative to the microphone array, The coarse two-dimensional positioning center within the working plane of the conveyor. This characterizes the positional uncertainty covariance of the original output of the acoustic model. The acoustic coarse localization results are used for anomaly search, acoustically reliable candidate region construction, constrained approach of mobile inspection robots, and inspection phase switching.

[0009] Furthermore, a preferred implementation is provided in which the acoustic deployment covariance adapted to the actual deployment scenario is obtained in step 3. Based on the acoustic deployment covariance Determine acoustically reliable candidate regions The method is as follows:

[0010] Define acoustic risk summary quantity as:

[0011] in, Used to describe the overall diffusion level of the current coarse acoustic localization results within the two-dimensional working plane; Let the working plane area of ​​the conveyor be denoted as Then the acoustically reliable candidate region is represented as:

[0012] in, Let be the radius of the acoustically reliable candidate region, and express it as: , The deployment coefficient is determined by the target coverage level or empirical statistics of independent calibration sets.

[0013] Furthermore, a preferred embodiment is provided, wherein in step 4, the acoustically reliable candidate region described in step 3 is... The method for projecting the image onto a thermal infrared image and determining the local region of interest within the thermal infrared image is as follows: Let the local thermal infrared input region be The visual branch output is represented as:

[0014] in, For estimating the center of thermal anomalies in the normalized coordinate space of a local area of ​​interest (ROI), This represents the positional uncertainty covariance of the original output of the visual model. The thermal anomaly center is used to characterize the visual location of suspected faulty idlers under near-field conditions.

[0015] Furthermore, a preferred embodiment is provided, wherein step 5 specifically includes: The image domain deployment covariance is obtained by posterior scaling of the uncertainty in the original output of the visual model.

[0016] Let the homography matrix from the thermal infrared image coordinate system obtained from calibration to the conveyor working plane coordinate system be... Then the image point and the working plane point satisfy:

[0017] in, It is a homogeneous scaling factor. Image domain coordinates, Let the coordinates be the coordinates of the working plane; denote the non-homogeneous mapping as... The estimate of the visual prediction point in the working plane is:

[0018] At the prediction point Nearby, the image domain covariance is propagated to the working plane coordinate system, resulting in:

[0019] in, For mapping functions exist Jacobian matrix at the location, This represents the visual positioning uncertainty matrix in the working plane coordinate system. That is, the thermal infrared visual positioning results are represented as near-field observations in a unified working plane coordinate system. This information is used for subsequent audio-visual fusion decisions and inspection termination determinations.

[0020] Furthermore, a preferred embodiment is provided, wherein the current inspection stage in step 6 includes an acoustic-dominated stage, an acoustic-visual handover stage, and a visual-dominated stage, wherein the acoustic observation quality includes at least one of time delay correlation peak ratio, signal-to-noise ratio, or observation confidence; and the visual observation quality includes any one of thermal anomaly contrast, image domain uncertainty, and local observation quality index.

[0021] Furthermore, a preferred embodiment is provided, wherein the inspection termination conditions in step 7 include: the uncertainty level of the current positioning output is lower than a preset risk convergence threshold, the change in the current positioning output position at multiple consecutive observation times is lower than a preset position stability threshold, and the current positioning output source meets the output source qualification; the output source qualification includes stable visual positioning results formed in the visual-dominated stage, positioning results formed by precision domain probability fusion under low conflict conditions, or positioning results formed by covariance cross fusion under unknown correlation and moderate conflict conditions.

[0022] Option 2: A fault location system for belt conveyor idlers based on audio-visual dual-modal fusion. This system is implemented based on the method described in Option 1. The system includes a mobile inspection robot, a microphone array, a thermal infrared camera, a pose measurement unit, a processor, and a memory; the memory stores a computer program. The processor is configured to invoke the computer program to perform the following operations: obtain acoustic coarse localization results based on the multi-channel acoustic signals acquired by the microphone array, and construct acoustically reliable candidate regions based on the acoustic coarse localization results; project the acoustically reliable candidate regions onto the thermal infrared images acquired by the thermal infrared camera to obtain the thermal anomaly center and its image domain uncertainty characterization; transform the thermal anomaly center and its image domain uncertainty characterization to a unified working plane coordinate system; select a precision domain probability fusion, covariance cross fusion, or high-risk conservative backoff strategy based on the current inspection stage, the consistency of acoustic and visual observations, and the uncertainty level to generate the current localization output; and determine the inspection termination based on the uncertainty level of the current localization output, the positional changes during continuous observation, and the preset output source eligibility conditions.

[0023] Option 3: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in Option 1.

[0024] Option 4: A computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method described in Option 1.

[0025] The advantages of this invention are: This invention uses the fault sound source direction, two-dimensional coarse localization center and its acoustic uncertainty characterization obtained by the microphone array to construct an acoustically reliable candidate region, and uses the acoustically reliable candidate region to constrain the approach range of the mobile inspection robot. This transforms the acoustic coarse localization result from a single point position output into a spatial prior that serves constrained approach, near-field confirmation and stage switching, thereby reducing the unconstrained search range and improving the targeting and feasibility of the mobile inspection robot entering the suspected faulty roller area.

[0026] This invention determines the local region of interest in a thermal infrared image under acoustically reliable candidate region constraints, estimates the thermal anomaly center and its image domain uncertainty, and transforms the thermal infrared visual positioning results and their uncertainties to a unified working plane coordinate system based on the calibration relationship between the thermal infrared image coordinate system and the conveyor working plane coordinate system. This enables the acoustic coarse positioning results and the thermal infrared visual positioning results to be expressed, compared and fused at the same spatial scale, thereby reducing the impact of inconsistencies in coordinate apertures of different sensing modes on the positioning results.

[0027] This invention employs an adaptive probabilistic decision fusion mechanism that is both phase-aware and uncertainty-aware. Based on the current inspection phase, acoustic observation quality, visual observation quality, acoustic-visual position consistency, and uncertainty level, it adaptively selects precision domain probabilistic fusion, covariance cross-fusion, or a high-risk conservative backoff strategy. This enables the system to utilize complementary information from both modes when acoustic and visual observations are consistent, to obtain conservative positioning results when there are unknown correlations or moderate inconsistencies, and to trigger supplementary observations, local verification, or delayed output under high-risk conditions. This reduces the risk of positioning misjudgment caused by observation conflicts, unknown correlations, or forced fusion.

[0028] This invention constrains the inspection termination condition by considering the uncertainty level of the current positioning output, the position change during continuous observation, and the eligibility of the output source. The final idler fault location is output only when the risk converges, the position is stable, and the output source meets the eligibility requirements. The acoustic coarse positioning results during the acoustic-dominated stage are only used as the basis for approach guidance and transient decision-making within the stage, and are not used as the final idler fault location confirmation result. This avoids single accidental positioning convergence or acoustic coarse positioning results directly triggering the final docking confirmation, thereby improving the accuracy, stability, and safety of the final idler fault location output. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the belt conveyor idler roller fault location method based on audio-visual dual-modal fusion as described in Implementation Method 1.

[0030] Figure 2 This is a schematic diagram of acoustic coarse localization and acoustic reliable candidate region construction as described in Implementation Method 1.

[0031] Figure 3This is a schematic diagram of the thermal infrared near-field localization process under acoustically reliable candidate region constraints as described in Implementation Method 1.

[0032] Figure 4 This is a schematic diagram of the phased perception audio-visual fusion decision-making and inspection termination determination process described in Implementation Method 1.

[0033] Among them, belt conveyor 1, idler group 2, faulty idler 3, mobile inspection robot 4, microphone array 5, acoustic coarse positioning center 6, fault sound source direction 7, acoustic reliable candidate region 8, and conveyor working plane coordinate system 9. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0035] Implementation Method 1, see [link] Figures 1 to 4 This embodiment describes a method for fault location of belt conveyor idlers based on audio-visual dual-modal fusion. The method specifically includes the following steps: Step 1: Establish a mobile inspection and sensing platform and a unified working plane coordinate system.

[0036] A microphone array, a thermal infrared camera, and a pose measurement unit are installed on a mobile inspection robot, enabling the robot to move along the inspection path of a belt conveyor and establishing a unified working plane coordinate system within the conveyor's working area. This unified working plane coordinate system represents the acoustic coarse positioning results, the thermal infrared visual positioning results, and the final output of the idler roller fault location. The pose measurement unit obtains the mobile inspection robot's pose information along the inspection path, allowing the acoustic observation results, thermal infrared image positioning results, and the final positioning output to be expressed, compared, and fused within the same working plane coordinate system.

[0037] Step 2: Acquire multi-channel acoustic signals and obtain coarse acoustic localization results.

[0038] Multi-channel acoustic signals during the operation of the idler roller are acquired by a microphone array. Time delay correlation features, inter-channel phase difference features, and observation quality features are extracted from the multi-channel acoustic signals. Based on these features, the direction of the fault sound source, the two-dimensional coarse positioning center, and its acoustic uncertainty characterization are obtained.

[0039] In one implementation, a time is set The array multi-channel observations are as follows:

[0040] in, This represents the number of microphone channels. The acoustic branch output is represented as:

[0041] in, For the direction estimation of the fault sound source relative to the microphone array, The coarse two-dimensional positioning center within the working plane of the conveyor. This represents the positional uncertainty covariance of the original output of the acoustic model.

[0042] The acoustic coarse localization results are used for anomaly search, acoustically reliable candidate region construction, constrained approach of the mobile inspection robot, and inspection phase switching, and are not directly used as the final confirmation result of the idler roller fault location.

[0043] Step 3: Construct acoustically reliable candidate regions based on the acoustic coarse localization results.

[0044] The original position uncertainty covariance representation in the acoustic coarse localization results is posteriorly scaled to obtain an acoustic deployment covariance that is adapted to the actual deployment scenario. In one implementation, the acoustic deployment covariance is expressed as:

[0045] To facilitate candidate region construction and risk gating, the acoustic risk summary size is defined as:

[0046] in, Used to describe the overall diffusion level of the current coarse acoustic localization results within the two-dimensional working plane.

[0047] Let the working plane area of ​​the conveyor be denoted as Then the acoustically reliable candidate region is represented as:

[0048] in, Let be the radius of the acoustically reliable candidate region, and express it as:

[0049] in, The deployment coefficient is determined by the target coverage level or empirical statistics of independent calibration sets.

[0050] The acoustically reliable candidate region is used to define the search and approach range of the mobile inspection robot and provides spatial priors for thermal infrared near-field localization, phase switching, and audio-visual handover decisions. See also Figure 2The microphone array 5 obtains the fault sound source direction 7, the acoustic coarse localization center 6, and its positional uncertainty characterization based on the time delay correlation characteristics, inter-channel phase difference characteristics, and observation quality characteristics of the multi-channel acoustic signals. After posterior scale calibration, the acoustic deployment covariance is obtained, and the candidate region radius is determined accordingly. With the acoustic coarse localization center 6 as the center, a reliable acoustic candidate region 8 is constructed within the conveyor working plane coordinate system 9. The reliable acoustic candidate region 8 is used to constrain the approach range of the mobile inspection robot 4 and provides spatial priors for thermal infrared near-field localization and inspection phase switching. The acoustic coarse localization result is not directly used as the final confirmation result of the idler roller fault location.

[0051] Step 4: Perform thermal infrared near-field localization under acoustically reliable candidate region constraints.

[0052] Once the mobile inspection robot meets the near-field confirmation conditions, acoustically reliable candidate regions are identified based on the robot's pose information, the thermal infrared camera's extrinsic parameters, and the calibration relationship between the conveyor's working plane coordinate system and the thermal infrared image coordinate system. The image is projected onto a thermal infrared image, and a local region of interest (ROI) is determined based on the projected area and its extended area. Thermal anomaly features within the ROI are extracted, and the thermal anomaly center and its image domain uncertainty characterization are obtained. The near-field confirmation conditions include at least one of the following: the mobile inspection robot enters the acoustically reliable candidate region; or the distance between the mobile inspection robot and the acoustic coarse localization center is lower than a preset proximity threshold.

[0053] In one embodiment, the local thermal infrared input area is defined as... The visual branch output is represented as:

[0054] in, For estimating the center of thermal anomalies in the normalized coordinate space of a local area of ​​interest (ROI), This represents the positional uncertainty covariance of the original output of the visual model.

[0055] The thermal anomaly center is used to characterize the visual location of the suspected faulty idler under near-field conditions. The visual branch performs local localization and near-field confirmation under acoustically reliable candidate region constraints, but does not perform unconstrained global search.

[0056] Step 5: Convert the thermal infrared positioning results to a unified working plane coordinate system.

[0057] Based on the calibration relationship between the thermal infrared image coordinate system and the conveyor working plane coordinate system, the thermal anomaly center is mapped from the image coordinate system to the unified working plane coordinate system to obtain the visual positioning position; at the same time, the image domain uncertainty is propagated according to the calibration relationship to obtain the visual positioning uncertainty in the working plane coordinate system.

[0058] In one implementation, the uncertainty of the original output of the visual model is calibrated using a posterior scale to obtain the image domain deployment covariance:

[0059] Let the homography matrix from the thermal infrared image coordinate system obtained from calibration to the conveyor working plane coordinate system be... Then the image point and the working plane point satisfy:

[0060] in, It is a homogeneous scaling factor. Image domain coordinates, Let the coordinates be those of the working plane. The non-homogeneous mapping is denoted as... The estimate of the visual prediction point in the working plane is:

[0061] At the prediction point Nearby, the image domain covariance is propagated to the working plane coordinate system, resulting in:

[0062] in, For mapping functions exist Jacobian matrix at the location, This is the uncertainty matrix for visual positioning in the working plane coordinate system.

[0063] Therefore, the thermal infrared visual positioning results are represented as near-field observations in a unified working plane coordinate system. This information is used for subsequent audio-visual fusion decisions and inspection termination determinations.

[0064] Step 6: Perform adaptive probabilistic decision fusion with stage perception and uncertainty perception.

[0065] Based on the relative position between the mobile inspection robot and the acoustically reliable candidate region, the acoustic observation quality, the visual observation quality, the visual positioning uncertainty, and the acoustic-visual position difference, the current inspection stage is determined. Under a unified working plane coordinate system, based on the current inspection stage, the consistency between acoustic and visual observations, and the level of uncertainty, an adaptive selection strategy is chosen from precision domain probabilistic fusion, covariance cross-fusion, or a high-risk conservative backoff strategy to generate the current positioning output. The current inspection stage includes an acoustic-dominated stage, an acoustic-visual handover stage, and a visual-dominated stage. The acoustic observation quality can be determined by statistics such as the time-delay correlation peak ratio, signal-to-noise ratio, or observation confidence; the visual observation quality can be determined by thermal anomaly contrast, image domain uncertainty, and local observation quality indicators.

[0066] In a unified working plane coordinate system, acoustic observations and visual observations are respectively represented as:

[0067]

[0068] in, For acoustic coarse localization center, This is the covariance representation used in the fusion phase of acoustic observations; For visual positioning, This represents the covariance of visual observations in the working plane coordinate system.

[0069] Based on the current inspection phase Acoustic quality factor Visual quality factor Modulating the covariance of the two modes yields the equivalent covariance:

[0070]

[0071] in, and The modulation coefficients are related to the current stage state.

[0072] Further define the consistency gating quantity under joint uncertainty constraints:

[0073] And define the difference in audio-visual position:

[0074] During the audio-visual handover phase, the corresponding decision-making method is selected based on the consistency gating quantity, audio-visual position differences, and uncertainty level.

[0075] When the acoustic localization result and the visual localization result meet the consistency condition and the uncertainty level is lower than the preset threshold, the accuracy domain probabilistic fusion is used to obtain the current localization result:

[0076]

[0077] in, This represents the location after probability fusion in the precision domain. This is the corresponding uncertainty matrix.

[0078] When there is an unknown correlation or moderate inconsistency between acoustic and visual localization results, a covariance cross-fusion method is used to obtain a conservative localization result to suppress the underestimation of uncertainty under unknown correlation conditions.

[0079]

[0080] in, The value is determined by the current inspection phase, the difference in quality between the two modes, and the positional deviation.

[0081] when Exceeding the high-risk threshold or When the safety boundary is exceeded, forced fusion is not performed. Instead, a conservative rollback mode is entered, which retains the transient decision output with higher reliability in the current stage and triggers supplementary observation, local verification or delayed output.

[0082] In the acoustic-dominated phase, acoustic localization results are used for coarse localization guidance, candidate region generation, and constrained approach; in the acoustic-visual handover phase, a fusion or conservative decision-making method is selected based on the consistency and risk level between acoustic and visual localization results; in the visual-dominated phase, stable thermal infrared visual localization results are used as the basis for near-field confirmation.

[0083] Step 7: Make a risk-constrained inspection termination decision.

[0084] Based on the uncertainty level of the current positioning output, the positional changes during continuous observation, and the eligibility of the output source, determine whether the inspection termination conditions are met.

[0085] The current location output is denoted as:

[0086] in, For the first The current output position at each observation time. This represents the corresponding uncertainty matrix. The current output risk summary quantity is defined as:

[0087] When continuous Simultaneously satisfying the following within a number of observation times:

[0088]

[0089] Furthermore, when the current output source meets the termination qualification requirements, the final idler roller fault location is output, and the inspection termination determination is completed.

[0090] in, Indicates the risk convergence threshold. Indicates the position stability threshold. Indicates the duration of continuous stability.

[0091] The termination eligibility requirements include: stable visual positioning results formed during the vision-dominated phase, or positioning results formed after precision domain probability fusion or covariance cross fusion during the acoustic-visual handover phase; the acoustic coarse positioning results during the acoustic-dominated phase are only used as the basis for approach guidance and transient decision-making within the phase, and are not used as the final confirmation result of the idler roller fault location.

[0092] If the current positioning output does not meet the risk convergence condition, position stability condition, or output source qualification requirements, the mobile inspection robot will continue to execute the decision-making process of constrained approach, thermal infrared supplementary observation, local verification, or audio-visual handover until the inspection termination condition is met and the final roller fault location is output.

[0093] Through the above steps, this invention unifies acoustic anomaly perception, thermal infrared near-field confirmation, and acoustic-visual dual-modal fusion decision-making into the inspection process of belt conveyor idler movement. This transforms idler fault location from single-modal location into a continuous location process that includes acoustic coarse location guidance, acoustic reliable candidate region construction, thermal infrared near-field confirmation, stage perception and uncertainty perception adaptive probabilistic decision fusion, and inspection termination determination based on risk constraints. This improves the accuracy, stability, and engineering feasibility of idler fault location.

[0094] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. This is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0095] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for fault location of belt conveyor idler rollers based on audio-visual dual-modal fusion, characterized in that, The method includes the following steps: Step 1: Set up a microphone array, a thermal infrared camera and a pose measurement unit on the mobile inspection robot to enable the mobile inspection robot to run along the inspection path of the belt conveyor and establish a unified working plane coordinate system within the working area of ​​the belt conveyor. Step 2: Use the microphone array set in Step 1 to collect multi-channel acoustic signals during the operation of the idler roller. Extract time delay correlation features, inter-channel phase difference features, and observation quality features from the multi-channel acoustic signals. Based on the extracted time delay correlation features, inter-channel phase difference features, and observation quality features, obtain the acoustic coarse localization result. The acoustic coarse localization result includes the direction of the fault sound source, the two-dimensional coarse localization center, and its acoustic uncertainty characterization. Step 3: Perform posterior scale calibration on the original position uncertainty covariance representation in the acoustic coarse localization results obtained in Step 2 to obtain an acoustic deployment covariance that is adapted to the actual deployment scenario. And based on the acoustic deployment covariance Determine acoustically reliable candidate regions ; Step 4: Once the mobile inspection robot meets the near-field confirmation conditions, based on the pose information of the mobile inspection robot, the extrinsic parameters of the thermal infrared camera, and the calibration relationship between the coordinate system of the conveyor working plane and the coordinate system of the thermal infrared image, the acoustically reliable candidate region mentioned in Step 3 is selected. The image is projected onto a thermal infrared image, and a local region of interest is determined in the thermal infrared image. Thermal anomaly features within the local region of interest are extracted, and the thermal anomaly center and its image domain uncertainty characterization are obtained. Step 5: Based on the calibration relationship between the thermal infrared image coordinate system and the conveyor working plane coordinate system, map the thermal anomaly center from the image coordinate system to the unified working plane coordinate system to obtain the visual positioning position; at the same time, propagate the image domain uncertainty according to the calibration relationship to obtain the visual positioning uncertainty in the working plane coordinate system. Step 6: Based on the mobile inspection robot and the acoustically reliable candidate region The relative positions between them, acoustic observation quality, visual observation quality, visual positioning uncertainty, and acoustic-visual position differences determine the current inspection stage; and under a unified working plane coordinate system, a probabilistic decision fusion with stage perception and uncertainty perception is performed. Based on the current inspection stage, acoustic-visual observation consistency and uncertainty level, a precision domain probabilistic fusion, covariance cross fusion, or high-risk conservative backoff strategy is selected to generate the current positioning output. Step 7: Based on the uncertainty level of the current positioning output, the position change during continuous observation, and the preset output source eligibility conditions, determine whether the inspection termination conditions are met. If the current positioning output does not meet the risk convergence conditions, position stability conditions, or preset output source eligibility conditions, the mobile inspection robot continues to execute constrained approach, thermal infrared supplementary observation, local verification, or audio-visual handover decision-making until the inspection termination conditions are met and the final idler roller fault location is output.

2. The method for fault location of belt conveyor idler rollers based on audio-visual dual-modal fusion according to claim 1, characterized in that, Step 2 involves acquiring multi-channel acoustic signals during the idler roller's operation, extracting time delay correlation features, inter-channel phase difference features, and observation quality features from the multi-channel acoustic signals, and obtaining coarse acoustic positioning results based on the extracted time delay correlation features, inter-channel phase difference features, and observation quality features. Set time The array multi-channel observations are as follows: in, The number of microphone channels; the acoustic branch output is represented as: in, For the direction estimation of the fault sound source relative to the microphone array, The coarse two-dimensional positioning center within the working plane of the conveyor. This characterizes the positional uncertainty covariance of the original output of the acoustic model. The acoustic coarse localization results are used for anomaly search, acoustically reliable candidate region construction, constrained approach of the mobile inspection robot, and inspection phase switching, and are not directly used as the final confirmation result of the idler roller fault location.

3. The method for fault location of belt conveyor idler rollers based on audio-visual dual-modal fusion according to claim 1, characterized in that, Step 3 yields the acoustic deployment covariance adapted to the actual deployment scenario. Based on the acoustic deployment covariance Determine acoustically reliable candidate regions The method is as follows: Define acoustic risk summary quantity as: in, Used to describe the overall diffusion level of the current coarse acoustic localization results within the two-dimensional working plane; Let the working plane area of ​​the conveyor be... Then the acoustically reliable candidate region is represented as: in, Let be the radius of the acoustically reliable candidate region, and express it as: , The deployment coefficient is determined by the target coverage level or empirical statistics of independent calibration sets.

4. The method for fault location of belt conveyor idler rollers based on audio-visual dual-modal fusion according to claim 1, characterized in that, In step 4, the acoustically reliable candidate region described in step 3 is... The method for projecting the image onto a thermal infrared image and determining the local region of interest within the thermal infrared image is as follows: Let the local thermal infrared input region be The visual branch output is represented as: in, For estimating the center of thermal anomalies in the normalized coordinate space of a local area of ​​interest (ROI), This represents the positional uncertainty covariance of the original output of the visual model. The thermal anomaly center is used to characterize the visual location of suspected faulty idlers under near-field conditions.

5. The method for fault location of belt conveyor idler rollers based on audio-visual dual-modal fusion according to claim 1, characterized in that, Step 5 specifically includes: The image domain deployment covariance is obtained by posterior scaling of the uncertainty in the original output of the visual model. Let the homography matrix from the thermal infrared image coordinate system obtained from calibration to the conveyor working plane coordinate system be... Then the image point and the working plane point satisfy: in, It is a homogeneous scaling factor. Image domain coordinates, Let the coordinates be the coordinates of the working plane; denote the non-homogeneous mapping as... The estimate of the visual prediction point in the working plane is: At the prediction point Nearby, the image domain covariance is propagated to the working plane coordinate system, resulting in: in, For mapping functions exist Jacobian matrix at the location, This represents the visual positioning uncertainty matrix in the working plane coordinate system. That is, the thermal infrared visual positioning results are represented as near-field observations in a unified working plane coordinate system. This information is used for subsequent audio-visual fusion decisions and inspection termination determinations.

6. The method for fault location of belt conveyor idler rollers based on audio-visual dual-modal fusion according to claim 1, characterized in that, The current inspection phase described in step 6 includes an acoustic-dominated phase, an acoustic-visual handover phase, and a visual-dominated phase. The acoustic observation quality includes at least one of the following: time delay correlation peak ratio, signal-to-noise ratio, or observation confidence. The visual observation quality includes any one of the following: thermal anomaly contrast, image domain uncertainty, and local observation quality index.

7. The method for fault location of belt conveyor idler rollers based on audio-visual dual-modal fusion according to claim 1, characterized in that, The inspection termination conditions in step 7 include: the uncertainty level of the current positioning output is lower than the preset risk convergence threshold, the change in the current positioning output position at multiple consecutive observation times is lower than the preset position stability threshold, and the current positioning output source meets the output source qualification conditions; the preset output source qualification conditions include stable visual positioning results formed in the visual-dominated stage, positioning results formed by precision domain probability fusion when the acoustic positioning results and visual positioning results meet the preset consistency conditions and the uncertainty level is lower than the preset threshold, or positioning results formed by covariance cross fusion when the acoustic positioning results and visual positioning results have unknown correlation or moderate inconsistency.

8. A fault location system for belt conveyor idlers based on audio-visual dual-modal fusion, characterized in that, The system is implemented based on the method described in claim 1, and includes a mobile inspection robot, a microphone array, a thermal infrared camera, a pose measurement unit, a processor, and a memory; the memory stores a computer program. The processor is configured to invoke the computer program to perform the following operations: obtain acoustic coarse localization results based on the multi-channel acoustic signals acquired by the microphone array, and construct acoustically reliable candidate regions based on the acoustic coarse localization results; project the acoustically reliable candidate regions onto the thermal infrared images acquired by the thermal infrared camera to obtain the thermal anomaly center and its image domain uncertainty characterization; transform the thermal anomaly center and its image domain uncertainty characterization to a unified working plane coordinate system; and select a precision domain probability fusion, covariance cross fusion, or high-risk conservative backoff strategy based on the current inspection stage, the consistency of acoustic and visual observations, and the uncertainty level to generate the current localization output. The system also determines the termination of the inspection based on the uncertainty level of the current positioning output, the positional changes during continuous observation, and the preset eligibility conditions of the output source.

9. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

10. A computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method of any one of claims 1-7.

Images