Belt conveyor coal flow detection method based on monocular vision

Abstract

The invention discloses a belt conveyor coal flow detection method based on monocular vision, and the method comprises the steps: 1, setting a monocular camera which is used for collecting a conveying belt and a coal flow image, and determining the relation between internal and external parameters of the camera and a world coordinate system; 2, obtaining a coal flow foreground image; 3, feature point detection and description are carried out on the coal flow foreground image, and feature points are matched between adjacent frames to obtain matching points; 4, recovering the three-dimensional coordinates of the feature points by using the matching points and a monocular camera perspective projection model in combination with the motion information of the conveying belt, and obtaining the point cloud data of the surface of the coal flow; and 5, carrying out surface fitting on the point cloud data, and calculating the volume and mass flow of coal passing through a coal flow interface in unit time by calculating the sectional area of the coal flow and combining the movement speed of the conveyor belt. A binocular or multi-view vision system and an expensive and complex traditional metering device are not needed, and the system cost and the installation and maintenance difficulty are reduced.

Description

Technical Field

[0001] The invention relates to the technical field of monitoring and controlling a main transport system of a coal mine, and in particular to a method for detecting coal flow of a belt conveyor based on monocular vision. Background Art

[0002] In the process of coal production and transportation, belt conveyors are widely used due to their continuous, efficient and stable conveying capacity. With the development of smart mines, the requirements for real-time detection and measurement of coal flow on conveyor belts are constantly increasing. Traditional coal flow measurement methods (such as nuclear scales and belt scales) are not only expensive, complex to install, and have high maintenance costs, but are also easily affected by factors such as vibration and changes in belt slack, resulting in reduced measurement accuracy and reliability.

[0003] In recent years, with the development of computer vision and image processing technology, research on coal quantity estimation by installing cameras and shooting and analyzing coal flow on conveyor belts has attracted attention. Although binocular or multi-camera vision methods can reconstruct the three-dimensional surface of coal flow using parallax information, they require multiple cameras, are complex to calibrate, are inconvenient to arrange, and are expensive. In contrast, the acquisition of three-dimensional information of coal flow based on monocular vision still faces technical challenges, because monocular cameras cannot directly restore depth information from static images. To this end, it is necessary to use the known motion information of the conveyor belt and the perspective projection model, combined with the matching of image feature points between adjacent frames, so as to obtain the three-dimensional structure of the coal flow surface under monocular conditions and realize accurate monitoring of coal flow. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of low accuracy and low reliability of the coal flow measurement method in the prior art, and to provide a belt conveyor coal flow detection method based on monocular vision.

[0005] The purpose of the present invention is achieved through the following technical solutions: A method for detecting coal flow of a belt conveyor based on monocular vision comprises the following steps: Step 1, setting a monocular camera for collecting images of the conveyor belt and coal flow, and determining the relationship between the internal and external parameters of the camera and the world coordinate system; Step 2: The monocular camera collects images of the conveyor belt and the coal flow, and performs background segmentation and dynamic area extraction on the conveyor belt and the coal flow images to obtain a foreground image of the coal flow; Step 3, detecting and describing feature points of the coal flow foreground image, and matching feature points between adjacent frames to obtain matching points; Step 4: Using the matching points and the monocular camera perspective projection model, combined with the conveyor belt motion information, the three-dimensional coordinates of the feature points are restored to obtain the coal flow surface point cloud data; Step 5: Perform surface fitting on the point cloud data, and calculate the volume and mass flow rate of coal passing through the coal flow interface per unit time by calculating the cross-sectional area of the coal flow and combining with the conveyor belt movement speed.

[0006] Preferably, in step 2, preprocessing is also performed on the conveyor belt and coal flow images, and the preprocessing includes denoising and distortion correction.

[0007] Preferably, in step 1, the internal parameter matrix K of the camera is: where f x , f y are the focal lengths in the horizontal and vertical directions on the imaging plane, and (c x , c y ,) are the coordinates of the principal point; The external parameter matrix of the camera is: where R represents the rotation matrix, t represents the translation vector, X C , Y C , Z C represent the three-dimensional coordinates of the camera coordinate system, and X w , Y w , Z w represent the three-dimensional coordinates of the world coordinate system.

[0008] Preferably, in step 2, background segmentation and dynamic region extraction are performed on the conveyor belt and coal flow images to obtain the coal flow foreground image, specifically: Use the semantic segmentation technology based on nnUNet to train the image, construct a segmentation model adapted to the coal flow scenario, and then optimize the network architecture and hyperparameters of the segmentation model; Input the conveyor belt and coal flow images into the trained segmentation model to obtain the pixel-level segmentation result.

[0009] Preferably, step 3 is specifically: Adopt the LOFTR module to perform feature point matching on the segmented coal flow and conveyor belt regions to generate matching pairs

[0010] Preferably, after obtaining the matching points in step 3, double filtering is also performed on the matching points, and the double filtering includes: Eliminate incorrect matching points through simple similarity and spatial constraints; Based on the known motion parameters of the conveyor belt, eliminate the matching points that do not conform to the conveyor belt motion law.

[0011] Preferably, based on the known motion parameters of the conveyor belt, eliminating the matching points that do not conform to the conveyor belt motion law is specifically: The conveyor belt moves uniformly at a known speed v b in a certain direction. For the three-dimensional coordinates (X w , Y w , Z w ) of a point on the surface of the coal flow, the relationship projected onto the image coordinate system in the camera coordinate system can be expressed as: During the time interval Δt from the k-th frame to the (k + l)-th frame, a certain point on the conveyor belt undergoes a known translation Δs = v b ·Δt relative to the world coordinates of the camera. For correctly matched feature points, their displacements on the imaging plane should conform to the projection characteristics of the conveyor belt movement. Further filter out incorrect matches in the set of matching points: Compare the pixel coordinate displacements (Δu, Δv) of the matching points between adjacent frames with the projected displacement range predicted by the conveyor belt movement, and eliminate the matching points with too large deviations from the predicted values, thereby obtaining a set of reliable matching feature point pairs.

[0012] Preferably, step 4 is specifically as follows: The imaging of the k-th frame and the (k + 1)-th frame satisfies: In the normalized coordinate system, p k =(x k , y k , z k ) T and p k+1 =(x k+1 , y k+1 , z k+1 ) T correspond to the normalized coordinates of the imaging plane. Expand the above formula and use the inverse solution of the internal parameter matrix to obtain the relationship in the normalized coordinate system: Considering the transformation relationship between the world coordinates and the camera coordinates, given the movement displacement (ΔX w , ΔY w , ΔZ w ) of the conveyor belt, the rotation matrix R, the translation vector t, the conveyor belt speed v b , and the time interval Δt, use the displacement difference of the feature points between two frames to solve for the depth Z w and the corresponding X w , Y w by solving a system of equations.

[0013] Preferably, step 5 is specifically as follows: By performing surface fitting on the three-dimensional coordinates of several feature points, a continuous three-dimensional description of the coal flow surface is reconstructed. Delaunay triangulation is used to fit the sparsely reconstructed three-dimensional feature points. Based on the contour data of the coal flow surface and the empty conveyor belt after fitting, the cross-sectional area A of the coal flow is calculated at any specified cross-section. cross ; Combined with the known speed of the conveyor belt, the volumetric flow rate of the coal flow through the cross-section is expressed as: Q v = A cross · v b Further combined with the coal density ρ, the mass flow rate of the coal flow is calculated: Q m = ρ · Q v = ρ · A cross · v v ..

[0014] The beneficial effects of the present invention are as follows: The present invention can achieve three-dimensional reconstruction and flow rate estimation of coal flow by using a monocular camera and known belt movement information, without the need for a binocular or multiocular vision system and expensive and complex traditional metering devices, reducing the system cost and the difficulty of installation and maintenance.

[0015] The method for screening matching points based on the fusion of vision and motion information effectively improves the reliability and accuracy of feature point matching, and solves the problem of missing depth information in monocular vision when estimating coal flow. Brief Description of the Drawings

[0016] Figure 1 is a flowchart of the present invention. Detailed Embodiments

[0017] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art.

[0018] In addition, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present application. However, those skilled in the art will realize that the technical solutions of the present application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be used. In other cases, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present application.

[0019] The flowcharts shown in the accompanying drawings are merely illustrative and not necessarily include all the content and operations / steps, nor are they necessarily executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined. Therefore, the actual execution order may change according to the actual situation.

[0020] Embodiment: A method for detecting the coal flow rate of a belt conveyor based on monocular vision, as Figure 1 shown, includes the following steps: Step 1, set up a monocular camera for collecting images of the conveyor belt and coal flow, and determine the relationship between the internal and external parameters of the camera and the world coordinate system; Step 2, the monocular camera collects images of the conveyor belt and coal flow, and performs background segmentation and dynamic region extraction on the images of the conveyor belt and coal flow to obtain a coal flow foreground image; Step 3, detect and describe feature points in the coal flow foreground image, and match the feature points between adjacent frames to obtain matching points; Step 4, use the matching points and the monocular camera perspective projection model, combined with the conveyor belt movement information, to restore the three-dimensional coordinates of the feature points and obtain the coal flow surface point cloud data; Step 5, perform surface fitting on the point cloud data, and calculate the volume and mass flow rate of the coal passing through the coal flow interface per unit time by calculating the cross-sectional area of the coal flow and combining the conveyor belt movement speed.

[0021] In the said Step 2, the images of the conveyor belt and coal flow are also preprocessed, and the preprocessing includes denoising and distortion correction.

[0022] In the said Step 1, the camera internal parameter matrix K is: where f x , f y are the focal lengths in the horizontal and vertical directions on the imaging plane, (c x , c y ,) are the principal point coordinates; The camera external parameter matrix is: where R represents the rotation matrix, t represents the translation vector, X C , Y C , Z C represent the three-dimensional coordinates of the camera coordinate system, and X w , Y w , Z w represent the three-dimensional coordinates of the world coordinate system.

[0023] In step 2, background segmentation and dynamic region extraction are performed on the conveyor belt and coal flow images to obtain the coal flow foreground image. Specifically: Use the semantic segmentation technology based on nnUNet to train the image, construct a segmentation model adapted to the coal flow scenario, and then optimize the network architecture and hyperparameters of the segmentation model; Input the conveyor belt and coal flow images into the trained segmentation model to obtain a pixel-level segmentation result. This model can distinguish the belt, coal flow, and background regions, providing an accurate foreground mask for subsequent analysis.

[0024] The nnUNet model automatically extracts and learns the color, texture, and shape features of the image through deep learning, and can generate accurate segmentation results. The coal flow pixel region is obtained by nnUNet segmentation, effectively removing background interference and laying a foundation for the 3D reconstruction and subsequent analysis of the coal flow surface. This technical method does not require manual design of segmentation rules, is applicable to coal flow segmentation in complex scenarios, and has the advantages of high robustness and automation.

[0025] Step 3 is specifically as follows: Use the LOFTR (Local Feature TRansformer local feature transformer) module to perform feature point matching on the segmented coal flow and conveyor belt regions to generate matching pairs Different from traditional ORB, SIFT, or SURF feature methods, LOFTR uses a deep learning architecture based on Transformer to optimize the matching within the global range through a bidirectional interactive attention mechanism to reduce the occurrence of false matches, and can automatically extract feature points and their corresponding feature descriptors within the segmented coal flow and conveyor belt regions. This module performs feature detection on the above specific regions and extracts the geometric structure and texture information of the image.

[0026] In step 3, after obtaining the matching points, double filtering is also performed on the matching points. The double filtering includes: Eliminate incorrect matching points through simple similarity and spatial constraints; Based on the known motion parameters of the conveyor belt, eliminate the matching points that do not conform to the conveyor belt motion law.

[0027] The elimination of the matching points that do not conform to the conveyor belt motion law based on the known motion parameters of the conveyor belt is specifically as follows: The conveyor belt moves uniformly at a known speed v b in a certain direction. For the three-dimensional coordinates (X w , Y w , Z w ) of a certain point on the coal flow surface, the relationship projected onto the image coordinate system in the camera coordinate system can be expressed as: In the interval Δt from the kth frame to the k+1th frame, a point on the belt undergoes a known translation Δs=v relative to the world coordinates of the camera. b ·Δt, for the correctly matched feature points, their displacement on the imaging plane should be consistent with the belt motion projection characteristics, and then further filter out the wrong matches in the matching point set: compare the pixel coordinate displacement (Δu, Δv) of the matching points in adjacent frames with the projection displacement range predicted by the belt motion, and eliminate the matching points that deviate too much from the predicted value, so as to obtain a set of credible matching feature point pairs.

[0028] The step 4 is specifically as follows: The imaging of the kth frame and the k+1th frame satisfies: In the normalized coordinate system, p k =(x k y k , z k ) T and p k+1 =(x k+1 ·y k+1 , z k+1 ) T Corresponding to the normalized coordinates of the imaging plane, the above formula is expanded and the inverse solution is used using the internal parameter matrix to obtain the relationship in the normalized coordinate system: Considering the transformation relationship between the world coordinates and the camera coordinates, the known motion displacement of the conveyor belt (ΔX w , △Y w , ΔZ w ), rotation matrix R, translation vector t, conveyor belt speed v b , time interval △t, using the displacement difference of the feature point between the two frames, the depth Z of the feature point is calculated by the simultaneous equations w and the corresponding X w , Y w To solve.

[0029] Taking the equation of the x-direction of the image coordinate system as an example, the constraint relationship can be expressed as: Combining the two equations and the equation of the y direction of the image coordinate system, we can solve for X w , Y w , Z w .

[0030] The step 5 is specifically as follows: By performing surface fitting on the three-dimensional coordinates of several feature points, a continuous three-dimensional description of the coal flow surface is reconstructed. Delaunay triangulation is used to fit the sparsely reconstructed three-dimensional feature points. Based on the contour data of the coal flow surface and the empty conveyor belt after fitting, the cross-sectional area A of the coal flow is calculated at any specified cross-section. cross ; Combined with the known speed of the conveyor belt, the volumetric flow rate of the coal flow through the cross-section is expressed as: Q v = A cross · v b Further combined with the coal density ρ, the mass flow rate of the coal flow is calculated: Q m = ρ · Q v = ρ · A cross · v b ..

[0031] According to the measurement requirements, multiple cross-sections can be set on the coal flow surface for analysis to improve the measurement accuracy. This embodiment provides a low-cost, easy-to-install, and easy-to-maintain online monitoring technical means for the detection of coal flow rate in the main coal transportation system of coal mines through real-time and accurate measurement of the coal flow rate on the conveyor.

[0032] After considering the specification and the disclosed embodiments herein, those skilled in the art will readily conceive of other embodiments of the present application. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include the common general knowledge or conventional technical means in the technical field not disclosed in the present application.

[0033] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A method for detecting the coal flow rate of a belt conveyor based on monocular vision, characterized in that It includes the following steps: Step 1: Set up a monocular camera for collecting images of the conveyor belt and coal flow, and determine the relationship between the internal and external parameters of the camera and the world coordinate system; Step 2: The monocular camera collects images of the conveyor belt and coal flow, and performs background segmentation and dynamic region extraction on the images of the conveyor belt and coal flow to obtain a coal flow foreground image; Step 3: Detect and describe feature points in the coal flow foreground image, and match the feature points between adjacent frames to obtain matching points; Step 4: Utilize the matching points and the monocular camera perspective projection model, combined with the conveyor belt movement information, to restore the three-dimensional coordinates of the feature points and obtain the coal flow surface point cloud data; Step 5: Perform surface fitting on the point cloud data, and calculate the cross-sectional area of the coal flow and combine it with the conveyor belt movement speed to obtain the volume and mass flow rates of the coal passing through the coal flow interface per unit time.

2. The coal flow rate detection method for a belt conveyor based on monocular vision according to claim 1, characterized in that, In the said Step 2, preprocessing is also performed on the images of the conveyor belt and coal flow, and the preprocessing includes denoising and distortion correction.

3. A method for detecting the coal flow rate of a belt conveyor based on monocular vision according to claim 1 or 2, characterized in that, In the said Step 1, the camera internal parameter matrix K is: where f x , f y are the focal lengths in the horizontal and vertical directions on the imaging plane, (c x , c y ,) are the principal point coordinates; The camera external parameter matrix is: where R represents the rotation matrix, t represents the translation vector, and X C , Y C , Z C represent the three-dimensional coordinates in the camera coordinate system, and X w , Y w , Z w represent the three-dimensional coordinates in the world coordinate system.

4. A method for detecting the coal flow rate of a belt conveyor based on monocular vision according to claim 1 or 2, characterized in that, In the said Step 2, performing background segmentation and dynamic region extraction on the images of the conveyor belt and coal flow to obtain a coal flow foreground image specifically includes: Using the semantic segmentation technology based on nnUNet to train the image, construct a segmentation model adapted to the coal flow scenario, and then optimize the network architecture and hyperparameters of the segmentation model; Input the images of the conveyor belt and coal flow into the trained segmentation model to obtain a pixel-level segmentation result.

5. A method for detecting the coal flow rate of a belt conveyor based on monocular vision according to claim 1, characterized in that, The said Step 3 specifically is: Use the LOFTR module to perform feature point matching on the segmented coal flow and conveyor belt areas to generate matching pairs 6. The coal flow rate detection method for a belt conveyor based on monocular vision according to claim 3, characterized in that, In the said Step 3, after obtaining the matching points, double filtering is also performed on the matching points, and the double filtering includes: Eliminating incorrect matching points through simple similarity and spatial constraints; Based on the known movement parameters of the conveyor belt, eliminating the matching points that do not conform to the conveyor belt movement law.

7. A method for detecting the coal flow rate of a belt conveyor based on monocular vision according to claim 6, characterized in that, The eliminating the matching points that do not conform to the conveyor belt movement law based on the known movement parameters of the conveyor belt specifically is: The conveyor belt moves uniformly at a known speed v b in a certain direction. For the three-dimensional coordinates (X w , Y w , Z w ) of a certain point on the surface of the coal flow, the relationship projected onto the image coordinate system in the camera coordinate system can be expressed as: During the interval Δt from the k-th frame to the k+1-th frame, the world coordinates of a certain point on the conveyor belt relative to the camera undergo a known translation Δs = v b ·Δt. For correctly matched feature points, their displacements on the imaging plane should conform to the projection characteristics of the conveyor belt movement. Furthermore, false matches are further filtered in the set of matched points: the pixel coordinate displacements (Δu, Δv) of the matched points in adjacent frames are compared with the projection displacement range predicted by the conveyor belt movement, and the matched points with too large deviations from the predicted values are removed, thereby obtaining a set of reliable matched feature point pairs.

8. A method for detecting the coal flow rate of a belt conveyor based on monocular vision according to claim 3, characterized in that, The said Step 4 specifically is: The imaging of the k-th frame and the (k + 1)-th frame satisfies: In the normalized coordinate system, p k =(x k , y k , z k ) T and p k+1 =(x k+1 , y k+1 , z k+1 ) T correspond to the normalized coordinates of the imaging plane. Expand the above formula and use the inverse solution of the intrinsic matrix to obtain the relationship in the normalized coordinate system: Considering the transformation relationship between world coordinates and camera coordinates, given the movement displacement of the conveyor belt (ΔX w , ΔY w , ΔZ w ), rotation matrix R, translation vector t, conveyor belt speed v b , time interval Δt, using the displacement difference of feature points between two frames, by solving a system of equations for the depth Z w of the feature points and the corresponding X w , Y w are solved.

9. The belt conveyor coal flow detection method based on monocular vision according to claim 3, characterized in that The said Step 5 specifically is: By performing surface fitting on the three-dimensional coordinates of several feature points to reconstruct the continuous three-dimensional description of the coal flow surface, using Delaunay triangulation to fit the sparsely reconstructed three-dimensional feature points, and based on the contour data of the coal flow surface and the empty belt after fitting, calculate the cross-sectional area A of the coal flow at any specified cross-section cross ; Combined with the known speed of the conveyor belt, the volume flow rate of the coal flow through the cross-section is expressed as: Q v = A cross · v b Further combined with the coal density ρ, calculate the mass flow rate of the coal flow: Q m = ρ·Q v = ρ·A cross ·v b .

Images