Non-contact measurement-based combined detection device and method for the width and height of coal piles

Abstract

The present invention relates to the field of non-contact measurement technologies, and particularly to a combined detection device and method for the width and height of coal piles based on non-contact measurement. The present invention includes the following contents: S1. Use a monitoring device to collect real-time images of the coal dropping point and determine the positions of the binocular cameras; S2. Calibrate and adjust the binocular cameras to obtain standard original coal pile images of the left and right cameras, and then obtain the width of the coal pile by processing a single image; S3. Process the left and right coal pile images obtained in S2 according to the PSSNet network to obtain the actual height of the coal pile; S4. Combine the width and height indicators of the coal pile to jointly determine whether there is a coal pile. The specific judgment rules are as follows: Set a first threshold to judge whether the width exceeds the preset threshold. If it exceeds, execute the height coal pile strategy; if it does not exceed, repeat the operation of obtaining the coal pile width in S2. Compared with the prior art, the present invention provides a new coal pile detection device and adopts a non-contact measurement method combining monocular and binocular, effectively solving the problems that it is difficult for a monocular to achieve accurate coal pile detection and the large amount of calculation for binocular measurement of the coal pile height.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and particularly to a device and method for jointly detecting the width and height of a coal pile based on non-contact measurement. Background Art

[0002] The terrain of underground transportation is complex and the transportation distance is relatively long. Belt conveyors are widely used in underground transportation due to their large transportation capacity, long transportation distance, high efficiency, and continuous transportation characteristics. Since the underground transportation distance is too long, multiple belt conveyors need to be used for combined transportation. However, during the transfer process, the falling speed of coal is too fast, and it is easy to generate coal piles before the unloading conveyor belt has time to transport it away. At present, the detection of coal piles in mine belt conveyors mainly uses coal pile sensors. During the working process, the sensors are easily affected by harsh environments such as coal dust and moisture, and often cannot detect coal pile fault signals in time. The sensors are easily mis-triggered by falling materials, and the use effect and reliability are not very satisfactory. The sensor protection can only give an early warning after a coal pile occurs. After the early warning, the belt conveyor is stopped, and the accumulated materials need to be manually cleaned, which is extremely time-consuming and laborious. Therefore, early warning and early protection are very important.

[0003] With the development of industrial technology and the continuous progress of coal comprehensive automation, image detection technology based on machine vision has gradually become a new development direction. Machine vision perception technology, as a typical non-contact intelligent sensing technology, involves fields such as artificial intelligence, computer science, image processing, and pattern recognition. By simulating the visual function of humans, as an extension of the human eye, relying on excellent monitoring and detection calculation speed, accuracy, and high efficient transferability, it measures structured scenes from the images of objective things, extracts information, processes and understands it, and finally realizes the detection and control of specified targets. The application of machine vision perception technology in coal piles mainly uses monocular cameras.

[0004] There are mainly two methods for a monocular camera to detect a coal pile: directly measuring the height dimension of the coal pile and directly measuring the width dimension of the coal pile. To detect the height of the coal pile, by simply calibrating the camera, the conversion parameters from the world coordinate system to the image coordinate system are derived. However, there are large errors in the measurement of irregular objects such as coal piles by monocular measurement technology, and it is not easy to improve. To detect the width of the coal pile, the width of the coal is a regular plane, and the monocular measurement of the size is relatively accurate. However, simply exceeding the threshold of the width cannot directly be used as a judgment condition for a coal pile, and it can only be an early warning that there may be a coal pile. Measuring the height of the coal pile to detect the coal pile is a good direction. Binocular vision has certain advantages in obtaining the size of irregular objects, but the amount of calculation required for binocular detection is large. When using binocular height measurement, the existing stereo matching networks either cannot capture context information well to achieve accurate matching. To solve the above problems, the present invention also proposes a method for jointly detecting the width and height of a coal pile based on non-contact measurement. Summary of the Invention

[0005] The object of the present invention is to propose a combined detection device and method for the width and height of a coal pile based on non-contact measurement to solve the problems raised in the background art.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A combined detection method for the width and height of a coal pile based on non-contact measurement, including the following:

[0008] S1. Use a monitoring device to collect real-time coal dropping point images and determine the positions of the binocular cameras;

[0009] S2. Calibrate and calibrate the binocular cameras to obtain the original coal pile images of the standard left and right cameras, and then obtain the width of the coal pile by processing a single picture;

[0010] S3. Process the left and right coal pile images obtained in S2 according to the PSSNet network to obtain the actual height of the coal pile;

[0011] S4. Combine the width and height indicators of the coal pile to jointly judge whether there is a coal pile. The specific judgment rules are as follows:

[0012] Set a first threshold to judge whether the width exceeds the preset threshold. If it exceeds, execute the height coal pile strategy; if not, repeat the operation of obtaining the width of the coal pile in S2.

[0013] Preferably, the S1 specifically includes the following:

[0014] Use a monitoring device to obtain the position of the coal dropping point on the belt conveyor to obtain a set x of multiple possible coal dropping points;

[0015] Perform training processing on the obtained set x of coal dropping points and predict the best coal dropping point as the target point;

[0016] Install and set binocular cameras at a position 10 - 30 cm away from the best coal dropping point along the running direction of the belt.

[0017] Preferably, the conveyor belt of the belt conveyor includes an upper belt and a lower belt, and the coal dropping point is located on the lower belt, indicating the falling point where the coal on the upper belt is transported to the lower belt.

[0018] Preferably, the S2 specifically includes the following:

[0019] Obtain the original coal pile images of the standard left and right cameras through camera calibration and zero value calibration, and use the YOLOv5 algorithm to detect the width of the coal pile image obtained by the camera.

[0020] Preferably, the camera calibration and zero calibration are implemented based on the Zhang Zhengyou calibration method, and the specific operations are as follows:

[0021] Use the Camera Calibrator module in Matlab to calibrate the internal and external parameters of the binocular camera, and obtain the focal length f and the baseline B of the camera;

[0022] Use the calibrated binocular camera to obtain the conveyor belt image when it is unloaded, detect the distance from the bottom of the belt groove to the camera, and adjust the distance from the bottom of the belt to the camera to zero by the relative distance method according to the actual distance;

[0023] After the adjustment is completed, take the original images of the left and right cameras when piling up coal.

[0024] Preferably, the modified YOLOv5 algorithm is used to obtain the image width, and the specific operations are as follows:

[0025] The YOLOv5 algorithm is an object detection algorithm. It is trained using a coal dataset. When testing, it can detect coal piles and form prior boxes. The width of the prior box is approximately equal to the width of the coal pile. The pixel coordinates of each corner of the rectangular prior box can be obtained through the YOLOv5 algorithm, and thus the pixel width of the prior box can be obtained. According to the principle of pinhole imaging, the actual width of the prior box can be calculated. The formula is:

[0026] W = (D × P) / F

[0027] W is the actual width, D is the distance from the camera to the belt (known), P is the pixel width, and F is the focal length (known when calibrating the camera). Writing the formula into the YOLOv5 algorithm can obtain the actual width of the coal pile.

[0028] Preferably, the specific content of S3 is as follows:

[0029] Use the stereo matching algorithm PSSNet to match the original images obtained by the left and right cameras to obtain a disparity map, obtain the depth value of each pixel point through the principle of triangulation, and obtain the depth value of the highest point;

[0030] The network structure of the stereo matching algorithm PSSNet is divided into:

[0031] Feature extraction module, and the feature extraction uses the pyramid squeeze attention structure;

[0032] Cost volume construction module, and use the feature maps obtained from the left and right images after passing through the feature extraction network to construct Cost Volume;

[0033] Feature fusion module, and use 3D convolution to extract the information between the left and right feature maps and different disparity levels to obtain the Cost Volume after feature fusion;

[0034] The parallax calculation module upsamples the Cost Volume after feature fusion to the original resolution and finds the parallax value with the smallest matching error.

[0035] Preferably, the pyramid squeeze attention structure of the feature extraction module is specifically composed of a new residual block with a pyramid structure, namely PSABlock. The PSA module divides the input feature map into S groups along the channel dimension. Each group uses convolutions with different kernel sizes to construct a pyramid structure, and then obtains channel attention values through a channel attention mechanism. In this way, PSA fuses context information of different scales and generates better pixel-level attention. The PSA module is used to replace the first and second convolutional blocks in the residual block to form a new residual block structure, PSABlock; the new feature extraction module is composed of four residual block structures, PSABlock, with channel numbers of 32, 64, 128, and 128 respectively.

[0036] Preferably, the specific content of S4 is as follows:

[0037] Set 80% of the maximum coal stacking width when operating normally without coal accumulation as the first threshold, and determine whether the obtained coal stacking width data exceeds the first threshold. If it exceeds the first threshold, issue a warning to remind the staff and execute the high coal stacking strategy; if it does not exceed the first threshold, repeat the operation of obtaining the coal stacking width in S2.

[0038] Preferably, the high coal stacking strategy specifically means: setting a coal stacking height threshold according to national standards and the actual scenario, referred to as the second threshold; comparing the coal stacking height obtained through the PSSNet network with the second threshold, and if it exceeds the threshold, send a protection signal and stop the conveyor belt.

[0039] Furthermore, it protects a non-contact measurement-based combined detection device for coal stacking width and height that applies the above method.

[0040] Compared with the prior art, the present invention provides a non-contact measurement-based combined detection method for coal stacking width and height, having the following beneficial effects:

[0041] (1) Based on the non-contact measurement method to measure the coal stacking width and height, the present invention proposes a new method for jointly discriminating the coal stacking situation by combining single-eye width recognition and binocular height recognition, solving the problem that it is difficult to achieve accurate coal stacking detection with a single eye; at the same time, it also adopts a method combined with width recognition, reducing the problem of large computational complexity in binocular measurement and realizing coal stacking detection.

[0042] (2) The present invention uses a stereo matching algorithm to measure the coal stacking height and proposes a stereo matching algorithm, PSSNet, based on a pyramid squeeze attention network.

[0043] (3) The proposed PSSNet algorithm of the present invention combines the pyramid structure and the residual structure, and uses the context information to enrich the feature map, which can help the correspondence estimation and significantly improve the matching in the ill-posed region. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flowchart of a method for jointly detecting the width and height of stockpiled coal based on non-contact measurement mentioned in Embodiment 1 of the present invention;

[0045] Figure 2 It is a schematic structural diagram of a belt conveyor transportation system and a device for jointly detecting the width and height of stockpiled coal based on non-contact measurement mentioned in Embodiment 1 of the present invention;

[0046] Figure 3 It is a schematic diagram of the placement of the real-time coal dropping point and the binocular camera mentioned in Embodiment 1 of the present invention;

[0047] Figure 4 It is a schematic diagram of the network structure of the stereo matching algorithm PSSNet mentioned in Embodiment 1 of the present invention;

[0048] Figure 5 It is a schematic diagram of the structure of the new residual block structure PSABlock mentioned in Embodiment 1 of the present invention;

[0049] Figure 6 It is a schematic diagram of the structure of the PSA module mentioned in Embodiment 1 of the present invention;

[0050] Figure 7 It is a result diagram of the width of stockpiled coal measured by using the modified YOLO v5 algorithm mentioned in Embodiment 1 of the present invention;

[0051] Figure 8 It is a result diagram of the height measured by using the stereo matching algorithm PSSNet mentioned in Embodiment 1 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] In order to make the objectives and advantages of the present invention clearer and more understandable, the present invention will be further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention and do not limit the protection scope of the present invention.

[0054] Embodiment 1:

[0055] Please refer to Figure 1, the present invention proposes a combined detection method for the width and height of coal piles based on non-contact measurement, including the following content:

[0056] Step 1: The monitoring device collects real-time images of the coal dropping point and determines the position of the binocular camera. The specific method is as follows:

[0057] Please refer to Figures 2-3 , the monitoring facility is a part of the conveyor belt transportation system and is installed above the conveyor belt to achieve image acquisition in the entire area; since there is a coal dropping point during the hierarchical transportation of the belt conveyor, and coal piles are likely to appear at the coal dropping point due to the change in the speed of the belt conveyor, it is necessary to monitor through the monitoring device to find the coal dropping point. Generally, the point vertically corresponding to the highest point of the piled material is selected as the coal dropping point. Since there is falling coal material on one side of the coal dropping point close to the upper belt, it affects the accuracy of obtaining the depth values of each point by stereo matching and it is not easy to identify the true height of the coal pile. Generally, a binocular camera is set at 20 cm from the best coal dropping point along the running direction of the belt. The specific setting is as Figures 2-3 shown.

[0058] The method for obtaining the coal dropping point image in Step 1 of this embodiment is as follows: First, obtain the simulated coal dropping point information on the conveyor belt in the historical conveyor transportation system to determine the predicted coal dropping point data set x; then, train the predicted coal dropping point data set x to determine the position of the best predicted coal dropping point; finally, set the camera according to the position of the coal dropping point on the conveyor belt.

[0059] Step 2: Calibrate the camera to obtain the standard original images of the coal pile for the left and right cameras, and obtain the width of the coal pile. The specific method is as follows:

[0060] To obtain the standard original images, camera calibration and zero-value calibration are required. Camera calibration is the basis for image distortion correction and stereo calibration. The key to obtaining depth information through stereo matching lies in image distortion correction and stereo calibration. Eliminating image distortion can enable the machine to better identify the feature information of the image, and performing stereo calibration can make the imaging planes of the two images parallel. Zero-value calibration can convert the obtained depth information into height information. After camera calibration and zero-value calibration, the original pictures of the left and right cameras are obtained. Then, first detect the width of the coal pile for the picture obtained by the right camera. The width detection uses the YOLO v5 algorithm to detect coal, and the principle of small hole imaging is used to calculate the width of the detection box instead of the width of the coal pile, as Figure 7 shown as the width detection result.

[0061] The camera calibration in the second step of this example mainly includes distortion correction and stereo rectification. Distortion correction is achieved by establishing mathematical models for radial and tangential distortions and then combining them into a polynomial distortion mathematical model, and the image can be corrected through a calibration lattice. For stereo rectification, the Bouguet stereo rectification algorithm is used to rectify the left and right images obtained above. After rectification, the camera focal length f and the baseline length B are obtained. For zero calibration, first take an image when the conveyor belt is empty, perform stereo matching to obtain the depth from the lowest point of the belt to the camera, and set the height of this point to zero, which is called the zero point. When taking pictures with coal piles, the height is obtained by subtracting the depth of the highest point from the depth of the zero point.

[0062] Step 3: Process the left and right coal pile images according to the PSSNet network to obtain the actual coal pile height, specifically as follows:

[0063] First, a convolutional neural network is used to extract the features of the two images. After feature extraction, the features of the two images are cascaded to construct a 4D matching cost volume. The cost volume is regularized through 3D convolution. The actual effect of the regularization module directly affects the matching accuracy. After regularization, the softmax operation is used to calculate the probability of the corresponding pixel disparity, and the true disparity value is calculated and displayed using the disparity regression function. After obtaining the disparity value, the depth value is obtained using the principle of triangulation. The formula is:

[0064] D = (f × B) / d

[0065] where D is the depth value and d is the disparity value. The structure diagram of the PSSNet network is as Figure 4 shown. In the feature extraction module, it consists of three convolutions with a kernel size of 3*3 and 32 channels and four residual blocks with 32, 64, 128, and 128 channels respectively. In the residual block, the PSA module is used to replace the first and second convolutional blocks in the residual block to form a new residual block structure PSABlock, as Figure 5 shown. The PSA module is as Figure 6 shown. By splicing the convolution results of different-sized convolutional kernels, a pyramid-shaped feature map is formed, and then an attention mechanism is applied to this feature map to extract more abundant feature information.

[0066] Step 4: Combine the width and height of the coal pile to jointly determine whether there is a coal pile, specifically as follows:

[0067] Since the algorithm for the coal pile height is relatively cumbersome and requires a large amount of calculation, while the acquisition of the coal pile width is relatively simple but the detection is inaccurate and cannot be used as the sole standard for coal pile detection, a joint judgment method using width and height is adopted. First, a threshold is set, and the obtained coal pile width value is input to determine whether the coal pile width exceeds the threshold. The width result is as Figure 7As shown, if it is a warning to remind the staff and conduct high-level coal stacking detection. If not, continue to run. To conduct high-level coal stacking detection, a height threshold needs to be set, and the height value obtained by using the stereo matching algorithm is input to determine whether the coal stacking height exceeds the threshold. The height detection result is as Figure 8 shown. If it is, alarm and stop the belt conveyor. If not, continue to run. The specific identification steps are as Figure 1 shown.

[0068] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered within the protection scope of the present invention.

Claims

1. A combined detection method for the width and height of coal piles based on non-contact measurement, characterized in that It includes the following contents: S1. Use a monitoring device to collect real-time coal dropping point images and determine the positions of binocular cameras; S2. Calibrate and calibrate the binocular cameras to obtain standard original coal stacking images of the left and right cameras, and then obtain the coal stacking width by processing single pictures; S3. Process the left and right coal stacking images obtained in S2 according to the PSSNet network to obtain the actual coal stacking height; specifically, it includes the following contents: Obtain a disparity map by matching the original images obtained by the left and right cameras through the stereo matching algorithm PSSNet, obtain the depth values of each pixel point through the principle of triangulation, and obtain the depth value of the highest point; The network structure of the stereo matching algorithm PSSNet is divided into: Feature extraction module, and pyramid squeeze attention structure is used for feature extraction; the pyramid squeeze attention structure is specifically composed of a new residual block PSABlock with a pyramid structure; PSABlock divides the input feature map into S groups from the channel dimension, and each group uses convolutions with different kernel sizes to construct a pyramid structure, and then obtains channel attention values through the channel attention mechanism. By doing so, PSABlock fuses context information of different scales and generates better pixel-level attention; use PSABlock to replace the first and second convolutional blocks in the residual block to form a new residual block structure; use the PSABlock module to replace the residual structure and pyramid structure in PSMNet, and the new feature extraction module is composed of four residual block structures PSABlock with channel numbers of 32, 64, 128, and 128 respectively; Cost volume construction module, construct CostVolume using the feature maps obtained from the left and right images after passing through the feature extraction network; Feature fusion module, use 3D convolution to extract information between the left and right feature maps and different disparity levels to obtain the Cost Volume after feature fusion; Disparity calculation module, upsample the Cost Volume after feature fusion to the original resolution and find the disparity value with the smallest matching error; S4. Combine the width and height indicators of the coal stack to jointly judge whether coal stacking occurs. The specific judgment rules are as follows: Set a first threshold, judge whether the width exceeds the preset threshold. If it exceeds, execute the height coal stacking strategy; if it does not exceed, repeat the operation of obtaining the coal stacking width in S2.

2. The combined detection method for the width and height of coal piles based on non-contact measurement according to claim 1, characterized in that The specific content of S1 is as follows: Use a monitoring device to obtain the positions of coal dropping points on the belt conveyor to obtain a set x of multiple possible coal dropping points; Perform training processing on the obtained coal dropping point set x and predict the best coal dropping point as the target point; Install and set a binocular camera at a position 10 - 30 cm away from the best coal dropping point along the running direction of the belt.

3. The combined detection method for the width and height of coal piles based on non-contact measurement according to claim 2, characterized in that, The conveyor belt of the belt conveyor includes an upper belt and a lower belt, and the coal dropping point is located on the lower belt, indicating the landing point where the coal on the upper belt is transported to the lower belt.

4. The combined detection method for the width and height of coal piles based on non-contact measurement according to claim 1, characterized in that The specific content of S2 is as follows: Obtain standard original coal stacking images of the left and right cameras through camera calibration and zero value calibration, and use the modified YOLOv5 algorithm to detect the coal stacking width of the coal stacking images obtained by the cameras.

5. The combined detection method for the width and height of coal piles based on non-contact measurement according to claim 4, wherein, The camera calibration and zero-value calibration are implemented based on Zhang Zhengyou's calibration method, and the specific operations are as follows: Use the Camera Calibrator module in Matlab to complete the calibration of the internal and external parameters of the binocular camera, and obtain the focal length of the camera and the baseline B; Use the calibrated binocular camera to obtain the conveyor belt image when it is unloaded, detect the distance from the bottom of the belt groove to the camera, and adjust the distance from the bottom of the belt to the camera to zero through the relative distance method according to the actual distance; After the adjustment is completed, take the original images of the left and right cameras when coal is piled up.

6. The combined detection method for the width and height of coal piles based on non-contact measurement according to claim 4, characterized in that The specific operation of obtaining the image width using the modified YOLOv5 algorithm is as follows: The YOLOv5 algorithm is an object detection algorithm. It is trained using a coal dataset. When testing, it detects the coal pile and forms a priori boxes. The width of the priori box is approximately equal to the width of the coal pile; the pixel coordinates of each corner of the rectangular priori box are obtained through the YOLOv5 algorithm, and thus the pixel width of the priori box is obtained; according to the principle of pinhole imaging, the actual width of the priori box is calculated. The specific calculation formula is: W=(D×P) / F Where W is the actual width, D is the distance from the camera to the belt, P is the pixel width, and F is the focal length; write the formula into the YOLOv5 algorithm to obtain the actual width of the piled coal.

7. The combined detection method for the width and height of coal piles based on non-contact measurement according to claim 1, wherein The specific content of S4 is as follows: Set 80% of the maximum coal stacking width when the coal conveyor is running normally without coal piling as the first threshold, and judge whether the obtained coal piling width data exceeds the first threshold. If it exceeds the first threshold, issue a warning to remind the staff and execute the high coal piling strategy; if it does not exceed the first threshold, repeat the operation of detecting the coal piling width in S2.

8. The combined detection method for the width and height of coal piles based on non-contact measurement according to claim 7, wherein The high coal piling strategy specifically refers to: setting the coal piling height threshold according to national standards and the actual scenario, which is called the second threshold; comparing the coal piling height obtained through the PSSNet network with the second threshold. If it exceeds the threshold, send a protection signal and stop the conveyor belt.

Images