Scraper conveyor upward-moving and downward-sliding monitoring method and device based on point cloud registration

Abstract

The embodiment of the invention provides a method and device for monitoring upward and downward movement of a scraper conveyor based on point cloud registration, and the method comprises the steps: setting a target on the wall of a transportation roadway, obtaining a depth image of a target region, and carrying out the filtering and hole filling of the depth image of the target region, and obtaining a processed depth image of the target region; based on a world coordinate system established by taking image acquisition equipment as an original point, carrying out coordinate conversion on pixel coordinates and depth values in the depth image of the processed target area to obtain a three-dimensional point cloud; sequentially carrying out denoising, segmentation, down-sampling and clustering division processing on the three-dimensional point cloud to obtain a first target point cloud; based on the obtained target template point cloud and the first target point cloud, template point cloud features and target point cloud features are extracted; registering the target point cloud features and the template point cloud features for multiple times to obtain registered target point cloud; and based on the registered target point cloud and the world coordinate system, the upward-sliding and downward-sliding offset of the scraper conveyor is determined.

Description

Technical Field

[0001] The present invention relates to the field of mine monitoring safety, and in particular to, but not limited to, a method and device for monitoring the upward and downward movement of a scraper conveyor based on point cloud registration. Background Art

[0002] As the running track of the coal mining machine and the main equipment for mine transportation, the scraper conveyor plays an important role in the modern coal mining process. In the actual working process, whether it is a working face with a large inclination angle, a working face with a small inclination angle or even a nearly horizontal working face, the scraper conveyor has problems of ups and downslides to varying degrees. Under the action of external forces such as gravity and thrust, the scraper conveyor will generate an "upward" or "downward" component in the inclination direction of the coal mining working face, resulting in unreasonable overlap between the head of the scraper conveyor and other equipment on the comprehensive mining face, leading to the instability of the supporting structure of the "three machines" and the deflection of the traction hydraulic support, which will cause the working conditions to deteriorate and weaken the support effect of the roof, and ultimately cause safety hazards. Underground workers need to stop the machine many times to deal with problems such as "pulling back coal" and "transfer overlap" caused by ups and downslides. Frequent addition and subtraction of chutes not only increases the labor intensity of the maintenance team workers, but also affects the coal mining operations of the production team. The greater the inclination of the coal seam, the more difficult it is to control the front scraper conveyor. When ups and downslides occur, a large amount of manpower and material resources are required for fault diagnosis and equipment maintenance, which will greatly reduce the production efficiency of the coal mine.

[0003] The normal operation of the scraper conveyor is the key to safe and efficient mining in the fully mechanized working face. It is particularly important to avoid the problem of scraper conveyor sliding up and down. With the great progress made in the research of coal mine machinery and equipment in recent years, the existing methods for measuring the displacement of sliding up and down are mainly distributed in the field of laser ranging. The limitations of laser measurement in complex environments are large. The measurement data will fluctuate greatly due to the uneven surface of the tunnel wall and the vibration of the coal mining machine. When there are foreign objects blocking it, it is impossible to distinguish whether the obtained data is valid monitoring data, which is easy to cause erroneous information feedback. The reliability of the obtained data is low, and the measurement results lack universality and three-dimensional information. Summary of the invention

[0004] Based on the problems in the related art, an embodiment of the present invention provides a method and device for monitoring the upward and downward movement of a scraper conveyor based on point cloud registration.

[0005] The technical solution of the embodiment of the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides a method for monitoring the upward and downward movement of a scraper conveyor based on point cloud registration, the method comprising:

[0007] A target is set on the wall of the transport lane, a depth image of the target area is obtained, and the depth image of the target area is filtered and hole-filled to obtain a processed depth image of the target area;

[0008] Based on a world coordinate system established with the image acquisition device as the origin, coordinate transformation is performed on the pixel coordinates and depth values ​​in the processed depth image of the target area to obtain a three-dimensional point cloud;

[0009] The three-dimensional point cloud is processed by denoising, segmenting, downsampling and clustering in sequence to obtain a first target point cloud;

[0010] Based on the acquired target template point cloud and the first target point cloud, extract template point cloud features and target point cloud features; and perform multiple registrations on the target point cloud features and the template point cloud features to obtain a registered target point cloud;

[0011] Based on the registered target point cloud and the world coordinate system, the upward and downward displacement of the scraper conveyor is determined.

[0012] In a second aspect, an embodiment of the present invention provides a device for monitoring the upward and downward movement of a scraper conveyor based on point cloud registration, the device comprising:

[0013] An acquisition module is used to set a target on the wall of the transport lane, acquire a depth image of the target area, and filter and fill holes in the depth image of the target area to obtain a processed depth image of the target area;

[0014] A conversion module, for performing coordinate conversion on pixel coordinates and depth values ​​in the processed target area depth image based on a world coordinate system established with the image acquisition device as the origin, to obtain three-dimensional point cloud data;

[0015] A processing module, used for sequentially performing denoising, segmentation, downsampling and clustering processing on the three-dimensional point cloud to obtain a first target point cloud;

[0016] A registration module, for extracting template point cloud features and target point cloud features based on the acquired target template point cloud and the first target point cloud; and performing multiple registrations on the target point cloud features and the template point cloud features to obtain a registered target point cloud;

[0017] The determination module is used to determine the upward and downward sliding offset of the scraper conveyor based on the aligned target point cloud and the world coordinate system.

[0018] In some embodiments, the acquisition module is also used to perform median filtering on the target area depth image to obtain a target area depth image from which noise is removed; fill the hole information in the target area depth image from which noise is removed through a preset image restoration algorithm, and set a search radius for restoration to obtain the processed target area depth image.

[0019] In some embodiments, the acquisition module is also used to obtain the intrinsic parameter matrix and size information of the image acquisition device; create a camera coordinate system based on the intrinsic parameter matrix and the size information; use a preset function to establish a transformation matrix between the camera coordinate system and the world coordinate system; based on the transformation matrix, perform coordinate transformation on the pixel coordinates and depth values ​​in the processed target area depth image to obtain the three-dimensional point cloud.

[0020] In some embodiments, the processing module is also used to perform statistical outlier removal and denoising processing on the three-dimensional point cloud to obtain a denoised three-dimensional point cloud; use a preset random sampling iterative algorithm to segment the denoised three-dimensional point cloud to obtain a segmented three-dimensional point cloud; perform voxel grid downsampling processing on the segmented three-dimensional point cloud to obtain a downsampled three-dimensional point cloud; perform point cloud clustering processing on the downsampled three-dimensional point cloud to obtain the first target point cloud.

[0021] In some embodiments, the registration module is further used to use a preset KD tree search algorithm to traverse the neighboring points of each point in the target template point cloud and the first target point cloud, determine the local plane based on a preset search radius and a maximum number of neighboring points, and estimate the normal of the local plane based on the direction and position of the neighboring points; perform FPFH feature extraction on the target template point cloud and the first target point cloud based on the normal to obtain local features of the target template point cloud and local features of the first target point cloud; use a preset random sampling iterative algorithm, based on a preset maximum distance threshold, to perform FPFH feature extraction on the target template point cloud and the first target point cloud The features and local features of the first target point cloud are randomly sampled to obtain a point cloud after successful feature matching; a rigid body transformation is performed on the point cloud after successful feature matching, and it is determined whether the transformed feature point cloud is within the preset maximum distance threshold, and the initial registration position is determined by iteratively querying the transformation with the most feature matching points; based on the initial registration position and the maximum allowed distance, the nearest neighbor point pair between the target template point cloud and the first target point cloud is determined, and the optimal matching parameters of the nearest neighbor point pair are calculated by a preset optimization error function to ensure that the difference between the nearest neighbor point pairs is minimized.

[0022] In some embodiments, the determination module is also used to fit the registered target point cloud to a spherical surface, and define the average value of the coordinates of all points in the registered target point cloud as the initial center of the sphere, and define the average value of the average center distance of all points in the registered target point cloud as the initial radius; calculate the initial Euclidean distance from each point to the initial center of the sphere; calculate the difference between the Euclidean distance and the initial radius to obtain a residual vector; use the least squares method to optimize the minimum sum of squares of the residual function to obtain the center coordinates and radius; based on the world coordinate system, calculate the real-time monitoring distance between the center coordinates and the coordinate origin; calculate the difference between the real-time monitoring distance and the initial Euclidean distance, and determine the upward and downward offset of the scraper conveyor based on the difference.

[0023] In the third aspect, an embodiment of the present invention provides a scraper conveyor up-and-down sliding monitoring device based on point cloud registration, comprising: a memory for storing executable instructions; and a processor for executing the executable instructions stored in the memory to implement the above-mentioned scraper conveyor up-and-down sliding monitoring method based on point cloud registration.

[0024] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned method for monitoring the upward and downward movement of a scraper conveyor based on point cloud registration.

[0025] The method and device for monitoring the ups and downslides of a scraper conveyor based on point cloud registration provided by the embodiment of the present invention, on the one hand, use a three-axis pan-tilt auxiliary installation when real-time monitoring of the depth image of the target area, add a timer to extract frame images at intervals, and overcome the influence of the jitter of the scraper conveyor; on the other hand, use a median filtering method to denoise the depth image, use a Telea algorithm to fill the depth image, and use a statistical outlier denoising method to remove the point cloud image noise, thereby solving the negative impact of environmental factors on the point cloud quality and obtaining a point cloud image with better quality; thirdly, use a variety of preprocessing methods to process the target point cloud, extract point cloud features through the FPFH algorithm, and RANSAC coarse registration provides an initial position for ICP fine registration. The final point cloud matching accuracy can reach more than 95%, and real-time monitoring of the ups and downslides of the scraper conveyor can effectively reduce the occurrence of unsafe accidents underground and improve mine mining efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of a method for monitoring the upward and downward movement of a scraper conveyor based on point cloud registration provided by an embodiment of the present invention;

[0027] Figure 2 It is a flow chart of another method for monitoring the upward and downward movement of a scraper conveyor based on point cloud registration provided by an embodiment of the present invention;

[0028] Figure 3 It is a schematic diagram of the equipment arrangement structure provided by an embodiment of the present invention;

[0029] Figure 4 is a target area depth image obtained according to an embodiment of the present invention;

[0030] Figure 5 This is a comparison diagram of the hole filling effect using the Telea algorithm provided by an embodiment of the present invention;

[0031] Figure 6 is a schematic diagram of a world coordinate system provided by an embodiment of the present invention;

[0032] Figure 7 is a point cloud image converted from a depth image provided by an embodiment of the present invention;

[0033] Figure 8 This is a denoising effect diagram using a statistical outlier algorithm provided by an embodiment of the present invention;

[0034] Fig. 9 This is an effect diagram of segmenting a target point cloud using the RANSAC algorithm provided by an embodiment of the present invention;

[0035] Fig.10 This is a processing effect diagram using voxel grid downsampling provided by an embodiment of the present invention;

[0036] Fig.11 This is a diagram showing the effect of using DBSCAN clustering processing provided by an embodiment of the present invention;

[0037] Fig.12 This is a rough RANSAC registration effect diagram of the template point cloud and the target point cloud provided by an embodiment of the present invention;

[0038] Fig.13 is a flow chart of ICP algorithm precise registration provided by an embodiment of the present invention;

[0039] Fig.14 This is an ICP precise registration effect diagram provided by an embodiment of the present invention;

[0040] Fig.15 This is the final effect diagram of point cloud recognition and registration provided by an embodiment of the present invention;

[0041] Fig.16 A schematic diagram of the composition structure of a scraper conveyor up-and-down sliding monitoring device based on point cloud registration provided in an embodiment of the present invention;

[0042] Fig.17 A schematic diagram of the composition structure of a scraper conveyor up-and-down sliding monitoring device based on point cloud registration provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0044] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present invention have the same meaning as those commonly understood by those skilled in the art to which the embodiments of the present invention pertain. The terms used in the embodiments of the present invention are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0045] The following describes an exemplary application of the scraper conveyor up-and-down monitoring device based on point cloud registration according to an embodiment of the present invention. The scraper conveyor up-and-down monitoring device based on point cloud registration provided by the embodiment of the present invention can be implemented as a terminal or a server. In one implementation, the scraper conveyor up-and-down monitoring device based on point cloud registration provided by the embodiment of the present invention can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, and mobile devices; in another implementation, the scraper conveyor up-and-down monitoring device based on point cloud registration provided by the embodiment of the present invention can also be implemented as a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiment of the present invention. Next, an exemplary application of the upward and downward sliding monitoring device for a scraper conveyor based on point cloud registration when it is implemented as a server will be described.

[0046] The embodiment of the present invention provides a method for monitoring the upward and downward movement of a scraper conveyor based on point cloud registration, see Figure 1 , Figure 1 is a flow chart of a method for monitoring the upward and downward movement of a scraper conveyor based on point cloud registration provided by an embodiment of the present invention, which is combined with Figure 1 The steps shown are explained.

[0047] Step S110, setting a target on the wall of the transport lane, acquiring a depth image of the target area, and filtering and filling holes in the depth image of the target area to obtain a processed depth image of the target area.

[0048] In some embodiments, multiple targets may be set on the wall of the transport lane, and a specific depth imaging device (such as a lidar, a structured light camera, etc.) may be used to photograph the target area to obtain a depth image of the area.

[0049] In some embodiments, in the depth image, filtering can smooth the fluctuations of the depth value and reduce errors caused by factors such as device accuracy and ambient light. Here, the filtering method may include mean filtering, median filtering, Gaussian filtering, etc., and the filtering method depends on the characteristics of the depth image and the type of noise. In the depth image, some holes or missing areas may appear due to factors such as device limitations, occlusions, or lighting conditions. The purpose of hole filling is to fill these missing areas through interpolation or other methods to make the depth image more complete and accurate. The hole filling method can be based on interpolation of the depth values ​​of neighboring pixels, or it can use a more complex image restoration algorithm.

[0050] In the present invention, targets are set at a certain distance on the wall of the transport lane. The present invention uses a three-axis gimbal bracket to install a depth camera to assist in image acquisition. The depth camera is used to obtain the depth image of the area where the spherical target is located, and the median filter is used for denoising and hole filling to ensure the integrity of the depth image information; the depth image is obtained, the depth image is filtered, and the hole filling is used to ensure the integrity of the image information.

[0051] Step S120 , based on a world coordinate system established with the image acquisition device as the origin, coordinate transformation is performed on the pixel coordinates and depth values ​​in the processed target area depth image to obtain a three-dimensional point cloud.

[0052] In the embodiment of the present invention, the world coordinate system (XYZ axis) is established with the image acquisition device as the origin, the X axis and the Y axis represent the horizontal direction and the vertical direction respectively, and the Z axis represents the depth direction, that is, the distance from the camera to the target.

[0053] In a depth image, each pixel has a corresponding coordinate (u, v) and a depth value z. The coordinate (u, v) indicates the position of the pixel in the image, while the depth value z indicates the depth (i.e., distance) of the point corresponding to the pixel in three-dimensional space.

[0054] In the present invention, the pixel coordinates and depth values ​​in the depth image are converted into point cloud data in three-dimensional space, and coordinate transformation is required. This process involves converting the image coordinate system (uv coordinate system) into the world coordinate system (XYZ coordinate system). This can be achieved by parsing the intrinsic parameter matrix and extrinsic parameter matrix of the camera. The intrinsic parameter matrix is ​​used to describe the internal properties of the camera (such as focal length, optical center position, etc.), while the extrinsic parameter matrix describes the position and direction of the camera in the world coordinate system. Through coordinate transformation, each pixel in the depth image can be converted into a point in three-dimensional space, and its XYZ coordinates are obtained. The collection of these points constitutes three-dimensional point cloud data.

[0055] In the present invention, a world coordinate system is established with the camera as the origin, and the pixel coordinates and depth values ​​in the depth image are converted into (x, y, z) coordinates in three-dimensional space to generate a data set containing a large number of three-dimensional points.

[0056] Step S130, performing denoising, segmentation, downsampling and clustering processing on the three-dimensional point cloud in sequence to obtain a first target point cloud.

[0057] In an embodiment of the present invention, the converted point cloud image is pre-processed by denoising, segmentation, downsampling, and clustering, and the target point cloud is segmented therefrom. Furthermore, the point cloud data is denoised to reduce the noise point cloud existing after the depth image is converted into the point cloud image. The denoised point cloud is segmented to extract the target point cloud separately. The target point cloud is down-sampled to reduce the number of point clouds and improve the running speed of subsequent steps. The down-sampled point cloud is clustered to further screen the target point cloud.

[0058] Step S140, based on the acquired target template point cloud and the first target point cloud, extract template point cloud features and target point cloud features; and perform multiple registrations on the target point cloud features and the template point cloud features to obtain a registered target point cloud.

[0059] Step S150, determining the upward and downward sliding offset of the scraper conveyor based on the registered target point cloud and the world coordinate system.

[0060] The method and device for monitoring the ups and downslides of a scraper conveyor based on point cloud registration provided by the embodiment of the present invention, on the one hand, use a three-axis pan-tilt auxiliary installation when real-time monitoring of the depth image of the target area, add a timer to extract frame images at intervals, and overcome the influence of the jitter of the scraper conveyor; on the other hand, use a median filtering method to denoise the depth image, use a Telea algorithm to fill the depth image, and use a statistical outlier denoising method to remove the point cloud image noise, thereby solving the negative impact of environmental factors on the point cloud quality and obtaining a point cloud image with better quality; thirdly, use a variety of preprocessing methods to process the target point cloud, extract point cloud features through the FPFH algorithm, and RANSAC coarse registration provides an initial position for ICP fine registration. The final point cloud matching accuracy can reach more than 95%, and real-time monitoring of the ups and downslides of the scraper conveyor can effectively reduce the occurrence of unsafe accidents underground and improve mine mining efficiency.

[0061] In some embodiments, the above step S110 may be implemented in the following manner:

[0062] First, the target area depth image is subjected to median filtering to obtain a target area depth image with noise removed. Secondly, the hole information in the target area depth image with noise removed is filled by a preset image restoration algorithm, and a search radius is set for restoration to obtain the processed target area depth image.

[0063] In some embodiments, the above step S120 can be implemented by following the steps S121 to S124:

[0064] Step S121, obtaining the intrinsic parameter matrix and size information of the image acquisition device.

[0065] Step S122: Create a camera coordinate system based on the intrinsic parameter matrix and the size information.

[0066] Step S123: using a preset function to establish a conversion matrix between the camera coordinate system and the world coordinate system.

[0067] Step S124 , based on the transformation matrix, coordinate transformation is performed on the pixel coordinates and depth values ​​in the processed target area depth image to obtain the three-dimensional point cloud.

[0068] In some embodiments, the above step S130 can be implemented by the following steps S131 to S134:

[0069] Step S131 , performing statistical outlier removal and denoising processing on the three-dimensional point cloud to obtain a denoised three-dimensional point cloud.

[0070] Step S132: using a preset random sampling iterative algorithm to segment the denoised three-dimensional point cloud to obtain a segmented three-dimensional point cloud.

[0071] Step S133 , performing voxel grid downsampling processing on the segmented three-dimensional point cloud to obtain a down-sampled three-dimensional point cloud.

[0072] Step S134, performing point cloud clustering processing on the downsampled three-dimensional point cloud to obtain the first target point cloud.

[0073] In some embodiments, the above step S140 can be implemented by the following steps S141 to S143:

[0074] Step S141, using a preset KD tree search algorithm to traverse the neighboring points of each point in the target template point cloud and the first target point cloud, determine the local plane based on a preset search radius and a maximum number of neighboring points, and estimate the normal of the local plane based on the direction and position of the neighboring points; based on the normal, perform FPFH feature extraction on the target template point cloud and the first target point cloud to obtain local features of the target template point cloud and local features of the first target point cloud.

[0075] Step S142, using a preset random sampling iterative algorithm, based on a preset maximum distance threshold, randomly sample the local features of the target template point cloud and the local features of the first target point cloud to obtain a point cloud after successful feature matching; perform a rigid body transformation on the point cloud after successful feature matching, and determine whether the transformed feature point cloud is within the preset maximum distance threshold, and determine the initial alignment position by iteratively querying the transformation with the most feature matching points.

[0076] Step S143, based on the initial registration position and the maximum allowed distance, determine the nearest neighbor point pair between the target template point cloud and the first target point cloud, and calculate the optimal matching parameters of the nearest neighbor point pair through a preset optimization error function to ensure that the difference between the nearest neighbor point pairs is minimized.

[0077] In some embodiments, the above step S150 can be implemented in the following manner:

[0078] First, fit the registered target point cloud to a sphere, and define the average value of all point coordinates of the registered target point cloud as the initial sphere center, and define the average value of the average center distance of all points of the registered target point cloud as the initial radius; calculate the initial Euclidean distance from each point to the initial sphere center; calculate the difference between the Euclidean distance and the initial radius to obtain the residual vector; use the least squares method to optimize the minimum sum of squares of the residual function to obtain the sphere center coordinates and radius. Secondly, based on the world coordinate system, calculate the real-time monitoring distance between the sphere center coordinates and the coordinate origin. Finally, calculate the difference between the real-time monitoring distance and the initial Euclidean distance, and determine the upward and downward sliding offset of the scraper conveyor based on the difference.

[0079] The following describes an exemplary application of an embodiment of the present invention in a practical application scenario.

[0080] The present invention proposes another method for monitoring the upward and downward displacement of a scraper conveyor based on point cloud registration, such as Figure 2 As shown in the figure, firstly, a spherical target is arranged on the tunnel wall, and the target point cloud image is obtained by the depth camera arranged at the head of the scraper conveyor. The target point cloud is identified by the ICP point cloud registration method, and the distance change between the target point cloud and the scraper conveyor is monitored. This method can accurately identify and locate the target, solve the problem of incorrect feedback caused by laser ranging due to occlusion, etc., and can accurately measure the upward and downward offset of the scraper conveyor, and alarm when the offset exceeds the threshold, so as to prevent the upward and downward sliding of the scraper conveyor and improve the efficiency of coal mining.

[0081] Step 210: Setting targets on the tunnel wall and acquiring a depth image of the target area.

[0082] Combination Figure 3 As shown, targets are set on the walls of the transport lanes. When the camera acquires the depth image, there is a problem of the working vibration of the scraper conveyor, which affects the quality of the image acquired by the camera. The present invention uses a three-axis pan-tilt bracket to install a depth camera to assist in depth image acquisition. The camera is installed at the head of the scraper conveyor to ensure that the camera can acquire complete image information of the lane wall. The depth camera is used to acquire the depth image of the area where the spherical target is located, and the median filter is used for denoising and hole filling to ensure the integrity of the depth image information.

[0083] Step 220: Establish a world coordinate system with the camera as the origin, convert the pixel coordinates and depth values ​​in the depth image into (x, y, z) coordinates in three-dimensional space, and generate a data set containing a large number of three-dimensional points.

[0084] Step 230: Perform denoising, segmentation, downsampling, and clustering preprocessing on the converted point cloud image to segment the target point cloud therefrom.

[0085] Step 240: extract the local surface features of the target point cloud, perform preliminary alignment between the template point cloud and the target point cloud through RANSAC coarse registration, and then use ICP fine registration to accurately align the template point cloud and the target point cloud.

[0086] Step 250: Use the least square method to fit the center of the target point cloud and calculate the distance between the center of the target and the camera.

[0087] In some embodiments, the above step 210 may be implemented through steps 211 to 212:

[0088] Step 211: Combine Figure 4 As shown, a depth camera is used at the head position of the scraper conveyor to obtain a depth image, and the noise in the depth image is removed by median filtering. The median filter is set to a 7×7 window to retain the image edges and contours.

[0089] Step 212: Combine Figure 5 As shown, the Telea algorithm is used to fill the hole information in the depth image, and the search radius (inpaint_radius) is set for repair.

[0090] In some embodiments, the above step 220 may be implemented by:

[0091] Combination Figure 6 As shown, read the camera's intrinsic matrix and image size information, create a pinhole camera model, use the R_camera_to_world function to complete the transformation between the camera coordinate system and the world coordinate system, and set the scale factor depth_scale parameter to 1000, that is, the original unit of the transformed point cloud image is meters. Figure 7 As shown, the function o3d.geometry.RGBDImage.create_from_color_and_depth is used to convert the depth image into a point cloud image.

[0092] In some embodiments, the above step 230 may be implemented through steps 231 to 234:

[0093] Step 231: De-noising the point cloud data to reduce the noise point cloud that exists after the depth image is converted into the point cloud image.

[0094] Further, step 231 can be implemented in the following manner:

[0095] Combination Figure 8 As shown in the figure, the point cloud data is subjected to statistical outlier removal and denoising. Assuming that the point cloud data set X with discrete points is i ,i=1,2,...,n}, point cloud data after filtering out discrete points X={x i,i=1,2,...,m} For any point, let S i Represents x i The average distance from all points in X' to their respective neighbors S i Obeying Gaussian distribution, calculated by global average distance, the discrete points can be considered to be small relative to the main point cloud, that is, the mean and standard deviation of X and X' are the same. Let μ be its mean and σ be its standard deviation, then record it as S(μ,σ), and have the following formula:

[0096]

[0097] All points in the point cloud whose distance to their neighbors is outside the range of (μ-std·σ,μ+std·σ) are considered outliers. The number of neighboring points nb_neighbors and the standard deviation ratio std_ratio are adjusted to remove outlier point clouds.

[0098] Step 232: Segment the denoised point cloud and extract the target point cloud separately.

[0099] Further, step 232 can be implemented in the following manner:

[0100] Combination Fig. 9 As shown in the figure, the RANSAC algorithm is used to segment the denoised point cloud data, and a small part of the data is randomly sampled to estimate the model parameters. This model is used to test all the data, and the data points that meet the model are regarded as inliers, and the data points that do not meet the model are regarded as outliers. Through multiple iterations of random sampling and estimation of model parameters, the model with the largest number of inliers is finally obtained as the final estimation result. Use the function cloud.segment_plane for plane segmentation, set the parameters of the maximum distance threshold distance_threshold, the number of points of the fitted plane ransac_n, and the maximum number of iterations num_iterations, segment the point cloud data, and segment the target point cloud.

[0101] Step 233: Downsample the target point cloud to reduce the number of point clouds and increase the running speed of subsequent steps.

[0102] Further, step 233 can be implemented in the following manner:

[0103] Combination Fig.10 As shown, the target point cloud after segmentation is subjected to voxel grid downsampling to reduce the number of point clouds, improve the calculation efficiency of subsequent steps while maintaining the overall shape of the point cloud. The side length of the small grid is L. Determination method:

[0104]

[0105] In the above formula, α is the scale factor, which adjusts the side length of the small cube grid, s is the scale coefficient, and g is the number of points in the point cloud data in the small grid.

[0106] Step 234: cluster the downsampled point cloud to further screen the target point cloud.

[0107] Further, step 234 can be implemented in the following manner:

[0108] Combination Fig.11 As shown, the point cloud data after downsampling is subjected to point cloud DBSCAN clustering processing. The cluster_dbscan function is used to set the maximum distance eps between points in the point cloud data and the minimum number of points in the point cloud cluster min_points to complete the division of the point cloud clusters.

[0109] In some embodiments, the above step 240 may be implemented through steps 241 to 242:

[0110] Step 241: Obtain the spherical target template point cloud, and extract the target point cloud and template point cloud features.

[0111] In some embodiments, a point cloud image of the target is obtained as a template point cloud under good environmental conditions. A hybrid KD tree search method is used to find neighboring points of each point in the target point cloud and the template point cloud, and a search radius radius and a maximum number of neighboring points max_nn are specified to determine a local plane, and a local plane normal is estimated based on the direction and position of the neighboring points. FPFH feature extraction is performed based on the existing local plane normal features to find corresponding points between the two point clouds.

[0112] Step 242: Perform ICP registration to accurately identify the target point cloud position.

[0113] In some embodiments, in combination Fig.12 As shown in the figure, based on the local features of the point cloud extracted by FPFH, the RANSAC algorithm is used to randomly sample the feature points between the two point clouds, set the maximum distance threshold, perform rigid body transformation on the points where the feature matching succeeds, and determine whether the distance of the feature points after the transformation is within the distance threshold range. The transformation with the most feature matching points is iteratively found as the initial position of the ICP alignment.

[0114] Combination Fig.13 As shown, the ICP algorithm is used to refine the registration. Based on the initial position obtained by the RANSAC coarse registration, the template point cloud X and the target point cloud Y are found according to the maximum allowed distance max_correspondence_distance. The nearest neighbor point pair (X i ,Y i), the optimal matching parameters O and P are calculated by optimizing the error function to minimize the difference between the point pairs.

[0115]

[0116] In the above formula, n is the number of nearest neighboring point pairs, X i is a point in the template point cloud X, Y i is a point in the target point cloud Y, X i Template point cloud and Y i is the point with the shortest distance, O and P are the optimal rotation matrix and optimal translation matrix calculated by the error function, respectively, to complete point cloud recognition and registration.

[0117] In this example, the maximum allowable distance parameter max_correspondence_distance is adjusted to 0.04. The final processing effect is combined with Fig.14 shown.

[0118] In some embodiments, the above step 250 may be implemented by:

[0119] Combination Fig.15 As shown in the figure, the registered template point cloud is fitted to the sphere, the average value of all point coordinates is defined as the initial sphere center a, the average value of the average center distance of all points is defined as the initial radius r, the Euclidean distance l from each point to the initial sphere center is calculated, the Euclidean distance l is subtracted from the initial radius r of the sphere to obtain the residual vector, and the least squares method is used to optimize the minimum sum of squares of the residual function to obtain the sphere center coordinates and radius. The distance between the sphere center coordinates and the coordinate origin is calculated in the world coordinate system created in the above steps, and the upward and downward offset of the scraper conveyor is determined by calculating the difference between the real-time monitoring distance and the initial distance.

[0120] Fig.16 is a schematic diagram of the composition structure of the scraper conveyor up-and-down sliding monitoring device based on point cloud registration provided by an embodiment of the present invention, such as Fig.16As shown, the scraper conveyor up-and-down sliding monitoring device 1600 based on point cloud registration includes: an acquisition module 1601, which is used to set a target on the wall of the transport lane, obtain a depth image of the target area, and filter and fill the hole of the target area depth image to obtain a processed target area depth image; a conversion module 1602, which is used to perform coordinate conversion on the pixel coordinates and depth values ​​in the processed target area depth image based on the world coordinate system established with the image acquisition device as the origin to obtain three-dimensional point cloud data; a processing ...2, which is used to perform coordinate conversion on the pixel coordinates and depth values ​​in the processed target area depth image based on the world coordinate system established with the image acquisition device as the origin to obtain three-dimensional point cloud data; a processing module 1603, which is used to perform coordinate conversion on the pixel coordinates and depth values ​​in the processed target area depth image based on the world coordinate system established with the image acquisition device as the origin to obtain three-dimensional point cloud data; a processing module 1604, which is used to perform coordinate conversion on the pixel coordinates and depth values ​​in the processed target area depth image based on the world coordinate system established with the image acquisition device as the origin to obtain three-dimensional point cloud data; a processing module 1605, which is used to perform coordinate conversion on the pixel coordinates and depth values ​​in the processed target area depth image based on the world coordinate system established with the image acquisition device as the origin to obtain three-dimensional point cloud data; a processing module 1606, which is used to perform coordinate conversion on the pixel coordinates and depth values ​​in the processed target area depth image based on the world coordinate system established with the image acquisition device as the Module 1603 is used to perform denoising, segmentation, downsampling and clustering processing on the three-dimensional point cloud in sequence to obtain a first target point cloud; the registration module 1604 is used to extract template point cloud features and target point cloud features based on the acquired target template point cloud and the first target point cloud; and to perform multiple registrations on the target point cloud features and the template point cloud features to obtain a registered target point cloud; the determination module 1605 is used to determine the upward and downward sliding offset of the scraper conveyor based on the registered target point cloud and the world coordinate system.

[0121] In some embodiments, the acquisition module is also used to perform median filtering on the target area depth image to obtain a target area depth image from which noise is removed; fill the hole information in the target area depth image from which noise is removed through a preset image restoration algorithm, and set a search radius for restoration to obtain the processed target area depth image.

[0122] In some embodiments, the acquisition module is also used to obtain the intrinsic parameter matrix and size information of the image acquisition device; create a camera coordinate system based on the intrinsic parameter matrix and the size information; use a preset function to establish a transformation matrix between the camera coordinate system and the world coordinate system; based on the transformation matrix, perform coordinate transformation on the pixel coordinates and depth values ​​in the processed target area depth image to obtain the three-dimensional point cloud.

[0123] In some embodiments, the processing module is also used to perform statistical outlier removal and denoising processing on the three-dimensional point cloud to obtain a denoised three-dimensional point cloud; use a preset random sampling iterative algorithm to segment the denoised three-dimensional point cloud to obtain a segmented three-dimensional point cloud; perform voxel grid downsampling processing on the segmented three-dimensional point cloud to obtain a downsampled three-dimensional point cloud; perform point cloud clustering processing on the downsampled three-dimensional point cloud to obtain the first target point cloud.

[0124] In some embodiments, the registration module is further used to use a preset KD tree search algorithm to traverse the neighboring points of each point in the target template point cloud and the first target point cloud, determine the local plane based on a preset search radius and a maximum number of neighboring points, and estimate the normal of the local plane based on the direction and position of the neighboring points; perform FPFH feature extraction on the target template point cloud and the first target point cloud based on the normal to obtain local features of the target template point cloud and local features of the first target point cloud; use a preset random sampling iterative algorithm, based on a preset maximum distance threshold, to perform FPFH feature extraction on the target template point cloud and the first target point cloud The features and local features of the first target point cloud are randomly sampled to obtain a point cloud after successful feature matching; a rigid body transformation is performed on the point cloud after successful feature matching, and it is determined whether the transformed feature point cloud is within the preset maximum distance threshold, and the initial registration position is determined by iteratively querying the transformation with the most feature matching points; based on the initial registration position and the maximum allowed distance, the nearest neighbor point pair between the target template point cloud and the first target point cloud is determined, and the optimal matching parameters of the nearest neighbor point pair are calculated by a preset optimization error function to ensure that the difference between the nearest neighbor point pairs is minimized.

[0125] In some embodiments, the determination module is also used to fit the registered target point cloud to a spherical surface, and define the average value of the coordinates of all points in the registered target point cloud as the initial center of the sphere, and define the average value of the average center distance of all points in the registered target point cloud as the initial radius; calculate the initial Euclidean distance from each point to the initial center of the sphere; calculate the difference between the Euclidean distance and the initial radius to obtain a residual vector; use the least squares method to optimize the minimum sum of squares of the residual function to obtain the center coordinates and radius; based on the world coordinate system, calculate the real-time monitoring distance between the center coordinates and the coordinate origin; calculate the difference between the real-time monitoring distance and the initial Euclidean distance, and determine the upward and downward offset of the scraper conveyor based on the difference.

[0126] It should be noted that the description of the device of the embodiment of the present invention is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present invention for understanding.

[0127] It should be noted that, in the embodiment of the present invention, if the above-mentioned method for monitoring the ups and downs of a scraper conveyor based on point cloud registration is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for a terminal to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present invention is not limited to any specific combination of hardware and software.

[0128] Correspondingly, an embodiment of the present invention provides a scraper conveyor up-and-down sliding monitoring device based on point cloud registration. Fig.17 is a schematic diagram of the composition structure of a scraper conveyor up-and-down sliding monitoring device based on point cloud registration provided by an embodiment of the present invention, such as Fig.17 As shown, the scraper conveyor ups and downslide monitoring device 1700 based on point cloud registration includes at least: a processor 1701 and a computer-readable storage medium 1702 configured to store executable instructions, wherein the processor 1701 generally controls the overall operation of the scraper conveyor ups and downslide monitoring device 1700 based on point cloud registration. The computer-readable storage medium 1702 is configured to store instructions and applications executable by the processor 1701, and can also cache data to be processed or processed by each module in the processor 1701 and the scraper conveyor ups and downslide monitoring device 1700 based on point cloud registration, which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0129] An embodiment of the present invention provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will be caused to execute the method provided by the embodiment of the present invention, for example, Figure 1 The method shown.

[0130] In some embodiments, the storage medium can be a computer-readable storage medium, for example, a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disk, or a compact disk read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above memories.

[0131] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0132] As an example, executable instructions may, but need not necessarily, correspond to a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As an example, executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed at multiple locations and interconnected by a communication network.

[0133] The above description is only an embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement and improvement made within the spirit and scope of the present invention are included in the protection scope of the present invention.

[0134] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present invention, the size of the serial number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention. The serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0135] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, an element defined by the statement "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed.

[0136] The above is only an embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for monitoring the upward and downward movement of a scraper conveyor based on point cloud registration, characterized in that: The method comprises: A target is set on the wall of the transport lane, a depth image of the target area is obtained, and the depth image of the target area is filtered and hole-filled to obtain a processed depth image of the target area; Based on a world coordinate system established with the image acquisition device as the origin, coordinate transformation is performed on the pixel coordinates and depth values ​​in the processed depth image of the target area to obtain a three-dimensional point cloud; The three-dimensional point cloud is processed by denoising, segmenting, downsampling and clustering in sequence to obtain a first target point cloud; Based on the acquired target template point cloud and the first target point cloud, extract template point cloud features and target point cloud features; and perform multiple registrations on the target point cloud features and the template point cloud features to obtain a registered target point cloud; Based on the registered target point cloud and the world coordinate system, the upward and downward displacement of the scraper conveyor is determined.

2. The method according to claim 1, characterized in that After filtering and hole filling the target area depth image, a processed target area depth image is obtained, including: Performing median filtering on the target area depth image to obtain a target area depth image with noise removed; The hole information in the target area depth image after the noise is removed is filled by a preset image restoration algorithm, and a search radius is set for restoration to obtain the processed target area depth image.

3. The method according to claim 1, characterized in that The method of performing coordinate transformation on the pixel coordinates and depth values ​​in the processed target area depth image based on a world coordinate system established with the image acquisition device as the origin to obtain a three-dimensional point cloud includes: Obtaining the intrinsic parameter matrix and size information of the image acquisition device; Creating a camera coordinate system based on the intrinsic parameter matrix and the size information; A conversion matrix between the camera coordinate system and the world coordinate system is established using a preset function; Based on the transformation matrix, coordinate transformation is performed on the pixel coordinates and depth values ​​in the processed target area depth image to obtain the three-dimensional point cloud.

4. The method according to claim 1, characterized in that: The step of sequentially performing denoising, segmentation, downsampling and clustering processing on the three-dimensional point cloud to obtain a first target point cloud includes: Performing statistical outlier removal and denoising processing on the three-dimensional point cloud to obtain a denoised three-dimensional point cloud; Using a preset random sampling iterative algorithm, segmenting the denoised three-dimensional point cloud to obtain a segmented three-dimensional point cloud; Performing voxel grid downsampling processing on the segmented three-dimensional point cloud to obtain a downsampled three-dimensional point cloud; Point cloud clustering is performed on the downsampled three-dimensional point cloud to obtain the first target point cloud.

5. The method according to claim 1, characterized in that extracting template point cloud features and target point cloud features based on the acquired target template point cloud and the first target point cloud; The target point cloud features and the template point cloud features are registered multiple times to obtain a registered target point cloud, including: Using a preset KD tree search algorithm to traverse the neighboring points of each point in the target template point cloud and the first target point cloud, determine a local plane based on a preset search radius and a maximum number of neighboring points, and estimate the normal of the local plane based on the direction and position of the neighboring points; Performing FPFH feature extraction on the target template point cloud and the first target point cloud based on the normal to obtain local features of the target template point cloud and local features of the first target point cloud; A preset random sampling iterative algorithm is used to randomly sample the local features of the target template point cloud and the local features of the first target point cloud based on a preset maximum distance threshold to obtain a point cloud after successful feature matching; a rigid body transformation is performed on the point cloud after successful feature matching, and it is determined whether the transformed feature point cloud is within the preset maximum distance threshold, and the transformation with the most feature matching points is determined as the initial registration position by iteratively querying; Based on the initial registration position and the maximum allowed distance, the nearest neighbor point pair between the target template point cloud and the first target point cloud is determined, and the optimal matching parameters of the nearest neighbor point pair are calculated by a preset optimization error function to ensure that the difference between the nearest neighbor point pairs is minimized.

6. The method according to claim 1, characterized in that The step of determining the upward and downward displacement of the scraper conveyor based on the registered target point cloud and the world coordinate system comprises: Fit the registered target point cloud to a spherical surface, define the average value of all point coordinates of the registered target point cloud as the initial sphere center, and define the average value of the average center distances of all points of the registered target point cloud as the initial radius; calculate the initial Euclidean distance from each point to the initial sphere center; calculate the difference between the Euclidean distance and the initial radius to obtain a residual vector; use the least squares method to optimize the minimum sum of squares of the residual function to obtain the coordinates and radius of the sphere center; Based on the world coordinate system, calculating the real-time monitoring distance between the spherical center coordinates and the coordinate origin; The difference between the real-time monitoring distance and the initial Euclidean distance is calculated, and the upward and downward sliding offset of the scraper conveyor is determined based on the difference.

7. A scraper conveyor up-and-down sliding monitoring device based on point cloud registration, characterized in that: The device comprises: An acquisition module is used to set a target on the wall of the transport lane, acquire a depth image of the target area, and filter and fill holes in the depth image of the target area to obtain a processed depth image of the target area; A conversion module, for performing coordinate conversion on pixel coordinates and depth values ​​in the processed target area depth image based on a world coordinate system established with the image acquisition device as the origin, to obtain three-dimensional point cloud data; A processing module, used for sequentially performing denoising, segmentation, downsampling and clustering processing on the three-dimensional point cloud to obtain a first target point cloud; A registration module, for extracting template point cloud features and target point cloud features based on the acquired target template point cloud and the first target point cloud; and performing multiple registrations on the target point cloud features and the template point cloud features to obtain a registered target point cloud; The determination module is used to determine the upward and downward sliding offset of the scraper conveyor based on the aligned target point cloud and the world coordinate system.

Images