A trench excavation shovel entry point detection system and method based on visual point cloud processing

Abstract

The application provides a trench excavation spading point detection system and method based on visual point cloud processing, and relates to the technical field of excavation spading point detection.The system comprises a traffic cone, a binocular camera and a processing module.The method comprises the following steps: obtaining a trench site point cloud, performing coordinate transformation, preprocessing, segmentation and trench region extraction on the obtained trench site point cloud to obtain a trench region point cloud, designing feature information of a spading point of trench excavation by using a global gradient consistency function based on the trench region point cloud, detecting the spading point based on the feature information of the spading point, and obtaining 3D coordinates of the spading point.The application can quickly calculate excavation parameters, facilitate error analysis of the spading point, improve spading point detection precision, and improve spading point calculation efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of digging shovel entry point detection, and in particular to a trench digging shovel entry point detection system and method based on visual point cloud processing. Background Art

[0002] Excavators are one of the most widely used types of construction machinery. A key technology for autonomous excavation operations lies in the perception of key areas during the excavation process, especially the detection and 3D positioning of key targets such as the shovel entry point at the beginning of the excavation cycle, which enables subsequent motion planning.

[0003] Currently, trench excavation shovel entry point detection methods basically adopt the manual establishment of shovel entry position marks and determine the shovel entry point by human eyes. The existing technical methods cannot provide accurate environmental information, which will cause errors in the shovel entry point and reduce the accuracy of shovel entry point detection. Summary of the Invention

[0004] In order to solve the above problems, the purpose of the present invention is to propose a trench excavation shovel entry point detection system and method based on visual point cloud processing, which can quickly calculate excavation parameters, facilitate analysis of shovel entry point errors, improve shovel entry point detection accuracy, and improve shovel entry point calculation efficiency.

[0005] In order to achieve the above technical objectives, the present invention provides a trench excavation shovel entry point detection system based on visual point cloud processing, comprising: a traffic cone, a binocular camera, and a processing module;

[0006] The traffic cones are placed in the trench excavation shovel entry point area, and the binocular camera is installed above the excavator cab and forms a set angle with the top plane of the cab. The binocular camera collects image information of the traffic cones and obtains a point cloud of the excavation site. The processing module is electrically connected to the binocular camera, and the processing module processes the image information of the traffic cones collected by the binocular camera to detect the trench excavation shovel entry point.

[0007] A trench excavation shovel entry point detection method based on visual point cloud processing, applied to the trench excavation shovel entry point detection system based on visual point cloud processing, comprises the following steps:

[0008] S1, obtain the point cloud of the excavation site;

[0009] S2, coordinate transformation, preprocessing, segmentation, and extraction of the trench area of ​​the acquired excavation site point cloud to obtain the trench area point cloud;

[0010] S3, based on the trench area point cloud, using a global gradient consistency function to design feature information of the shovel entry point for trench excavation;

[0011] S4, detecting the scooping point based on the feature information of the scooping point, and acquiring the 3D coordinates of the scooping point.

[0012] Preferably, the step S1 of obtaining a point cloud of the excavation site area includes:

[0013] A binocular camera is used to obtain three-dimensional point cloud data of the excavation site, including obtaining left and right images, extracting feature points from the left and right images using a CNN network, calculating the disparity of the feature points in each image, and obtaining the 3D depth and three-dimensional point cloud coordinates of each feature point in the camera space based on camera parameters and triangulation technology.

[0014] Preferably, the step S2 of obtaining the point cloud of the excavation site area further includes:

[0015] Establish the excavator's base coordinate system, the excavator's working device coordinate system, and the camera coordinate system. The excavator's base coordinate system is O0-X0Y0Z0. The excavator's base coordinate system is a global coordinate system with the intersection of the excavator chassis's rotation center and the crawler ground plane as its origin. During the excavation operation, the coordinate system is always fixed to the ground and remains unchanged, where the X0 axis points to the front of the crawler and the Y0 axis points outward perpendicular to the crawler length.

[0016] The coordinate system of the excavator's working device is O1-X1Y1Z1, which is a local coordinate system with the hinge point between the boom rotation center and the vehicle body as the origin;

[0017] The camera coordinate system is O C -X C Y C Z C , refers to the coordinate system of the binocular camera with the left camera as the reference. During the excavation operation, the camera coordinate system changes as the vehicle rotates.

[0018] Preferably, in step S2, coordinate transformation is performed on the acquired excavation site point cloud, including:

[0019] The camera coordinate system O C -X C Y C Z C The coordinates of the 3D point are transformed from the working device coordinate system O1-X1Y1Z1 to the global base coordinate system O0-X0Y0Z0. The coordinate transformation of this process is realized by the homogeneous transformation matrix, which is as follows:

[0020]

[0021] Where (x0, y0, z0) represents the coordinates of the point in the global base coordinate system, (X C ,Y C ,ZC ) represents the coordinates of a point in the camera coordinate system, (c x ,c y ,c z ) represents the origin O C of the camera coordinate system, (a1, d1) represents the coordinates in the work device coordinate system O1-X1Y1Z1, a1 represents the length of the connecting rod between the rotation joint axis Z0 and the axis Z1, and d1 represents the connecting rod offset distance between the axis X0 and the axis X1.

[0022] Preferably, in the step S2, the obtained excavation site point cloud is preprocessed, including:

[0023] The excavation site point cloud is filtered by using a cloth filtering algorithm.

[0024] Based on the result of filtering the excavation site point cloud by using the cloth filtering algorithm, an abnormal value is removed by a statistical filter to form a new excavation site point cloud, specifically including: setting the k-nearest neighbors P of a point p, the position of each point is subject to a Gaussian distribution with a standard deviation σ k and a mean value u k , for a given point set P, calculating the average distance d i of each point p i ∈P to the neighbor points, if the average distance i of the point p is greater than a given threshold value, the point is determined to be an abnormal value and is removed, and the point set P composed of the remaining points * is recorded as follows:

[0025]

[0026] In the formula, l b =(u k -α·σ k ) and u b =(u k +α·σ k ), α is a factor affecting the required density of the point cloud, according to experience, k is set to 8 and α is set to 3.

[0027] After removing the abnormal value by the statistical filter, the new excavation site point cloud is sampled by using a voxel filter.

[0028] Preferably, in the step S2, the obtained excavation site point cloud is segmented, including:

[0029] The mining site point cloud is voxelized, and the voxelization of the mining site point cloud includes dividing the mining site point cloud into small three-dimensional cubes. In each voxel, the points therein form a surface with different shapes, including planes, curved surfaces or intersecting surfaces. The surface patches are globally clustered to extract the segmented planes.

[0030] Preferably, in step S2, extracting the groove area from the acquired excavation site point cloud includes:

[0031] After segmenting the obtained excavation site point cloud, the plane part of the excavation site point cloud is segmented out, and the extracted ground plane part point cloud P ** Remove it and get the point cloud P of the groove area C , written as:

[0032] P C =P * \P ** (3)

[0033] Based on the extracted point cloud of the trench area, subsequent shovel entry point detection and other excavation parameter calculations are performed.

[0034] Preferably, S3, based on the trench area point cloud, designing feature information of the shovel entry point for trench excavation using a global gradient consistency function, includes:

[0035] Establish a global gradient consistency function to describe the shovel entry point of the excavation area, specifically including when the shovel entry point is the real shovel entry point, the distribution of the contour points of the excavation area on both sides of the k point relative to the straight line and straight lines With the smallest overall deviation, the straight lines formed by points j and i in the left and right regions and the endpoints n and 0 are calculated on both sides of the shovel entry point k. and straight lines Gradient and The global gradient consistency function of point k is obtained by calculating the cumulative value of the gradient deviation of the data points on both sides of point k using the following formula (11): for

[0036]

[0037] Where, and are the straight lines on both sides of point k and straight lines gradient;

[0038] Finally, combined with the previous analysis, the condition is met The data point k in the smallest case is the detection result of the shovel entry point, which is expressed as follows

[0039]

[0040] From formula (11) and formula (12), the index k of the trench digging operation can be determined, and then the index is used to determine the cutting-in point and its three-dimensional coordinates.

[0041] Compared with the prior art, the present application has the beneficial effects that: the trench digging cutting-in point detection system and method based on visual point cloud processing are proposed, which comprises a traffic cone, a binocular camera and a processing module; by obtaining a point cloud of a digging site, coordinate transformation, preprocessing, segmentation and extraction of a trench area are performed on the obtained point cloud of the digging site to obtain a point cloud of the trench area, feature information of the cutting-in point of the trench digging is designed based on a global gradient consistency function, the cutting-in point is detected based on the feature information of the cutting-in point, and 3D coordinates of the cutting-in point are obtained, which can quickly calculate digging parameters, facilitate analysis of the error of the cutting-in point, improve the detection accuracy of the cutting-in point and improve the calculation efficiency of the cutting-in point. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0043] Figure 1 The trench digging cutting-in point detection method based on visual point cloud processing according to the present application is shown in the flowchart.

[0044] Figure 2 The stereo camera and the excavation coordinate system according to the present application are shown in the schematic diagram.

[0045] Figure 3 The cutting-in point detection method according to the present application is shown in the flowchart.

[0046] Figure 4 The global gradient variation of the trench digging area contour point according to the present application is shown in the schematic diagram.

[0047] Figure 5 The cutting-in point positioning error of the excavation operation according to the present application is shown in the schematic diagram.

[0048] Figure 6 The excavation area three-dimensional point cloud diagram according to the present application is shown in the schematic diagram.

[0049] Figure 7 The structure block diagram of the trench digging cutting-in point detection system based on visual point cloud processing according to the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0051] like Figure 7 As shown, the present invention provides a trench excavation shovel entry point detection system 12 based on visual point cloud processing, comprising: a traffic cone 1202, a binocular camera 1201, and a processing module 1203;

[0052] The traffic cones are placed in the trench excavation shovel entry point area, and the binocular camera is installed above the excavator cab and forms a set angle with the top plane of the cab. The binocular camera collects image information of the traffic cones and obtains a point cloud of the excavation site. The processing module is electrically connected to the binocular camera, and the processing module processes the image information of the traffic cones collected by the binocular camera to detect the trench excavation shovel entry point.

[0053] Because it is difficult to place auxiliary information markers in trench excavation areas, robust features can be constructed to describe the entry point, and a binocular camera can be used to obtain its position information. First, a three-dimensional point cloud of the working environment is acquired using a binocular camera, and outliers in the data are removed based on the characteristics of the excavation operation and the working range. Then, the state of the excavation area is determined by fusing the two-dimensional image data with the characteristics of the excavation operation. The entry point of the excavation area is extracted in real time from the image and its corresponding three-dimensional spatial coordinates are obtained. The position coordinates of the entry point are then corrected. Finally, the entry point is determined by combining the excavation motion constraints. The position coordinates of the excavation target in the camera system are converted to the global base coordinate system of the current excavator according to the excavation task, and the motion parameters such as path points and joint angles in the excavator's pose space and joint space are obtained.

[0054] Combining the "trench excavation + side unloading" mode of trench operation conditions and the depth perception capability of binocular vision, a point cloud processing algorithm for trench excavation area is proposed to segment the trench area. The shovel entry point detection feature is constructed to realize the shovel entry point detection. Finally, through field tests, the error between this scheme and the actual measurement is analyzed and compared.

[0055] like Figure 1As shown, the present invention also provides a trench excavation shovel entry point detection method based on visual point cloud processing, which is applied to the trench excavation shovel entry point detection system based on visual point cloud processing, comprising the following steps:

[0056] S1, obtain the point cloud of the excavation site;

[0057] Furthermore, the step S1 of obtaining the point cloud of the excavation site area includes:

[0058] Using binocular vision technology, a binocular camera acquires 3D point cloud data of the excavation site. This involves acquiring left and right images, extracting feature points from these images using a CNN network, calculating the disparity of the feature points in each image, and obtaining the 3D depth and 3D point cloud coordinates of each feature point in camera space based on camera parameters and triangulation technology.

[0059] Binocular vision technology can be used to acquire 3D point cloud data of trench excavation sites. First, left and right images are acquired. Feature points are extracted from these images using an improved CNN network (Neural Depth technology). This method calculates disparity in a low-resolution cost volume. High-frequency details are re-introduced hierarchically using a compact pixel-to-pixel refinement network to learn an upsampling function, thereby enhancing feature extraction. The disparity of this feature in each image is then calculated. Finally, the 3D depth and 3D point cloud coordinates of each feature point in camera space are obtained based on camera parameters and triangulation techniques.

[0060] Furthermore, the step S1 of obtaining the point cloud of the excavation site area further includes:

[0061] Establish the excavator's base coordinate system, the excavator's working device coordinate system, and the camera coordinate system. The excavator's base coordinate system is O0-X0Y0Z0. The excavator's base coordinate system is a global coordinate system with the intersection of the excavator chassis's rotation center and the crawler track's ground plane as its origin. During the excavation operation, the coordinate system is always fixed to the ground and remains unchanged, with the X0 axis pointing directly in front of the crawler track and the Y0 axis pointing outward perpendicular to the crawler track's length.

[0062] The coordinate system of the excavator's working device is O1-X1Y1Z1, which is a local coordinate system with the hinge point between the boom rotation center and the vehicle body as the origin;

[0063] The camera coordinate system is O C -X C Y C Z C , refers to the coordinate system of the binocular camera with the left camera as the reference. During the excavation operation, the camera coordinate system changes as the vehicle rotates.

[0064] In the experimental platform, the camera is installed above the cab and at a fixed angle with the top plane of the cab. The angle needs to be adjusted in advance before use, so that the camera field of view completely covers the trench and unloading area and the corresponding front area during excavation. As shown in Figure 2 The coordinate systems in the excavator are defined as follows: the base coordinate system O0-X0Y0Z0 of the excavator is a global coordinate system with the intersection of the center of rotation of the excavator chassis and the ground plane as the origin (the coordinate system remains fixed with the ground and remains unchanged during the excavation operation), wherein the X0 axis points to the front of the track, and the Y0 axis is perpendicular to the length direction of the track. The working device coordinate system O1-X1Y1Z1 of the excavator is a local coordinate system with the hinge point of the boom rotation center and the vehicle body as the origin. The camera coordinate system O C -X C Y C Z C refers to the coordinate system of the binocular camera with the left camera as the reference, and the camera coordinate system changes with the rotation of the vehicle during the excavation operation.

[0065] S2, coordinate transformation, preprocessing, segmentation, and extraction of the trench area point cloud obtained from the excavation site;

[0066] Further, in the step S2, the obtained excavation site point cloud is subjected to coordinate transformation, which includes: in order to facilitate the calculation of the key point position coordinates and the working device joint space parameters during the excavation process, the coordinates of the lower 3D points are transformed from the camera coordinate system O C -X C Y C Z C to the global base coordinate system O0-X0Y0Z0 through the working device coordinate system O1-X1Y1Z1. The coordinate transformation in this process is written in the following form through the homogeneous transformation matrix:

[0067]

[0068] In the formula, (x0, y0, z0) represents the coordinates of the point in the global base coordinate system, (X C ,Y C ,Z C ) represents the coordinates of the point in the camera coordinate system, and (c x ,c y ,c z ) represents the origin O C of the camera coordinate system. The coordinates in the working device coordinate system O1-X1Y1Z1, a1 represents the length of the connecting rod between the rotation joint axis Z0 and the axis Z1, and d1 represents the connecting rod offset distance between the axis X0 and the axis X1.

[0069] Through the above-mentioned homogeneous coordinate transformation, the coordinates of the unloading point during the excavation operation can be transformed from the camera coordinate system to the global base coordinate system.

[0070] Furthermore, in step S2, the obtained excavation site point cloud is preprocessed, including:

[0071] Using a cloth filtering algorithm to filter the point cloud of the excavation site;

[0072] Based on the result of filtering the mining site cloud using the cloth filtering algorithm, outliers are removed by statistical filtering to form a new mining site cloud, specifically including: setting the position of each point in the k neighbors P closest to point p to obey the standard deviation σ k and the mean is u k Gaussian distribution, for a given point set P, calculate each point p i ∈P the average distance d to neighboring points i , if point p i The average distance When it is greater than a given threshold, the point is judged as an outlier and removed, and the remaining points The point set P composed of * It is written as follows:

[0073]

[0074] Where, l b =(u k -α·σ k ) and u b =(u k +α·σ k ), α is a factor that affects the required density of the point cloud. According to experience, k is set to 8 and α is set to 3;

[0075] After removing outliers through statistical filters, the new excavation site point cloud is sampled using a voxel filter.

[0076] The point cloud of the excavation site usually obtained includes the ground, the excavation trench, the working device of the excavator and other equipment. Directly processing the original point cloud will produce large errors, so the original point cloud needs to be preprocessed.

[0077] To effectively segment the trench area, we first filter out objects outside the ground. Traditional filtering algorithms mostly distinguish ground points from non-ground points based on slope and elevation changes. The cloth filtering algorithm, however, uses a "spring-mass" model. First, the point cloud is flipped, and then, assuming a piece of cloth falls from above due to gravity, the final piece of cloth that falls represents the current terrain. This paper sets the grid size to 0.1m and the classification elevation difference threshold to 0.9m.

[0078] After statistical filtering to remove outliers, in order to improve the computational efficiency of subsequent processing and maintain the geometric structure information of the point cloud, this paper uses a voxel filter to downsample the filtered point cloud. In this paper, the resolution vr of the 3D voxel is set to 0.06m.

[0079] Furthermore, in step S2, the acquired excavation site point cloud is segmented, including:

[0080] The mining site point cloud is voxelized, wherein the voxelization includes dividing the mining site point cloud into small three-dimensional cubes, wherein the points in each voxel form a surface with different shapes, such as a plane, a curved surface, or an intersecting surface, and globally clustering the surface patches to extract the segmented plane.

[0081] Furthermore, in the step S2, the trench area is extracted from the obtained excavation site point cloud, including: after segmenting the obtained excavation site point cloud, the plane part of the excavation site point cloud is segmented, and the extracted ground plane part point cloud P is extracted. ** Remove it and get the point cloud P of the groove area C , written as:

[0082] P C =P * \P ** (3)

[0083] Based on the extracted point cloud of the trench area, subsequent shovel entry point detection and other excavation parameter calculations are performed.

[0084] Voxel is the basic unit of three-dimensional space gridding. Voxelization is the conversion of the geometric representation of an object into a voxel representation that is closest to the object. It achieves an approximation of the geometric shape of the object at a certain spatial resolution. In order to achieve voxelization, the octree structure is used to discretize the entire point cloud into three-dimensional voxels v∈V of equal size (V is the entire point cloud), so that the octree can be used to index unorganized points and simplify the point cloud set through grid representation. Since the granularity of the voxel affects the performance of the segmentation, the generation of the octree structure is entirely for the purpose of creating different levels of voxel resolution (i.e. size). Compared with the point-based data structure, the voxel structure of the point cloud simplifies the representation of complex scenes. To some extent, voxelization is also a dimensionality reduction sampling. The tree structure used to divide the space can accelerate the search process and improve the efficiency of point traversal.

[0085] Selection of plane patches: After voxelization, the entire point cloud has been divided into small three-dimensional cubes. In each voxel, the points form a surface with different shapes, such as a plane, a curved surface, or an intersecting surface. For plane segmentation, there are two conditions: (1) a plane can only be composed of voxels with a plane; (2) a plane model can be represented by a cluster of discontinuous planes. Obviously, by selecting voxels with a plane, these voxels may not be continuous, but better plane fitting results can be obtained. In this point cloud processing method, voxels that meet the above two conditions are called patches.

[0086] The patch f∈F is represented by the center and normal vector of the voxel, and the patch selection is achieved by fitting a plane to the surface of the voxel. For the point set Pv within the voxel v, the coordinates of k 3D points {p1,p1,…,pk} are used to calculate the 3D structure tensor M of the points in the neighborhood, which is expressed as the position covariance matrix M of the given 3D point [5]:

[0087]

[0088] Where, is the geometric center of the voxel, calculated as follows:

[0089]

[0090] The eigenvalues ​​and eigenvectors of M provide the points p∈P in V v Geometric information, the eigenvector corresponding to the smallest eigenvalue represents the normal vector of the surface, which can be used to distinguish the surface type at point p.

[0091] When performing global clustering on plane patches, the plane patch set f∈F is the residual from all points to the plane. The voxels v*∈V that are smaller than a given threshold Δr are composed of:

[0092]

[0093] Wherein, the threshold Δr is empirically set to one-quarter of the voxel size, and the residual di is the distance from point pi to the fitted plane.

[0094] Global Clustering of Facets: Once faces are selected from the voxels, they are clustered into clusters, and all points in the same cluster are used to estimate the plane model. Clustering of faces is transformed into the problem of constructing and partitioning a point cloud using a graphical model. The main advantage of graphical models is that they can explicitly represent the relationship between points using mathematical structures and infer the underlying distribution from known information. Since local graphical models only contain information about the local geometric structure of the point cloud, a global graphical model is needed to construct the structural features of the faces.

[0095] In the global graph model G(N,W,E), all patches F are treated as nodes N, and adjacent patches are connected by edges E with weight W. Assuming that the nodes (i.e. patches) are independent of each other, the weight w(i,j)∈W of the edge Eij between nodes Ni and Nj is calculated by measuring the normal vector in a multiplicative form. and position difference (i.e. spatial distance) It is defined by the angular difference between:

[0096]

[0097] Where, δ a and δ p are parameters that control the importance of angle difference and spatial distance respectively. And ensure that w(i,j)∈[0,1].

[0098] An efficient graph segmentation algorithm is used for 3D point cloud segmentation. The partition S divides the nodes N (i.e., patches) into clusters C∈S corresponding to the connected faces in the graph. First, each node Ni is considered to be a cluster Ci, and the edges connecting the two nodes are sorted in ascending order of weight. Then, the graph segmentation is performed iteratively by evaluating the maximum internal difference Ii within the partition Si and the external difference between clusters Ci and Cj. The maximum internal difference in a cluster is related to the maximum weight of the edges between the nodes included in the cluster. The external difference is the minimum weight of the edge connecting two faces from different factions in the graph. If the difference between two clusters is less than the maximum difference within a cluster, then the two clusters are merged. After merging the two clusters, the maximum internal difference I of the cluster is updated and increased by a term Cij associated with the number of faces in the newly merged cluster. Specifically, for nodes Ni∈Cm and Nj∈Cn of an edge Eij, if Eij has the minimum weight of all possible edges connecting nodes from different clusters Cm and Cn, and the weight wij of Eij is greater than a threshold τmn, then Cm and Cn are merged. The weight w(i,j) is related to the external difference between Cm and Cn as follows:

[0099]

[0100] In the formula, |C| represents the number of nodes in C, and δ represents the constant that determines the initial threshold. m |=1 and |C n |=1, then define τ mn =δ.

[0101] Plane positioning: Once a segmentation plane is extracted, its position and orientation can be described by an OBB or AABB bounding box [8]. AABB in space is defined using a parallelepiped, whose basis vectors are aligned with the coordinate axes; OBB is also a parallelepiped, but its faces and edges are parallel to the principal eigenvectors of the point set. It is closer to the object than AABB and can significantly reduce the number of bounding boxes. Therefore, OBB boxes are used to describe the segmentation plane.

[0102] The construction of an OBB requires specifying a center point p(xc, yc, zc) and three orthogonal eigenvectors n(nx, ny, nz) for position and orientation, respectively. The OBB dimensions require width, height, and depth. The OBB is a minimal closed 3D bounding box with minimal metrics in the X, Y, and Z dimensions. Therefore, when extracting a plane from the segmentation, the OBB constrains the plane's possible orientation and position.

[0103] Plane model estimation: 3D HT algorithms are often used for plane detection. Given a set of points P, a parameterized plane model is estimated. The Hesse normal plane model is defined by a point p(xp, yp, zp) on the plane, the plane normal vector np(nx, ny, nz), and the signed distance ρ from the origin O of the coordinate system to the plane [9]:

[0104] ρ=x p n x +y p n y +z p n z (9)

[0105] Where nx, ny, and nz are calculated as follows:

[0106]

[0107] Where α is the angle between the projection axis of the normal vector np on the XY plane and the X axis, and γ is the angle between the normal vector np and the Z axis.

[0108] For a point p in the Cartesian space, there are m planes Ak(ak, yk, pk) passing through the point, k = {1, 2,..., m}. These planes can be represented as points in the Hough space (a, y, p), i.e. a point in the Cartesian space is transformed into a surface formed by multiple points in the Hough space. If multiple surfaces in the Hough space intersect at the same point, it means that multiple points in the Cartesian space are coplanar, where the point Amwith the most intersecting surfaces corresponds to the plane with the most points, i.e. the target plane.

[0109] For the three parameters a, y and p in the Hough space, the range of the first two parameters is [0, 2p] and the range of the third parameter is the distance from any point in the bounding box B(xc, yc, zc, w, h, d) to the origin, where (xc, yc, zc) is the center of the bounding box and (w, h, d) is the size of the bounding box. The OBB that partitions the point cloud also approximates the size of the plane, i.e. the possible plane direction is limited in the range between two opposite diagonals in the OBB cross section. For the two ranges of the plane direction along the X and Y axes, the possible plane direction along the Y axis ranges from amin to amax with a difference of Da, and the possible plane direction along the X axis ranges from ymin to ymax with a difference of Dy. Therefore, the range of the first two parameters in the Hough space is set to [amin, amax] and [ymin, ymax], respectively.

[0110] The plane model is estimated using the points in the patch, and the plane segmentation of the point cloud is not complete because some points are not assigned to any plane. To optimize the segmented planes, the points that are not assigned to a plane but located in the OBB of the plane need to be locally clustered. The previously unassigned points are first assigned to the initially estimated plane model, and then more points are used to estimate the plane model. After this processing, the plane part of the point cloud is segmented.

[0111] Extraction of the trench region point cloud: After the point cloud segmentation, the plane part in the excavation region point cloud can be segmented out. Under the constraint of a non-strict plane fitting threshold, the segmented plane part point cloud occupies the majority in the excavation region point cloud, so the trench region can be extracted. Generally, the excavation site point cloud is mainly composed of the ground plane part, the slope and the pit bottom plane, and what is needed is the trench region composed of the slope and the pit bottom. Therefore, the extracted ground plane part point cloud P ** is removed, and the point cloud of the trench region P C is obtained, denoted as:

[0112] P C = P * \P ** (3)

[0113] Based on the extracted point cloud of the trench area, subsequent shovel entry point detection and other excavation parameter calculations can be performed.

[0114] S3, based on the trench area point cloud, using a global gradient consistency function to design feature information of the shovel entry point for trench excavation;

[0115] Furthermore, in S3, based on the trench area point cloud, feature information of the shovel entry point for trench excavation is designed using a global gradient consistency function, including:

[0116] Establish a global gradient consistency function to describe the shovel entry point of the excavation area, specifically including when the shovel entry point is the real shovel entry point, the distribution of the contour points of the excavation area on both sides of the k point relative to the straight line and straight lines With the smallest overall deviation, the straight lines formed by points j and i in the left and right regions and the endpoints n and 0 are calculated on both sides of the shovel entry point k. and straight lines Gradient and The global gradient consistency function of point k is obtained by calculating the cumulative value of the gradient deviation of the data points on both sides of point k using the following formula (11): for

[0117]

[0118] Where, and are the straight lines on both sides of point k and straight lines gradient;

[0119] Finally, combined with the previous analysis, the condition is met The data point k in the smallest case is the detection result of the shovel entry point, which is expressed as follows

[0120]

[0121] According to equations (11) and (12), the index k of the shovel entry point of the trench excavation operation can be determined, and then the shovel entry point and its three-dimensional coordinates can be determined according to the index.

[0122] Detection of the shovel entry point in trench excavation: During manual excavation, the excavation point is usually determined manually based on the excavation conditions and requirements. For autonomous excavation, a detection algorithm for the initial excavation point (i.e., the shovel entry point) in the working device plane during the excavation process is proposed to help the system complete the detection of the excavation point in the trench operation environment. The overall process of shovel entry point detection is as follows: Figure 3 shown.

[0123] Shovel entry point detection feature construction: During trenching operations with manually operated excavators, the digging point is typically determined manually based on the excavation conditions and requirements. In autonomous excavation, it is necessary to automatically detect the initial digging point (i.e., the shovel entry point) during the excavation process to assist the autonomous system in completing digging point detection during trenching tasks.

[0124] like Figure 4 As shown in Figure 1, the ideal excavation area outline is roughly composed of three parts: the trench bottom, the excavation area slope (red outline), and the ground. In this paper, the shovel entry point is defined as the location where the excavation area slope intersects the trench bottom in the O0-X0Z0 plane.

[0125] After field tests and data analysis, a global gradient consistency function is proposed to describe the shovel entry point of the excavation area. When the possible shovel entry point is the actual shovel entry point, the distribution of the excavation area contour points on both sides of the k point is relative to the straight line. and straight lines It has the smallest overall deviation. In order to facilitate calculation, the local gradient of the contour points of the mining area on both sides of the k point is used and The cumulative deviation value is used to evaluate the overall deviation of the slope profile data points in the excavation operation area. First, on both sides of the possible shovel entry point k, the straight lines formed by each point j and i in the left and right areas and the endpoints n and 0 are calculated. and straight lines Gradient and Then, the cumulative gradient deviation of the data points on both sides of point k is calculated by the following formula (13) to obtain the global gradient consistency function of point k: for

[0126]

[0127] Where, and are the straight lines on both sides of point k and straight lines gradient.

[0128] Finally, combined with the previous analysis, the condition is met The data point k in the smallest case is the detection result of the shovel entry point, which is expressed as follows

[0129]

[0130] According to equations (13) and (14), the index k of the shovel entry point of the trench excavation operation can be determined, and then the shovel entry point and its three-dimensional coordinates can be determined based on the index.

[0131] S4, detecting the scooping point based on the feature information of the scooping point, and acquiring the 3D coordinates of the scooping point.

[0132] Shovel entry point detection accuracy assessment: After obtaining the positioning coordinates of the shovel entry point of the excavation operation, its positioning error needs to be analyzed. The positioning error of the shovel entry point is defined as the difference between the actual position of the shovel entry point and the position of the detected shovel entry point, such as Figure 5 The actual entry point position is obtained by manual measurement using a laser rangefinder; the entry point position detected by visual inspection is calculated using a positioning algorithm. To facilitate the analysis of the entry point error, the positioning error is analyzed only on the measured distance, as shown in the following formula.

[0133] ε p =|P p -P g | (15)

[0134] Where εp represents the positioning error of the shovel entry point, Pp represents the three-dimensional position coordinates of the shovel entry point detected by vision, and Pg represents the three-dimensional position coordinates of the shovel entry point actually measured.

[0135] Test results and analysis

[0136] 3D point cloud reconstruction of the excavation area: The shovel entry point detection test in this section was completed on the independently developed SWE50E autonomous excavator platform. The shovel entry point detection was performed using a ZED 2i stereo camera with an image resolution of 1920×1080 pixels.

[0137] The 3D point cloud of the excavation area and the shovel entry point detection algorithm verification effect are as follows Figure 6 As shown in Figure 2, the 3D point cloud has been transformed into the base coordinate system using Equation (2). The 3D point cloud fully represents the three important components of the excavation area, namely the upper ground, the excavation area slope, and the pit ground. Line A represents the plane where the working device is located, that is, the plane where the shovel entry point is searched. From the figure, we can see that the excavation depth increases from 0m to -1m, which represents the change in pit depth.

[0138] Detection results and analysis of the shovel entry point: To facilitate the evaluation of the shovel entry point detection accuracy, the experiment found the corresponding point of the shovel entry point image detection result in the actual environment, and calculated the deviation between the actual value of the shovel entry point and the visual detection value using formula (15). The manual measurement results were obtained using a laser rangefinder (accuracy 1mm). The seven frames of images are images of the contours of the working area in different forms during a round of excavation (including 7 shovel excavations). As shown in Table 1, within the range of 5m, as the distance increases, the binocular ranging error increases, and the maximum measurement error is less than 80mm, with an average error of 46.2mm. For general excavation operation requirements, the positioning results can meet the application requirements.

[0139] Table 1 Binocular absolute positioning error of shovel entry point

[0140]

[0141]

[0142] The present invention proposes a trench excavation shovel entry point detection system and method based on visual point cloud processing, including: traffic cones, binocular cameras, and a processing module. By acquiring an excavation site point cloud, the acquired excavation site point cloud is subjected to coordinate transformation, preprocessing, segmentation, and trench area extraction to obtain a trench area point cloud. Based on the trench area point cloud, a global gradient consistency function is used to design feature information of trench excavation shovel entry points. Based on the feature information of the shovel entry points, the shovel entry points are detected and their 3D coordinates are obtained. The system can quickly calculate excavation parameters, facilitate analysis of shovel entry point errors, improve shovel entry point detection accuracy, and improve shovel entry point calculation efficiency.

[0143] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0144] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. However, such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A trench digging shovel entry point detection method based on visual point cloud processing, applied to a trench digging shovel entry point detection system based on visual point cloud processing, characterized in that: The following steps are involved: S1, obtain the point cloud of the excavation site; Obtain a point cloud of the excavation site area, including: Establish the excavator's base coordinate system, the excavator's working device coordinate system, and the camera coordinate system. The excavator's base coordinate system is O0-X0Y0Z0. The excavator's base coordinate system is a global coordinate system with the intersection of the excavator chassis's rotation center and the crawler ground plane as its origin. During the excavation operation, the coordinate system is always fixed to the ground and remains unchanged, where the X0 axis points to the front of the crawler and the Y0 axis points outward perpendicular to the crawler length. The coordinate system of the excavator's working device is O1-X1Y1Z1, which is a local coordinate system with the hinge point between the boom rotation center and the vehicle body as the origin; The camera coordinate system is O C -X C Y C Z C , refers to the coordinate system of the binocular camera with the left camera as the reference. During the excavation operation, the camera coordinate system changes as the vehicle rotates; S2, coordinate transformation, preprocessing, segmentation, and extraction of the trench area of ​​the acquired excavation site point cloud to obtain the trench area point cloud; S3, based on the trench area point cloud, using a global gradient consistency function to design feature information of the shovel entry point for trench excavation; S4, detecting the shoveling point based on the feature information of the shoveling point, and obtaining the 3D coordinates of the shoveling point; The trench excavation shovel entry point detection system based on visual point cloud processing includes: traffic cones, binocular cameras, and a processing module; The traffic cones are placed in the trench excavation shovel entry point area, and the binocular camera is installed above the excavator cab and forms a set angle with the top plane of the cab. The binocular camera collects image information of the traffic cones and obtains a point cloud of the excavation site. The processing module is electrically connected to the binocular camera, and the processing module processes the image information of the traffic cones collected by the binocular camera to detect the trench excavation shovel entry point.

2. The trench excavation shovel entry point detection method based on visual point cloud processing according to claim 1 is characterized in that: The step S1, obtaining a point cloud of the excavation site area, includes: A binocular camera is used to obtain three-dimensional point cloud data of the excavation site, including obtaining left and right images, extracting feature points from the left and right images using a CNN network, calculating the disparity of the feature points in each image, and obtaining the 3D depth and three-dimensional point cloud coordinates of each feature point in the camera space based on camera parameters and triangulation technology.

3. The trench excavation shovel entry point detection method based on visual point cloud processing according to claim 1, characterized in that: In step S2, coordinate transformation is performed on the acquired point cloud of the excavation site, including: The camera coordinate system O C -X C Y C Z C The coordinates of the 3D point are transformed from the working device coordinate system O1-X1Y1Z1 to the global base coordinate system O0-X0Y0Z0. The coordinate transformation of this process is realized by the homogeneous transformation matrix, which is as follows: Where (x0, y0, z0) represents the coordinates of the point in the global base coordinate system, (X C ,Y C ,Z C ) represents the coordinates of the point in the camera coordinate system, (c x ,c y ,c z ) represents the origin O of the camera coordinate system C In the coordinate system O1-X1Y1Z1 of the working device, a1 represents the length of the connecting rod between the rotary joint axis Z0 and the axis Z1, and d1 represents the offset of the connecting rod between the axis X0 and the axis X1.

4. The trench excavation shovel entry point detection method based on visual point cloud processing according to claim 1, characterized in that: In step S2, the obtained excavation site point cloud is preprocessed, including: Using a cloth filtering algorithm to filter the point cloud of the excavation site; Based on the result of filtering the mining site cloud using the cloth filtering algorithm, outliers are removed by statistical filtering to form a new mining site cloud, specifically including: setting the position of each point in the k neighbors P closest to point p to obey the standard deviation σ k and the mean is u k Gaussian distribution, for a given point set P, calculate each point p i ∈P the average distance d to neighboring points i , if point p i The average distance When it is greater than a given threshold, the point is judged as an outlier and removed, and the remaining points The point set P composed of * It is written as follows: Where, l b =(u k -α·σ k ) and u b =(u k +α·σ k ), α is a factor that affects the required density of the point cloud. According to experience, k is set to 8 and α is set to 3; After removing outliers through statistical filters, the new excavation site point cloud is sampled using a voxel filter.

5. The trench excavation shovel entry point detection method based on visual point cloud processing according to claim 1, characterized in that: In step S2, the acquired mining site point cloud is segmented, including: The mining site point cloud is voxelized, and the voxelization of the mining site point cloud includes dividing the mining site point cloud into small three-dimensional cubes. In each voxel, the points therein form a surface with different shapes, including planes, curved surfaces or intersecting surfaces. The plane patches are globally clustered and the segmented planes are extracted.

6. The trench excavation shovel entry point detection method based on visual point cloud processing according to claim 4, characterized in that: In step S2, the trench area is extracted from the acquired excavation site point cloud, including: After segmenting the obtained excavation site point cloud, the plane part of the excavation site point cloud is segmented, and the extracted ground plane part point cloud P is extracted. ** Remove it and get the point cloud P of the groove area C , written as: P C =P * \P ** (3) Based on the extracted point cloud of the trench area, subsequent shovel entry point detection and other excavation parameter calculations are performed.

7. The trench excavation shovel entry point detection method based on visual point cloud processing according to claim 1, characterized in that: S3, based on the trench area point cloud, using a global gradient consistency function to design feature information of the shovel entry point for trench excavation, including: Establish a global gradient consistency function to describe the shovel entry point of the excavation area, specifically including when the shovel entry point is the real shovel entry point, the distribution of the contour points of the excavation area on both sides of the k point relative to the straight line and straight lines With the smallest overall deviation, the straight lines formed by points j and i in the left and right regions and the endpoints n and 0 are calculated on both sides of the shovel entry point k. and straight lines Gradient and The global gradient consistency function of point k is obtained by calculating the cumulative value of the gradient deviation of the data points on both sides of point k using the following formula (11): for Where, and are the straight lines on both sides of point k and straight lines gradient; Finally, combined with the previous analysis, the condition is met The data point k in the smallest case is the detection result of the shovel entry point, which is expressed as follows According to equations (11) and (12), the index k of the shovel entry point of the trench excavation operation can be determined, and then the shovel entry point and its three-dimensional coordinates can be determined according to the index.

Images