Methods, devices, electronic equipment, and media for tracking digging points on excavators.

Abstract

This application provides a method, apparatus, electronic device, and storage medium for tracking excavation points on an excavator, applicable to the field of excavator technology. The method includes: acquiring multiple consecutive frames of images captured by a camera during excavator movement; acquiring the first image coordinates of a target excavation point to be tracked that is not within the observable range of a lidar on the T-th frame image; calculating the second image coordinates of the target excavation point based on a first homography matrix between the T-th and T+1-th frames and the first image coordinates of the target excavation point; and determining the position of the target excavation point on the T+1-th frame image based on the second image coordinates of the target excavation point. This method can calculate the target excavation point in the subsequent frame image using the coordinates of the target excavation point in the previous frame and the homography matrix between the previous and subsequent frames, thereby significantly improving the accuracy of tracking the target excavation point.

Description

Technical Field

[0001] This application relates to the field of excavator technology, and more particularly to a method, apparatus, electronic device, and medium for tracking the digging point of an excavator. Background Technology

[0002] Remote control of excavators can solve the problem of harsh and dangerous construction environments, and the most common method currently is to use automated control technology to complete remote control operations. Specifically, the excavator's sensors detect the surrounding environment and transmit the corresponding two-dimensional image to the remote control device. When the remote operator selects a target excavation point on the two-dimensional image, the remote control device needs to obtain the corresponding three-dimensional coordinates of the target excavation point. Then, the remote control device uses control signals to control the movement of the excavator's tires, boom, arm, and bucket. Through their combined movement, the excavator arm reaches the target excavation point and then moves in both horizontal and vertical planes to complete the excavation task.

[0003] Existing excavator sensing devices typically include cameras and solid-state LiDAR. Cameras capture two-dimensional images of the surrounding environment, while LiDAR captures three-dimensional images. Since the remote operator selects the target excavation point on the two-dimensional image, the remote control device, upon obtaining the three-dimensional coordinates of the target excavation point, can convert these coordinates into three-dimensional coordinates based on the projection mapping relationship between the three-dimensional image captured by the LiDAR and the two-dimensional image captured by the camera. Then, based on the obtained three-dimensional coordinates, the device controls the movement of various parts of the excavator.

[0004] However, the camera's shooting range is typically much larger than that of a solid-state LiDAR. Only when the target excavation point is within the shooting range of both the camera and the LiDAR can the 3D coordinates of the target excavation point be obtained from the 2D image. If the remote operator selects the target excavation point outside the LiDAR's shooting range, the 3D coordinates of the target excavation point cannot be obtained. Usually, when the target excavation point is outside the LiDAR's shooting range, the excavator needs to be controlled to move towards the target excavation point so that it falls as close as possible within the LiDAR's shooting range. Therefore, tracking the target excavation point during this movement becomes a critical problem that needs to be solved. Summary of the Invention

[0005] In view of this, this application provides a method, device, electronic device and storage medium for tracking the digging point of an excavator, thereby solving the problem in the prior art that when the target digging point is not within the shooting range of the excavator's solid-state lidar, the remote control device cannot accurately track the target digging point as the excavator moves toward the target digging point.

[0006] The first aspect of this application provides a method for tracking the digging point of an excavator, wherein the excavator is equipped with a camera and a lidar, and the relative positions of the camera and lidar are fixed. The observation range of the camera covers the observation range of the lidar. The method includes:

[0007] During the excavator's movement, multiple consecutive frames of images captured by the camera are obtained.

[0008] Obtain the first image coordinates of the target excavation point to be tracked on the T-th frame image, where the target excavation point is not within the observable range of the lidar, and T is a positive integer greater than 0.

[0009] Based on the first homography matrix between the T-th frame image and the T+1-th frame image, and the first image coordinates of the target excavation point, calculate the second image coordinates of the target excavation point on the T+1-th frame image.

[0010] Based on the second image coordinates of the target excavation point, determine the position of the target excavation point on the T+1 frame image.

[0011] A second aspect of this application provides a control device for an excavator, wherein a camera and a lidar are mounted on the excavator, and the relative positions of the camera and lidar are fixed. The observation range of the camera covers the observation range of the lidar. The control device includes:

[0012] The acquisition unit is used to acquire multiple consecutive frames of images captured by the camera during the movement of the excavator.

[0013] The acquisition unit is also used to acquire the first image coordinates of the target excavation point to be tracked on the T-th frame image, wherein the target excavation point is not within the observable range of the lidar, and T is a positive integer greater than 0.

[0014] The processing unit is used to calculate the second image coordinates of the target excavation point on the T+1 frame image based on the first homography matrix between the T-th frame image and the T+1-th frame image and the first image coordinates of the target excavation point.

[0015] The determining unit is used to determine the position of the target excavation point on the T+1 frame image based on the second image coordinates of the target excavation point.

[0016] A third aspect of this application also provides an electronic device, including: a processor and a memory. Wherein:

[0017] The memory stores the instructions that the computer executes.

[0018] The processor executes computer execution instructions, causing the electronic device to perform the excavation point tracking method for the excavator as described in the first aspect above.

[0019] A fourth aspect of this application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the excavation point tracking method for an excavator as described in the first aspect above.

[0020] The technical solution provided in this application embodiment controls the excavator to move towards the target excavation point when the selected target excavation point on the two-dimensional image is not within the observable range of the lidar. During the excavator's movement, the image coordinates of the target excavation point on the two-dimensional image are first acquired. Then, based on the homography matrix between the two-dimensional image and the next frame of the two-dimensional image, the image coordinates of the target excavation point on the next frame of the two-dimensional image are calculated, thereby determining the position of the target excavation point on the next frame of the two-dimensional image, thus achieving the purpose of tracking the target excavation point. In this application embodiment, the traditional pixel matching method is no longer used to track the target excavation point, avoiding the problem of large pixel matching errors caused by similar visual textures in the excavator's working scene and unclear target excavation points, which prevents accurate tracking of the target excavation point. This method can calculate the homography matrix between two adjacent frames of images by tracking corner points with high brightness and obvious contrast. Then, based on the homography matrix and the coordinates of the target excavation point on the previous frame of the image, the target excavation point in the next frame of the image is calculated, greatly improving the accuracy of tracking the target excavation point. In this way, the accurate three-dimensional coordinates of the target excavation point can be obtained based on the accurately tracked two-dimensional coordinates of the target excavation point, thereby achieving the purpose of accurately completing the excavation task. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the structure of a camera's shooting range provided in an embodiment of this application;

[0023] Figure 2 A flowchart illustrating a method for tracking the digging point of an excavator, provided as an embodiment of this application;

[0024] Figure 3 A flowchart illustrating a remote control method for an excavator provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of the structure of a control device for an excavator provided in an embodiment of this application;

[0026] Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] In view of this, this application provides a method, device, electronic device and storage medium for tracking the digging point of an excavator, thereby solving the problem in the prior art that when the target digging point is not within the shooting range of the excavator's solid-state lidar, the remote control device cannot accurately track the target digging point as the excavator moves toward the target digging point.

[0028] To enable those skilled in the art to better understand the technical solutions of this application, the application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. However, this application can be implemented in many other ways different from those described above. Therefore, based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described herein. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0030] First, the technical background of this application will be explained:

[0031] Remote control of excavators can solve the problem of harsh and dangerous construction environments, and the most common method currently is to use automated control technology to complete remote control operations. Specifically, the excavator's sensors detect the surrounding environment and transmit the corresponding two-dimensional image to the remote control device. When the remote operator selects a target excavation point on the two-dimensional image, the remote control device needs to obtain the corresponding three-dimensional coordinates of the target excavation point. Then, the remote control device uses control signals to control the movement of the excavator's tires, boom, arm, and bucket. Through their combined movement, the excavator arm reaches the target excavation point and then moves in both horizontal and vertical planes to complete the excavation task.

[0032] Existing excavator sensing devices typically include cameras and solid-state LiDAR. Cameras capture two-dimensional images of the surrounding environment, while LiDAR captures three-dimensional images. When a remote control device operates the excavator, the operator first selects a target excavation point on the two-dimensional image, and then the remote control device moves the excavator's end effector to that point. Understandably, the remote control device needs to plan the movement route based on the three-dimensional coordinates of the target excavation point. If the target excavation point is also within the LiDAR's field of view, the two-dimensional coordinates of the target excavation point can be converted into three-dimensional coordinates based on the projection mapping relationship between the three-dimensional image captured by the LiDAR and the two-dimensional image captured by the camera.

[0033] However, the camera's shooting range is typically much larger than that of a solid-state LiDAR. The target excavation point selected by the remote operator may not be within the LiDAR's shooting range. Therefore, it's impossible to obtain the three-dimensional coordinates of the target excavation point from the two-dimensional image. Usually, when the target excavation point is outside the LiDAR's shooting range, the excavator needs to be controlled to move towards the target excavation point so that it first falls within the LiDAR's shooting range, and then the two-dimensional coordinates of the target excavation point are transformed according to the aforementioned projection mapping relationship. Therefore, how the remote control device tracks the target excavation point during movement becomes a crucial problem to solve.

[0034] To address the aforementioned technical problems, this application provides a method, apparatus, electronic device, and storage medium for tracking excavation points using an excavator. When the target excavation point is not within the observable range of a lidar, the excavator is controlled to move towards the target excavation point. During the excavator's movement, the image coordinates of the target excavation point on the previous frame of a two-dimensional image are first acquired. Then, based on the homography matrix between the previous and subsequent two-dimensional images, the image coordinates of the target excavation point on the next frame of a two-dimensional image are calculated, thereby determining the position of the target excavation point on the next frame of a two-dimensional image, thus achieving the purpose of tracking the target excavation point. The method, apparatus, terminal, and computer-readable storage medium of this application will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0035] Before introducing the excavation point tracking method provided in the embodiments of this application, the shooting range of the camera and solid-state LiDAR will be introduced first. Figure 1 This is a schematic diagram illustrating the structure of a camera's shooting range, provided as an embodiment of this application. Figure 1As shown, 101 represents the camera's shooting range, and 102 represents the solid-state LiDAR's shooting range. It's understandable that both the camera and the LiDAR on the excavator are fixedly installed, meaning their relative positions are fixed. Therefore, the shooting angles of the camera and LiDAR are consistent at all times. Generally, the camera's shooting range is larger, while the LiDAR's is smaller, and the camera's shooting range will cover the LiDAR's. Therefore, in each image captured by the camera, the area that the LiDAR can capture is also fixed, namely area 102 in the figure. That is, in any fixed-size image captured by the camera, area 102 is the part that the LiDAR can capture, and the area outside of area 102 is the part that the LiDAR cannot capture. It's understandable that a location can only obtain its three-dimensional coordinates when it is illuminated by the LiDAR. Therefore, only the two-dimensional coordinates of the pixels in area 102 can be converted into three-dimensional coordinates according to the projection mapping relationship between the LiDAR and the camera. Pixels outside region 102 are not within the imaging range of the solid-state LiDAR, so their two-dimensional coordinates cannot be converted into three-dimensional coordinates. Therefore, region 102 is also referred to as the effective data region of the image.

[0036] Understandably, when the target excavation point selected by the remote operator of the excavator is outside area 102, the three-dimensional coordinates of the target excavation point cannot be obtained. Only when the target excavation point is within area 102 can its two-dimensional coordinates in the image be converted into three-dimensional coordinates. Therefore, when the target excavation point is outside area 102, the excavator needs to be moved, thereby moving the camera and solid-state LiDAR on the excavator to bring the target excavation point into the shooting range of the camera and solid-state LiDAR, that is, into area 102. Therefore, during the movement, the position of the target excavation point in each frame of the image needs to be tracked continuously until the target excavation point falls into area 102. Once the target excavation point falls into area 102 in a certain frame of the image, the two-dimensional coordinates of the target excavation point in that frame of the image are converted into three-dimensional coordinates according to the projection mapping relationship between the solid-state LiDAR and the camera. Then, based on the three-dimensional coordinates of the target excavation point, a motion trajectory is planned, and the excavator's end effector is controlled to move to the target excavation point to carry out subsequent excavation operations.

[0037] Based on the above description Figure 2 This is a flowchart illustrating a method for tracking the digging point of an excavator, provided as an embodiment of this application. Figure 2 As shown, the excavation point tracking method includes the following steps:

[0038] 201. During the movement of the excavator, control the camera to capture multiple consecutive frames of images.

[0039] Understandably, during the remote control of an excavator, the remote operator needs to determine the target excavation point on the real-time image captured by the camera, and then plan the movement path based on that target excavation point to control the excavator's movement so that the excavator's end effector moves to the target excavation point. Specifically, if the target excavation point selected by the remote operator on the 2D image is within the observation range of the solid-state lidar, the 2D coordinates are directly converted to 3D coordinates based on the projection mapping relationship between the 2D image captured by the camera and the 3D image captured by the solid-state lidar. However, if the target excavation point is not within the observation range of the solid-state lidar, the excavator needs to be controlled to move (driving both the camera and the solid-state lidar) so that the target excavation point falls within the observation range of the solid-state lidar. In this case, during the excavator's movement, the camera needs to capture multiple frames of images in real time to track whether the target excavation point falls within the observation range of the solid-state lidar.

[0040] 202. Obtain the first image coordinates of the target excavation point to be tracked in the previous frame image.

[0041] During the tracking of target mining points, if the target mining point in the previous frame is not yet within the valid data area of ​​the previous frame (the target mining point is not within the observable range of the LiDAR), it is necessary to continue tracking the target mining point in the next frame. In this embodiment, the target mining point in the next frame is no longer found based on pixel matching, but is calculated based on the transformation relationship between the previous and next frames. Therefore, it is first necessary to obtain the first image coordinates (two-dimensional coordinates) of the target mining point in the previous frame, and then calculate the second image coordinates of the target mining point in the next frame based on the transformation relationship between the two frames.

[0042] 203. Based on the homography matrix between the previous and next frame images and the first image coordinates of the target excavation point in the previous frame image, calculate the second image coordinates of the target excavation point in the next frame image.

[0043] A homography matrix is ​​used to characterize the transformation relationship between two planes. Using the homography matrix, the two-dimensional coordinates of the target excavation point in the next frame image can be calculated based on the two-dimensional coordinates of the target excavation point in the previous frame image. In this way, the position of the target excavation point in the next frame image can be obtained, thus completing the tracking of the target excavation point.

[0044] 204. Determine the position of the target excavation point in the next frame image based on the second image coordinates of the target excavation point.

[0045] Based on the above description, after calculating the two-dimensional coordinates (second image coordinates) of the target excavation point on the next frame image, the position of the target excavation point on the next frame image can be determined. At this time, if the target excavation point has fallen into the effective data area of ​​the next frame image, then the second image coordinates of the target excavation point can be transformed into three-dimensional coordinates according to the coordinate mapping relationship between the camera and the LiDAR, and then path planning can be performed based on the three-dimensional coordinates of the target excavation point.

[0046] However, if the target tracking point is still not within the valid data area of ​​the current frame, it is necessary to continue tracking the target mining point in the next frame. The tracking process is similar to the steps described above, that is, calculating the two-dimensional coordinates of the target mining point in the next frame based on the homography matrix between the two frames, thereby determining the position of the target mining point in the next frame. Tracking of the target mining point ends when it falls into the valid data area of ​​a frame.

[0047] The technical solution provided in this application embodiment controls the excavator to move towards the target excavation point when the selected target excavation point on the two-dimensional image is not within the observable range of the lidar. During the excavator's movement, the image coordinates of the target excavation point on the two-dimensional image are first acquired. Then, based on the homography matrix between the two-dimensional image and the subsequent two-dimensional image, the image coordinates of the target excavation point on the next two-dimensional image are calculated, thereby determining the position of the target excavation point on the next two-dimensional image and achieving the purpose of tracking the target excavation point. In this application embodiment, the traditional pixel matching method is no longer used to track the target excavation point, avoiding the problem of large pixel matching errors due to similar visual textures in the excavator's working scene and unclear target excavation points, thus failing to accurately track the target excavation point. This method calculates the target excavation point in the subsequent image based on the homography matrix between two consecutive image frames and the coordinates of the target excavation point on the previous image, greatly improving the accuracy of tracking the target excavation point. This allows for the accurate three-dimensional coordinates of the target excavation point to be obtained based on the accurately tracked two-dimensional coordinates, thereby achieving the goal of accurately completing the excavation task.

[0048] In conjunction with the above embodiments, Figure 3 This is a flowchart illustrating a remote control method for an excavator provided in an embodiment of this application. It is understood that... Figure 2 The illustrated embodiment describes the tracking of a target excavation point. The purpose of tracking the target excavation point is to obtain its three-dimensional coordinates in order to control the excavator's end effector to move to that point for excavation. Therefore, Figure 3 The illustrated embodiment will describe the entire remote control process of the excavator. For example... Figure 3 As shown, the remote control process includes the following steps:

[0049] 301. Control the movement of the excavator and control the camera to continuously capture multiple frames of images during the movement.

[0050] If the target excavation point determined by the remote operator is not within the effective data range of the real-time image, the excavator needs to be moved to bring the target excavation point into the observation range of the solid-state lidar. During the excavator's movement, the camera needs to capture multiple frames of images in real time to track whether the target excavation point has fallen into the observation range of the solid-state lidar.

[0051] 302. Preprocess each frame of the image and select corner points in the previous frame of the preprocessed image.

[0052] Before tracking, each frame of the image captured by the camera needs to be preprocessed. For example, adaptive local histogram equalization is required to increase image contrast. This improves corner matching accuracy when calculating the homography matrix between two frames, thereby increasing the precision of the homography matrix.

[0053] After preprocessing, it is necessary to calculate the homography matrix between two adjacent image frames. Steps 303 to 305 describe the process of determining the homography matrix. When calculating the homography matrix between two image frames, multiple corner points are first selected on the previous image frame. Then, these corner points are tracked on the next image frame, and the offset of each corner point is used to determine the offset between the two images, thus obtaining the transformation relationship between the two images. For example, multiple pixels on the previous image frame can be extracted as corner points at fixed intervals, and it is necessary to ensure that the extracted corner points are evenly distributed across the entire image.

[0054] 303. Perform pixel matching on the previous frame and the next frame, and determine the corner matching point corresponding to the corner point in the next frame.

[0055] Specifically, when tracking corner points, it's necessary to determine the corresponding corner point in the next frame using traditional pixel matching methods. Understandably, to improve the accuracy of the homography matrix, it's necessary to improve the corner point matching precision. Therefore, when selecting corner points, pixels with high brightness and high contrast can be chosen.

[0056] 304. Obtain the second image coordinates of each corner matching point in the next frame image.

[0057] After matching the corner points, it's necessary to obtain the two-dimensional coordinates (second image coordinates) of these corner points in the next frame. Understandably, the number of selected corner points is limited, but to improve the accuracy of the homography matrix calculation, iterative calculations are required. Therefore, a large amount of corner and corner matching point data is needed. For example, optical flow vectors can be used to expand the number of corner points.

[0058] Specifically, assuming the coordinates of the selected corner point in frame t are (x, y), the light intensity of this corner point can be represented as I(x, y, t). Then, this corner point moves a distance of (dx, dy) and takes time dt to reach frame t+1. The light intensity of the matching point of this corner point in frame t+1 can then be represented as I(x+dx, y+dy, t+dt). Since the corner point and its matching point are the same pixel, and the light intensity of the same pixel remains unchanged before and after the movement, therefore:

[0059] I(x,y,t)=I(x+dx,y+dy,t+dt)

[0060] Performing a Taylor expansion on the above formula, we get:

[0061]

[0062] Where ε represents a second-order infinitesimal term, therefore, the above formula can be obtained as follows:

[0063]

[0064] Assume u and v are the velocity vectors of the optical flow along the x and y directions, respectively, that is:

[0065]

[0066] At this time, Let X, Y, and T represent the partial derivatives of the gray level of a pixel in the image, respectively. Then:

[0067] I X u+I y v+I t =0

[0068] Among them, I x I y and I tBoth can be obtained from image data, allowing the calculation of the optical flow vector (u, v). This optical flow vector can then be used to characterize pixel position deviation. It's understandable that all adjacent pixels have the same motion; that is, within a predetermined region around a pixel, all pixels possess the same optical flow vector. Therefore, the coordinates of the matching points of pixels surrounding a selected corner point in the next frame can be determined based on the optical flow vector corresponding to that corner point. This allows for the expansion of corner points and their matching points to calculate the homography matrix between two image frames.

[0069] 305. Calculate the homography matrix between the previous and next frames based on the first image coordinates of the corner point in the previous frame and the second image coordinates of the corner point matching point in the next frame.

[0070] Once the coordinates of the corner points in the previous frame and the coordinates of the matching corner points in the next frame are determined, the homography matrix between the two frames can be calculated. First, the corner points in the previous frame are... The corner matching point of the next frame is The homography matrix H is at this time:

[0071]

[0072] Multiplying the above formula by a non-zero factor, and setting h9 equal to 1, we obtain the following linear relationship:

[0073] h1x1+h2y1+h3-h7x1x2-h8x1x2=x2

[0074] h4x1+h5y1+h6-h7x1y2-h8y1y2=y2

[0075] From the above formula, we can see that a set of corner points and corner matching points can construct two constraints. Therefore, a homography matrix with 8 degrees of freedom can be solved by using a system of linear equations about the homography matrix H through four sets of corner points and corner matching points. The values ​​of h1 to h8 are calculated using the system of linear equations, thus obtaining the homography matrix.

[0076] Therefore, when calculating the homography matrix, four groups of corner points are selected for calculation each time. To improve the accuracy of the homography matrix, iterative calculations can be performed, and the homography matrix can be corrected based on the results of multiple calculations. For example, nonlinear optimization can also be used to optimize the homography matrix H, assuming P′ t，k P is a two-dimensional point transformed from a corner point in frame t using the homography matrix H, while P t+1，k These are the corner matching points corresponding to the corner points in the (t+1)th frame of the image. Therefore, the accuracy of solving the homography matrix H can be improved by minimizing the reprojection error of the n sets of tracking points. The specific formula is as follows:

[0077]

[0078] To solve the aforementioned least squares problem, the homography matrix can be used as an optimization variable, and the Levenberg-Marquardt method (LM) can be used to calculate a more accurate homography matrix.

[0079] Understandably, the homography matrix between two adjacent frames is calculated based on the two-dimensional coordinates of corner points in the two frames. Therefore, during corner tracking, some poor-quality tracking points can be filtered out. Consequently, the number of corner points in subsequent images will decrease. When the number of corner points in a frame is less than a set threshold, corner points can be reselected in areas with fewer corner points to ensure that each tracking image has a uniformly distributed and sufficient number of corner points.

[0080] 306. Based on the homography matrix between the previous and next frames and the two-dimensional coordinates of the target mining point in the previous frame, calculate the two-dimensional coordinates of the target mining point in the next frame. After obtaining the homography matrix, it can be used to calculate the two-dimensional coordinates of the target mining point in the next frame. Specifically, first, convert the coordinates P(x, y) of the target mining point in the previous frame into homogeneous term form P(x, y, 1), and then use the optimized homography matrix H to transform it to calculate the target mining point P in the next frame. t , specifically

[0081] P t =HP

[0082] Then, it is normalized to obtain the two-dimensional coordinates of the target excavation point on the next frame image.

[0083] 307. Based on the second image coordinates of the target excavation point, determine whether the target excavation point is within the valid data area of ​​the next frame image.

[0084] After calculating the two-dimensional coordinates (second image coordinates) of the target excavation point on the next image frame, the position of the target excavation point on the next image frame can be determined. At this point, if the target excavation point has fallen within the valid data area of ​​the next image frame, then the second image coordinates of the target excavation point can be transformed into three-dimensional coordinates according to the coordinate mapping relationship between the camera and the LiDAR. If it has not fallen within the valid data area, then tracking continues.

[0085] Based on the coordinate mapping relationship between the camera and the lidar, the second image coordinates of the target excavation point are transformed into three-dimensional coordinates.

[0086] Once the target excavation point enters the effective data area of ​​the image, it is necessary to transform the second image coordinates of the target excavation point into three-dimensional coordinates based on the coordinate mapping relationship between the camera and the LiDAR. Specifically, a rectangular frame is constructed by extending 10 pixels up, down, left, and right from the center of the two-dimensional coordinates, and then the 3D point cloud data of the projected points within the rectangular frame is calculated. Wherein, if P represents the relative extrinsic parameters of the solid-state lidar and the camera. L P is a three-dimensional point in the lidar coordinate system. C For a 3D point in the corresponding camera coordinate system, Z is point P. C The distance to the camera's optical center, K is the camera's intrinsic parameter, and (u,v) is a two-dimensional point projected onto the image plane. If (u,v) is within the rectangle, then its corresponding point P... C For the point inside the box, then:

[0087]

[0088]

[0089] Next, a KD-tree is constructed based on the point cloud within the bounding box, and clustering is performed according to Euclidean distance. If the number of clusters is 0, it means that there is no laser point cloud data in the clicked area. If the number of clusters is 1, the cluster center is calculated as the 3D point corresponding to the 2D point. If the number of clusters exceeds 1, only the point cloud of the cluster with the most points is retained as the current object point cloud, and the cluster center is calculated as the 3D point corresponding to the 2D point.

[0090] 308. Control the excavator's end effector to move to the target excavation point based on the three-dimensional coordinates of the target excavation point.

[0091] Next, the three-dimensional point is converted to the excavator coordinate system, and the end position is controlled to reach the three-dimensional coordinates through the control algorithm, and the boom, arm, and bucket are controlled to perform digging operations.

[0092] Based on the above method embodiments, Figure 4 This is a schematic diagram of the structure of a control device for an excavator provided in an embodiment of this application, as shown below. Figure 4 As shown, the excavator is equipped with a camera and a lidar, with their relative positions fixed. The camera's observation range covers the lidar's observation range. The control device includes:

[0093] The acquisition unit 401 is used to acquire a series of multiple frames of images captured by the camera during the movement of the excavator.

[0094] The acquisition unit 401 is also used to acquire the first image coordinates of the target excavation point to be tracked on the T-th frame image, wherein the target excavation point is not within the observable range of the lidar, and T is a positive integer greater than 0.

[0095] Processing unit 402 is used to calculate the second image coordinates of the target excavation point on the T+1 frame image based on the first homography matrix between the T-th frame image and the T+1-th frame image and the first image coordinates of the target excavation point.

[0096] The determining unit 403 is used to determine the position of the target excavation point on the T+1 frame image based on the second image coordinates of the target excavation point.

[0097] In an alternative embodiment, the device further includes a control unit 404.

[0098] The processing unit 402 is further configured to, if the target excavation point is within the effective data area of ​​the T+1 frame image, convert the second image coordinates of the target excavation point into three-dimensional coordinates according to the coordinate mapping relationship between the camera and the lidar. The effective data area corresponds to the observation range of the lidar.

[0099] Control unit 404 is used to control the movement of the excavator according to three-dimensional coordinates so that the end of the excavator reaches the target excavation point.

[0100] In an optional implementation, the processing unit 402 is further configured to, if the target excavation point is not within the valid data area of ​​the T+1 frame image, continue to track the position of the target excavation point on the T+2 frame image based on the second homography matrix between the T+1 frame image and the T+2 frame image and the second image coordinates.

[0101] In an optional implementation, the acquisition unit 401 is further configured to select multiple corner points on the T-frame image and acquire the first image coordinates corresponding to each corner point.

[0102] The determining unit 403 is also used to perform pixel matching on the T+1 frame image to determine the corner matching point corresponding to the corner point on the T+1 frame image.

[0103] The acquisition unit 401 is also used to acquire the second image coordinates corresponding to each corner matching point.

[0104] The processing unit 402 is further configured to calculate the first homography matrix between the T-th frame image and the T+1-th frame image based on the first image coordinates corresponding to the corner point and the second image coordinates corresponding to the corner point matching point.

[0105] In an optional implementation, the processing unit 402 is further configured to perform local histogram equalization processing on the T-th frame image, the local histogram equalization processing being used to increase the image contrast of the T-th frame image.

[0106] In an optional implementation, the acquisition unit 401 is specifically used to select multiple corner points on the T-th frame image after the image contrast is increased, based on the pixel brightness value and a preset fixed distance. The multiple corner points are evenly distributed on the T-th frame image.

[0107] In an optional implementation, the determining unit 403 is specifically used to obtain at least four target corner points among a plurality of corner points, and to determine the target corner point matching point for each of the at least four target corner points.

[0108] The processing unit 402 is specifically used to calculate the first homography matrix between the T-th frame image and the T+1-th frame image based on the first image coordinates of at least four target corner points and the second image coordinates corresponding to the matching points of at least four target corner points.

[0109] In an optional implementation, the processing unit 402 is further configured to calculate the second image coordinates corresponding to the corner calculation point of each corner point based on the first image coordinates corresponding to each corner point and the first homography matrix.

[0110] The determining unit 403 is also used to determine the coordinate error corresponding to each corner point based on the second image coordinates corresponding to each corner point matching point and the second image coordinates corresponding to each corner point calculation point.

[0111] The processing unit 402 is also used to optimize the first homography matrix based on the coordinate errors corresponding to all corner points.

[0112] In an optional implementation, the determining unit 403 is further configured to filter corner matching points and determine a target corner matching point among the corner matching points. Pixel matching is performed on the (T+2)th frame image to determine the secondary corner matching point corresponding to the target corner matching point on the (T+2)th frame image, and the third image coordinates corresponding to the secondary corner matching point are obtained.

[0113] The processing unit 402 is also used to calculate the second homography matrix between the T+1 frame image and the T+2 frame image based on the second image coordinates corresponding to the target corner matching point and the third image coordinates corresponding to the secondary corner matching point.

[0114] In an optional implementation, the acquisition unit 401 is further configured to select target pixels on the T+1 frame image. Based on a preset number, the target pixels are used to supplement the target corner matching points.

[0115] The technical solution provided in this application embodiment controls the excavator to move towards the target excavation point when the selected target excavation point on the two-dimensional image is not within the observable range of the lidar. During the excavator's movement, the image coordinates of the target excavation point on the two-dimensional image are first acquired. Then, based on the homography matrix between the two-dimensional image and the next frame of the two-dimensional image, the image coordinates of the target excavation point on the next frame of the two-dimensional image are calculated, thereby determining the position of the target excavation point on the next frame of the two-dimensional image, thus achieving the purpose of tracking the target excavation point. In this application embodiment, the traditional pixel matching method is no longer used to track the target excavation point, avoiding the problem of large pixel matching errors caused by similar visual textures in the excavator's working scene and unclear target excavation points, which prevents accurate tracking of the target excavation point. This method can calculate the homography matrix between two adjacent frames of images by tracking corner points with high brightness and obvious contrast. Then, based on the homography matrix and the coordinates of the target excavation point on the previous frame of the image, the target excavation point in the next frame of the image is calculated, greatly improving the accuracy of tracking the target excavation point. In this way, the accurate three-dimensional coordinates of the target excavation point can be obtained based on the accurately tracked two-dimensional coordinates of the target excavation point, thereby achieving the purpose of accurately completing the excavation task.

[0116] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software through processing element calls; they can be fully implemented in hardware; or some modules can be implemented through processing element calls in software, while others are implemented in hardware. Moreover, these modules can be fully or partially integrated together, or implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through the integrated logic circuits in the hardware of the processor element or through software instructions.

[0117] The following describes an electronic device provided by an embodiment of this application. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device 800 may be equipped with... Figure 4 The control device for the excavator described in the corresponding embodiment is used to implement Figures 1 to 3 The functions correspond to those in the embodiments. Specifically, the electronic device 800 includes: a receiver 801, a transmitter 802, a processor 803, and a memory 804 (wherein the number of processors 803 in the execution device 800 can be one or more). Figure 5(Taking a processor as an example), the processor 803 may include an application processor 8031 ​​and a communication processor 8032. In some embodiments of this application, the receiver 801, transmitter 802, processor 803, and memory 804 may be connected via a bus or other means.

[0118] Memory 804 may include read-only memory and random access memory, and provides instructions and data to processor 803. A portion of memory 804 may also include non-volatile random access memory (NVRAM). Memory 804 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0119] The processor 803 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0120] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 803. Processor 803 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 803 or by instructions in software form. Processor 803 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 803 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 804, and processor 803 reads the information from memory 804 and, in conjunction with its hardware, completes the steps of the above method.

[0121] Receiver 801 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 802 can be used to output digital or character information through the first interface; transmitter 802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 802 may also include a display device such as a display screen.

[0122] In this embodiment of the application application, the application processor 8031 ​​in the processor 803 is used to execute... Figures 1 to 3 The corresponding embodiment describes a method for tracking the digging point of an excavator. It should be noted that the specific manner in which the application processor 8031 ​​executes each step differs from that in this application. Figures 1 to 3 The various method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figures 1 to 3 The corresponding method embodiments are the same, and for details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0123] This application also provides a chip for executing instructions, which is used to execute the technical solution of the file data storage method in the above embodiments.

[0124] This application also provides a computer-readable storage medium storing computer instructions. When these computer instructions are executed on a server, the server performs the technical solution of the excavator digging point tracking method described in the above embodiments.

[0125] This application also provides a computer program product, including a computer program, which, when executed by a processor, is used to perform the technical solution of the excavation point tracking method for excavators described in the above embodiments.

[0126] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to general-purpose or special-purpose servers.

[0127] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0128] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A method of tracking a digging point with respect to a power shovel, characterized by, The excavator is provided with a camera and a laser radar, and the relative positions of the camera and the laser radar are fixed; the observation range of the camera covers the observation range of the laser radar, and the method comprises the following steps: During the movement of the excavator, a plurality of continuous image frames captured by the camera are acquired; A first image coordinate of a target digging point to be tracked on a Tth image frame is acquired; the target digging point is not within the observable range of the laser radar; the T is a positive integer greater than 0; A second image coordinate of the target digging point on a (T+1)th image frame is calculated according to a first homography matrix between the Tth image frame and the (T+1)th image frame and the first image coordinate of the target digging point; The position of the target digging point on the (T+1)th image frame is determined according to the second image coordinate of the target digging point; If the target digging point is within an effective data area of the (T+1)th image frame, the second image coordinate of the target digging point is converted into a three-dimensional coordinate according to the coordinate mapping relationship between the camera and the laser radar; the effective data area corresponds to the observation range of the laser radar; the three-dimensional coordinate is used to control the movement of the excavator so that the end of the excavator reaches the target digging point; If the target digging point is not within the effective data area of the (T+1)th image frame, the excavator is controlled to move towards the target digging point according to a second homography matrix between the (T+1)th image frame and a (T+2)th image frame and the second image coordinate, and the position of the target digging point on the (T+2)th image frame is continuously tracked.

2. The method of claim 1, wherein, The method further comprises the following steps: A plurality of corner points on the Tth image frame are selected, and a first image coordinate corresponding to each corner point is acquired; Pixel matching is performed on the (T+1)th image frame to determine a corner point matching point corresponding to each corner point on the (T+1)th image frame, and a second image coordinate corresponding to each corner point matching point is acquired; The first homography matrix between the Tth image frame and the (T+1)th image frame is calculated according to the first image coordinate corresponding to each corner point and the second image coordinate corresponding to each corner point matching point.

3. The method of claim 2, wherein, Before the step of selecting corner points on the Tth image frame, the method further comprises the following step: The Tth image frame is subjected to local histogram equalization processing; the local histogram equalization processing is used to increase the image contrast of the Tth image frame.

4. The method of claim 3, wherein, The step of selecting corner points on the Tth image frame comprises the following step: A plurality of corner points are selected on the Tth image frame after the image contrast is increased according to pixel brightness values and a preset fixed distance; the plurality of corner points are uniformly distributed on the Tth image frame.

5. The method of claim 4, wherein, The step of calculating the first homography matrix between the Tth image frame and the (T+1)th image frame according to the first image coordinate corresponding to each corner point and the second image coordinate corresponding to each corner point matching point comprises the following steps: At least four target corner points are acquired from the plurality of corner points; Target corner point matching points of each corner point in the at least four target corner points are respectively determined; According to the first image coordinates of the at least four target corner points and the second image coordinates corresponding to the at least four target corner point matching points, the first homography matrix between the Tth frame image and the T+1th frame image is calculated.

6. The method of claim 5, wherein, The method further comprises: According to the first image coordinates corresponding to each corner point and the first homography matrix, the second image coordinates corresponding to the corner point calculation points corresponding to each corner point are calculated. According to the second image coordinates corresponding to each corner point matching point and the second image coordinates corresponding to the corner point calculation points, the coordinate error corresponding to each corner point is determined. According to the coordinate errors corresponding to all corner points, the first homography matrix is optimized.

7. The method according to any one of claims 5 to 6, characterized in that, The method further comprises: The corner point matching points are screened to determine target corner point matching points in the corner point matching points. Pixel matching is performed on the T+2th frame image to determine secondary corner point matching points corresponding to the target corner point matching points on the T+2th frame image, and third image coordinates corresponding to the secondary corner point matching points are obtained. According to the second image coordinates corresponding to the target corner point matching points and the third image coordinates corresponding to the secondary corner point matching points, the second homography matrix between the T+1th frame image and the T+2th frame image is calculated.

8. The method of claim 7, wherein, The method further comprises: A target pixel point is selected on the T+1th frame image. According to a preset number, the target pixel point is used to supplement the target corner point matching points.

9. A control device for a power shovel, characterized by comprising: The excavator is provided with a camera and a laser radar, and the relative positions of the camera and the laser radar are fixed; the observation range of the camera covers the observation range of the laser radar; and the control device comprises: An acquisition unit is configured to acquire continuous multiple frames of images captured by the camera during the movement of the excavator. The acquisition unit is further configured to acquire first image coordinates of a target digging point to be tracked on a Tth frame of image; the target digging point is not within the observable range of the laser radar; and T is a positive integer greater than 0. A processing unit is configured to calculate second image coordinates of the target digging point according to a first homography matrix between the Tth frame of image and a T+1th frame of image and the first image coordinates of the target digging point; and the homography matrix is used to indicate the mapping relationship between the Tth frame of image and the T+1th frame of image. A determination unit is configured to determine the position of the target digging point on the T+1th frame of image according to the second image coordinates of the target digging point. The processing unit is further configured to, if the target digging point is within an effective data area of the T+1th frame of image, convert the second image coordinates of the target digging point into three-dimensional coordinates according to the coordinate mapping relationship between the camera and the laser radar; the effective data area corresponds to the observation range of the laser radar. A control unit is configured to control the movement of the excavator according to the three-dimensional coordinates, so that the end of the excavator reaches the target digging point. The processing unit is further configured to, if the target digging point is not in the valid data region of the T+1th frame image, control the excavator to move to the target digging point according to a second homography matrix between the T+1th frame image and a T+2th frame image and the second image coordinate, and continue to track the position of the target digging point on the T+2th frame image.

10. An electronic device, comprising: The method comprises: a processor, a memory, and computer program instructions stored on the memory and executable on the processor; the processor executes the computer program instructions to implement the method for tracking a digging point of an excavator according to any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method for tracking a digging point of an excavator according to any one of claims 1 to 8.

Images