A UAV system for roaming on a vertical facade and a method for constructing an environmental model

Abstract

The present invention discloses a drone system for roaming operations on a vertical facade and a method for constructing an environmental model, comprising a drone and a fixed bracket, wherein a horizontal belt is installed at the upper end of the fixed bracket, the belt is driven by a stepper motor, a guide wheel is installed at the lower side of the belt, and a first traction wheel and a second traction wheel are installed on the left and right sides of the fixed bracket, respectively, and the drone is respectively connected with electric wires, a flexible pipe for feeding during operation, and a safety rope, wherein the electric wires and the safety rope are connected to the first traction wheel as a bundle passing around the guide wheel, and the flexible pipe for feeding during operation is connected to the second traction wheel. The friction force with the facade can be increased by using the drone's counter-torque, and this new method is a replacement for the traditional suction cup type and negative pressure adsorption type facade operation robot; the drone's positive torque and visual SLAM technology are used to cross obstacles on the facade; the smoothness of the operation is ensured, which is conducive to improving the operation efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle operation, and specifically relates to an unmanned aerial vehicle system for roaming operation on a vertical facade and a method for constructing an environmental model. Background Art

[0002] As the name implies, facade drones are robots that walk on facades. They have a wide range of application scenarios, such as spraying, installing, and cleaning building facades. They can replace the Spider-Man that we often see. At present, there are many brands of facade operation drones on the market. For example, BOBOT, R-Storm, Yingxing Intelligence and other companies have corresponding products, but most of them use negative pressure adsorption and suction cup adsorption to stick the robot to the facade for operation. There is often a lack of adhesion with the facade, and it may fall off, posing a safety hazard. In addition, there are protruding frames on the facades of building glass walls, which constitute obstacles to movement. The current facade operation robots still have the problem of difficulty in crossing obstacles. Summary of the invention

[0003] The purpose of the present invention is to provide a UAV system for roaming on a vertical facade and a method for constructing an environmental model to solve the problems raised in the above-mentioned background technology.

[0004] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a drone system for roaming operations on a vertical facade, comprising a drone and a fixed bracket installed on the upper part of the facade, a horizontal belt is installed on the upper end of the fixed bracket, the belt is driven by a stepper motor, a guide wheel is installed on the lower side of the belt, and a first traction wheel and a second traction wheel are installed on the left and right sides of the fixed bracket respectively, the drone is respectively connected with electric wires, a flexible pipe for feeding materials during operation, and a safety rope, the electric wires and the safety rope are combined into a bundle and passed around the guide wheel to be connected to the first traction wheel, the flexible pipe is connected to the second traction wheel, and the drone uses counter-torque to press on the vertical facade.

[0005] The present invention also discloses a method for constructing an environment model of a drone roaming on a vertical facade, comprising the following steps:

[0006] S1. Obtaining images: obtaining real-time image data captured by the onboard binocular camera of the drone;

[0007] S2. Build the front-end vertical elevation visual odometry: Expand the single-layer direct method to the pyramid-like multi-layer direct method, use the coarse-to-fine process to calculate the relative motion, and then you can get the binocular camera pose derived by the multi-layer direct method.

[0008] S3, back-end optimization: The back-end is mainly filtering and nonlinear optimization algorithms. The back-end optimization uses the improved sliding window BA method to estimate the state of the entire system from noisy data, as well as the maximum a posteriori probability estimate of the uncertainty of this state. The state includes both the trajectory of the drone itself and the constructed pose graph;

[0009] S4, loop detection: The purpose is to eliminate drift. This task is accomplished by using the carried binocular vision sensor, that is, using the loop detection algorithm of the image appearance to complete the similarity loop detection. In the loop detection algorithm of the appearance, for image A and image B, it is necessary to calculate the similarity score between them: s(A,B)=||AB||. The similarity score will be within a certain range. When the similarity score is greater than a certain amount, a loop occurs.

[0010] S5. Mapping: Dense mapping using binocular vision sensors is to convert image data into point clouds based on the estimated camera pose, and then stitch them together to obtain a point cloud map composed of discrete points. On this basis, the surface of the vertical facade is estimated, and triangular meshes and patches are used for mapping. In addition, obstacle information is drawn on the map and navigation is performed on the map, and an occupied grid map can be established through voxels.

[0011] S6. Adjust attitude and position information: Use the UAV's facade vision technology to estimate and adjust its own flight status and position information in real time;

[0012] S7. Work: When the facade work area is identified, the external working tools on the drone will automatically start working.

[0013] Preferably, the binocular cameras image the same spatial coordinate point, and the pixel coordinates are recorded as p1 and p2 respectively. The principle of the direct method is to find the position of p2 according to the estimated value of the current camera's posture. However, if the camera posture is not good enough, the appearance of p2 will be significantly different from that of p1. In order to reduce this difference, the camera posture is optimized to find p2 that is more similar to p1.

[0014] Preferably, the BA method includes establishing a graph optimization model, constructing an overall Hessian matrix, and then marginalizing all landmark points to solve.

[0015] Compared with the prior art, the beneficial effects of the present invention are: the drone is lifted by a safety rope, and the horizontal and vertical movements of the drone are controlled by the horizontal movement and extension of the safety rope to prevent the drone from falling off, thereby improving safety; the reverse torque of the drone can be used to increase the friction with the facade, and this new method is a replacement for the traditional suction cup type and negative pressure adsorption type facade operation robots; the positive torque of the drone and the visual SLAM technology are used to cross obstacles on the facade to ensure smooth operation, which is conducive to improving operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a structural schematic diagram of the UAV system of the present invention;

[0017] Figure 2 Instant localization and mapping of vertical facades;

[0018] Figure 3 is a control flow chart of the present invention;

[0019] Figure 4 It is a schematic diagram of the direct method;

[0020] Figure 5 Removing keyframes for the sliding window will destroy the diagonal block structure of the landmark part;

[0021] Figure 6 It is a schematic diagram of the posture graph.

[0022] In the figure: 1. UAV; 2. Belt; 3. First traction wheel; 4. Second traction wheel; 5. Fixed bracket; 6. Guide wheel. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0024] The present invention discloses a drone system for roaming operation on a vertical facade, comprising a drone 1 and a fixed bracket 5 installed on the upper part of the facade, a horizontal belt 2 is installed on the upper end of the fixed bracket 5, the belt 2 is driven and operated by a stepping motor, a guide wheel 6 is installed on the lower side of the belt 2, a first traction wheel 3 and a second traction wheel 4 are installed on the left and right sides of the fixed bracket 5, respectively, the drone 1 is connected with electric wires, a flexible pipe and a safety rope, respectively, the electric wires and the safety rope are combined into a bundle and passed around the guide wheel 6 to be connected to the first traction wheel 3, the flexible pipe is connected to the second traction wheel 4, the drone 1 is pressed on the vertical facade by means of a counter torque, the belt 2 controls the drone 1 to move in the X direction, the retraction and release of the safety rope pulls the drone 1 to move in the Y direction, and the flexible pipe is used to transport consumables required for the operation.

[0025] The present invention also discloses a method for constructing an environment model of a drone roaming on a vertical facade, which specifically includes the following steps:

[0026] S1. Obtaining images: obtaining real-time image data captured by the onboard binocular camera of the drone;

[0027] S2. Constructing the front vertical elevation visual odometry: Expanding the single-layer direct method to the pyramid-like multi-layer direct method, using the coarse-to-fine process to calculate the relative motion, the UAV binocular camera pose derived by the multi-layer direct method can be obtained;

[0028] The task of the vertical facade visual odometer is to estimate the movement of the camera between adjacent images of the facade, as well as the appearance of the local map. In facade visual SLAM, the front-end is more related to the research field of computer vision, such as feature extraction and matching of images. Due to the errors in the visual odometer, cumulative drift is inevitable, so back-end optimization and loop detection are required. A multi-layer direct method is used to construct a vertical facade visual odometer. In optical flow, it is first necessary to track the position of feature points, and then determine the movement of the camera based on these positions. It is difficult to ensure global optimality using such a two-step solution. The present invention adjusts the result of tracking the position of feature points in determining the movement of the camera.

[0029] like Figure 4 As shown, consider a spatial point Р and two cameras at two moments. The world coordinates of P are [X, Y, Z], and it is imaged on two cameras, and the pixel coordinates are recorded as p1 and p2.

[0030] The goal is to find the relative pose transformation from the first camera to the second camera. Taking the first camera as the reference system, let the rotation and translation of the second camera be R,t (the corresponding Lie group is T). At the same time, the internal parameters of the two cameras are the same, denoted by K. For clarity, write the complete projection equation:

[0031]

[0032]

[0033] Where Z is the depth of P, Z2 is the depth of P in the second camera coordinate system, which is the third coordinate value of RP+t.

[0034] In the feature point method, since the pixel positions of p1 and p2 are known through matching descriptions, the reprojection position can be calculated. However, in the direct method, since there is no feature matching, there is no way to know which p2 corresponds to the same point as p1. The idea of ​​the direct method is to find the position of p2 based on the estimated value of the current camera's pose. However, if the camera pose is not good enough, the appearance of p2 will be significantly different from that of p1. Therefore, in order to reduce this difference, the camera pose is optimized to find p2 that is more similar to p1. This can also be done by solving an optimization problem, but this time it is not the reprojection error that is minimized, but the photometric error, that is, the brightness error of the two pixels of P:

[0035] e=I1(p1)-I2(p2)

[0036] Here e is a scalar. Similarly, the optimization target is the second norm of the error, temporarily taking the unweighted form, which is:

[0037] The reason for this optimization is still based on the grayscale invariance assumption. Assume that the grayscale of a spatial point is constant under different viewing angles. If there are many (for example, N) spatial points P, then the entire camera pose estimation problem becomes:

[0038]

[0039] The optimization variable here is the camera pose T, instead of optimizing the movement of each feature point like optical flow. In order to solve this optimization problem, it is necessary to analyze the derivative relationship of the error e as the camera pose T changes. Therefore, two intermediate variables are defined:

[0040]

[0041] Here q is the coordinate of Р in the second camera coordinate system, and u is its pixel coordinate. Obviously q is a function of T, and u is a function of q, and thus also a function of T. Consider the left perturbation model of Lie algebra, using the first-order Taylor expansion, because

[0042] e(T)=I1(p1)-I2(u),

[0043] so

[0044]

[0045] Where δξ is the left perturbation of T. It can be seen that the first-order derivative is divided into three terms due to the chain rule, and these three terms are easy to calculate.

[0046] 1. is the pixel gradient at u.

[0047] 2. is the derivative of the projection equation about the three-dimensional point in the camera coordinate system. Let q = [X, Y, Z] T , the derivative is

[0048] 3. is the derivative of the transformed three-dimensional point pair transformation:

[0049] Since the last two items are only related to the three-dimensional point q but not to the image, they need to be merged together.

[0050]

[0051] Based on this 2×6 matrix, the Jacobian matrix of the error relative to the Lie algebra can be derived:

[0052]

[0053] For the problem of N points, this method can be used to calculate the optimized Jacobian matrix, and then the Gauss-Newton method or Levenberg-Marquardt method can be used to calculate the increments and iteratively solve the problem. So far, the entire process of estimating the position and posture of the drone stereo camera by the direct method has been derived. We then expand the single-layer direct method to the pyramid-like multi-layer direct method, and use the Coarse-to-fine process to calculate the relative motion, and we can derive the entire process of the drone stereo camera position and posture derived by the multi-layer direct method.

[0054] S3, back-end (non-linear) optimization: The back-end is mainly filtering and non-linear optimization algorithms. The back-end optimization uses the sliding window BA method to estimate the state of the entire system from noisy data, as well as the maximum a posteriori probability estimate of the uncertainty of this state. The state here includes both the trajectory of the drone itself and the constructed posture graph. Figure 5 As shown, assuming that x1 sees the landmarks y1 to y4, then before processing, the Hessian matrix of the BA problem should be like Figure 4 Left Figure 1 Similarly, there are non-zero blocks in the columns y1 to y4 of the row x1, indicating that x1 has seen them. Now consider marginalizing x1, then the Schur elimination process is equivalent to eliminating several non-zero matrix blocks on the off-diagonal lines through matrix row and column operations. Obviously, this will cause the landmark point matrix block in the lower right corner to no longer be a non-diagonal matrix. This process is called filling in marginalization. The original sliding window BA method is not suitable for large-scale SLAM mapping systems. For this reason, it is necessary to do the following Figure 6 As shown in the optimization, when the landmark points in BA are no longer optimized and they are only considered as constraints on the pose nodes, a pose graph with a much smaller computational scale is obtained;

[0055] S4, loop detection: Simply put, it is to eliminate drift. This task is accomplished by using the carried binocular vision sensor, that is, using the loop detection algorithm of the image appearance to complete the similarity loop detection. In the loop detection algorithm of the appearance, for image A and image B, it is necessary to calculate the similarity score between them: s(A,B)=||AB||. This score will be within a certain range. When it is greater than a certain amount, we believe that a loop has occurred.

[0056] Simply put, it is to eliminate drift. This task is accomplished by using the carried binocular vision sensor, that is, the image itself. For example, the similarity of the image is used to complete loop detection. The front end provides feature point extraction and trajectory, map initial values, and the back end is responsible for optimizing all these data. However, if only the key frames at adjacent times are considered like the visual odometer, then the errors generated previously will inevitably accumulate to the next moment, causing cumulative errors in the entire SLAM, and the long-term estimation results will be unreliable.

[0057] S5. Mapping: Before block matching, it is a common preprocessing method to perform an image-to-image transformation. This is because we assume that the small image blocks remain unchanged when the camera moves, and this assumption can be maintained when the camera is translated (the example data set is basically such an example), but it is difficult to maintain when the camera rotates significantly. In particular, when the camera rotates around the optical center, a black-on-the-bottom and white-on-the-top image may become a black-on-the-top and white-on-the-bottom image block, causing the correlation to directly become negative (although it is still the same block). To prevent this from happening, it is usually necessary to take into account the motion between the reference frame and the current frame before block matching;

[0058] S6. Adjust attitude and position information: The drone uses the drone facade vision technology to estimate and adjust its own flight status and position information in real time;

[0059] S7. Work: When the facade work area is identified, the external working tools on the drone will automatically start working.

[0060] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing an environment model by a drone roaming on a vertical facade, characterized in that: The following steps are involved: S1. Obtaining images: obtaining real-time image data captured by the onboard binocular camera of the drone; S2. Build a front-end vertical elevation visual odometry: Expand the single-layer direct method to a pyramid-like multi-layer direct method, use the Coarse-to-fine process to calculate the relative motion, and then you can get the binocular camera pose derived by the multi-layer direct method. S3, back-end optimization: The back-end is mainly filtering and nonlinear optimization algorithms. The back-end optimization uses the sliding window BA method to estimate the state of the entire system from noisy data, as well as the maximum a posteriori probability estimate of the uncertainty of this state. The state includes both the trajectory of the drone itself and the constructed pose graph; S4, loop detection: The purpose is to eliminate drift. This task is accomplished by using the carried binocular vision sensor, that is, using the loop detection algorithm of the image appearance to complete the similarity loop detection. In the loop detection algorithm of the appearance, for image A and image B, it is necessary to calculate the similarity score between them: s(A,B)=||AB||. The similarity score will be within a certain range. When the similarity score is greater than a certain amount, a loop occurs. S5. Mapping: Dense mapping using binocular vision sensors is to convert image data into point clouds based on the estimated camera pose, and then stitch them together to obtain a point cloud map composed of discrete points. On this basis, the surface of the vertical facade is estimated, and triangular meshes and patches are used for mapping. In addition, obstacle information is drawn on the map and navigation is performed on the map, and an occupied grid map can be established through voxels. S6. Adjust attitude and position information: Use the facade vision technology to estimate and adjust the flight status and position information in real time; S7. Work: When the facade work area is identified, the external working tools on the drone will automatically start working.

2. The method for constructing an environment model by a drone roaming on a vertical surface according to claim 1, characterized in that: The binocular cameras image the same spatial coordinate point, and the pixel coordinates are recorded as p 1 ,p 2 The principle of the direct method is to find p based on the estimated value of the current camera's pose 2 But if the camera pose is not good enough, 2 The appearance and p 1 There will be a significant difference. In order to reduce this difference, the camera position is optimized to find the position that matches p 1 More similar p 2 .

3. The method for constructing an environment model by a drone roaming on a vertical surface according to claim 1, characterized in that: The BA method includes establishing a graph optimization model, constructing an overall Hessian matrix, and then marginalizing all landmark points to solve.

Images