Blade interior autonomous crawling sensing method and system based on robot

Abstract

The invention provides a robot-based blade interior autonomous crawling sensing method and system, and the method comprises the steps: carrying a multi-sensor array on a robot, the multi-sensor array comprises a binocular fisheye camera array and a ToF sensor array, controlling the robot to start from a path travel starting point, obtaining a panoramic depth image based on the multi-sensor array, and transmitting the panoramic depth image to the robot; recognizing an obstacle based on the panoramic depth image, constructing a three-dimensional point cloud map based on the obstacle, avoiding the obstacle based on the three-dimensional point cloud map, generating a feasible path in the blade, and enabling the robot to move based on the feasible path until the robot reaches a path travel terminal point or a minimum space in which the robot cannot pass. Autonomous crawling perception of the robot in the blade can be ensured, movement of the robot in the blade is ensured, and detection is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fan blade detection, and in particular to a blade internal autonomous crawling perception method and system based on a robot. BACKGROUND

[0002] The blades of a wind turbine (referred to as a fan) are prone to erosion by strong winds, rain, ultraviolet light and the like due to being in an outdoor environment for a long time, resulting in defects such as wear, cracks and corrosion, which can have a serious impact on the operation of the fan.

[0003] At present, personnel working at a high position enter the blade internal cavity from the hub of the unit to closely inspect the internal condition of the blade, but this detection method is highly dependent on the experience of the inspection personnel and is highly subjective, and due to the complex internal structure of the blade, it is prone to high missed detection rates and detection errors; in related technologies, a method of detecting the internal condition of the blade by a robot is proposed, but the prior art cannot flexibly control the robot to autonomously crawl and perceive in the internal blade, resulting in the robot being unable to move in the internal blade and making it difficult to implement detection work in the internal blade. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application aims to provide a robot-based internal blade autonomous crawling perception method and system, which autonomously crawls and perceives in the internal blade to ensure that the robot moves in the internal blade and implements detection.

[0005] According to a first aspect of the present application, a robot-based internal blade autonomous crawling perception method is provided, comprising: Step 1: mounting a multi-sensor array on a robot, the multi-sensor array comprising a binocular fisheye camera array and a ToF sensor array, and controlling the robot to start from a path starting point; Step 2: obtaining a panoramic depth image based on the multi-sensor array; Step 3: identifying obstacles based on the panoramic depth image; Step 4: constructing a three-dimensional point cloud map based on the obstacles; Step 5: generating a feasible path in the internal blade based on the three-dimensional point cloud map and avoiding the obstacles; Step 6: controlling the robot to avoid the obstacles in real time; Step 7: the robot moves based on the feasible path, and steps 2 to 6 are repeated until the robot reaches the path end point or the minimum space that cannot be passed through.

[0006] In some embodiments of the present application, based on the foregoing scheme, obtaining a panoramic depth image based on the multi-sensor array comprises: taking the coordinate system of the binocular fisheye camera array as a unified coordinate system; acquiring a ToF depth map based on the ToF sensor array, and acquiring a first depth value corresponding to each depth point based on the ToF depth map; acquiring an initial disparity map based on the binocular fisheye camera array, and acquiring a second depth value corresponding to each depth point based on the initial disparity map; for each depth point, performing confidence weighted fusion based on the first depth value and the second depth value to obtain a third depth value; aggregating the third depth values of all the depth points to obtain a panoramic depth image.

[0007] In some embodiments of the present application, based on the foregoing scheme, based on the panoramic depth image, an obstacle is identified, comprising: plane segmentation is performed on the panoramic depth image to extract all non-planar regions; for each non-planar region, a 3D edge detection is used to locate a convex structure; different kinds of obstacles are respectively pre-defined to correspond to 3D templates, a local point cloud of the convex structure is matched by using an ICP algorithm, each non-planar region is pre-defined as a target obstacle according to a matching score of the local point cloud of the convex structure and each 3D template, and the target obstacle is an obstacle corresponding to a 3D template with the highest score; an RGB image of each non-planar region is acquired based on a ToF sensor array, and HOG features or color histogram features are extracted based on the RGB image; whether each non-planar region is a false obstacle or a real obstacle is determined according to the HOG features or the color histogram features; if the non-planar region is a real obstacle, the non-planar region is output as the target obstacle, a position and a size of the non-planar region are output, and defect information of the non-planar region is identified, the defect information including a defect type and a defect position.

[0008] In some embodiments of the present application, based on the foregoing scheme, based on the obstacle, a three-dimensional point cloud map is constructed, comprising: the obstacle is labeled on the panoramic depth image, dynamic interference points are removed, and a sparse point cloud map is generated, wherein the method for removing the dynamic interference points is: a relative pose of the robot between adjacent frames of the third depth map is output by using an inertial measurement unit (IMU), and a search radius is used to reduce a epipolar search range from a full image to a predicted region; a re-projection error of a feature point that is successfully matched is acquired, and the feature point with the re-projection error greater than a preset pixel error is determined as a dynamic interference point and removed; based on the sparse point cloud map, dense depth estimation is performed to construct a three-dimensional point cloud map, wherein the method for the dense depth estimation is: Superpixel segmentation is performed on the sparse point cloud map, a quadratic surface is fitted for the point cloud in each superpixel, abnormal points are detected through the fitting residual, and the abnormal points are replaced with surface interpolation; A depth probability distribution is maintained for each superpixel, and each frame of sparse point cloud map is updated; An energy function is constructed based on the blade body, and the energy function is iteratively solved using the Gauss-Newton method, so that the depth of each superpixel conforms to the overall geometric constraint, and a three-dimensional point cloud map is obtained.

[0009] In some embodiments of the present application, based on the foregoing scheme, based on the three-dimensional point cloud map, a feasible path inside the blade is generated by avoiding obstacles, including: Based on the three-dimensional point cloud map, a voxel grid is converted, and based on the robot radius and the safety margin, the grid where the obstacle is located is dilated, and the area where the slope is greater than the slope threshold is marked as an impassable area; The cumulative cost from the starting point of the path to the current node and the heuristic function are set, and the weighted sum of the path cost and the heuristic function is the total cost; Avoiding obstacles, a first path with the minimum total cost is generated; Dubins curve interpolation is performed on the feasible path to ensure that the turning radius is greater than the turning radius threshold, and a second path is obtained; The grid slope through which the first path passes is detected, and if the grid slope is greater than the grid slope threshold, an alternative path is searched in the contour direction or a detour point is inserted to obtain discrete path points; The discrete path points are fitted into a multi-order B-spline curve; The multi-order B-spline curve is optimized according to the curvature and slope dynamic speed limit to generate a feasible path inside the blade.

[0010] In some embodiments of the present application, based on the foregoing scheme, the feasible path is optimized according to the motion constraint, including curvature smoothing optimization, slope constraint optimization, and dynamic obstacle avoidance optimization, wherein the method of curvature smoothing optimization is: Curve interpolation is performed on the feasible path, and if the turning radius is less than the turning radius threshold, the sharp turning line segment is replaced with a circular arc; The method of slope constraint optimization is: The grid slope through which the path passes is detected, and if the grid slope is greater than the grid slope threshold, an alternative path is searched in the contour direction or a detour point is inserted; Dynamic obstacle avoidance optimization uses D* Lite real-time update, and when a new obstacle is detected, the cost change is propagated from the current node in reverse, and the feasible path is dynamically corrected.

[0011] In some embodiments of the present application, based on the foregoing scheme, step 1 further comprises using a depth sensor to obtain a high-precision point cloud of the path travel starting point, extracting ISS key points or FPFH descriptors, and storing the high-precision point cloud and the feature descriptors into a local database of the robot. Further comprising: Step 8: If the depth sensor detects that the space in front is less than a set threshold, or the robot reaches a fault-tolerant range of the path travel end point coordinates, the robot is triggered to automatically return along the feasible path; When the robot returns, the surrounding environment point cloud is obtained in real time by the ToF sensor array, and the high-precision point cloud of the path return starting point and the real-time collected point cloud are matched using the ICP algorithm, and the return starting point is the travel end point. If the registration is unsuccessful, displacement error compensation is performed, the deviation of the ICP pose and the cumulative pose of the odometry is compared, if the translation error or the rotation error is greater than a set threshold, repositioning is triggered until the translation error or the rotation error is less than or equal to the set threshold, then the registration is successful, and the current pose is output.

[0012] According to a second aspect of the present application, a robot-based internal blade autonomous crawling perception device is provided, comprising: A mounting module for mounting a multi-sensor array on the robot, the multi-sensor array comprising a binocular fisheye camera array and a ToF sensor array, and controlling the robot to start from a path travel starting point; An acquisition module for acquiring panoramic depth images based on the multi-sensor array; An identification module for identifying obstacles based on the panoramic depth images; A construction module for constructing a three-dimensional point cloud map based on the obstacles; A generation module for generating a feasible path inside the blade based on the three-dimensional point cloud map and avoiding obstacles; A control module for controlling the robot to avoid obstacles in real time; A movement module for the robot to move based on the feasible path, and repeating the acquisition module to the generation module until the robot reaches the path end point or the minimum space that cannot be passed through.

[0013] According to a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the robot-based internal blade autonomous crawling perception method described above.

[0014] According to a fourth aspect of the present application, a non-transitory computer readable storage medium is provided, having a computer program stored thereon, and the computer program is executed by a processor to implement the robot-based internal blade autonomous crawling perception method described above.

[0015] Compared with the prior art, the present application has the following advantages and beneficial effects: (1) The method and system for autonomous crawling perception inside the blade based on the robot provided in the present application combines the rich visual information of the fisheye camera and the accurate depth perception of ToF, identifies the obstacles while maintaining a wide range of coverage, and then controls the robot to avoid the obstacles, so as to realize autonomous crawling perception inside the blade and ensure the movement of the robot inside the blade and realize detection.

[0016] (2) The method and system for autonomous crawling perception inside the blade based on the robot provided in the present application acquires a ToF depth map based on a ToF sensor array, acquires a first depth value corresponding to each depth point based on the ToF depth map, acquires an initial disparity map based on a binocular fisheye camera array, acquires a second depth value corresponding to each depth point based on the initial disparity map, performs confidence weighted fusion based on the first depth value and the second depth value to acquire a third depth value, and aggregates the third depth values of all depth points to acquire a panoramic depth image, so that the obtained panoramic depth image has high precision and is more conducive to the identification of obstacles.

[0017] (3) The method and system for autonomous crawling perception inside the blade based on the robot provided in the present application identify obstacles, improve detection efficiency and accuracy, and are more conducive to the perception of the robot to the inside of the blade. In addition, identifying obstacles is more conducive to identifying defects on the blade.

[0018] (4) The method and system for autonomous crawling perception inside the blade based on the robot provided in the present application identify obstacles, store the point cloud of the obstacles, form a three-dimensional point cloud map with semantic labels, and enhance path planning and obstacle avoidance capabilities. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are included to provide a further understanding of the present application and constitute a part of this application, illustrate certain illustrative embodiments of the present application and together with the general description of the present application given above and the detailed description of the present application given below, serve to explain the present application. In the drawings: Figure 1 A flowchart of a method for autonomous crawling perception inside a blade based on a robot according to the present embodiment; Figure 2 A schematic diagram of a system for autonomous crawling perception inside a blade based on a robot according to the present embodiment; Figure 3 A schematic diagram of an electronic device according to the present embodiment. DETAILED DESCRIPTION

[0020] In order to make the technical problems to be solved by the present application, technical solutions and beneficial technical effects more clearly understood, the following will be further described in detail in combination with the drawings and a plurality of exemplary embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the protection scope of the present application.

[0021] In addition, the described features, structures or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, a number of specific details are provided to give a sufficient understanding of the embodiments of the present application. However, one skilled in the art will realize that the technical solutions of the present application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be used. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid obscuring aspects of the present application.

[0022] The block diagram shown in the drawings is only a functional entity, which does not necessarily correspond to a physically independent entity. That is, these functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0023] The flowchart shown in the drawings is only an exemplary illustration, which does not necessarily include all contents and operations / steps, and is not necessarily executed in the order described. For example, some operations / steps can be further decomposed, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to the actual situation.

[0024] In the description of the present application, it should be understood that the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0025] Please refer to Figure 1 The flowchart shown in the drawings is a flowchart of a blade internal autonomous crawling perception method based on a robot of the present embodiment, and the present embodiment provides a blade internal autonomous crawling perception method based on a robot, which comprises: Step 1: mounting a multi-sensor array on the robot, the multi-sensor array comprising a binocular fisheye camera array and a ToF sensor array, and controlling the robot to start from the path starting point.

[0026] In some embodiments of the present embodiment, due to the complex internal structure of the leaf, the multi-sensor array includes a dual fisheye camera array and a ToF sensor array, which combines the rich visual information of the fisheye camera and the accurate depth perception of the ToF, identifies obstacles while maintaining a wide range of coverage, and realizes high-precision three-dimensional environment reconstruction, thereby facilitating the generation of a feasible path inside the leaf and the autonomous crawling and perception of the robot inside the leaf, ensuring the movement of the robot inside the leaf and achieving detection.

[0027] In some embodiments of the present embodiment, the dual fisheye camera array includes two sets of dual fisheye cameras installed back-to-back, and the ToF sensor array includes a plurality of ToF sensors that perceive the robot's surroundings in a ring shape. In the present embodiment, the ToF sensor array includes four ToF sensors arranged in a cross shape, which track pose changes in real time. The ToF sensors used in the present embodiment are Microsoft Azure Kinect and Intel RealSense, which can output ToF depth maps and RGB images.

[0028] In some embodiments of the present embodiment, an inertial measurement unit (IMU) is also included to assist in multi-frame data alignment.

[0029] In some embodiments of the present embodiment, FPGA is used to trigger all sensors to synchronize data collection, NTP / PTP protocol is used to align timestamps, and IMU data is used as a motion compensation reference to achieve software synchronization. A unified coordinate system for all sensors is established based on the dual fisheye camera as the main coordinate system.

[0030] In some embodiments of the present embodiment, the robot is a drone, and the two sets of dual fisheye cameras installed back-to-back are installed in the middle of the drone's fuselage, and the four ToF sensors arranged in a cross shape are installed at the four corners of the drone's fuselage.

[0031] Step 2: Obtain a panoramic depth image based on the multi-sensor array.

[0032] In some embodiments of the present embodiment, due to the relatively complex internal structure of the leaf, in order to perform three-dimensional modeling, it is necessary to collect 360° panoramic depth images centered on the robot at a fixed frequency. Based on the panoramic depth image, obstacles can be identified. The panoramic depth image is obtained based on the multi-sensor array, including: Step 201: Take the coordinate system of the dual fisheye camera array as the unified coordinate system.

[0033] Step 202: Obtain a ToF depth map based on the ToF sensor array, and obtain a first depth value corresponding to each depth point based on the ToF depth map.

[0034] In some embodiments of the present embodiment, the ToF points are projected onto the fisheye camera coordinate system for ToF depth map alignment, and the formula is as follows:

[0035] wherein, represents the coordinates of the ToF point cloud in the camera coordinate system; represents the rotation matrix from the ToF coordinate system to the camera coordinate system; represents the original point cloud coordinates measured by the ToF sensor; represents the translation vector from the ToF coordinate system to the camera coordinate system.

[0036] Step 203: Obtain an initial disparity map based on the binocular fisheye camera array, and obtain a second depth value corresponding to each depth point based on the initial disparity map.

[0037] In some embodiments of the present embodiment, since the binocular fisheye camera array adopts two sets of binocular fisheye cameras installed back-to-back, it is necessary to perform HDR fusion on fisheye images of different exposure degrees, and combine distortion correction to ensure the alignment accuracy of fisheye images of different exposure degrees and avoid synthetic artifacts.

[0038] In some embodiments of the present embodiment, an SMG (Semi-Global Matching) algorithm is used to generate an initial disparity map of the two sets of binocular fisheye cameras installed back-to-back, and then a second depth value corresponding to each depth point is obtained through depth calculation based on the initial disparity map and polar geometry.

[0039] Step 204: For each depth point, confidence weighted fusion is performed based on the first depth value and the second depth value to obtain a third depth value.

[0040] In some embodiments of the present embodiment, in order to ensure the accuracy of the finally obtained depth value, it is necessary to perform confidence weighted fusion based on the first depth value and the second depth value to obtain a third depth value of each depth point. The formula of the confidence weighted fusion is as follows:

[0041] wherein, represents the fused depth value; represents the depth value estimated by the binocular camera; represents the depth value measured by the ToF sensor; represents the binocular depth confidence weight (such as based on SSIM or matching error), represents the ToF depth confidence weight (such as based on ranging noise variance), and the weight needs to be dynamically adjusted in combination with the confidence index.

[0042] Step 205: Aggregate the third depth values of all depth points to obtain a panoramic depth image.

[0043] The collected panoramic depth image is preprocessed, including denoising, distortion correction, image enhancement, etc., to improve the quality of the panoramic depth image.

[0044] Step 3: Based on the panoramic depth image, the obstacle is identified.

[0045] In some embodiments of the present embodiment, the obstacles that need to be identified for the robot movement in the blade are mainly the following types: obstacles such as main beams, webs, stiffeners, bolts, supports, internal support structures, etc. By identifying the obstacles, the inside of the blade is perceived, the detailed information of the inside of the blade is obtained, and then a three-dimensional point cloud map is constructed, and a robot feasible path feature is generated. The robot feasible path feature is the channel between the main beam and the web and the unsealed internal space of the shell, and has a certain safety distance from the obstacle.

[0046] In some embodiments of the present embodiment, the obstacles are distinguished based on the panoramic depth image.

[0047] In some embodiments of the present embodiment, RANSAC (Random Aample Consensus) or region growing method is used for plane segmentation of the panoramic depth image, and all non-planar regions and all planar regions are extracted. The present embodiment does not limit this.

[0048] In some embodiments of the present embodiment, based on the panoramic depth image, the obstacle is identified, including: Step 301: For each non-planar region, a 3D edge detection is used to locate the convex structure.

[0049] In some embodiments of the present embodiment, the 3D edge is a discontinuity detection method based on normal vector, and the present embodiment does not limit this.

[0050] Step 302: Predefine different kinds of obstacles corresponding to 3D templates respectively, match the local point cloud of the convex structure through the ICP (Iterative Closest Point) algorithm, and according to the matching score of the local point cloud of the convex structure with each 3D template, each non-planar region is preset as a target obstacle. The target obstacle is the obstacle corresponding to the 3D template with the highest score.

[0051] In some embodiments of the present embodiment, the 3D template is a CAD model or a point cloud library.

[0052] Step 303: Obtain the RGB image of each non-planar region based on the ToF sensor array, and extract the HOG feature or color histogram feature based on the RGB image.

[0053] Step 304: Determine whether each non-planar region is a false obstacle or a real obstacle according to the HOG feature or color histogram feature.

[0054] Step 305: If the non-planar region is a real obstacle, output the non-planar region as a target obstacle, output the position and size of the non-planar region, and identify the defect information of the non-planar region, the defect information including defect type and defect position, and the defect type usually including wear, crack, corrosion, etc.

[0055] In this way, by identifying the obstacle, it is convenient for subsequent construction of a three-dimensional point cloud map.

[0056] Step 4: Based on the obstacle, a three-dimensional point cloud map is constructed.

[0057] In some embodiments of the present embodiment, the method for constructing a three-dimensional point cloud map based on the obstacle specifically comprises: Step 401: Label the obstacle on the panoramic depth image to generate a sparse point cloud map; In some embodiments of the present embodiment, the sparse point cloud map is optimized to eliminate dynamic interference points. Generating the sparse point cloud map further comprises: Using an inertial measurement unit (IMU) to output the relative pose of the robot between adjacent frames of the third depth map Within the search radius, the epipolar search range is reduced from the full image to the predicted area, and the formula of the search radius is as follows:

[0058] wherein, is the velocity estimate, is the IMU noise variance, and the coefficient According to the performance of the inertial measurement unit (IMU), the motion state and the environmental disturbance, the optimization is performed.

[0059] For the matched feature points, the re-projection error of the feature points is calculated:

[0060] wherein, r represents the re-projection error of the feature points; is the 2D pixel coordinate of the feature point of the previous frame; is the 2D pixel coordinate of the feature point of the current frame; is the camera projection function; is the adjacent frame pose transformation matrix estimated by the IMU; is the 3D world coordinate of the feature point of the previous frame.

[0061] If r is greater than 3 , is the pixel error standard deviation, and is determined as a dynamic interference point and is eliminated.

[0062] In some embodiments of the present embodiment, the obstacles are labeled on the panoramic depth image to generate a sparse point cloud map, and the sparse point cloud map is also based on an inertial measurement unit (IMU) to obtain a pose graph SLAM, and constraints about adjacent frame ICP matching residuals, IMU pre-integration residuals, and cylindrical prior residuals are added to the pose graph SLAM.

[0063] Specifically, an optimization objective function is established:

[0064] wherein, represents a pose to be optimized; represents an ICP matching residual; represents an IMU pre-integration residual; represents a cylindrical prior constraint residual. The optimization is performed using a g2o or Ceres solver.

[0065] In this way, the environment perception accuracy of the sparse point cloud map can be significantly improved.

[0066] In some embodiments of the present embodiment, a surface normal vector and a curvature around a feature point can also be extracted to construct a geometry-appearance joint descriptor , and the calculation formula is:

[0067] wherein, represents a SIFT feature descriptor of the feature point ; is a local surface normal vector; is a local curvature value.

[0068] In some embodiments of the present embodiment, a KD-Tree is used to accelerate the nearest neighbor search, and feature points with similar geometric properties are preferentially matched.

[0069] In some embodiments of the present embodiment, since the internal stiffeners of the blade are usually periodically distributed in a grid, a virtual template feature can be constructed to realize stiffener prior utilization. Specifically, the intersection points of the stiffeners are manually labeled in the initial frame, and the grid nodes are tracked in the subsequent frames through graph matching. A grid model is fitted using RANSAC to complete the feature points in the occluded area.

[0070] Step 402: based on the sparse point cloud map, dense depth estimation is performed to construct a three-dimensional point cloud map, wherein the method of dense depth estimation is: The sparse point cloud map is subjected to superpixel segmentation, the point cloud in each superpixel is fitted with a quadratic surface, abnormal points are detected through fitting residuals, and the abnormal points are replaced with interpolated values of the surface to avoid the influence of the abnormal points on the three-dimensional point cloud map.

[0071] Specifically, the method for fitting a quadratic surface to the point cloud within each superpixel is as follows: Using SLIC superpixel segmentation, fit a quadratic surface to the point cloud within each superpixel:

[0072] in, , , , , , , , , , Represents the coefficients of the quadratic surface; Represents the coordinates of the point cloud.

[0073] Maintain a depth probability distribution for each superpixel Update the sparse point cloud map for each frame using the following formula: ,

[0074] in, Indicates the first The average depth of the frame; Indicates the first Frame depth standard deviation Indicates the first Depth measurement values ​​of frames, if multiple consecutive frames >3 This is determined to be dynamic interference; therefore, the update of this pixel is frozen. The standard deviation of the measurement noise in the (k+1)th frame is calibrated by the sensor. This represents the updated depth standard deviation.

[0075] An energy function is constructed based on the blade body, and the Gauss-Newton method is used to iteratively solve the energy function so that the depth of each superpixel conforms to the overall geometric constraints, thus obtaining a 3D point cloud map.

[0076] Specifically, the blade body is set to a cylindrical shape, and an energy function is constructed as follows:

[0077] in, Represents the optimized pixels Depth value; Represents the optimized pixels Depth value; This represents the depth value measured by the sensor at the i-th pixel. Where is the radius of the cylinder. For pixels The angle difference between them For adjacent pixel edge sets, This represents the regularization weight, used to balance the importance of data terms (sensor measurements) and geometric constraint terms (cylindrical priors).

[0078] Thus, by constructing an energy function based on the main body of the blade and using the Gauss-Newton method to iteratively solve the energy function, the depth value within the superpixel can be forced to conform to the actual shape of the blade, avoiding local distortion caused by independent optimization. Furthermore, the Gauss-Newton method for iteratively solving the energy function efficiently converges to the optimal solution, making the point cloud more closely resemble the three-dimensional structure of the real blade.

[0079] Step 5: Based on the 3D point cloud map, avoid obstacles and generate feasible paths inside the blade.

[0080] In some implementations of this embodiment, based on a 3D point cloud map, an improved A* algorithm is used to generate feasible paths inside the blade, specifically including: According to robot radius and safety margin Expand the obstacle grid by a size of [value missing]. Mark the slope > The area is impassable; and dynamic obstacles detected in real time are avoided; Each node n Represented as ( x , y , ),in For the robot's orientation (discretized into 8 directions or continuous values), consider the robot's nonholonomic constraints (such as differential drive models), allowing only changes in adjacent angles. ; The cumulative cost from the origin to the current node is used as the path cost. ,include:

[0081] in, Indicates the distance traveled (Euclidean or Manhattan distance); Indicates the absolute value of altitude change, penalizing steep slopes; This represents the risk factor of approaching an obstacle (inversely proportional to distance); the weighting coefficient needs to be optimized based on the robot's capabilities, environmental risks, and mission objectives. Weights representing the distance traveled. Weights representing the absolute value of height changes The weight representing the risk factor of approaching an obstacle; Heuristic functions :

[0082] in, Angle alignment weight (usually 0.1-0.3). (Guarantee the optimality of A*).

[0083] Starting point of travel Add to Open List settings Perform a loop search and select from the open list. Minimum node ;like For the target point, backtrack to generate a path; otherwise, generate all feasible child nodes (that satisfy motion constraints and are not occupied by obstacles); for each child node... Calculate temporary costs ,like Not on the open list or Update the cost and record the parent node; continue until the target point is found or the open list is empty.

[0084] In some implementations of this embodiment, feasible paths are optimized based on motion constraints, including curvature smoothing optimization, slope constraint optimization, and dynamic obstacle avoidance optimization.

[0085] Among them, curvature smoothing optimization is achieved through Dubins path correction: feasible paths are interpolated using Dubins curves to ensure the turning radius. Calculate the minimum turning radius for each path segment. Replace the sharp turn segment with an arc, the calculation formula is:

[0086] in, Indicates the radius of the arc. Indicates the turning radius threshold. Indicates the length of the line segment in a sharp turn. Indicates the angle of a sharp turn segment.

[0087] Slope constraint optimization employs height profile adjustment to adjust the grid slope along the path. If the test is performed, , If the slope threshold is set, then an alternative path is searched along the contour line direction, or a detour point is inserted.

[0088] The dynamic obstacle avoidance optimization uses D* Lite for real-time updates. When a new obstacle is detected, the cost change is propagated back from the current node to dynamically correct the feasible path.

[0089] In this embodiment, post-processing and verification of feasible paths are also included, fitting discrete path points into 3rd-order B-spline curves to ensure continuity:

[0090] in, As control points, For sample B, the basis function is used.

[0091] Discretely sample the smoothed feasible path and check whether each point overlaps with the expanded obstacle grid.

[0092] According to curvature R and slope Dynamic speed limit:

[0093] in, This indicates the maximum speed designed for the robot; This indicates the coefficient of friction between the tire and the ground. g Represents gravitational acceleration; Indicates the turning radius (determined by the path curvature); This indicates the motor's continuous output power; m Indicates the robot's mass; Indicates the slope angle.

[0094] Step 6: Control the robot to avoid obstacles in real time.

[0095] Step 7: The robot moves based on feasible paths, repeating steps 2 to 6 until the robot reaches the end of the path or the smallest impassable space.

[0096] In some embodiments of this example, step 1 further includes using a depth sensor to obtain a high-precision point cloud of the starting point of the path, extracting ISS key points or FPFH descriptors, and storing the high-precision point cloud and feature descriptors into the robot's local database. Also includes: Step 8: If the depth sensor detects that the space in front is less than the set threshold, or the robot reaches the fault tolerance range of the end point coordinates of the path, then the robot is triggered to automatically return along the feasible path. When the robot returns, it acquires the surrounding environment point cloud in real time through a ToF sensor array, and uses the ICP algorithm to match the high-precision point cloud at the path return starting point with the real-time acquired point cloud. If registration fails, displacement error compensation is performed. The deviation between the ICP pose and the odometer cumulative pose is compared. If the translation error or rotation error is greater than the set threshold, repositioning is triggered. Registration is successful when the translation error or rotation error is less than or equal to the set threshold, and the current pose is output.

[0097] In some implementations of this embodiment, fault tolerance mechanisms are also included: offline storage of path key points to ensure that the robot can still return when the network is disconnected; switching to visual SLAM for detection when the ToF sensor array fails; triggering visual relocalization when the IMU drift is too large; and issuing a human assistance signal after the robot has attempted to detour multiple times.

[0098] Robot autonomous return path planning: Following the pose sequence of the stored travel path, the process is executed in reverse. During the return process, the D* Lite dynamic programming algorithm is used to update the return path in real time for dynamic obstacles.

[0099] In some embodiments of this example, the robot is equipped with a depth sensor and an IMU odometry. The depth sensor detects the distance in front of the robot to determine whether the minimum spatial conditions for triggering a return-to-home call are met. The IMU odometry verifies whether the robot has reached the target point by comparing the coordinate difference between the current pose and the target point pose to see if it is within the tolerance range.

[0100] In some implementations of this embodiment, key point extraction is also included: key points, such as inflection points, are extracted from the feasible path of the return path. This can reduce the storage volume because it is not necessary to store the detailed information of the entire path, but only to retain key inflection points and other information.

[0101] In some implementations of this embodiment, smoothing processing is also included: connecting key points using B-spline curves to ensure the path conforms to the minimum turning radius. Constraints. B-spline curves can generate smooth paths, and their shape can be adjusted using control points to satisfy the robot's kinematic constraints, such as curvature. .

[0102] In some implementations of this embodiment, real-time obstacle avoidance is also included, which fuses ToF (near field) and binocular (far field) data to update the obstacle map. The ToF sensor is mainly used for near-range obstacle detection, while the binocular vision sensor can acquire depth information at a greater distance. Combining the two can provide a more comprehensive perception of obstacles in the surrounding environment.

[0103] In some embodiments of this example, the robot's speed is also dynamically adjusted, and the calculation formula is as follows: in, The current path curvature formula, The maximum speed of the robot, The coefficient of friction, The acceleration is due to gravity. This ensures the robot maintains a suitable speed during its return journey, allowing for both rapid return and safety.

[0104] In some embodiments of this example, the robot's control strategy employs a differential drive model: the left and right wheel speeds are calculated based on the path curvature, using the following formula:

[0105] in, This refers to the wheel track.

[0106] In this way, the robot can adjust the speed of its left and right wheels according to the curvature of the path, achieving precise steering.

[0107] In some implementations of this embodiment, the return-to-home termination condition includes setting return-to-home start point identification and termination logic, including: Step 801: Offline phase return-to-home starting point template construction, using a depth camera to scan at the return-to-home starting point location to obtain high-precision point cloud Q; Step 802: Feature extraction: Extract ISS key points or FPFH descriptors to enhance matching robustness; store point cloud Q and feature descriptors in the robot's local database.

[0108] Step 803: Online phase return matching, real-time data acquisition, and acquisition of current environmental point cloud P through ToF sensor; Step 804: ICP Registration: Use KD-Tree to accelerate the nearest point search and solve for the transformation matrix. The successful conditions are that the registration error (RMSE) is less than the ICP matching error threshold and the overlap rate is greater than the set threshold. Step 805: Pose Output: If registration is successful, output the current pose. ; Step 806: Displacement error compensation: Error detection: Compare the deviation between the ICP pose and the accumulated pose of the odometry. ; Threshold trigger: If translation error Greater than the translation error threshold or rotation error Rotation error threshold triggers repositioning; Error compensation: If translation error If the error exceeds the translation error threshold, the odometer drift is corrected using EKF filtering. Step 807: Failure Handling Mechanism ICP registration failed: Enable dead reckoning based on IMU and wheel velocity gauges, and continue searching for the return point; initiate visual relocation.

[0109] Multiple matching failures: Issue an alert and request manual intervention.

[0110] Example 2 Please seeFigure 1 As shown, Embodiment 2 provides a robot-based autonomous crawling sensing device for the interior of a blade, comprising: The module is used to mount a multi-sensor array on the robot, which includes a binocular fisheye camera array and a ToF sensor array, to control the robot to start from the starting point of the path. The acquisition module is used to acquire panoramic depth images based on a multi-sensor array; The recognition module is used to identify obstacles based on panoramic depth images; The building module is used to construct 3D point cloud maps based on obstacles; The generation module is used to generate feasible paths inside the blades based on a 3D point cloud map, avoiding obstacles. The control module is used to control the robot to avoid obstacles in real time; The movement module is used for the robot to move based on a feasible path. The acquisition module is repeatedly transferred to the generation module until the robot reaches the end of the path or the smallest space where it cannot pass.

[0111] Specifically, this embodiment corresponds one-to-one with the above method embodiments. The functions of each module have been described in detail in the corresponding method embodiments, so they will not be repeated here.

[0112] Example 3 Example 3 provides an electronic device, such as Figure 3 As shown, it includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The feature is that when the processor executes the program, it implements the above-mentioned robot-based autonomous crawling perception method inside the blade.

[0113] Example 4 Example 4 provides a non-transitory computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the robot-based autonomous crawling perception method inside the blade as described above.

[0114] The memory in this embodiment of the invention is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0115] The robot-based autonomous crawling perception method for the interior of blades disclosed in this invention can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the robot-based autonomous crawling perception method for the interior of blades can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in this invention. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly represented as execution by a hardware decoding processor, or as execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, specifically a memory. The processor reads information from the memory and, in conjunction with its hardware, completes the steps of the robot-based autonomous crawling perception method for the interior of blades provided in this invention.

[0116] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned methods.

[0117] It is understood that memory can be volatile or non-volatile, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Sync Link Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0118] The above embodiments are merely illustrative examples of the technical solutions of the present invention. The methods involved in the present invention are not limited to those described in the above embodiments, but are defined by the scope of the claims. Any modifications, additions, or equivalent substitutions made by those skilled in the art based on these embodiments are within the scope of protection claimed by the claims of the present invention.

Claims

1. A robot-based autonomous crawling perception method for the interior of a blade, characterized in that, include: Step 1: Equip the robot with a multi-sensor array, which includes a binocular fisheye camera array and a ToF sensor array, and control the robot to start from the starting point of the path; Step 2: Acquire panoramic depth images based on the multi-sensor array; Step 3: Identify obstacles based on the panoramic depth image; Step 4: Construct a 3D point cloud map based on the obstacles; Step 5: Based on the 3D point cloud map, avoid the obstacles and generate a feasible path inside the blade; Step 6: Control the robot to avoid obstacles in real time; Step 7: The robot moves based on the feasible path, repeating steps 2 to 6 until the robot reaches the end of the path or the smallest impassable space.

2. The method according to claim 1, characterized in that, Panoramic depth images are acquired using a multi-sensor array, including: The coordinate system of the binocular fisheye camera array is used as a unified coordinate system; A ToF depth map is obtained based on the ToF sensor array, and a first depth value corresponding to each depth point is obtained based on the ToF depth map; An initial disparity map is obtained based on the binocular fisheye camera array, and a second depth value corresponding to each depth point is obtained based on the initial disparity map. For each depth point, a third depth value is obtained by performing a confidence-weighted fusion based on the first depth value and the second depth value; The third depth values ​​of all depth points are aggregated to obtain the panoramic depth image.

3. The method according to claim 1, characterized in that, Based on panoramic depth images, obstacles are identified, including: Perform planar segmentation on the panoramic depth image and extract all non-planar regions; For each of the aforementioned non-planar regions, 3D edge detection is used to locate the protruding structure; Different types of obstacles are predefined into 3D templates. The local point cloud of the protruding structure is matched with the ICP algorithm. Based on the matching score between the local point cloud of the protruding structure and each 3D template, each non-planar region is preset as a target obstacle. The target obstacle is the obstacle corresponding to the 3D template with the highest score. Based on the ToF sensor array, RGB images of each non-planar region are acquired, and HOG features or color histogram features are extracted from the RGB images. Each non-planar region is determined to be a false obstacle or a real obstacle based on HOG features or color histogram features. If the non-planar region is a real obstacle, output that the non-planar region is a target obstacle, and output the position and size of the non-planar region, and identify the defect information of the non-planar region, including the defect type and defect location.

4. The method according to claim 1, characterized in that, Based on the obstacles, a 3D point cloud map is constructed, including: Obstacles are marked on the panoramic depth image, dynamic interference points are removed, and a sparse point cloud map is generated. The method for removing dynamic interference points is as follows: The relative pose of the robot between adjacent frames of the third depth map is output by the inertial measurement unit (IMU), and the epipolar search range is reduced from the entire map to the predicted area based on the search radius. Obtain the reprojection error of successfully matched feature points, and identify feature points with reprojection errors greater than the preset pixel error as dynamic interference points and remove them. Based on sparse point cloud maps, dense depth estimation is performed to construct 3D point cloud maps. The method for dense depth estimation is as follows: Superpixel segmentation is performed on the sparse point cloud map. A quadratic surface is fitted to the point cloud within each superpixel. Outliers are detected by the fitting residual and replaced with surface interpolation. Maintain a depth probability distribution for each superpixel and update the sparse point cloud map for each frame; An energy function is constructed based on the blade body, and the Gauss-Newton method is used to iteratively solve the energy function so that the depth of each superpixel conforms to the overall geometric constraints, thus obtaining a 3D point cloud map.

5. The method according to claim 1, characterized in that, Based on a 3D point cloud map, a feasible path is generated inside the blade to avoid the obstacles, including: Based on the three-dimensional point cloud map converted into a voxel mesh, the grid containing the obstacle is expanded according to the robot radius and safety margin, and areas with a slope greater than the slope threshold are marked as impassable areas. Define the cumulative cost from the starting point of the path to the current node and a heuristic function, and calculate the total cost by weighting the path cost and the heuristic function. Avoid the obstacles and generate the first path with the minimum total cost; Perform Dubins curve interpolation on the feasible path to ensure that the turning radius is greater than the turning radius threshold, and obtain the second path; The grid slope along the first path is detected. If the grid slope is greater than the grid slope threshold, an alternative path is searched along the contour line or a detour point is inserted to obtain discrete path points. Fit discrete path points into multi-order B-spline curves; The multi-order B-spline curve is optimized by dynamically limiting the speed based on curvature and slope to generate feasible paths inside the blade.

6. The method according to claim 5, characterized in that: Feasible path optimization based on motion constraints includes curvature smoothing optimization, slope constraint optimization, and dynamic obstacle avoidance optimization. The curvature smoothing optimization method is as follows: Perform curve interpolation on feasible paths; if the turning radius is less than the turning radius threshold, replace sharp turn segments with circular arcs. The method for slope constraint optimization is as follows: The grid slope along the path is detected. If the grid slope is greater than the grid slope threshold, an alternative path is searched along the contour line direction, or a detour point is inserted. The dynamic obstacle avoidance optimization uses D* Lite for real-time updates. When a new obstacle is detected, the cost change is propagated back from the current node to dynamically correct the feasible path.

7. The method according to claim 1, characterized in that: Step 1 further includes using a depth sensor to acquire a high-precision point cloud of the starting point of the path, extracting ISS key points or FPFH descriptors, and storing the high-precision point cloud and feature descriptors into the robot's local database. Also includes: Step 8: If the depth sensor detects that the space ahead is less than the set threshold, or if the robot reaches the tolerance range of the end point coordinates of the path, then the robot is triggered to automatically return along the feasible path; When the robot returns, it acquires the surrounding environmental point cloud in real time through a ToF sensor array. The ICP algorithm is used to match the high-precision point cloud of the return path starting point with the real-time acquired point cloud. The return starting point is the destination. If registration fails, displacement error compensation is performed. The deviation between the ICP pose and the odometer cumulative pose is compared. If the translation error or rotation error is greater than the set threshold, repositioning is triggered. Registration is successful when the translation error or rotation error is less than or equal to the set threshold, and the current pose is output.

8. A robot-based autonomous crawling sensing device for the interior of a blade, characterized in that, include: The module is used to mount a multi-sensor array on the robot, the multi-sensor array including a binocular fisheye camera array and a ToF sensor array, to control the robot to start from the starting point of the path; The acquisition module is used to acquire panoramic depth images based on the multi-sensor array; The recognition module is used to identify obstacles based on the panoramic depth image; A construction module is used to build a 3D point cloud map based on the obstacles; The generation module is used to generate a feasible path inside the blade based on the three-dimensional point cloud map, avoiding the obstacles. The control module is used to control the robot to avoid obstacles in real time; The movement module is used for the robot to move based on the feasible path, and to repeatedly obtain information from the acquisition module to the generation module until the robot reaches the end of the path or the smallest space where it cannot pass.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the robot-based autonomous crawling perception method for the interior of blades as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot-based autonomous crawling perception method for the interior of blades as described in any one of claims 1 to 7.

Images