Unmanned aerial vehicle-mechanical arm cooperative dynamic reconstruction and grabbing method and system

Abstract

The invention relates to the technical field of cooperative control of unmanned aerial vehicles and mechanical arms. The invention provides an unmanned aerial vehicle-mechanical arm collaborative dynamic reconstruction and grabbing method and system. The method comprises the steps that depth images are obtained in real time and sent to a target detection module, and a global voxel map is constructed through incremental fusion; calculating each unoccupied voxel in the global voxel map to obtain a comprehensive uncertainty value, and constructing a shielding uncertainty field; selecting a next-generation optimal observation viewpoint, and controlling the unmanned aerial vehicle to move towards the next-generation optimal observation viewpoint; the global voxel map is updated in real time in the process that the unmanned aerial vehicle flies to the next-generation optimal observation viewpoint, the online incremental grabbing quality evaluation and trajectory re-planning process is executed in parallel, and the optimal grabbing strategy is determined; and when the optimal strategy condition is met, the unmanned aerial vehicle is controlled to hover, the mechanical arm is driven to execute the finally planned grabbing track, and grabbing of the target object is completed.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of collaborative control technology of unmanned aerial vehicles (UAVs) and robotic arms, and particularly to a method and system for collaborative dynamic reconstruction and grasping of UAVs and robotic arms. Background Technology

[0002] The collaborative system of drones and robotic arms integrates the wide operating range of drones with the precision operation capabilities of robotic arms, providing a new technological approach for performing autonomous grasping tasks in complex environments. Such systems have broad application prospects in disaster relief, high-altitude equipment maintenance, warehousing and logistics, and other fields.

[0003] Existing drone-robotic arm collaborative operation technologies still have significant limitations. Most systems rely on pre-built complete environmental maps for planning, and this approach, based on static environmental assumptions, struggles to adapt to unknown or dynamically changing unstructured scenarios. When unknown obstacles exist in the environment or the target object shifts, the system's reliability and success rate decrease significantly. Traditional methods typically employ a sequential workflow of "reconstruct first, then plan." The 3D reconstruction process and grasping planning are independent, potentially failing to collect crucial information for grasping decisions during the reconstruction phase, such as the object's specific gripping surface. Only after reconstruction is complete may it be discovered that a feasible grasping plan cannot be generated. This architecture results in a waste of time and resources. At the perception level, existing view planning methods primarily aim to improve the overall model integrity, lacking task orientation. For grasping tasks, occluded areas often contain key grasping features, but existing systems lack proactive perception and exploration mechanisms for such task-related uncertainties.

[0004] The aforementioned problems collectively result in existing systems exhibiting poor adaptability, delayed decision-making, and low operational efficiency when dealing with real-world complex environments, severely hindering their widespread application in practical scenarios. Summary of the Invention

[0005] To address the above issues, this invention employs an online incremental grasping quality assessment and trajectory replanning mechanism. This allows the system to continuously and in parallel evaluate candidate grasping poses in real time, even when the 3D environmental model is incomplete. This enables the formation of the optimal grasping strategy to proceed synchronously with the construction of the environmental model, without waiting for the final perfection of the global model. This significantly reduces the total task time from target discovery to grasping execution.

[0006] According to embodiments of the present invention, a method and system for collaborative dynamic reconstruction and grasping of unmanned aerial vehicles (UAVs) and robotic arms are provided.

[0007] In a first aspect of the invention, a method for collaborative dynamic reconstruction and grasping using a drone and a robotic arm is provided. The method includes: Step S01: Scan the work scene using a depth camera mounted on the drone, acquire depth images in real time, send them to the target detection module, and incrementally fuse them to build a global voxel map; Step S02: For each unoccupied voxel in the global voxel map, calculate the comprehensive uncertainty value by multiplying its corresponding occlusion uncertainty metric and task relevance weight, and construct an occlusion uncertainty field; Step S03: Based on the occlusion uncertainty field, sample candidate viewpoints in the flight space of the UAV, use the evaluation function to calculate the total entropy of the uncertainty field of each candidate viewpoint, select the candidate viewpoint with the minimum total entropy as the next generation best observation viewpoint, and control the UAV to move towards the next generation best observation viewpoint; Step S04: While the UAV is flying toward the next-generation best observation point, update the global voxel map in real time, and execute the online incremental grasping quality assessment and trajectory replanning process in parallel to determine the optimal grasping strategy; Step S05: When the conditions of the optimal strategy are met, control the drone to hover and drive the robotic arm to execute the final planned grasping trajectory to complete the grasping of the target object.

[0008] Furthermore, the occlusion uncertainty measure mentioned in step S02 is determined by calculating the degree of obstruction by occupied voxels on the line of sight from the current camera optical center to the unoccupied voxel. Specifically, for each unoccupied voxel in the map, the line of sight from the current camera optical center to the voxel is simulated. If the line of sight passes through a voxel marked as occupied before reaching the target voxel, the target voxel is considered to be in an occluded state, and its uncertainty value increases according to the degree of occlusion. Unoccupied visible area voxels are assigned a lower uncertainty value.

[0009] Furthermore, the task relevance weight mentioned in step S02 is based on the identified potential grasping target. The unoccupied voxels around the grasping target are assigned a weight coefficient higher than that of the scene background voxels. The specific steps are as follows: the system runs a lightweight target detection algorithm in parallel to initially identify potential grasping targets; and assigns a weight coefficient higher than that of the scene background voxels to the unoccupied voxels around the potential grasping target object, especially the area where it may be occluded.

[0010] Furthermore, the evaluation function described in step S03 is the information gain function.

[0011] Furthermore, the specific steps of step S04 are as follows: Step S041: For the target object that has been identified but whose model is not yet complete, sample several six-degree-of-freedom candidate grasping poses in real time and continuously within its currently reconstructed surface and adjacent space; Step S042: For each candidate grasping pose, a multi-dimensional weighted fusion calculation is performed through an online evaluation module to obtain its comprehensive grasping quality score; Step S043: Based on the comprehensive grasping quality score of all candidate grasping poses, filter and update the global optimal grasping strategy at the current moment in real time.

[0012] Furthermore, the evaluation dimensions include, but are not limited to: Geometric stability score: Based on the local point cloud data of the target object in the current voxel map, analyze the distribution of contact points between the gripper and the object under the gripping pose, calculate the classic gripping quality index, and evaluate the physical stability of the gripping. Accessibility rating: Combining the current hovering pose of the drone, the kinematic model and joint constraints of the robotic arm, and the real-time updated global voxel map, preliminary, lightweight motion planning and collision detection are performed to assess the possibility and difficulty of the robotic arm reaching the grasping pose without collision. Model integrity confidence: The reliability of the grasping decision is calculated based on the average uncertainty of the corresponding voxel region in the occlusion uncertainty field on which the grasping pose depends. If the grasping point is located in a high uncertainty region, i.e., the region may not have been observed or is still severely occluded, the model integrity confidence of the grasping pose will be lowered.

[0013] Furthermore, the optimal strategy conditions described in step S05 include: Quality compliance conditions: The overall grasping quality score of the current global optimal grasping strategy exceeds the first preset threshold, and the drone is in a stable hovering state; Optimal efficiency condition: The system predicts that the improvement in quality score of the globally optimal crawling strategy due to continuing to execute viewpoint exploration is less than the second preset threshold.

[0014] When one of the conditions is met, it is considered that the optimal strategy has been achieved.

[0015] In a second aspect, the present invention provides a system for collaborative dynamic reconstruction and grasping of a drone and a robotic arm. The system includes: a drone platform, a multi-degree-of-freedom robotic arm, an onboard computing unit, a depth camera, and a flight control system.

[0016] Furthermore, the onboard computing unit is configured to: Run the SLAM and 3D reconstruction algorithm to perform step S01; Construct an occlusion uncertainty field and perform viewpoint planning to execute steps S02 and S03; Run the online incremental crawling quality assessment and trajectory replanning algorithm to perform step S04; Coordinate the control of the drone and the robotic arm to ultimately execute step S05.

[0017] Furthermore, the airborne computing unit integrates a target detection module, which is used to identify potential capture targets in the global voxel map in real time and provide input for the task relevance weight calculation in step S02.

[0018] This invention employs an online incremental grasping quality assessment and trajectory replanning mechanism, which allows the system to evaluate candidate grasping poses in parallel and continuously in real time even when the 3D environmental model is incomplete. This enables the formation of the optimal grasping strategy to proceed synchronously with the construction of the environmental model, without waiting for the final perfection of the global model, thus significantly reducing the total task time from target discovery to grasping execution.

[0019] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description.

[0020] The beneficial effects of this invention are: 1. By introducing an occlusion uncertainty field that incorporates task relevance weights, the UAV's viewpoint planning always revolves around the grasping task requirements. The system can intelligently identify areas that are crucial to the grasping decision, especially the parts of the target object that are currently occluded, and prioritize exploring these areas. This task-oriented perception mechanism avoids wasting observation resources in irrelevant background areas, and ensures that each viewpoint movement can acquire visual information that is valuable for completing the grasping task to the greatest extent, thereby significantly shortening the perception time required to formulate a feasible grasping strategy. 2. An online incremental grasping quality assessment and trajectory replanning mechanism is adopted, which allows the system to evaluate candidate grasping poses in parallel and continuously in real time even when the 3D environmental model is not yet complete. Parallel processing enables the formation of the optimal grasping strategy to be synchronized with the construction of the environmental model. Once the system passes the online evaluation module, which integrates geometric stability, reachability and model confidence, it determines that there is a high-quality and feasible grasping solution and can immediately trigger the grasping action without waiting for the final completion of the global model. This design completely eliminates the "waiting time" in the traditional serial process and greatly reduces the total task time from target discovery to grasping execution. 3. Online incremental evaluation is a continuously running closed-loop process. Whether due to cognitive updates caused by incomplete initial models or changes in the actual poses of objects in the environment, the system can dynamically adjust its optimal strategy through real-time sampling of new grasping poses and continuously updated multimodal scores. Specifically, when a grasping point becomes unreachable due to newly observed obstacles or its stability decreases due to object model updates, the system can immediately switch to another better alternative. This ability to continuously re-evaluate and optimize based on real-time perception information enables the system to effectively cope with uncertainties in real-world scenarios and demonstrates excellent environmental adaptability. Attached Figure Description

[0021] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. Wherein: Figure 1 A flowchart of a method for collaborative dynamic reconstruction and grasping of a drone and a robotic arm according to an embodiment of the present invention is shown; Figure 2 An isometric view of grasping a cluttered warehouse box according to Embodiment 1 of the present invention is shown; Figure 3 A side sectional view of a messy warehouse box grabbing method according to Embodiment 1 of the present invention is shown; Figure 4 A schematic diagram of a global voxel map according to Embodiment 1 of the present invention is shown; Figure 5 A schematic diagram of the map after incremental update according to Embodiment 1 of the present invention is shown; Figure 6 An initial map schematic diagram of the retrieval of relief supplies from disaster ruins according to Embodiment 2 of the present invention is shown; Figure 7 An updated map schematic diagram according to Embodiment 2 of the present invention is shown. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] According to an embodiment of the present invention, a method and system for collaborative dynamic reconstruction and grasping of unmanned aerial vehicles (UAVs) and robotic arms are proposed. The method adopts an online incremental grasping quality assessment and trajectory replanning mechanism, which allows the system to evaluate candidate grasping poses in parallel and continuously in real time when the three-dimensional environmental model is not yet complete. This enables the formation of the optimal grasping strategy to be synchronized with the construction of the environmental model, without waiting for the final perfection of the global model, and significantly reduces the total task time from target discovery to grasping execution.

[0024] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0025] Figure 1 This is a schematic flowchart illustrating a method for collaborative dynamic reconstruction and grasping using a drone and a robotic arm, according to an embodiment of the present invention. The method includes: Step S01: Scan the work scene using a depth camera mounted on the drone, acquire depth images in real time, send them to the target detection module, and incrementally fuse them to build a global voxel map; Step S02: For each unoccupied voxel in the global voxel map, calculate the comprehensive uncertainty value by multiplying its corresponding occlusion uncertainty metric and task relevance weight, and construct an occlusion uncertainty field; Step S03: Based on the occlusion uncertainty field, sample candidate viewpoints in the flight space of the UAV, use the evaluation function to calculate the total entropy of the uncertainty field of each candidate viewpoint, select the candidate viewpoint with the minimum total entropy as the next generation best observation viewpoint, and control the UAV to move towards the next generation best observation viewpoint; Step S04: While the UAV is flying toward the next-generation best observation point, update the global voxel map in real time, and execute the online incremental grasping quality assessment and trajectory replanning process in parallel to determine the optimal grasping strategy; Step S05: When the conditions of the optimal strategy are met, control the drone to hover and drive the robotic arm to execute the final planned grasping trajectory to complete the grasping of the target object.

[0026] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0027] To provide a clearer explanation of the above-mentioned method for collaborative dynamic reconstruction and grasping of a drone and a robotic arm, three specific embodiments are described below. However, it is worth noting that these embodiments are only for better illustrating the present invention and do not constitute an improper limitation of the present invention.

[0028] The following three specific examples will further illustrate a method for collaborative dynamic reconstruction and grasping between a drone and a robotic arm: Step S01: Scan the work scene using a depth camera mounted on the drone, acquire depth images in real time, send them to the target detection module, and incrementally fuse them to build a global voxel map.

[0029] After the system starts, the drone equipped with a depth camera scans the target area. Using the onboard depth camera, such as an RGB-D camera, it continuously acquires depth images of the scene. These depth images are fed in real-time into a simultaneous localization and mapping (SLAM) or pure 3D reconstruction process, incrementally fusing and updating a global voxel map. This voxel map serves as the foundational representation for the entire system's environmental cognition; each voxel contains its occupancy probability, color information, and uncertainty values ​​calculated in subsequent steps.

[0030] Step S02: For each unoccupied voxel in the global voxel map, calculate the comprehensive uncertainty value by multiplying its corresponding occlusion uncertainty metric and task relevance weight, and construct an occlusion uncertainty field.

[0031] Occlusion uncertainty is determined by calculating the degree to which occupied voxels obstruct the line of sight from the current camera optical center to the unoccupied voxel. For each unoccupied voxel in the map, the line of sight from the current camera optical center to that voxel is simulated. If the line of sight passes through voxels marked as occupied before reaching the target voxel, the target voxel is considered occluded, and its uncertainty value increases according to the degree of occlusion, such as the number of occupied voxels penetrated or the depth difference. Unoccupied visible voxels are assigned lower uncertainty values.

[0032] The task relevance weight is based on the identified potential grasping targets, assigning a weight coefficient higher than that of scene background voxels to the unoccupied voxels surrounding the target. To concentrate perception resources on task-critical regions, the system runs a lightweight target detection algorithm in parallel, such as a point cloud-based clustering or deep learning model, to initially identify potential grasping targets. Subsequently, unoccupied voxels surrounding these targets, especially areas that may be occluded, are assigned a weight coefficient higher than that of scene background voxels. Finally, the overall uncertainty value of each voxel is the product of its occlusion uncertainty measure and the task relevance weight. In this way, a region that is crucial to the grasping task but is currently occluded will exhibit an extremely high overall uncertainty value.

[0033] Step S03: Based on the occlusion uncertainty field, sample candidate viewpoints in the flight space of the UAV, use the evaluation function to calculate the total entropy of the uncertainty field of each candidate viewpoint, select the candidate viewpoint with the minimum total entropy as the next generation best observation viewpoint, and control the UAV to move towards the next generation best observation viewpoint.

[0034] The evaluation function used is the information gain function, which quantifies the reduction in overall uncertainty that can be observed from each candidate viewpoint, weighted by task relevance.

[0035] For each candidate viewpoint, the system simulates the set of voxels it can observe through ray casting and calculates the reduction in the total entropy of the uncertainty field resulting from observing these voxels, i.e., the information gain. The system selects the candidate viewpoint that maximizes the information gain as the next-generation optimal observation viewpoint and generates a safe path to guide the UAV toward it. This mechanism ensures that the UAV always proactively flies to locations that minimize the unknown state of the critical area of ​​the grasping mission.

[0036] Step S04: While the UAV is flying toward the next-generation best observation point, update the global voxel map in real time, and execute the online incremental grasping quality assessment and trajectory replanning process in parallel to determine the optimal grasping strategy.

[0037] Step S04 and step S03 are executed synchronously and in parallel. This is the core of breaking the sequential process of rebuilding first and then planning. It continuously evaluates and optimizes the crawling strategy even when the environment model is still incomplete. The specific steps are as follows: Step S041: For the target object that has been identified but whose model is not yet complete, sample several six-degree-of-freedom candidate grasping poses in real time and continuously within its currently reconstructed surface and adjacent space; Step S042: For each candidate grasping pose, a comprehensive grasping quality score is obtained by weighted fusion calculation of multiple evaluation dimensions through an online evaluation module. The evaluation dimensions include, but are not limited to: Geometric stability rating: Based on the local point cloud data of the target object in the current voxel map, analyze the distribution of contact points between the gripper and the object under the gripping pose, calculate classic gripping quality indicators such as force closure index and disturbance rejection, and evaluate the physical stability of the gripping. Reachability score: Combining the current hovering pose of the drone, the kinematic model and joint constraints of the robotic arm, and the real-time updated global voxel map, preliminary lightweight motion planning and collision detection are performed using fast random tree (RRT) or its variant algorithms to assess the possibility and difficulty of the robotic arm reaching the grasping pose without collision. Model integrity confidence: The reliability of the grasping decision is calculated based on the average uncertainty of the corresponding voxel region in the occlusion uncertainty field on which the grasping pose depends. If the grasping point is located in a high uncertainty region, i.e., the region may not have been observed or is still severely occluded, the model integrity confidence of the grasping pose will be lowered because the actual geometry may not match the current reconstruction. Step S043: Based on the comprehensive grasping quality score of all candidate grasping poses, filter and update the global optimal grasping strategy at the current moment in real time.

[0038] Step S05: When the conditions of the optimal strategy are met, control the drone to hover and drive the robotic arm to execute the final planned grasping trajectory to complete the grasping of the target object.

[0039] The conditions for the optimal strategy include: Quality compliance conditions: The overall grasping quality score of the current globally optimal grasping strategy exceeds the first preset threshold, and the drone is in a stable hovering state, which indicates that the system has found a high-confidence, executable grasping solution; Optimal efficiency condition: The system predicts that the improvement in quality score of the globally optimal crawling strategy by continuing to perform viewpoint exploration is less than the second preset threshold. This means that the "gain" of continuing to explore is no longer high. From the perspective of time efficiency, the current best solution should be executed immediately.

[0040] When one of the conditions is met, it is considered that the optimal strategy has been achieved. The drone first fine-tunes to the final grasping hover position, and then the robotic arm executes the grasping action according to the optimal grasping strategy to complete the task.

[0041] Through the closed-loop execution of steps S01 to S05, a fundamental transformation is achieved from passive perception to active task-driven perception, and from serial operation to parallel decision-making, enabling the UAV-robotic arm system to truly possess the ability to efficiently and intelligently complete grasping tasks in dynamic and unknown environments.

[0042] Example 1: Picking up boxes from a cluttered warehouse.

[0043] Scenario Description: In an automated warehouse, multiple cardboard boxes are haphazardly stacked on shelves. A drone equipped with a six-DOF robotic arm receives an instruction to grab a specific-sized box—the target box. This target box is located in the lower-middle layer of the stack, and its front label area, which is available for grabbing, is partially obscured by the boxes above it. Figure 2-3 As shown.

[0044] The drone performs a preliminary scan of the shelving area from a safe distance, quickly constructing a sparse global voxel map, such as... Figure 4 As shown, the drone identified multiple container-shaped objects.

[0045] The system calculates the uncertainty of the initial map. At this point, the visible surface uncertainty of all containers is low. However, the front of the target container is obscured by the container above, resulting in extremely high occlusion uncertainty for its voxels. Furthermore, since these voxels belong to the identified target containers, they are assigned high task relevance weights. Therefore, in the fused uncertainty field, the front area of ​​the target container becomes a significant "information hotspot."

[0046] Instead of uniformly scanning the entire row of shelves, the drone calculates the optimal viewing point that most effectively allows it to "see" the front of the target box, guided by an uncertainty field. This viewing point might be a slightly forward-facing position, providing an angled view of the obscured front. The drone then autonomously flies towards this viewing point.

[0047] During flight, the online grasp evaluation module began operating. Based on the initial sparse model, it sampled the poses from the top and sides for grasping. The evaluation yielded the following results: Top gripping: Low geometric stability score, poor load-bearing capacity at the top center of the carton, and may be uneven; Side gripping: Low accessibility score, the robotic arm needs to be very close to other cargo boxes, and the risk of collision is high; Therefore, the combined capture quality score of both schemes did not reach the trigger threshold.

[0048] As the drone approaches the new viewpoint, the depth camera begins acquiring point cloud data of the front of the target container. The map is incrementally updated, such as... Figure 5 As shown, the frontal voxel changes from not occupying (unknown) to occupying (cargo box surface), and its uncertainty value decreases sharply.

[0049] The online evaluation module immediately sampled the gripping pose on the new, more complete model. A gripping point was found located in the center of the front, near the reinforcing rib: Geometric stability rating: High, flat front, good gripping force closure; Accessibility rating: High; the robotic arm extends from the front of the drone, with a wide movement path and no collisions. Model integrity confidence: High, the region has just been observed with high quality and uncertainty has been eliminated; The overall grasping quality score of the grasping pose quickly jumped and exceeded the preset threshold.

[0050] The drone arrived at the viewpoint and hovered stably. At this point, the quality standards were immediately met.

[0051] The system triggers the grasping process. The robotic arm extends according to the planned trajectory, successfully adsorbs or grips the front of the target box, and completes the grasping.

[0052] Challenges faced by traditional methods: 1. Using drones to perform detailed reconstruction of the entire scene takes an extremely long time.

[0053] 2. Even after reconstruction, the planning algorithm may find that the top and sides of the target box are visible, but the top is uneven and the sides are close to other boxes, making it difficult for the robotic arm to achieve stable, collision-free grasping. The optimal grasping surface, the front view, is precisely what is missing in the initial model.

[0054] 3. The system may therefore determine that the capture has failed, or the operator needs to re-specify the scanning area, which is inefficient.

[0055] The advantages of this embodiment are evident: Increased efficiency: Instead of wasting time rebuilding the entire shelf, the system went straight to the point and solved the key problem of missing information.

[0056] Decision intelligence: The system recognizes the shortcomings of the initial grasping plan during the perception process and actively seeks the optimal solution, rather than exposing the problem afterward.

[0057] High success rate: By proactively eliminating uncertainties in key areas of the task, the final crawling strategy is ensured to be both stable and feasible.

[0058] Example 2: Retrieving relief supplies from disaster ruins.

[0059] Scene Description: In the post-earthquake ruins, a first-aid kit is trapped under a concrete slab, with only a corner protruding. The environment is extremely unstructured, with slight disturbances such as a gentle breeze.

[0060] The drone conducted exploratory scans over the ruins, acquiring depth images in real time and incrementally fusing them to construct an initial global voxel map. This initial map was filled with unknowns and occlusions, such as... Figure 6 As shown.

[0061] The drone identified a corner of the first aid kit as having features of a man-made object from a scattered point cloud.

[0062] In an uncertain field, the portion of the first-aid kit that is pinned down becomes a core region with high uncertainty and high weight. The system also marks the areas around key obstacles that may affect accessibility, such as the edges of concrete slabs, as regions of high uncertainty.

[0063] The drone was guided to a low-altitude hovering position where it could more clearly observe the exposed part of the first aid kit and the surrounding obstructions.

[0064] The online assessment module began operation. Since most of the target was buried, the available grasping poses were very limited. The pose for directly grasping the exposed portion was assessed first. The accessibility score indicated that the robotic arm needed to penetrate deep beneath the concrete slab at a specific angle, posing a collision risk; therefore, the score was moderate.

[0065] At this moment, a slight breeze caused the drone to shake slightly, resulting in minor changes to the reconstructed map. This reflects the dynamic environment.

[0066] Once the map was updated, the system immediately re-evaluated the previous candidate grasping poses. It discovered that one pose, due to a slight displacement of the drone, now had a better collision-free path, such as... Figure 7 As shown.

[0067] The system dynamically updates this pose to the current optimal strategy. At the same time, model integrity confidence is used to quantify risk: the system knows that the geometry under the concrete slab is uncertain, so when planning the robot arm trajectory, it commands it to approach at a more cautious speed and reserves a safety distance.

[0068] Despite the uncertainties in the environment, the overall grasping quality score of the current optimal grasping strategy has reached an acceptable threshold. The system determines that continuing to move the drone may bring more unstable factors, thus satisfying the efficiency optimization condition, that is, the marginal benefit of continuing to explore is less than the risk of executing the current plan.

[0069] The system promptly triggered the grabbing action. The robotic arm carefully and adaptively grabbed the exposed portion of the first aid kit and removed it.

[0070] Challenges faced by traditional methods: 1. The environment is completely unknown and extremely complex, making global and precise reconstruction impractical and dangerous.

[0071] 2. The first aid kit is severely obscured, and traditional view planning may not be able to effectively find it, or even if it is found, planning may be abandoned due to the incomplete model.

[0072] 3. Static planning cannot adapt to the slight changes that may occur in the ruin structure.

[0073] The advantages of this embodiment are evident: Coping with unstructured environments: The system does not rely on precise prior maps and can identify targets from fragmented information and actively explore them.

[0074] Dynamic adaptability: It can adapt to minute changes in UAV pose and incremental updates to the environmental model, adjust the strategy in real time, and exhibits strong robustness.

[0075] Risk-controlled decision-making: By clearly quantifying decision risks through model integrity confidence and adopting corresponding conservative strategies during execution, safe operation in extreme environments has been achieved.

[0076] Example 3: Installation of working tools on high-altitude cables.

[0077] Scenario Description: A vibration damper needs to be installed on a high-voltage power cable. The background of the cable is open, but high precision is required, and the environment is subject to changes in lighting and wind disturbance.

[0078] The drone flies close to the work site. After the initial scan, the voxel map of the cable may be sparse or discontinuous.

[0079] Uncertainty fields will highlight the missing line segments in these point clouds. The system assigns task-related weights to these key feature areas, including the cable itself and the interior of the vibration damper slot to be installed.

[0080] The drone was guided to the best viewpoint for observing the broken cable and the inside of the slot, specifically to repair these missing parts of the model that were critical to the mission.

[0081] In this scenario, the online evaluation module demonstrates support for precise manipulation. It samples not only the gripping pose of the shock absorber but also the final assembly pose, specifically the pose where the slot aligns with the cable.

[0082] The online assessment module plays a crucial role: Geometric stability: Assess whether the shock absorber will slip or rotate when it is gripped; Accessibility: Assess whether the entire motion trajectory from the gripper point to the assembly point is smooth and collision-free; Model integrity confidence: The system checks whether the model of the cable in the slot docking area is complete and whether the uncertainty is low enough. If the uncertainty in this area is high, the evaluation module will give the assembly strategy a low score and may drive the viewpoint planning in reverse, requiring the UAV to move again to obtain better observation data in this area until the confidence level is met.

[0083] The system will only trigger the final grabbing and installation action when the model integrity confidence of all key points on the assembly path reaches a very high threshold and the comprehensive score meets the quality standards.

[0084] The drone hovered with extremely high precision, and the robotic arm executed a precise trajectory that had been evaluated with high confidence, successfully installing the shock absorber onto the cable.

[0085] Challenges faced by traditional methods: 1. Cables are thin and long objects that are easily lost or broken during 3D reconstruction, leading to failure of the grab planning.

[0086] 2. The robotic arm needs to precisely align the slot of the anti-vibration hammer with the cable, which requires extremely high accuracy in pose estimation.

[0087] 3. Traditional methods may not be able to generate feasible and sophisticated operational strategies after reconstructing an incomplete cable model.

[0088] The advantages of this embodiment are evident: Task-oriented perception accuracy: The system is not only satisfied with seeing objects, but also strives for accurate reconstruction of key features of the task, such as the inside of the card slot and the cable interface.

[0089] Supports complex operation sequences: The online evaluation mechanism can cover the entire operation chain from grabbing to assembly, ensuring end-to-end feasibility.

[0090] Quality Assurance: By binding model integrity confidence to a high trigger threshold, a high first-time success rate is ensured in high-precision tasks, avoiding assembly failures or equipment damage caused by perception errors.

[0091] Through the above three differentiated embodiments, the advantages of this invention compared to traditional technologies are fully and profoundly demonstrated: 1. By using task-oriented occlusion uncertainty field, limited perception resources are concentrated on the most critical areas, significantly shortening the time from the start of reconnaissance to finding a feasible solution.

[0092] 2. By using online incremental capture quality assessment and trajectory replanning, the decision-making process is brought forward and runs in parallel with perception, eliminating the waiting window in the serial process and realizing simultaneous observation, thinking, and planning.

[0093] 3. Through continuous reassessment and multimodal scoring mechanisms, especially model integrity confidence, the system can confidently face uncertainties such as model incompleteness and dynamic environmental changes, and make risk-aware, adaptive intelligent decisions, thus demonstrating extremely high success rate and reliability in real, unstructured, and complex environments.

[0094] Based on the same inventive concept, this invention also proposes a system for collaborative dynamic reconstruction and grasping using a drone and a robotic arm. This system includes: a drone platform, a multi-degree-of-freedom robotic arm, an onboard computing unit, a depth camera, and a flight control system. The onboard computing unit is configured to: Run the SLAM and 3D reconstruction algorithm to perform step S01; Construct an occlusion uncertainty field and perform viewpoint planning to execute steps S02 and S03; Run the online incremental crawling quality assessment and trajectory replanning algorithm to perform step S04; Coordinate the control of the drone and the robotic arm to ultimately execute step S05.

[0095] Specifically, the airborne computing unit integrates a target detection module, which is used to identify potential capture targets in the global voxel map in real time and provide input for the task relevance weight calculation in step S02.

[0096] The present invention proposes a system for collaborative dynamic reconstruction and grasping of UAV-robotic arm, which adopts an online incremental grasping quality assessment and trajectory replanning mechanism. This allows the system to evaluate candidate grasping poses in parallel and continuously in real time when the 3D environmental model is not yet complete. This enables the formation of the optimal grasping strategy to proceed synchronously with the construction of the environmental model, without waiting for the final perfection of the global model, thus significantly reducing the total task time from target discovery to grasping execution.

[0097] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0098] Regarding the limitation of the scope of protection of this invention, those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solution of this invention are still within the scope of protection of this invention.

Claims

1. A method for collaborative dynamic reconstruction and grasping using a drone and a robotic arm, characterized in that, The method includes: Step S01: Scan the work scene using a depth camera mounted on the drone, acquire depth images in real time, send them to the target detection module, and incrementally fuse them to build a global voxel map; Step S02: For each unoccupied voxel in the global voxel map, calculate the comprehensive uncertainty value by multiplying its corresponding occlusion uncertainty metric and task relevance weight, and construct an occlusion uncertainty field; Step S03: Based on the occlusion uncertainty field, sample candidate viewpoints in the flight space of the UAV, use the evaluation function to calculate the total entropy of the uncertainty field of each candidate viewpoint, select the candidate viewpoint with the minimum total entropy as the next generation best observation viewpoint, and control the UAV to move towards the next generation best observation viewpoint; Step S04: While the UAV is flying toward the next-generation best observation point, update the global voxel map in real time, and execute the online incremental grasping quality assessment and trajectory replanning process in parallel to determine the optimal grasping strategy; Step S05: When the conditions of the optimal strategy are met, control the drone to hover and drive the robotic arm to execute the final planned grasping trajectory to complete the grasping of the target object.

2. The method for collaborative dynamic reconstruction and grasping of a drone and a robotic arm according to claim 1, characterized in that, The occlusion uncertainty measure mentioned in step S02 is determined by calculating the degree of obstruction by occupied voxels on the line of sight from the current camera optical center to the unoccupied voxel. Specifically, for each unoccupied voxel in the map, the line of sight from the current camera optical center to the voxel is simulated. If the line of sight passes through a voxel marked as occupied before reaching the target voxel, the target voxel is considered to be in an occluded state, and its uncertainty value increases according to the degree of occlusion. Unoccupied visible area voxels are assigned a lower uncertainty value.

3. The method for collaborative dynamic reconstruction and grasping of a drone and a robotic arm according to claim 1, characterized in that, The task relevance weight mentioned in step S02 is based on the identified potential grasping target. The unoccupied voxels around the grasping target are assigned a weight coefficient higher than that of the scene background voxels. The specific steps are as follows: the system runs a lightweight target detection algorithm in parallel to initially identify potential grasping targets; and assigns a weight coefficient higher than that of the scene background voxels to the unoccupied voxels around the potential grasping target object, especially the area where it may be occluded.

4. The method for collaborative dynamic reconstruction and grasping of a drone and a robotic arm according to claim 1, characterized in that, The evaluation function mentioned in step S03 is the information gain function.

5. The method for collaborative dynamic reconstruction and grasping of a drone and a robotic arm according to claim 1, characterized in that, The specific steps of step S04 are as follows: Step S041: For the target object that has been identified but whose model is not yet complete, sample several six-degree-of-freedom candidate grasping poses in real time and continuously within its currently reconstructed surface and adjacent space; Step S042: For each candidate grasping pose, a multi-dimensional weighted fusion calculation is performed through an online evaluation module to obtain its comprehensive grasping quality score; Step S043: Based on the comprehensive grasping quality score of all candidate grasping poses, filter and update the global optimal grasping strategy at the current moment in real time.

6. The method for collaborative dynamic reconstruction and grasping of a drone and a robotic arm according to claim 5, characterized in that, The evaluation dimensions mentioned include, but are not limited to: Geometric stability score: Based on the local point cloud data of the target object in the current voxel map, analyze the distribution of contact points between the gripper and the object under the gripping pose, calculate the classic gripping quality index, and evaluate the physical stability of the gripping. Accessibility rating: Combining the current hovering pose of the drone, the kinematic model and joint constraints of the robotic arm, and the real-time updated global voxel map, preliminary, lightweight motion planning and collision detection are performed to assess the possibility and difficulty of the robotic arm reaching the grasping pose without collision. Model integrity confidence: The reliability of the grasping decision is calculated based on the average uncertainty of the corresponding voxel region in the occlusion uncertainty field on which the grasping pose depends. If the grasping point is located in a high uncertainty region, i.e., the region may not have been observed or is still severely occluded, the model integrity confidence of the grasping pose will be lowered.

7. The method for collaborative dynamic reconstruction and grasping of a drone and a robotic arm according to claim 1, characterized in that, The optimal strategy conditions mentioned in step S05 include: Quality compliance conditions: The overall grasping quality score of the current global optimal grasping strategy exceeds the first preset threshold, and the drone is in a stable hovering state; Optimal efficiency condition: The system predicts that the improvement in quality score of the globally optimal crawling strategy due to continuing to execute viewpoint exploration is less than the second preset threshold. When one of the conditions is met, it is considered that the optimal strategy has been achieved.

8. A system for implementing the method of any one of claims 1-7, characterized in that, The system includes: a drone platform, a multi-degree-of-freedom robotic arm, an onboard computing unit, a depth camera, and a flight control system.

9. A system for collaborative dynamic reconstruction and grasping of a drone and a robotic arm according to claim 8, characterized in that, The onboard computing unit is configured to: Run the SLAM and 3D reconstruction algorithm to perform step S01; Construct an occlusion uncertainty field and perform viewpoint planning to execute steps S02 and S03; Run the online incremental crawling quality assessment and trajectory replanning algorithm to perform step S04; Coordinate the control of the drone and the robotic arm to ultimately execute step S05.

10. A system for collaborative dynamic reconstruction and grasping of a drone and a robotic arm according to claim 8, characterized in that, The airborne computing unit integrates a target detection module, which is used to identify potential capture targets in the global voxel map in real time and provide input for the task relevance weight calculation in step S02.

Images