A robot perception method and system under an occlusion scene

Abstract

The application relates to the technical field of robots, and provides a robot perception method and system in a shielding scene, which comprises the following steps: S1, collecting target data and environment data, obtaining target visibility, target pose confidence and line-of-sight shielding degree according to the target data and the environment data, and weighting and fusing the target visibility, the target pose confidence and the line-of-sight shielding degree to obtain a shielding index for quantifying the shielding degree of the target; S2, assigning a corresponding perception strategy, a pose and joint optimization scheme and a trajectory planning mode to the target according to the shielding index; and S3, controlling a mechanical arm to execute adjustment of joint configuration, end pose and motion trajectory according to the assigned perception strategy, pose and joint optimization scheme and trajectory planning mode. The application can realize adaptive perception strategy and iterative optimization of an end pose based on the shielding degree of the target, and improves the active regulation and control capability of the robot in the shielding scene.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more specifically, to a robot perception method and system in occluded scenarios. Background Technology

[0002] Current mainstream robot vision guidance systems generally adopt a serial open-loop architecture of perception, planning, and control. In actual production scenarios, factors such as workpiece stacking, interference from surrounding tooling equipment, occlusion by the robotic arm's own links, and dynamic intervention by operators lead to widespread visual occlusion. Occlusion directly causes the missing visible area of ​​the target and insufficient number of feature points, which in turn leads to a decrease in pose estimation accuracy or even solution failure. These are the core technical bottlenecks restricting the robot's ability to operate in complex scenarios.

[0003] In existing serial open-loop architectures, the sensing and control systems operate independently and transmit data unidirectionally, lacking a bidirectional collaborative mechanism based on occlusion states. Specifically, the degree of occlusion is often only considered an internal state variable of the sensing end, unable to be used as a control input to drive the robotic arm to adjust its own state. Furthermore, the joint configuration, end-effector posture, and motion trajectory of the robotic arm are designed solely around the task at hand, completely disregarding their impact on the sensing effect. This creates a disconnect between occlusion perception and end-effector control, forcing the robot to passively endure occlusion interference in occluded scenarios, unable to actively reduce the degree of occlusion through posture fine-tuning, thus diminishing the robot's proactive control capabilities in occluded environments. Summary of the Invention

[0004] To improve the robot's active control capability in occluded scenarios, this invention provides a robot perception method and system for occluded scenarios, the specific technical solution of which is as follows: A robot perception method in occluded scenes includes the following steps: S1. Collect target data and environmental data. Based on the target data and environmental data, obtain target visibility, target pose confidence and line-of-sight occlusion. Weightedly fuse target visibility, target pose confidence and line-of-sight occlusion to obtain an occlusion index used to quantify the degree of target occlusion. S2, based on the occlusion index, assigns the corresponding perception strategy, posture and joint optimization scheme and trajectory planning mode to the target; S3 controls the robotic arm to adjust joint configuration, end effector posture, and motion trajectory based on the assigned perception strategy, posture and joint optimization scheme, and trajectory planning mode.

[0005] The robot perception method in the occlusion scenario integrates target visibility, target pose confidence, and line-of-sight occlusion to obtain an occlusion index for quantifying the degree of target occlusion. Based on the occlusion index, the method assigns corresponding perception strategies, posture and joint optimization schemes, and trajectory planning modes to the target. This enables adaptive perception strategies driven by the degree of target occlusion and iterative optimization of the end-effector posture, thereby improving the robot's active control capability in occlusion scenarios.

[0006] Preferably, step S2 specifically includes: S21, if the occlusion index is not greater than the preset first index threshold, execute the conventional precision perception strategy, acquire the color image and depth map of the target through the RGB-D camera, obtain the 6D pose based on the depth map, and the robotic arm maintains the current configuration and posture and moves normally along the planned trajectory. S22, if the occlusion index is greater than the preset first index threshold and not greater than the preset second index threshold, the occlusion area is geometrically completed and matched based on the visible area features to obtain coarse-grained pose. At the same time, the robot end pitch angle and yaw angle are adjusted under the allowable pose error of the task to maximize the visible area of ​​the target for iterative optimization. S23, if the occlusion index is greater than the preset second index threshold, generate several candidate observation viewpoints in the reachable space around the target and estimate the occlusion index of each viewpoint, obtain the path length between the end point and the viewpoint, obtain the optimal viewpoint with the minimum comprehensive cost based on the occlusion index and the path length, and perform multi-view point cloud stitching and pose fusion based on the optimal viewpoint. Among them, the preset first exponential threshold is less than the preset second exponential threshold.

[0007] Preferably, step S23 specifically includes: S231, Based on the optimal viewpoint, plan the local motion path of the robotic arm, control the robotic arm to move along the local motion path to the optimal viewpoint and obtain the second visual point cloud data under the optimal viewpoint; S232, using the camera coordinate system of the initial viewpoint as a reference, transforms the second visual point cloud data to the same coordinate system; S233: Extract local geometric features of visible point clouds from two perspectives respectively, establish initial corresponding point pairs through nearest neighbor search and eliminate mismatches, then minimize the point-to-surface distance error between the visible point clouds of the two visions to obtain the relative pose, and finally perform point cloud fusion to generate the target point cloud. S234, estimate the target pose at each viewpoint and obtain the corresponding occlusion index, obtain the confidence weight of the target pose based on the corresponding occlusion index, fuse the rotation component of the target pose and fuse the position component of the target pose based on the confidence weight to obtain the fused pose; S235 If the equivalent occlusion index or pose accuracy of the fused pose meets the requirements, the final pose is output; otherwise, based on the current fused pose, the next optimal viewpoint is planned and iterative optimization is performed based on the optimal viewpoint.

[0008] Preferably, step S2 further includes: S24, Identify different occlusion types based on target data and environmental data, including static environmental occlusion, dynamic environmental occlusion, robotic arm self-occlusion, and target self-occlusion; S25, for both static and dynamic environmental occlusion, the Euler angles of the robotic arm's end effector are used as decision variables. Under the premise of satisfying preset task constraints, the end effector's posture is optimized by minimizing the occlusion index under the current posture. S26, In response to self-occlusion of the robotic arm, the degree of self-occlusion is minimized by adjusting the joint angle while keeping the end-effector's position unchanged; S27, In response to target self-blocking, within the process allowance, adjust the approach angle and contact position of the end effector to avoid the blocking part.

[0009] Preferably, step S2 further includes: S28, Based on the static environmental map, plan the initial global path and perform scrolling window optimization along the initial global path; S29, obtain the real-time occlusion index and the future occlusion index, and adjust the local trajectory and pose according to the real-time occlusion index and the future occlusion index.

[0010] A robot perception system for occluded environments includes: The occlusion index acquisition module is used to collect target data and environmental data, and obtain target visibility, target pose confidence and line-of-sight occlusion based on the target data and environmental data. The target visibility, target pose confidence and line-of-sight occlusion are weighted and fused to obtain the occlusion index used to quantify the degree of occlusion of the target. The target strategy allocation module is used to allocate corresponding perception strategies, pose and joint optimization schemes, and trajectory planning modes to the target based on the occlusion index. The robotic arm execution module is used to control the adjustment of the joint configuration, end effector posture, and motion trajectory of the robotic arm according to the assigned perception strategy, posture and joint optimization scheme, and trajectory planning mode.

[0011] Preferably, the target strategy allocation module includes: The conventional perception strategy unit is used to execute the conventional precision perception strategy when the occlusion index is not greater than the preset first index threshold. It acquires the color image and depth image of the target through the RGB-D camera, obtains the 6D pose based on the depth image, and the robotic arm maintains the current configuration and posture and moves normally along the planned trajectory. The pose iteration optimization unit is used to perform geometric completion of the occlusion area and match based on the visible area features when the occlusion index is greater than the preset first index threshold and not less than the preset second index threshold to obtain coarse-grained pose. At the same time, it adjusts the robot end pitch angle and yaw angle under the allowable pose error of the task to perform iterative optimization with the goal of maximizing the visible area of ​​the target. The multi-view pose fusion unit is used to generate several candidate observation viewpoints in the reachable space around the target when the occlusion index is greater than the preset second index threshold, estimate the occlusion index of each viewpoint, obtain the path length between the end point and the viewpoint, obtain the optimal viewpoint with the minimum comprehensive cost based on the occlusion index and the path length, and perform multi-view point cloud stitching and pose fusion based on the optimal viewpoint. Among them, the preset first exponential threshold is less than the preset second exponential threshold.

[0012] Preferably, the multi-view pose fusion unit includes: The point cloud data acquisition subunit is used to plan the local motion path of the robotic arm based on the optimal viewpoint, control the robotic arm to move along the local motion path to the optimal viewpoint, and acquire the second visual point cloud data under the optimal viewpoint. The point cloud data alignment sub-unit is used to transform the second-view point cloud data to the same coordinate system, based on the camera coordinate system of the initial viewpoint. The target point cloud generation subunit is used to extract local geometric features of the visible point clouds from two perspectives respectively. It establishes initial corresponding point pairs through nearest neighbor search and removes mismatches. Then, it minimizes the point-to-surface distance error between the visible point clouds of the two visions to obtain the relative pose. Finally, it performs point cloud fusion to generate the target point cloud. The pose fusion acquisition subunit is used to estimate the target pose under each viewpoint and obtain the corresponding occlusion index. Based on the corresponding occlusion index, the confidence weight of the target pose is obtained. The rotation component of the target pose is fused and the position component of the target pose is fused according to the confidence weight to obtain the fused pose. The multi-pose fusion subunit is used to output the final pose when the equivalent occlusion index or pose accuracy of the fused pose meets the requirements; otherwise, it plans the next optimal viewpoint based on the current fused pose and performs iterative optimization based on the optimal viewpoint.

[0013] Preferably, the target strategy allocation module further includes: The occlusion type identification unit is used to identify different occlusion types based on target data and environmental data. The occlusion types include static environmental occlusion, dynamic environmental occlusion, robotic arm self-occlusion, and target self-occlusion. The end-effector posture optimization unit is used to optimize the end-effector posture for both static and dynamic environmental occlusion. Using the Euler angles of the robotic arm end effector as the decision variable, it optimizes the end-effector posture by minimizing the occlusion index under the current posture, while meeting preset task constraints. For self-occlusion of the robotic arm, it minimizes the degree of self-occlusion by adjusting the joint angles while keeping the end-effector operation posture unchanged. For target self-occlusion, it adjusts the approach angle and contact position of the end effector within the process allowance to avoid the occluded parts.

[0014] Preferably, the target strategy allocation module further includes: The global path planning unit is used to plan an initial global path based on a static environmental map and perform rolling window optimization along the initial global path. The trajectory and pose adjustment unit is used to acquire the real-time occlusion index and the future occlusion index, and adjust the local trajectory and pose according to the real-time occlusion index and the future occlusion index. Attached Figure Description

[0015] The invention will be further understood from the following description taken in conjunction with the accompanying drawings. The components in the drawings are not necessarily drawn to scale, but rather the emphasis is on illustrating the principles of the embodiments. In different views, the same reference numerals designate corresponding parts.

[0016] Figure 1 This is a schematic diagram of the overall process of a robot perception method in an occluded scene provided in one embodiment of the present invention; Figure 2 This is a flowchart illustrating the specific method for obtaining step S2 in one embodiment of the present invention. Figure 1 ; Figure 3 This is a flowchart illustrating the specific method of step S23 in one embodiment of the present invention; Figure 4 This is a flowchart illustrating the specific method of step S2 in one embodiment of the present invention. Figure 2 ; Figure 5 This is a flowchart illustrating the specific method of step S2 in one embodiment of the present invention. Figure 3 . Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to its embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of the invention.

[0018] Current mainstream robot vision guidance systems generally adopt a serial open-loop architecture of perception, planning, and control. Specifically, the vision perception unit independently completes image acquisition, target detection, and pose calculation; the motion planning unit generates the robot arm's motion trajectory using the target pose as input and the task as constraint; and then the motion control unit drives the robot arm to perform the task along the trajectory. This serial open-loop architecture is technically mature and operates stably in structured scenarios where the target is fully visible and the environment is fixed, and is currently the standard configuration for industrial robots.

[0019] However, in actual production scenarios, factors such as workpiece stacking, interference from surrounding tooling and equipment, occlusion by the robotic arm's own linkages, and dynamic intervention by operators lead to widespread visual occlusion. Occlusion directly causes missing visible areas of the target and insufficient number of feature points, resulting in decreased pose estimation accuracy or even solution failure. These are core technical bottlenecks restricting the robot's ability to operate in complex scenarios. Existing technologies for addressing occlusion mainly focus on optimization along two independent directions. One is the passive completion approach at the perception end. This approach uses methods such as introducing a prior CAD model of the target, training a deep learning network for point cloud completion, and fusing multi-view data to algorithmically complete the geometric information of the occluded area, attempting to recover the complete pose of the target from limited visible information. However, this type of method is essentially a retrospective remedy for occlusion results, and it generally only works under low to medium occlusion levels. When the occlusion level is too high or exceeds a threshold, the completion error will amplify dramatically, pose reliability cannot be guaranteed, and it cannot fundamentally reduce the occurrence of occlusion.

[0020] Another approach is physical obstacle avoidance routing at the control end. When an obstacle is detected encroaching on the work path, trajectory replanning is triggered to bypass the obstacle and avoid collisions with the robotic arm. However, this type of method only focuses on the physical obstacle avoidance of the robotic arm itself and does not incorporate camera line-of-sight and target perception quality into the planning and optimization objectives. It not only fails to improve the observation angle by actively adjusting the robotic arm's state, but may even further exacerbate line-of-sight obstruction due to trajectory detours, leading to a further decline in perception quality.

[0021] Understandably, in the existing serial open-loop architecture, the sensing system and control system operate independently and transmit data in one direction only, lacking a bidirectional collaborative mechanism based on occlusion states. Specifically, the degree of occlusion is often only considered as an internal state variable of the sensing end, and cannot be used as a control input to drive the robotic arm to adjust its own state. Furthermore, the joint configuration, end-effector posture, and motion trajectory of the robotic arm are designed solely around the task at hand, completely disregarding their impact on the sensing effect. This creates a disconnect between occlusion perception and end-effector control, forcing the robot to passively endure occlusion interference in occluded scenarios. It cannot actively reduce the degree of occlusion through posture fine-tuning, reducing the robot's proactive control capabilities in occluded scenarios. Ultimately, this leads to frequent work interruptions, limiting the robot's application scope and operational efficiency in complex industrial scenarios.

[0022] To improve the robot's active control capabilities in occluded scenarios, such as Figure 1 As shown, an embodiment of the present invention provides a robot perception method in an occluded scene, comprising the following steps: S1. Collect target data and environmental data. Based on the target data and environmental data, obtain target visibility, target pose confidence and line-of-sight occlusion. Weightedly fuse target visibility, target pose confidence and line-of-sight occlusion to obtain an occlusion index used to quantify the degree of target occlusion.

[0023] Target data includes, but is not limited to, real-time visual data, target 3D point cloud and model, key features, prior semantics, and process data. Real-time visual data includes RGB color images of the target area and pixel-aligned depth images, which can be acquired by an RGB-D camera and preprocessed through camera intrinsic parameter calibration and color-depth extrinsic parameter alignment. Target 3D point cloud and model include real-time visible target point cloud and pre-stored standard CAD complete point cloud / triangular mesh model. Key features mainly include target geometric key points and functional reference surfaces. Prior semantics and process data include target category, size specifications, weight, and operational process constraints, which are generally pre-stored in the material and process library.

[0024] Environmental data includes, but is not limited to, static environmental structure data, dynamic environmental obstacle data, and global spatial depth data. Static environmental structure data includes the 3D geometric model / point cloud and spatial coordinates of fixed obstacles such as workbenches and fixtures. Dynamic environmental obstacle data includes the category, position, speed, and direction vectors of moving objects in the scene, such as operators and AGVs. Global spatial depth data includes the depth map of the camera's complete field of view and the global 3D point cloud map of the scene.

[0025] Target visibility can be obtained by calculating the proportion of the visible point cloud to the visible area of ​​the complete target based on the prior CAD model of the target. It reflects the physical degree to which the target body is occluded. For example, target visibility can be defined as 1 - target visible area size / theoretical size of the target's complete area. The target visible area size is determined based on the number of visible point clouds, and the theoretical size of the target's complete area is determined based on the total number of points in the target's standard CAD point cloud. Target visibility is between 0 and 1, where 0 indicates that the target is fully visible and 1 indicates that the target is completely occluded.

[0026] Target pose confidence primarily reflects the actual impact of occlusion on perception accuracy, i.e., the reliability of the current perceived pose. It is determined by estimating the target pose, obtaining the keypoint reprojection error and feature matching rate, and then basing the confidence score on the keypoint reprojection error. Target pose confidence ranges from 0 to 1, where 0 indicates the pose estimation is completely reliable and the error is within acceptable limits, and 1 indicates the pose estimation is completely ineffective. For example, target pose confidence = ,in, These are the average Euclidean distance between the keypoint coordinates and the image-detected keypoint coordinates, and the preset error threshold for complete pose failure. Alternatively, during the pose solving process, the number of inliers that meet the reprojection error threshold and the total number of matched feature points can be counted among the successfully matched feature points to determine the target pose confidence = 1 - number of inliers / total number of matched feature points.

[0027] The degree of line-of-sight obstruction reflects the extent to which the line of sight is blocked. It can be determined based on the intersection ratio of the line connecting the camera's optical center to the target's center with environmental obstacles and robotic arm links. It ranges from 0 to 1, where 0 indicates that all lines of sight from the camera to the target are completely unobstructed, and 1 indicates that the line of sight is completely blocked.

[0028] The occlusion index is between 0 and 1, where 0 indicates that the target is fully visible and 1 indicates that the target is completely occluded.

[0029] As a preferred technical solution, since line-of-sight occlusion simultaneously reduces pose accuracy and visibility, the coupled deterioration effect of target visibility, target pose confidence, and line-of-sight occlusion can be considered. An occlusion coupling deterioration coefficient, representing the nonlinear amplification error of multiple occlusions, is obtained based on target visibility, target pose confidence, and line-of-sight occlusion. The occlusion index is obtained by weighted fusion of these factors. The occlusion coupling deterioration coefficient can be defined as the product of target visibility, target pose confidence, and line-of-sight occlusion, ensuring that when low target visibility, poor pose confidence, and severe line-of-sight occlusion coexist, the occlusion hazard is nonlinearly amplified, thus addressing the problem of traditional linear weighting underestimating the risk of severe occlusion.

[0030] Based on this, the occlusion index = λ1 × target visibility + λ2 × target pose confidence + λ3 × line-of-sight occlusion + (1 + λ4 × occlusion coupling deterioration coefficient), where λ1, λ2, and λ3 are the weighted weights of target visibility, target pose confidence, and line-of-sight occlusion, and λ4 is the normalized adjustment coefficient.

[0031] In addition, an environment-viewpoint complementary gain coefficient can be introduced to characterize the occlusion mitigation capability brought about by multi-viewpoint and attitude optimization. Generally, the more effective observation views there are, the larger the environment-viewpoint complementary gain coefficient and the smaller the viewpoint overlap rate. For example, the environment-viewpoint complementary gain coefficient is positively correlated with (number of effective observation views × (1 - viewpoint overlap rate)). The final occlusion index = λ1 × target visibility + λ2 × target pose confidence + λ3 × line-of-sight occlusion + (1 + λ4 × occlusion coupling deterioration coefficient) + (1 - λ5 × environment-viewpoint complementary gain coefficient). λ5 is a normalization adjustment coefficient. Since multi-viewpoint complementarity and attitude optimization can offset some occlusion errors, based on the environment-viewpoint complementary gain coefficient, the more complex the occlusion, the stronger the compensation.

[0032] S2 assigns corresponding perception strategies, posture and joint optimization schemes, and trajectory planning modes to the target based on the occlusion index.

[0033] As a preferred technical solution, such as Figure 2 As shown, step S2 specifically includes: S21, if the occlusion index is not greater than the preset first index threshold, execute the conventional precision perception strategy, acquire the target's color image and depth map through the RGB-D camera, obtain the 6D pose based on the depth map, and the robotic arm maintains the current configuration and posture, moving normally along the planned trajectory.

[0034] First, RGB-D data calibration, alignment, and preprocessing are performed: based on the camera intrinsic and color-depth extrinsic calibration results, the depth map is mapped to the RGB image coordinate system and denoised. Pixel-by-pixel back projection is performed through a pinhole camera model to generate the original 3D point cloud in the camera coordinate system. Next, based on the RGB-D data calibration, alignment, and preprocessing, 2D target instance segmentation is performed. The segmentation mask is mapped to the depth map / original point cloud, and the point cloud containing only the target is cropped. Then, target point cloud preprocessing is performed, including outlier filtering, voxel downsampling, and normal estimation. Finally, the initial pose is solved, iteratively optimized, and the coordinate system is transformed to obtain the 6D pose.

[0035] S22, if the occlusion index is greater than the preset first index threshold and not greater than the preset second index threshold, perform geometric completion on the occluded area and match it based on the visible area features to obtain coarse-grained pose. At the same time, adjust the robot end-effector pitch angle and yaw angle under the allowable pose error of the task to perform iterative optimization with the goal of maximizing the visible area of ​​the target.

[0036] The process of geometrically completing the occluded region and matching it based on the features of the visible region to obtain coarse-grained pose specifically includes: extracting and preprocessing the visible point cloud of the target based on the target CAD model, then performing CAD model consistency pruning to geometrically complete the occluded region; starting from the complete prior model, restoring the theoretical visible model that is completely consistent with the currently observed occlusion morphology to avoid registration drift caused by matching part of the point cloud with the complete model; then extracting local geometric features from the actual visible point cloud and the candidate theoretical visible point cloud respectively; finally, performing nearest neighbor matching on the extracted local geometric features to obtain initial corresponding point pairs, and then performing mismatch elimination and pose optimization to obtain the coarse-grained pose.

[0037] The iterative optimization of adjusting the robot's end effector pitch and yaw angles under the permissible attitude error of the task to maximize the visible area of ​​the target includes: using the angular offset of the end effector relative to the working reference attitude (such as pitch angle offset and yaw angle offset) as variables, and taking maximizing the visible area of ​​the target, i.e. minimizing the target occlusion index, as the optimization objective, and using the gradient descent method to iteratively solve the problem under the premise of satisfying the constraints of working attitude tolerance, end effector position, joint limit and obstacle avoidance.

[0038] Here, the gradient descent method is used for iterative solution, which includes: obtaining the occlusion index gradient under the current pose, updating the pose angle along the negative gradient direction, simulating the visible area under different poses based on the target coarse pose and environment model, completing gradient calculation and iteration, until the occlusion index decrease is less than the gradient threshold such as 0.01 or the pose change is less than the pose change threshold such as 0.2°, determining that the iteration has converged and outputting the optimal end pose and the corresponding occlusion index.

[0039] S23, if the occlusion index is greater than the preset second index threshold, generate several candidate observation viewpoints in the reachable space around the target and estimate the occlusion index of each viewpoint, obtain the path length between the end point and the viewpoint, obtain the optimal viewpoint with the minimum comprehensive cost based on the occlusion index and the path length, and perform multi-view point cloud stitching and pose fusion based on the optimal viewpoint.

[0040] First, clarify the mathematical expression of the viewpoint and the feasible domain boundary to avoid generating invalid candidate points. Based on this, generate several candidate observation viewpoints within the reachable space around the target. For example, a heuristic sampling method based on occlusion bias can be used to generate candidate observation viewpoints, that is, based on the current occlusion direction and type, dense sampling is performed in the region opposite to the occlusion, while sparse sampling is performed in other regions.

[0041] Based on the pre-stored environment model, target CAD model, and robotic arm kinematic model, the occlusion situation under each candidate viewpoint is simulated forward. By obtaining the corresponding target visibility, target pose confidence, and line-of-sight occlusion, and performing weighted fusion, the occlusion index of the candidate observation viewpoint is obtained.

[0042] The occlusion index and path length are weighted and fused to obtain a comprehensive cost. Iterative optimization is then performed based on this comprehensive cost. When the occlusion index of a candidate viewpoint is no greater than a preset first index threshold, the viewpoint with the shortest path is directly selected as the optimal solution.

[0043] The preset first exponential threshold is less than the preset second exponential threshold. Here, the preset first exponential threshold and the preset second exponential threshold can be set to 0.3 and 0.7, respectively.

[0044] Specifically, such as Figure 3 As shown, step S23 specifically includes: S231, based on the optimal viewpoint, plans the local motion path of the robotic arm, controls the robotic arm to move from the initial viewpoint along the local motion path to the optimal viewpoint, and obtains the second visual point cloud data under the optimal viewpoint.

[0045] Based on the optimal viewpoint, RRT can be used. The local motion path of the robotic arm is planned using either the Rapidly-exploring Random Tree Star (RTS) or artificial potential field method, ensuring no collisions or continuous occlusion during the motion. After the robotic arm reaches its position, it is stabilized for 1-2 frames to eliminate jitter. Then, RGB images, depth maps, and point cloud data from that viewpoint are acquired, and the target pose of the camera at that viewpoint in the robot's base coordinate system is calculated to facilitate subsequent coordinate system unification.

[0046] S232 transforms the second visual point cloud data to the same coordinate system, using the camera coordinate system of the initial viewpoint as a reference.

[0047] Using the camera coordinate system of the initial viewpoint as a reference, the second visual point cloud data under the optimal viewpoint is transformed to the same coordinate system to achieve coarse alignment of the point clouds, resulting in the second visual point cloud data in the first viewpoint coordinate system. Specifically, the target pose estimated independently for each viewpoint can be obtained. Combined with the camera extrinsic participation of the robot arm's forward kinematics, the relative transformation matrix between the two viewpoint cameras can be derived. Then, the second visual point cloud data under the optimal viewpoint is transformed to the camera coordinate system of the initial viewpoint using the relative transformation matrix.

[0048] S233: Extract local geometric features of visible point clouds from two perspectives respectively, establish initial corresponding point pairs through nearest neighbor search and eliminate mismatches, then minimize the point-to-surface distance error between the visible point clouds of the two visions to obtain the relative pose, and finally perform point cloud fusion to generate the target point cloud.

[0049] Local geometric features are extracted from the visible point cloud data under the initial viewpoint and the obtained second-view point cloud data in the first-viewpoint coordinate system. Initial corresponding point pairs are established through nearest neighbor search, and mismatches are eliminated using RANSAC (Random Sample Consensus). The point-to-surface ICP (Iterative Closest Point) algorithm is used to iteratively optimize the relative transformation to minimize the point-to-surface distance error between the visible point clouds of the two views, obtaining an accurate relative pose or relative transformation matrix. Voxel downsampling is performed to remove duplicates in overlapping regions, and non-overlapping regions (complementary occlusion regions) are directly merged to generate a more complete target point cloud matrix.

[0050] S234, estimate the target pose at each viewpoint and obtain the corresponding occlusion index, obtain the confidence weight of the target pose based on the corresponding occlusion index, fuse the rotation component of the target pose and fuse the position component of the target pose based on the confidence weight to obtain the fused pose.

[0051] Specifically, the confidence weight of a target pose is 1 minus the corresponding occlusion index. For the rotation component, it can be fused using spherical linear interpolation or quaternion weighted averaging to obtain the final rotation pose.

[0052] The equivalent occlusion index of the fused pose characterizes the overall reliability of the final fused pose, and it can be determined based on the weighted average of the occlusion indices corresponding to the target pose under each view. Preferably, a baseline occlusion index is first obtained based on the weighted average of the occlusion indices corresponding to the target pose under each view. Then, the proportion of the intersection to the union of the visible target regions from each view is calculated to obtain the visual average overlap rate. The baseline occlusion index is positively corrected based on the visual average overlap rate. This is because the lower the visual average overlap rate, the stronger the complementarity between the two viewpoints. For example, the equivalent occlusion index = baseline occlusion index × (1 - fusion gain coefficient × (1 - visual average overlap rate)), where the fusion gain coefficient is a preset parameter, generally between 0 and 0.5.

[0053] S235 If the equivalent occlusion index or pose accuracy of the fused pose meets the requirements, the final pose is output; otherwise, based on the current fused pose, the next optimal viewpoint is planned and iterative optimization is performed based on the optimal viewpoint.

[0054] For example, viewpoint planning can be stopped and the final pose can be output when the equivalent occlusion index of the fused pose is no greater than a preset first index threshold or the pose accuracy meets the requirements. If the requirements are still not met, the next optimal viewpoint can be planned again based on the current fused pose, and iterative optimization can be performed based on the optimal viewpoint until the conditions are met or the maximum number of viewpoints is reached.

[0055] In some cases, if the occlusion index is too high and vision is nearly ineffective due to severe occlusion, end-effector tactile / force sensors can be used to directly contact the key geometric features of the target. This physical contact allows for precise geometric constraints to be obtained, thus strongly complementing visual information. For example, when the occlusion index is greater than a preset second index threshold but not greater than a third index threshold (e.g., 0.9), coarse-grained pose estimation can be output using vision, combined with tactile feedback for fine-tuning of key features. If the occlusion index is greater than the third index threshold, tactile feedback dominates pose estimation, with vision serving as an auxiliary verification. Key geometric features include edges, corners, the center of positioning holes, and reference planes. Priority is given to key geometric features within visually occluded areas to supplement missing visual information.

[0056] A robotic arm can drive a six-dimensional force sensor / array of haptic sensors at the end effector to slowly approach the target surface along the normal direction. When the force sensor detects that the contact force exceeds a threshold, the movement immediately stops and the current position and attitude of the end effector are recorded. Using the torque information and calibration parameters of the force sensor, the 3D coordinates of the contact point in the tool coordinate system and the surface normal direction are calculated. All planned feature points are then detected sequentially to obtain the set of contact point coordinates and the set of normals. Then, the haptic pose is solved. By establishing a matching relationship between the contact point and the corresponding key geometric feature points on the CAD model, a pose solution algorithm based on point-to-surface correspondence is used to minimize the distance error and normal direction error between the contact point and the model feature points. Finally, the estimated target pose and the corresponding confidence level are output.

[0057] The target pose obtained based on tactile and visual estimation is weighted and fused according to their respective uncertainties, i.e., covariance, using a Bayesian framework to obtain the optimal pose.

[0058] S3 controls the robotic arm to adjust joint configuration, end effector posture, and motion trajectory based on the assigned perception strategy, posture and joint optimization scheme, and trajectory planning mode.

[0059] Based on the assigned perception strategy, attitude and joint optimization scheme, and trajectory planning mode, control commands are generated, such as end-effector pitch / yaw fine-tuning, zero-space joint configuration adjustment, optimal observation viewpoint target pose, local replanning motion trajectory, and operational constraints. The decision layer parses the commands and verifies the constraints, and then controls the robotic arm to adjust the joint configuration, end-effector attitude, and motion trajectory. Simultaneously, during the movement, the robotic arm's body parameters, such as joint speed and angle, as well as raw data from the end-effector six-dimensional force / torque sensor, are collected and fed back to the perception layer, thus forming a closed loop of occlusion assessment, perception adaptation, attitude optimization, trajectory replanning, and execution feedback.

[0060] In summary, the robot perception method in occlusion scenarios integrates target visibility, target pose confidence, and line-of-sight occlusion to obtain an occlusion index for quantifying the degree of target occlusion. Based on the occlusion index, the method assigns corresponding perception strategies, pose and joint optimization schemes, and trajectory planning modes to the target. This enables adaptive perception strategies driven by the degree of target occlusion and iterative optimization of the end-effector pose, thereby improving the robot's active control capabilities in occlusion scenarios.

[0061] In one embodiment, such as Figure 4 As shown, step S2 further includes: S24. Identify different occlusion types based on target data and environmental data. The occlusion types include static environmental occlusion, dynamic environmental occlusion, robotic arm self-occlusion, and target self-occlusion.

[0062] Specifically, the parallel detection method can identify different occlusion types. That is, within the same perception cycle, based on the robot arm kinematic data, environmental visual / depth data, and target CAD model, static environmental occlusion, dynamic environmental occlusion, robot arm self-occlusion, and target self-occlusion are independently detected and identified.

[0063] For static environmental occlusion, a ray casting method can be used. N representative points are uniformly sampled from the camera's optical center onto the target surface. Then, probe rays are emitted, and each ray is checked to see if it intersects with an obstacle collider in a pre-constructed set of static obstacles. The target area corresponding to the intersecting ray is defined as the static occlusion area. Finally, the proportion of the static occlusion area to the target's complete theoretical area is calculated to obtain the static occlusion contribution. If the static occlusion contribution is greater than a corresponding preset threshold, it is determined to be static environmental occlusion.

[0064] For dynamic environmental occlusion, a point cloud difference method can be used. First, the real-time scene point cloud is registered with the pre-stored static point cloud and then subtracted to obtain a difference point cloud. Then, it is determined whether the difference point cloud is located on the line-of-sight path between the camera's optical center and the target. If so, the target area corresponding to the difference point cloud located on the line-of-sight path is marked as the dynamic occlusion area. Finally, the proportion of the dynamic occlusion area to the complete theoretical area of ​​the target is calculated to obtain the dynamic occlusion contribution. If the dynamic occlusion contribution is greater than the corresponding preset threshold, it is determined to be dynamic environmental occlusion.

[0065] For self-occlusion by a robotic arm, a mask projection method can be used to project the 3D model of the robotic arm onto the image plane, generating a semantic mask for the robotic arm. The self-occluded region is obtained by calculating the intersection of the robotic arm semantic mask and the target mask. The proportion of the self-occluded region to the total theoretical area of ​​the target is then calculated to obtain the self-occlusion contribution of the robotic arm. If the self-occlusion contribution is greater than a corresponding preset threshold, it is determined to be self-occlusion by the robotic arm. For target self-occlusion, the target self-occlusion contribution can be obtained by calculating the proportion of the number of occluded key features to the total number of key features or the proportion of the self-occluded area. If this proportion is greater than a corresponding preset threshold, it is determined to be target self-occlusion.

[0066] If all types of occlusion exist simultaneously, the type with the largest occlusion contribution can be used as the current occlusion type.

[0067] S25, for both static and dynamic environmental occlusion, using the Euler angles of the robotic arm's end effector as the decision variable, and under the premise of satisfying preset task constraints, optimizes the end effector posture by minimizing the occlusion index under the current posture.

[0068] The Euler angles of the robotic arm's end effector include pitch, yaw, and roll angles. Preset task constraints include, but are not limited to, the permissible attitude range for the operation task (e.g., grasping angle deviation ≤ ±15°), joint limits, and obstacle avoidance distance. With minimizing the occlusion index under the current attitude as the optimization objective, a gradient descent method is employed, guided by the gradient of the occlusion degree with respect to the attitude angles, to optimize the end effector attitude iteratively until the optimal attitude is obtained.

[0069] S26 addresses self-occlusion in robotic arms by minimizing the degree of self-occlusion by adjusting joint angles while maintaining the end-effector's operational posture.

[0070] For redundant robotic arms with 7 or more degrees of freedom, the end effector pose corresponds to multiple joint solutions. Therefore, null-space projection can be used to adjust the joint angles to eliminate self-occlusion while keeping the end effector pose completely unchanged.

[0071] S27, In response to target self-blocking, within the process allowance, adjust the approach angle and contact position of the end effector to avoid the blocking part.

[0072] For situations where the target is self-obstructed, such as a workpiece hole being blocked by its own protrusion, the approach angle and contact position of the end effector can be adjusted within the allowable range of the process to avoid the obstructed area and select a visible and operable feature surface to complete the operation.

[0073] In this way, the self-occlusion of the robotic arm can be eliminated while keeping the end-effector's position unchanged, thereby reducing the position estimation error caused by self-occlusion without affecting the accuracy of the operation process.

[0074] In one embodiment, such as Figure 5 As shown, step S2 further includes: S28, based on the static environmental map, plans the initial global path and performs scrolling window optimization along the initial global path.

[0075] Based on static environmental maps, an improved RRT is adopted. The algorithm plans the initial global path, prioritizing paths with less obstruction and better line of sight.

[0076] S29, obtain the real-time occlusion index and the future occlusion index, and adjust the local trajectory and pose according to the real-time occlusion index and the future occlusion index.

[0077] Specifically, the local trajectory and pose can be dynamically adjusted based on the real-time occlusion index and the predicted future occlusion index, thereby achieving rolling window optimization along the initial global path. The future occlusion index can be determined by collecting historical occlusion indices and performing least squares fitting.

[0078] In some cases, if it is predicted that the occlusion index will exceed the threshold at a certain time in the future, attitude adjustment or trajectory offset actions can be initiated in advance to avoid the lag of correction after occlusion occurs.

[0079] In this way, by adjusting the local trajectory and pose based on the real-time occlusion index and the future occlusion index, the occlusion risk can be predicted in advance and the trajectory can be actively corrected. This enables a predictive occlusion trajectory planning mechanism, avoiding downtime and global replanning after occlusion occurs, thereby improving the overall operational efficiency in occlusion scenarios.

[0080] One embodiment of the present invention also provides a robot perception system in an occlusion scenario, including an occlusion index acquisition module, a target strategy allocation module, and a robotic arm execution module.

[0081] The occlusion index acquisition module is used to collect target data and environmental data. Based on the target data and environmental data, it obtains target visibility, target pose confidence, and line-of-sight occlusion. The target visibility, target pose confidence, and line-of-sight occlusion are weighted and fused to obtain the occlusion index used to quantify the degree of target occlusion.

[0082] The target strategy allocation module is used to allocate corresponding perception strategies, posture and joint optimization schemes, and trajectory planning modes to the target based on the occlusion index; the robotic arm execution module is used to control the adjustment of the joint configuration, end-effector posture, and motion trajectory of the robotic arm based on the allocated perception strategies, posture and joint optimization schemes, and trajectory planning modes.

[0083] The target strategy allocation module includes a conventional perception strategy unit, a pose iteration optimization unit, and a multi-view pose fusion unit.

[0084] The conventional perception strategy unit is used to execute the conventional precision perception strategy when the occlusion index is not greater than the preset first index threshold. It acquires the color image and depth image of the target through the RGB-D camera, obtains the 6D pose based on the depth image, and the robotic arm maintains the current configuration and posture and moves normally along the planned trajectory.

[0085] The pose iteration optimization unit is used to perform geometric completion of the occluded area and match it based on the visible area features when the occlusion index is greater than the preset first index threshold and not less than the preset second index threshold to obtain coarse-grained pose. At the same time, it adjusts the robot end pitch angle and yaw angle under the allowable pose error of the task to perform iterative optimization with the goal of maximizing the visible area of ​​the target.

[0086] The multi-view pose fusion unit is used to generate several candidate observation viewpoints in the reachable space around the target and estimate the occlusion index of each viewpoint when the occlusion index is greater than the preset second index threshold. It obtains the path length between the end point and the viewpoint, obtains the optimal viewpoint with the minimum comprehensive cost based on the occlusion index and the path length, and performs multi-view point cloud stitching and pose fusion based on the optimal viewpoint. The preset first index threshold is less than the preset second index threshold.

[0087] The multi-view pose fusion unit includes a point cloud data acquisition subunit, a point cloud data alignment subunit, a target point cloud generation subunit, a fused pose acquisition subunit, and a multi-pose fusion subunit.

[0088] The point cloud data acquisition subunit is used to plan the local motion path of the robotic arm based on the optimal viewpoint, control the robotic arm to move along the local motion path to the optimal viewpoint, and acquire the second visual point cloud data under the optimal viewpoint; the point cloud data alignment subunit is used to transform the second visual point cloud data to the same coordinate system based on the camera coordinate system of the initial viewpoint.

[0089] The target point cloud generation subunit is used to extract local geometric features of the visible point clouds from two different perspectives. It establishes initial corresponding point pairs through nearest neighbor search and eliminates mismatches. Then, it minimizes the point-to-surface distance error between the visible point clouds of the two visions to obtain the relative pose. Finally, it performs point cloud fusion to generate the target point cloud.

[0090] The fusion pose acquisition subunit is used to estimate the target pose under each viewpoint and obtain the corresponding occlusion index. Based on the corresponding occlusion index, the confidence weight of the target pose is obtained. The rotation component of the target pose is fused, and the position component of the target pose is fused based on the confidence weight to obtain the fused pose.

[0091] The multi-pose fusion subunit is used to output the final pose when the equivalent occlusion index or pose accuracy of the fused pose meets the requirements; otherwise, it plans the next optimal viewpoint based on the current fused pose and performs iterative optimization based on the optimal viewpoint.

[0092] As a preferred technical solution, the target strategy allocation module also includes an occlusion type recognition unit, an end-effector posture optimization unit, a global path planning unit, and a trajectory posture adjustment unit.

[0093] The occlusion type recognition unit is used to identify different occlusion types based on target data and environmental data. The occlusion types include static environmental occlusion, dynamic environmental occlusion, robotic arm self-occlusion, and target self-occlusion. End-effector posture optimization unit. This unit optimizes the end-effector posture for both static and dynamic environmental occlusion, using the Euler angles of the robotic arm's end effector as the decision variable, while meeting preset task constraints. The optimization objective is to minimize the occlusion index under the current posture. For self-occlusion, it minimizes the degree of self-occlusion by adjusting joint angles while maintaining the end-effector's operational pose. Finally, for target self-occlusion, it adjusts the approach angle and contact position of the end effector within process tolerances to avoid occlusion.

[0094] To address target self-occlusion, such as workpiece holes being blocked by their own protrusions, the approach angle and contact position of the end effector can be adjusted within the allowable process range to avoid the obstruction and select a visible and operable feature surface to complete the task. In this way, self-occlusion of the robotic arm can be eliminated while maintaining the end effector's working posture completely, thereby reducing the posture estimation error caused by self-occlusion without affecting the accuracy of the work process.

[0095] The global path planning unit is used to plan an initial global path based on the static environmental map and perform scrolling window optimization along the initial global path; the trajectory and pose adjustment unit is used to obtain the real-time occlusion index and the future occlusion index, and adjust the local trajectory and pose according to the real-time occlusion index and the future occlusion index.

[0096] In this way, by adjusting the local trajectory and pose based on the real-time occlusion index and the future occlusion index, the occlusion risk can be predicted in advance and the trajectory can be actively corrected. This enables a predictive occlusion trajectory planning mechanism, avoiding downtime and global replanning after occlusion occurs, thereby improving the overall operational efficiency in occlusion scenarios.

[0097] In summary, the robot perception system in the occlusion scenario integrates target visibility, target pose confidence, and line-of-sight occlusion to obtain an occlusion index for quantifying the degree of target occlusion. Based on the occlusion index, it assigns corresponding perception strategies, posture and joint optimization schemes, and trajectory planning modes to the target. This enables adaptive perception strategies driven by the degree of target occlusion and iterative optimization of the end-effector posture, thereby improving the robot's active control capability in occlusion scenarios.

[0098] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A robot perception method in occluded scenarios, characterized in that, Includes the following steps: S1. Collect target data and environmental data. Based on the target data and environmental data, obtain target visibility, target pose confidence and line-of-sight occlusion. Weightedly fuse target visibility, target pose confidence and line-of-sight occlusion to obtain an occlusion index used to quantify the degree of target occlusion. S2, based on the occlusion index, assigns the corresponding perception strategy, posture and joint optimization scheme and trajectory planning mode to the target; S3 controls the robotic arm to adjust joint configuration, end effector posture, and motion trajectory based on the assigned perception strategy, posture and joint optimization scheme, and trajectory planning mode.

2. The robot perception method in an occluded scene as described in claim 1, characterized in that, Step S2 specifically includes: S21, if the occlusion index is not greater than the preset first index threshold, execute the conventional precision perception strategy, acquire the color image and depth map of the target through the RGB-D camera, obtain the 6D pose based on the depth map, and the robotic arm maintains the current configuration and posture and moves normally along the planned trajectory. S22, if the occlusion index is greater than the preset first index threshold and not greater than the preset second index threshold, the occlusion area is geometrically completed and matched based on the visible area features to obtain coarse-grained pose. At the same time, the robot end pitch angle and yaw angle are adjusted under the allowable pose error of the task to maximize the visible area of ​​the target for iterative optimization. S23, if the occlusion index is greater than the preset second index threshold, generate several candidate observation viewpoints in the reachable space around the target and estimate the occlusion index of each viewpoint, obtain the path length between the end point and the viewpoint, obtain the optimal viewpoint with the minimum comprehensive cost based on the occlusion index and the path length, and perform multi-view point cloud stitching and pose fusion based on the optimal viewpoint. Among them, the preset first exponential threshold is less than the preset second exponential threshold.

3. The robot perception method in an occluded scene as described in claim 2, characterized in that, Step S23 specifically includes: S231, Based on the optimal viewpoint, plan the local motion path of the robotic arm, control the robotic arm to move along the local motion path to the optimal viewpoint and obtain the second visual point cloud data under the optimal viewpoint; S232, using the camera coordinate system of the initial viewpoint as a reference, transforms the second visual point cloud data to the same coordinate system; S233: Extract local geometric features of visible point clouds from two perspectives respectively, establish initial corresponding point pairs through nearest neighbor search and eliminate mismatches, then minimize the point-to-surface distance error between the visible point clouds of the two visions to obtain the relative pose, and finally perform point cloud fusion to generate the target point cloud. S234, estimate the target pose at each viewpoint and obtain the corresponding occlusion index, obtain the confidence weight of the target pose based on the corresponding occlusion index, fuse the rotation component of the target pose and fuse the position component of the target pose based on the confidence weight to obtain the fused pose; S235 If the equivalent occlusion index or pose accuracy of the fused pose meets the requirements, the final pose is output; otherwise, based on the current fused pose, the next optimal viewpoint is planned and iterative optimization is performed based on the optimal viewpoint.

4. The robot perception method in an occluded scene as described in claim 3, characterized in that, Step S2 also includes: S24, Identify different occlusion types based on target data and environmental data, including static environmental occlusion, dynamic environmental occlusion, robotic arm self-occlusion, and target self-occlusion; S25, for both static and dynamic environmental occlusion, the Euler angles of the robotic arm's end effector are used as decision variables. Under the premise of satisfying preset task constraints, the end effector's posture is optimized by minimizing the occlusion index under the current posture. S26, In response to self-occlusion of the robotic arm, the degree of self-occlusion is minimized by adjusting the joint angle while keeping the end-effector's position unchanged; S27, In response to target self-blocking, within the process allowance, adjust the approach angle and contact position of the end effector to avoid the blocking part.

5. A robot perception method in an occluded scene as described in claim 4, characterized in that, Step S2 also includes: S28, Based on the static environmental map, plan the initial global path and perform scrolling window optimization along the initial global path; S29, obtain the real-time occlusion index and the future occlusion index, and adjust the local trajectory and pose according to the real-time occlusion index and the future occlusion index.

6. A robot perception system for occluded scenarios, characterized in that, include: The occlusion index acquisition module is used to collect target data and environmental data, and obtain target visibility, target pose confidence and line-of-sight occlusion based on the target data and environmental data. The target visibility, target pose confidence and line-of-sight occlusion are weighted and fused to obtain the occlusion index used to quantify the degree of occlusion of the target. The target strategy allocation module is used to allocate corresponding perception strategies, pose and joint optimization schemes, and trajectory planning modes to the target based on the occlusion index. The robotic arm execution module is used to control the adjustment of the joint configuration, end effector posture, and motion trajectory of the robotic arm according to the assigned perception strategy, posture and joint optimization scheme, and trajectory planning mode.

7. A robot perception system in an occluded scenario as described in claim 6, characterized in that, The target strategy allocation module includes: The conventional perception strategy unit is used to execute the conventional precision perception strategy when the occlusion index is not greater than the preset first index threshold. It acquires the color image and depth image of the target through the RGB-D camera, obtains the 6D pose based on the depth image, and the robotic arm maintains the current configuration and posture and moves normally along the planned trajectory. The pose iteration optimization unit is used to perform geometric completion of the occlusion area and match based on the visible area features when the occlusion index is greater than the preset first index threshold and not less than the preset second index threshold to obtain coarse-grained pose. At the same time, it adjusts the robot end pitch angle and yaw angle under the allowable pose error of the task to perform iterative optimization with the goal of maximizing the visible area of ​​the target. The multi-view pose fusion unit is used to generate several candidate observation viewpoints in the reachable space around the target when the occlusion index is greater than the preset second index threshold, estimate the occlusion index of each viewpoint, obtain the path length between the end point and the viewpoint, obtain the optimal viewpoint with the minimum comprehensive cost based on the occlusion index and the path length, and perform multi-view point cloud stitching and pose fusion based on the optimal viewpoint. Among them, the preset first exponential threshold is less than the preset second exponential threshold.

8. A robot perception system in an occluded scenario as described in claim 7, characterized in that, The multi-view pose fusion unit includes: The point cloud data acquisition subunit is used to plan the local motion path of the robotic arm based on the optimal viewpoint, control the robotic arm to move along the local motion path to the optimal viewpoint, and acquire the second visual point cloud data under the optimal viewpoint. The point cloud data alignment sub-unit is used to transform the second-view point cloud data to the same coordinate system, based on the camera coordinate system of the initial viewpoint. The target point cloud generation subunit is used to extract local geometric features of the visible point clouds from two perspectives respectively. It establishes initial corresponding point pairs through nearest neighbor search and removes mismatches. Then, it minimizes the point-to-surface distance error between the visible point clouds of the two visions to obtain the relative pose. Finally, it performs point cloud fusion to generate the target point cloud. The pose fusion acquisition subunit is used to estimate the target pose under each viewpoint and obtain the corresponding occlusion index. Based on the corresponding occlusion index, the confidence weight of the target pose is obtained. The rotation component of the target pose is fused and the position component of the target pose is fused according to the confidence weight to obtain the fused pose. The multi-pose fusion subunit is used to output the final pose when the equivalent occlusion index or pose accuracy of the fused pose meets the requirements; otherwise, it plans the next optimal viewpoint based on the current fused pose and performs iterative optimization based on the optimal viewpoint.

9. A robot perception system in an occluded scenario as described in claim 8, characterized in that, The target strategy allocation module also includes: The occlusion type identification unit is used to identify different occlusion types based on target data and environmental data. The occlusion types include static environmental occlusion, dynamic environmental occlusion, robotic arm self-occlusion, and target self-occlusion. The end-effector posture optimization unit is used to optimize the end-effector posture for both static and dynamic environmental occlusion. Using the Euler angles of the robotic arm end effector as the decision variable, it optimizes the end-effector posture by minimizing the occlusion index under the current posture, while meeting preset task constraints. For self-occlusion of the robotic arm, it minimizes the degree of self-occlusion by adjusting the joint angles while keeping the end-effector operation posture unchanged. For target self-occlusion, it adjusts the approach angle and contact position of the end effector within the process allowance to avoid the occluded parts.

10. A robot perception system in an occluded scenario as described in claim 9, characterized in that, The target strategy allocation module also includes: The global path planning unit is used to plan an initial global path based on a static environmental map and perform rolling window optimization along the initial global path. The trajectory and pose adjustment unit is used to acquire the real-time occlusion index and the future occlusion index, and adjust the local trajectory and pose according to the real-time occlusion index and the future occlusion index.

Images