Tail end shielding sensing and visual angle switching method and system for teleoperation hot-line work robot

Abstract

The invention discloses a tail end shielding sensing and visual angle switching method and system for a teleoperation hot-line work robot. The method comprises the steps that a mechanical arm three-dimensional geometric model is loaded, and hand-eye calibration is executed for initialization; the shielding degree and the center position are calculated in real time based on geometric projection and depth information comparison; when the shielding degree exceeds a shielding degree threshold value or the end effector deviates from a preset image center area at the central position of an image plane and exceeds a centroid deviation threshold value, determining an optimal pan-tilt angle through a local view angle search and optimization algorithm; adopting smooth path planning and proportional-integral-differential control to drive the holder to rotate; and the visual angle is adjusted and optimized through motion prediction and a dynamic threshold value. The method does not need to depend on external image features, adapts to a complex environment, is high in automation degree, smooth in visual angle switching and excellent in real-time performance, provides reliable visual feedback for high-precision hot-line work, and remarkably improves operation efficiency and safety.

Description

Technical Field

[0001] This application belongs to the field of robot vision and teleoperation technology, specifically relating to a method and system for end-effector occlusion perception and viewpoint switching of a teleoperated live-line working robot, which is applicable to the visual feedback optimization of the end effector of the robotic arm in power system maintenance. Background Technology

[0002] Live-line working is a crucial aspect of power system maintenance, involving high-risk tasks such as connecting high-voltage power lines, overhauling equipment, and troubleshooting, directly impacting the stability and safety of the power grid. In recent years, with advancements in robotics technology, teleoperated robotic arms have gradually become the mainstream tool for live-line work. During teleoperation, visual feedback is key for the operator to perceive and control the robotic arm's trajectory, especially the accurate perception of the end effector's position, posture, and operational status, which directly determines the success of the task. However, the multi-joint structure and complex geometry of the robotic arm often cause the arm itself to obscure the end effector during dynamic movement, preventing the camera from fully capturing the end effector image. This obscuring problem reduces the operator's accuracy in perceiving the end effector's position, potentially leading to misoperation or even equipment damage or safety accidents.

[0003] To address the issue of end-point occlusion, existing technologies typically first achieve occlusion detection and then eliminate occlusion by adjusting the viewing angle.

[0004] In terms of occlusion perception, existing technologies mostly rely on image segmentation or edge detection techniques to determine occlusion by identifying the overlapping area between the end effector and the robotic arm body. However, existing technologies are highly sensitive to environmental factors such as changes in lighting, background noise, and interference from power lines. In complex scenarios of outdoor live-line work, dynamic environments and varying backgrounds significantly weaken the robustness of algorithms, making it difficult to accurately distinguish occlusion areas with complex geometries.

[0005] In terms of eliminating occlusion, existing technologies mainly attempt the following solutions: First, multi-camera fusion is used to capture the end effector from different angles to reduce the probability of occlusion. However, additional cameras increase hardware costs and system complexity. The computational overhead of view synchronization and data fusion is large, and the processing latency is difficult to meet the real-time requirements of highly dynamic operations. Second, the camera is mounted on a two-degree-of-freedom gimbal to dynamically avoid occlusion by adjusting the posture. However, the view switching based on heuristic rules is highly dependent on the operator's experience, has a low degree of automation, and the switching process may not be smooth enough, affecting the operator's continuous perception.

[0006] In summary, existing technologies have the following shortcomings in live-line working scenarios: image processing methods are sensitive to environmental changes and lack robustness, making them difficult to adapt to complex outdoor scenarios; multi-camera fusion schemes are computationally complex and cannot meet the real-time requirements of high-dynamic operations; manual or semi-automatic perspective adjustment has a low degree of automation and is difficult to adapt to the dynamic pose changes of the robotic arm; perspective switching lacks smooth control, which may interfere with the continuity of the operator's visual feedback.

[0007] Therefore, there is an urgent need for an efficient, robust, real-time and automated occlusion detection and view adjustment method that can dynamically eliminate the occlusion of the end effector in complex live-line working environments, ensuring that it is clearly visible in the camera image, while taking into account real-time performance, stability and smoothness, and providing reliable visual feedback support for high-precision teleoperation. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this application provides a method and system for end-effector occlusion perception and viewpoint switching in a teleoperated live-line working robot. By loading a three-dimensional geometric model of the robotic arm and combining it with hand-eye calibration, the occlusion status of the end effector is perceived in real time, solving the problem of visual feedback failure caused by the robotic arm itself occluding the end effector, thus ensuring continuous and clear observation of the working tool itself. Furthermore, local viewpoint search and gradient optimization are employed to dynamically adjust the gimbal angle. Through path planning and proportional-integral-derivative control, the optimal observation viewpoint is actively searched and planned, ensuring the smoothness and real-time performance of viewpoint switching. This improves the unobstructed, reliable, and smooth observation of the end effector in complex environments.

[0009] The technical solution adopted in this application is as follows.

[0010] The first aspect of this application provides a method for end-effector obstruction detection and viewpoint switching of a remotely operated live-line working robot, the method comprising the following steps:

[0011] S1: Initialization phase, load the 3D geometric model of the robotic arm of the teleoperated live-line working robot, perform hand-eye calibration to determine the relative pose of the camera and the robotic arm, and configure the camera parameters;

[0012] S2: Based on camera parameters and the relative pose of the camera and the robotic arm, the end effector and body of the three-dimensional geometric model are projected onto the image plane to generate corresponding region masks and depth information. The occlusion area is identified according to the region mask and the depth information, and the occlusion degree of the end effector and its center position on the image plane are calculated in real time. The occlusion degree is the proportion of the occluded area to the total area of ​​the end effector.

[0013] S3: Determine whether the optimization triggering condition is met based on the degree of occlusion and the center position of the end effector in the image plane. If the optimization triggering condition is met, determine the optimal gimbal angle based on the degree of occlusion and the center position of the end effector to optimize the gimbal viewing angle and obtain the optimal gimbal angle. The optimization triggering condition is: the degree of occlusion exceeds the occlusion degree threshold or the center position of the end effector in the image plane deviates from the preset image center area by more than the centroid deviation threshold.

[0014] S4: Based on the optimal gimbal angle, plan the view switching path, and drive the gimbal to rotate to the optimal gimbal angle according to the view switching path.

[0015] Optionally, in S3, determining the optimal gimbal angle based on the degree of occlusion and the center position of the end effector includes:

[0016] Set the search range around the current gimbal angle, and generate multiple candidate viewpoints within the search range according to the preset step size and preset gimbal angle constraint range;

[0017] For each candidate viewpoint, calculate its corresponding occlusion degree and centroid deviation, and perform weighted fusion of the occlusion degree and centroid deviation to form a comprehensive evaluation index value, and evaluate each candidate viewpoint;

[0018] The candidate viewpoint corresponding to the minimum comprehensive evaluation index value is selected as the optimal gimbal angle.

[0019] Optionally, a strategy combining grid search and gradient optimization is adopted to select the candidate viewpoint corresponding to the minimum comprehensive evaluation index value as the optimal gimbal angle. The grid search strategy determines whether the minimum comprehensive evaluation index value is less than the score threshold. If so, the candidate viewpoint corresponding to the minimum comprehensive evaluation index value is determined as the optimal gimbal angle. The gradient optimization strategy calculates the numerical gradient of the minimum comprehensive evaluation index value if the minimum comprehensive evaluation index value is greater than or equal to the score threshold. The numerical gradient is multiplied by the learning rate and iterated until the iteration condition or stopping condition is met. The angle corresponding to the optimal comprehensive evaluation index value obtained by the iteration is determined as the optimal gimbal angle.

[0020] Optionally, the occlusion area can be identified based on the region mask and depth information using the following formula, and the degree of occlusion of the end effector can be calculated in real time:

[0021]

[0022] M overlap =M ee ∩M body

[0023] In the formula, S curTo indicate the degree of occlusion, 1 represents complete occlusion of the end effector or the end effector is completely removed from the field of view, and N represents the degree of occlusion. ee M represents the total number of pixels at the end. ee M represents the end effector mask. body Represents the ontology mask, N occluded This represents the total number of end effector pixels occluded by the robotic arm body in the overlapping region, where II(·) is the indicator function, and M... overlap This represents the overlapping region mask, (u,v) represents the pixel coordinates on the image plane, and D... ee This indicates that a depth map of the end effector pixels is generated based on the depth information from the 3D spatial points corresponding to the end effector in the 3D geometric model to the camera. D body A depth map of the body pixels is generated by representing the depth information from the 3D spatial points corresponding to the body part of the 3D geometric model to the camera.

[0024] For pixel coordinates (u,v)∈M on the image plane overlap If D ee (u,v)>D body (u,v) indicates that the pixel is occluded by the body, thus identifying the occluded area.

[0025] Optionally, in S4, based on the optimized gimbal angle, the planned viewpoint switching path includes:

[0026] Plan the linear interpolation path from the current angle to the optimal gimbal angle using the following formula:

[0027]

[0028] In the formula, (θ p (t),θ t (t) represents the optimal gimbal angle, (θ) p_cur ,θ t_cur ) represents the current angle, represents the interpolated angle, T represents the transition time, which is the time taken to complete the viewpoint switching path, and t represents the elapsed time since the gimbal started rotating.

[0029] Optionally, the method further includes:

[0030] S5: Feedback optimization phase, periodically verify the effect of view optimization, predict future pose based on the movement trend of the robotic arm and optimize the gimbal view in advance according to the predicted future pose, dynamically adjust the occlusion degree threshold and centroid deviation threshold to adapt to different task requirements.

[0031] Optionally, in S5, predicting the future pose based on the robotic arm's motion trend and optimizing the gimbal's viewing angle in advance based on the predicted future pose includes:

[0032] The rate of change of the joint angles of the robotic arm is calculated, and the pose change of the end effector is predicted based on the rate of change using the Jacobian matrix to obtain the predicted pose; wherein, the Jacobian matrix is ​​used to represent the linear relationship between the velocity of the end effector and the velocity of each joint of the robotic arm;

[0033] The degree of occlusion and the comprehensive evaluation index value are calculated based on the predicted pose. If the difference between the comprehensive evaluation index value of the predicted pose and the comprehensive evaluation index value of the current pose is less than a preset threshold, the gimbal is driven to perform optimization in advance according to the predicted pose.

[0034] Optionally, S3 further includes: setting a cooling time, wherein the cooling time is the interval between two adjacent gimbal view optimizations.

[0035] Optionally, S3 also includes setting a gimbal angle change threshold. When the gimbal angle change rate is less than the gimbal angle change threshold, no viewpoint switching is performed; otherwise, the gimbal viewpoint is switched according to the optimal gimbal angle.

[0036] A second aspect of this application provides a system for end-effector obstruction detection and viewpoint switching of a remotely operated live-line working robot, the system comprising:

[0037] The initialization module is used to load the three-dimensional geometric model of the robotic arm of the teleoperated live-line working robot and configure the camera parameters during the initialization phase.

[0038] The occlusion perception module is used to project the end effector and body of the three-dimensional geometric model onto the image plane based on camera parameters, generate corresponding region masks and depth information, identify occlusion regions based on the region masks and depth information, and calculate the occlusion degree of the end effector and its center position on the image plane in real time; the occlusion degree is the proportion of the occluded region to the total end effector region;

[0039] The viewpoint optimization module is used to determine whether the optimization triggering condition is met based on the degree of occlusion and the center position of the end effector in the image plane. If the optimization triggering condition is met, the optimal gimbal angle is determined based on the degree of occlusion and the center position of the end effector to optimize the gimbal viewpoint and obtain the optimal gimbal angle. The optimization triggering condition is: the degree of occlusion exceeds the occlusion degree threshold or the center position of the end effector in the image plane deviates from the preset image center area by more than the centroid deviation threshold.

[0040] The gimbal control module is used to plan a viewpoint switching path based on the optimal gimbal angle, and drive the gimbal to rotate to the optimal gimbal angle according to the viewpoint switching path.

[0041] Optionally, the system further includes:

[0042] The feedback optimization module is used to periodically verify the effect of the view optimization. It predicts the future pose based on the movement trend of the robotic arm and optimizes the gimbal view in advance according to the predicted future pose. It dynamically adjusts the occlusion degree threshold and the centroid deviation threshold to adapt to different task requirements.

[0043] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when loaded onto the processor, implements the aforementioned method for end-effector occlusion sensing and viewpoint switching of a teleoperated live-line working robot.

[0044] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for end-effector occlusion detection and viewpoint switching of a teleoperated live-line working robot.

[0045] Compared with the prior art, the beneficial effects of this application include at least the following:

[0046] This application employs a projection and depth comparison method based on a three-dimensional geometric model. Compared to existing occlusion perception methods based on image segmentation or edge detection, which are sensitive to changes in lighting, background noise, and complex geometric shapes and are difficult to work stably in complex and ever-changing outdoor live-line working scenarios, this application directly calculates the degree of occlusion using the joint state of the robotic arm and the camera pose. It does not rely on image features, has strong environmental adaptability, overcomes the limitations of image processing methods, and significantly improves robustness to environmental interference.

[0047] This application achieves automatic tracking and viewing angle adjustment of the PTZ by real-time occlusion rate calculation and online viewing angle optimization. It can dynamically eliminate occlusion without operator intervention, ensuring that the end effector is always clearly visible. It has a high degree of automation, reduces the operator's burden, and improves the convenience and consistency of operation.

[0048] This application ensures smooth gimbal rotation without jitter through smooth path planning and proportional-integral-derivative control. Compared with existing dynamic view adjustment methods, where frequent or abrupt view switching may interfere with the operator's visual perception and affect task continuity, this application provides smooth view switching and improves the user experience. At the same time, this application avoids unnecessary view changes by setting jitter thresholds and optimizing caching mechanisms, providing the operator with smooth and stable visual feedback. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0050] Figure 1 This is a schematic flowchart of a method for end-effector occlusion detection and viewpoint switching of a live-line working robot provided in an embodiment of this application.

[0051] Figure 2 This is a schematic diagram of an occlusion rate calculation process provided in an embodiment of this application;

[0052] Figure 3 This is a flowchart of a perspective optimization process provided in an embodiment of this application;

[0053] Figure 4 This is a schematic diagram of gimbal control provided in an embodiment of this application;

[0054] Figure 5 This is a schematic diagram of a software and hardware platform and data interaction provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. The embodiments described in this application are merely some embodiments of this application, and not all embodiments. Based on the spirit of this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of this application.

[0056] Combination Figure 1 As shown, Embodiment 1 of this application provides a method for end-effector occlusion detection and viewpoint switching of a live-line working robot. For example... Figure 1 As shown, the method flow consists of five main steps: initialization, real-time occlusion rate calculation, online viewpoint optimization, gimbal control, and feedback optimization. The specific implementation methods of each step are described below.

[0057] S1: Initialization phase, load the 3D geometric model of the robotic arm, perform hand-eye calibration to determine the relative pose of the camera and the robotic arm, and configure camera parameters, occlusion threshold and gimbal control parameters.

[0058] The initialization phase prepares the necessary basic data and parameters for system operation, and the specific implementation is as follows:

[0059] S1.1: Load the 3D geometric model of the robotic arm containing detailed geometric information of the base, joints, links and end effector, and calculate the initial joint state through forward kinematics to determine the spatial pose of each component.

[0060] Specifically, the geometric model of the robotic arm is loaded using URDF (Unified Robot Description Format), which includes 3D mesh information of the base, joints, links, and end effector. The 3D mesh information file format is STL. The 3D geometric model of the robotic arm describes the geometry and kinematic relationships of each component.

[0061] The initial angles of the robotic arm joints are obtained as q = [q1, q2, ..., q6], and the initial pose T of the end effector is calculated based on the initial angles of the robotic arm joints using a forward kinematics algorithm. ee :

[0062]

[0063] Where q1 represents the initial angle of the first joint of the robotic arm, q i R represents the initial angle of the i-th joint of the robotic arm. ee Let t be the rotation matrix of the robotic arm joints. ee Let f be the translation vector of the robotic arm joint. FK () is the forward kinematics algorithm function, used to calculate the initial pose of the end effector based on the initial angle of the robotic arm joints.

[0064] It should be noted that by establishing the robot's own digital model and kinematic description, the system can know "what the robot looks like" and "how its arms move".

[0065] S1.2: Perform hand-eye calibration, and calculate the initial pose of the camera relative to the robot arm base based on the initial pose of the end effector in S1.1 using the calibration algorithm.

[0066] Using the Tsai-Lenz calibration algorithm, based on the relative pose of the camera and the robotic arm base acquired via the calibration board in S1.1, the initial transformation matrix is ​​obtained:

[0067]

[0068] In the formula, T basec0 R represents the spatial transformation relationship from the robot's base coordinate system to the camera's initial c0 coordinate system. c0 Let be the initial rotation matrix, representing the initial pose of the camera relative to the robot base, t c0 Let be the initial translation vector, representing the initial position of the camera relative to the robot base.

[0069] It should be noted that hand-eye calibration establishes the spatial relationship between the robot and the camera, so that the robot model can be correctly projected into the camera image.

[0070] Understandably, the hand-eye calibration process requires driving the robot to move to multiple different, precisely known poses. These "precisely known" pose data are obtained by calculating the robot's pose data based on the real-time reading of the robot's joint angles using the positive kinematics model established in S1.1.

[0071] In this embodiment, the calibration accuracy requirement can be set to within 1 mm, which can ensure the accuracy of subsequent projection calculations.

[0072] It should be noted that calibration accuracy is typically assessed by comparing the error between actual measured values ​​and theoretically calculated values. Specifically, calibration is performed from multiple perspectives using a calibration board, recording the image coordinates acquired by the camera and the position coordinates of the robotic arm's end effector, followed by error analysis. Accuracy is usually calculated using the least squares method to determine the reprojection error, which is the distance between the actual image point and the projected point. Controlling the calibration accuracy to within 1 millimeter means we expect the final reprojection error to be no more than 1 millimeter, ensuring the accuracy of subsequent projection calculations and the overall system precision.

[0073] S1.3: Configure camera parameters, occlusion threshold, and gimbal control parameters.

[0074] Configure camera parameters, including image resolution, focal length, and center position; set occlusion thresholds, angle of view optimization weight factors, and dynamic parameters for gimbal control; and provide a unified reference framework for system operation.

[0075] S1.3 specifically includes:

[0076] S1.3.1: Configure camera parameters.

[0077] Specifically, configuring camera parameters includes configuring the following camera intrinsic parameter matrix:

[0078]

[0079] Among them, f x f represents the pixel value of the focal length in the x-direction of the image. y This represents the pixel value of the focal length in the y-direction of the image, (c x ,c y ) represents the pixel coordinates of the image center.

[0080] In this embodiment of the disclosure, f is set x 600 pixels, f yThe pixel value is 700 pixels, and the pixel coordinates of the image center are (c x ,c y = (540, 360), the image resolution is configured as 1080×720 pixels.

[0081] The image center region is configured by defining a circle with the pixel coordinates of the image center as the origin and a radius of r = 72 pixels as the image center region. The image center region can be set to occupy approximately 10% of the image's width and height; however, the specific method for setting the image center region is not limited in this embodiment.

[0082] S1.3.2: Set the occlusion threshold and the weighting factor for viewpoint optimization.

[0083] The weighting factors for viewpoint optimization are parameters used to measure the importance of different factors during the optimization process. This application defines two main optimization objectives: occlusion degree and end effector centering error. Therefore, the weighting factors determine the relative importance of these two factors during optimization. The viewpoint optimization weighting factors include an occlusion degree weight w1 and a centroid deviation weight w2. The occlusion degree weight w1 measures the degree to which the end effector is occluded; a higher w1 indicates greater emphasis on resolving the occlusion problem during optimization. The centroid deviation weight w2 measures the degree of deviation of the end effector from the image, i.e., whether the end effector is centered in the image.

[0084] In this embodiment of the disclosure, an occlusion degree threshold s is set. th The occlusion rate weight w1 can be 0.7, and the centering error weight w2 can be 0.3.

[0085] Understandably, by adjusting these weighting factors, it is possible to flexibly control the optimization algorithm to prioritize reducing occlusion or ensuring the end effector is centered in different scenarios.

[0086] S1.3.3: Set the dynamic parameters for gimbal control.

[0087] In this embodiment, the gimbal used is a two-degree-of-freedom servo system that supports horizontal rotation (Pan, θ). p (∈[-180°, 180°]) and pitch rotation (Tilt, θ) t The angle accuracy is 0.1°, ∈[-90°, 90°]. The proportional-integral-derivative (PID) controller parameter is k. p =1.0, k i =0.1, k d =0.05, the transition time is set to 0.5 seconds.

[0088] S2: Real-time acquisition of the current joint angle and gimbal angle of the robotic arm, projection of the end effector and body of the three-dimensional geometric model onto the image plane, generation of corresponding region mask and depth information, identification of occlusion areas based on region mask and depth information, and real-time calculation of the occlusion degree of the end effector and its center position in the image.

[0089] It should be noted that the image plane is a virtual two-dimensional plane. Through mathematical transformations of perspective projection, the robotic arm model in three-dimensional space is drawn onto this two-dimensional plane. In this embodiment, the occlusion rate is calculated by comparing the depth information of each component after projection on this image plane. The image plane can be the two-dimensional plane containing the image captured by the camera on the gimbal.

[0090] Calculating the real-time occlusion level can dynamically sense the occlusion status of the end effector, such as... Figure 2 As shown, S2 specifically includes:

[0091] S2.1: Real-time acquisition of the current joint angles of the robotic arm and the current angle of the gimbal, and calculation of the spatial pose of the end effector and each link using forward kinematics.

[0092] Specifically, the current joint angle q = [q1, q2, ..., q6] is obtained through a joint encoder, with an accuracy of 0.01°. The spatial pose of the end effector and each link is calculated using a forward kinematics algorithm.

[0093] T i =f FK (q,i),i∈{base,link1,...,link5,ee} (4)

[0094] In the formula, T i Let f be the pose matrix, representing the position and orientation of the i-th component of the robotic arm relative to the coordinate system of the robotic arm base. The component includes, but is not limited to, the base, links, or end effectors. FK () represents the positive kinematics algorithm function, f FK (q,i) represents the calculation of the pose T of the component based on the input robot arm joint angle vector q and the specified component index i. i q represents the current joint angle of the robotic arm, and i represents the index of the robotic arm component, with a value range of {"base","link1",...,"link5","ee"}, representing the base, each link, and the end effector, respectively. T i Vertex p of the 3D model defined in the local coordinate system of each link link_i The coordinates are uniformly transformed to the common coordinate system of the robotic arm base, that is, p link_i Multiply by T i Get p w .

[0095] S2.1: Combine the hand-eye calibration results and the gimbal angle to calculate the current pose of the camera relative to the robotic arm base, and dynamically update the pose of the camera relative to the robotic arm base.

[0096] Specifically, the current angle (θ) of the gimbal is obtained through the encoder of the gimbal servo motor. p_cur ,θ t_cur The accuracy of the gimbal angle is 0.1°, where θ p_cur θ represents the current horizontal rotation angle of the gimbal. t_cur This indicates the current pitch and rotation angle of the gimbal.

[0097] Update the camera's pose relative to the robotic arm base using the following formula:

[0098]

[0099] In the formula, R represents the current pose transformation matrix of the camera coordinate system relative to the robot arm base coordinate system. This matrix changes in real time as the gimbal rotates. c_cur t represents the rotation part of the matrix, i.e., the current rotation matrix of the camera. c0 This represents the initial translation vector of the camera obtained from hand-eye calibration, i.e., the positional relationship between the camera and the robotic arm base when the gimbal angle is zero. This vector remains unchanged during the rotation of the gimbal and serves as the reference for all subsequent real-time pose calculations.

[0100] The rotation matrix is:

[0101] R c_cur =R c0 ·R p (θ p_cur )·R t (θ t_cur (6)

[0102] In the formula, R c0 The initial rotation matrix represents the initial attitude of the camera relative to the robot base, specifically the rotation matrix at the initial position or zero position, i.e., when both the pan and tilt angles are 0. This initial rotation matrix is ​​pre-measured and calculated in step S1 using hand-eye calibration; it defines the fixed rotational relationship between the camera mounting bracket, i.e., the pan-tilt base, and the robotic arm base. Throughout the operation, R... c0 It is a constant, R p (θ p_cur ) indicates the current horizontal rotation θ of the gimbal. p_cur The resulting rotation matrix, R t (θ t_cur ) indicates the current pitch and rotation θ of the gimbal. t_cur The resulting rotation matrix.

[0103] Horizontal rotation matrix R p The following expression:

[0104]

[0105] In the formula, θ p This represents the horizontal rotation angle variable of the gimbal.

[0106] Pitch and rotation matrix R t The following expression:

[0107]

[0108] In the formula, θ t This represents the pitch and rotation angle variable of the gimbal.

[0109] S2.3: Project the end effector and body of the robotic arm's 3D geometric model onto the image plane to generate the corresponding region mask and depth information.

[0110] Specifically, the end effector and body mesh of the URDF model are projected onto the image plane, wherein the body part includes a link and a base.

[0111] It is understood that the depth information is the distance from a point on the 3D geometric model to the camera, or the distance along the camera's viewing direction.

[0112] The projection operation applies to all parts of the 3D model, including the end effector and the body. The same projection transformation is performed on every 3D vertex of the model, mapping it onto a 2D image plane. The final result is two sets of pixel coordinates: one set consists of the pixel coordinates of all 3D points belonging to the end effector after projection; the other set consists of the pixel coordinates of all 3D points belonging to the robotic arm body after projection. Both sets of pixels reside in the same image plane coordinate system.

[0113] Specifically, when a 3D geometric model is virtually rendered onto the image plane, depth information is generated using the Z-buffer algorithm. Simply put, when each point of the model is projected onto a pixel in the image, the algorithm calculates and records the depth information of that point from the camera. If multiple points are projected onto the same pixel, the Z-buffer will only retain the depth information of the point closest to the camera.

[0114] Specifically, for a 3D vertex of any link in a 3D model, the pose matrix T is first used. i Multiplying the result by the 3D vertex and transforming it to the robot arm base coordinate system yields p. w Then for p w Perform subsequent camera projection calculations.

[0115] Specifically, the three-dimensional points p of the three-dimensional geometric model of the robotic arm w =[x w ,y w ,z w ,1] T Points transformed to camera coordinate system:

[0116]

[0117] In the formula, p w p represents the coordinates of a grid point on the URDF model in the coordinate system of the robot arm base. c p w The coordinates in the camera coordinate system, x w p w The x-coordinate and y-coordinate in the coordinate system of the robot arm base. w p w In the coordinate system of the robot arm base, the y-coordinate, z-coordinate w p w The z-coordinate and x-coordinate in the coordinate system of the robot arm base. c p w The x-coordinate and y-coordinate in the camera coordinate system c p w In the camera coordinate system, the y-coordinate, z-coordinate c p w The z-coordinate in the camera coordinate system.

[0118] Specifically, the end effector and body of the robotic arm's 3D geometric model are projected from perspective onto the image plane using the camera intrinsic parameter matrix:

[0119]

[0120] In the formula, u and v represent points p in three-dimensional space. c The two-dimensional pixel coordinates on the image plane after projection through the camera's intrinsic parameters K, where K represents the camera's intrinsic parameter matrix, which contains information such as the camera's focal length and image center, and f represents the camera's focal length.

[0121] It should be noted that the two-dimensional pixel coordinates (u,v) are obtained through (10), and z is ignored in this process. c ≤0 points.

[0122] The rasterization algorithm is used to generate corresponding masks from the two sets of pixel coordinates, and the masks include an end effector mask M. ee and the body mask M body Among them, for the end effector mask M ee A pixel value of 1 indicates that it belongs to the end region, meaning that pixel (u,v) is not covered by the end effector; body mask Mbody A pixel value of 1 indicates that it belongs to the link or base area, and that pixel (u,v) is covered by the robotic arm body.

[0123] S2.4: By comparing the depth information and region mask of the end effector and the body, the occlusion area is identified and the degree of occlusion is calculated. The degree of occlusion is the proportion of the occluded area to the total area of ​​the end effector.

[0124] Calculate the distance from the camera to the 3D spatial point corresponding to the end effector of the 3D geometric model, and generate a depth map D of the end effector pixels. ee Calculate the distance from the camera to the corresponding 3D spatial points of the body part of the 3D geometric model to generate a depth map D of the body pixels. body .

[0125] During the generation process, for multiple 3D points projected onto the same pixel, the Z-buffer algorithm is used to retain the depth value of the point closest to the camera. For example, to generate a depth map D... ee First, an infinite depth value is initialized for each pixel in the image. When a triangular facet of the model is rasterized and covers a pixel, the depth of that facet at that pixel location is calculated, which is the distance to the camera. If this newly calculated depth value is less than the depth value already stored for the current pixel, the pixel's depth value is updated to this smaller new value. After traversing all facets, each pixel in the depth map stores the depth of the point closest to the camera at that location.

[0126] For the end effector mask M ee With the body mask M body Find the union of the sets to obtain the mask of the overlapping regions:

[0127] M overlap =M ee ∩M body (11)

[0128] In the formula, M ee M represents the end effector mask. body Represents the ontology mask; for pixel coordinates (u,v)∈M in a two-dimensional image overlap If D ee (u,v)>D body If (u,v), then the pixel is occluded by the body.

[0129] Calculate the number of pixels whose end effector is occluded:

[0130]

[0131] In the formula, N occludedThis represents the total number of end effector pixels occluded by the robotic arm body in the overlapping region; II(·) is an indicator function, with a value of 1 when the condition is met; (u,v) represents the pixel coordinates on the image plane; M overlap This represents the mask for overlapping regions.

[0132] Calculate the total number of pixels N at the end. ee :

[0133]

[0134] In the formula, N ee M represents the total number of pixels at the end. ee This represents the end effector mask, and (u,v) represents the pixel coordinates on the image plane.

[0135] Calculate the degree of occlusion using the following formula:

[0136]

[0137] In the formula, S cur To indicate the degree of occlusion, 1 represents complete occlusion of the end effector or the end effector is completely removed from the field of view, and N represents the degree of occlusion. ee This indicates the total number of pixels at the end. It ensures that the system's automated viewpoint optimization function can be reliably triggered to correct for lost visual feedback.

[0138] S2.5: Calculate the center position of the end effector in the image plane.

[0139] Specifically, the centroid coordinates (u) of the end effector in the image plane are calculated using the following formula. cur ,v cur ):

[0140]

[0141] In the formula, This represents the sum of the horizontal u-coordinates of all pixels within the end effector region. This represents the sum of the vertical v-coordinates of all pixels within the end effector region.

[0142] If N ee =0 indicates that the end is completely invisible. The centroid coordinates are set to the image center coordinates. (u) cur ,v cur ) = (540, 360).

[0143] Preferably, but not limited to, a graphics processor is used for the projection and depth map generation of S2.3. Specifically, a graphics processor, such as, but not limited to, an NVIDIA RTX 3060, is used to accelerate mesh projection and depth map generation, and an OpenGL rendering pipeline is employed, resulting in a single computation time of approximately 20 milliseconds. Furthermore, the initial projection resolution is set to 540×360, and the verification phase resolution is set to 1080×720.

[0144] Furthermore, by using multi-threading to separate joint state acquisition from occlusion rate calculation, the two processes can be made to operate without blocking each other.

[0145] Furthermore, the transformation results of the mesh vertices in the robot arm base coordinate system are cached to reduce redundant calculations.

[0146] In this embodiment, a graphics processor is used to accelerate the projection process, and multi-threaded processing and dynamic resolution adjustment are combined to ensure real-time performance and improve computational efficiency.

[0147] S3: Determine whether the optimization triggering condition is met based on the degree of occlusion and the center position of the end effector. If the optimization triggering condition is met, optimize the viewing angle by determining the optimal gimbal angle based on the degree of occlusion and the center position of the end effector. The optimization triggering condition is: the degree of occlusion exceeds the occlusion degree threshold or the center position of the end effector deviates from the center area of ​​the image by more than the centroid deviation threshold.

[0148] When the occlusion rate or centroid deviation exceeds a threshold, viewpoint optimization is triggered, dynamically adjusting the gimbal angle to eliminate occlusion and center the end effector. For example... Figure 3 As shown, the specific implementation is as follows:

[0149] S3.1: Determine whether the optimized triggering conditions are met based on the degree of occlusion and the center position of the end effector.

[0150] Specifically, the occlusion threshold s th The value can be 0.2. When the degree of occlusion is greater than the occlusion threshold, it indicates that the end effector is excessively occluded.

[0151] Calculate the pixel distance from the center of the end effector to the center of the image:

[0152]

[0153] Where (u0, v0) represents the coordinates of the image center, and d pixel_cur The pixel distance from the center of the end effector to the center of the image, (u cur ,v cur ) represents the coordinates of the center position of the end effector.

[0154] Specifically, the image center can take the value (u0, v0) = (540, 360), and the centroid deviation threshold r can be set to 72. If the pixel distance d from the center of the actuator to the image center... pixel_cur >r=72 indicates that the end deviates from the central region.

[0155] Preferably, but not limitingly, a cooling time is set, which is the interval between two adjacent gimbal view optimizations. For example, if the cooling time is set to 0.3 seconds, when the system triggers a view optimization, a 0.3-second timer will start. Before the timer expires, even if the degree of occlusion is greater than the occlusion degree threshold or the center position of the end effector deviates from the center of the image area by more than the centroid deviation threshold, the system will not start a new round of optimization. This can avoid frequent triggering and effectively prevent the gimbal from adjusting too frequently due to slight shaking of the robotic arm or sensor noise.

[0156] It should be noted that those skilled in the art can set specific values ​​for the occlusion degree threshold and the centroid deviation threshold according to actual applications. This application does not limit the specific values ​​for the occlusion degree threshold and the centroid deviation threshold.

[0157] S3.2: Set a local search range around the current gimbal angle, and generate multiple candidate views within the search range according to preset compensation and preset gimbal angle constraint range.

[0158] Specifically, at the current gimbal angle (θ) p_cur ,θ t_cur Set the following local search range in the vicinity:

[0159] Pan:θ p ∈[θ p_cur -10°, θ p_cur +10°

[0160] Tilt:θ t ∈[θ t_cur -10°, θ t_cur +10°] (17)

[0161] Step size set to δθ p =δθ t =2°, then the number of candidate viewpoints generated is expressed as follows:

[0162]

[0163] Set angle constraints and angle range:

[0164] θ p =max(-θ) p_min min(θ) p_max ,θ p_cur +δθp ·kp))

[0165] θ t =max(-θ) t_min min(θ) t_max ,θ t_cur +δθ t ·kt)) (19)

[0166] In the formula, kp and kt represent the discrete candidate angles generated within the defined search range, and the limit range of horizontal rotation is [-θ]. t_min ,θ t_max The pitch and rotation limits are [-θ]. p_min ,θ p_max ].

[0167] It should be noted that kp and kt are integer indices or iteration variables used to generate discrete candidate angles within a defined search range. For example, in an 11x11 grid search, for horizontal rotation, the value range of kp is [-5, -4, ..., 0, ..., 4, 5], and for pitch rotation, the value range of kt is also [-5, -4, ..., 0, ..., 4, 5].

[0168] When kp is 0, θ p_cur +δθ p ·kp represents the current angle θ p_cur When kp changes from -5 to 5, the step size δθ is adjusted accordingly. p Step size δθ p Taking a value of 2°, it generates a value from θ. p_cur -10° to θ p_cur +10° 11 candidate angles.

[0169] By setting constraint angles, it can be ensured that all generated candidate angles will not exceed the physical limit of the gimbal hardware, such as the mechanical rotation limit of the gimbal's servo motor. For example, the range of horizontal rotation is [-180°, 180°], and the range of pitch rotation is [-90°, 90°].

[0170] It should be noted that those skilled in the art can set specific values ​​for the local search range and step size according to actual applications, and this application does not limit the specific values ​​for the local search range and step size.

[0171] S3.3: For each candidate viewpoint, calculate its corresponding occlusion degree and centroid deviation, and form a comprehensive evaluation index by weighted fusion of the occlusion degree and centroid deviation to evaluate each candidate viewpoint.

[0172] Specifically, S3.3 includes:

[0173] S3.3.1: For each candidate viewpoint (θ) p，i ,θ t，j ), calculate the camera pose according to S2.1:

[0174]

[0175] The degree of occlusion s is calculated using the method in S2. i，j and the center position of the end effector (u) i，j ,v i，j And centroid deviation:

[0176]

[0177] The comprehensive evaluation index value is formed by weighted fusion of occlusion degree and centroid deviation, and expressed by the following formula:

[0178]

[0179] Among them, Score i，j S represents the comprehensive evaluation index value. i，j Indicates the degree of occlusion, d pixel，i，j w1 represents the centroid deviation, and w1 represents the degree of occlusion S. i，j The weights, w2 represents the centroid deviation d pixel，i，j The weights are used to measure the degree of deviation of the end effector in the image. A smaller overall evaluation metric indicates a better candidate viewpoint. This is achieved by calculating the centroid deviation d. pixel，i，j The centroid deviation is normalized and scaled to the range [0,1] by the ratio of the maximum pixel distance from the end effector center position to the image center.

[0180] Specifically, w1 = 0.7, w2 = 0.3,

[0181] It should be noted that those skilled in the art can set the weights of the degree of occlusion and the centroid deviation according to actual needs. The specific values ​​of the weights of the degree of occlusion and the centroid deviation are not limited in the embodiments of this disclosure.

[0182] Furthermore, S3.4 employs a two-stage optimization strategy. First, a grid search is used to quickly filter candidate viewpoints. If the results are unsatisfactory, gradient optimization is used to fine-tune the angles. S3.4 specifically includes:

[0183] S3.4.1: Grid search: Select the candidate viewpoint corresponding to the minimum comprehensive evaluation index. If the minimum comprehensive evaluation index value is less than the score threshold, it is determined as the optimal gimbal angle.

[0184] Specifically, 121 candidate perspectives were evaluated, and the perspective with the lowest comprehensive evaluation index value was selected:

[0185]

[0186] In practice, the computation time is controlled within 100 milliseconds through GPU parallel processing.

[0187] The score threshold can be set to 0.3. i，j Less than 0.3, corresponding to S i，j <2,d pixel，i，j <150, use directly

[0188] S3.4.2: Gradient Descent: If the minimum comprehensive evaluation index value is greater than or equal to the score threshold, calculate the numerical gradient of the minimum comprehensive evaluation index value, multiply the numerical gradient by the learning rate and iterate until the iteration condition is met, and determine the angle corresponding to the optimal comprehensive evaluation index value as the optimal gimbal angle.

[0189] Specifically, if Score i，j ≥0.3, with Using initial values, calculate the numerical gradient:

[0190]

[0191] Where ε is 1°, (θ) p ,θ t ) represents the gimbal angle coordinates during the iteration process.

[0192] Update the viewpoint by multiplying the numerical gradient by the learning rate:

[0193]

[0194] Where η represents the learning rate. The learning rate can be 0.5°, iterated 10 times, or the gradient norm. Stop iterating when the time comes.

[0195] S3.5: Set the gimbal angle change threshold. If the gimbal angle change rate is less than the gimbal angle change threshold, the view will not be switched; otherwise, the view will be updated according to the optimal gimbal angle.

[0196] like and Maintain the current viewpoint to avoid frequent switching and prevent frequent viewpoint jitter.

[0197] Preferably, but not restrictively, candidate views are assigned to GPU threads for batch occlusion rate calculation.

[0198] In some embodiments, the total optimization time is controlled within a preset time, such as 100 milliseconds, by using GPU threads, thereby improving the optimization speed.

[0199] Preferably, but not restrictively, historical optimization results should be cached. Specifically, the results of the five most recent optimizations should be cached to speed up subsequent calculations.

[0200] If the score difference for the new perspective is less than 0.05, the historical result is reused. The comprehensive evaluation index value corresponding to the optimal perspective found in this optimization process is compared with the comprehensive evaluation index value corresponding to the historical best perspective stored in the cache, and the difference between the two is calculated to obtain the score difference for the new perspective. For example, the system caches the best results found in the last 5 optimizations. If the comprehensive evaluation index value of the current optimal perspective is 0.15, while the best comprehensive evaluation index value in the historical cache is 0.12, then the difference is 0.03. Because 0.03 is less than the set threshold of 0.05, it means that the new perspective found this time has not significantly improved performance. For stability, the new result is discarded, and the historical best perspective is used instead. It is understandable that the comparison is between the currently found perspective and the historical best perspective, not just the two most recent ones.

[0201] As one of the most prominent substantive features of this application, this application uses the comparison of 3D model projection and depth information to quantitatively perceive occlusion in real time, and determines the optimal gimbal viewpoint by comprehensively evaluating index values, thereby eliminating perceived occlusion and realizing viewpoint switching. This application does not rely on image features, has strong environmental adaptability, overcomes the limitations of image processing methods, and significantly improves robustness to environmental interference.

[0202] S4: Based on the optimized gimbal angle, plan the view switching path. The view switching path is used to drive the gimbal to rotate and update the camera pose in real time.

[0203] Furthermore, such as Figure 4 As shown, the pan-tilt control in S4 specifically includes:

[0204] S4.1: Based on the optimal gimbal angle obtained from S3 optimization, plan a smooth interpolation path from the current angle to the optimal gimbal angle.

[0205] Specifically, the plan is based on the following formula, starting from the current angle (θ). p_cur ,θ t_cur To the optimal gimbal angle (θ) p * ,θ t * Linear interpolation path:

[0206]

[0207] In the formula, (θ p (t),θ t (t) represents the optimal gimbal angle, (θ) p_cur ,θ t_cur(θ) represents the current angle, T represents the transition time, and t represents the elapsed time since the path planning, i.e., the moment the gimbal begins to rotate. p * ,θ t * The angle () represents the linear interpolation angle in the planned path. Specifically, T is set to 0.5 seconds, and T can be divided into 50 steps, each step being 10 milliseconds. This ensures that the viewpoint switching process is smooth and does not interfere with the operator's observation.

[0208] The transition time is a preset duration for the gimbal to complete one viewpoint switch. Specifically, after the system determines a new optimal target viewpoint, to avoid interference to the operator caused by the instantaneous change in visual imagery, this embodiment does not directly command the gimbal to immediately switch to the target position. Instead, this embodiment plans a smooth motion trajectory from the current angle to the target angle according to the method in S4, and the transition time is the time taken to complete this trajectory. This parameter setting is to ensure the smoothness and continuity of viewpoint switching in this invention, and is a crucial step in solving the problem of "abrupt viewpoint switching" in the background art.

[0209] S4.2: Drive the gimbal servo motor in real time based on the error between the optimal gimbal angle and the current angle.

[0210] First, a PID controller is used to drive the gimbal servo motor, and the control law is as follows:

[0211]

[0212] Where e(t) represents the difference between the target angle and the actual angle at time t, and k p k represents the scaling parameter. i Denotes the integration parameter, k d Let u(t) represent the differential parameter, and u(t) represent the controller output signal. p (t), u t (t) is composed of. The error is calculated using the following formula: e p (t)=θ p * -θ p (t), e t (t)=θ t * -θ t (t).

[0213] In this embodiment of the disclosure, parameter k p The value is 1.0, k i The value is 0.1, k d The value is set to 0.005, and the angle error e(t) is controlled within 0.2°. Then the output u is... p (t), u t(t) is converted into a servo motor pulse signal.

[0214] Understandably, the PID controller runs on the main computing platform, such as the Jetson AGXXavier in this embodiment. Its workflow is as follows: after the viewpoint optimization module calculates the target angle of the gimbal, the PID control algorithm receives this target value and compares it with the current angle read back in real-time from the gimbal encoder. By calculating the error between the two, the algorithm generates precise control commands, including voltage / pulse signals, which are then sent to the gimbal's servo motor driver for execution. PID control enables smooth viewpoint switching and maintains high precision and stability in rotation.

[0215] S4.3: Updates camera pose in real time during gimbal rotation.

[0216] During rotation, the gimbal angle (θ) is read in real time. p (t),θ t (t)), update camera pose:

[0217]

[0218] Specifically, the update frequency can be 100Hz.

[0219] By updating the camera pose during the rotation of the gimbal, the continuity of projection calculation can be ensured, making the calculation of occlusion rate more accurate and further improving the degree of automation.

[0220] S4.4: After the rotation is completed, re-execute S3.1 to verify whether the degree of occlusion and the center position of the end effector meet the optimization trigger conditions. If the optimization trigger conditions are met, a new round of optimization is triggered.

[0221] Specifically, if the occlusion degree is ≤0.2 and the pixel distance from the center of the end effector to the center of the image is ≤72, the adjustment is successful; otherwise, a new round of optimization is triggered, and the search range is expanded to ±0.2°.

[0222] S5: Periodically verify the effectiveness of the viewing angle adjustment. Predict future poses based on the robotic arm's motion trends and optimize the gimbal's viewing angle in advance. Dynamically adjust the occlusion threshold and centroid deviation threshold to adapt to different task requirements. Feedback optimization ensures the system adapts to dynamic changes. The specific implementation is as follows:

[0223] S5.1: Verify at fixed intervals whether the degree of occlusion of the current viewpoint and the center position of the end effector meet the optimized triggering conditions.

[0224] Specifically, the occlusion level and the end effector center position are calculated every 100 milliseconds to verify whether the occlusion level is ≤0.2 and the pixel distance from the end effector center position to the image center is ≤72.

[0225] S5.2: Calculate the rate of change of the joint angle of the robotic arm, predict the pose change of the end effector based on the Jacobian matrix according to the rate of change, obtain the predicted pose, and calculate the degree of occlusion and the comprehensive evaluation index value according to the predicted pose. If the difference between the comprehensive evaluation index value of the predicted pose and the comprehensive evaluation index value of the current pose is less than a preset threshold, then drive the gimbal in advance according to the predicted pose.

[0226] Specifically, the rate of change of the robotic arm joint angle is calculated using the following formula:

[0227]

[0228] Predict the pose change of the end effector based on the rate of change using the following formula:

[0229]

[0230] in, The Jacobian matrix describes the linear relationship between the velocity of the end effector and the velocities of the robotic arm's joints. The value of this linear relationship matrix depends on the current posture of the robotic arm, i.e., the current joint angle q. Since the joint angle q changes in real time during the robotic arm's movement, the Jacobian matrix is ​​recalculated in real time based on the latest joint angle q(t).

[0231] Specifically, the preset threshold value is 0.1. Those skilled in the art can set the specific value of the preset threshold according to the actual application, and this disclosure does not impose any restrictions on it.

[0232] It's understandable that the S5.1 cycle verification and the S5.2 motion prediction can be two parallel, complementary logical branches. For example, assuming the current viewpoint is excellent, occlusion is zero, and the end effector is centered, then S5.1 verification passes. However, the system detects rapid movement of the robotic arm through S5.2 and predicts that in 0.2 seconds, a link will severely obstruct the end effector. Even if the S5.1 result meets the conditions, the S5.2 prediction will still trigger viewpoint optimization in advance, driving the gimbal to rotate, thus preventing the unfavorable viewpoint from appearing 0.2 seconds later. These two steps can run simultaneously in the background. If either branch determines that adjustment is needed, whether it has already occurred or will occur in the future, it will initiate or affect viewpoint optimization, jointly ensuring the continuity and quality of visual feedback.

[0233] As one of the outstanding real-time features of this application, the Jacobian matrix calculated in real time by analyzing the change rate of the robotic arm joint angle predicts the pose change of the end effector in a short period of time, thereby predicting the future occlusion trend. Based on this, the gimbal view is calculated and optimized in advance to reduce response latency, thereby achieving proactive and predictive occlusion avoidance and improving the real-time performance and smoothness of the system.

[0234] S5.3: Dynamically adjust the occlusion threshold and centroid deviation threshold according to different stages of the task.

[0235] Specifically, depending on the different stages of the task, including but not limited to coarse positioning when approaching the target or fine operation when grasping the wire, the occlusion threshold and centroid deviation threshold are dynamically adjusted to balance visual clarity and computational efficiency.

[0236] Specifically, for complex environments such as areas with dense power lines, environmental geometry modeling or image segmentation techniques can be combined to further optimize the accuracy of occlusion detection. In the coarse localization stage, for example, when approaching power lines, a setting is used... th =0.3, r=120 pixels. For fine-tuning stages such as grasping, set s... th =0.1, r=60 pixels.

[0237] If continuous optimization fails, the threshold is gradually reduced: s th ←s th -0.05, untils th ≥0.05.

[0238] Combination Figure 5 As shown, Embodiment 2 of this application provides a system for end-effector obstruction detection and viewpoint switching of a remotely operated live-line working robot, which operates the end-effector obstruction detection and viewpoint switching method of the remotely operated live-line working robot as described in Embodiment 1. The system includes:

[0239] The initialization module is used to load the 3D geometric model of the robotic arm of the teleoperated live-line working robot and configure the camera parameters during the initialization phase.

[0240] The initialization module can be an embedded computing platform, such as Jetson AGX Xavier, to load the URDF model, perform hand-eye calibration through ROS nodes, and configure camera and gimbal parameters.

[0241] The occlusion perception module is used to project the end effector and body of the three-dimensional geometric model onto the image plane based on camera parameters, generate corresponding region masks and depth information, identify occlusion regions based on the region masks and depth information, and calculate the occlusion degree of the end effector and its center position on the image plane in real time; the occlusion degree is the proportion of the occluded region to the total end effector region.

[0242] Hardware: GPU supports real-time rendering, industrial camera (resolution downsampled to 640×480).

[0243] Software: Implements mesh projection, depth comparison, and occlusion rate calculation at a frequency of 20Hz.

[0244] The viewing angle optimization module is used to determine whether the optimization triggering condition is met based on the degree of occlusion and the center position of the end effector in the image plane. If the optimization triggering condition is met, the optimal gimbal angle is determined based on the degree of occlusion and the center position of the end effector to optimize the gimbal viewing angle and obtain the optimal gimbal angle. The optimization triggering condition is: the degree of occlusion exceeds the occlusion degree threshold or the center position of the end effector in the image plane deviates from the preset image center area by more than the centroid deviation threshold.

[0245] The view optimization module performs parallel computation on 121 candidate views, combining grid search and gradient descent, with an optimization time of less than 100 milliseconds.

[0246] The gimbal control module is used to plan a viewpoint switching path based on the optimal gimbal angle, and drive the gimbal to rotate to the optimal gimbal angle according to the viewpoint switching path.

[0247] The cloud control module can be a two-degree-of-freedom servo gimbal with a mechanical delay of approximately 100 milliseconds. It is driven by a PID controller, and the path planning is executed in 50 steps.

[0248] Optionally, the system further includes:

[0249] The feedback optimization module is used to periodically verify the effect of the view optimization. It predicts the future pose based on the movement trend of the robotic arm and optimizes the gimbal view in advance according to the predicted future pose. It dynamically adjusts the occlusion degree threshold and the centroid deviation threshold to adapt to different task requirements.

[0250] Regarding the system in the above embodiments, the specific manner in which each unit performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0251] Embodiment 3 of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, it implements the end-effector occlusion sensing and viewpoint switching method of the teleoperated live-line working robot described in Embodiment 1.

[0252] Embodiment 4 of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for end-effector occlusion detection and viewpoint switching of a remotely operated live-line working robot according to Embodiment 1.

[0253] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0254] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0255] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of the claims of this application.

Claims

1. A method for end-effector obstruction detection and viewpoint switching of a remotely operated live-line working robot, characterized in that, The method includes the following steps: S1: Initialization phase, load the 3D geometric model of the robotic arm of the teleoperated live-line working robot, perform hand-eye calibration to determine the relative pose of the camera and the robotic arm, and configure the camera parameters; S2: Based on camera parameters and the relative pose of the camera and the robotic arm, the end effector and body of the three-dimensional geometric model are projected onto the image plane to generate corresponding region masks and depth information. The occlusion area is identified according to the region mask and the depth information, and the occlusion degree of the end effector and its center position on the image plane are calculated in real time. The occlusion degree is the proportion of the occluded area to the total area of ​​the end effector. S3: Determine whether the optimization triggering condition is met based on the degree of occlusion and the center position of the end effector in the image plane. If the optimization triggering condition is met, determine the optimal gimbal angle based on the degree of occlusion and the center position of the end effector to optimize the gimbal viewing angle and obtain the optimal gimbal angle. The optimization triggering condition is: the degree of occlusion exceeds the occlusion degree threshold or the center position of the end effector in the image plane deviates from the preset image center area by more than the centroid deviation threshold. S4: Based on the optimal gimbal angle, plan the view switching path, and drive the gimbal to rotate to the optimal gimbal angle according to the view switching path.

2. The method for end-effector obstruction detection and viewpoint switching of a remotely operated live-line working robot according to claim 1, characterized in that: In S3, determining the optimal gimbal angle based on the degree of occlusion and the center position of the end effector includes: Set the search range around the current gimbal angle, and generate multiple candidate viewpoints within the search range according to the preset step size and preset gimbal angle constraint range; For each candidate viewpoint, calculate its corresponding occlusion degree and centroid deviation, and perform weighted fusion of the occlusion degree and centroid deviation to form a comprehensive evaluation index value, and evaluate each candidate viewpoint; The candidate viewpoint corresponding to the minimum comprehensive evaluation index value is selected as the optimal gimbal angle.

3. The method for end-effector obstruction detection and viewpoint switching of a remotely operated live-line working robot according to claim 2, characterized in that: A strategy combining grid search and gradient optimization is adopted to select the candidate viewpoint corresponding to the minimum comprehensive evaluation index value as the optimal gimbal angle. The grid search strategy determines whether the minimum comprehensive evaluation index value is less than the score threshold. If so, the candidate viewpoint corresponding to the minimum comprehensive evaluation index value is determined as the optimal gimbal angle. The gradient optimization strategy calculates the numerical gradient of the minimum comprehensive evaluation index value if the minimum comprehensive evaluation index value is greater than or equal to the score threshold. The numerical gradient is multiplied by the learning rate and iterated until the iteration condition or stopping condition is met. The angle corresponding to the optimal comprehensive evaluation index value obtained by the iteration is determined as the optimal gimbal angle.

4. The method for end-effector obstruction detection and viewpoint switching of a remotely operated live-line working robot according to claim 1, characterized in that: The occlusion area is identified based on the region mask and depth information using the following formula, and the degree of occlusion of the end effector is calculated in real time: M overlap =M ee ∩M body In the formula, S cur To indicate the degree of occlusion, 1 represents complete occlusion of the end effector or the end effector is completely removed from the field of view, and N represents the degree of occlusion. ee M represents the total number of pixels at the end. ee M represents the end effector mask. body Represents the ontology mask, N occluded This represents the total number of end effector pixels occluded by the robotic arm body in the overlapping region, where II(·) is the indicator function, and M... overlap This represents the overlapping region mask, (u,v) represents the pixel coordinates on the image plane, and D... ee This indicates that a depth map of the end effector pixels is generated based on the depth information from the 3D spatial points corresponding to the end effector in the 3D geometric model to the camera. D body A depth map of the body pixels is generated by representing the depth information from the 3D spatial points corresponding to the body part of the 3D geometric model to the camera. For pixel coordinates (u,v)∈M on the image plane overlap If D ee (u,v)>D body (u,v) indicates that the pixel is occluded by the body, thus identifying the occluded area.

5. The method for end-effector obstruction detection and viewpoint switching of a remotely operated live-line working robot according to claim 1, characterized in that: In S4, based on the optimized gimbal angle, the planned viewpoint switching path includes: Plan the linear interpolation path from the current angle to the optimal gimbal angle using the following formula: In the formula, (θ p (t),θ t (t) represents the optimal gimbal angle, (θ) p_cur ,θ t_cur () indicates the current angle. The interpolation angle is represented by T, the transition time is the time taken to complete the viewpoint switching path, and t is the elapsed time since the gimbal started rotating.

6. The method for end-effector obstruction detection and viewpoint switching of a remotely operated live-line working robot according to claim 1, characterized in that: The method further includes: S5: Feedback optimization phase, periodically verify the effect of view optimization, predict future pose based on the movement trend of the robotic arm and optimize the gimbal view in advance according to the predicted future pose, dynamically adjust the occlusion degree threshold and centroid deviation threshold to adapt to different task requirements.

7. The method for end-effector obstruction detection and viewpoint switching of a remotely operated live-line working robot according to claim 6, characterized in that: S5 includes predicting future pose based on the robotic arm's motion trend and optimizing the gimbal's viewing angle in advance based on the predicted future pose, including: The rate of change of the joint angles of the robotic arm is calculated, and the pose change of the end effector is predicted based on the rate of change using the Jacobian matrix to obtain the predicted pose; wherein, the Jacobian matrix is ​​used to represent the linear relationship between the velocity of the end effector and the velocity of each joint of the robotic arm; The degree of occlusion and the comprehensive evaluation index value are calculated based on the predicted pose. If the difference between the comprehensive evaluation index value of the predicted pose and the comprehensive evaluation index value of the current pose is less than a preset threshold, the gimbal is driven to perform optimization in advance according to the predicted pose.

8. The method for end-effector obstruction detection and viewpoint switching of a remotely operated live-line working robot according to claim 1, characterized in that: S3 also includes: setting a cooling time, which is the interval between two adjacent gimbal view optimizations.

9. The method for end-effector obstruction detection and viewpoint switching of a remotely operated live-line working robot according to claim 1, characterized in that: S3 also includes setting a gimbal angle change threshold. When the gimbal angle change rate is less than the gimbal angle change threshold, the viewpoint is not switched; otherwise, the gimbal viewpoint is switched according to the optimal gimbal angle.

10. A system for end-effector occlusion detection and viewpoint switching of a remotely operated live-line working robot using the end-effector occlusion detection and viewpoint switching method according to any one of claims 1 to 9, characterized in that, The system includes: The initialization module is used to load the three-dimensional geometric model of the robotic arm of the teleoperated live-line working robot and configure the camera parameters during the initialization phase. The occlusion perception module is used to project the end effector and body of the three-dimensional geometric model onto the image plane based on camera parameters, generate corresponding region masks and depth information, identify occlusion regions based on the region masks and depth information, and calculate the occlusion degree of the end effector and its center position on the image plane in real time; the occlusion degree is the proportion of the occluded region to the total end effector region; The viewpoint optimization module is used to determine whether the optimization triggering condition is met based on the degree of occlusion and the center position of the end effector in the image plane. If the optimization triggering condition is met, the optimal gimbal angle is determined based on the degree of occlusion and the center position of the end effector to optimize the gimbal viewpoint and obtain the optimal gimbal angle. The optimization triggering condition is: the degree of occlusion exceeds the occlusion degree threshold or the center position of the end effector in the image plane deviates from the preset image center area by more than the centroid deviation threshold. The gimbal control module is used to plan a viewpoint switching path based on the optimal gimbal angle, and drive the gimbal to rotate to the optimal gimbal angle according to the viewpoint switching path.

11. The end-effector obstruction detection and viewpoint switching system for a remotely operated live-line working robot according to claim 10, characterized in that: The system also includes: The feedback optimization module is used to periodically verify the effect of the view optimization. It predicts the future pose based on the movement trend of the robotic arm and optimizes the gimbal view in advance according to the predicted future pose. It dynamically adjusts the occlusion degree threshold and the centroid deviation threshold to adapt to different task requirements.

12. An electronic device, comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the end-effector occlusion sensing and viewpoint switching method for a teleoperated live-line working robot according to any one of claims 1-9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the steps of the end-effector occlusion perception and viewpoint switching method for any one of claims 1-9 of the teleoperated live-line working robot.

Images