Dual-arm cooperative vision optimization method, device, equipment and system

Abstract

The application relates to the technical field of robot cooperative control and visual perception, in particular to a dual-arm cooperative visual optimization method, device, equipment and system for a substation operation and maintenance scene. The method comprises the following steps: acquiring an end task instruction and a target feature pose of a main operating arm; based on the end task instruction and the target feature pose, a multi-objective optimization function containing a field of view centering degree, an observation distance and dynamic anti-shielding is constructed; and according to the multi-objective optimization function and a null space control method, an optimization control instruction of an auxiliary visual arm is generated. The method solves the problems that an existing fixed global camera or an end visual is easily shielded by a mechanical arm in a narrow and dense environment, a field of view is insufficient in close-range operation, and it is difficult to balance continuous global perception and operation safety, and improves the environmental perception integrity, close-range operation safety and dual-arm cooperative control effect.

Description

Technical Field

[0001] This application relates to the fields of robot collaborative control and visual perception technology, and in particular to a dual-arm collaborative visual optimization method, device, equipment and system. Background Technology

[0002] In the operation and maintenance scenarios of high-voltage substations, robotic arms need to perform high-precision end-effector operations in confined spaces while simultaneously sensing the surrounding environment in real time to avoid collisions with live conductors or obstacles. Therefore, there is an urgent need for a dual-arm collaborative control method that can actively adjust the visual arm's pose during dynamic operations and achieve unobstructed global observation to ensure operational accuracy and system safety.

[0003] In existing technologies, visual perception for collaborative operation of two arms is typically achieved by fixing a global camera to the chassis of the robotic arm or the working environment, or by directly integrating the vision system into the end effector of the main manipulator.

[0004] However, existing end-effector cameras on main control arms have limited field of view due to close-range operation, making them unable to perceive the surrounding environment and increasing the risk of collisions. Summary of the Invention

[0005] This application provides a dual-arm collaborative vision optimization method, device, equipment, and system for substation operation and maintenance scenarios. It addresses the vision collaboration needs of the main operating arm when performing dynamic operations in substation operation and maintenance scenarios. Based on the task objectives and target object pose information of the main operating arm, the observation posture of the auxiliary vision arm is comprehensively optimized. This enables the auxiliary vision arm to balance target observation quality, observation stability, and operational collaboration when cooperating with the main operating arm, thereby improving the continuous global perception capability and close-range operation safety in complex and narrow environments.

[0006] Firstly, this application provides a dual-arm collaborative vision optimization method for substation operation and maintenance, including:

[0007] Acquire the end-effector command and target feature pose of the main manipulator;

[0008] Based on the end-task command and the target feature pose, a multi-objective optimization function is constructed, which includes field of view centering, observation distance, and dynamic anti-occlusion.

[0009] Based on the multi-objective optimization function and the zero-space control method, optimized control commands for the auxiliary vision arm are generated.

[0010] In one possible implementation, based on the end-task command and the target feature pose, a multi-objective optimization function is constructed, including field-of-view centering, observation distance, and dynamic anti-occlusion.

[0011] Construct a kinematic capsule envelope model of the main manipulator;

[0012] Based on the terminal task instructions, the kinematic capsule envelope model is dynamically updated to obtain a dynamic obstacle model;

[0013] Based on the dynamic obstacle model and the target feature pose, a multi-objective optimization function is constructed, which includes field of view centering, observation distance, and dynamic anti-occlusion.

[0014] In one possible implementation, constructing the kinematic capsule envelope model of the main manipulator includes:

[0015] Obtain the joint angle data of the main manipulator;

[0016] Based on the joint angle data and the forward kinematics calculation method, the end positions of each link in the main operating arm are calculated;

[0017] Based on the end positions of each link in the main control arm, a kinematic capsule envelope model is constructed.

[0018] In one possible implementation, based on the dynamic obstacle model and the target feature pose, a multi-objective optimization function is constructed, including field-of-view centering, observation distance, and dynamic anti-occlusion, comprising:

[0019] Based on the target feature pose, the line-of-sight vector, coordinate deviation, and expected observation distance of the visual arm are determined;

[0020] Based on the dynamic obstacle model and the line-of-sight vector, a dynamic anti-occlusion cost is generated;

[0021] Based on the visual interaction matrix and the coordinate deviation, a field-of-view centering cost is generated;

[0022] Based on the gradient descent method and the desired observation distance, an observation distance cost is generated.

[0023] Based on the dynamic anti-occlusion cost, the field-of-view centering cost, and the observation distance cost, a multi-objective optimization function is constructed.

[0024] In one possible implementation, obtaining the target feature pose of the main manipulator includes:

[0025] Environmental point cloud data is collected using the image acquisition device on the main operating arm;

[0026] Based on the environmental point cloud data, the pixel coordinates of the target features are extracted;

[0027] Based on the camera pose transformation matrix, the pixel coordinates of the target feature are converted into the target feature pose in the global base coordinate system.

[0028] In one possible implementation, based on the multi-objective optimization function and the null space control method, optimized control instructions for the auxiliary vision arm are generated, including:

[0029] Calculate the gradient vector based on the multi-objective optimization function;

[0030] A pseudo-inverse algorithm based on the visual interaction matrix is ​​used to generate the main task speed command.

[0031] According to the null space control method, the gradient vector is superimposed on the main task speed command to generate the optimized control command for the auxiliary vision arm.

[0032] In one possible implementation, optimized control instructions for the auxiliary vision arm are generated, including:

[0033] The weight coefficients of the multi-objective optimization function are dynamically adjusted based on reinforcement learning algorithms, or an avoidance path is generated based on the unknown future linkage of the main manipulator determined by a motion prediction model.

[0034] Secondly, this application provides a dual-arm collaborative vision optimization device for substation operation and maintenance, comprising:

[0035] The main manipulator data module is used to acquire the end-effector task commands and target feature poses of the main manipulator.

[0036] The optimization function construction module is used to construct a multi-objective optimization function that includes field of view centering, observation distance, and dynamic anti-occlusion based on the end task instruction and the target feature pose;

[0037] The optimization instruction generation module is used to generate optimization control instructions for the auxiliary vision arm based on the multi-objective optimization function and the null space control method.

[0038] Thirdly, this application provides a dual-arm collaborative vision optimization device, including: a memory and a processor;

[0039] The memory stores computer-executed instructions;

[0040] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0041] Fourthly, this application provides a dual-arm collaborative vision optimization system, including a computer program, a main manipulator arm, and an auxiliary vision arm;

[0042] When the computer program is executed by the processor, it implements the first aspect and / or various possible implementations of the first aspect as described above.

[0043] This application provides a dual-arm collaborative vision optimization method, device, equipment, and system for substation operation and maintenance scenarios. It acquires the end-effector task command and target feature pose of the main operating arm; based on the end-effector task command and target feature pose, it constructs a multi-objective optimization function including field-of-view centering, observation distance, and dynamic anti-occlusion; and generates optimized control commands for the auxiliary vision arm according to the multi-objective optimization function and the zero-space control method. This solves the problems of existing fixed global cameras or end-effector vision systems being easily obstructed by robotic arms in narrow and dense environments, insufficient field of view during close-range operations, and difficulty in balancing continuous global perception and operational safety. It improves the integrity of environmental perception, the safety of close-range operations, and the effectiveness of dual-arm collaborative control. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0045] Figure 1 This is a flowchart illustrating an embodiment of the dual-arm collaborative vision optimization method provided in this application.

[0046] Figure 2 This is a flowchart illustrating Embodiment 2 of the dual-arm collaborative vision optimization method provided in this application;

[0047] Figure 3 This is a flowchart illustrating Embodiment 3 of the dual-arm collaborative vision optimization method proposed in this application;

[0048] Figure 4 A three-dimensional diagram illustrating the relationship between the shortest distance and the line-of-sight vector provided for this application;

[0049] Figure 5 A flowchart illustrating the dual-arm decoupling control law provided in this application;

[0050] Figure 6 A schematic diagram of the dual-arm collaborative vision optimization device provided in this application;

[0051] Figure 7 This is a schematic diagram of the structure of the dual-arm collaborative vision optimization device provided in this application.

[0052] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0053] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0054] This application applies to high-voltage substation operation and maintenance scenarios. Its network architecture includes a main operating arm (performing tasks such as temperature measurement and partial discharge detection), an auxiliary vision arm (equipped with an RGB-D camera), and a central control unit (running optimization algorithms). The main operating arm needs to perform high-precision operations on high-voltage equipment (such as circuit breaker contacts and surge arresters) within confined spaces, while the auxiliary vision arm needs to dynamically adjust its pose to provide an unobstructed global observation view. The system needs to process point cloud data, kinematic models, and optimize control commands in real time to ensure the safety and efficiency of the two arms working collaboratively in complex environments.

[0055] In existing technologies, fixed global cameras are easily obstructed by links during robotic arm movement, leading to target loss; end-effector cameras lose their global perception capability due to excessively close working distance, increasing the risk of collisions. For example, in a substation temperature measurement scenario, the end-effector camera of the main operating arm has a narrowed field of view due to its proximity to high-voltage contacts, making it unable to detect the relative position of the elbow to other equipment; while fixed cameras are obstructed by links during arm extension, failing to provide effective environmental information. Furthermore, existing technologies lack real-time obstacle avoidance mechanisms for dynamic obstacles (such as moving robotic arm links), resulting in lagging vision-motion collaborative planning and difficulty in adapting to the rapidly changing operational needs in substation operation and maintenance.

[0056] Based on the aforementioned technical challenges, this application addresses the high safety requirements of substation operation and maintenance. First, it identifies the limitations of fixed cameras and end-point cameras: fixed cameras cannot adapt to dynamic occlusion, and end-point cameras cannot cover the entire environment. Subsequently, it proposes decoupling the pose adjustment of the auxiliary vision arm from the motion trajectory of the main operating arm, calculating the optimal observation position in real time through a dynamic optimization function. Building upon this, null-space projection technology is introduced to ensure that the secondary tasks of the vision arm (anti-occlusion and distance optimization) do not affect the primary task (centering the field of view). Finally, by constructing a kinematic capsule envelope model and a dynamic obstacle model, real-time prediction of occlusion risks on the main operating arm links is achieved. Combined with gradient descent, optimized control commands are generated to complete the active visual control of the two arms in coordination.

[0057] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0058] Figure 1 This is a flowchart illustrating an embodiment of the dual-arm collaborative vision optimization method provided in this application. Figure 1 As shown, the method includes:

[0059] S101. Obtain the end-effector command and target feature pose of the main manipulator.

[0060] Among them, the end-effector command of the main manipulator is used to characterize the operation action that the main manipulator needs to perform and its motion constraints, and serves as the input for the subsequent auxiliary vision arm to perform collaborative observation planning.

[0061] In the embodiments of this application, the end-point task instruction may be an operation instruction issued by the control unit for temperature measurement, partial discharge detection, contact alignment, close-range inspection, or other operations for the operation and maintenance of live equipment in the substation, or it may include pose or motion constraint information related to the operation.

[0062] The target feature pose is used to characterize the position and attitude information of the observed object in space. It can correspond to high-voltage circuit breaker contacts, disconnector contact points, surge arrester connection terminals, meter feature areas, or local structural areas that need to be measured at close range.

[0063] In the embodiments of this application, the target feature pose can be obtained through existing visual perception or pose determination methods to form target feature pose data that can be used for subsequent optimization calculations.

[0064] In practice, the control unit in the dual-arm collaborative vision optimization system receives the work plan from the host computer and generates end-effector task instructions for the main manipulator based on the work plan. The control unit can also determine the work constraint information related to subsequent collaborative observations by combining the current state of the main manipulator.

[0065] At the same time, it is also necessary to collect environmental point cloud data based on the image acquisition device on the main manipulator to extract the pixel coordinates of the target features; and based on the camera pose transformation matrix, convert the pixel coordinates of the target features into the target feature pose in the global base coordinate system.

[0066] Among them, environmental point cloud data is used to characterize the three-dimensional geometric relationship between the target area and its surrounding obstacles.

[0067] Pixel coordinates can be extracted from the image region corresponding to the point cloud data and correspond one-to-one with the position of the target feature in the field of view.

[0068] The camera pose transformation matrix can be obtained through hand-eye calibration and used to convert the target position in the image domain into pose information in the global space.

[0069] In practical implementation, the image acquisition device on the main manipulator can be installed near the end effector, with its lens facing the target work area, so that it can simultaneously acquire images and depth information of the surrounding environment during the movement of the main manipulator. The image acquisition device can be any of a structured light camera, a binocular camera, or an RGB-D (Red Green Blue-Depth) camera. The output data is processed by the point cloud reconstruction module to form an environmental point cloud data stream.

[0070] Based on the environmental point cloud data, the system segments and identifies the region where the target features are located, then extracts the pixel coordinates corresponding to the target features, and associates the pixel coordinates with camera intrinsic parameters and depth information to obtain the two-dimensional position of the target in the camera's field of view.

[0071] Subsequently, by combining the camera pose transformation matrix obtained from calibration, the camera coordinates corresponding to the pixel coordinates are transformed to the global base coordinate system, and the target feature pose is output. The pose includes at least the position parameters of the target feature, and may also include the attitude parameters if necessary.

[0072] During operation, the image acquisition device on the main manipulator continuously acquires environmental point cloud data. The system determines the pixel coordinates of the target feature regions in the point cloud, and then completes the coordinate system transformation through the camera pose transformation matrix, thereby obtaining the target feature pose unified in the global base coordinate system. This pose result can be directly used as the input for the subsequent construction of multi-objective optimization functions, enabling the auxiliary vision arm to complete field of view adjustment, observation distance control, and dynamic anti-occlusion constraint calculation under a unified spatial reference.

[0073] After adopting the above method, the image information of the target features can be stably mapped into global spatial pose information. The target position and attitude obtained by the main manipulator during dynamic operation are more convenient to interface with the control model, thereby providing a consistent spatial reference for subsequent auxiliary visual arm collaborative observation and path optimization.

[0074] Through the steps of this application, the system's control unit simultaneously obtains the task requirements of the main manipulator and the target spatial pose within the same control cycle, so that the subsequent adjustment of the auxiliary vision arm no longer depends on a fixed viewpoint or simple end-effector close-range observation, but is based on the real-time coupling of the operation action and the target spatial relationship.

[0075] This application directly addresses the working conditions where the main operating arm moves continuously in the narrow environment of a power station and the target area may be obstructed by connecting rods. By using the task command at the end of the main operating arm and the target feature pose as unified inputs, the subsequent optimization process can be carried out around the real working object and the real-time working path, thereby providing a calculable state basis for continuous global perception.

[0076] S102. Based on the end-task command and target feature pose, construct a multi-objective optimization function that includes field of view centering, observation distance, and dynamic anti-occlusion.

[0077] The multi-objective optimization function is used to comprehensively evaluate whether the current observation pose of the auxiliary vision arm meets the requirements of collaborative operation. Its core inputs include the real-time motion state of the main manipulator, the target feature pose, and the current camera pose of the auxiliary vision arm.

[0078] Field of view centering is used to characterize the degree of deviation of target features from the field of view observed by the auxiliary visual arm, and can constitute the corresponding cost term.

[0079] The observation distance is used to characterize whether the spatial distance between the auxiliary vision arm and the target is within the preset observation range, and can constitute the corresponding cost term.

[0080] Dynamic anti-occlusion is used to characterize the risk of the main manipulator linkage obstructing the line of sight of the auxiliary vision arm. It can be used to construct corresponding cost terms based on the relationship between the movement state of the main manipulator and the line of sight of the auxiliary vision arm.

[0081] In this embodiment, the multi-objective optimization function can be expressed as a weighted sum of each cost term, wherein the weight parameters are set by the control unit according to the task type, equipment level and main operating arm movement speed. The smaller the function value, the closer the current pose of the auxiliary vision arm is to the observation state of being unobstructed, centered and distance-adapted.

[0082] In its specific implementation, the control unit assesses the risk of line-of-sight occlusion by combining the motion state of the main manipulator and the positional relationship of the auxiliary vision arm relative to the target. As the occlusion risk increases, the dynamic anti-occlusion cost increases accordingly to guide the auxiliary vision arm in adjusting its observation pose. The field-of-sight centering cost is constructed based on the target's deviation from the current field of view to characterize whether the target remains in a suitable observation position. The observation distance cost is constructed based on the distance relationship between the auxiliary vision arm and the target, used to bring the current observation distance closer to the required observation distance for the operation.

[0083] In the existing technology, the root cause of the failure of fixed global cameras is that the viewing angle is fixed and cannot bypass the occlusion of the main operating arm itself, while the root cause of the failure of end vision is that the field of view shrinks when close to the target and there is a lack of global environmental reference.

[0084] This application's steps incorporate field-of-view centering, observation distance, and dynamic anti-occlusion into a unified cost function, transforming the issues of target clarity, distance, and occlusion by linkages into a single, real-time solvable optimization problem. Therefore, the movement of the auxiliary vision arm is no longer a simple target tracking action, but an active pose adjustment process driven by multiple observation constraints. This enables the maintenance of continuous, visible target observation even in environments with continuous main arm movement, dense equipment, and confined spaces.

[0085] S103. Generate optimized control commands for the auxiliary vision arm based on the multi-objective optimization function and the zero-space control method.

[0086] Among them, the auxiliary vision arm is used to carry image acquisition equipment and undertake active observation tasks. Its optimized control commands can be joint speed commands, joint increment commands, or end effector Cartesian speed commands.

[0087] The null space control method is used to map constraints such as observation distance optimization and dynamic anti-occlusion to the redundant degrees of freedom of the auxiliary vision arm while ensuring the achievement of the main observation task. This allows multiple targets to be executed in a coordinated manner under the same control framework. Among them, the main observation task corresponds to the centering control of the target in the image. The mapping relationship between the target deviation and the camera motion velocity can be established through the visual interaction matrix. Then, the basic velocity components that satisfy the main task can be obtained by calculating the pseudo-inverse of the visual interaction matrix.

[0088] The gradient of the multi-objective optimization function is used to characterize which direction the current auxiliary vision arm pose adjustment can reduce the overall cost, while the null projection ensures that the additional optimization components will not violate the main task's constraint on the target centering, and finally generates optimized control commands that can be directly issued to the auxiliary vision arm actuator.

[0089] In practice, the system's control unit first constructs a target feature error vector based on the current observation results and then calculates the corresponding visual interaction matrix according to the camera imaging model. The visual interaction matrix describes the changes in the target within the observation field of view as the camera moves. The control unit performs a pseudo-inverse operation on the visual interaction matrix to obtain the camera spatial velocity command vmain (Artificial Intelligence) that satisfies the image centering task.

[0090] If the degrees of freedom of the auxiliary vision arm are greater than the minimum degrees of freedom required to complete the main vision task, there is a redundant degree of freedom space. The control unit continues to calculate the gradient of the multi-objective optimization function J with respect to the joint variables or end-effector pose variables of the auxiliary vision arm, forming an additional optimization direction vsub. This additional optimization direction includes the direction of reducing observation distance error and the direction of dynamic anti-occlusion adjustment. When the distance between the camera and the target is not well matched, the corresponding distance correction component in vsub increases; when the main manipulator link is close to the line of sight, the corresponding avoidance component in vsub increases.

[0091] Subsequently, the control unit constructs a null-space projection matrix N = I - L + L, where I is the identity matrix, L is the visual interaction matrix, and L + is its pseudo-inverse. This projection matrix is ​​used to map the additional optimization directions to a redundant space that does not affect the main visual task. The control unit generates a comprehensive camera speed command for the auxiliary visual arm based on vcmd = vmain + λNvsub, where λ is an additional task adjustment coefficient used to control the intensity of the null-space optimization components.

[0092] If the target is already roughly centered in the field of view, vmain gradually decreases. At this point, the distance and anti-occlusion optimization components in null space can still function, allowing the auxiliary vision arm to find a safer observation position while maintaining the target's stable centering. After the integrated camera velocity command is generated, it is converted into joint velocity commands through the inverse mapping of the auxiliary vision arm's Jacobian matrix. These commands are then processed in conjunction with the motion constraints of each joint to obtain the final optimized control command, which is then sent to the auxiliary vision arm's servo driver for execution.

[0093] At the execution level, the auxiliary vision arm receives optimized control commands and adjusts its joint angles according to the control cycle. After each joint update, the image acquisition device reacquires the target image, and the control unit recalculates the target deviation, observation distance, and the relative relationship between the line of sight of the main manipulator and the auxiliary vision arm, thus forming a closed-loop iterative control.

[0094] If a target is detected to temporarily leave the edge of the field of view during a certain period, the control unit first maintains the weight of the main task, so that the auxiliary vision arm can complete the target re-acquisition first; if the risk of occlusion is detected to be increasing, the gradient corresponding to the dynamic anti-occlusion is increased, driving the auxiliary vision arm to adjust the observation pose along the null space direction, thereby restoring the unobstructed observation path.

[0095] For close-range operation scenarios of live equipment in substations, this closed-loop control process can run synchronously with the main operating arm control cycle, enabling the auxiliary vision arm to adjust its observation posture in real time in accordance with the continuous movement of the main operating arm.

[0096] This step achieves task decoupling through zero-space control, directly addressing the control challenges in a dual-arm collaborative environment. If the auxiliary vision arm is controlled solely for target centering, it may remain at its original field of view and cause occlusion when the main manipulator's linkage crosses the vicinity of the line of sight; if the auxiliary vision arm is moved solely for anti-occlusion, the target may deviate from the image center and affect the accuracy of the observation.

[0097] This application sets target centering as the primary task and observation distance and dynamic anti-occlusion as secondary tasks in zero space. This allows the auxiliary vision arm to perform obstacle avoidance and distance correction using redundant degrees of freedom without disrupting continuous target alignment. Therefore, it can continuously provide unobstructed and appropriately scaled observation images when the main manipulator is performing temperature measurement, detection, or close-range operations.

[0098] The dual-arm collaborative vision optimization method provided in this application integrates the main operating arm's operational requirements, target spatial pose, main operating arm's motion state, and auxiliary vision arm's observation state into a real-time optimization control link. The auxiliary vision arm can continuously adjust its observation pose around the target and actively avoid occlusion paths based on the main operating arm's dynamic movement. Therefore, in scenarios with dense equipment, narrow passages, and continuous main operating arm movement in substations, it maintains the target's stable visibility in the imaging field of view, ensures the camera maintains the necessary observation distance for the task, and generates corresponding avoidance control quantities when the linkage approaches the line of sight. This supports the main operating arm in completing close-range detection, temperature measurement, and continuous global perception and operational safety assurance during operation.

[0099] Figure 2 This is a flowchart illustrating Embodiment 2 of the dual-arm collaborative vision optimization method provided in this application. Figure 2 As shown, in Figure 1 Based on the implementation examples, and using the end-task instructions and target feature poses, a multi-objective optimization function is constructed, including field-of-view centering, observation distance, and dynamic anti-occlusion, comprising:

[0100] S201. Construct the kinematic capsule envelope model of the main manipulator.

[0101] Among them, the kinematic capsule envelope model is used to characterize the spatial occupancy of each link of the main operating arm. After the links are geometrically simplified by using the capsule, a unified spatial constraint can be provided for occlusion prediction and obstacle avoidance calculation.

[0102] In one possible implementation, the joint angle data of the main manipulator is obtained; based on the joint angle data and the forward kinematics calculation method, the end positions of each link in the main manipulator are calculated; and a kinematic capsule envelope model is constructed according to the end positions of each link in the main manipulator.

[0103] Among them, joint angle data is used to characterize the posture state of each joint of the main manipulator at the current moment. It is usually collected in real time by the angle encoder at each joint, or calculated by the control system based on the motion control command and feedback state of the main manipulator.

[0104] Forward kinematics calculation is used to map joint angle data into the pose information of the link in space. In this embodiment, a forward kinematics solution method based on DH (Denavit-Hartenberg) parameters can be adopted to transform the input position transformation matrix of each joint rotation angle, link length, link torsion angle and joint offset, thereby outputting the end position of each link.

[0105] A link is a mechanical connection between adjacent joints in the main control arm, and its end position is used to determine the extension direction and occupied area of ​​the link in space.

[0106] In practical implementation, after the system control unit acquires the joint angle data of the main operating arm, it inputs the data into the pre-established kinematic model of the main operating arm and calculates the coordinate transformation relationship step by step according to the DH parameters corresponding to each joint to obtain the spatial coordinates from the base to the end of each link.

[0107] Subsequently, a capsule envelope is constructed using the end positions of each link as endpoints and the geometric connection relationships of adjacent links. The axis of the capsule corresponds to the centerline of the link, and the hemispherical surfaces at both ends of the capsule correspond to the spatial expansion regions at the link ends. The radius of the capsule is set based on the link dimensions and safety redundancy. This envelope model is updated synchronously with the actual posture of the main manipulator, ensuring that the spatial occupancy of the links at each moment can be continuously represented.

[0108] For example, a global base coordinate system for the substation robot chassis is defined. The coordinate system of the end effector of the main manipulator Visual arm camera coordinate system .

[0109] The position of the target detection point in the camera coordinate system is The transformation matrix of the camera in the global state is obtained through hand-eye calibration and forward kinematics. Calculate its global position vector :

[0110]

[0111] The position of the camera's optical center in the global coordinate system is denoted as Define the line-of-sight vector Let the unit vector be the direction from the camera's optical center to the target being manipulated, as shown in the following formula:

[0112]

[0113] To prevent the main control arm from obstructing the view, the main control arm... Each link is simplified to a radius of... The cylindrical body (capsule envelope).

[0114] Let the first The coordinates of the two joints of the link are respectively and The connecting rod axis can be expressed as a parametric equation:

[0115]

[0116] The spatial ray formed by combining the line-of-sight vectors:

[0117]

[0118] Calculate the line-of-sight vector in real time using the spatial geometric distance formula. To the main operating arm The shortest vertical distance between the central axes of each link .

[0119] The kinematic capsule envelope model constructed in this application can convert the discrete joint postures of the main manipulator into a continuous geometric envelope, providing a unified spatial representation basis for subsequent dynamic obstacle model updates. Since the model is directly generated from joint angle data and forward kinematics results, it can reflect the changes in link position in real time with the movement of the main manipulator and output envelope results suitable for occlusion analysis and safety constraint calculation, thereby supporting the construction of subsequent multi-objective optimization functions.

[0120] S202. Based on the end-task instructions, dynamically update the kinematic capsule envelope model to obtain the dynamic obstacle model.

[0121] In this step, the system's control unit combines the joint drive quantity, speed command, and attitude adjustment quantity corresponding to the end-effector task command to make real-time corrections to the position, orientation, and envelope length of the capsule, so that it changes synchronously with the main manipulator when performing grasping, alignment, or temperature measurement actions, thereby obtaining a dynamic obstacle model.

[0122] S203. Based on the dynamic obstacle model and target feature pose, a multi-objective optimization function is constructed, which includes field of view centering, observation distance, and dynamic anti-occlusion.

[0123] In this step, the system's control unit inputs the dynamic obstacle model and the target feature pose into a multi-objective optimization function, calculates the deviation between the field of view centering cost and the target feature point, the observation distance between the auxiliary visual arm and the target, and the occlusion relationship between the capsule and the visual optical axis, and uses these three as optimization terms to participate in a unified solution, thereby outputting an objective function expression that simultaneously satisfies the field of view centering, distance constraints, and dynamic anti-occlusion constraints.

[0124] In one possible implementation, the system first calculates the spatial position of the target point in the coordinate system of the auxiliary vision arm based on the target feature pose, and obtains the line of sight vector by connecting the optical axis direction of the auxiliary vision arm with the target point. At the same time, the difference between the target pixel coordinates and the image center coordinates is used as the coordinate deviation, and then the expected observation distance is determined by combining the safety observation scale between substation equipment.

[0125] The dynamic obstacle model is formed by the superposition of capsules of each link of the main control arm. The capsule can be composed of a cylindrical segment and two hemispherical segments at both ends. Its radius value corresponds to the external dimensions of the link. The model is updated in real time as the joint angle changes.

[0126] The system constructs a dynamic anti-occlusion cost based on the minimum distance between the line-of-sight vector and the dynamic obstacle model. When the link envelope approaches the line-of-sight channel, the cost increases and an avoidance constraint is formed.

[0127] The field-of-view centering cost is obtained by mapping the coordinate deviation to camera motion error using the visual interaction matrix, which makes the target features closer to the image center.

[0128] The observation distance cost is based on the difference between the current distance and the expected observation distance, and the auxiliary vision arm pose is iteratively updated using the gradient descent method until the distance error converges.

[0129] The aforementioned dynamic anti-occlusion costs, field-of-view centering costs, and observation distance costs are weighted and summed to form a multi-objective optimization function, which is output as the optimization control evaluation quantity for the auxiliary visual arm. This function is updated synchronously during the movement of the main operating arm, enabling the auxiliary visual arm to maintain an appropriate observation distance while keeping the target centered, and to adjust the viewing angle in real time according to the dynamic obstacle model to avoid occlusion. Using this method, the auxiliary visual arm can continuously acquire target information in the confined space of a substation, reduce observation interruptions caused by linkage occlusion, and make image tracking and target alignment more stable in close-range operations.

[0130] Figure 3 This is a flowchart illustrating Embodiment 3 of the dual-arm collaborative vision optimization method proposed in this application. Figure 3 As shown, in Figure 1 Based on the implementation examples, and according to the multi-objective optimization function and the null space control method, optimized control instructions for the auxiliary vision arm are generated, including:

[0131] S301. Calculate the gradient vector based on the multi-objective optimization function.

[0132] The gradient vector represents the optimization direction obtained from the multi-objective optimization function, which is used to drive the auxiliary vision arm to adjust its pose.

[0133] In practice, the multi-objective optimization function is composed of a weighted average of the field of view centering cost, the observation distance cost, and the dynamic anti-occlusion cost. The control unit performs differentiation or difference calculation on this function according to the current joint state and image features to obtain the gradient vector representing the descent direction.

[0134] S302. Based on the pseudo-inverse algorithm of the visual interaction matrix, generate the main task speed command.

[0135] The pseudo-inverse algorithm is used to solve the inverse mapping relationship of the visual interaction matrix in order to calculate the motion control quantity from the image error.

[0136] The main task speed command is used to ensure that the auxiliary vision arm completes the motion control output for the main task of centering the field of view.

[0137] In the specific implementation, the visual interaction matrix is ​​established by the current image features and the camera pose. Its pseudo-inverse is solved by singular value decomposition. Combined with the image feature error, the camera spatial velocity command is generated and used as the main task velocity command.

[0138] S303. Based on the zero-space control method, the gradient vector is superimposed on the main task speed command to generate the optimized control command for the auxiliary vision arm.

[0139] At a specific time, the gradient vector is mapped to the redundant degrees of freedom space corresponding to the main task constraints using the null space control method. This is then superimposed with the main task velocity command to output an optimized control command in the form of joint velocity. After this control command is sent to the auxiliary vision arm, the auxiliary vision arm adjusts its observation pose according to the optimized direction while maintaining the consistency of the main task motion.

[0140] In one possible implementation, the optimized control instructions for generating the auxiliary vision arm can be the weight coefficients of a multi-objective optimization function that are dynamically adjusted based on a reinforcement learning algorithm.

[0141] Among them, the reinforcement learning algorithm is used to adaptively update the weight coefficients in the multi-objective optimization function. The weight coefficients are used to characterize the relative importance of field of view centering, observation distance and dynamic anti-occlusion in the optimization objective.

[0142] In this embodiment, the motion state of the main operating arm, the target feature pose, and the distribution of environmental obstacles can be used as state variables, and the control result after combining weight coefficients can be used as action variables. A reward function is constructed based on the target centering degree, occlusion risk change, and observation stability after the auxiliary vision arm outputs the result, so that the weight coefficients can be adjusted in real time according to the openness of the substation site, the linkage rotation amplitude, and the target approach state.

[0143] In densely obscured areas, the system increases the weight of the dynamic anti-obscuration term and increases the weight of the observation distance term when the target distance changes significantly, thereby outputting optimized control commands adapted to the current scene. The aforementioned weight updates can be performed online by the control unit within each control cycle. In practical applications, other models of this component can also be selected, and this application does not limit this choice.

[0144] In another possible implementation, the optimized control commands for generating the auxiliary vision arm can be based on a motion prediction model to determine that the future links of the main manipulator are unknown, and generate an avoidance path.

[0145] Among them, the motion prediction model is used to determine the future link position, attitude or space occupied by the master arm based on the end-effector task command and historical motion data.

[0146] In practice, the control unit can input the joint angles, angular velocities and end effector traject ...

[0147] The avoidance path is output as the trajectory of the auxiliary vision arm's end effector and combined with zero-space control commands, enabling the auxiliary vision arm to complete attitude replanning outside the area that the main manipulator is about to occupy. The prediction model can employ a temporal neural network, a recursive regression model, or a Kalman prediction model. In practical applications, other models of this component can also be selected, and this application does not impose any limitations on this.

[0148] In this way, the auxiliary vision arm can change the response intensity to different constraints when the weight coefficient is adaptively adjusted, and can also replan the avoidance trajectory in advance after predicting the future linkage space of the main operating arm. This enables the auxiliary vision arm to maintain continuous observation and a safe distance in narrow electrical equipment environments, and the output control commands have stronger scene adaptability and foresight.

[0149] The embodiments of this application ensure that the primary task speed command prioritizes the field-of-view centering requirement, while the gradient vector, after null space mapping, does not interfere with the primary task constraints, but only drives the convergence of the secondary optimization target. Therefore, the auxiliary vision arm can continuously obtain stable observations during close-range operations at substations and maintain synchronous perception of the target area and its surrounding environment.

[0150] Figure 4 A three-dimensional diagram illustrating the relationship between the shortest distance and the line-of-sight vector provided in this application. (See diagram below.) Figure 4 As shown, in Figure 2 Based on the previous examples, a multi-objective optimization function is constructed to ensure that the vision arm always maintains the optimal global observation angle. ,include:

[0151] (1) Calculate the cost of centering the field of view :

[0152] Utilizing visual servo feature errors ( The current pixel coordinates of the target on the image plane. Minimize it (with the image center as the center):

[0153]

[0154] (2) Calculate the cost of the optimal observation distance :

[0155] To avoid the visual arm degenerating into "local perception" when too close or causing "insufficient resolution" when too far, the desired observation distance must be maintained. :

[0156]

[0157] (3) Calculate the cost of dynamic anti-shading :

[0158] When the main manipulator moves and approaches the line-of-sight vector, its potential energy increases sharply, forcing the vision arm to actively avoid it. A safe avoidance threshold is set. :

[0159]

[0160] The overall objective function is:

[0161]

[0162] in These are adaptive weighting coefficients.

[0163] Figure 5 A flowchart illustrating the dual-arm decoupling control law provided in this application is shown. Figure 5 As shown, in Figure 3 and Figure 4 Based on this, the constructed multi-objective optimization function is decoupled for task priority.

[0164] The main task of the visual arm is to ensure that the field of view is centered (i.e., minimized). The primary task is to ensure that the target is not lost; the secondary task is to ensure the optimal observation distance (optimization). ) and active anti-occlusion avoidance (optimized) By using null space projection technology, attitude adaptive adjustment can be achieved by utilizing the gradient descent of the secondary task without interfering with the main task.

[0165] (1) Main control arm control law:

[0166] The main operating arm is primarily responsible for performing main operational tasks such as temperature measurement or partial discharge measurement in the substation. Let its desired end-effector velocity be... The joint velocity command is directly obtained from the Jacobian pseudo-inverse:

[0167]

[0168] in, Jacobian matrix for the main manipulator It is its Moore-Penrose pseudo-reversal.

[0169] (2) Visual arm master task control law:

[0170] To minimize the field-of-view centering cost function Image-based visual servoing (IBVS) is introduced. Let the coordinates of the current image feature points be... The desired image center coordinates are The characteristic error is .

[0171] Constructing a visual interaction matrix (Image Jacobian) This makes the image feature velocity and camera space velocity... The relationship satisfies:

[0172]

[0173] To make the error It exhibits exponential convergence, and the desired camera spatial velocity command is designed as follows:

[0174] ,

[0175] in, For the corresponding The positive proportional gain coefficient, It is the pseudo-inverse of the visual interaction matrix.

[0176] At this point, the joint velocities required by the primary task of the vision arm are:

[0177]

[0178] (3) Gradient descent for secondary tasks of the vision arm:

[0179] For tasks involving optimal observation distance and dynamic anti-occlusion, the chain rule is used to calculate the objective function with respect to the visual arm joint. gradient vector:

[0180] 1) Distance optimization gradient : Calculate the deviation of the current distance from the optimal distance The partial derivatives of .

[0181]

[0182] in, For the visual arm Jacobian matrix The first three lines (i.e., the translated Jacobi part).

[0183] 2) Anti-occlusion repulsion gradient When the line-of-sight vector is relative to the distance between the main arm and the link. At that time, the repulsive potential gradient is triggered:

[0184]

[0185] in,

[0186] (4) Dual-arm coordinated zero-space projection integrated control law:

[0187] The control commands of the primary task are combined with the negative gradient of the secondary cost function, and the projection operator is used. To ensure that the gradient descent of the secondary task does not interfere with the centering of the field of view of the primary task, the integrated visual arm joint control law is as follows:

[0188]

[0189] In the above formula, This ensures that the camera is always pointed at the target, achieving smooth tracking; This ensures that obstacle avoidance and distance adjustment by the camera occur only in redundant degrees of freedom that do not affect the main tracking task. (Coefficient) For corresponding adaptive weights The control gain mapping can be dynamically amplified according to the degree of linkage approximation. To ensure emergency avoidance.

[0190] Figure 6 This is a schematic diagram of the dual-arm collaborative vision optimization device provided in this application. Figure 6 As shown, the dual-arm collaborative vision optimization device 60 provided in this embodiment is used for substation operation and maintenance, and includes:

[0191] The main manipulator data module 601 is used to acquire the end-effector task commands and target feature poses of the main manipulator.

[0192] The optimization function construction module 602 is used to construct a multi-objective optimization function that includes field of view centering, observation distance, and dynamic anti-occlusion based on the end-task instructions and target feature pose.

[0193] The optimization instruction generation module 603 is used to generate optimization control instructions for the auxiliary vision arm based on the multi-objective optimization function and the zero-space control method.

[0194] In one possible implementation, the optimization function construction module 602 is further specifically used for:

[0195] Construct a kinematic capsule envelope model of the main manipulator;

[0196] Based on the end-task instructions, the kinematic capsule envelope model is dynamically updated to obtain the dynamic obstacle model;

[0197] Based on the dynamic obstacle model and target feature pose, a multi-objective optimization function is constructed, which includes field of view centering, observation distance, and dynamic anti-occlusion.

[0198] In one possible implementation, the optimization function construction module 602 is further specifically used for:

[0199] Obtain the joint angle data of the main manipulator;

[0200] Based on joint angle data and positive kinematics calculation, the end positions of each link in the main manipulator are calculated.

[0201] Based on the end positions of each link in the main control arm, a kinematic capsule envelope model is constructed.

[0202] In one possible implementation, the optimization function construction module 602 is further specifically used for:

[0203] Based on the target feature pose, determine the visual arm's gaze vector, coordinate deviation, and expected observation distance;

[0204] Based on the dynamic obstacle model and line-of-sight vector, a dynamic anti-occlusion cost is generated.

[0205] Based on the visual interaction matrix and coordinate deviation, a field-of-view centering cost is generated.

[0206] Based on gradient descent and the desired observation distance, an observation distance cost is generated.

[0207] A multi-objective optimization function is constructed based on dynamic anti-occlusion cost, field-of-view centering cost, and observation distance cost.

[0208] In one possible implementation, the main operating arm data module 601 is further specifically used for:

[0209] Environmental point cloud data is collected using the image acquisition device on the main operating arm;

[0210] Based on environmental point cloud data, extract the pixel coordinates of target features;

[0211] Based on the camera pose transformation matrix, the pixel coordinates of the target feature are transformed into the target feature pose in the global base coordinate system.

[0212] In one possible implementation, the optimized instruction generation module 603 is further specifically used for:

[0213] Calculate the gradient vector based on the multi-objective optimization function;

[0214] A pseudo-inverse algorithm based on the visual interaction matrix is ​​used to generate the main task speed command.

[0215] Based on the zero-space control method, the gradient vector is superimposed on the main task speed command to generate the optimized control command for the auxiliary vision arm.

[0216] In one possible implementation, optimized control instructions for the auxiliary vision arm are generated, including:

[0217] The weight coefficients of the multi-objective optimization function can be dynamically adjusted based on reinforcement learning algorithms, or an avoidance path can be generated based on the unknown future linkage of the main control arm determined by a motion prediction model.

[0218] The dual-arm collaborative vision optimization device provided in this embodiment extracts the current operational intent and target space status of the main operating arm in real time through the main operating arm data module, providing the auxiliary vision arm with a perception basis synchronized with actual operation and maintenance actions. The optimization function construction module integrates field of view centering, observation distance, and dynamic anti-occlusion into a multi-objective optimization function, enabling the auxiliary vision arm to consider clear target imaging, appropriate shooting scale, and avoidance requirements of the main operating arm linkage and surrounding electrical equipment when following observation. The optimization command generation module further combines the zero-space control method to output optimized control commands, thereby driving the auxiliary vision arm to continuously adjust the observation posture without interfering with the main operating arm's predetermined task execution. This makes it less likely for the target to leave the field of view and reduces the probability of occlusion during operation, thereby improving the continuous global perception capability and close-range operation safety in the narrow and dense environment of the substation. Therefore, it is beneficial to ensure the stable operation of detection, temperature measurement, and fine operations.

[0219] Figure 7 This is a schematic diagram of the structure of the dual-arm collaborative vision optimization device provided in this application. Figure 7 As shown, the dual-arm collaborative vision optimization device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the dual-arm collaborative vision optimization device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0220] The device can be deployed in the control unit of the substation operation and maintenance robot. The memory stores the program, parameters and target model corresponding to the dual-arm collaborative vision optimization method. After the processor calls the computer to execute the instructions, it can calculate around the end task instructions and target feature pose of the main operating arm, and further generate the optimization control instructions of the auxiliary vision arm, so that the auxiliary vision arm can synchronously adjust the observation pose during the continuous operation of the main operating arm.

[0221] Since the processor executes the above method, it can take into account the centering of the field of view, the observation distance and the dynamic anti-occlusion constraint, so that the target area, the robotic arm itself and the surrounding electrical equipment can obtain more continuous visual coverage, thereby reducing the risk of occlusion, target loss and collision. Therefore, it can improve the safety, stability and accuracy of close-range detection, temperature measurement and operation in narrow and dense substation environments.

[0222] In the specific implementation process, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to execute the above-mentioned dual-arm collaborative vision optimization method.

[0223] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0224] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0225] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0226] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0227] This application also provides a dual-arm collaborative vision optimization system, including a computer program, a main manipulator arm, and an auxiliary vision arm;

[0228] The computer program implements the methods provided above when executed by the processor.

[0229] The main operating arm is responsible for the detection, temperature measurement and close-range operation of live equipment such as high-voltage circuit breakers, disconnect switches and surge arresters. The auxiliary vision arm independently adjusts the observation posture. The computer program coordinates the movements of the two arms and visual perception accordingly, so as to maintain continuous observation of the target area, the robotic arm body and surrounding obstacles during the continuous movement of the main operating arm.

[0230] When the computer program is executed by the processor, it realizes the multi-objective optimization and control generation in the above method, enabling the auxiliary vision arm to perform collaborative tracking around the field of view centering, observation distance and dynamic anti-occlusion, thereby reducing the risk of observation interruption caused by the main operating arm linkage occlusion and the limited field of view of the end vision. This allows the system to take into account both close-range operation accuracy and global environmental perception capability, thereby improving the safety, stability and target observation reliability of operation in narrow and dense substation scenarios.

[0231] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the aforementioned dual-arm collaborative vision optimization method.

[0232] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0233] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0234] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0235] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0236] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0237] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0238] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0239] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A dual-arm collaborative vision optimization method, characterized in that, For substation operation and maintenance, the method includes: Acquire the end-effector command and target feature pose of the main manipulator; Based on the end-task command and the target feature pose, a multi-objective optimization function is constructed, which includes field of view centering, observation distance, and dynamic anti-occlusion. Based on the multi-objective optimization function and the zero-space control method, optimized control commands for the auxiliary vision arm are generated.

2. The dual-arm collaborative vision optimization method according to claim 1, characterized in that, Based on the end-task command and the target feature pose, a multi-objective optimization function is constructed, including field-of-view centering, observation distance, and dynamic anti-occlusion. Construct a kinematic capsule envelope model of the main manipulator; Based on the terminal task instructions, the kinematic capsule envelope model is dynamically updated to obtain a dynamic obstacle model; Based on the dynamic obstacle model and the target feature pose, a multi-objective optimization function is constructed, which includes field of view centering, observation distance, and dynamic anti-occlusion.

3. The dual-arm collaborative vision optimization method according to claim 2, characterized in that, Constructing the kinematic capsule envelope model of the main manipulator includes: Obtain the joint angle data of the main manipulator; Based on the joint angle data and the forward kinematics calculation method, the end positions of each link in the main operating arm are calculated; Based on the end positions of each link in the main control arm, a kinematic capsule envelope model is constructed.

4. The dual-arm collaborative vision optimization method according to claim 2, characterized in that, Based on the dynamic obstacle model and the target feature pose, a multi-objective optimization function is constructed, including field-of-view centering, observation distance, and dynamic anti-occlusion, comprising: Based on the target feature pose, the line-of-sight vector, coordinate deviation, and expected observation distance of the visual arm are determined; Based on the dynamic obstacle model and the line-of-sight vector, a dynamic anti-occlusion cost is generated; Based on the visual interaction matrix and the coordinate deviation, a field-of-view centering cost is generated; Based on the gradient descent method and the desired observation distance, an observation distance cost is generated. Based on the dynamic anti-occlusion cost, the field-of-view centering cost, and the observation distance cost, a multi-objective optimization function is constructed.

5. The dual-arm collaborative vision optimization method according to claim 1, characterized in that, Obtain the target feature pose of the main manipulator, including: Environmental point cloud data is collected using the image acquisition device on the main operating arm; Based on the environmental point cloud data, the pixel coordinates of the target features are extracted; Based on the camera pose transformation matrix, the pixel coordinates of the target feature are converted into the target feature pose in the global base coordinate system.

6. The dual-arm collaborative vision optimization method according to claim 1, characterized in that, Based on the multi-objective optimization function and the null space control method, optimized control instructions for the auxiliary vision arm are generated, including: Calculate the gradient vector based on the multi-objective optimization function; A pseudo-inverse algorithm based on the visual interaction matrix is ​​used to generate the main task speed command. According to the null space control method, the gradient vector is superimposed on the main task speed command to generate the optimized control command for the auxiliary vision arm.

7. The dual-arm collaborative vision optimization method according to claim 6, characterized in that, Generate optimized control instructions for the auxiliary vision arm, including: The weight coefficients of the multi-objective optimization function are dynamically adjusted based on reinforcement learning algorithms, or an avoidance path is generated based on the unknown future linkage of the main manipulator determined by a motion prediction model.

8. A dual-arm collaborative vision optimization device, characterized in that, Used for substation operation and maintenance, including: The main manipulator data module is used to acquire the end-effector task commands and target feature poses of the main manipulator. The optimization function construction module is used to construct a multi-objective optimization function that includes field of view centering, observation distance, and dynamic anti-occlusion based on the end task instruction and the target feature pose; The optimization instruction generation module is used to generate optimization control instructions for the auxiliary vision arm based on the multi-objective optimization function and the null space control method.

9. A dual-arm collaborative vision optimization device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A dual-arm collaborative vision optimization system, characterized in that, Includes computer programs, a main control arm, and an auxiliary vision arm; When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images