Transformer oil taking docking method and system fusing deep learning and force feedback

Abstract

The application belongs to the technical field of electric power robots. A transformer oil taking docking method and system fusing deep learning and force feedback are proposed. A lightweight multi-task model is used to synchronously analyze images and point clouds, to obtain an oil taking port semantic mask, key points and an initial pose. A voxel map containing forbidden areas is constructed based on the mask, and a collision-free trajectory is planned. Cartesian impedance control is used in the initial contact stage to convert force deviation into pose correction to achieve flexible coarse adjustment. After the force is stable, the system switches to a force-position hybrid closed-loop mode, and the joint amount is dynamically corrected according to the real-time force error to maintain torque stability. Through the deep cooperation of vision and force control, the application realizes full-process adaptive control from macro obstacle avoidance to micro precise docking, effectively avoids rigid collision, and significantly improves the precision and flexibility of oil taking operations in complex environments.

Description

Technical Field

[0001] This invention relates to the field of electric robot technology, specifically to a transformer oil sampling docking method and system that integrates deep learning and force feedback. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In the field of power system operation and maintenance, transformer oil sampling is a crucial step in monitoring equipment health and preventing major faults. With the advancement of smart grid construction, utilizing robots to replace manual labor in high-risk oil sampling operations has become an industry trend. Existing automated oil sampling solutions mainly rely on robotic arm systems, the core of which lies in acquiring the oil sampling port location information through vision sensors to guide the robotic arm to complete the docking operation. Traditional technical approaches often employ visual algorithms based on geometric features or manually designed features, combined with point cloud data acquired by depth cameras, to calculate the target pose through point-line matching or template matching. Simultaneously, to cope with the contact force impact during the docking process, some solutions introduce basic force control strategies, attempting to add a layer of force protection mechanism on top of visual guidance. These technologies have, to a certain extent, automated oil sampling operations, constituting the main technical form of current robotic arm-assisted operation and maintenance, laying the foundation for improving power inspection efficiency.

[0004] However, existing technologies still suffer from a core problem when facing the complex and ever-changing actual operating conditions of substations: the disconnect between visual perception and force control execution makes it difficult to achieve high-precision flexible docking. Traditional visual algorithms are not robust to changes in lighting, equipment aging, and background interference, and often lack semantic understanding capabilities, making it difficult to accurately locate the oil intake port under robot pose deviations caused by rugged terrain. More importantly, existing technologies have failed to establish a deep collaborative mechanism between visual perception and force feedback. The vision system only provides a static initial pose and cannot dynamically correct the trajectory based on real-time force information at the moment of contact. Meanwhile, simple force control lacks global pose prior, making it prone to rigid collisions or alignment failures when the robotic arm contacts the high-precision docking channel. This disconnect between perception and control makes it difficult for the robotic arm to adaptively complete the entire process from rough approach to fine contact in complex environments with multi-dimensional position and angle deviations, failing to meet the stringent requirements of millimeter-level precision and compliance for oil intake port sealing docking. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a transformer oil sampling docking method and system that integrates deep learning and force feedback. By constructing a deep closed-loop collaborative mechanism of visual perception and force control execution, the accuracy, compliance, and success rate of oil sampling operations under complex working conditions are significantly improved.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a transformer oil sampling docking method that integrates deep learning and force feedback.

[0008] A transformer oil sampling and docking method integrating deep learning and force feedback includes the following process:

[0009] Multi-task visual perception and pose calculation steps: acquire real-time image and point cloud data of the oil tank panel, input it into a lightweight multi-task deep learning model, and simultaneously output the semantic segmentation mask and initial three-dimensional pose parameters of the oil outlet. Based on the hand-eye conversion matrix, the initial three-dimensional pose parameters are converted into target pose information in the tool coordinate system of the robotic arm.

[0010] Visual-guided collision-free trajectory planning steps: Construct a 3D voxel map containing restricted areas based on semantic segmentation mask, combine target pose information and robotic arm kinematic constraints, and use path planning algorithm to generate a collision-free reference trajectory from the current position to the oil outlet;

[0011] Impedance-compliant contact control steps: Control the robotic arm to move along a collision-free reference trajectory. When the detected contact force is less than the preset initial threshold, the deviation between the measured contact force and the expected contact force is converted into a position and attitude correction amount based on the Cartesian space impedance model, and the robotic arm end is driven to complete flexible contact and coarse attitude adjustment.

[0012] Force-position hybrid closed-loop fine-tuning steps: When the measured contact force is stable within the desired contact force range, switch to closed-loop PID control mode, output joint control quantity based on the real-time error between the measured contact force and the desired contact force, dynamically correct the robot arm motion trajectory to maintain torque stability during docking, and complete high-precision docking of the oil intake port.

[0013] In one implementation of the first aspect of the present invention, during the training process of the lightweight multi-task deep learning model, a weighted fusion multi-task loss function is used for joint optimization. The weighted fusion multi-task loss function is as follows:

[0014] ;

[0015] in, This is the total loss value. The cross-entropy loss is used for semantic segmentation branches. The mean squared error loss for the keypoint detection branch. The reprojection error loss is used for the 3D pose estimation branch. , , These are the dynamic weight coefficients for each branch.

[0016] As a further limitation of the first aspect of the invention, the cross-entropy loss of semantic segmentation branches The calculation formula is:

[0017] ;

[0018] in, Total number of pixels For the number of categories, For pixels Category The true label, Predict pixels for the model Category The probability of.

[0019] As a further limitation of the first aspect of the present invention, the mean square error loss of the key point detection branch The calculation formula is:

[0020]

[0021] in, The total number of key points. For the first The true coordinates of each key point For the first Predicted coordinates of key points This represents the Euclidean distance.

[0022] As a further limitation of the first aspect of the invention, the reprojection error loss of the 3D pose estimation branch The calculation formula is:

[0023] ;

[0024] in, For the first The three-dimensional coordinates of the key points For rotation matrix, It is a translation vector. This is a projection function that converts three-dimensional coordinates into two-dimensional image coordinates. For the first The true two-dimensional coordinates of each key point.

[0025] As a further limitation of the first aspect of the present invention, the dynamic weighting coefficient , , During training, the system adaptively adjusts the loss based on the rate of change of the validation set loss for each branch. If the rate of decrease of the validation loss for a certain branch is lower than a preset threshold, the dynamic weight coefficient corresponding to that branch is increased.

[0026] In one implementation of the first aspect of the present invention, in the force-position hybrid closed-loop fine-tuning step, the discretized control formula of the closed-loop PID regulation mode is:

[0027] ;

[0028] in, For the first Real-time control quantity for each control cycle. , , These are the proportional, integral, and differential coefficients, respectively. For the first The real-time error between the actual force and the expected force in each control cycle To control the cycle.

[0029] In one implementation of the first aspect of the present invention, during the transition from the impedance compliant contact control step to the force-position hybrid closed-loop fine-tuning step, a first-order low-pass filtering strategy is used to fuse the control quantities, and the fusion formula is as follows:

[0030] ;

[0031] in, For the control quantity that is ultimately sent to the execution layer, To control the output quantity by impedance. For the output of the PID controller, The weighting coefficients increase linearly from 0 to 1.

[0032] Secondly, the present invention provides a transformer oil sampling docking system that integrates deep learning and force feedback.

[0033] A transformer oil sampling docking system integrating deep learning and force feedback, comprising:

[0034] The multi-task visual perception and pose calculation unit is configured to: acquire real-time images and point cloud data of the oil tank panel, input them into a lightweight multi-task deep learning model, simultaneously output the semantic segmentation mask and initial three-dimensional pose parameters of the oil outlet, and convert the initial three-dimensional pose parameters into target pose information in the robotic arm tool coordinate system based on the hand-eye conversion matrix.

[0035] The visually guided collision-free trajectory planning unit is configured to: construct a 3D voxel map containing restricted areas based on semantic segmentation masks, combine target pose information and robotic arm kinematic constraints, and use path planning algorithms to generate a collision-free reference trajectory from the current position to the oil outlet;

[0036] The impedance compliant contact control unit is configured to: control the robotic arm to move along a collision-free reference trajectory; when the detected contact force is less than a preset initial threshold, convert the deviation between the measured contact force and the expected contact force into a position and attitude correction amount based on the Cartesian space impedance model, and drive the end of the robotic arm to complete compliant contact and coarse attitude adjustment.

[0037] The force-position hybrid closed-loop fine-tuning unit is configured to switch to closed-loop PID control mode when the measured contact force is stable within the desired contact force range. Based on the real-time error between the measured contact force and the desired contact force, the joint control quantity is output to dynamically correct the movement trajectory of the robotic arm to maintain torque stability during the docking process and complete high-precision docking of the oil intake port.

[0038] Thirdly, the present invention provides an oil extraction robot, comprising: a processor and a computer-readable storage medium;

[0039] A processor, adapted to execute computer programs;

[0040] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the transformer oil sampling docking method integrating deep learning and force feedback, which is the first aspect of this invention.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] This invention innovatively proposes a transformer oil sampling docking method integrating deep learning and force feedback. By constructing a deep closed-loop collaborative mechanism of visual perception and force control execution, it deeply integrates the initial pose prior provided by vision with real-time force feedback, solving the problems of easy collision under dynamic deviations and poor adaptability to complex working conditions caused by single visual guidance. It achieves millimeter-level accuracy and torque stability for oil sampling port docking, improving the accuracy, compliance, and success rate of oil sampling operations under complex working conditions, and meeting the high-precision operation requirements of transformer oil sample collection. Addressing the problems of poor robustness and fragmented perception and control in traditional visual algorithms, this invention utilizes a lightweight multi-task deep learning model to simultaneously output semantic segmentation masks, key point coordinates, and 3D pose. This not only effectively overcomes illumination changes and environmental interference, achieving precise semantic locking of the oil sampling port, but also, through the construction of a voxel map of restricted areas, addresses the issue from the source... The invention plans a collision-free safety trajectory above the head, solving the problem of collisions easily occurring under dynamic deviations in single vision guidance. It innovatively designs a dual-stage control strategy of "impedance-compliant contact" and "force-position hybrid closed-loop fine-tuning," deeply integrating the initial pose prior provided by vision with real-time force feedback. In the initial contact phase, a Cartesian space impedance model is used to achieve coarse posture adjustment and flexible buffering, eliminating rigid impacts. After stable contact, it switches to a force-position hybrid closed-loop mode, dynamically correcting joint motion based on real-time force errors and adaptively compensating for multi-dimensional position and angle deviations. This end-to-end collaborative control enables the robotic arm to autonomously complete the entire process from macroscopic obstacle avoidance to microscopic precision docking in unstructured environments, ensuring millimeter-level accuracy and torque stability at the oil port docking, and solving the problem of docking failure or equipment damage caused by the disconnect between perception and control in existing technologies.

[0043] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0044] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0045] Figure 1 A schematic diagram of the framework of a transformer oil sampling docking method integrating deep learning and force feedback provided as an exemplary embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a transformer oil sampling docking system integrating deep learning and force feedback, provided as an exemplary embodiment of the present invention;

[0047] Figure 3 A schematic diagram of a computer device provided for an exemplary embodiment of the present invention. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0050] This implementation proposes a transformer oil sampling and docking method that integrates deep learning and force feedback, such as... Figure 1 The diagram shows the overall control architecture, which consists of a closed-loop collaborative system formed by a "visual perception loop" on the left and a "force feedback loop" on the right through a bidirectional dynamic coupling mechanism. The visual perception loop acquires images at a frequency of 30fps, and simultaneously completes semantic segmentation, key point detection, and pose estimation through a lightweight multi-task model. Every 100ms, it outputs the target pose and drives the RRT* algorithm to plan a collision-free path, guiding the robotic arm for coarse positioning. The force feedback loop relies on a 1000Hz high-frequency six-dimensional force sensor. During the contact transition phase, it uses impedance compliance control to achieve flexible buffering and initial attitude adjustment, entering a stable state. After the initial operation phase, the system switches to PID control mode and performs fine adjustments in conjunction with force / torque error compensation. The two loops are linked in real time through a two-way interactive mechanism of "abnormal force-triggered visual repositioning" and "dynamic adjustment of force threshold for pose deviation". When the force control detects an abnormal impact, it triggers the vision system to recalculate the pose. The pose drift identified by the vision end will also dynamically correct the threshold parameters of the force control loop, thus forming a fully adaptive closed loop of perception-decision-execution. This ensures high-precision, high-compliance, and high-robust automatic docking of the oil tap in complex unstructured environments.

[0051] In this implementation, to obtain rich and representative data, an Intel RealSense D455 depth camera is used to acquire data from the fuel tank panel. This camera integrates an infrared structured light module and an RGB camera, capable of simultaneously outputting color and depth images with a resolution of 1280×720. The depth measurement range is 0.1-10 meters, with an accuracy of ±2mm. During the acquisition process, a robotic arm is controlled to move the camera around the fuel tank panel, covering a distance of 0.5-2 meters and a ±45° field of view. Scene data is acquired at 200 different locations, with 10 sets of continuous images and point cloud data acquired at each location, totaling 2000 sets of raw data.

[0052] The labeling process utilizes the LabelMe tool for pixel-level semantic segmentation and labeling of the oil extraction port. This involves precisely outlining the target edge using closed polygons, defining two types of labels: "oil extraction port" and "background." The output JSON file is converted into a pixel-level mask with the same dimensions as the original image using a script. Simultaneously, point cloud data is extracted using a depth camera. Preprocessing operations such as pass-through filtering and voxel mesh filtering from the PCL (Point Cloud Library) library are used to remove outliers and redundant data. Then, the least squares method is used to fit the target geometry, accurately labeling the 3D coordinates of the target's center and corner points. After labeling, the data is divided into training, validation, and test sets in an 8:1:1 ratio. A correspondence is established between the detected target keypoint 2D coordinates and the pre-labeled 3D model keypoints. The rotation matrix R and translation vector t are solved by minimizing the reprojection error. To improve computational stability, an iterative PnP algorithm is used to optimize the pose parameters.

[0053] In this implementation, to achieve collaborative optimization of semantic segmentation, keypoint detection, and 3D pose estimation, and to avoid problems such as insufficient feature extraction and gradient competition between tasks caused by single-task training, this invention designs a joint training strategy combining a weighted fusion-type multi-task loss function with dynamic weight adjustment. This strategy integrates the learning requirements of each branch task through a unified loss optimization objective, and dynamically adjusts the weight coefficients of each task according to the training progress. This ensures both the model's learning priority for the core task and the convergence speed and accuracy of each task, ultimately achieving end-to-end high-precision learning of the fuel tank nozzle's "identification-localization-pose estimation".

[0054] Design a multi-task loss function to achieve collaborative optimization among tasks:

[0055] (1);

[0056] The definitions and functions of the loss functions for each branch are as follows:

[0057] The cross-entropy loss function for the semantic segmentation branch measures the difference between the predicted category and the true label. The formula is:

[0058] (2);

[0059] In the formula, N is the total number of pixels, and C is the number of categories. Let i be the true label (0 or 1) for class c. To predict probabilities, this loss function enables the model to accurately learn the pixel-level feature differences between the target and the background, achieving high-precision segmentation of the target area and the background in the fuel tank panel.

[0060] Here is the mean squared error (MSE) loss function for the keypoint detection branch, used to constrain the distance between the predicted keypoint coordinates and the true coordinates. The formula is:

[0061] (3);

[0062] In the formula, M represents the total number of key points. and The first The actual coordinates and predicted coordinates of each key point. This is the square of the Euclidean distance. This loss function minimizes coordinate deviation, enabling the model to accurately locate the key positions of the target, providing reliable baseline coordinate data for subsequent 3D pose estimation.

[0063] Let be the reprojection error loss function of the 3D pose estimation branch. This function optimizes the rotation matrix R and translation vector t by minimizing the deviation between the reprojected coordinates of keypoints in the predicted pose and their true 2D coordinates. The formula is as follows:

[0064] (4);

[0065] In the formula, For the first 3D coordinates of key points The projection function converts 3D coordinates to 2D image coordinates. This loss function focuses on optimizing the transformation accuracy of the target in the "3D space-2D image" coordinate system, ensuring that the robotic arm can accurately perform the oil retrieval and docking task based on the estimated pose information.

[0066] During training, to balance the convergence speed and optimization priority of each task, an adaptive weight adjustment algorithm is used to dynamically adjust the weight coefficients. , and :

[0067] In the initial phase (the first 50 epochs), to accelerate the overall convergence of the model, priority is given to ensuring the learning efficiency of semantic segmentation (the basic task). , , Convergence Phase (50-300 epochs): Weight coefficients are adjusted in real-time based on the changes in validation set loss for each task. If the rate of decrease in validation loss for a task is less than 0.001 / epoch, its weight coefficient is increased; conversely, it is decreased. This dynamically balances the optimization priorities of different tasks, avoiding overfitting or underfitting of a single task. This joint training strategy integrates multi-task learning objectives through a unified loss framework and adapts to changes in task characteristics during the training process by dynamically adjusting weights. This effectively avoids gradient competition and interference between multiple tasks, enabling the model to achieve high-precision convergence in semantic segmentation, keypoint detection, and 3D pose estimation, meeting the real-time perception requirements of the robotic arm during the oil sampling robot's docking process.

[0068] This implementation achieves vision-force collaborative operation, specifically including:

[0069] Step 1: Visually guided path planning.

[0070] The trained lightweight multi-task model is deployed to the robotic arm controller, and the depth camera captures real-time images of the fuel tank panel at 30fps. The visual processing flow is as follows:

[0071] The input image is first processed by the semantic segmentation branch, and the output is a pixel category prediction map of the target and obstacles. The contours of the target and obstacles are extracted by threshold segmentation and morphological operations (dilation and erosion). The two-dimensional contours are converted into three-dimensional voxel maps using the Octomap library to mark the prohibited areas in the robot arm's motion space.

[0072] The keypoint detection branch locates the center and corner points of the target and obtains the 3D coordinates of the keypoints by combining the depth image. The 3D pose estimation branch calculates the target's 6D pose in the camera coordinate system based on the keypoint correspondences using the PnP algorithm, and then uses a pre-calibrated hand-eye transformation matrix calibrated using the AXYZ four-position method. Transform the target pose to the robotic arm tool coordinate system;

[0073] Based on Octomap voxel maps and target pose information, RRT* is used for collision-free motion trajectory planning. During the planning process, the kinematic constraints of the robotic arm joints are considered to generate a smooth and safe motion path.

[0074] Step 2: Dynamic adjustment of force feedback.

[0075] A six-dimensional force sensor is installed at the end effector of the robotic arm, which can collect six-dimensional force information at the end effector in real time. , , , , , The sampling frequency is 1kHz, and the force feedback control is divided into two stages:

[0076] A six-dimensional force sensor is installed between the end effector flange and the actuator of the robotic arm, which can detect three-dimensional force at the end effector. , , ) and three-dimensional torque ( , , The system acquires information at high frequency, with a sampling frequency of 1kHz, providing a high-real-time and high-precision sensing foundation for contact force control. Before data acquisition, zero-point calibration, force / torque linear calibration, and coordinate system alignment calibration are performed. Simultaneously, force signal preprocessing is completed through a combination of second-order low-pass filtering and Kalman filtering, effectively suppressing signal noise in the substation's electromagnetic environment and ensuring the reliability of the force feedback data. Considering the entire process of the robotic arm from approaching the target to stable operation, the force feedback control is divided into two stages: impedance compliance control and closed-loop PID regulation. Seamless switching between the two stages is achieved through contact force threshold determination, balancing the compliance of the contact process with the accuracy of the force value during operation.

[0077] Impedance compliance control is suitable for the transition phase from the robotic arm's approach to the target to initial contact. The triggering condition is that the robotic arm moves within 20mm of the target, and the six-dimensional force sensor detects that the contact force is less than a preset initial threshold. The core of this stage is to establish a force-position mapping relationship, converting the contact force signal into an end-effector position / attitude correction command, thereby achieving flexible buffering of contact impact and quickly correcting the initial attitude deviation of the end-effector to avoid rigid collisions that could damage the target or actuator.

[0078] Impedance control is based on the Cartesian space impedance model, with preset stiffness K and damping B parameters that match the characteristics of the target operation. The control formula is as follows:

[0079] (5);

[0080] In the formula, The desired threshold of contact force. For the actual contact force measured by the sensor, This is the Cartesian spatial position correction. The change in end-effector velocity is calculated using this formula. The position / attitude correction is then converted into angle correction commands for each joint of the robotic arm via the Jacobian matrix and sent to the execution layer to achieve compliant end-effector motion, thus completing the initial impact absorption and coarse attitude adjustment.

[0081] Closed-loop PID control is applicable to the operational phase after the robotic arm has made stable contact with the target. The trigger condition is that the sensor detects that the contact force is stable within ±5% of the desired threshold for 10ms, and after determining stable contact, the system automatically switches to this phase. The core of this phase is the construction of a real-time force feedback closed-loop control system. By comparing the deviation between the actual contact force / torque and the desired operational threshold, the PID algorithm outputs control quantities in real time to correct the robotic arm's trajectory, ensuring that the force / torque remains stable within the industry standard's allowable range throughout the operation, meeting the precise operational requirements of the transformer oil sampling robot's oil sampling port docking.

[0082] The PID controller takes force / torque deviation as input and outputs joint speed or torque control quantity. The core continuous control formula is:

[0083] (6);

[0084] In the formula, For real-time control of quantities, , , These are the proportional, integral, and differential coefficients, respectively. To account for the real-time error between the actual force and the desired force, and considering the discrete nature of computer control, the continuous formula is discretized to adapt to a 1kHz control cycle (T=0.001s). The discretized formula is as follows:

[0085] (7);

[0086] In the formula, For the first One control cycle, Output control quantity for the current cycle. This represents the current periodic force error.

[0087] The PID parameters were determined using the Ziegler-Nichols tuning method. First, the integral and derivative components were turned off, and then the proportional gain was gradually increased until the system exhibited constant-amplitude oscillations. The critical proportional gain was then recorded. With critical oscillation period Then, it is calculated using empirical formulas. , , Finally, the parameters were fine-tuned based on the actual working conditions of the oil tank and oil outlet to ensure that the system response time was <5ms, there was no overshoot, and the steady-state error was <±3N. At the same time, joint kinematic constraints were applied to the control output to avoid exceeding the allowable range of the robot arm joint speed and torque.

[0088] In this implementation, to avoid the force / position shock generated during the switching between impedance compliance control and closed-loop PID regulation, which would affect operational stability, a smooth transition strategy using a first-order low-pass filter is designed. The core formula is:

[0089] (8);

[0090] in, The control quantity that is ultimately sent to the execution layer. To control the output quantity by impedance. For the output of the PID controller, The weighting coefficient increases linearly from 0 to 1. During the switching process, α increases linearly from 0 to 1, with a transition time of 5ms, ensuring that the contact force fluctuation at the moment of switching is <±1N, achieving a seamless connection between the two stages without impact, and guaranteeing the continuity and stability of force feedback control.

[0091] This implementation also constructs a dual closed-loop mechanism of "visual perception - force control execution - feedback correction" to achieve collaborative operation between vision and force control. The vision system updates the target pose information every 100ms. When it detects a change in the target's position due to factors such as vibration or human touch, it replans the robotic arm's motion trajectory. The force control system adjusts the contact force at a high frequency of 1ms to ensure stable operating force. During operation, if the force control system detects abnormal force, it immediately triggers an emergency stop signal and feeds back to the vision system to reassess the target state; conversely, the vision system can also dynamically adjust the expected force threshold of the force control system according to changes in the target pose, forming a close interactive collaboration.

[0092] In summary, this invention constructs diverse datasets using multi-view images and point cloud data, trains lightweight multi-task deep learning models to achieve high-precision target recognition and pose estimation, effectively overcoming the limitations of single sensors and insufficient texture information. While fully utilizing image semantic features and point cloud geometric features, it effectively improves the accuracy and robustness of 3D target detection and pose estimation in complex scenes. Simultaneously, in the end-effector force control system, a dual closed-loop mechanism of "visual perception - force control execution - feedback correction" is constructed, and a multi-degree-of-freedom robotic arm force constraint trajectory control and force / position hybrid control model are designed to achieve adaptive adjustment of the motion trajectory of the high-precision interface docking channel. This effectively improves the accuracy and safety of the robotic arm operation, maintaining real-time target positioning and precise robotic arm operation even under complex lighting and dynamic interference, meeting the precision operation requirements of the transformer oil sampling robot.

[0093] Optionally, in some other implementations, the Cartesian space impedance model in formula (5) adopts a preset fixed stiffness. However, in actual oil sampling and docking processes, the aging of the oil sampling port sealing ring material, changes in the coefficient of friction caused by oil adhesion, and slight deviations in the robotic arm's approach angle all lead to dynamic changes in the optimal contact stiffness. Fixed stiffness cannot reconcile the contradiction between "quickly eliminating positional deviations" (requiring high stiffness) and "avoiding rigid impact damage to the sealing surface" (requiring low stiffness). Especially at the moment of contact, excessive stiffness can easily cause the sealing surface to crush, while insufficient stiffness leads to slow convergence of docking vibrations.

[0094] In view of this, optionally, after the smooth switching of the control quantity is completed by formula (8), an adaptive variable stiffness factor can be introduced. The algorithm dynamically corrects the stiffness matrix in the impedance model. It utilizes the volatility of real-time force signals and the uncertainty of visual pose estimation to construct a stability criterion.

[0095] The dynamic stiffness adjustment formula is defined as follows:

[0096] (9);

[0097] (10);

[0098] (11);

[0099] in, The equivalent stiffness coefficient adjusted at different times, in units ; Initial preset stiffness coefficient, unit ; Stiffness attenuation factor, dimensionless, range of values . The larger the value, the lower the stiffness, and the more compliant the system. Force-visual weighting coefficients are dimensionless and typically take the value of 1. This is used to balance the effects of force signal fluctuations and visual errors; The number of sampling points in the sliding window is dimensionless (e.g., taking...). (corresponding to 10ms data). Past Measured contact force per sampling period, in units ; Average contact force within the sliding window, unit ; Expected contact force threshold, unit ; The reprojection error value of the 3D pose estimation branch output by the current frame visual model (directly reusing the calculation result in Equation 4). ),unit (pixel) or the normalized dimensionless error value; The safety threshold for reprojection error, in units of Consistent; The equivalent mass and damping coefficient of the system, respectively, are in units of and ; Position and attitude correction amount, in units or .

[0100] When the contact force fluctuates drastically (large numerator) or the visual localization reliability is low ( When (large), Increase Automatic lowering makes the robotic arm behave more "flexibly," effectively absorbing high-frequency vibrations and handling positioning uncertainties, preventing hard collisions. When contact is stable and vision is clear, Approaching 0, Restore to This improves system stiffness, accelerates the elimination of steady-state errors, and enhances docking efficiency. The calculated... The original impedance control circuit will be replaced in real time. Parameters used for position correction in the next control cycle. The calculation forms a millisecond-level dynamic closed loop of "perception-decision-execution".

[0101] Alternatively, in some other implementations, visual guidance provides the macroscopic trajectory, while force feedback provides the microscopic correction. However, there may be a time scale mismatch between the two. When the robotic arm approaches at high speed or the sealing ring undergoes elastic deformation, pure feedback control (PID / impedance) has an inherent hysteresis, causing the robotic arm end to produce tentative small amplitude oscillations at the contact surface, prolonging the fine-tuning time, and in extreme cases, it may cause system divergence due to excessive hysteresis compensation.

[0102] In view of this, optionally, a feedforward compensator can be constructed to predict the trajectory correction required at the next moment by utilizing the coupling relationship between the trend term of the visual pose prediction and the force control historical residual, and then superimposed on the output of formula (8).

[0103] The trajectory feedforward compensation formula is defined as follows:

[0104] (12);

[0105] (13);

[0106] in, No. Periodic joint space feedforward compensation, unit (radian); The final master control command (joint angle or speed) is issued to the actuator, in units. or ; The smoothed control quantity calculated by formula (8) has units equal to... Consistent; The feedforward gain coefficient is dimensionless and is used to adjust the compensation strength (usually). ); robotic arm in current configuration The inverse Jacobian matrix below, unit or (Mapping Cartesian space quantities back to joint space). No. The rotation matrix of the time-mapping visual solution is a dimensionless orthogonal matrix; The sampling time interval of the vision system (approximately 0.033 seconds), in units of... ; The target relative motion velocity vector calculated from visual optical flow or point cloud registration, in units. If the target is stationary, this item reflects the estimated approach speed of the robotic arm relative to the target; Force residual integral gain coefficient, unit This is used to convert the accumulated force error into a displacement trend. Historical moment Force control error ( ),unit ; The length of the integration window is dimensionless.

[0107] Calculated The output of formula (8) is directly superimposed on the output of formula (8). Above, the final joint drive commands are generated. This guides the robotic arm to make "predictive" movement adjustments within the next millisecond cycle.

[0108] Figure 2 A transformer oil sampling docking system integrating deep learning and force feedback is shown, including:

[0109] The multi-task visual perception and pose calculation unit 201 is configured to: acquire real-time images and point cloud data of the oil tank panel, input them into a lightweight multi-task deep learning model, and simultaneously output the semantic segmentation mask, key point two-dimensional coordinates and initial three-dimensional pose parameters of the oil outlet, and convert the initial three-dimensional pose parameters into target pose information in the robotic arm tool coordinate system based on the hand-eye conversion matrix.

[0110] The visually guided collision-free trajectory planning unit 202 is configured to: construct a three-dimensional voxel map containing prohibited areas based on semantic segmentation masks, combine target pose information and robotic arm kinematic constraints, and use path planning algorithms to generate a collision-free reference trajectory from the current position to the oil outlet;

[0111] The impedance compliant contact control unit 203 is configured to: control the robotic arm to move along a collision-free reference trajectory; when the contact force is detected to be less than a preset initial threshold, convert the deviation between the measured contact force and the expected contact force into a position and attitude correction amount based on the Cartesian space impedance model, and drive the end of the robotic arm to complete compliant contact and coarse attitude adjustment.

[0112] The force-position hybrid closed-loop fine-tuning unit 204 is configured to switch to closed-loop PID control mode when the measured contact force is stable within the desired contact force range. Based on the real-time error between the measured contact force and the desired contact force, the joint control quantity is output to dynamically correct the movement trajectory of the robotic arm to maintain torque stability during the docking process and complete high-precision docking of the oil intake port.

[0113] It is understood that the aforementioned units can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of the present invention. The aforementioned units are based on logical functional division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of the present invention, the system may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0114] According to another embodiment of the present invention, the system of this embodiment can be constructed by running a computer program (including program code) capable of performing the steps involved in the corresponding method of the present invention on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, loaded into the aforementioned computing device through the computer-readable recording medium, and run therein.

[0115] Figure 3 An oil-retrieving robot is shown, including an oil-retrieving robotic arm. The oil-retrieving robot includes a processor 301, a communication interface 302, and a computer-readable storage medium 303. The processor 301, communication interface 302, and computer-readable storage medium 303 can be connected via a bus or other means.

[0116] The communication interface 302 is used to receive and send data. The computer-readable storage medium 303 can be stored in the memory of the electronic device. The computer-readable storage medium 303 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the computer-readable storage medium 303.

[0117] The processor 301 is the computing and control core of the electronic device. It is suitable for implementing one or more instructions, specifically for loading and executing one or more instructions to achieve the corresponding method flow or corresponding function.

[0118] Processor 301 is configured to perform the following procedure:

[0119] Multi-task visual perception and pose calculation steps: acquire real-time image and point cloud data of the oil tank panel, input it into a lightweight multi-task deep learning model, and simultaneously output the semantic segmentation mask and initial three-dimensional pose parameters of the oil outlet. Based on the hand-eye conversion matrix, the initial three-dimensional pose parameters are converted into target pose information in the tool coordinate system of the robotic arm.

[0120] Visual-guided collision-free trajectory planning steps: Construct a 3D voxel map containing restricted areas based on semantic segmentation mask, combine target pose information and robotic arm kinematic constraints, and use path planning algorithm to generate a collision-free reference trajectory from the current position to the oil outlet;

[0121] Impedance-compliant contact control steps: Control the robotic arm to move along a collision-free reference trajectory. When the detected contact force is less than the preset initial threshold, the deviation between the measured contact force and the expected contact force is converted into a position and attitude correction amount based on the Cartesian space impedance model, and the robotic arm end is driven to complete flexible contact and coarse attitude adjustment.

[0122] Force-position hybrid closed-loop fine-tuning steps: When the measured contact force is stable within the desired contact force range, switch to closed-loop PID control mode, output joint control quantity based on the real-time error between the measured contact force and the desired contact force, dynamically correct the robot arm motion trajectory to maintain torque stability during docking, and complete high-precision docking of the oil intake port.

[0123] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0124] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic cable, digital cable) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0125] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A transformer oil sampling and docking method integrating deep learning and force feedback, characterized in that, Includes the following processes: Multi-task visual perception and pose calculation steps: acquire real-time image and point cloud data of the oil tank panel, input it into a lightweight multi-task deep learning model, and simultaneously output the semantic segmentation mask and initial three-dimensional pose parameters of the oil outlet. Based on the hand-eye conversion matrix, the initial three-dimensional pose parameters are converted into target pose information in the tool coordinate system of the robotic arm. Visual-guided collision-free trajectory planning steps: Construct a 3D voxel map containing restricted areas based on semantic segmentation mask, combine target pose information and robotic arm kinematic constraints, and use path planning algorithm to generate a collision-free reference trajectory from the current position to the oil outlet; Impedance-compliant contact control steps: Control the robotic arm to move along a collision-free reference trajectory. When the detected contact force is less than the preset initial threshold, the deviation between the measured contact force and the expected contact force is converted into a position and attitude correction amount based on the Cartesian space impedance model, and the robotic arm end is driven to complete flexible contact and coarse attitude adjustment. Force-position hybrid closed-loop fine-tuning steps: When the measured contact force is stable within the desired contact force range, switch to closed-loop PID control mode, output joint control quantity based on the real-time error between the measured contact force and the desired contact force, dynamically correct the robot arm motion trajectory to maintain torque stability during docking, and complete high-precision docking of the oil intake port.

2. The transformer oil sampling and docking method integrating deep learning and force feedback as described in claim 1, characterized in that, During the training of the lightweight multi-task deep learning model, a weighted fusion multi-task loss function is used for joint optimization. The weighted fusion multi-task loss function is as follows: ； in, This is the total loss value. The cross-entropy loss is used for semantic segmentation branches. The mean squared error loss for keypoint detection branches. The reprojection error loss is used for the 3D pose estimation branch. , , These are the dynamic weight coefficients for each branch.

3. The transformer oil sampling and docking method integrating deep learning and force feedback as described in claim 2, characterized in that, Cross-entropy loss of semantic segmentation branches The calculation formula is: ； in, Total number of pixels For the number of categories, For pixels Category The true label, Predict pixels for the model Category The probability of.

4. The transformer oil sampling and docking method integrating deep learning and force feedback as described in claim 2, characterized in that, Mean square error loss of key point detection branch The calculation formula is: in, The total number of key points. For the first The true coordinates of each key point For the first Predicted coordinates of key points This represents the Euclidean distance.

5. The transformer oil sampling and docking method integrating deep learning and force feedback as described in claim 2, characterized in that, Reprojection error loss in 3D pose estimation branch The calculation formula is: ； in, For the first The three-dimensional coordinates of the key points Let be a rotation matrix. It is a translation vector. This is a projection function that converts three-dimensional coordinates into two-dimensional image coordinates. For the first The true two-dimensional coordinates of each key point.

6. The transformer oil sampling and docking method integrating deep learning and force feedback as described in claim 2, characterized in that, Dynamic weighting coefficients , , During training, the system adaptively adjusts the loss based on the rate of change of the validation set loss for each branch. If the rate of decrease of the validation loss for a certain branch is lower than a preset threshold, the dynamic weight coefficient corresponding to that branch is increased.

7. The transformer oil sampling and docking method integrating deep learning and force feedback as described in claim 1, characterized in that, In the force-position hybrid closed-loop fine-tuning step, the discretized control formula for the closed-loop PID regulation mode is: ； in, For the first Real-time control quantity for each control cycle. , , These are the proportional, integral, and differential coefficients, respectively. For the first The real-time error between the actual force and the expected force in each control cycle To control the cycle.

8. The transformer oil sampling and docking method integrating deep learning and force feedback as described in claim 1, characterized in that, During the transition from the impedance-compliant contact control step to the force-position hybrid closed-loop fine-tuning step, a first-order low-pass filter strategy is used to fuse the control inputs. The fusion formula is as follows: ； in, For the control quantity that is ultimately sent to the execution layer, To control the output quantity by impedance. For the output of the PID controller, The weighting coefficients increase linearly from 0 to 1.

9. A transformer oil sampling docking system integrating deep learning and force feedback, characterized in that, include: The multi-task visual perception and pose calculation unit is configured to: acquire real-time images and point cloud data of the oil tank panel, input them into a lightweight multi-task deep learning model, simultaneously output the semantic segmentation mask and initial three-dimensional pose parameters of the oil outlet, and convert the initial three-dimensional pose parameters into target pose information in the robotic arm tool coordinate system based on the hand-eye conversion matrix. The visually guided collision-free trajectory planning unit is configured to: construct a 3D voxel map containing restricted areas based on semantic segmentation masks, combine target pose information and robotic arm kinematic constraints, and use path planning algorithms to generate a collision-free reference trajectory from the current position to the oil outlet; The impedance compliant contact control unit is configured to: control the robotic arm to move along a collision-free reference trajectory; when the detected contact force is less than a preset initial threshold, convert the deviation between the measured contact force and the expected contact force into a position and attitude correction amount based on the Cartesian space impedance model, and drive the end of the robotic arm to complete compliant contact and coarse attitude adjustment. The force-position hybrid closed-loop fine-tuning unit is configured to switch to closed-loop PID control mode when the measured contact force is stable within the desired contact force range. Based on the real-time error between the measured contact force and the desired contact force, the joint control quantity is output to dynamically correct the movement trajectory of the robotic arm to maintain torque stability during the docking process and complete high-precision docking of the oil intake port.

10. An oil-collecting robot, characterized in that, include: Processor and computer-readable storage media; A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the transformer oil sampling docking method integrating deep learning and force feedback as described in any one of claims 1 to 8.

Images