Manipulator end effector control method fusing imitation learning and reinforcement learning

Abstract

The embodiment of the application relates to the technical field of control, and provides a manipulator end effector control method fusing imitation learning and reinforcement learning, the method comprising the following steps: acquiring expert demonstration data to construct an imitation learning data set; training a policy network by using the imitation learning data set, and making the policy network reproduce the operation behavior of the expert by fusing imitation learning and reinforcement learning through supervised learning; state information input into the policy network further comprises joint flexibility features collected by a double feedback system composed of a motor encoder and a connecting rod end position sensor; integrating the policy network into a control system of a manipulator, generating a control instruction according to real-time sensed state information, and driving the end effector to complete an operation task corresponding to the control instruction. The embodiment of the application can perceive joint flexibility characteristics and efficiently optimize policy network parameters, obtain a manipulator end effector control strategy with operation precision, flexibility and stability, improve control precision, and optimize control effect.

Description

Technical Field

[0001] This application relates to the field of control technology, and more specifically to a control method for a robotic arm end effector that integrates imitation learning and reinforcement learning. Background Technology

[0002] The end effector, as the terminal component of a robotic arm that interacts with the work object or environment, is often referred to as the robot's hand. The performance of the end effector directly affects the intelligence level and operational capabilities of the entire robot system. Based on their form and function, end effectors can be mainly divided into grippers (such as mechanical grippers and vacuum suction cups), sensors (such as vision and force / torque sensors), and specialized process tools (such as welding torches and spray guns).

[0003] End effector control scenarios are widely distributed in industrial automation, medical robotics, space operations, and emerging service robots. Currently, the mainstream control method is mainly position-based control, which relies on precise kinematic models and is susceptible to systematic errors caused by manufacturing and calibration errors. Its rigid control characteristics cannot meet the requirements of compliant operation, and it is difficult to compensate for trajectory jitter caused by the dynamic coupling effect of the robotic arm. To address the shortcomings of position control in compliant operation, force-based control methods have been proposed. Force-based control methods directly control the contact force between the end effector and the working environment, enabling it to adapt to minute changes in the workpiece surface and ensuring operational safety. Although force control methods solve the rigidity problem of position control, they place extremely high demands on the accuracy and response speed of sensors. Sensor noise, calibration errors, and data transmission delays directly affect the stability and performance of the controller. Furthermore, there is a coupling problem between force control and position control methods. When it is necessary to control both position and force simultaneously (such as constraining axial force and controlling in-plane position in hole-shaft assembly), the controller implementation becomes quite complex. Furthermore, for industrial robots with large motion mass and elastic and frictional joints, achieving high-quality direct force control is extremely difficult. Indirect force control is usually achieved by using an additional force-controlled end effector, which increases the complexity of the system structure and the difficulty of control.

[0004] In conclusion, there is an urgent need for a completely new technical solution to address at least one technical problem existing in the current technology. Summary of the Invention

[0005] This application provides a robotic arm end effector control method that integrates imitation learning and reinforcement learning. It can sense the joint flexibility characteristics and efficiently optimize the policy network parameters to obtain a robotic arm end effector control strategy that combines operational accuracy, compliance, and stability, thereby improving control accuracy and optimizing control performance.

[0006] In a first aspect, embodiments of this application provide a method for controlling a robotic arm end effector that integrates imitation learning and reinforcement learning, the method comprising: A dataset for imitation learning is constructed by acquiring expert demonstration data, wherein the expert demonstration data includes state information and corresponding action information; The policy network is trained using the imitation learning dataset, and supervised learning enables the policy network to reproduce the expert's operational behavior by fusing imitation learning and reinforcement learning. The state information input to the strategy network also includes: joint flexibility features collected by a dual feedback system consisting of a motor encoder and a link end position sensor, wherein the joint flexibility features include at least the positional deviation between the motor end and the link end. The trained policy network is integrated into the control system of the robotic arm, and control commands are generated based on real-time perceived state information to drive the end effector to complete the operation tasks corresponding to the control commands.

[0007] Secondly, embodiments of this application provide a robotic arm end effector control device that integrates imitation learning and reinforcement learning, having the function of implementing the robotic arm end effector control method that integrates imitation learning and reinforcement learning provided in the first aspect above. The function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0008] In one embodiment, the robotic arm end effector control device that integrates imitation learning and reinforcement learning includes: The input / output module is configured to acquire expert demonstration data to construct an imitation learning dataset, wherein the expert demonstration data includes state information and corresponding action information. The processing module is configured to train a policy network using the imitation learning dataset, and to enable the policy network to reproduce the expert's operational behavior through supervised learning by fusing imitation learning and reinforcement learning. The state information input to the policy network also includes joint flexibility features collected by a dual feedback system consisting of a motor encoder and a link end position sensor, the joint flexibility features including at least the positional deviation between the motor end and the link end. The trained policy network is integrated into the control system of the robotic arm, and control commands are generated based on the real-time perceived state information to drive the end effector to complete the operation task corresponding to the control commands.

[0009] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the robotic arm end effector control method that integrates imitation learning and reinforcement learning as described in the first aspect.

[0010] Fourthly, embodiments of this application provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robotic arm end effector control method that integrates imitation learning and reinforcement learning as described in the first aspect.

[0011] Compared to existing technologies, the policy network in this embodiment can quickly learn high-quality operating patterns through expert demonstration data, and then fine-tune them through reinforcement learning to adapt to differences in actual scenarios, forming a control strategy that combines accuracy and compliance. The two-stage learning model, which integrates imitation learning and reinforcement learning, further balances data efficiency and environmental adaptability. The flexible features provided by the dual feedback system further provide direct mechanical body state basis for policy optimization. Therefore, the control strategy obtained in this embodiment can not only accurately reproduce expert operations, but also respond to joint deformation and environmental changes in real time. This can improve the control accuracy, stability, and compliance of the robotic arm end effector, adapting to the needs of multiple scenarios such as industrial assembly and medical operations, while reducing system deployment costs and equipment wear risks. In summary, this embodiment can sense joint flexibility characteristics and efficiently optimize policy network parameters to obtain a robotic arm end effector control strategy that combines operational accuracy, compliance, and stability, improving control accuracy and optimizing control effects. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the robotic arm end effector control method that integrates imitation learning and reinforcement learning, as exemplified in this application. Figure 2 This is a schematic diagram of the structure of the robotic arm end effector control device that integrates imitation learning and reinforcement learning according to an embodiment of this application. Figure 3 This is a schematic diagram of the structure of a computing device according to an embodiment of this application; Figure 4 This is a schematic diagram of a server structure in one embodiment of this application. Detailed Implementation

[0013] This application provides a robotic arm end effector control method that integrates imitation learning and reinforcement learning. It can be applied to scenarios requiring high precision and compliance, such as industrial precision assembly, minimally invasive medical surgery, and space operation equipment control, and is compatible with various robotic arm control systems. The system may include an image processing device and a robotic arm end effector control device, which can be integrated or deployed independently. The image processing device performs noise reduction, enhancement, and feature extraction preprocessing on scene images acquired by a vision sensor, outputting a target image adapted to the subsequent end effector control scenario. The robotic arm end effector control device executes the core control process, specifically including acquiring expert demonstration data and constructing an imitation learning dataset containing state information and corresponding action information. This dataset is used to train a strategy network through supervised learning to reproduce expert operational behavior. The input state information of the strategy network includes joint flexibility features (at least including the positional deviation between the motor end and the link end) collected by a dual feedback system consisting of a motor encoder and a link end position sensor. Finally, the trained strategy network is integrated into the robotic arm control system, generating control commands based on real-time perceived state information and driving the end effector to complete the corresponding operation task. The control device can be an independent control model, a terminal device or server that deploys the model, or it can be integrated into the main control unit of the robotic arm.

[0014] The solutions provided in this application involve core technologies such as imitation learning, reinforcement learning, dual-feedback flexible sensing, gradient-free parameter optimization, and fusion of visual and mechanical ontological features. By combining bio-inspired optimization algorithms with intelligent learning frameworks, the contradiction between accuracy, compliance, and system complexity in existing robotic arm control is resolved. Detailed explanations are provided below: In existing technologies, the mainstream control methods for end effectors are mainly position-based control methods. These methods rely on precise kinematic models and are susceptible to systematic errors caused by manufacturing and calibration mistakes. Their rigid control characteristics cannot meet the requirements of compliant operation, and they struggle to compensate for trajectory jitter caused by the dynamic coupling effect of the robotic arm. To address the shortcomings of position control in compliant operation, force-based control methods have been proposed. Force-based control methods directly control the contact force between the end effector and the working environment, enabling it to adapt to minute changes in the workpiece surface and ensuring operational safety. While force control methods solve the rigidity problem of position control, they place extremely high demands on sensor accuracy and response speed. Sensor noise, calibration errors, and data transmission delays directly affect the stability and performance of the controller. Furthermore, there is a coupling problem between force control and position control methods. When simultaneous control of position and force is required (such as constraining axial force while controlling in-plane position in hole-shaft assembly), the controller implementation becomes quite complex. Moreover, for industrial robots with large moving masses and elastic and frictional joints, achieving high-quality direct force control is exceptionally difficult. Typically, additional force-controlled end effectors are used for indirect force control, which increases the complexity of the system structure and the difficulty of control.

[0015] To overcome the problems of cumulative errors and the inability to detect flexible deformation in robotic arm sensors (such as encoders), existing technologies also introduce external high-precision measurement systems (such as optical motion capture systems) as a recommended positioning method. This closed-loop control method based on external measurement can improve positioning accuracy through optical motion capture systems. However, the optimized positioning data needs to be transmitted to the robotic arm controller via communication interfaces (such as VRPN and ROS). The overall control latency includes not only the latency of the motion capture system but also communication latency and controller computation latency, resulting in poor real-time performance. Furthermore, optical motion capture systems are complex to deploy, require specific site conditions (such as avoiding strong light and metal reflections), and the camera's field of view cannot be obstructed, making them difficult to widely apply in practical robotic arm service scenarios.

[0016] Compared to existing technologies, this application provides a method for controlling a robotic arm end effector that integrates imitation learning and reinforcement learning. The method includes: acquiring expert demonstration data to construct an imitation learning dataset, wherein the expert demonstration data includes state information and corresponding action information; training a policy network using the imitation learning dataset, and using supervised learning to enable the policy network to reproduce the expert's operational behavior through the integration of imitation learning and reinforcement learning; wherein the state information input to the policy network further includes joint flexibility features collected by a dual feedback system consisting of a motor encoder and a link end position sensor, wherein the joint flexibility features include at least the positional deviation between the motor end and the link end; integrating the trained policy network into the robotic arm's control system, generating control commands based on the real-time perceived state information, and driving the end effector to complete the operational task corresponding to the control commands.

[0017] This application embodiment can sense the joint flexibility characteristics and efficiently optimize the strategy network parameters to obtain a robotic arm end effector control strategy that combines operational accuracy, compliance, and stability, thereby improving control accuracy and optimizing control performance. Furthermore, this application embodiment can further reduce the operational risks caused by technical defects in traditional control methods, such as trajectory jitter, force-pose coupling conflicts, and low parameter optimization efficiency, providing a better user experience. In addition, this application embodiment can further reduce the cost of expert demonstration data acquisition, robotic arm equipment wear and tear, and the probability of workpiece damage.

[0018] In some implementations, the robotic arm end effector control method that integrates imitation learning and reinforcement learning can be implemented by a server, a terminal device, or a control system consisting of a server and a terminal device.

[0019] It should be noted that the server involved in the embodiments of this application can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0020] The terminal device involved in this application embodiment can be a device that provides voice and / or data connectivity to a user, a handheld device with wireless connectivity, or other processing devices connected to a wireless modem. In practical applications, the terminal device can be connected to a robotic arm as an external controller. Alternatively, it can be directly mounted in the robotic arm as a hardware control module, or installed in the robotic arm's control system as a software control module, such as a dedicated control program for the robotic arm's end effector.

[0021] Reference Figure 1 , Figure 1 This is a flowchart illustrating a robotic arm end effector control method integrating imitation learning and reinforcement learning, provided as an embodiment of this application. The method includes the following steps 101 to 103: Step 101: Obtain expert demonstration data to construct an imitation learning dataset.

[0022] In this embodiment of the application, the expert demonstration data includes status information and corresponding action information.

[0023] It is worth noting that, for example, the acquisition of expert demonstration data relies on a high-fidelity teleoperation platform. Experienced operators guide the robotic arm's end effector to complete tasks such as precision assembly and flexible grasping using a master-slave teleoperation device. During operation, a hardware synchronization mechanism triggers collaborative acquisition from multiple sensors. A vision sensor continuously captures real-time image sequences containing workpiece shape, spatial position, and environmental background. Joint encoders accurately record the rotation angle and angular velocity changes of each joint. The dual-feedback sensing system consists of a motor output shaft encoder and a high-precision displacement sensor at the end of the linkage, simultaneously measuring the instantaneous positions of the motor and linkage ends and calculating their dynamic deviation. This deviation, as a core characteristic of joint flexibility, directly reflects the elastic deformation of the reducer and the internal deformation caused by load disturbances. Simultaneously, the control system captures the raw control signals output by the master device in real time, converting them through kinematic mapping into the desired pose adjustment of the end effector in Cartesian space, including linear displacement components along the coordinate axes and attitude adjustment components around the rotation axis.

[0024] In this embodiment, state information is defined as a multimodal observation set that comprehensively describes the current working environment, body configuration, and internal physical dynamics of the robotic arm. Its connotation includes the scene semantics and target spatial relationship provided by visual perception, the real-time configuration state represented by joint kinematic parameters, and the joint flexible dynamic characteristics revealed by the dual feedback system.

[0025] In this embodiment, action information is defined as end-effector motion commands generated by expert decision-making under specific state observations. Essentially, it reflects the human operator's intention to achieve the task objective and their adaptive adjustment strategies to cope with environmental disturbances. All collected data is strictly aligned according to high-precision hardware timestamps, forming a continuous temporal sequence of state and action pairs. After removing high-frequency noise from sensors and eliminating transient abnormal sampling points through moving average filtering, effective operation segments are divided according to the task logic stages. Finally, a well-structured and temporally rigorous imitation learning dataset is constructed, providing a high-quality sample foundation for policy network training that combines environmental diversity, operational standardization, and physical realism.

[0026] In step 101, the construction of expert demonstration data relies on a multi-source sensor collaborative acquisition mechanism. State information is obtained from real-time images of the task scene acquired by a vision sensor, providing environmental semantics and spatial relationships with target objects. Joint encoders synchronously record the angles and angular velocities of each joint, characterizing the kinematic state of the robotic arm. A dual-feedback system consists of an encoder at the motor output end and a high-precision position sensor at the end of the link. Millisecond-level synchronous sampling calculates the positional deviation between the motor end and the link end; this deviation serves as a core parameter of the joint flexibility characteristics, directly reflecting the elastic deformation of the reducer and the dynamic deformation caused by load disturbances. Action information corresponds to the Cartesian space position increment and attitude rotation of the end effector generated by expert operations at each state moment. Data acquisition is completed through remote operation equipment or a high-precision teaching system. All sensor data is strictly aligned using hardware timestamps to form a continuous temporal state-action pair sequence. After the raw data undergoes moving average filtering to eliminate high-frequency noise and remove instantaneous sensor outliers, it is divided into effective segments according to task stages, and structured samples are generated based on time windows. The final imitation learning dataset not only contains traditional kinematic information, but more importantly, it embeds the physical essence of joint flexibility, enabling the dataset to characterize the intrinsic dynamics of the robotic arm and providing a high-quality prior knowledge foundation for the policy network to learn the flexibility perception and compensation mechanism.

[0027] As an optional embodiment, in step 101, the step of acquiring expert demonstration data to construct an imitation learning dataset includes: the expert demonstration data consisting of a sequence of state-action pairs arranged in chronological order. The state information includes task scene images captured by a visual sensor, robotic arm joint angles, robotic arm joint angular velocities, and joint flexibility features collected by a dual feedback system. The action information consists of position and attitude adjustment commands for the end effector in the corresponding state.

[0028] In the above optional embodiments, the construction of expert demonstration data relies on a multi-source sensor collaborative acquisition system. The state information consists of four core parameters: The task scene images captured by the visual sensor provide a visual representation of the environmental semantics and spatial relationship of the target object, typically high-resolution color images or depth image sequences. The robotic arm joint angles record the instantaneous physical quantity of the rotational position of each joint, accurately describing the current configuration of the robotic arm in radians. The robotic arm joint angular velocity characterizes the dynamic rate of change of each joint's motion, measured in radians per second, reflecting the acceleration characteristics during motion. The joint flexibility characteristics acquired by the dual feedback system are obtained through synchronous measurement by the motor output shaft encoder and the high-precision displacement sensor at the end of the connecting rod. The core parameter is the positional deviation between the motor end and the connecting rod end at the same timestamp. This deviation quantifies the elastic deformation of the joint reducer, transmission clearance, and internal deformation dynamics caused by load disturbances in the form of angles or linear displacement.

[0029] Specifically, motion information is defined as end effector control commands generated by experts under corresponding state observations. It includes position adjustment (linear displacement components along three-dimensional coordinate axes) and attitude adjustment (small attitude change increments based on rotation matrices or quaternions) in Cartesian space, reflecting the expert's precise intent to the task objective and adaptive response to environmental disturbances.

[0030] In the process of constructing the imitation learning dataset, a standardized work process is first executed. The operator guides the robotic arm to complete tasks such as precision assembly or compliant grasping using a high-fidelity telemanipulation master hand. All sensor data and motion commands are strictly synchronized with hardware-level timestamps to ensure temporal alignment. The raw data is filtered using a moving average to eliminate high-frequency noise from the sensors, remove instantaneous abnormal sampling points, and segmented into logically complete operation segments based on task start and end signals. The state parameters of each time step are integrated into a multimodal feature vector, which is paired with the synchronously recorded motion commands to form a state-action pair sequence. Finally, the dataset is organized in chronological order into a structured dataset, which not only fully preserves the temporal dynamic characteristics of expert operations, but also, crucially, embeds joint flexibility features as physical priors into the data layer. This enables the dataset to represent the real dynamic behavior of the robotic arm, providing a high-fidelity training foundation for the policy network to learn flexible perception and adaptive compensation mechanisms.

[0031] Step 102: Train the policy network using the imitation learning dataset, and enable the policy network to reproduce the expert's operational behavior through supervised learning by fusing imitation learning and reinforcement learning.

[0032] The state information input to the strategy network also includes: joint flexibility features collected by a dual feedback system consisting of a motor encoder and a connecting rod end position sensor, wherein the joint flexibility features include at least the positional deviation between the motor end and the connecting rod end.

[0033] In step 102, the training of the policy network relies on the structured state-action pairs in the imitation learning dataset, and its input state information is composed of deep fusion of multimodal physical perception data. The task scene images captured by the visual sensor are normalized and converted into feature tensors, which carry the visual representation of the environmental semantics and the spatial relationship of the target object. The angles of each joint of the robotic arm describe the current configuration state in radians, and the joint angular velocity quantifies the rate of change of motion dynamics in radians per second. The joint flexibility feature provided by the dual feedback system is obtained in real time through a hardware synchronization mechanism by acquiring the position readings of the encoder on the motor output shaft and the position readings of the high-precision displacement sensor at the end of the link, and calculating the instantaneous deviation between the two within a millisecond time window. This deviation is quantified in the form of angular deviation or micrometer-level linear displacement to reflect the degree of internal deformation of the joint reducer caused by elastic deformation, transmission clearance and load disturbance, directly reflecting the physical characteristics of the robotic arm dynamic system.

[0034] At the data organization level, each training sample in the imitation learning dataset integrates the above parameters into a unified state vector, in which joint flexibility features are embedded as an independent and key dimension into the input representation, forming a spatiotemporally aligned multimodal observation sequence with visual features and joint kinematic parameters.

[0035] During the supervised learning phase, the policy network uses this state vector as input and establishes a nonlinear mapping relationship between the joint flexibility state and expert action commands through end-to-end training. This allows the network to implicitly learn the expert's compensation strategy for joint deformation during operation. For example, when an increase in the deviation of a specific joint position is detected, the policy automatically fine-tunes the end-effector trajectory reference point to avoid accumulated errors. This not only ensures that the policy network has physical perception capabilities when reproducing expert behavior but also provides an accurate state representation foundation for the subsequent reinforcement learning fine-tuning phase. This enables the policy to dynamically optimize control commands based on real-time flexibility features, fundamentally improving the adaptability and control robustness to the inherent dynamic characteristics of the robotic arm.

[0036] Step 102 first performs imitation learning pre-training, inputting state information into the multimodal fusion policy network. This network adopts a hierarchical processing architecture, with convolutional branches parsing visual images to extract environmental features, and fully connected branches fusing numerical states such as joint angles, angular velocities, and joint flexibility features. The feature modulation module adaptively weights and integrates multi-source information. Training aims to optimize the mean squared error between predicted actions and expert actions, using gradient descent to enable the policy network to initially reproduce expert operational logic, especially learning how to adjust actions to avoid deformation effects under specific flexibility features. After pre-training, the policy network serves as the initial policy for reinforcement learning fine-tuning, employing a parameter-efficient fine-tuning strategy. The parameters of the feature extraction and action generation modules are frozen, and only the adaptation layer weights or attention mechanism parameters in the feature modulation module are unlocked for optimization. Simultaneously, a value network is constructed to evaluate the long-term value of state-action, and a hybrid experience replay pool is maintained to synchronously store offline expert data and simulation interaction data. In the fine-tuning phase, iterative optimization is performed based on a composite objective function. Reinforcement learning terms drive the strategy to pursue cumulative task rewards, while behavior cloning regularization terms constrain the strategy output to avoid deviating from expert behavior. A vibration suppression penalty term based on the square integral of acceleration is embedded in the reward function, guiding the strategy to actively learn and suppress residual vibrations induced by flexibility. This fusion training mechanism enables the policy network to inherit the efficiency of expert knowledge while enhancing its adaptability to dynamic disturbances through environmental interaction, achieving a shift from repetitive operation to optimized control.

[0037] As an optional embodiment, in step 102, a policy network is trained using the imitation learning dataset. Supervised learning enables the policy network to reproduce the expert's operational behavior by fusing imitation learning and reinforcement learning. This includes: using state information from the imitation learning dataset as input, training the policy network through supervised learning, and minimizing the mean squared error between the predicted action output by the policy network and the standard action labeled in the expert demonstration data as the optimization objective. Furthermore, the supervised-trained policy network is used as the policy network for reinforcement learning fine-tuning. A parameter-efficient fine-tuning strategy is employed, where specific parameters in the feature modulation module are unlocked and optimized while freezing the main structural parameters of the policy network.

[0038] Specifically, in the supervised learning pre-training phase, the policy network takes structured multimodal state information from the imitation learning dataset as input. This state information integrates task scene images captured by the visual sensor, real-time angle and angular velocity parameters of each joint of the robotic arm, and joint flexibility features provided by the dual feedback system. The joint flexibility features are calculated from position data synchronously collected by the encoder on the motor output shaft and the high-precision displacement sensor at the end of the link. Its core is the instantaneous position deviation between the motor end and the link end within the time window. This deviation physically characterizes the internal deformation dynamics of the joint reducer caused by elastic deformation, transmission backlash, and load disturbance.

[0039] After the policy network generates predicted actions through forward propagation, a loss function is constructed using the mean squared error between the predicted actions and the expert-annotated standard actions. The network parameters are then iteratively optimized using backpropagation and gradient descent algorithms. This process, based on the principle of behavior cloning, allows the policy network to implicitly learn the compensation decision logic of experts under different flexibility states. For example, when joint flexibility features indicate increased deformation of a specific joint, the network automatically establishes a mapping relationship for fine-tuning the end-effector trajectory reference points. This stage of training is efficient and stable, obtaining an initial policy with basic operational capabilities, reducing the sample consumption of reinforcement learning from scratch, and laying a physical perception foundation for subsequent optimization.

[0040] Entering the reinforcement learning fine-tuning phase, the pre-trained policy network is used as the initial policy, and a parameter-efficient fine-tuning strategy is employed for targeted optimization. Specifically, the policy network is structurally divided into a feature extraction module, a feature modulation module, and an action generation module. During fine-tuning, all parameters of the feature extraction and action generation modules are strictly frozen, and only specific parameters in the feature modulation module are unlocked for updates. These parameters include the adaptation layer weight parameters, the projection matrix parameters involved in the attention mechanism, or the gating parameters in the gated recurrent unit.

[0041] Optionally, the strategy further optimizes the online data generated by the interaction between offline expert data synchronously provided by the hybrid experience replay pool and the simulation environment, using a composite objective function for iteration. The reinforcement learning objective term drives the strategy to pursue long-term cumulative rewards, while the behavior cloning regularization term constrains the strategy output to not deviate from the expert action distribution. Simultaneously, the reward function embeds a vibration suppression penalty term based on the square integral of acceleration. This mechanism enables the strategy to retain the core expert knowledge while specifically enhancing its adaptability to joint flexibility perturbations through environmental interaction. For example, it dynamically adjusts the smoothness of movements to suppress residual vibrations when abnormal flexibility features are detected. Thus, efficient parameter fine-tuning reduces computational resource consumption, avoids training instability and catastrophic forgetting risks caused by full parameter updates, and ensures that the strategy optimization process mainly involves adaptive adjustments related to physical characteristics.

[0042] Overall, the two-stage training framework organically integrates imitation learning and reinforcement learning. The supervised learning stage efficiently internalizes expert operations and flexible compensation experience, while the reinforcement learning fine-tuning stage, through parameter-oriented optimization and physical perception reward design, enhances the policy network's adaptability to the inherent dynamic characteristics of the robotic arm with minimal disturbance to initial knowledge. Joint flexibility, as a key state dimension throughout the entire training process, prompts the policy to explicitly establish a causal relationship between deformation perception and motion adjustment, fundamentally alleviating the trajectory jitter and positioning drift problems caused by neglecting joint flexibility in traditional position control. This approach avoids the strong dependence on high-precision force sensors and the complexity of force-position coupling design inherent in pure force control, and also overcomes the limitations of real-time performance and environmental adaptability in deploying external measurement systems in industrial settings. It provides reliable technical support for achieving high-precision, high-stability, and highly environmentally adaptable collaborative control of robotic arm end effectors in demanding scenarios such as precision assembly and compliant grasping.

[0043] Further optionally, when optimizing the policy network, the weights of each part in the objective function of the policy network are adjusted by a regularization coefficient. The objective function consists of the following parts: a reinforcement learning objective term, which maximizes the value network's score for the action taken by the current policy; and a behavior cloning regularization term, which constrains the deviation between the policy network's output action and the expert demonstration action, so that the policy network's output action does not deviate from the action in the expert demonstration data.

[0044] In the policy network optimization process, the objective function adopts a composite structure to organically unify the reinforcement learning exploration capability and expert knowledge constraints. The reinforcement learning objective term is guided by maximizing the long-term cumulative score of the policy action by the value network, driving the policy to continuously optimize the decision path in environmental interactions. The behavior cloning regularization term constructs a soft constraint mechanism by quantifying the Euclidean distance deviation between the policy output action and the expert demonstration action, preventing the policy from deviating from the core logical framework of expert operations during fine-tuning. The regularization coefficient serves as a dynamic adjustment hub, adaptively adjusting the weight ratio of the two terms according to the training progress. The coefficient is set higher in the initial stage of training to strengthen the guiding role of the behavior cloning regularization term, enabling the policy to conduct limited exploration within the safe operating domain constructed by the expert demonstration, effectively avoiding training oscillations and invalid sampling caused by random actions.

[0045] As the strategy convergence improves, the weight of the learning objective term is gradually reduced according to a preset decay curve, progressively strengthening the dominant weight. This incentivizes the strategy to adaptively optimize for dynamic factors such as joint flexibility perturbations and subtle environmental changes, while retaining the essence of expert operation. For example, in precision assembly tasks, the initial high-coefficient constraints ensure the strategy strictly follows the expert alignment trajectory, avoiding part collisions due to minor deviations. Later, after coefficient decay, the strategy can autonomously learn to fine-tune contact force and trajectory smoothness when abnormal joint flexibility characteristics are detected, achieving a higher assembly success rate.

[0046] This improves training stability and avoids the convergence difficulties and overfitting risks associated with sparse rewards in pure reinforcement learning. A dynamic balance between exploration and exploitation is constructed, enabling the policy to inherit the efficient operation of experts in typical scenarios while also possessing generalization capabilities for unstructured environments. Deep coupling with the joint flexibility features provided by the dual feedback system allows regularization constraints to focus on physical property-related action adjustments, enhancing the policy's adaptability to the robotic arm's intrinsic dynamics. Overall, seamlessly integrating expert prior knowledge with environmental interactive learning forms a physically-perceived policy evolution path, laying the algorithmic foundation for the synergistic improvement of accuracy, stability, and adaptability of the robotic arm's end effector in complex work scenarios.

[0047] Optionally, the reward function in the reinforcement learning fine-tuning phase is composed of the following terms: a direction reward term, which provides a positive reward when the end effector's motion direction points to the target position and a negative reward when it deviates; a velocity reward term, which dynamically rewards or penalizes based on the reasonableness of the velocity approaching the target; a final success reward term, which triggers a high one-time reward when the end effector enters the target pose tolerance range; and a vibration suppression penalty term, constructed based on the time integral of the square of the end effector's acceleration, used to suppress residual vibrations caused by joint flexibility.

[0048] Understandably, during the fine-tuning phase of reinforcement learning, the reward function constructs a refined guidance and physical constraint mechanism for policy behavior through the collaborative design of multi-dimensional reward items.

[0049] The directional reward is dynamically calculated based on the spatial angle between the current motion vector of the end effector and the target position vector. When the motion direction accurately approaches the target, a gradient positive incentive is provided, and a proportional penalty is applied when the direction deviates. This mechanism guides the strategy to learn efficient path planning logic, effectively suppresses ineffective wandering and trajectory oscillation, and ensures that the motion process always converges around the task target.

[0050] The speed bonus is dynamically assigned based on the degree of matching between the instantaneous speed of the end effector approaching the target and the preset ideal speed curve. When the speed is too fast, negative adjustment is applied to avoid impact risk and vibration excitation. When the speed is too slow, appropriate incentive is given to maintain task efficiency, prompting the strategy to generate a smooth motion trajectory that conforms to the dynamic characteristics.

[0051] The final success reward, as a key sparse signal, triggers a high one-time reward when the end effector's pose accurately enters the preset tolerance range. This clearly marks the task completion node, strengthens the strategy's ultimate understanding of goal achievement, and solves the learning efficiency problem in sparse reward environments.

[0052] The vibration suppression penalty term is constructed by the squared value of the three-axis acceleration of the real-time integral end effector. This integral value physically represents the intensity of kinetic energy fluctuation and residual vibration energy during the motion process. When the strategy output causes a sudden acceleration change or high-frequency oscillation, the penalty term increases, driving the strategy to actively avoid operation modes that are prone to vibration during the action generation stage.

[0053] Deformation caused by joint flexibility is often accompanied by abnormal acceleration fluctuations. The penalty mechanism prompts the strategy to optimize motion smoothness in advance when abnormal flexibility characteristics are detected. For example, in precision assembly scenarios, the strategy learns to appropriately reduce acceleration before approaching the contact point, thereby suppressing residual vibrations caused by the elastic deformation of the reducer from the source.

[0054] The four reward items mentioned above are organically integrated: the direction and speed reward item constructs a process guidance framework, the success reward item establishes goal orientation, and the vibration suppression penalty item embeds physical constraints. Together, they shape a control strategy that combines task efficiency, motion smoothness, and system stability. This enhances the strategy's adaptability to the inherent dynamic characteristics of the robotic arm, effectively alleviating the trajectory jitter and positioning drift problems caused by neglecting joint flexibility in traditional position control. Simultaneously, it avoids the strong dependence on high-precision force sensors and the complexity of force-position coupling design in pure force control methods. It achieves high-precision compliant control without the need for external optical measurement equipment, providing solid algorithmic support for the reliable operation of industrial robots in unstructured environments.

[0055] As an optional embodiment, in step 102, the state information (including visual data, robotic arm body parameter data, and joint flexibility feature data collected by the dual feedback system) in the imitation learning dataset is preprocessed to normalize and eliminate dimensional differences. Next, the trainable parameters of the policy network (including weight parameters of the feature extraction module, feature modulation module, and action generation module) are mapped into multiple independent parameter vectors, each corresponding to an optimization unit, forming a parallel optimization set. For each parameter vector, symmetrical virtual parameter samples are constructed on both sides of the current parameter vector to simulate a dual-angle perception mechanism. The parameter vector and the virtual samples on both sides are substituted into the policy network, and the preprocessed state information is input to generate predicted actions. The mean square error between each predicted action and the standard action is calculated as an adaptation evaluation index. Furthermore, by comparing the adaptation evaluation index of the virtual samples on both sides of the same parameter vector, the update direction and step size of the same parameter vector are determined, allowing direct optimization without calculating gradients. Finally, the parallel optimization set is processed synchronously to traverse the parameter space until the mean square error corresponding to all parameter vectors converges to a preset threshold, so that the policy network forms a target policy for reproducing expert operation behavior.

[0056] Specifically, in the above embodiments, firstly, the state information and corresponding standard actions in the imitation learning dataset are preprocessed. The visual data, robotic arm body parameter data, and joint flexibility feature data collected by the dual feedback system in the state information are normalized to eliminate dimensional differences between different types of data and ensure the stability of the training process. Next, a parallel parameter optimization mechanism is constructed, mapping the trainable parameters in the policy network to multiple independent parameter vectors. Each parameter vector corresponds to a complete policy network parameter configuration, forming a parameter exploration set. For each parameter vector, two virtual parameter combinations are set on either side of the current parameter value. Each parameter combination is substituted into the policy network, inputting the preprocessed state information to generate the corresponding predicted action. The mean square error between the predicted action and the expert standard action is calculated, and this mean square error is used as the fitness evaluation index for the parameter combination. Then, based on the difference in fitness evaluation indices between the two virtual parameter combinations, the update direction and step size of the current parameter vector are determined. If the mean square error of one virtual parameter combination is smaller, the parameter vector is adjusted along the direction with the smaller mean square error. If the difference between the two sides is less than a set difference threshold, the current update step size is maintained and the exploration direction is fine-tuned to achieve direct parameter optimization without gradients. Finally, the parallel optimization set is processed synchronously to traverse the parameter space, forming a group collaborative optimization mechanism, while configuring an adaptive step size strategy. For example, a larger step size is used in the early stage of optimization for global exploration, and the step size is gradually reduced as the iteration progresses, focusing on high-quality parameter regions for local fine optimization. The above parameter update process is continuously iterated until the mean squared error corresponding to all parameter vectors converges to a preset threshold, so that the policy network forms a basic policy that can reproduce the expert's operating behavior.

[0057] It is worth noting that in step 102, the policy network training employs a two-stage fusion optimization mechanism. The supervised learning stage uses the imitation learning dataset as input, minimizing the mean squared error between the predicted action and the expert's standard action, enabling the policy network to initially grasp the operational logic including joint flexibility feature perception. The reinforcement learning fine-tuning stage uses the pre-trained network as the initial policy, implementing efficient parameter fine-tuning by unlocking specific parameters in the feature modulation module for optimization. In the parameter optimization stage, swarm intelligence optimization methods such as the beetle swarm search algorithm or the hybrid frog-jumping algorithm can be optionally introduced. The beetle swarm search algorithm simulates the biological antennae perception mechanism, achieving efficient directional exploration in a high-dimensional parameter space. The hybrid frog-jumping algorithm divides the parameter population into multiple meme groups, dynamically balancing global exploration and local fine-grained search capabilities through intra-group local co-evolution and inter-group global information mixing. These algorithms effectively overcome the limitations of traditional gradient methods that easily get trapped in local optima in non-convex optimization, and are particularly suitable for optimizing complex dynamic parameter spaces introduced by joint flexibility features. By combining a composite reward function consisting of directional reward, speed regulation, task success reward, and vibration suppression penalty term based on the square integral of acceleration, the policy network enhances its adaptability to joint deformation disturbances and motion smoothness while retaining expert operation, ultimately generating an end-point control strategy that combines operational accuracy, trajectory stability, and environmental robustness.

[0058] For example, the Beetle Group Search (BSA) algorithm can be used to accelerate parameter optimization in policy networks. This algorithm simulates the foraging behavior of longhorn beetles using their two tentacles to sense odor gradients, mapping the parameter vectors of the policy network to the beetle's position in the search space. For each individual beetle, a pair of virtual tentacles are placed on either side of its current position. The direction and step size of parameter updates are determined by comparing the fitness values ​​(i.e., prediction errors) corresponding to the positions of the left and right tentacles. Its core position update formula is as follows: ,in The iteration step size, For symbolic functions, This is the solution from the previous iteration. This is the solution for the current iteration. This is the position of the right antenna. The position of the left antenna. and These are the fitness values ​​for the right and left antennae, respectively. > Return 1 if the problem is minimized, otherwise return -1. (If the value is smaller, move to the right). This method eliminates the need to calculate gradients, achieving an efficient direct search. Utilizing the inherent parallel search mechanism of the longhorn beetle swarm algorithm, multiple longhorn beetle individuals are initialized to form a population, collaboratively exploring the parameter space and further accelerating convergence. An adaptive step-size strategy is employed, using a larger step size for global exploration in the early stages of optimization, and gradually decreasing the step size with iterations to achieve refined local exploration, thus balancing convergence speed and accuracy.

[0059] Thus, through the above steps, the left and right antenna positions can be generated for each candidate solution, the difference in the sign of the objective function response values ​​can be compared to determine the search direction, and the parameter space can be updated in a directional manner by combining an adaptive decay step size. This mechanism enables the algorithm to efficiently balance global exploration and local exploitation capabilities in a high-dimensional non-convex parameter space, effectively avoiding the defect of gradient-based methods that are prone to getting trapped in local optima, and is particularly suitable for complex dynamic optimization scenarios with the introduction of joint flexibility features. At the same time, a hybrid frog-leaping algorithm can be selected, which further enhances the robustness and convergence efficiency of parameter search by mixing local co-evolution within meme groups with global information between groups. Both types of swarm intelligence optimization methods focus on the efficient adjustment of specific parameters in the feature modulation module. While retaining expert operation, they improve the policy's adaptability to joint deformation perturbations and motion smoothness, and finally generate an end-effector control policy with high accuracy, strong stability and environmental robustness.

[0060] For example, the hybrid frog leaping algorithm (SFLA) can be used to optimize the training of a policy network. First, the parameter vectors of the entire policy network are encoded as the positions of individual frogs, and the entire frog population is randomly initialized. Then, the entire population is divided into multiple memeplexes. Within each memeplex, individual frogs perform independent local searches and information exchanges, simulating the foraging behavior of frogs in a local pond. After several local evolutions, frogs from all memeplexes are re-mixed for global information exchange. This mechanism of grouping, local search, and global mixing effectively balances the algorithm's global exploration capability (avoiding premature entrapment in local optima) and local search capability (accelerating convergence near potential optima). It is particularly suitable for optimizing high-dimensional, nonlinear policy network parameter spaces, and through co-evolution within memeplexes, it may discover solutions more robust than traditional PSO.

[0061] Further optionally, in the above embodiments, it is assumed that the reinforcement learning fine-tuning stage employs a deep deterministic policy gradient algorithm. Based on this, the supervised-trained policy network is used as the policy network for reinforcement learning fine-tuning. A parameter-efficient fine-tuning strategy is adopted, which, while freezing the main structural parameters of the policy network, unlocks and optimizes specific parameters in the feature modulation module, including: The supervised training policy network is divided into a feature extraction module responsible for extracting state features from the original input, a feature modulation module responsible for adaptively adjusting the state features, and an action generation module responsible for outputting control commands. During the reinforcement learning fine-tuning stage, the network parameters of the feature extraction module and the action generation module are frozen. Specific parameters in the feature modulation module are unlocked and optimized. These specific parameters include at least one of the following: adaptation layer weight parameters, query projection matrix parameters, key projection matrix parameters, value projection matrix parameters in the attention mechanism, and gating parameters of the gated recurrent unit. The specific parameters are then weighted and fused into the original parameters of the feature modulation module to achieve targeted adjustment of the policy network.

[0062] This optimizes the network regions most sensitive to joint flexibility disturbances, enabling the policy network to specifically enhance its response to physical signals such as positional deviations between the motor and linkage ends. For example, when the dual feedback system detects abnormal joint deformation, the optimized attention parameters automatically increase the weights of the flexibility feature channels, guiding the policy to generate pre-trajectory compensation and smooth motion adjustments. While reducing the size of trainable parameters and computational overhead, the policy network can quickly adapt to the dynamic characteristics of new tasks while inheriting expert behavior patterns. Combined with the synergistic optimization of the policy network and value network in the deep deterministic policy gradient algorithm, the final generated control strategy achieves synergistic improvements in trajectory accuracy, vibration suppression capability, and environmental adaptability, providing an efficient and reliable control solution for robotic arms in demanding scenarios such as precision assembly and compliant interaction.

[0063] Optionally, the supervised training policy network can be used as the policy network for reinforcement learning fine-tuning. A parameter-efficient fine-tuning strategy can be employed. Before unlocking and optimizing only certain specific parameters while freezing the main network parameters, a value network can be constructed to evaluate the long-term cumulative value of state-action pairs, providing a quantitative evaluation signal for policy optimization. A hybrid experience replay pool is maintained, consisting of a pre-collected offline expert data pool and a data pool generated in real-time during online interaction. During the reinforcement learning fine-tuning stage, the policy gradient is calculated based on the quantitative evaluation signal provided by the value network, and the policy network is iteratively optimized with the goal of maximizing long-term cumulative rewards. The optimized policy network is then integrated into the robotic arm's control system for real-time generation of control commands.

[0064] Understandably, during the fine-tuning phase of reinforcement learning, the construction of the value network provides crucial long-term reward evaluation for policy optimization. This network employs an architecture that trains in conjunction with the policy network, continuously updating parameters through a temporal difference learning mechanism to accurately predict the total reward that any combination of states and actions can accumulate in future tasks. Its input integrates current multimodal state information and corresponding action instructions, outputting a quantified evaluation value that provides stable and reliable optimization direction guidance for policy gradient calculation. For example, in assembly tasks, the value network can identify the potential contribution of minute trajectory adjustments to subsequent insertions, guiding the policy to focus on high-value decision paths.

[0065] The hybrid experience replay pool employs a dual-track fusion mechanism of offline and online data. The offline expert data pool fully preserves the high-quality demonstration trajectories collected during the imitation learning phase, including implicit compensation operations by experts for joint flexibility perturbations under typical working conditions, providing a safe and reliable basis for strategy fine-tuning. The online interactive data pool dynamically collects new experiences generated by strategies in the simulation environment, covering adaptive attempts to deal with unknown perturbations and strategies for handling boundary conditions. These two types of data are dynamically mixed and stored according to an importance sampling strategy, and a priority replay mechanism is introduced, assigning higher sampling weights to samples with larger temporal difference errors, improving the utilization efficiency of key learning signals, and effectively alleviating the sample efficiency bottleneck of pure online learning and the risk of offline data distribution deviation.

[0066] The strategy optimization process aims to maximize long-term cumulative rewards. The algorithm samples experience batches from a hybrid experience replay pool, first updating the value network parameters to approximate the true reward function. Then, it fixes the value network and performs gradient ascent updates on specific parameters unlocked in the feature modulation module based on the policy gradient signal it outputs. Within this efficient parameter fine-tuning framework, it precisely focuses on key areas of physical perception, enabling the strategy to retain expert-level operation while specifically enhancing its responsiveness to joint flexibility features acquired by the dual-feedback system. After the optimized strategy network is deployed to the robotic arm control system, it can generate control commands in real-time by fusing vision, joint kinematics, and flexibility features, achieving high-precision trajectory tracking and active vibration suppression without the need for external measurement equipment.

[0067] Overall, the synergistic design of the value network and the hybrid experience replay pool constructs an optimized closed loop for knowledge inheritance and environmental adaptation. The value network provides unbiased long-term reward evaluation, enhancing the physical rationality of policy gradients. The hybrid data mechanism balances the stability of prior knowledge with exploration diversity, accelerating convergence and enhancing policy robustness. It effectively avoids the shortcomings of traditional reinforcement learning, such as large training oscillations and low sample efficiency, enabling the policy network to deeply internalize joint flexibility compensation logic and vibration suppression strategies with minimal perturbation of initial knowledge. This helps assist robotic arms in achieving a synergistic improvement in accuracy, stability, and environmental adaptability in demanding scenarios such as precision assembly and compliant interaction.

[0068] Step 103: Integrate the trained policy network into the control system of the robotic arm, generate control commands based on the real-time sensed state information, and drive the end effector to complete the operation task corresponding to the control command.

[0069] Step 103 above deploys the optimized strategy network to the robotic arm's embedded control system. During operation, the system acquires visual images, joint kinematic parameters, and joint flexibility features output by the dual feedback system in real time during control cycles. After standardized preprocessing, these are input into the strategy network. The strategy network outputs end-effector motion commands and impedance control parameters in real time. The motion commands are converted into target joint angles through inverse kinematics calculation and executed by the underlying servo driver. The impedance parameters dynamically adjust the end-effector stiffness and damping coefficient, enabling the robotic arm to exhibit adaptive compliance characteristics when in contact with the environment. When the dual feedback system detects an abnormal increase in joint flexibility features, the strategy network automatically fine-tunes the trajectory reference point and optimizes motion smoothness based on training, suppressing vibration excitation at its source. This integrated solution achieves adaptive precision control of the robotic arm's end-effector without relying on external optical measurement equipment, reducing system deployment complexity and environmental constraints, while ensuring trajectory smoothness and safety during high-speed motion and contact operations.

[0070] Optionally, the control commands output by the policy network further include impedance parameters for controlling the end effector's interaction force. A reward function corresponding to the impedance parameters is constructed, the reward function containing optimization terms for the impedance parameters, to guide the policy in learning compliant interaction behavior.

[0071] Specifically, the control commands output by the strategy network synchronously include impedance parameters for regulating the interaction force at the end effector, specifically covering the stiffness coefficient and damping coefficient, which together define the dynamic force response characteristics of the robotic arm end effector when it comes into contact with the environment.

[0072] To guide the policy learning task towards adaptive and compliant interaction behavior, an impedance parameter optimization term is specifically constructed in the reward function. This optimization term consists of two collaborative parts: a penalty mechanism for impedance parameters deviating from the task's preset ideal value, where the preset ideal value is dynamically configured based on the material characteristics of the manipulated object, the task stage, and the joint flexibility state perceived by the dual feedback system. For example, a lower stiffness is set in the precision assembly insertion stage to allow for minor deformation compensation, while a moderate stiffness is set in the polishing task to maintain contact force stability. The impedance change smoothness reward mechanism applies negative adjustment to abrupt changes in stiffness and damping coefficients, encouraging continuous and smooth parameter adjustments. During reinforcement learning fine-tuning, the policy network maximizes the composite reward including this optimization term, autonomously learning strategies such as pre-adjusting impedance parameters before contact and dynamically fine-tuning based on real-time flexibility characteristics during contact, gradually mastering compliant interaction techniques such as buffering impacts, suppressing vibrations, and adapting to surface fluctuations. This enables robotic arms to achieve adaptive force control solely based on their own perception without the need for external high-precision force sensors. This improves operational safety and task robustness in scenarios sensitive to interactive forces, such as minimally invasive medical procedures, precision assembly, and human-machine collaboration. At the same time, it avoids the complex design requirements of force-position coupling controllers in traditional force control methods, reducing system hardware costs and deployment complexity.

[0073] Further optionally, when the robotic arm is a cable-driven or continuous robotic arm, the control instructions output by the policy network also include instructions for regulating tendon tension; and the reward function also includes a reward term for tension smoothness.

[0074] Optionally, the output of the strategy network includes motion control commands for the end effector, used to adjust the impedance parameters of the robotic arm's end effector stiffness and damping in real time. When the robotic arm is cable-driven or a continuum configuration, it outputs tendon tension regulation commands. The reward function includes: a task-based reward, a penalty for impedance parameters deviating from a preset ideal value, and a reward for the smoothness of the tendon tension change rate.

[0075] It is worth noting that in the context of tethered or continuous manipulators, the output instruction set of the policy network is further expanded. In addition to end-effector motion commands and impedance parameters, a reference instruction sequence for regulating the tension of each driving tendon is generated simultaneously. The kinematic and dynamic behavior of the manipulator is highly dependent on the coordinated distribution of tendon tension. Tension imbalance can easily lead to structural relaxation, local buckling, or end-effector jitter, directly affecting operational accuracy and structural safety. To guide the policy in learning the temporal smoothness and task adaptability of tension regulation, a tension smoothness reward term is added to the reward function. This reward term is evaluated by a continuity index (such as the absolute value of the first-order difference or acceleration characteristics) that quantifies the rate of change of tendon tension within adjacent control cycles. Negative regulation is applied to abrupt adjustments, encouraging the policy to generate a gradual and stable tension evolution trajectory.

[0076] During the fine-tuning process of reinforcement learning, the policy network autonomously learns multi-objective cooperative strategies through composite reward signals. In the trajectory tracking stage, it predicts tension requirements to maintain structural stiffness; in the contact interaction stage, it dynamically fine-tunes the tension distribution based on the joint flexibility characteristics perceived by the dual feedback system; and during motion state transitions (such as acceleration to uniform speed), it smoothly transitions tension commands to suppress vibration excitation.

[0077] For example, in minimally invasive surgery, a continuous endoscope needs to maintain a uniform distribution of tendon tension during bending and movement to avoid excessive tightness causing tissue compression or excessive looseness leading to shaky vision. When flexibly grasping fragile objects, the strategy adjusts the end-effector impedance and tendon tension in tandem based on object deformation feedback to ensure precise grasping force and gradual tension changes. This mechanism is deeply coupled with the perception of joint flexibility characteristics. When the dual-feedback system detects abnormal positional deviations, the system works in tandem to optimize tension commands and trajectory corrections, constructing a closed-loop control system of motion-force-tension.

[0078] This improves the trajectory stability and structural reliability of rope-driven and continuous robotic arms in confined spaces, effectively suppresses body vibration and end effector jitter caused by sudden tension changes, extends the service life of tendon components, and enhances human-robot interaction safety. It provides a control solution that combines precision, compliance, and robustness for flexible robots in highly sensitive scenarios such as medical intervention, precision testing, and unstructured environment operations.

[0079] As an optional embodiment, in step 101, before acquiring expert demonstration data to construct the imitation learning dataset, the state information input from the visual sensor can be analyzed using a deep neural network to determine the 3D pose of the manipulated object and locate key feature points. Then, a reference motion trajectory for the end effector is planned based on these key feature points. In trajectory planning, joint flexibility features collected by the dual feedback system are invoked in real time, and the pose parameters of the reference motion trajectory are dynamically corrected by combining the positional deviation between the motor end and the link end in the joint flexibility features. Finally, the corrected reference motion trajectory is added as prior knowledge to the state information of the policy network.

[0080] Understandably, the visual guidance trajectory generation step can be optionally performed before acquiring expert demonstration data to construct the imitation learning dataset.

[0081] For high-precision applications with small sample sizes, such as medical robot surgery or on-orbit operations of space robotic arms, a deep neural network model based on neural radiation field optimization (NeRF-Pose) is used to process multi-view RGB image sequences acquired by visual sensors. This model eliminates the need for pre-calibration of camera intrinsic and extrinsic parameters, directly outputting the 3D spatial pose and coordinates of sparse key feature points of the manipulated object through implicit scene representation. Based on the spatial topological relationships of the key feature points, a reference motion trajectory for the end effector is planned, and this trajectory is used as the state input dimension of the policy network embedded with structured prior knowledge. During trajectory generation, joint flexibility features acquired by a dual-feedback system are fused in real time. Specifically, positional deviation data between the motor end and the link end are used to construct a dynamic flexibility compensation term, which corrects the pose parameters of the reference trajectory online, effectively offsetting trajectory deviations caused by elastic deformation of the joint reducer, thus improving trajectory execution accuracy and operational safety.

[0082] For sensor-noise-sensitive scenarios such as precision industrial assembly or surface polishing, a joint network architecture consisting of a 3D mask autoencoder (3D-MAE) and a lightweight pose regression head is adopted. This architecture first uses 3D-MAE to perform unsupervised pre-training on point cloud or RGB-D data input from a visual sensor, reconstructing robust global and local geometric features of the target object through masked regions. The pose regression head outputs 3D pose and key feature point coordinates based on extracted features, simultaneously suppressing ambient lighting changes and sensor noise interference. In the feature fusion stage, joint flexibility features (including positional deviations between the motor and connecting rod ends) acquired in real-time by a dual-feedback system are used as physical priors and concatenated with visual features at the channel level to construct a vision-ontology joint representation. Key feature points located based on this joint representation generate a reference motion trajectory, which is then used as prior knowledge to supplement the policy network state input. In the trajectory planning stage, the flexibility compensation term dynamically adjusts the compensation intensity based on the joint representation, performing real-time pose correction on the reference trajectory to accurately compensate for trajectory deviations caused by joint flexibility, enhancing the system's planning robustness and execution consistency in noisy environments.

[0083] Further optionally, in the above embodiments, the step of dynamically correcting the pose parameters of the reference motion trajectory by real-time invocation of the joint flexibility features collected by the dual feedback system and combining the positional deviation between the motor end and the connecting rod end in the joint flexibility features includes: The pose estimation deviation between the 3D pose of the manipulated object and the corresponding target pose in the reference motion trajectory is determined. A dynamic compensation function is obtained by co-fitting the pose estimation deviation and the position deviation, with the pose estimation deviation and position deviation as inputs and the trajectory correction amount as output. The weight coefficients of the dynamic compensation function are adaptively adjusted according to the motion state of the end effector, including acceleration, constant speed, and deceleration. During acceleration, the flexibility compensation weight is increased to suppress deformation accumulation; during constant speed, the compensation weight is kept stable; and during deceleration, the compensation weight is gradually reduced to avoid residual vibration. The path point coordinates and attitude angles in the reference motion trajectory are corrected point-by-point based on the trajectory correction amount to generate a dynamic trajectory adapted to the joint flexibility characteristics.

[0084] It is worth noting that during the trajectory planning process, the pose parameters of the reference motion trajectory are dynamically corrected by first calculating the pose estimation deviation between the three-dimensional pose of the manipulated object and the target pose in the reference trajectory. This deviation quantifies the geometric difference between visual perception and planning intent.

[0085] Subsequently, the pose estimation bias is co-fitted with the positional deviations of the motor end and the link end in the joint flexibility features collected in real time by the dual feedback system to construct a dynamic compensation function. This function takes two types of physical deviations as input and outputs trajectory correction through nonlinear mapping. Its fitting process can be combined with an online learning mechanism to continuously optimize the compensation logic, so that the correction strategy can adaptively evolve with the progress of the task.

[0086] To enhance temporal adaptability, the system dynamically adjusts the weighting coefficients of the compensation function based on the real-time motion state of the end effector. During the acceleration phase, joint flexibility deformation accumulates rapidly with increasing acceleration. At this time, the flexibility compensation weight is increased, and corrections are applied in advance to suppress deformation development. For example, in precision assembly tasks, when the end effector starts from a standstill, the system enhances the compensation intensity to ensure the accuracy of the initial trajectory segment. During the constant speed phase, deformation tends to reach dynamic equilibrium, and the weights are maintained to ensure a smooth and continuous trajectory. During the deceleration phase, to avoid excessive correction that could trigger residual vibrations, the weighting coefficients gradually decay according to a preset curve to achieve a smooth transition. For example, during deceleration before inserting the workpiece, the flexibility compensation intensity is reduced to prevent end effector jitter from affecting contact quality. Based on the trajectory correction amount after adaptive weight adjustment, the three-dimensional coordinates and attitude angles of each path point in the reference trajectory are refined point-by-point to generate a dynamic trajectory that matches the current flexibility state of the robotic arm in real time.

[0087] The aforementioned mechanism deeply integrates external visual perception and internal ontological physical perception, proactively compensating for systematic offsets caused by joint flexibility at the trajectory planning source, thereby improving trajectory tracking accuracy and stability under high-speed motion. This effectively suppresses end-effector jitter and positioning drift caused by deformation accumulation, and balances the timeliness of correction and the stability of vibration suppression through a weighted adaptive strategy of motion state perception. This provides a physically-perceived trajectory generation guarantee for high-precision, high-reliability scenarios such as minimally invasive operations in medical robots and on-orbit assembly of space robotic arms, while reducing reliance on high-cost external measurement equipment.

[0088] Further optionally, in the above embodiments, based on the joint modeling features of 3D-MAE, the deviation error between the predicted position and the actual measured position of the key feature points of the manipulated object is calculated, and this error is used as the visual constraint for trajectory correction. A flexible deformation model is constructed based on the position deviation between the motor end and the connecting rod end. This model quantifies the influence of joint flexibility on the pose of the end effector and outputs the deformation compensation amount. The weight of the flexible compensation term is dynamically adjusted according to the synergistic relationship between the visual constraint error and the deformation compensation amount. When the visual constraint error is small, the weight of the deformation compensation amount is increased to prioritize compensating for the influence of joint flexibility. When the visual constraint error is large, the weight of the deformation compensation amount is decreased to prioritize correcting the pose estimation deviation. Finally, the adjusted flexible compensation term is superimposed on the corresponding path points of the reference trajectory, and the corrected trajectory segment is smoothly transitioned through an interpolation algorithm to avoid robot arm vibration caused by abrupt trajectory changes, while ensuring that the corrected trajectory meets the kinematic constraints of the end effector.

[0089] Optionally, in the above embodiments, the NeRF-Pose network outputs the real-time three-dimensional pose estimation result of the manipulated object and calculates the deviation value between the pose and the corresponding target pose in the reference trajectory. Then, the deviation value is co-modeled with the position deviation between the motor end and the connecting rod end to construct a dynamic compensation function. The compensation function takes the pose estimation deviation and the position deviation as input and outputs a trajectory correction amount. The weight coefficients of the compensation function are adaptively adjusted according to the motion state (acceleration / constant speed / deceleration) of the end effector. During the acceleration phase, the flexibility compensation weight is increased to suppress deformation accumulation; during the constant speed phase, the compensation weight is kept stable; and during the deceleration phase, the compensation weight is gradually reduced to avoid residual vibration. Finally, based on the trajectory correction amount, the path point coordinates and attitude angles of the reference trajectory are corrected point by point to generate a dynamic trajectory adapted to the joint flexibility characteristics.

[0090] Optionally, the policy network is trained using the imitation learning dataset within the meta-learning framework, enabling the policy network to quickly adapt to different tasks. This includes: during the meta-training phase, training a meta-policy model with general adaptability using expert demonstration data from multiple related tasks; when faced with a new task, acquiring small sample demonstration data for the new task, and the meta-policy model rapidly adjusting internally to generate a policy network for the new task by combining the small sample demonstration data; and inputting the flexibility parameters of the robotic arm as task context into the meta-policy model to improve the generalization ability of the policy network across different robotic arm configurations. The flexibility parameters of the robotic arm include joint flexibility features and / or tendon characteristic parameters.

[0091] Specifically, within the meta-learning framework, the training of the policy network achieves rapid adaptation across tasks and configurations through a two-stage mechanism. The meta-training stage integrates expert demonstration data from multiple related tasks, covering different operational scenarios (such as precision assembly and flexible grasping) and robotic arm configurations with differentiated physical characteristics (including rigid industrial arms, cable-driven arms, and continuum arms). The meta-optimization algorithm trains the meta-policy model, enabling it to internalize adaptive prior knowledge shared across tasks and flexible perception generalization capabilities.

[0092] When deployed to a new task, only a small sample of demonstration data (typically several successful operation trajectories) needs to be collected for that task. The meta-policy model then combines the demonstration data with the context of the robotic arm's flexibility parameters to perform rapid internal parameter adjustments, generating a customized policy network for the new task. The robotic arm's flexibility parameters, as the core task context, are explicitly encoded and input into the meta-policy model. These include joint flexibility features (positional deviation between the motor end and the link end) collected in real time by a dual feedback system, as well as tendon characteristic parameters specific to cable-driven or continuous robotic arms (such as preload coefficient and elastic modulus). This contextual mechanism enables the meta-policy model to establish a physical mapping between flexibility parameters and policy behavior.

[0093] During meta-training, the model learns that high-flexibility configurations require enhanced trajectory smoothness and vibration suppression weights, while low-flexibility configurations can focus on trajectory tracking accuracy. When adapting to new tasks, even when faced with robotic arm configurations not seen during training, the model can generate physically reasonable strategies based on the context of the input flexibility parameters.

[0094] For example, when replacing a new type of flexible endoscopic robotic arm in a medical setting, the meta-policy model can quickly generate a compliant control strategy adapted to the configuration by inputting its tendon characteristic parameters and a small number of puncture demonstrations.

[0095] When switching assembly tasks with different load characteristics in an industrial setting, only a small amount of teaching trajectory for the new workpiece is required, and the strategy can adapt by combining the current joint flexibility features.

[0096] This enhances the generalization ability and deployment efficiency of the policy network across tasks and configurations, reduces the amount of data required for adapting to new tasks to a small sample level, significantly shortens the on-site debugging cycle, and strengthens the policy's ability to perceive and adapt to the physical characteristics of the robotic arm. It effectively avoids the risk of control mismatch caused by configuration differences, and provides an efficient solution for the rapid deployment and high-reliability operation of multiple robotic arm models in unstructured environments.

[0097] In this embodiment, a dual-feedback system is used to explicitly perceive joint flexibility characteristics and integrate them throughout the entire chain of data construction, strategy training, and real-time control to construct a physical characteristic-driven intelligent control. The imitation learning stage internalizes the implicit compensation experience of experts regarding flexible deformation, while the reinforcement learning fine-tuning stage enhances the robustness of the strategy to dynamic disturbances through efficient parameter optimization and vibration suppression reward mechanisms. This method fundamentally alleviates the trajectory jitter and positioning drift caused by neglecting joint flexibility in traditional position control, avoids the strong dependence on high-precision force sensors and the complexity of force-position coupling design in pure force control, and also overcomes the limitations of external measurement systems in terms of real-time performance and industrial site applicability. The resulting control strategy, while ensuring operational accuracy, improves the compliant interaction capability and operational stability of the robotic arm in precision tasks such as assembly and grinding, providing a practical technical path for highly reliable operation of industrial robots in unstructured environments.

[0098] The above describes a robotic arm end effector control method that integrates imitation learning and reinforcement learning in the embodiments of this application. The following describes a robotic arm end effector control device that integrates imitation learning and reinforcement learning to perform the above method.

[0099] See Figure 2 ,like Figure 2 The diagram shows a structural schematic of a robotic arm end effector control device that integrates imitation learning and reinforcement learning. The robotic arm end effector control device integrating imitation learning and reinforcement learning in this embodiment can achieve the above-mentioned... Figure 1 The steps of the robotic arm end effector control method integrating imitation learning and reinforcement learning are executed in the corresponding embodiments. The functions implemented by the robotic arm end effector control device integrating imitation learning and reinforcement learning can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware. The robotic arm end effector control device integrating imitation learning and reinforcement learning may include an input / output module 201 and a processing module 202. The functional implementation of the processing module 202 and the input / output module 201 can be referred to Figure 1The operations performed in the corresponding embodiments will not be described in detail here. For example, the processing module 202 can be used to control the sending, receiving, and acquiring operations of the input / output module 201.

[0100] The input / output module 201 is configured to acquire expert demonstration data to construct an imitation learning dataset, wherein the expert demonstration data includes state information and corresponding action information. The processing module 202 is configured to train a policy network using the imitation learning dataset, and to enable the policy network to reproduce the expert's operational behavior through supervised learning by fusing imitation learning and reinforcement learning. The state information input to the policy network also includes joint flexibility features collected by a dual feedback system consisting of a motor encoder and a link end position sensor, the joint flexibility features including at least the positional deviation between the motor end and the link end. The trained policy network is integrated into the control system of the robotic arm, and control commands are generated based on the real-time perceived state information to drive the end effector to complete the operation task corresponding to the control commands.

[0101] In this embodiment, the robotic arm end effector control device that integrates imitation learning and reinforcement learning can sense the joint flexibility characteristics and efficiently optimize the strategy network parameters to obtain a robotic arm end effector control strategy that combines operational accuracy, compliance, and stability. This reduces the operational risks caused by technical defects such as trajectory jitter in traditional position-based control, sensor dependence and force-pose coupling conflict in force-based control, and insufficient real-time performance of external measurement control, thereby improving the control accuracy of the robotic arm end effector and optimizing the control effect.

[0102] The above describes the robotic arm end effector control device 20 that integrates imitation learning and reinforcement learning in the embodiments of this application from the perspective of modular functional entities. The following describes the robotic arm end effector control device that integrates imitation learning and reinforcement learning in the embodiments of this application from the perspective of hardware processing.

[0103] It should be noted that, Figure 2 The physical device corresponding to the input / output module 201 shown can be a transceiver, radio frequency circuit, communication module, and input / output (I / O) interface, etc., and the physical device corresponding to the processing module 202 can be a processor.

[0104] Figure 2 The devices shown can all have the following characteristics: Figure 3 The structure shown, when Figure 2 The robotic arm end effector control device 20 shown integrates imitation learning and reinforcement learning, and has the following characteristics: Figure 3 When the structure shown is used, Figure 3The processor and transceiver in the device can perform the same or similar functions as the processing module 202 and input / output module 201 provided in the aforementioned device embodiments. Figure 3 The memory stores the computer programs that the processor needs to call when executing the above-mentioned robotic arm end effector control method that combines imitation learning and reinforcement learning.

[0105] This application also provides a server; please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1100 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1122 (e.g., one or more processors) and memory 1132, and one or more storage media 1130 (e.g., one or more mass storage devices) for storing application programs 1142 or data 1144. The memory 1132 and storage media 1130 can be temporary or persistent storage. The program stored in the storage media 1130 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the CPU 1122 may be configured to communicate with the storage media 1130 and execute the series of instruction operations in the storage media 1130 on the server 1100.

[0106] Server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input / output interfaces 1158, and / or one or more operating systems 1141, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0107] The steps performed by the server in the above embodiments can be based on this Figure 4 The structure of server 1100 is shown. For example, in the above embodiment, it consists of... Figure 4 The server architecture or other type of device shown. For example, the central processing unit 1122 performs the following operations by calling instructions in memory 1132: An imitation learning dataset is constructed by acquiring expert demonstration data through input / output interface 1158, wherein the expert demonstration data includes state information and corresponding action information; a policy network is trained using the imitation learning dataset, and supervised learning enables the policy network to reproduce the expert's operational behavior by fusing imitation learning and reinforcement learning; wherein the state information input to the policy network also includes joint flexibility features collected by a dual feedback system consisting of a motor encoder and a link end position sensor, wherein the joint flexibility features include at least the positional deviation between the motor end and the link end; the trained policy network is integrated into the control system of the robotic arm, and control commands are generated based on the real-time perceived state information to drive the end effector to complete the operation task corresponding to the control command.

[0108] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0110] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0112] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0113] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0114] The technical solutions provided in the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. A method for controlling the end effector of a robotic arm that integrates imitation learning and reinforcement learning, characterized in that, The method includes: A dataset for imitation learning is constructed by acquiring expert demonstration data, wherein the expert demonstration data includes state information and corresponding action information; The policy network is trained using the imitation learning dataset, and supervised learning enables the policy network to reproduce the expert's operational behavior by fusing imitation learning and reinforcement learning. The state information input to the strategy network also includes: joint flexibility features collected by a dual feedback system consisting of a motor encoder and a link end position sensor, wherein the joint flexibility features include at least the positional deviation between the motor end and the link end. The trained policy network is integrated into the control system of the robotic arm, and control commands are generated based on real-time perceived state information to drive the end effector to complete the operation tasks corresponding to the control commands.

2. The robotic arm end effector control method integrating imitation learning and reinforcement learning according to claim 1, characterized in that, The step of training a policy network using the imitation learning dataset and then using supervised learning to enable the policy network to reproduce the expert's operational behavior by fusing imitation learning and reinforcement learning includes: Using the state information in the imitation learning dataset as input, a policy network is trained through supervised learning. The optimization objective is to minimize the mean square error between the predicted action output by the policy network and the standard action labeled in the expert demonstration data. Using the supervised training policy network as the policy network for reinforcement learning fine-tuning, a parameter-efficient fine-tuning strategy is adopted. Under the premise of freezing the main structural parameters of the policy network, specific parameters in the feature modulation module are unlocked and optimized.

3. The robotic arm end effector control method integrating imitation learning and reinforcement learning according to claim 2, characterized in that, The step of using state information from the imitation learning dataset as input, training a policy network through supervised learning, and minimizing the mean squared error between the predicted action output by the policy network and the standard action labeled in the expert demonstration data as the optimization objective includes: Normalize the state information and corresponding standard actions in the imitation learning dataset to eliminate dimensional differences; The trainable parameters of the policy network are mapped to multiple sets of independent parameter vectors, each set of parameter vectors corresponds to an optimization unit, forming a parallel optimization set; For each set of parameter vectors, symmetrical virtual parameter samples are constructed on both sides of the current parameter vector to simulate a dual-angle sensing mechanism. The parameter vector and the virtual samples on both sides are substituted into the policy network, and the preprocessed state information is input to generate the predicted action. The mean square error between each predicted action and the standard action is calculated as the fitness evaluation index. By comparing the fitness evaluation index of virtual samples on both sides of the same parameter vector, the update direction and step size of the same parameter vector can be determined, and direct optimization can be completed without calculating the gradient. The parallel optimization set is processed synchronously to traverse the parameter space until the mean square error corresponding to all parameter vectors converges to a preset threshold, so that the policy network forms a target policy for reproducing expert operation behavior.

4. The robotic arm end effector control method integrating imitation learning and reinforcement learning according to claim 2, characterized in that, The supervised training policy network is used as the policy network for reinforcement learning fine-tuning. A parameter-efficient fine-tuning strategy is employed, which, while freezing the main structural parameters of the policy network, unlocks and optimizes specific parameters in the feature modulation module, including: The supervised training policy network is divided into a feature extraction module, a feature modulation module, and an action generation module. During the reinforcement learning fine-tuning phase, the network parameters of the feature extraction module and the action generation module are frozen; Unlock and optimize specific parameters in the feature modulation module, including at least one of the following: adaptation layer weight parameters, query projection matrix parameters, key projection matrix parameters, value projection matrix parameters, and gate parameters of the gated loop unit; The specific parameters are weighted and fused into the original parameters of the feature modulation module to achieve targeted adjustment of the policy network.

5. The robotic arm end effector control method integrating imitation learning and reinforcement learning according to claim 2, characterized in that, The method of using a supervised-trained policy network as a policy network for reinforcement learning fine-tuning, employing a parameter-efficient fine-tuning strategy, includes, before unlocking and optimizing only certain specific parameters while freezing the main parameters of the network: Construct a value network to evaluate the long-term cumulative value of state-action pairs and provide quantitative evaluation signals for policy optimization; Maintain a hybrid experience replay pool, which is composed of a pre-collected offline expert data pool and a data pool generated in real time during online interaction; During the fine-tuning phase of reinforcement learning, the policy gradient is calculated based on the quantitative evaluation signal provided by the value network, and the policy network is iteratively optimized with the goal of maximizing long-term cumulative rewards. The optimized strategy network is integrated into the control system of the robotic arm to generate control commands in real time.

6. The robotic arm end effector control method integrating imitation learning and reinforcement learning according to claim 1, characterized in that, Before acquiring expert demonstration data to construct the imitation learning dataset, the following steps are also included: By analyzing the state information input from the vision sensor through a deep neural network, the three-dimensional pose of the object being manipulated is determined and key feature points are located. The reference motion trajectory of the end effector is planned based on the key feature points; In trajectory planning, the joint flexibility features collected by the dual feedback system are called in real time, and the positional deviation between the motor end and the connecting rod end in the joint flexibility features is combined to dynamically correct the pose parameters of the reference motion trajectory. The corrected reference trajectory is added as prior knowledge to the state information of the policy network.

7. The robotic arm end effector control method integrating imitation learning and reinforcement learning according to claim 6, characterized in that, In the trajectory planning, the joint flexibility features collected by the dual feedback system are invoked in real time. Combined with the positional deviation between the motor end and the connecting rod end in the joint flexibility features, the pose parameters of the reference motion trajectory are dynamically corrected, including: Determine the pose estimation deviation between the three-dimensional pose of the manipulated object and the corresponding target pose in the reference motion trajectory; Based on the pose estimation deviation and the position deviation, a dynamic compensation function is obtained by co-fitting. The dynamic compensation function takes the pose estimation deviation and the position deviation as inputs and outputs the trajectory correction amount. The weight coefficients of the dynamic compensation function are adaptively adjusted according to the motion state of the end effector; wherein the motion state of the end effector includes acceleration, constant speed, and deceleration; the flexible compensation weight is increased during the acceleration phase to suppress deformation accumulation, the compensation weight is kept stable during the constant speed phase, and the compensation weight is gradually reduced during the deceleration phase to avoid residual vibration. Based on the trajectory correction amount, the coordinates of the path points and the attitude angles in the reference motion trajectory are corrected point by point to generate a dynamic trajectory that adapts to the joint flexibility characteristics.

8. The robotic arm end effector control method integrating imitation learning and reinforcement learning according to claim 1, characterized in that, The control commands output by the strategy network also include impedance parameters for controlling the end effector interaction force of the robotic arm's end effector. Construct a reward function corresponding to the impedance parameter, the reward function containing an optimization term for the impedance parameter, which is used to guide the policy to learn compliant interaction behavior.

9. The robotic arm end effector control method integrating imitation learning and reinforcement learning according to claim 1, characterized in that, When the robotic arm is a cable-driven or continuous robotic arm, the control commands output by the strategy network also include commands for regulating tendon tension; and The reward function also includes a reward term for tension smoothness.

10. The robotic arm end effector control method integrating imitation learning and reinforcement learning according to claim 1, characterized in that, Within the meta-learning framework, a policy network is trained using the aforementioned imitation learning dataset, enabling the policy network to rapidly adapt to different tasks, including: During the meta-training phase, a meta-policy model with general adaptability is trained using expert demonstration data from multiple related tasks. When faced with a new task, a small sample demonstration data of the new task is obtained. The meta-policy model is then rapidly adjusted internally and combined with the small sample demonstration data to generate a policy network for the new task. The flexibility parameters of the robotic arm are input into the meta-policy model as the task context to improve the generalization ability of the policy network across different robotic arm configurations. The flexibility parameters of the robotic arm include joint flexibility features and / or tendon characteristic parameters.

Images