Method for robot arm approaching valve and end pose alignment based on deep reinforcement learning

Abstract

The application discloses a method for valve approaching and end posture alignment of a mechanical arm based on deep reinforcement learning, and belongs to the technical fields of artificial intelligence, intelligent robot control and deep reinforcement learning. The method comprises the following steps: a physical simulation model containing a mechanical arm and a valve is constructed, and a spatial coordinate system of a simulation environment is established; according to the physical simulation model and the spatial coordinate system, an environment reset condition, a state space S, an action space A and a reward function R of reinforcement learning training are defined; a pre-constructed policy network is trained by using a proximal policy optimization algorithm, so as to obtain a control strategy model for controlling the mechanical arm to approach the valve and complete end posture alignment; and the trained control strategy model is used to control the mechanical arm to perform valve approaching and end posture alignment operations. The method can control the mechanical arm to enter a standard and compliant pre-grasping state before valve screwing, and effectively reduce the risk caused by posture divergence, physical interference or tripping in the subsequent screwing process.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, intelligent robot control, and deep reinforcement learning, and more specifically to a method for aligning the posture of a robotic arm approaching a valve with its end effector based on deep reinforcement learning. Background Technology

[0002] In recent years, with the improvement of industrial automation, the maintenance needs in fields such as intelligent manufacturing, unmanned factories, and chemical pipeline maintenance have been increasing. Robotic arms are widely used in complex and delicate operations such as valve turning. Using robotic arms to replace manual labor in high-risk valve operations effectively improves operational safety and efficiency. At the same time, the maturity of deep reinforcement learning and simulation technologies has reduced the reliance on traditional complex trajectory planning, enabling reinforcement learning-trained robotic arm systems to possess powerful end-to-end execution capabilities, facilitating large-scale deployment in large industrial facilities.

[0003] However, large-angle continuous turning, a core component of valve operation, has always been a research challenge in the field of robot control. In industrial environments with significant randomness, if reinforcement learning training is conducted solely using pure kinematic feedback, neural networks are prone to "catastrophic forgetting" and pose singularities. Especially in the initial training phase, the lack of a unified and standardized initial pose definition can lead to confusion in the agent's exploration direction, resulting in slow convergence or even divergence. Furthermore, imprecise grasping postures can cause the robotic arm to slip or the gripper to disengage during valve turning. As the turning angle increases, if a strictly orthogonal grasping posture cannot be maintained, motion deviations will increase significantly, leading to a sharp rise in the risk of operational failure. Therefore, establishing a standardized training starting point, providing a stable and compliant initial state for reinforcement learning through preset pose constraints, is a crucial prerequisite for ensuring efficient training convergence and obtaining a high-performance policy model. Summary of the Invention

[0004] The purpose of this invention is to propose a method for aligning the posture of a robotic arm approaching a valve with its end effector based on deep reinforcement learning. This method aims to solve the technical problems of slow training convergence, unstable strategies, and easy operation failures caused by the lack of standardized pre-grasping posture guidance when training a robotic arm to directly perform valve turning tasks in the prior art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] This invention provides a method for aligning the posture of a robotic arm approaching a valve with its end effector based on deep reinforcement learning, comprising the following steps: Construct a physical simulation model that includes a robotic arm and valves, and establish the spatial coordinate system of the simulation environment; Based on the physical simulation model and spatial coordinate system, the environment reset conditions, state space S, action space A, and reward function R for reinforcement learning training are defined. Based on the environment reset conditions, state space S, action space A and reward function R of the reinforcement learning training, the pre-constructed policy network is trained using the proximal policy optimization algorithm to obtain a control policy model that controls the robotic arm to approach the valve and complete the end-effector posture alignment. Using the trained control strategy model, the robotic arm is controlled to perform an operation to approach the valve and align its end effector.

[0007] In one embodiment, the environment reset conditions for defining reinforcement learning training include: At the beginning of each training round, the joint positions of the robotic arm are reset to a fixed initial reference pose. The joint velocity is reset to zero; At the start of each training round, a random offset is applied to the initial position of the valve in the world coordinate system. ,in It follows a uniform distribution within a preset range; End of round condition When the time step reaches or exceeds the preset time limit, the timeout ends and the environment is reset; otherwise, the system continues to explore. Noise initialization conditions: These are used to apply bounded random perturbations to relevant physical parameters during training, so that the policy network is exposed to diverse dynamic conditions during simulation training, thereby narrowing the gap between the perception and dynamic distribution in simulation and reality.

[0008] In one embodiment, the state space S is expressed as follows:

[0009] In the formula, Indicates the state of the robotic arm itself. This indicates the relative orientation of the valve and the clamp. Indicates the action state at the previous time step; The state of the robotic arm itself The expression is as follows:

[0010] in: , These represent the position and velocity of the six degrees of freedom joints of the robotic arm, respectively. It indicates the position of the end effector's driven joint, which can be directly read by querying the joint state tensor of the underlying physics engine in real time; , These represent the position coordinates and quaternion orientation of the end effector in the world coordinate system, respectively, which are obtained in real time from the forward kinematics of the robotic arm. The expression for the relative orientation state of the valve and the clamp is as follows:

[0011] The three-dimensional spatial position of the valve handle in the world coordinate system is obtained through coordinate transformation. The formula is as follows:

[0012] in, This represents the relative position vector between the end gripper and the valve handle. This represents the homogeneous transformation matrix of the valve base coordinate system relative to the world coordinate system; through spatial vector difference operations, the relative position vector between the end gripper and the valve handle is calculated in real time by subtracting the current gripper coordinate components from the target handle coordinate components. ; The action state at the previous time step The action instruction vector output by the policy network at the previous time step. .

[0013] In one embodiment, the action space A is expressed as follows:

[0014] in: This represents the six-degree-of-freedom pose increment command of the robotic arm's end effector in a spatial coordinate system, including the translation increment of the spatial position. and the rotational increment of spatial attitude ; This indicates the opening and closing control command of the end effector of the robotic arm, representing the continuous opening and closing degree from fully open (0) to fully closed (1).

[0015] In one embodiment, the reward function R is used to guide the robotic arm end effector to approach and align with the valve handle in a standard pre-grasping pose, and includes at least: A proximity reward is calculated based on the distance between the current position of the end effector and the position of a standard pre-grab target. An attitude constraint reward is calculated based on the axial vector alignment between the local coordinate system of the end effector and the local coordinate system of the valve handle. A penalty term calculated based on the smoothness of the robotic arm's movements.

[0016] In one embodiment, the standard pre-fetch target location is determined in the following manner:

[0017] The position of the valve handle in the world coordinate system Based on the ideal grasping point, the three-dimensional pose offset vector from the grasping point to the valve handle target point is: The position of the end effector in the standard pre-grabbing posture is calculated by combining the offset vector. .

[0018] In one embodiment, the proximity reward item The calculation method is as follows:

[0019]

[0020] Calculate the current position of the end effector. Compared with the standard pre-grabbing posture position The Euclidean distance d between them; Rewards are calculated based on distance d, when d is less than or equal to a preset threshold. When d is greater than 1, the first reward calculation formula is used; when d is greater than 1. When the first reward calculation formula is used, the reward value provided by the first reward calculation formula is higher than the reward value of the second reward calculation formula at the same distance.

[0021] In one embodiment, the attitude constraint reward item Alignment is measured by calculating the dot product of the unit vectors between the preset axes of the local coordinate system of the end gripper and the preset axes of the local coordinate system of the valve handle, as expressed below:

[0022]

[0023]

[0024] in, , Let represent the unit vectors of the y-axis and z-axis of the local coordinate system of the robotic arm's end effector gripper, respectively. , Let x and y represent the unit vectors of the local coordinate system of the valve handle, respectively. This is a sign function used to preserve the original directional characteristics of the angle between vectors.

[0025] In one embodiment, the penalty item This includes penalties for differences in motion vectors between adjacent time steps and / or penalties for robot joint velocities, expressed as follows:

[0026] in, and These represent the action instruction vectors output by the policy network at the current time step and the previous time step, respectively. Let be the joint velocity vector of the robotic arm. and These represent the penalty weights for the rate of change of motion and the penalty weights for joint velocity, respectively.

[0027] In one embodiment, training using the proximal policy optimization algorithm includes: The exploration is performed synchronously in multiple parallel simulation sub-environments to collect interaction trajectory data; The advantage function is calculated using the generalized advantage estimation algorithm; The policy network parameters are optimized and updated based on the pruning and substitution objective function.

[0028] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method for aligning the robotic arm's approach to a valve and its end effector posture based on deep reinforcement learning. It can intuitively characterize the geometric posture of the robotic arm's end effector and the valve through pure kinematic description and spatial three-dimensional vector projection verification. On the one hand, it uses spatial posture constraint rewards for reinforcement learning training to constrain the robotic arm to reach the pre-grasping posture, realizing the pre-grasping action of any valve within the working range of the robotic arm. On the other hand, it simulates real noise conditions and enhances adaptability through random perturbations in training. The pre-grasping posture output by the obtained model can avoid catastrophic forgetting problems such as posture distortion and gripper disengagement in subsequent valve turning training, thereby improving the stability of future turning training.

[0029] This invention offers strong control stability and superior algorithm generalization, enabling its widespread application in pre-grabbing training for various heterogeneous valves, valves with different initial approach angles, and valves subject to dynamic observation disturbances. It provides a reliable underlying technical solution for precise positioning and universal grasping of robotic arms in complex industrial scenarios. For example, complex industrial scenarios can be categorized into the following typical cases: (1) Target heterogeneity and scale variation scenarios: The model can be directly generalized to heterogeneous valve pre-grabbing tasks with different geometric shapes, such as long strip handle valves, irregular handles, different sizes and specifications, and different material surfaces; (2) Strict pose constraint scenario: For valves with arbitrary non-orthogonal tilt angles installed in actual factory buildings, the strategy network can adaptively adjust according to the current observation to ensure that the end gripper of the robotic arm can eventually approach the plane where the valve handle is located in a relatively vertical posture. (3) Dynamic differences and underlying perception noise scenarios: The strategy model can effectively overcome the objective gap between the simulation environment and the real physical world, resist the random noise of the visual sensor caused by electrical interference, slight vibration of equipment or fluctuation of basic lighting in the real industrial site, and has strong robustness to the changes in joint friction coefficient and end load deviation caused by long-term operation of the robotic arm. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0031] Figure 1 This is a flowchart of a method for aligning the posture of a robotic arm approaching a valve with its end effector, based on deep reinforcement learning, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the initial posture and standard pre-grasping posture of the robotic arm provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the geometric principle of the end-effector attitude constraint reward item provided in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] See Figure 1 As shown, this invention provides a method for aligning the robotic arm's approach to a valve with its end effector posture based on deep reinforcement learning. This method, based on deep reinforcement learning, defines the initial and final conditions, state space, action space, and reward function for the end effector posture constraints. It then uses a proximal policy optimization algorithm to train a policy model for aligning the robotic arm's approach to the valve with its posture. The method includes the following steps S1 to S4: S1. Construct a physical simulation model that includes a robotic arm and valves, and establish the spatial coordinate system of the simulation environment; S2. Based on the physical simulation model and spatial coordinate system, define the environment reset conditions, state space S, action space A, and reward function R for reinforcement learning training; S3. Based on the environment reset conditions, state space S, action space A and reward function R of the reinforcement learning training, the pre-constructed policy network is trained using the proximal policy optimization algorithm to obtain a control policy model that controls the robotic arm to approach the valve and complete the end-effector posture alignment. S4. Using the trained control strategy model, control the robotic arm to perform an operation to approach the valve and align the end effector.

[0034] This method constructs a unified posture for valve grasping based on distance and 3D spatial vector projection verification. It then combines end-effector posture constraints and reward functions such as motion smoothness penalties to perform pure kinematic evaluation and training on the reinforcement learning policy network. This method can control the robotic arm to enter a standard and compliant pre-grabbing state before valve turning, effectively reducing the risks caused by posture divergence, physical interference, or disengagement during subsequent turning.

[0035] The following is a further detailed explanation of each of the above steps. The method in this embodiment includes the following steps: Step 1, as follows Figure 1 As shown in Figure S1, a physical simulation model of the robotic arm and valve is constructed, and the spatial coordinate system of the simulation environment is established.

[0036] A physical simulation model of the robotic arm and valve is constructed based on a high-fidelity URDF file. The model encapsulates complete geometric collision bodies, multibody dynamic parameters such as mass and inertia matrix, as well as joint physical constraints.

[0037] Based on the Isaac Lab simulation framework, N parallel and independent sets of physical sub-environments are constructed. For each parallel environment A spatial coordinate system is established: the world coordinate system is used as the globally unique static reference coordinate system, fixed at the origin of the simulation environment; the origin of the robot arm base coordinate system is aligned and coincident with the origin of the world coordinate system; the valve is placed within the three-dimensional effectively accessible workspace of the robot arm, and its spatial coordinates are represented by the position of the valve base coordinate system origin. .

[0038] Step 2: Based on the physical simulation model and spatial coordinate system, define the environment reset conditions, state space S, action space A, and reward function R for reinforcement learning training.

[0039] Step 2.1, as follows Figure 1 As shown in Figure S2, based on the physical simulation environment obtained in step 1, the reset conditions for the reinforcement learning training of the robotic arm are defined, including: initial conditions. and termination conditions .

[0040] For each parallel environment built in step 1 Strict boundary constraints are set at the beginning and end of each training round.

[0041] Initial conditions of the round Includes robotic arm initialization conditions Valve initialization conditions and noise initialization conditions The expression is as follows:

[0042]

[0043]

[0044] To address the deterministic pose reset constraint for the robotic arm: at the start of each round, reset the positions of the robotic arm's multi-degree-of-freedom joints to their initial poses. ,like Figure 2 As shown in the attached figure (A), the speed of the multi-degree-of-freedom joints of the robotic arm is reset to 0 to ensure that the robotic arm is initially stationary.

[0045] For the domain randomization constraint of the valve: at the start of the round, set the initial coordinates of the valve... Apply a three-dimensional random position offset based on a uniform distribution Obtain the fixed position coordinates of the target valve in this round. ,in These are the minimum and maximum values ​​of the offset, respectively, which restrict the valve position within the workspace of the robotic arm.

[0046] To simulate real-world noise during initialization: Considering the subsequent Sim-to-Real transfer of the model on the UR5 robotic arm, bounded random perturbations need to be applied to physical parameters such as the joint friction coefficient, end effector mass parameters, and visual sensor noise during training. This ensures the policy network encounters diverse dynamic conditions during simulation training, narrowing the gap between simulation and real-world perception and dynamic distribution. This initialization includes three aspects, with specific perturbation formulas as follows:

[0047] in, and These are the nominal value vector and the simulated disturbance value vector of the friction coefficient of the multi-joints of the UR5 robotic arm, respectively. This is the Hadamard product (i.e., matrix multiplication). and These represent the nominal value and the simulated disturbance value of the end effector mass, respectively. and These are visual sensor observation data under ideal conditions and actual simulated observation data after noise injection, respectively. , To represent the relative perturbation ratio of the corresponding physical parameters, samples were taken from the interval [missing information]. uniform distribution ,in , The preset maximum disturbance boundary (e.g., setting) Indicates quality is acceptable (floatation), ensuring the boundedness of physical parameter disturbances; To simulate visual sensor noise caused by real-world camera shake and lighting distortion, samples were taken from the point where the image was truncated to... Zero-mean Gaussian distribution within the range This ensures that extreme noise will not cause the simulation physics engine to crash.

[0048] End of round condition The expression is as follows: in, The maximum physical interaction time step threshold means that the end condition of a round is only related to the physical interaction time: when the time step reaches or exceeds the preset time limit, the round ends and the environment is reset; otherwise, the system continues to explore.

[0049] Step 2.2, as follows Figure 1 As shown in Figure S2, the state space for the reinforcement learning training of the robotic arm is defined based on the physical simulation environment obtained in step 1. .

[0050] For each parallel environment built in step 1 From the state of the robotic arm itself Relative orientation of valve and clamp Previous time step action state The three aspects jointly construct the reinforcement learning observation state space. ,Right now .

[0051] The above-mentioned robotic arm's own state The expression is as follows:

[0052] in: , These represent the position and velocity of the six degrees of freedom joints of the robotic arm, respectively. It indicates the position of the end effector's driven joint, which can be directly read by querying the joint state tensor of the underlying physics engine in real time; , These represent the position coordinates and quaternion orientation of the end effector in the world coordinate system, respectively, which are obtained in real time from the forward kinematics of the robotic arm.

[0053] The expression for the relative orientation of the valve and the clamp is as follows:

[0054] The three-dimensional spatial position of the valve handle in the world coordinate system is obtained through coordinate transformation. The formula is as follows:

[0055] Specifically, the homogeneous coordinate representations of the local positions of the end effector and valve handle are directly read by querying the spatial state tensor of the underlying physics engine. That is, the relative position vector and the homogeneous transformation matrix of the valve base coordinate system relative to the world coordinate system. By subtracting the current gripper coordinates from the target handle coordinates using spatial vector difference operations, the relative position vector between the end gripper and the valve handle is calculated in real time. .

[0056] Previous time step action state It is the action instruction vector output by the policy network at the previous time step. This data is automatically cached by the Isaac Lab Environment Manager.

[0057] Step 2.3, as follows Figure 1 As shown in Figure S2, the motion space for reinforcement learning training of the robotic arm is defined based on the physical simulation environment obtained in step 1. .

[0058] For each parallel environment built in step 1 The six-dimensional spatial pose increment command of the robotic arm's end effector gripper is concatenated with the one-dimensional opening and closing control command to form a continuous seven-dimensional vector, i.e., the motion space. The expression is as follows:

[0059] in: This represents the six-degree-of-freedom pose increment command of the robotic arm's end effector in a spatial coordinate system, including the translation increment of the spatial position. and the rotational increment of spatial attitude . This indicates the opening and closing control command of the end effector of the robotic arm, representing the continuous opening and closing degree from fully open (0) to fully closed (1).

[0060] The strategy network outputs seven-dimensional action space data based on differential inverse kinematics. At each time step... Next, the six-dimensional spatial pose increment The target setpoint is input to the differential inverse kinematics controller within the simulation environment. Utilizing the Jacobian matrix and its pseudo-inverse of the robotic arm, it is frequently calculated into a target angle sequence for the underlying drive joints, thereby controlling the robotic arm's pose in real time. This allows for the real-time control of the robotic arm's posture. The one-dimensional continuous opening and closing commands are then processed... Linear mapping is applied to the angle range of the robotic arm's end effector gripper, enabling real-time control of the end effector gripper's opening and closing.

[0061] Step 2.4, as follows Figure 1 As shown in Figure S2, based on the physical simulation environment obtained in step 1, the reward function for the reinforcement learning training of the robotic arm is defined. .

[0062] For each parallel environment built in step 1 Design reward function The robotic arm is guided to autonomously approach the valve and align its posture. The reward function is designed from three aspects. ,include: 1) Proximity reward calculated based on the distance between the current position of the end effector and the position of a standard pre-grab target. It defines a clear and measurable standard pre-grabbing posture for valve operation of the robotic arm, providing a stable and reliable starting point for subsequent high-precision operations such as screwing, and effectively reducing the risk of operation failure due to poor initial posture.

[0063] 2) Attitude constraint reward based on the axial vector alignment between the local coordinate system of the end effector and the local coordinate system of the valve handle ; 3) Penalty term based on the smoothness calculation of robotic arm movements .

[0064] Its expression is as follows:

[0065] in, to These represent the weights of each reward function.

[0066] (1) Proximity to valve reward The expression used to urge the robotic arm's end effector to approach the valve handle is as follows:

[0067]

[0068]

[0069] like Figure 2 Under the standard pre-grabbing posture shown in the attached figure (B), This is the pre-grabbing pose offset vector mapped to a three-dimensional coordinate system, used to characterize the spatial relative position from the ideal grab point to the valve handle target point; This is the location of the end effector in the standard pre-grab posture, calculated using this offset vector; The distance threshold for triggering double rewards. For example, when the distance d is less than or equal to the threshold. When this happens, double the reward can be given to encourage accurate arrival.

[0070] (2) End-point constraint attitude reward This is used to characterize the relative posture of the robotic arm's end effector gripper and the valve handle. The reward function guides the end effector gripper to adjust and maintain an orthogonal grasping posture relative to the valve handle during approach by calculating the dot product (i.e., cosine similarity) between unit vectors on the local coordinate axes. The final effect is as follows: Figure 2 The attached diagram (B) shows the geometric principle of this reward item. Figure 3 As shown, where , These are the unit vectors of the x-axis, y-axis, and z-axis in the local coordinate system of the robotic arm's end effector gripper. , These are the unit vectors of the x-axis, y-axis, and z-axis in the local coordinate system of the valve handle, respectively.

[0071] Since all the vectors involved in the operation are unit vectors, the dot product is the cosine of the angle between the two vectors:

[0072]

[0073] The cosine value of the aforementioned angle directly measures the alignment of the corresponding local coordinate axes: the closer the cosine value is to 1, the more parallel and aligned the coordinate axes are in the same direction. By maximizing the cosine similarity in these two orthogonal directions, the three-dimensional rotational degrees of freedom of the end effector can be effectively constrained, ensuring that it maintains the expected perpendicular orientation relative to the valve plane.

[0074] To enhance the gradient guidance signal during the pose alignment process, the cosine value is further squared, and simultaneously... The function is signed, and the original directional features of the vector angle are preserved. The final expression for the end-effector posture constraint reward is as follows:

[0075] The standard pre-grab posture requires that the z-axis of the end effector be parallel and in the same direction as the y-axis of the valve handle, and that the y-axis of the end effector be antiparallel to the x-axis of the valve handle; the reward is achieved by calculating the dot product of the unit vectors of the corresponding axes.

[0076] (3) Motion smoothness penalty is designed to suppress the high-frequency jitter and violent movement of the robotic arm in response to the differences in motion instructions between adjacent time steps, so that the generated motion trajectory is smoother and more stable, which meets the control requirements of the actual physical system.

[0077] Its reward function The expression is as follows:

[0078] in, and These are the action instruction vectors output by the policy network at the current time step and the previous time step, respectively. Let be the joint velocity vector of the robotic arm. and These are the penalty weights for the rate of change of motion and the penalty weights for joint speed, respectively.

[0079] Step 3, as follows Figure 1 As shown in Figure S3, iterative training is performed using a memoryless proximal strategy optimization algorithm.

[0080] Based on the parallel environment constructed in step 1 and the reinforcement learning parameters designed in step 2, the control policy is trained through multiple rounds of iterative training using a memoryless proximal policy optimization algorithm. The training process includes: 1) Simultaneously execute exploration and collect interaction trajectory data in multiple parallel simulation sub-environments; 2) Calculate the advantage function using the generalized advantage estimation algorithm; 3) Optimize and update the policy network parameters based on the pruning and substitution objective function.

[0081] The specific process is as follows: The overall network uses a multilayer perceptron as the backbone, and the parameters are constructed as follows: Policy network With parameters value network To ensure the model is lightweight, both the policy network and the value network adopt a "memoryless" architecture, meaning that the network relies only on the current observation state when making decisions. This design avoids introducing temporal memory structures such as recurrent neural networks (RNNs) or long short-term memory (LSTMs), and does not use historical state information as network input. The advantages of this design are: in a simulation environment that satisfies the Markov decision process assumption, the current observed state already contains all the information needed for decision-making; the memoryless strategy effectively avoids noise interference from historical information, thus significantly simplifying the network structure, reducing the number of training parameters and computational overhead, and enabling more efficient learning of the direct mapping relationship from state to action.

[0082] The two-layer policy network uses a multilayer perceptron as its backbone. Both layers have three fully connected layers with [256, 128, 128] neurons in their hidden layers, and employ exponential linear units. As an activation function.

[0083] In terms of hardware and computing resource configuration, this training is based on the Ubuntu 22.04 operating system, using an NVIDIA GeForce RTX 4090 GPU (24GB VRAM) as the computing acceleration hardware, equipped with an Intel Xeon processor and 64GB of system memory. For the optimization algorithm configuration, the Adam optimizer is used uniformly for gradient descent updates. The batch size is set to 128; the initial learning rates for the policy network and value network are... Reward Discount Factor Smoothing decay parameter for generalized dominance estimation Cutting tolerance overparameter .

[0084] The model is trained iteratively using a multi-round alternating cycle of "parallel exploration - advantage estimation - truncation update". A total of 5000 training iterations are planned, resulting in a total simulation interaction step count of [number missing]. step: (1) Environment Interaction and Advantage Estimation Stage: In each iteration of training, in each parallel environment Independently carry out exploration tasks based on initial conditions Introducing random perturbations, the agent interacts with the physical simulation environment in real time to generate interaction trajectory experience under the current environment. ,in These represent the observed state, the action performed, and the immediate reward at time t, respectively. Indicates the next moment, i.e. The observation state at any given time. At the data input end, the observation state space... Data of different dimensions are mapped to a standard distribution space with zero mean using an empirical normalization method, which accelerates the gradient convergence of network weights.

[0085] For all interaction trajectory experiences, a global experience summary is performed based on a concurrent sampling mechanism, and the action advantage function at each time step is calculated using the generalized advantage estimation method. Its expression is as follows:

[0086] in, Represents the total length of the data collection trajectory. Represents the current time step. The summation symbol represents the upper limit of the operation, which is to define the time window sequence for accumulating future rewards; the summation symbol indicates that from the current moment to the end of this trajectory (i.e., all remaining time steps in the future), the entire time series difference error is accumulated. For timing difference error, As a discount factor, This is a parameter for smooth attenuation.

[0087] (2) Network truncation update phase: After obtaining the global advantage evaluation within the current training cycle, the trajectory data is dynamically divided into multiple mini-batches for multi-round (usually set to 10 epochs) cyclic training, and the objective function is replaced by a pruning mechanism with truncation. Perform policy network parameters The gradient optimization is expressed as follows:

[0088] in, Based on experience and expectations, The ratio of new to old strategies. For the clipping function, This is the preset cutting tolerance hyperparameter.

[0089] Step 4, as follows Figure 1 As shown in Figure S4, the trained control strategy model is used to control the robotic arm to perform an operation of approaching the valve and aligning the end effector. Based on the training configuration and process in step 3, the model iterates through approximately 3000 rounds (corresponding to approximately...). When the simulation steps reach a certain number, the expected cumulative return of the strategy can quickly climb and reach a stable convergence state, completing the direct mapping from zero-based to a high-success-rate strategy. When the pruning replacement objective loss function of the strategy network tends to stabilize and the expected cumulative return of the environment reaches the convergence upper limit, the strategy network with globally optimized parameters is frozen and extracted, and the control strategy model of the robotic arm approaching the valve and aligning its posture is output.

[0090] To achieve Sim-to-Real model transfer, an online adaptive mechanism based on real-time kinematic feedback is adopted during the policy deployment phase. This involves feeding back data from the joint encoders and end-effector force / torque sensors of the real robotic arm to the policy inference pipeline in real time. A differential inverse kinematics controller is used to fine-tune the pose increment of the policy output online, compensating for unmodeled dynamic effects and geometric assembly errors in the real environment. To address potential sensor data loss or delays during real-world deployment, the policy model employs a fixed-time-step synchronous inference mechanism. Within each control cycle, it makes independent decisions based solely on the current observation frame, avoiding error accumulation caused by time-series dependencies. Finally, the control policy model is deployed on a real UR5 robotic arm, controlling it to perform the valve approach and end-effector pose alignment operation within the workspace. Based on the real-time sensed valve pose and robotic arm state, action commands are output in real time, driving the robotic arm to move efficiently, stably, and accurately to the standard pre-grasping pose.

[0091] This embodiment introduces randomization of valve position during training and uses pure kinematic features and geometric relationships as the basis for state and reward. This allows the trained policy to be independent of the appearance or absolute coordinates of a specific valve model, enabling it to generalize to valves in different positions within the workspace, and even to valves of different sizes / models, thus possessing strong practicality. Specifically, by decomposing the complex valve operation task and designing a reward function based on spatial geometric constraints, a robust policy for "pre-alignment" is trained using deep reinforcement learning. This significantly reduces the agent's ineffective exploration space, guiding it to efficiently learn compliant motion strategies and solving the slow convergence and policy divergence problems commonly found in end-to-end training. This method provides crucial posture assurance for subsequent fine operations such as grasping and turning, improving the automation level, success rate, and generalization ability of the entire valve operation task.

[0092] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0093] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for aligning the posture of a robotic arm approaching a valve with its end effector based on deep reinforcement learning, characterized in that, Includes the following steps: Construct a physical simulation model that includes a robotic arm and valves, and establish the spatial coordinate system of the simulation environment; Based on the physical simulation model and spatial coordinate system, the environment reset conditions, state space S, action space A, and reward function R for reinforcement learning training are defined. Based on the environment reset conditions, state space S, action space A and reward function R of the reinforcement learning training, the pre-constructed policy network is trained using the proximal policy optimization algorithm to obtain a control policy model that controls the robotic arm to approach the valve and complete the end-effector posture alignment. Using the trained control strategy model, the robotic arm is controlled to perform an operation to approach the valve and align its end effector.

2. The method according to claim 1, characterized in that, The defined environment reset conditions for reinforcement learning training include: At the beginning of each training round, the joint positions of the robotic arm are reset to a fixed initial reference pose. The joint velocity is reset to zero; At the start of each training round, a random offset is applied to the initial position of the valve in the world coordinate system. ,in It follows a uniform distribution within a preset range; End of round condition When the time step reaches or exceeds the preset time limit, the timeout ends and the environment is reset; otherwise, the system continues to explore. Noise initialization conditions: These are used to apply bounded random perturbations to relevant physical parameters during training, so that the policy network is exposed to diverse dynamic conditions during simulation training, thereby narrowing the gap between the perception and dynamic distribution in simulation and reality.

3. The method according to claim 1, characterized in that, The state space S is expressed as follows: In the formula, Indicates the state of the robotic arm itself. This indicates the relative orientation of the valve and the clamp. Indicates the action state at the previous time step; The state of the robotic arm itself The expression is as follows: in: , These represent the position and velocity of the six degrees of freedom joints of the robotic arm, respectively. It indicates the position of the end effector's driven joint, which can be directly read by querying the joint state tensor of the underlying physics engine in real time; , These represent the position coordinates and quaternion orientation of the end effector in the world coordinate system, respectively, which are obtained in real time from the forward kinematics of the robotic arm. The expression for the relative orientation state of the valve and the clamp is as follows: The three-dimensional spatial position of the valve handle in the world coordinate system is obtained through coordinate transformation. The formula is as follows: in, This represents the relative position vector between the end gripper and the valve handle. This represents the homogeneous transformation matrix of the valve base coordinate system relative to the world coordinate system; through spatial vector difference operations, the relative position vector between the end gripper and the valve handle is calculated in real time by subtracting the current gripper coordinate components from the target handle coordinate components. ; The action state at the previous time step The action instruction vector output by the policy network at the previous time step. .

4. The method according to claim 1, characterized in that, The expression for the action space A is as follows: in: This represents the six-degree-of-freedom pose increment command of the robotic arm's end effector in a spatial coordinate system, including the translation increment of the spatial position. and the rotational increment of spatial attitude ; This indicates the opening and closing control command of the end effector of the robotic arm, representing the continuous opening and closing degree from fully open (0) to fully closed (1).

5. The method according to claim 1, characterized in that, The reward function R is used to guide the robotic arm end effector to approach and align with the standard pre-grasp pose of the valve handle, and it includes at least: A proximity reward is calculated based on the distance between the current position of the end effector and the position of a standard pre-grab target. An attitude constraint reward is calculated based on the axial vector alignment between the local coordinate system of the end effector and the local coordinate system of the valve handle. A penalty term calculated based on the smoothness of the robotic arm's movements.

6. The method according to claim 5, characterized in that, The standard pre-grabbing target location is determined in the following way: The position of the valve handle in the world coordinate system Based on the ideal grasping point, the three-dimensional pose offset vector from the grasping point to the valve handle target point is: The position of the end effector in the standard pre-grabbing posture is calculated by combining the offset vector. .

7. The method according to claim 5, characterized in that, The proximity reward item The calculation method is as follows: Calculate the current position of the end effector. Compared with the standard pre-grabbing posture position The Euclidean distance d between them; Rewards are calculated based on distance d, when d is less than or equal to a preset threshold. When d is greater than 1, the first reward calculation formula is used; when d is greater than 1. When the first reward calculation formula is used, the reward value provided by the first reward calculation formula is higher than the reward value of the second reward calculation formula at the same distance.

8. The method according to claim 5, characterized in that, The attitude constraint reward item Alignment is measured by calculating the dot product of the unit vectors between the preset axes of the local coordinate system of the end gripper and the preset axes of the local coordinate system of the valve handle, as expressed below: in, , Let represent the unit vectors of the y-axis and z-axis of the local coordinate system of the robotic arm's end effector gripper, respectively. , Let x and y represent the unit vectors of the local coordinate system of the valve handle, respectively. This is a sign function used to preserve the original directional characteristics of the angle between vectors.

9. The method according to claim 5, characterized in that, The penalty item This includes penalties for differences in motion vectors between adjacent time steps and / or penalties for robot joint velocities, expressed as follows: in, and These represent the action instruction vectors output by the policy network at the current time step and the previous time step, respectively. Let be the joint velocity vector of the robotic arm. and These represent the penalty weights for the rate of change of motion and the penalty weights for joint velocity, respectively.

10. The method according to claim 1, characterized in that, The training using the near-end policy optimization algorithm includes: The exploration is performed synchronously in multiple parallel simulation sub-environments to collect interaction trajectory data; The advantage function is calculated using the generalized advantage estimation algorithm; The policy network parameters are optimized and updated based on the pruning and substitution objective function.

Images