On-orbit operation reinforcement learning training method for immersive augmented reality environment

Abstract

An on-orbit operation reinforcement learning training method of an immersive augmented reality environment, based on the immersive augmented reality environment, a virtual-real integrated coordinate transformation and imaging model is constructed, and the state representation of the real device and the virtual scene is unified; a polynomial reward function containing task completion degree, safety, smoothness and action constraint is designed; and proximal policy optimization and distributed alternating direction method of multipliers are used for efficient joint training of policy network and value network. After training, the strategy can be directly deployed in the same augmented reality environment for evaluation, and the average return, success rate, collision rate and end error are used as comprehensive evaluation indexes to realize the closed loop of training and evaluation.

Description

Technical Field

[0001] This invention relates to an on-orbit operation reinforcement learning training method for immersive augmented reality environments, belonging to the field of space on-orbit servicing and intelligent control technology. Background Technology

[0002] On-orbit servicing and space robot operation tasks include on-orbit assembly, extravehicular maintenance, target acquisition and docking, etc., which involve complex operating environments, large communication latency, and high real-time requirements. Existing training methods mainly include: pure software simulation platforms based on traditional graphics rendering, ground experimental platforms based on physical equipment, and script-based task contingency plan evaluation. These methods have the following shortcomings: (1) It is difficult to simultaneously achieve the coupling between visual immersion and real control equipment, and there is a lack of unified mathematical representation between the operator's head and hand posture and the virtual on-orbit target; (2) Reinforcement learning strategies are mostly trained in offline simulation environments, and the state vectors The failure to fully incorporate real sensor feedback resulted in a significant performance degradation when the strategy was migrated to a real-world system. (3) The training process and deployment evaluation are often separate systems, lacking the ability to perform closed-loop verification in the same augmented reality environment, making it difficult to intuitively evaluate the security and interpretability of the strategy.

[0003] Therefore, there is an urgent need for a method that can unify virtual and real spaces in immersive augmented reality environments, construct state vectors and action vectors containing multimodal information, and efficiently train and evaluate on-orbit operation strategies through a reasonable reward function and distributed reinforcement learning algorithms. Summary of the Invention

[0004] The technical problem solved by this invention is: addressing the difficulties in ground-based experimental platforms for physical equipment and script-based mission plan evaluation in the current technology, a reinforcement learning training method for on-orbit operation in an immersive augmented reality environment is proposed.

[0005] The present invention solves the above-mentioned technical problem through the following technical solution: An on-orbit operational reinforcement learning training method for immersive augmented reality environments includes: Construct an augmented reality environment, perform augmented reality environment coordinate calibration based on on-orbit operation mission information, and obtain real space coordinates and virtual space coordinates; An imaging model of a virtual camera in an augmented reality environment is constructed based on virtual spatial coordinates to construct a state vector at a specified time, and an action vector is obtained based on the state vector. Construct action smoothing terms, action safety terms, and action constraint terms to design multi-objective reward functions; Define an optimization objective function to optimize the multi-objective reward function; Construct policy networks and value networks and perform distributed training; Based on the multi-objective reward function and the distributed trained policy network and value network, a shearing policy objective is constructed. The network loss of the value network is calculated using the shearing policy objective to update the value network parameters, and this loss is used as an evaluation metric to measure whether the policy training has converged, thus performing policy convergence evaluation. Once the strategy convergence evaluation is passed, the overall performance of the on-orbit operation task is judged to meet the standards. If it meets the standards, the strategy network parameters in the distributed environment are adjusted, and the augmented Lagrangian function is constructed using the adjusted strategy network parameters. After training the policy network parameters using the augmented Lagrangian function, scoring parameters are calculated, and a comprehensive score is obtained based on each scoring parameter to determine whether the on-orbit operation policy corresponding to the current policy network parameters is the optimal policy.

[0006] The augmented reality environment consists of multi-planar stereoscopic display units, used for on-orbit operation task information acquisition and operation control. Based on the on-orbit operation task information, the augmented reality environment coordinates are calibrated, and the real spatial coordinates of each display unit are determined through this calibration. ; Virtual space coordinates For real space coordinates The mapped coordinates, where: In the formula, Let be the rigid homogeneous transformation matrix from the real space coordinate system to the virtual space coordinate system. Let be the three-dimensional rotation matrix of this transformation, representing the attitude rotation relationship between the virtual and real coordinate systems. Represents a three-dimensional special orthogonal group. Let be the three-dimensional translation vector of this transformation, representing the spatial offset of the origin of the virtual and real coordinate systems. Represents a three-dimensional real vector space. for Zero-dimensional vector.

[0007] The imaging model of the virtual camera is as follows: In the formula, Let be the homogeneous pixel coordinate vector of the image plane. , The intrinsic parameter matrix of the virtual camera. Let be the extrinsic homogeneous transformation matrix from the real space coordinate system to the virtual camera coordinate system. , The virtual camera images are respectively in , Equivalent focal length along the axial direction , The virtual camera main point is located at , The pixel coordinates of the axis; Obtain two-dimensional pixel coordinates from the imaging model. ; Two-dimensional pixel coordinates This is used to achieve precise visual alignment between real physical devices and augmented reality virtual fields of view; by mapping three-dimensional spatial coordinates to two-dimensional views, the relative pose of the target is extracted according to the mapping relationship, and an environmental state vector containing sensor perception information is constructed, ensuring that the observation state received by the policy network is consistent with the real on-orbit operating environment.

[0008] The specified time is Construct a state vector at a specified time. Obtain the sensing part of the state vector The pose of the on-orbit target relative to the end effector of the robotic arm is determined based on the vector elements of the sensing components. ; Determine the one-dimensional scalar based on on-orbit operation mission information. According to one-dimensional scalar Update the state vector, and construct the action vector using the updated state vector. : In the formula, For the end translation increment, For end-effector attitude increment, This refers to the amount of control for the opening and closing of the gripper.

[0009] The motion smoothing term, motion safety term, and motion constraint term are respectively set as follows: The multi-objective reward function is: In the formula, For motion smoothing penalty items, To smooth out the penalty coefficient, and They are time points and time The output action vector, This is a penalty item for safe actions. As a safety penalty coefficient, The minimum permissible safe distance threshold, The actual safe distance at the current moment. For action constraint penalties, To constrain the penalty coefficient, The dimension of the action vector. For the first Time of the first The actual output action, For the first The maximum allowable amplitude of a dimensional action. The total reward for each moment, Rewards for completing the task. These are the reward weights for task completion, action safety, and action smoothness, respectively.

[0010] The optimization objective function is defined according to the discount factor, and the optimization objective function is as follows: In the formula, The discount factor is determined based on the planning cycle length of the on-orbit operation mission and the degree of importance attached to long-term reward returns.

[0011] The policy network is optimized using a Gaussian distribution parameterization strategy, and the policy network is as follows: In the formula, For parameters Under constraints, by state Output Action random strategy, and Let be the mean and covariance matrices of a Gaussian distribution, respectively, where: By updating the policy network parameters Make the objective function Maximize network parameters using update strategies Find the optimal on-orbit control strategy that yields the maximum long-term expected cumulative reward; During the distributed training of the policy network and value network, the parameters of the policy network are updated. The subsequent policy network is responsible for generating control actions, and the value network is responsible for evaluating the value of the current state; after training converges, the weight parameters of the optimal policy network are obtained. Furthermore, the system utilizes the weight parameters of the optimal policy network to generate a sequence of control commands for the on-orbit robotic arm in a real-world environment.

[0012] The value network is used to estimate the state value of a multi-objective reward function. Used to approximate the true value The expected state value obtained by fitting the neural network is: In the formula, For state The true expected long-term discounted return; Indicates the current control strategy Mathematical expectation under the trajectory; Discount factor; From From this moment on, the future will begin. Instant rewards based on environmental feedback; Value networks through parameters Extract state features and perform mapping calculations to improve the output results. The output value represents the expected cumulative reward that the robotic arm can obtain from the current state, and serves as the evaluation benchmark for the strategy action. The time difference error is: The generalized advantage is estimated as follows: The construction probability ratio is: In the formula, for Time difference error (TD error) at any given moment. and Value networks in Time and The value of a state predicted at any given moment; for Generalized advantage estimation (GAE) at time. The time step truncation length for the advantage estimation. For smoothing parameters; This represents the ratio of the action probability of the robotic arm's current on-orbit control strategy to the old strategy. For the current policy network (parameters are) In state Take action below The probability, For the old policy network (parameters are) Take action under the same conditions The probability of.

[0013] The cutting strategy objective for: Cutting strategy target The parameters for updating the policy network are constructed based on the action probability ratio of the current on-orbit control strategy of the robotic arm to the old strategy and the generalized dominance estimate. ; During the update process, by introducing The shearing function limits the magnitude of a single policy update, preventing policy collapse during model training.

[0014] The network loss of the value network is based on the value network. Sure: in, The loss function of the value network, For value network parameters, Indicates time step Seeking expectations, for The actual cumulative discounted return target value from time point; the calculated network loss. Used to update the value network using gradient descent, and to evaluate the long-term value of the current state; During distributed training, the network loss value The downward trend and stable state are used as the basis for judging the strategy convergence assessment; when the value network loss converges and the strategy assessment is passed, the subsequent on-orbit operation performance assessment is triggered.

[0015] The method for adjusting policy network parameters in a distributed environment is as follows: adjust the global policy parameters. Decomposed into node local parameters , The decomposition method satisfies the consistency constraint. ; The augmented Lagrangian function is constructed as follows: In the formula, For penalty parameters, As a globally consistent auxiliary variable, For Lagrange multipliers; This represents the total number of distributed computing nodes. Number the current node; These are global policy network parameters. For the first Local policy parameters for each node; The objective function is the local objective function of the node. Node local parameters The local update method is as follows: In the formula, For the number of iterations, For the updated local policy network parameters, For the Lagrange multipliers before the update, For the updated global parameters; The global aggregation method in augmented Lagrange functions is: The multiplier update method is as follows: In the formula, For the updated global parameters, For the Lagrange multipliers before and after the update; An augmented Lagrangian function is constructed using the adjusted policy network parameters, and then the local policy network parameters among the distributed computing nodes are used to construct the function. With global aggregation parameters The consistency error is used to perform global collaborative optimization of policy parameters that diverge among nodes in the later stages of parallel reinforcement learning training at each node; by introducing a Lagrange multiplier penalty term, the local policy network parameters of each dispersed node are forced to converge to a unified global optimal parameter.

[0016] The policy network parameters are trained using an augmented Lagrangian function, and the scoring parameters for all policies are calculated, including average reward, success rate, collision rate, and end-position error, where: Average return: Success rate: Collision rate: End position error: The method for calculating the comprehensive score using scoring parameters is as follows: In the formula, The weights are used to select the optimal on-orbit operation strategy.

[0017] The advantages of this invention compared to the prior art are: (1) The present invention provides an on-orbit operation reinforcement learning training method for immersive augmented reality environment, which can directly construct a unified state-action space in the augmented reality scene composed of multi-plane stereo display, depth camera and motion capture, design a reward function that takes into account task completion, safety and motion smoothness, and combine near-end policy optimization and distributed solution to achieve efficient training and deployment evaluation of on-orbit operation strategy, and provide a technical basis for on-orbit service tasks; (2) This invention constructs a virtual-real fusion coordinate transformation, a unified state vector and action vector expression in an augmented reality environment, defines a multinomial reward function, and uses reinforcement learning algorithms to optimize the policy network and value network. Through the unified construction of transformation matrix, camera model and state vector, it realizes the fusion of real devices and virtual on-orbit targets under the same mathematical framework, enabling reinforcement learning strategies to be directly trained and deployed in immersive augmented reality scenarios. (3) This invention unifies task error, safety distance, action smoothing and constraint violation into the same optimization objective through a polynomial reward function, and combines near-end strategy optimization and distributed solution to achieve efficient, safe and transferable training of on-orbit operation strategies, providing a visual and quantifiable training and evaluation method for on-orbit service tasks. Attached Figure Description

[0018] Figure 1 A flowchart illustrating the on-orbit operation reinforcement learning training method based on an immersive augmented reality environment provided by this invention; Figure 2 This is a schematic diagram of the virtual-real fusion coordinate transformation and projection imaging model provided by the present invention; Figure 3 This is a schematic diagram of the integrated structure for distributed reinforcement learning training, deployment, and evaluation provided by the present invention. Detailed Implementation

[0019] An on-orbit reinforcement learning training method for immersive augmented reality environments is proposed. Based on the immersive augmented reality environment, a coordinate transformation and imaging model that integrates virtual and real elements is constructed to unify the state representation of real devices and virtual scenes. A multinomial reward function incorporating task completion, safety, smoothness, and action constraints is designed. Proximal policy optimization and distributed alternating direction multiplier method are used for efficient joint training of the policy network and value network. After training, the policy can be directly deployed in the same augmented reality environment for evaluation. A comprehensive score is obtained through indicators such as average reward, success rate, collision rate, and terminal error, achieving a closed loop between training and evaluation.

[0020] The specific steps of the on-orbit reinforcement learning training method for immersive augmented reality environments are as follows: Construct an augmented reality environment, perform augmented reality environment coordinate calibration based on on-orbit operation mission information, and obtain real space coordinates and virtual space coordinates; An imaging model of a virtual camera in an augmented reality environment is constructed based on virtual spatial coordinates to construct a state vector at a specified time, and an action vector is obtained based on the state vector. Construct action smoothing terms, action safety terms, and action constraint terms to design multi-objective reward functions; Define an optimization objective function to optimize the multi-objective reward function; Construct policy networks and value networks, and perform distributed training; Based on the multi-objective reward function and the distributed trained policy network and value network, a shearing policy objective is constructed. The network loss of the value network is calculated using the shearing policy objective, which is used to update the value network parameters and serves as an evaluation metric for whether the policy training has converged. Once the strategy convergence evaluation is passed, the overall performance of the on-orbit operation task (including control smoothness and end-point control accuracy) is judged to meet the standards. If the standards are met, the strategy network parameters in the distributed environment are adjusted, and the augmented Lagrangian function is constructed using the adjusted strategy network parameters. After training the policy network parameters using the augmented Lagrangian function, scoring parameters are calculated, and a comprehensive score is obtained based on each scoring parameter to determine whether the on-orbit operation policy corresponding to the current policy network parameters is the optimal policy.

[0021] The augmented reality environment consists of multi-planar stereoscopic display units, used for on-orbit operation task information acquisition and operation control. Based on the on-orbit operation task information, the augmented reality environment coordinates are calibrated, and the true spatial coordinates of each display unit are determined through this calibration. ; Virtual space coordinates For real space coordinates The mapped coordinates, where: In the formula, Let be the rigid homogeneous transformation matrix from the real space coordinate system to the virtual space coordinate system. Let be the three-dimensional rotation matrix of this transformation, representing the attitude rotation relationship between the virtual and real coordinate systems. It represents a special three-dimensional orthogonal group (i.e., rotation matrix space). Let be the three-dimensional translation vector of this transformation, representing the spatial offset of the origin of the virtual and real coordinate systems. Represents a three-dimensional real vector space. for Zero-dimensional vector.

[0022] The imaging model of the virtual camera is as follows: In the formula, Let be the homogeneous pixel coordinate vector of the image plane. , The intrinsic parameter matrix of the virtual camera. Let be the extrinsic homogeneous transformation matrix from the real space coordinate system to the virtual camera coordinate system. , The virtual camera images are respectively in , Equivalent focal length along the axial direction , The virtual camera main point is located at , Pixel coordinates of the axis. Pixel coordinates are obtained based on the imaging model. The imaging model and the calculated two-dimensional pixel coordinates This is used to achieve precise visual alignment between real physical devices and augmented reality virtual fields of view. By mapping three-dimensional spatial coordinates to a two-dimensional view, it provides the necessary geometric reference for subsequently extracting the relative pose of the operational target and constructing an environmental state vector containing sensor-perceived information, thereby ensuring that the observation state received by the policy network is highly consistent with the real on-orbit operational environment.

[0023] The specified time is Construct a state vector at a specified time. Obtain the sensing part of the state vector The pose of the on-orbit target relative to the end effector of the robotic arm is determined based on the vector elements of the sensing components. ; Determine the one-dimensional scalar based on on-orbit operation mission information. According to one-dimensional scalar Update the state vector, and construct the action vector using the updated state vector. : In the formula, For the end translation increment, For end-effector attitude increment, This refers to the amount of control for the opening and closing of the gripper.

[0024] The motion smoothing, motion safety, and motion constraint settings are respectively set as follows: The multi-objective reward function is: In the formula, For motion smoothing penalty items, To smooth out the penalty coefficient, and They are time points and time The output action vector, This is a penalty item for safe actions. As a safety penalty coefficient, The minimum permissible safe distance threshold, The actual safe distance at the current moment. For action constraint penalties, To constrain the penalty coefficient, The dimension of the action vector. For the first Time of the first The actual output action, For the first The maximum allowable amplitude of a dimensional action. The total reward for each moment, Rewards for completing the task. The reward weights are respectively for task completion, action safety, and action smoothness. Based on the discount factor definition, the objective function for optimization is: In the formula, The discount factor is determined based on the planning cycle length of the on-orbit operation mission and the degree of importance attached to long-term reward returns.

[0025] The policy network is optimized using a Gaussian distribution parameterization strategy. The policy network is as follows: In the formula, For parameters Under constraints, by state Output Action random strategy, and These are the mean and covariance matrices of the Gaussian distribution, respectively.

[0026] By updating the policy network parameters Make the objective function Maximize the optimal on-orbit control policy to obtain the maximum long-term expected cumulative reward; update the policy network parameters during the distributed training of the policy network and value network. The subsequent policy network is responsible for interactively generating control actions, while the value network is responsible for evaluating the value of the current state (i.e., representing the long-term expected cumulative reward that the robotic arm can obtain from the current state) and using this state value as a benchmark to measure the quality of actions. After training converges, the weight parameters of the optimal policy network are obtained. Furthermore, the system utilizes the weight parameters of the optimal policy network to generate a sequence of control commands for the on-orbit robotic arm in a real-world environment.

[0027] Value networks are used to estimate the state value of multi-objective reward functions. It is used to approximate the value of the true state. The neural network fits the expected state value formula as follows: In the formula, For state The true expected long-term discounted return; Indicates the current control strategy Mathematical expectation under the trajectory; Discount factor; From From this moment on, the future will begin. Immediate rewards based on environmental feedback. The value network uses parameters... Extract state features and perform mapping calculations to improve the output results. The output value represents the expected cumulative reward that the robotic arm can obtain from the current state, and serves as a benchmark for evaluating strategic actions. The value output by this network represents the "goodness" or "badness" of the current state, providing a benchmark for evaluating the advantages of subsequent strategic actions. The time difference error is: The generalized advantage is estimated as follows: The construction probability ratio is: In the formula, for Time difference error (TD error) at any given moment. and Value networks in Time and The value of a state predicted at any given moment; for Generalized advantage estimation (GAE) at time. The time step truncation length for the advantage estimation. For smoothing parameters; This represents the ratio of the action probability of the robotic arm's current on-orbit control strategy to the old strategy. For the current policy network (parameters are) In state Take action below The probability, For the old policy network (parameters are) Take action under the same conditions The probability of.

[0028] Cutting strategy target for: Cutting strategy target Based on the action probability ratio of the current on-orbit control strategy of the robotic arm to the old strategy and the generalized advantage estimate, a method is constructed to update the parameters of the policy network during the reinforcement learning training phase using the gradient ascent method. Introduction The shearing function can limit the magnitude of a single policy update, preventing policy collapse during training and thus ensuring the stability of the entire training process. The network loss of the value network is determined based on the value network... Sure: in, The loss function of the value network, For value network parameters, Indicates time step Seeking expectations, for The actual cumulative discounted return target value from time point onwards. The calculated network loss. This is used to update the value network via gradient descent, enabling it to more accurately assess the long-term value of the current state. Simultaneously, during distributed training, the network's loss value... The downward trend and stable state are directly used as the basis for judging "policy convergence evaluation". Only when the value network loss converges and the policy evaluation passes can the subsequent on-orbit operational performance assessment be triggered. The method for adjusting policy network parameters in a distributed environment is as follows: adjust the global policy parameters. Decomposed into node local parameters , The decomposition method satisfies the consistency constraint. ; The augmented Lagrangian function is constructed as follows: In the formula, For penalty parameters, It serves as a globally consistent auxiliary variable (i.e., a global aggregation parameter). For Lagrange multipliers; This represents the total number of distributed computing nodes. Number the current node; These are global policy network parameters. For the first Local policy parameters for each node; The objective function is the local objective function of the node. Node local parameters The local update method is as follows: In the formula, For the number of iterations, For the updated local policy network parameters, For the Lagrange multipliers before the update, For the updated global parameters; The global aggregation method in augmented Lagrange functions is: The multiplier update method is as follows: In the formula, For the updated global parameters, These are the Lagrange multipliers before and after the update.

[0029] An augmented Lagrangian function is constructed using the adjusted policy network parameters, and then the local policy network parameters among the distributed computing nodes are used to construct the function. With global aggregation parameters The consistency error is used to perform global collaborative optimization of policy parameters that diverge among nodes in the later stages of multi-node parallel reinforcement learning training. By introducing a Lagrange multiplier penalty term, the local policy network parameters of each dispersed node are forced to converge to a unified global optimal parameter, thereby solving the problem of inconsistent policy updates in distributed reinforcement learning and ensuring that the final output on-orbit operation policy has global optimality and strong robustness.

[0030] The policy network parameters are trained using an augmented Lagrangian function, and the scoring parameters for all policies are calculated, including average reward, success rate, collision rate, and end-position error, where: Average return: Success rate: Collision rate: End position error: The method for calculating the comprehensive score using scoring parameters is as follows: In the formula, The weights are used to select the optimal on-orbit operation strategy.

[0031] The following description, in conjunction with the accompanying drawings and preferred embodiments, provides further details: In the current embodiment, such as Figure 1 As shown, the on-orbit operational reinforcement learning training based on an immersive augmented reality environment is completed. The specific methods and steps include: (1) Augmented reality operation space calibration and multi-plane display configuration.

[0032] In implementation, the indoor operating space was selected as the real coordinate system. Virtual scene workspace is The set of point pairs is obtained by measuring with a calibration plate. The transformation matrix is ​​solved using the least squares method. ,make Minimum; Depth camera coordinates with actual coordinates Between Connection, to satisfy Thus, virtual coordinates are obtained. Multiple display planes are represented by their respective plane equations. Define the pose through homogeneous transformation The virtual canvas coordinates are mapped to the corresponding plane, and the projected pixels are obtained through a pinhole model and normalized. The adjacent planar splicing region uses a weighting function Perform weighted fusion to form By minimizing brightness error Adjust the gain of each channel.

[0033] (2) On-orbit operation task scenario construction and status-action data acquisition The orbital position and attitude of the on-orbit target are configured in a virtual scene and rendered in an augmented reality environment; the operator wears stereoscopic glasses and holds a controller, and the head and hand poses are captured by a motion capture system. , The robotic arm controller provides feedback on the current joint angle. Joint angular velocity And by positive kinematics Once the end-effector pose is obtained, the relative pose is further calculated. Thus, the state vector is constructed. Action vectors Used to drive virtual and real robotic arms, the end effector pose is updated to... Solving by inverse kinematics And check whether the joint angle meets the requirements. Otherwise, the violation will be recorded. .

[0034] (3) Enhance learning training and strategy deployment assessment During the training phase, the system operates at a fixed time step. Run multiple parallel environments, each generating a trajectory. Merge multiple trajectories into a dataset. By batch size Extract small batches of data for strategy and value updates; by Calculate the time difference error and utilize... The advantage estimate is obtained and substituted into the shear target. The parameters are updated using a stochastic gradient ascent strategy. Value networks adopt loss With learning rate renew In a distributed environment, each computing node calculates its local gradient and then... The equation is updated locally, and then by Complete global aggregation.

[0035] After training converges, the final policy will be... Deployed back to augmented reality environment to perform each typical on-orbit mission. This trial calculates the average return. Success rate Collision rate Average end error And calculate the overall score. The strategy with the highest score is selected as the standard strategy for this type of on-orbit operation task.

[0036] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

[0037] The contents not described in detail in this specification are common knowledge to those skilled in the art.

Claims

1. An on-orbit operational reinforcement learning training method for an immersive augmented reality environment, characterized in that... include: Construct an augmented reality environment, perform augmented reality environment coordinate calibration based on on-orbit operation mission information, and obtain real space coordinates and virtual space coordinates; An imaging model of a virtual camera in an augmented reality environment is constructed based on virtual spatial coordinates to construct a state vector at a specified time, and an action vector is obtained based on the state vector. Construct action smoothing terms, action safety terms, and action constraint terms to design multi-objective reward functions; Define an optimization objective function to optimize the multi-objective reward function; Construct policy networks and value networks and perform distributed training; Based on the multi-objective reward function and the distributed trained policy network and value network, a shearing policy objective is constructed. The network loss of the value network is calculated using the shearing policy objective to update the value network parameters, and this loss is used as an evaluation metric to measure whether the policy training has converged, thus performing policy convergence evaluation. Once the strategy convergence evaluation is passed, the overall performance of the on-orbit operation task is judged to meet the standards. If it meets the standards, the strategy network parameters in the distributed environment are adjusted, and the augmented Lagrangian function is constructed using the adjusted strategy network parameters. After training the policy network parameters using the augmented Lagrangian function, scoring parameters are calculated, and a comprehensive score is obtained based on each scoring parameter to determine whether the on-orbit operation policy corresponding to the current policy network parameters is the optimal policy.

2. The on-orbit operational reinforcement learning training method for an immersive augmented reality environment according to claim 1, characterized in that: The augmented reality environment consists of multi-planar stereoscopic display units, used for on-orbit operation task information acquisition and operation control. Based on the on-orbit operation task information, the augmented reality environment coordinates are calibrated, and the real spatial coordinates of each display unit are determined through this calibration. ; Virtual space coordinates For real space coordinates The mapped coordinates, where: In the formula, Let be the rigid homogeneous transformation matrix from the real space coordinate system to the virtual space coordinate system. Let be the three-dimensional rotation matrix of this transformation, representing the attitude rotation relationship between the virtual and real coordinate systems. Represents a three-dimensional special orthogonal group. Let be the three-dimensional translation vector of this transformation, representing the spatial offset of the origin of the virtual and real coordinate systems. Represents a three-dimensional real vector space. for Zero-dimensional vector.

3. The on-orbit operational reinforcement learning training method for an immersive augmented reality environment according to claim 1, characterized in that: The imaging model of the virtual camera is as follows: In the formula, Let be the homogeneous pixel coordinate vector of the image plane. , The intrinsic parameter matrix of the virtual camera. Let be the extrinsic homogeneous transformation matrix from the real space coordinate system to the virtual camera coordinate system. , The virtual camera images are respectively in , Equivalent focal length along the axial direction , The virtual camera main point is located at , The pixel coordinates of the axis; Obtain two-dimensional pixel coordinates from the imaging model. ; Two-dimensional pixel coordinates Used to achieve precise visual alignment between real physical devices and augmented reality virtual viewpoints; By mapping three-dimensional spatial coordinates to a two-dimensional view, the relative pose of the target is extracted according to the mapping relationship, and an environmental state vector containing sensor perception information is constructed, ensuring that the observation state received by the policy network is consistent with the actual on-orbit operating environment.

4. The on-orbit operational reinforcement learning training method for an immersive augmented reality environment according to claim 1, characterized in that: The specified time is Construct a state vector at a specified time. Obtain the sensing part of the state vector The pose of the on-orbit target relative to the end effector of the robotic arm is determined based on the vector elements of the sensing components. ; Determine the one-dimensional scalar based on on-orbit operation mission information. According to one-dimensional scalar Update the state vector, and construct the action vector using the updated state vector. : In the formula, For the end translation increment, For end-effector attitude increment, This refers to the amount of control for the opening and closing of the gripper.

5. The on-orbit operational reinforcement learning training method for an immersive augmented reality environment according to claim 4, characterized in that: The motion smoothing term, motion safety term, and motion constraint term are respectively set as follows: The multi-objective reward function is: In the formula, For motion smoothing penalty items, To smooth out the penalty coefficient, and They are time points and time The output action vector, This is a penalty item for safe actions. As a safety penalty coefficient, The minimum permissible safe distance threshold, The actual safe distance at the current moment. For action constraint penalties, To constrain the penalty coefficient, The dimension of the action vector. For the first Time of the first The actual output action, For the first The maximum allowable amplitude of a dimensional action. The total reward for each moment, Rewards for completing the task. These are the reward weights for task completion, action safety, and action smoothness, respectively.

6. The on-orbit operational reinforcement learning training method for an immersive augmented reality environment according to claim 5, characterized in that: The optimization objective function is defined according to the discount factor, and the optimization objective function is as follows: In the formula, The discount factor is determined based on the planning cycle length of the on-orbit operation mission and the degree of importance attached to long-term reward returns.

7. The on-orbit operational reinforcement learning training method for an immersive augmented reality environment according to claim 6, characterized in that: The policy network is optimized using a Gaussian distribution parameterization strategy, and the policy network is as follows: In the formula, For parameters Under constraints, by state Output Action random strategy, and Let be the mean and covariance matrices of a Gaussian distribution, respectively, where: By updating the policy network parameters Make the objective function Maximize network parameters using update strategies Find the optimal on-orbit control strategy that yields the maximum long-term expected cumulative reward; During the distributed training of the policy network and value network, the parameters of the policy network are updated. The subsequent policy network is responsible for generating control actions, and the value network is responsible for evaluating the value of the current state; after training converges, the weight parameters of the optimal policy network are obtained. Furthermore, the system utilizes the weight parameters of the optimal policy network to generate a sequence of control commands for the on-orbit robotic arm in a real-world environment.

8. The on-orbit operational reinforcement learning training method for an immersive augmented reality environment according to claim 7, characterized in that: The value network is used to estimate the state value of a multi-objective reward function. Used to approximate the true value The expected state value obtained by fitting the neural network is: In the formula, For state The true expected long-term discounted return; Indicates the current control strategy Mathematical expectation under the trajectory; Discount factor; From From this moment on, the future will begin. Instant rewards based on environmental feedback; Value networks through parameters Extract state features and perform mapping calculations to obtain the output result. The output value represents the expected cumulative reward that the robotic arm can obtain from the current state, and serves as the evaluation benchmark for the strategy action. The time difference error is: The generalized advantage is estimated as follows: The construction probability ratio is: In the formula, for Time difference error at any given moment and Value networks in Time and The value of a state predicted at any given moment; for Generalized advantage estimation of time, The time step truncation length for the advantage estimation. For smoothing parameters; This represents the ratio of the action probability of the robotic arm's current on-orbit control strategy to the old strategy. For the current policy network parameters In state Take action below The probability, Network parameters for the old strategy Take action under the same conditions The probability of.

9. The on-orbit operational reinforcement learning training method for an immersive augmented reality environment according to claim 8, characterized in that: The cutting strategy objective for: Cutting strategy target The parameters for updating the policy network are constructed based on the action probability ratio of the current on-orbit control strategy of the robotic arm to the old strategy and the generalized dominance estimate. ; During the update process, by introducing The shearing function limits the magnitude of a single policy update, preventing policy collapse during model training.

10. The on-orbit operational reinforcement learning training method for an immersive augmented reality environment according to claim 9, characterized in that: The network loss of the value network is based on the value network. Sure: in, The loss function of the value network, For value network parameters, Indicates time step Seeking expectations, for The actual cumulative discounted return target value from time point; the calculated network loss. Used to update the value network using gradient descent, and to evaluate the long-term value of the current state; During distributed training, the network loss value The downward trend and stable state are used as the basis for judging the strategy convergence assessment; when the value network loss converges and the strategy assessment is passed, the subsequent on-orbit operation performance assessment is triggered.

11. The on-orbit operational reinforcement learning training method for an immersive augmented reality environment according to claim 10, characterized in that: The method for adjusting policy network parameters in a distributed environment is as follows: adjust the global policy parameters. Decomposed into node local parameters , The decomposition method satisfies the consistency constraint. ; The augmented Lagrangian function is constructed as follows: In the formula, For penalty parameters, As a globally consistent auxiliary variable, For Lagrange multipliers; This represents the total number of distributed computing nodes. Number the current node; These are global policy network parameters. For the first Local policy parameters for each node; The objective function is the local objective function of the node. Node local parameters The local update method is as follows: In the formula, For the number of iterations, For the updated local policy network parameters, For the Lagrange multipliers before the update, For the updated global parameters; The global aggregation method in augmented Lagrange functions is: The multiplier update method is as follows: In the formula, For the updated global parameters, For the Lagrange multipliers before and after the update; An augmented Lagrangian function is constructed using the adjusted policy network parameters, and then the local policy network parameters among the distributed computing nodes are used to construct the function. With global aggregation parameters The consistency error is used to perform global collaborative optimization of policy parameters that diverge among nodes in the later stages of parallel reinforcement learning training at each node. By introducing a Lagrange multiplier penalty term, the local policy network parameters of each dispersed node are forced to converge to a unified global optimal parameter.

12. The on-orbit operational reinforcement learning training method for an immersive augmented reality environment according to claim 11, characterized in that: The policy network parameters are trained using an augmented Lagrangian function, and the scoring parameters for all policies are calculated, including average reward, success rate, collision rate, and end-position error, where: Average return: Success rate: Collision rate: End position error: The method for calculating the comprehensive score using scoring parameters is as follows: In the formula, The weights are used to select the optimal on-orbit operation strategy.

Images