Virtual assembly path planning guiding method based on policy gradient optimization algorithm

Abstract

The application discloses a virtual assembly path planning guide method based on a policy gradient optimization algorithm, and steps include: step 1, a part 3D model is established through 3ds MAX, the part is converted into an FBX file, and the FBX file is imported into a Unity 3D reinforcement learning training scene; step 2, an ML_agents module is used to create an experimental environment in the Unity 3D reinforcement learning training scene; step 3, a part agent decision model is established; step 4, the part agent decision model is optimized; and step 5, the optimized part agent decision model is used for path planning of the part. The method is applied to a virtual assembly path planning guide process based on deep reinforcement learning of the policy gradient optimization, greatly improves assembly efficiency and assembly quality, and provides a new idea for solving virtual assembly path planning.

Description

Technical Field

[0001] This invention belongs to the field of virtual assembly technology and relates to a virtual assembly path planning and guidance method based on a strategy gradient optimization algorithm. Background Technology

[0002] Virtual assembly is an important tool for modern industrial design, product verification, and user training. However, in immersive virtual assembly systems, the lack of realistic collision force perception and assembly positioning constraints, coupled with trainees' tendency to lose their sense of direction in a three-dimensional virtual environment, makes completing virtual assembly in application scenarios with complex parts and diverse assembly structures a heavy burden. This leads to cognitive fatigue among trainees during training, significantly reducing assembly efficiency and quality. Summary of the Invention

[0003] The purpose of this invention is to provide a virtual assembly path planning guidance method based on a strategy gradient optimization algorithm, which solves the problem that existing technologies are unable to achieve effective tactile feedback on the assembly of parts, leading to cognitive fatigue among trainees during training and increasing their cognitive burden.

[0004] The technical solution adopted in this invention is a virtual assembly path planning guidance method based on a strategy gradient optimization algorithm, implemented according to the following steps:

[0005] Step 1: Create a 3D model of the part in 3ds Max, convert the part into an FBX file, and import it into the Unity 3D reinforcement learning training scene;

[0006] Step 2: Use the ML_agents module to create the experimental environment in the Unity 3D reinforcement learning training scenario;

[0007] Step 3: Establish a decision-making model for the intelligent agent of the component;

[0008] Step 4: Optimize the decision-making model of the intelligent agent for the part;

[0009] Step 5: Use the optimized part agent decision model to perform path planning for the part.

[0010] The beneficial effect of this invention is that by applying deep reinforcement learning based on policy gradient optimization to the virtual assembly path planning guidance process, it effectively solves the cognitive burden caused by the lack of motion direction and complex assembly process in immersive virtual assembly systems. By using the policy gradient optimization algorithm, the parts to be assembled are transformed into intelligent parts agents with autonomous decision-making capabilities. The intelligent parts agents can make decisions on actions and perform path planning based on specific environmental states. This path is instructive for assembly, greatly improving assembly efficiency and quality. Compared with traditional intelligent algorithms, it is more efficient and convenient, providing a new approach to solving virtual assembly path planning. Attached Figure Description

[0011] Figure 1 This is a flowchart of the part assembly path planning method of the present invention;

[0012] Figure 2 This is a flowchart illustrating the workflow of the ML_agents module used in the method of this invention.

[0013] Figure 3 This is a visual diagram illustrating the assembly path planning training in the method of the present invention;

[0014] Figure 4 This is a schematic diagram of the path planning guidance process performed by the trained first-stage rotor component intelligent agent decision-making model in Embodiment 1 of the method of the present invention when it is generalized to other components.

[0015] Figure 5 This is a schematic diagram of the path planning guidance process performed by the trained three-stage rotor blade component intelligent agent decision-making model when generalizing to other components in Embodiment 2 of the method of the present invention.

[0016] Figure 6 This is a schematic diagram illustrating the path planning guidance process performed by the trained intelligent agent decision-making model of the transmission shaft component in Embodiment 3 of the present invention when it is generalized to other components. Detailed Implementation

[0017] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0018] This invention proposes a part assembly path guidance prompting method based on gradient policy optimization to solve the part assembly path guidance problem in traditional assembly teaching tasks. The principle is to use a policy gradient optimization algorithm in a reinforcement learning environment to train the part to make intelligent path planning decisions. The main idea is to treat the part to be assembled as a part intelligent agent and set action reward and punishment functions, action functions, and scene reset functions to train the part intelligent agent. Finally, the path planning decision model of the part intelligent agent converges. No matter where the part to be assembled is in the scene, the decision model can make the part intelligent agent approach the part to be assembled. In this process, an assembly guidance path is generated, which reduces the cognitive burden of compressor assembly training.

[0019] Reference Figure 1 The virtual assembly path planning and guidance method of the present invention is implemented according to the following steps:

[0020] Step 1: Create a 3D model of the part in 3ds Max, convert the part into an FBX file, and import it into the Unity 3D reinforcement learning training scene.

[0021] Step 2: Create the experimental environment in the Unity 3D reinforcement learning training scenario using the ML_agents module, such as... Figure 2 As shown, it is possible to create intelligent agents for parts in the experimental environment, train and optimize them, and initialize the training scenario.

[0022] Reference Figure 3 In the Unity 3D reinforcement learning training scenario, parts to be assembled are randomly generated. The part agent selects the action to be executed to approach the part to be assembled based on the probability distribution of the output action. Here, a greedy strategy is not adopted to select the action.

[0023] Reinforcement learning training scenarios involve the interaction process between the component agent and the environment, mainly including four elements: policy, reward, value function, and environment model, which are explained below:

[0024] 2.1) The strategy is the behavior of the component agent at a specific moment, that is, the action mapping taken by the component agent according to the current environmental state. It can take different actions and rewards according to the action transition function, and generate an action trajectory τ from the current state to the target state, expressed as:

[0025] τ={s1,a1,r1,s2,a2,r2,...,s T ,a T ,r T} (1)

[0026] Where T represents the current moment of the environment, and S1, S2, S3…S…T The environmental state sequence is a1, a2, a3…a T For the action sequence, r1, r2, ..., r T The sequence of rewards obtained for each action, and the state transition between adjacent states after action a, according to the state transition function;

[0027] 2.2) Rewards and punishments are the values ​​that the environment provides to the component agent. These rewards and punishments tell the component agent what it should and shouldn't do. During each training round, a complete action trajectory reward is generated, expressed as:

[0028]

[0029] R(τ) is the total action reward per round of training, t is the current moment in the reinforcement learning training scenario, and r t It is the reward value obtained from the action performed at the current moment;

[0030] 2.3) The value function refers to the sum of expected rewards obtained by a component agent starting from a certain state and running continuously. It is used to represent long-term trends. The expression of the value function is:

[0031]

[0032] Where θ are the neural network parameters. This is expressed as the sum of the probability P(τ|θ) of each trajectory occurring, multiplied by the total action reward R(τ) for each trajectory, and is approximately equal to the average of the total rewards for all N trajectories; τ n This represents the nth trajectory;

[0033] 2.4) The environment model refers to the external world in which the intelligent component interacts. The environment is the scene in which the intelligent component is located, including observable states, executable actions, and dynamic interactions between the intelligent component and the environment. Given a state and an action, the state and reward at the next moment caused by the action will be predicted.

[0034] Step 3: Establish a component intelligent agent decision-making model. The decision-making model takes environmental observation information as input using the CollectObservation() function and makes action decisions. It uses the AgentAction() function to execute the decided actions and the AddReward() function to reward or punish each action. When a training round ends, the AgentReset() function is called to initialize the scene. The specific process is as follows:

[0035] 3.1) The part agent decision model employs a stochastic strategy, requiring the calculation of the conditional probability of each action in a given state. In the training environment, observation data, such as the part agent's local coordinate system position and velocity information, is input via the `CollectObservation()` function. The part agent decision model calculates the predicted reward values ​​for various actions and then selects the appropriate action based on the probability distribution. The expression is:

[0036] a~π(s,a,θ)=p[a|s,θ] (4)

[0037] Here, policy π determines the choice of action a, and the choice of action a determines the transition probability of state s. The decisive method here is the mapping from state to action, and the AgentAction() function determines the current action of the part agent. The policy gradient optimization algorithm is used to adjust these parameters and find the optimal policy.

[0038] 3.2) Since the reward for the action is unknown at this point, we are still in the exploration phase and cannot determine the quality of the action. We need to make a judgment based on the reward function AddReward().

[0039] A) If the reward value of the action is positive or the subsequent state is of high value, then the action is considered a good action, and the part's intelligence will get closer and closer to the part to be assembled.

[0040] B) If the reward value of the action is negative, or the subsequent state caused by the action is of low value, which will cause the intelligent part to move further and further away from the part to be assembled, or to go beyond the scene range, then this action is regarded as a bad action. At this time, the training scene is initialized and the path planning training is carried out again.

[0041] 3.3) The scene is initialized using the AgentReset() function, and subsequent action selection is performed.

[0042] The part agent decision model will try to avoid using poor actions that result in sparse rewards and instead select actions with high rewards. When the part agent finally encounters the part to be assembled based on some good actions, it means that the path planning is successful. Then, the scene is initialized, the part agent decision model updates the decision model parameters using the gradient ascent method, and stores the action sequence in the experience pool.

[0043] The component agent decision model reduces the expected probability of actions taken on the previously failed assembly path, thus lowering the probability of negative reward trajectories. If assembly is successful, to increase the probability of positive reward trajectories, it essentially samples a series of actions, calculates the state value of the sequence at time t, and uses this value to update the policy. The expression for updating the policy is:

[0044]

[0045] Where α is the learning rate, The change in gradient;

[0046] Finally, the gradient of the parameter function of the part intelligent agent decision model is calculated, and its expression is:

[0047]

[0048] Among them, E π Let the expected value of the gradient change of the objective function be . Let θ be the gradient change of the objective function. The optimal θ is obtained through gradient ascent, thus yielding the intelligent agent decision model for the part.

[0049] Step 4: Optimize the decision-making model of the intelligent agent for the part.

[0050] After training is completed, the system will generate a neural network model file, i.e., a decision model. After the part agent successfully plans its path, the scene initialization is triggered. At this point, it does not mean that the part agent's decision model is optimal. Instead, it needs to be continuously trained in the scene to find better and better actions to obtain higher reward rewards. When the total reward obtained from training no longer increases after multiple rounds of experiments, it means that the part agent's decision model has converged and reached the optimal state.

[0051] Step 5: Use the optimized part intelligent agent decision model to perform path planning for the part.

[0052] The optimized part agent decision model is generalized to the remaining parts. The current state of the remaining part agents is input into the part agent decision model, which outputs the probability of each action. The part agent decision model guides the part agents to take actions based on the probability of the actions. Consecutive actions will generate a path, which is the optimal assembly path. Then, line segments generated by Unity 3D's built-in Trail Renderer component are used to provide visual prompts for the assembly path guidance.

[0053] After multiple rounds of training and optimization, the part agent decision-making model needs to use a function to cover the continuous state space, and then adjust the parameters of the part agent decision-making model to control the direction of change of the objective function, thereby achieving the optimal strategy. The final gradient is:

[0054]

[0055] Equation (7) selects a behavior through observation information and directly propagates it backward. Based on the reward feedback, it directly enhances or weakens the probability of selecting the behavior. A good behavior increases the probability of being selected next time, while a bad behavior weakens the probability of being selected next time. Through continuous actions, an optimal assembly path will eventually be planned, thus completing the process.

[0056] Example 1

[0057] The implementation process of the method of this invention will be described using the assembly path planning process of the first-stage and second-stage compressor rotors in an aero-engine as an example. (Refer to...) Figure 1 By constructing an environment for training the first-level rotor component intelligent agent, the first-level rotor component intelligent agent is trained to obtain an action decision model. This first-level component intelligent agent decision model is then applied to the path planning process of first-level rotor component assembly within the environment. The specific steps are as follows:

[0058] Step 1: Create 3D models of the primary and secondary rotors of the part in 3ds Max, convert the part into an FBX file, and import it into the Unity 3D reinforcement learning training scene.

[0059] Step 2, refer to Figure 2 In Unity 3D reinforcement learning training scenarios, the ML_agents module is used; refer to... Figure 4 It enables the creation, training, and optimization of intelligent agents for machine parts within a laboratory environment. The usage of the ML_agents module is as follows:

[0060] a) Create a virtual environment and define the interaction rules between the first-level rotor component intelligent agent and the environment.

[0061] b) Define the neural network structure, policy function, value function, etc. of the intelligent agent of the first-stage rotor component.

[0062] c) During the interaction between the first-level rotor component intelligent agent and the environment, the ML-agents module automatically collects data such as the state, actions and rewards of the first-level rotor component intelligent agent and saves it in an experience replay pool.

[0063] d) Use Unity 3D to start the training process. The ML-agents module will automatically read the data in the experience replay pool and use policy gradient ascent to train the policy function and value function of the first-level rotor part agent.

[0064] e) After training is completed, the ML-agents module will automatically save the trained first-level rotor component intelligent agent decision model, which can be generalized to other first-level rotor components, so that the other components also have the ability to plan their own paths.

[0065] Step 3: Set up the environment observation function CollectionObservations(). Set a secondary rotor as the part to be assembled in the environment. The CollectObservations() function needs to observe nine pieces of information: the local coordinates of the secondary rotor part, the local coordinates of the primary rotor part, the X-axis velocity component of the primary rotor part, the Z-axis velocity component of the primary rotor part, and the Y-axis velocity component of the primary rotor part. The above observed information is input into the decision model of the primary rotor part for action decision-making.

[0066] 3.1) Set the action function AgentAction(). During training, AgentAction() will be continuously called by the training algorithm to update the actions of the first-level rotor component agent. The action space trained in this step is the velocity components of the first-level rotor component agent in the X, Y, and Z axes, which are three continuous values. Through continuous interaction, the action function AgentAction() can gradually drive the first-level rotor component agent to move closer to the second-level rotor, thereby realizing the autonomous learning and strategy optimization of the first-level rotor component agent.

[0067] 3.2) Set a reward / penalty function AddReward() to evaluate the quality of each action performed by the part agent during its movement. Good actions increase the likelihood of being selected next time; conversely, bad actions decrease the likelihood. The design of the reward / penalty function AddReward() is as follows:

[0068] When the distance between the first-stage rotor component and the second-stage rotor exceeds 4, AddReward(-0.01f) is executed, resulting in a negative reward of -0.01.

[0069] When the distance between the first-stage rotor component and the second-stage rotor is less than 3, AddReward(0.01f) is executed, and a positive reward of 0.01 is obtained;

[0070] When the distance between the primary rotor component and the secondary rotor is less than 2, AddReward(0.07f) is executed, and a positive reward of 0.07 is obtained;

[0071] When the distance between the first-level rotor component agent and the second-level rotor is less than or equal to 1.42, AddReward(1f) is executed to obtain a positive reward of 1, which is considered as the assembly path planning guidance has been successful. The current round of training is ended using the Done() method and the scene is initialized using the AgentReset() function.

[0072] 3.3) The first-level rotor component intelligent agent will move to a new position through the scene reset function AgentReset().

[0073] Additionally, if the first-level rotor component agent goes out of range, the scene reset function AgentReset() will be triggered, which will initialize the first-level rotor component agent, reset the initial velocity of the first-level rotor component agent to zero, reset the initial angular velocity of the first-level rotor component agent to zero, set the initial local coordinates of the first-level rotor component agent to (0, 0, 0), and the position of the second-level rotor will also be randomly refreshed within a 4×4×4 range.

[0074] Step 4: After training, the system will generate a first-level rotor component intelligent agent decision model, which includes a neural network model and parameter information. This model can process the information perceived by the component intelligent agent and generate corresponding actions. This first-level rotor component intelligent agent decision model can then be attached to other components, allowing them to make action decisions through the neural network model within the first-level rotor component intelligent agent decision model.

[0075] Step 5: After completing steps 1 to 4, apply the trained decision model to the remaining parts for path planning to verify its generalization ability. Other first-stage and second-stage rotor parts were introduced for assembly path planning. The trained decision model was added to these other first-stage rotor parts, making them intelligent agents with decision-making capabilities. The endpoint of the first-stage rotor parts' path planning was set as the second-stage rotor. The total time and number of successful path planning attempts for the first-stage rotor parts were statistically analyzed over 100 attempts. The average time for each path planning attempt was 1.21 seconds, with a success rate of 89%. This demonstrates the effectiveness of the decision model's generalization ability.

[0076] Example 2

[0077] Taking the assembly process of a three-stage blade and a three-stage rotor as an example, the implementation process of this invention is illustrated. By constructing a trainable environment and training the three-stage blade as a component intelligent agent, a decision-making model for the three-stage blade component intelligent agent's actions is obtained. This decision-making model is then applied to the path planning process of component assembly within the environment. The specific steps are as follows:

[0078] Step 1: Create a 3D model of the three-stage blades and three-stage rotor parts in 3ds Max, convert the parts into FBX files, and import them into the Unity 3D reinforcement learning training scene.

[0079] Step 2, refer to Figure 2 In Unity 3D reinforcement learning training scenarios, the ML_agents module is used. (See reference...) Figure 5 It can create three-level rotor component agents in an experimental environment and train and optimize them. The usage of the ML_agents module is as follows:

[0080] a) Create a virtual environment and define the interaction rules between the three-level blade component intelligent agent and the environment.

[0081] b) Define the neural network structure, policy function, and value function of the intelligent agent for the three-stage blade component.

[0082] c) During the interaction between the third-level blade component intelligent agent and the environment, the ML-agents module automatically collects data such as the state, actions and rewards of the third-level blade component intelligent agent and saves it in an experience replay pool.

[0083] d) Use Unity 3D to start the training process. The ML-agents module will automatically read the data from the experience replay pool and use policy gradient ascent to train the policy function and value function of the part agent.

[0084] e) After training is completed, the ML-agents module will automatically save the trained three-level blade component intelligent agent decision model, which can be generalized to other components, enabling the other components to also have the ability to plan their own paths.

[0085] Step 3: Set up the environment observation function CollectionObservations(). Set up an assembly part and a part intelligence agent in the environment. The CollectObservations() function needs to observe nine pieces of information: the coordinates of the three-stage rotor part, the local coordinates of the three-stage blade part intelligence agent, the X-axis velocity component of the three-stage blade part intelligence agent, the Z-axis velocity component of the three-stage blade part intelligence agent, and the Y-axis velocity component of the three-stage blade part intelligence agent. The above observed information is input into the three-stage blade part intelligence agent decision model for action decision.

[0086] 3.1) Set the action function AgentAction(). During training, AgentAction() will be continuously called by the training algorithm to update the actions of the third-stage blade component agent. The action space trained in this step is the velocity components of the third-stage blade component agent in the X, Y, and Z axes, which are three continuous values. Through continuous interaction, the action function AgentAction() can gradually drive the third-stage blade component agent to move closer to the third-stage rotor, thereby realizing the autonomous learning and strategy optimization of the third-stage blade component agent.

[0087] 3.2) Set a reward / penalty function AddReward() to evaluate the quality of each action of the third-stage blade component's intelligent agent during its movement. Good actions increase the likelihood of being selected next time; conversely, bad actions decrease the likelihood. The reward / penalty function AddReward() is designed as follows:

[0088] When the distance between the third-stage blade component agent and the third-stage rotor exceeds 4, AddReward(-0.01f) is executed, and a negative reward of -0.01 is obtained;

[0089] When the distance between the third-stage blade component agent and the third-stage rotor is less than 3, AddReward(0.01f) is executed, and a positive reward of 0.01 is obtained;

[0090] When the distance between the third-stage blade component agent and the third-stage rotor is less than 2, AddReward(0.07f) is executed, and a positive reward of 0.07 is obtained;

[0091] When the distance between the third-stage blade component agent and the third-stage rotor is less than or equal to 1.42, AddReward(1f) is executed to obtain a positive reward of 1, which is considered as the assembly path planning guidance has been successful. The current round of training is ended using the Done() method and the scene is initialized using the AgentReset() function.

[0092] 3.3) The intelligent agent of the third-stage blade part will move to a new position through the scene reset function AgentReset().

[0093] Additionally, if the third-stage blade component agent goes out of range, the scene reset function AgentReset() will be triggered, which will initialize the third-stage blade component agent, reset the initial velocity of the third-stage blade component agent to zero, reset the initial angular velocity of the third-stage blade component agent to zero, set the initial local coordinates of the third-stage blade component agent to (0, 0, 0), and refresh the position of the third-stage rotor randomly within a 4×4×4 range.

[0094] Step 4: After training, the system will generate a decision-making model for the third-level blade component, which includes a neural network model and parameter information. This model can process the information perceived by the third-level blade component's intelligent agent and generate corresponding actions. This decision-making model can then be attached to the other third-level blade components, allowing them to make action decisions through the neural network model within it.

[0095] Step 5: After completing steps 1 to 4, apply the trained Level 3 decision model to the remaining Level 3 blade components for path planning to verify the generalization ability of the decision model. The assembly path planning of Level 3 rotor blades and Level 3 rotor components is introduced. The trained decision model is added to the other Level 3 rotor blades, making them intelligent components with decision-making capabilities. The endpoint of the Level 3 rotor blade path planning is set as the Level 3 rotor itself. The total time and number of successful path planning attempts for the Level 3 rotor blade component intelligent components are statistically analyzed over 100 attempts. The average time for each path planning attempt is 1.35 seconds, and the success rate is 85%. This demonstrates the effectiveness of the decision model's generalization ability.

[0096] Example 3

[0097] Taking the assembly process of the driveshaft and front axle as an example, the implementation process of this invention is illustrated. By constructing a trainable environment for the driveshaft as a component intelligent agent and training it, a motion decision model for the driveshaft component intelligent agent is obtained. This motion decision model is then applied to the path planning process of driveshaft assembly. The specific steps are as follows:

[0098] Step 1: Create 3D models of the drive shaft and front axle in 3ds Max, convert the drive shaft and front axle into FBX files, and import them into the Unity 3D reinforcement learning training scene.

[0099] Step 2, refer to Figure 2 In Unity 3D reinforcement learning training scenarios, the ML_agents module is used. (See reference...) Figure 6 It can create intelligent agents for drive shaft components in an experimental environment and train and optimize them. The usage of the ML_agents module is as follows:

[0100] a) Create a virtual environment and define the interaction rules between the drive shaft component agent and the environment.

[0101] b) Define the neural network structure, policy function, value function, etc. of the intelligent agent of the transmission shaft parts.

[0102] c) During the interaction between the intelligent agent of the drive shaft component and the environment, the ML-agents module automatically collects data such as the state, actions and rewards of the intelligent agent of the drive shaft component and saves it in an experience replay pool.

[0103] d) Use Unity 3D to start the training process. The ML-agents module will automatically read the data from the experience replay pool and use policy gradient ascent to train the policy function and value function of the part agent.

[0104] e) After training is completed, the ML-agents module will automatically save the trained intelligent agent decision model of the drive shaft part, which can be generalized to other parts, so that the other parts also have the ability to plan their own paths.

[0105] Step 3: Set up the environment observation function CollectionObservations(). In the environment, set a front axle as the part to be assembled and a drive shaft intelligent agent. The CollectObservations() function needs to observe nine pieces of information: the local coordinates of the front axle part, the local coordinates of the drive shaft intelligent agent, the X-axis velocity component of the drive shaft intelligent agent, the Z-axis velocity component of the drive shaft intelligent agent, and the Y-axis velocity component of the drive shaft intelligent agent. The above observed information is input into the drive shaft intelligent agent decision model for action decision.

[0106] 3.1) Set the action function AgentAction(). During training, AgentAction() will be continuously called by the training algorithm to update the actions of the drive shaft component agent. The action space trained in this step is the component of the drive shaft component agent's velocity in the X, Y, and Z axes, which are three continuous values. Through continuous interaction, the action function AgentAction() can gradually drive the drive shaft component agent to move closer to the coordinate position of the forward axle component, thereby realizing the component agent's autonomous learning and strategy optimization.

[0107] 3.2) Set a reward / penalty function AddReward() to evaluate the quality of each action of the intelligent agent moving the transmission shaft component. Good actions increase the likelihood of being selected next time; conversely, bad actions decrease the likelihood of being selected next time. The reward / penalty function AddReward() is designed as follows:

[0108] When the distance between the drive shaft component and the front axle exceeds 4, AddReward(-0.01f) is executed, resulting in a negative reward of -0.01.

[0109] When the distance between the drive shaft component and the front axle is less than 3, AddReward(0.01f) is executed, and a positive reward of 0.01 is obtained;

[0110] When the distance between the drive shaft component and the front axle is less than 2, AddReward(0.07f) is executed, and a positive reward of 0.07 is obtained;

[0111] When the distance between the drive shaft component agent and the front axle is less than or equal to 1.42, AddReward(1f) is executed, and a positive reward of 1 is obtained. This is considered as the assembly path planning guidance being successful. The Done() method is then used to end the current round of training, and the AgentReset() function is used to initialize the scene.

[0112] 3.3) The intelligent agent of the drive shaft part will move to a new position through the scene reset function AgentReset().

[0113] Additionally, if the driveshaft component agent goes out of range, the scene reset function AgentReset() will be triggered, which will initialize the driveshaft component agent, reset the initial velocity of the driveshaft component agent to zero, reset the initial angular velocity of the driveshaft component agent to zero, set the initial local coordinates of the driveshaft component agent to (0, 0, 0), and refresh the position of the front axle component randomly within a 4×4×4 range.

[0114] Step 4: After training, the system will generate a decision-making model for the driveshaft component, which includes a neural network model and parameter information. This model can process the information perceived by the driveshaft component's intelligent agent and generate corresponding actions. By attaching this decision-making model to the component's intelligent agent, the agent can make action decisions through the neural network model within it.

[0115] Step 5: After completing steps 1 to 4, apply the trained decision-making model to the remaining driveshaft parts for path planning to verify the generalization ability of the driveshaft part intelligent agent decision-making model. The assembly path planning between the remaining driveshafts and the front axle was introduced. The trained driveshaft part intelligent agent decision-making model was added to the remaining driveshaft parts, making them intelligent agents with decision-making capabilities. The endpoint of the driveshaft part path planning was set as the front axle. The total time and number of successful path planning attempts for the remaining driveshaft part intelligent agents were statistically analyzed over 100 attempts. The average time for each path planning attempt was 2.01 seconds, and the success rate was 76%. This demonstrates the effectiveness of the decision-making model's generalization ability.

[0116] To illustrate the practical effectiveness of the method of this invention, a set of assembly control experiments was set up to verify the path guidance capability of the training decision model of this invention. An assembly environment was constructed in the scenario, consisting of a 6×6 square table with three sets of compressor parts placed on it. The purpose of the experiment was to allow students who were new to virtual assembly to assemble the parts, and to verify the effective guidance capability of the decision model trained by this invention through two guidance methods. Two control groups were set up, each with two students. The assembly process was guided by text prompts, while the assembly process was guided by intelligent part path planning. The completion time was used as the evaluation index, and the experimental results are shown in Table 1 below.

[0117] Table 1. Comparison of efficiency between text prompts and path guidance prompts

[0118] Text prompts 1 201 6 33.5 Text prompts and guidance 2 142 3 47.3 Path planning guidance 1 96 7 13.7 Path planning guidance 2 94 5 18.8

[0119] The results in Table 1 show that the first user, guided by text prompts, completed 6 assembly steps in 201 seconds, with an average time of 33.5 seconds. The second user, also guided by text, completed only 3 assembly steps in 142 seconds, with an average time of 47.3 seconds. In contrast, the two users guided by intelligent paths completed 7 and 5 assembly steps in 96 and 94 seconds respectively, with average times of only 13.7 and 18.8 seconds. This comparison demonstrates that the guidance capability of the assembly path planning using the strategy gradient optimization algorithm is far superior to the text guidance capability in traditional virtual assembly processes, illustrating the effectiveness of the method presented in this invention.

[0120] In summary, this invention belongs to the field of virtual assembly and relates to the process of planning assembly paths for compressor parts using the policy gradient optimization algorithm in deep reinforcement learning. The parts to be assembled are considered as intelligent agents, and a neural network is used to fit the action decision function. The environment in which the intelligent agent exists takes the current state as input to the neural network and then outputs the probability of taking the next action. The intelligent agent selects its next action based on the probability. When using the neural network to approximate the value function, the current state is also taken as input, and then the value of that state is output. In this way, the intelligent agent can evaluate the quality of the current state based on its value, thereby making better decisions. The network parameters are updated using a backpropagation algorithm, continuously optimizing the policy function and value function of the intelligent agent, thus guiding the intelligent agent to correctly execute actions and plan an assembly path. This process uses the policy gradient optimization algorithm to train the intelligent agent, enabling it to select and execute the correct actions to generate the assembly guidance path.

Claims

1. A virtual assembly path planning guidance method based on a policy gradient optimization algorithm, characterized in that, Follow these steps: Step 1: Create a 3D model of the part in 3ds Max, convert the part into an FBX file, and import it into the Unity 3D reinforcement learning training scene; Step 2: Use the ML_agents module to create the experimental environment in the Unity 3D reinforcement learning training scenario; Step 3: Establish the intelligent agent decision-making model for the parts. The specific process is as follows: The decision model takes environmental observation information as input using the CollectObservation() function and makes action decisions. It then uses the AgentAction() function to execute the decided actions, and the AddReward() function to reward or penalize each action. At the end of a training round, the AgentReset() function is called to initialize the scene. The process consists of the following steps: 3.1) The part-specific intelligent agent decision-making model adopts a stochastic strategy, requiring the calculation of the conditional probability of each action in a given state. In the training environment, observation data is input through the CollectObservation() function, with the expression: （4） Where, the policy determines the behavior of the selection, and the selection of the behavior determines the transition probability of the state , here the deterministic method is the mapping from state to action, and the current action of the part agent is determined through the AgentAction() function, and the policy gradient optimization algorithm is used to adjust these parameters and find the optimal policy; 3.2) Determine based on the reward / penalty function AddReward(): A) If the reward value of the action is positive or the subsequent state is of high value, then the action is considered a good action, and the part's intelligence will get closer and closer to the part to be assembled. B) If the reward value of the action is negative, or the subsequent state caused by the action is of low value, which will cause the intelligent part to move further and further away from the part to be assembled, or to go beyond the scene range, then this action is regarded as a bad action. At this time, the training scene is initialized and the path planning training is carried out again. 3.3) The scene is initialized using the AgentReset() function, and subsequent action selection is performed. The part agent decision model will try to avoid using poor actions that result in sparse rewards and instead select actions with high rewards. When the part agent finally encounters the part to be assembled based on some good actions, it means that the path planning is successful. Then, the scene is initialized, the part agent decision model updates the decision model parameters using the gradient ascent method, and stores the action sequence in the experience pool. The component agent decision-making model reduces the expected probability of actions taken on the previously failed assembly path, thus lowering the probability of negative reward trajectories. If assembly is successful, to increase the probability of positive reward trajectories, it essentially involves sampling a series of actions and then calculating the sequence... The state value at time step is used to update the policy. The expression for updating the policy is: （5） wherein, is a learning rate, is a change in gradient; Finally, the gradient of the parameter function of the part intelligent agent decision model is calculated, and its expression is: （6） Wherein, is the expected value of the gradient change of the target function, is the gradient change of the target function, and the optimal is obtained by gradient ascent, and the part intelligent agent decision model is obtained. Step 4: Optimize the decision-making model of the intelligent agent for the part; Step 5: Use the optimized part agent decision model to perform path planning for the part.

2. The virtual assembly path planning guidance method based on the policy gradient optimization algorithm according to claim 1, characterized in that, Step 2, the specific process is as follows: In the Unity 3D reinforcement learning training scenario, random parts to be assembled are generated. The part agent selects the action to be executed to approach the part to be assembled based on the probability distribution of the output action. Here, a greedy strategy is not adopted to select the action. The reinforcement learning training scenario involves the interaction process between the component agent and the environment, including four elements: policy, reward, value function, and environment model, which are explained below: 2.1) The strategy is the behavior of the component agent in the current environmental state. That is, the action mapping taken by the component agent according to the current environmental state, can take different actions and rewards according to the action transfer function, and generate an action trajectory from the current state to the target state. The expression is: （1） in, For the current moment of the environment, It is a sequence of environmental states. For action sequences, r 1, r 2,…, r T The sequence of rewards for each action, with adjacent states following the sequence of actions. After the action is performed, the state transition occurs according to the state transition function; 2.2) Rewards and punishments are the values ​​that the environment provides to the component agent. These rewards and punishments tell the component agent what it should and should not do. During each training round, a complete action trajectory reward is generated, expressed as: （2） It is the sum of the action rewards for each round of training. t This is the current moment in a reinforcement learning training scenario. It is the reward value obtained from the action performed at the current moment; 2.3) The value function refers to the sum of expected rewards obtained by a component agent starting from a certain state and running continuously. It is used to represent long-term trends. The expression of the value function is: （3） in, These are neural network parameters. The probability of each trajectory occurring is represented as... Multiply by the sum of the motion rewards for each trajectory. The sum of these is approximately equal to the total. The average of the total rewards for each trajectory; This represents the nth trajectory; 2.4) The environment model refers to the external world in which the intelligent component interacts. The environment is the scene in which the intelligent component is located, including observable states, executable actions, and dynamic interactions between the intelligent component and the environment. Given a state and an action, the state and reward at the next moment caused by the action will be predicted.

3. The virtual assembly path planning guidance method based on the policy gradient optimization algorithm according to claim 1, characterized in that, Step 4, the specific process is as follows: After training is completed, the system will generate a neural network model file, i.e., a decision model. After the part agent successfully plans its path, the scene initialization is triggered. At this point, it does not mean that the part agent's decision model is optimal. Instead, it needs to be continuously trained in the scene to find better and better actions to obtain higher reward rewards. When the total reward obtained from training no longer increases after multiple rounds of experiments, it means that the part agent's decision model has converged and reached the optimal state.

4. The virtual assembly path planning guidance method based on the policy gradient optimization algorithm according to claim 1, characterized in that, Step 5, the specific process is as follows: The optimized part agent decision model is generalized to the remaining parts. The current state of the remaining part agents is input into the part agent decision model, which outputs the probability of each action. The part agent decision model guides the part agents to take actions based on the probability of the actions. Consecutive actions will generate a path, which is the optimal assembly path. Then, line segments generated by Unity 3D's built-in Trail Renderer component are used to provide visual prompts for the assembly path guidance. After multiple rounds of training and optimization, the part agent decision-making model needs to use a function to cover the continuous state space, and then adjust the parameters of the part agent decision-making model to control the direction of change of the objective function, thereby achieving the optimal strategy. The final gradient is: （7） Equation (7) selects an action through observation information and directly propagates it backward. Based on the reward feedback, it directly enhances or weakens the probability of selecting the action. A good action will increase the probability of being selected next time, while a bad action will weaken the probability of being selected next time. Through continuous actions, an optimal assembly path will eventually be planned.

Images