Ffc self-learning assembly method based on parallel reinforcement learning

Abstract

The application discloses a kind of FFC self-learning assembly methods based on parallel reinforcement learning, establishes including real physical assembly system and simulation system parallel learning system, and physical assembly system and simulation system are run in parallel, experience data in assembly process is gathered to experience pool of parallel learning system, simulation system is trained by experience data provided by physical assembly system in experience pool, and feedback guides real physical system to execute assembly task;Wherein, the physical information of physical assembly system is stored into parameter server for simulation system to train after being handled by Softmax classifier of parallel learning system.The self-learning assembly method of the application not only has the advantages of high assembly efficiency and learning efficiency, but also has the advantage of high assembly success rate.

Description

Technical Field

[0001] This invention relates to a self-learning assembly method for FFC based on parallel reinforcement learning. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, using robots to assemble parts has become a current trend. In the fields of mechanical manufacturing and 3C electronic product manufacturing, there are numerous assembly tasks. Currently, in actual production processes, manual trial-and-error methods are often used. However, this method suffers from drawbacks such as low production efficiency, susceptibility to fatigue errors, and poor quality consistency.

[0003] Flexible flat cables (FFCs) are mainly used in various electronic products for electrical connections between different components, especially moving and stationary parts, for signal transmission. Because FFCs are thin, typically only 0.3mm, they are easily deformed under external forces. Compared to the assembly of rigid components, this introduces more uncertainties and interference, such as the problem of asynchronous movement between the robotic arm and the FFC due to deformation. This makes conventional assembly methods in existing technologies unsuitable for FFC assembly. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the main objective of this invention is to provide a self-learning assembly method for FFC based on parallel reinforcement learning.

[0005] To achieve the above-mentioned main objectives, this invention provides a self-learning assembly method for FFC based on parallel reinforcement learning, wherein: a parallel learning system is established, comprising a real physical assembly system and a simulation system; the physical assembly system and the simulation system are run in parallel; experience data from the assembly process is collected into the experience pool of the parallel learning system; the simulation system is trained using the experience data provided by the physical assembly system in the experience pool, and provides feedback to guide the real physical system in executing assembly tasks; wherein the physical information of the physical assembly system is processed by the Softmax classifier of the parallel learning system and stored in a parameter server for the simulation system to train;

[0006] The physical assembly system includes a vibration isolation test bench and a robotic arm; the vibration isolation test bench is equipped with a connector fixture, and the connector is installed on the connector fixture; the end of the robotic arm is equipped with a gripper and a force sensor, the gripper is used to hold the FFC, and the force sensor is used to detect the first contact force along the assembly direction that the FFC bears during the assembly process to the connector;

[0007] The assembly method steps are as follows:

[0008] S1) Initialize the parameters in the parallel learning system;

[0009] S2) Start the Softmax classifier and physical assembly system;

[0010] S3) The gripper at the end of the robotic arm picks up the FFC1 to be assembled and brings it close to the connector to start assembly training. At the same time, the state information of the physical assembly system is processed by the Softmax classifier and stored in the parameter server.

[0011] S4) After the physical assembly system completes its first successful assembly, the simulation system is started, and the Q-table in the experience pool is automatically updated. The physical assembly system continues the next round of assembly training with reference to the Q-value in the experience pool, where:

[0012] If the number of learning steps exceeds the set maximum number of learning steps or the first contact force between FFC1 and the connector exceeds the set safety force threshold, the assembly fails. At this time, the robotic arm is controlled to return to the initial position and start the next round of learning. At the same time, the current state information of the physical assembly system is reprocessed by the Softmax classifier and stored in the parameter server to guide the simulation system to retrain and input the new Q value into the experience pool.

[0013] Assembly is successful when the robotic arm in the physical assembly system reaches the specified depth within the set maximum learning steps and the first contact force between FFC1 and the connector does not exceed the set safety threshold.

[0014] S5) After reaching the set number of rounds, the assembly training ends.

[0015] Furthermore, the assembly method also includes a feasibility verification step:

[0016] S6) The FFC2 to be assembled is picked up by the gripper at the end of the robotic arm and the assembly is performed in two or more rounds with reference to the information in the experience pool.

[0017] Furthermore, the learning process of a parallel learning system is a Markov decision process, including the agent, the environment, and the action space. state space And a reward and punishment strategy; at each time t, the agent selects an action from the action space. Interact with the environment and receive rewards. t ;

[0018] The state space is defined as follows:

[0019]

[0020] Among them, f x f x f z and τ z Let x, y, z and θ represent the force and torque along the x, y, and z axes, respectively.z These represent the displacement and angle of the robotic arm along the x, y, and z axes, respectively.

[0021] Action space is defined as:

[0022]

[0023] Where Δx, Δy, and Δθ represent the step size of movement and rotation along the x, y, and z axes, respectively;

[0024] At the start, the agent selects actions from the action space; learning efficiency is improved by maximizing cumulative rewards:

[0025]

[0026] Where γ is the decay rate, r is the current reward, and k represents the number of steps;

[0027] Reward and punishment strategies are used to evaluate the agent's state. The performance of action 'a' in the simulation system is evaluated using a Softmax classifier; the reward and penalty policy in the physical system is defined as follows:

[0028]

[0029] k max This represents the maximum number of steps in one learning round. If the agent's search steps exceed k... max When this happens, a reward of r = -0.5 will be received. "Danger occurs" indicates that f x f x f z and τ z If the set safe contact force threshold is exceeded during assembly, a reward of r = -1 will be obtained.

[0030] The Q-learning algorithm is used to select the most likely action, and the selection strategy is defined as follows:

[0031]

[0032] Where ξ is a random value between 0 and 1, ε∈(0,1) is the greedy coefficient; Q1(s,a) and Q2(s,a) are Q functions, which are updated randomly in a loop using the following formula:

[0033]

[0034] Here, ψ1 and ψ2 are the learning rates; in Q-learning, the Q-values ​​are recorded in the Q table; and The weighting coefficient is defined as follows:

[0035]

[0036] Wherein, the coefficient c is a constant.

[0037] Furthermore, the robotic arm in the physical assembly system is controlled by a force-position coupling controller and performs the following:

[0038] In step S1, the bit coupling controller is initialized;

[0039] In step S3, the robotic arm is controlled by a force-position coupling controller so that the second contact force between FFC1 and the connector reaches the desired contact force; wherein, the second contact force is a contact force perpendicular to the assembly direction.

[0040] Furthermore, the physical assembly system was assembled for the first time using a teaching-based method.

[0041] The present invention has the following beneficial effects:

[0042] The self-learning assembly method proposed in this invention has higher learning efficiency and anti-interference ability, which can effectively solve the uncertainty interference problem caused by deformation during the assembly of FFC flexible electronic components, and realize the effective and efficient assembly of FFC.

[0043] To more clearly illustrate the purpose, technical solution, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0044] Figure 1 This is a framework diagram of the parallel learning system according to an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the physical assembly system according to an embodiment of the present invention;

[0046] Figure 3 This is a comparison chart of the assembly efficiency of the self-learning assembly method of this invention embodiment with that of traditional Q-learning and deep-learning;

[0047] Figure 4 This is a comparison chart of the learning efficiency of the self-learning assembly method of this invention with traditional Q-learning and deep-learning. Detailed Implementation

[0048] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0049] The parallel learning system established by the FFC self-learning assembly method based on parallel reinforcement learning in the embodiment is as follows: Figure 1 As shown, it includes a real system, a simulation system, an experience pool, a parameter server, and a softmax classifier; among which, the real system is used to perform real assembly.

[0050] like Figure 2 As shown, the physical assembly system includes a vibration isolation test bench 1 and a robotic arm 2; the vibration isolation test bench 1 is equipped with a connector fixture 3, and the connector 4 is mounted on the connector fixture 3; the robotic arm 2 is equipped with a force-position coupling controller, and the end of the robotic arm 2 is equipped with a gripper 5 and a force sensor 6. The gripper 5 is, for example, a pneumatic gripper or an electric gripper, used to hold the FFC, and the force sensor 6 is used to detect the first contact force along the assembly direction that the FFC bears during the assembly process to the connector.

[0051] In this embodiment, by running the physical assembly system and the simulation system in parallel, the experience data of the assembly process is collected into the experience pool of the parallel learning system. The simulation system is trained using the experience data provided by the physical assembly system in the experience pool and provides feedback to guide the real physical system to perform assembly tasks. The physical information of the physical assembly system is processed by the Softmax classifier of the parallel learning system and stored in the parameter server for the simulation system to train.

[0052] In this embodiment, the entire parallel learning system includes 7 agents, while the simulation system has 6 agents. In order to eliminate the overestimation behavior in traditional Q-learning, each agent uses the weighted double Q-learning (WDQL) method for learning.

[0053] The learning process of the parallel learning system is a Markov decision process, which includes the agent, the environment, and the action space. state space And a reward and punishment strategy; at each time t, the agent selects an action from the action space. Interact with the environment and receive rewards. t ;

[0054] The state space is defined as follows:

[0055]

[0056] Among them, f x f x f z and τz Let x, y, z and θ represent the force and torque along the x, y, and z axes, respectively. z These represent the displacement and angle of the robotic arm along the x, y, and z axes, respectively.

[0057] Action space is defined as:

[0058]

[0059] Where Δx, Δy, and Δθ represent the step size of movement and rotation along the x, y, and z axes, respectively;

[0060] At the start, the agent selects actions from the action space; learning efficiency is improved by maximizing cumulative rewards:

[0061]

[0062] Where γ is the decay rate, r is the current reward, and k represents the number of steps;

[0063] Reward and punishment strategies are used to evaluate the agent's state. The performance of action 'a' in the simulation system is evaluated using a Softmax classifier; the reward and penalty policy in the physical system is defined as follows:

[0064]

[0065] k max This represents the maximum number of steps in one learning round. If the agent's search steps exceed k... max When this happens, a reward of r = -0.5 will be received. "Danger occurs" indicates that f x f x f z and τ z If the set safe contact force threshold is exceeded during assembly, a reward of r = -1 will be obtained.

[0066] The Q-learning algorithm is used to select the most likely action, and the selection strategy is defined as follows:

[0067]

[0068] Where ξ is a random value between 0 and 1, ε∈(0,1) is the greedy coefficient; Q1(s,a) and Q2(s,a) are Q functions, which are updated randomly in a loop using the following formula:

[0069]

[0070] Here, ψ1 and ψ2 are the learning rates; in Q-learning, the Q-values ​​are recorded in the Q table; and The weighting coefficient is defined as follows:

[0071]

[0072] Wherein, the coefficient c is a constant.

[0073] The specific steps of the assembly method in this embodiment are as follows:

[0074] S1) Initialize the parameters of the force-position coupling controller and the parallel learning system;

[0075] S2) Start the Softmax classifier and physical assembly system;

[0076] S3) The FFC1 to be assembled is gripped by the gripper at the end of the robotic arm and brought close to the connector. The robotic arm is controlled by the force-position coupling controller so that the second contact force between the FFC1 and the connector reaches the desired contact force. In this embodiment, the desired contact force is set to 5N. The second contact force is a contact force perpendicular to the assembly direction.

[0077] Assembly training begins, and the state information of the physical assembly system is processed by the Softmax classifier and stored in the parameter server.

[0078] S4) The physical assembly system is assembled for the first time using a teaching method to obtain the initial assembly position. Within the allowable error range, the initial assembly position can be randomly given according to the actual assembly position for the simulation system to refer to.

[0079] After the physical assembly system completes its first successful assembly, the simulation system is launched, and the Q-table in the experience pool is automatically updated. The physical assembly system then continues the next round of assembly training, referencing the Q-values ​​in the experience pool.

[0080] If the number of learning steps exceeds the set maximum number of learning steps (500 times) or the first contact force between FFC1 and the connector exceeds the set safety force threshold, the assembly fails. At this time, the robotic arm is controlled to return to the initial position and start the next round of learning. At the same time, the current state information of the physical assembly system is reprocessed by the Softmax classifier and stored in the parameter server to guide the simulation system to retrain and input the new Q value into the experience pool.

[0081] Assembly is successful when the robotic arm in the physical assembly system reaches the specified depth within the set maximum learning steps and the first contact force between FFC1 and the connector does not exceed the set safety threshold.

[0082] S5) After reaching the set number of rounds (48), the assembly training ends;

[0083] S6) Based on the assembly training results, the FFC2 to be assembled is picked up again by the gripper at the end of the robotic arm and the assembly is performed for two rounds with reference to the information in the experience pool. The feasibility of the parallel learning system is verified by the number of assembly steps.

[0084] like Figure 3-4 As shown, compared to traditional Q-learning and deep Q-learning, the self-learning assembly method proposed in this embodiment achieves a success rate of 92% during 50 rounds of learning and assembly, which is superior to Q-learning's 74% and deep Q-learning's 58%. Furthermore, assembly failures only occur in the initial learning phase. Figure 3 It can be seen that the method proposed in this embodiment reached convergence after 8 rounds, while the number of learning steps did not converge when using Q-learning and deep Q-learning. Therefore, the assembly efficiency of this embodiment is significantly higher, the learning efficiency is effectively improved, and the assembly success rate is significantly improved.

[0085] While the present invention has been disclosed above with reference to specific embodiments, these embodiments are not intended to limit the scope of the invention. Any person skilled in the art can make variations / modifications without departing from the scope of the invention; all equivalent variations / modifications made in accordance with the present invention should be covered by the protection scope of the present invention.

Claims

1. A self-learning assembly method for FFC based on parallel reinforcement learning, characterized in that: A parallel learning system is established, comprising a real physical assembly system and a simulation system. The physical assembly system and the simulation system run in parallel. The experience data from the assembly process is collected into the experience pool of the parallel learning system. The simulation system is trained using the experience data provided by the physical assembly system in the experience pool and provides feedback to guide the real physical system in executing assembly tasks. The physical information of the physical assembly system is processed by the Softmax classifier of the parallel learning system and stored in the parameter server for the simulation system to train. The physical assembly system includes a vibration isolation test bench and a robotic arm; the vibration isolation test bench is equipped with a connector fixture, and the connector is installed on the connector fixture; the end of the robotic arm is equipped with a gripper and a force sensor, the gripper is used to hold the FFC, and the force sensor is used to detect the first contact force along the assembly direction that the FFC bears during the assembly process to the connector; The assembly method steps are as follows: S1) Initialize the parameters in the parallel learning system; S2) Start the Softmax classifier and physical assembly system; S3) The gripper at the end of the robotic arm picks up the FFC1 to be assembled and brings it close to the connector to start assembly training. At the same time, the state information of the physical assembly system is processed by the Softmax classifier and stored in the parameter server. S4) After the physical assembly system completes its first successful assembly, the simulation system is started, and the Q-table in the experience pool is automatically updated. The physical assembly system continues the next round of assembly training with reference to the Q-value in the experience pool, where: If the number of learning steps exceeds the set maximum number of learning steps or the first contact force between FFC1 and the connector exceeds the set safety force threshold, the assembly fails. At this time, the robotic arm is controlled to return to the initial position and start the next round of learning. At the same time, the current state information of the physical assembly system is reprocessed by the Softmax classifier and stored in the parameter server to guide the simulation system to retrain and input the new Q value into the experience pool. Assembly is successful when the robotic arm in the physical assembly system reaches the specified depth within the set maximum learning steps and the first contact force between FFC1 and the connector does not exceed the set safety threshold. S5) After reaching the set number of rounds, the assembly training ends.

2. The FFC self-learning assembly method based on parallel reinforcement learning as described in claim 1, characterized in that: The assembly method also includes a feasibility verification step: S6) The FFC2 to be assembled is picked up by the gripper at the end of the robotic arm and the assembly is performed in two or more rounds with reference to the information in the experience pool.

3. The FFC self-learning assembly method based on parallel reinforcement learning as described in claim 1, characterized in that: The learning process of a parallel learning system is a Markov decision process, which includes the agent, the environment, and the action space. State space And reward and punishment policies; at each time t, the agent selects an action a∈ from the action space. Interact with the environment and receive rewards. t ; The state space is defined as: in, These represent the forces along the x, y, and z axes, respectively. The torque is represented along the z-axis, and x, y, and z represent the displacements of the robotic arm along the x, y, and z axes, respectively. Indicates the rotation angle about the z-axis; Action space is defined as: Where △x and △y represent the movement step size along the x and y axes, respectively, and △θ represents the rotation step size around the z axis; At the start, the agent selects actions from the action space; it improves learning efficiency by maximizing the accumulated reward. in, R is the decay rate, r is the current reward, and k represents the number of steps; k Indicates cumulative reward. r k This represents the reward at step k, where n is a quantifier. Reward and punishment strategies are used to evaluate the agent's state. The performance of action 'a' in the simulation system is evaluated using a Softmax classifier; the reward and penalty policy in the physical system is defined as follows: This represents the maximum number of steps in one learning round. If the agent's search steps exceed [a certain number], [the maximum number of steps is specified]. When this happens, you will receive a reward of r=-0.5, and "Danger occurs" indicates that... and If the set safe contact force threshold is exceeded during assembly, a reward of r=-1 will be obtained. The Q-learning algorithm is used to select the most likely action, and the selection strategy is defined as follows: in, A random value between 0 and 1 This is the greed coefficient; and This is the Q function.

4. The FFC self-learning assembly method based on parallel reinforcement learning as described in claim 1, characterized in that: The robotic arm in the physical assembly system is controlled by a force-position coupling controller and performs the following: In step S1, the bit coupling controller is initialized; In step S3, the robotic arm is controlled by a force-position coupling controller so that the second contact force between FFC1 and the connector reaches the desired contact force; wherein, the second contact force is a contact force perpendicular to the assembly direction.

5. The FFC self-learning assembly method based on parallel reinforcement learning as described in claim 1, characterized in that: The physical assembly system was assembled for the first time using a teaching-based method.

Images