Industrial robot digital twin model self-updating method based on reinforcement learning algorithm

Abstract

An industrial robot digital twinning model self-updating method based on a reinforcement learning algorithm comprises the following steps that firstly, various real-time motion parameters of an industrial robot are collected through a sensor to serve as a training set of a digital twinning model, and an optimal control strategy is obtained after the iterative training process; different behaviors of the industrial robot are classified, and balance processing is carried out on data in the collection process; 2, constructing an industrial robot digital twinning model, and constructing a tetrad of a digital twinning system; and step 3, based on a parameter updating process of a DQ-SAC algorithm, adding a Q value on the basis of the SAC algorithm to form a double-Q network, and finally forming a DQ-SAC network. According to the method, fault characteristic parameters of the twin model are corrected by optimizing a control strategy, and self-updating of the model is achieved.

Description

Technical Field

[0001] The present invention relates to the fields of industrial robots and digital twin technologies, and in particular to a digital twin model self-updating method based on a reinforcement learning algorithm. Background Art

[0002] Professor Michael Grieves proposed the concept of digital twin (DT) in the early days. When this concept was first proposed, it was used in the military and aerospace fields. Since digital twins have the characteristics of virtual-real integration, real-time interaction, iterative calculation and optimization, and full-factor / full-process / full-business data drive, they have been applied to all stages of the product life cycle, including product design, manufacturing, operation and maintenance.

[0003] In recent years, it has become a trend to combine fault prognostics and health management (PHM) with DT technology. For example, Wenjun Fan et al. proposed a nonlinear degradation inflection point prediction method with relaxation voltage as the characteristic sequence, established a remaining service life prediction framework combining the inflection point degradation characteristics, and jointly predicted the inflection point and remaining service life, which stably improved the prediction accuracy of the remaining service life. Pietro Foti et al. introduced the main life prediction methods for fatigue damage assessment of additively manufactured components under multi-axial fatigue loads. QW Li proposed a PHM system based on advanced digital twin technology for the 500-meter Aperture Spherical Radio Telescope (FAST). The PHM system uses DT's finite element analysis to evaluate the safety status of the FAST system cable net structure and predict its fatigue life, achieving effective condition-based maintenance, ensuring the healthy and safe operation of the structure, while improving maintenance efficiency and reducing costs.

[0004] In the field of robotics, Wenna, Wang, et al. proposed a DT model for three-dimensional path planning of a large-span articulated arm gantry robot, which ensures the control safety of the actual gantry robot, improves production efficiency, and has satisfactory real-time performance. Malik, Ali Ahmad, et al. explored the opportunities of using DT technology to solve the complexity of human-robot collaborative production systems. Lu, Weisheng, et al. developed a human-robot team DT framework for facility management to achieve sustainability and health in the built environment. Case studies show that the framework can safely and effectively incorporate robots into facility management tasks (such as patrolling, inspection, and cleaning), allowing humans to plan, supervise, manage, and cooperate with robots through the two-way mechanism of DT.

[0005] For example, Chinese invention patent application No. 202410931883.1 discloses a method, device and system for compensating the pose accuracy of a digital twin model of an industrial robot. The method includes: constructing a digital twin model of an industrial robot, and establishing a mapping relationship between the joint angles of the actual industrial robot and the mechanical arm of its digital twin model under the same terminal pose conditions through neural network training, collecting the joint angles of the actual industrial robot in real time, and performing data prediction based on a neural network method to obtain the driving joint angles of the digital twin model under the same terminal pose conditions as the actual robot, and using this joint angle to drive the digital twin model, thereby reducing the error between the digital twin model and the actual robot's terminal pose, and achieving synchronization between the physical entity of the industrial robot and the terminal pose of the digital twin model, thereby improving the reconstruction capability of the digital twin model and the error control performance of the ICBC robot. Its technical defects are: 1. The scheme requires a lot of training of the neural network to accurately predict the specific value of each joint angle, and the overall work has very high requirements for the accuracy of the predicted value. This makes the overall workload huge and requires a lot of computing resources as support. 2. This solution only collects multiple terminal posture states of industrial robots and their actual joint angle data to facilitate the subsequent forward and inverse kinematic solutions. Our invention combines the four data of real-time angle, velocity, acceleration, and torque of each joint to reduce the overall calculation amount. 3. The ultimate goal of this solution is to make the terminal posture of the digital twin model more accurate by compensating the angle of each joint of the digital twin model. The main purpose of our invention is to determine the next decision of the intelligent agent π through the maximum entropy increase of reinforcement learning. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention proposes a self-updating method for an industrial robot digital twin model based on a reinforcement learning algorithm, which corrects the fault characteristic parameters of the twin model by optimizing the control strategy to achieve self-updating of the model.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A method for self-updating a digital twin model of an industrial robot based on a reinforcement learning algorithm, the method comprising the following steps:

[0009] The first step is to collect training sets and parameters. The process is as follows:

[0010] Step (1.1) collects various real-time motion parameters of the industrial robot through sensors as the training set of the digital twin model, and obtains the optimal control strategy after an iterative training process;

[0011] Step (1.2) classifies the different behaviors of industrial robots and balances the data collected;

[0012] The second step is model predictive control modeling. The process is as follows:

[0013] Step (2.1) constructing a digital twin model of an industrial robot;

[0014] Step (2.2) constructs the quaternion of the digital twin system;

[0015] The third step is to update the parameters of the DQ-SAC algorithm. On the basis of the SAC algorithm, a Q value is added to form a dual-Q network. The process is as follows:

[0016] Step (3.1) Actor is a strategy network, which takes the current state as input and outputs the probability distribution of actions or continuous action values. The Critic network then evaluates the quality of the action and adjusts the strategy accordingly.

[0017] Step (3.2) adds a Q value to the SAC network to form a double Q network, and finally forms a DQ-SAC network.

[0018] The step (1.2) includes the following sub-steps:

[0019] Step (1.2.1) classifies different behaviors of real industrial robots;

[0020] For the normal operation data collection of industrial robots, one sample is a cycle of the set action movement; for the fault data collection, one sample is the data of 4 seconds before and after the fault moment; different faults are classified;

[0021] Step (1.2.2) balances the data during the collection process;

[0022] Through undersampling, a portion of samples with a similar number to other faults are randomly selected from the most frequent faults, and the training data set is reconstructed to obtain the final experimental samples;

[0023] After obtaining the experimental samples, the fault data features of different categories are extracted and learned, and then the corresponding fault data are labeled, and the DQ-SAC algorithm is used to self-update the digital twin.

[0024] In the step (2.1), reinforcement learning is used to realize the real-time working condition feedback of the twin model to the real system. In the process of simulating the behavior of the real physical system, the industrial robot digital twin model (DTM) inputs u t , the state variable x t , state parameter θ t Under the condition of performance index J, the twin system state variable x is obtained according to the state equationt The best estimate of It also provides the real output of the system y t Best estimate with twin system Therefore, the model tracking problem is described as follows,

[0025]

[0026] θ i =π(x t ,y t ,y t+1 ,u t+1 );

[0027]

[0028] The reinforcement learning framework models the tracking problem through the model predictive control process (MPC), and applies the maximum entropy deep reinforcement learning algorithm to find the performance indicator J that best matches the display physical model to update the parameters of the twin model performance, ultimately making the system's real output y t+1 The best estimator x of the twin system t+1 They can match each other to achieve the purpose of model tracking.

[0029] In the step (2.2), the industrial robot model is described by a quaternion (S, A, J, M), including the following sub-steps:

[0030] In step (2.2.1), the state set S in the quaternary is composed of the input, output and state elements of the real physical system. For an industrial robot, the state set S is:

[0031] S={I i ,U i ,P i ,x t};

[0032] Where I represents the real-time current signal, U represents the real-time voltage signal, and P represents the real-time power. The industrial robot is a six-axis robot. The subscript i represents the information corresponding to different joints. t is the state variable, including position, velocity and temperature information;

[0033] In step (2.2.2), A in the quadruple is the action set, including the twin system needs to monitor the motion state of the industrial robot in real time,

[0034]

[0035] Where J(θ) is the Jacobian matrix, which contains the angular velocity and linear velocity information of each axis at each moment; R(t)p(t) are the rotation matrix and translation vector respectively; is the linear acceleration, is the angular acceleration, the force f on each link i , torque n i and joint torque τ i ;

[0036] In step (2.2.3), J in the quadruple is the cost function used to evaluate the tracking effect, expressed as:

[0037]

[0038] Where L(x t ,u t ) is the stage cost at time t, F(x N ) is the terminal cost at the end of the forecast horizon;

[0039] In step (2.2.4), M in the quaternary is the mathematical model of system behavior. For the state data monitoring of the industrial robot digital twin system, it is expressed as

[0040] x t+1 =f(x t ,u t ,θ t )

[0041] y t+1 =g(x t ,u t ,θ t )

[0042] where x t is the state variable, which is the state of the system at time t; u t is the system input, the control input of the system at time t, and f is a function that describes the dynamics of the system.

[0043] In the step (3.2), Actor is a strategy network, which takes the current state as input and outputs the probability distribution of the action or the continuous action value. The Critic network then evaluates the quality of the action and adjusts the strategy. The formula is as follows:

[0044]

[0045] H(π(·s t ))=E[-logπ(·s t )]

[0046] Where J(π) is the objective function of the DQ-SAC algorithm, V(s t) is the state value function, Q(s t ,a t ) is the value function, the minimum value of the Soft-Q value is taken as two Q value functions parameterized by θ1 and θ2, which helps to avoid overestimating inappropriate Q values, and H(·) represents the entropy value. T represents the total number of time steps of interaction between the agent and the environment, ρ π Indicates that under strategy π (s t ,a t ), π is the current strategy, α is a hyperparameter that controls the randomness of the optimal strategy and the importance of weighting entropy for rewards, γ∈[0,1) is the discount factor, Calculate the information entropy distribution of state transition under the current strategy.

[0047] In the step (3.3), the Q value is a function value used in reinforcement learning to measure the expected return that can be obtained after taking action a in a certain state s. The Q value function Q(s,a) gives the expected value of the accumulated reward that can be obtained after selecting action a in state s. In the dual Q network, the update target selects the smaller one of the two Q value networks to reduce the risk of overestimation.

[0048]

[0049] in and are the two Q values ​​of the dual Q network, y is the update target with the minimum Q value, J Q (θ i ) is the loss function of the Q-value network;

[0050] The dual Q network can solve the overestimation problem in the traditional SAC algorithm. Through two independent Q value networks, it avoids the quality and convergence performance of the strategy caused by the overestimation of Q value, thereby improving the performance and stability of the learning algorithm.

[0051] The technical concept of the present invention is: to propose a self-updating method for the digital twin model of an industrial robot based on a reinforcement learning algorithm. First, the position, velocity, acceleration and torque information collected by the sensors on each axis of the industrial robot are collected over time; then the Markov decision process is used to constrain the established twin model of the industrial robot; finally, reinforcement learning is introduced, and the twin calculates the state variables and their best estimated values ​​based on the various sensor data obtained as input, and seeks the optimal control strategy to select the next action; the above process completes the state parameter update and realizes the self-updating of the digital twin model. On this basis, the present invention adds a Q value (Quality Value) to the SAC (Soft Actor-Critic) algorithm, constructs a double-Q network (Double-Q), and designs a self-updating method for the digital twin model based on a reinforcement learning algorithm.

[0052] Using the digital twin model as the training environment, the intelligent agent is iteratively trained, and the Actor and Critic networks are synchronously updated at the same time, and finally the converged optimal control strategy is obtained. The currently obtained optimal control strategy is applied when the digital twin is deployed; according to the real-time input of the industrial robot to the digital twin system, the real-time state parameter θ of the twin system is adjusted. t The performance index J is modified so that the digital twin system can respond to various information of the actual industrial robot system in real time and accurately.

[0053] The beneficial effects of the present invention are mainly manifested in that: in the application scenario of industrial robots, the physical entity system continuously inputs various monitoring parameters under specific working conditions to the twin system, and the twin model updates its own state using the latest data received. At the same time, the relevant fault characteristic parameters are updated and adjusted by the reinforcement learning algorithm when the working conditions change or an emergency occurs. Through the cooperation between the real system and the twin system, the constructed digital twin model can effectively track the response of the real system, accurately reflect the current working status of the physical entity, and complete the data interaction of the physical entity with the digital twin system. After reducing the response deviation from the real system, the twin model can continue to provide the latest system state estimation, provide real-time, accurate and reliable system information for other subsequent tasks, and thus provide support for the decisions needed for subsequent tasks. Under dangerous working conditions, the whole process simulation is carried out by updating the preset initial state and target task, and the hidden problem points of the verification project are found in advance, and the workflow is optimized before the physical entity system starts, avoiding the existing hidden dangers, and completing the interactive feedback between the digital twin model and the physical entity. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of the self-updating method of the digital twin model of an industrial robot based on the reinforcement learning algorithm;

[0055] Figure 2 This is a twin model tracking training effect diagram;

[0056] Figure 3 This is the DQ-SAC algorithm training effect diagram;

[0057] Figure 4 This is a comparison chart between the position response under DQ-SAC algorithm control and the actual response of the system;

[0058] Figure 5 This is a comparison chart between the speed response under DQ-SAC algorithm control and the actual response of the system;

[0059] Figure 6 This is a comparison chart between the acceleration response under DQ-SAC algorithm control and the actual response of the system;

[0060] Figure 7 It is a comparison chart between the torque response under DQ-SAC algorithm control and the actual response of the system. DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with the accompanying drawings.

[0062] Reference Figure 1 to Figure 7 , a self-updating method for a digital twin model of an industrial robot based on a reinforcement learning algorithm, the method comprising the following steps:

[0063] The first step is to collect training sets and parameters. The process is as follows:

[0064] Step (1.1) collects various real-time motion parameters of the industrial robot through sensors as the training set of the digital twin model, and obtains the optimal control strategy after an iterative training process;

[0065] The data used in this embodiment comes from the normal operation data of industrial robots and abnormal data at the moment of failure provided by the R&D laboratory of the Robot Division of a Shanghai Electric Co., Ltd., China. For the signals such as the rotation angle of each axis used in the present invention, the control cabinet collects and transmits data at a sampling frequency of 250Hz. The present invention obtains all operation failure samples of a certain model of industrial robot with a maximum load of 220kg from July 1 to July 31, 2023.

[0066] Step (1.2) classifies the different behaviors of industrial robots and balances the data collected;

[0067] The step (1.2) includes the following sub-steps:

[0068] Step (1.2.1) classifies different behaviors of real industrial robots;

[0069] Since the time that an industrial robot is in a normal state during actual operation is often much longer than that in an abnormal state and the sampling robot is a laboratory verification model, the external environment is intervened in order to increase the amount of various fault data. In actual situations, the number of faults is much smaller than the number presented in sample collection. The present invention only obtains part of the normal operation data within this time period to increase the number of positive samples in the experiment and reduce unnecessary overhead in the data collection process. All the above samples together constitute the industrial robot fault data set.

[0070] The data used in the present invention comes from the normal operation data of industrial robots and abnormal data at the moment of failure provided by the R&D laboratory of the Robot Division of a Shanghai Electric Co., Ltd., China. For the rotation angle, speed, acceleration, and torque signals of each axis of the present invention, the control cabinet collects and transmits data at a sampling frequency of 250Hz (i.e., sensor data is collected once every 4ms).

[0071] The present invention obtains all operation fault samples of a certain model of industrial robot with a maximum load of 220kg from July 1 to July 31, 2023. For normal operation data collection, one sample is a cycle of the set action movement (40 seconds, i.e., an array of 10,000 sampling moments). For fault data collection, one sample is the data of 4 seconds before and after the fault moment (i.e., an array of 2,000 sampling moments).

[0072] For the imbalance problem between normal operation data and fault data:

[0073] The working conditions for data collection are uncertain. It is possible that the robot will run all day without any failure, or it may fail frequently. The sample size is shown in Table 1. Since the amount of normal operation sample data is much larger than the amount of failure data sample data, only 2000 normal operation data samples were randomly selected for training.

[0074] The specific data distribution and the final experimental sample size are shown in Table 1.

[0075] Table 1

[0076]

[0077] For the imbalance problem between different fault data samples:

[0078] The three faults of the present invention also had the problem of sample imbalance during the sampling process. Label 2 quickly exceeded 200 groups, while label 1 and label 3 had very few occurrences in the two faults during the sampling process. Subsequently, by collecting fault data of other robots of the same model and the same action in the factory, 220 groups of samples were collected.

[0079] These two data imbalance problems have been taken into account during the training process, and the final sample size used for the experiment is shown in Table 2. The data imbalance problem that occurred during the sampling process was solved by reconstructing the dataset to avoid serious bias in the model.

[0080] The fault classification is shown in Table 2.

[0081] Table 2

[0082]

[0083] Note: i represents the i-th axis, i=0,1,2,3

[0084] Step (1.2.2) balances the data during the collection process;

[0085] There is a large class imbalance problem between the sample collection of the "axis i position out of limit" fault caused by the actual position exceeding the preset range and the other types of faults generated during the operation of the system. If this data is directly used for training, the resulting prediction model will inevitably have serious bias, causing the evaluation results to shift to the category with a larger number. In order to solve this problem, this embodiment randomly selects a portion of samples with a similar number of other faults from the "axis i position out of limit" fault through undersampling, reconstructs the training data set, and obtains the final experimental samples;

[0086] The final hyperparameters of the DQ-SAC network are shown in Table 3.

[0087] Table 3

[0088]

[0089]

[0090] After obtaining the experimental samples, the fault data features of different categories are extracted and learned, and then the corresponding fault data are labeled, and the DQ-SAC (Double-Q Soft Actor-Critic) algorithm is used to self-update the digital twin.

[0091] The second step is model predictive control modeling. The process is as follows:

[0092] Step (2.1) constructing a digital twin model of an industrial robot;

[0093] In step (2.1), the digital twin system needs to have the ability to update the twin model in real time and accurately to synchronize various emergencies when the industrial robot system is working. Therefore, reinforcement learning is used to realize the real-time working condition feedback of the twin model to the real system. In the process of simulating the behavior of the real physical system, the industrial robot digital twin model (Digital Twin Model, DTM) inputs u t , the state variable x t , state parameter θ t Under the condition of performance index J, the twin system state variable x is obtained according to the state equation t The best estimate of It also provides the real output of the system y t Best estimate with twin system Therefore, the model tracking problem is described as follows,

[0094]

[0095] θ i =π(x t ,y t ,y t+1 ,u t+1 );

[0096]

[0097] The reinforcement learning framework models the tracking problem through the model predictive control process (MPC), and applies the maximum entropy deep reinforcement learning algorithm to find the performance indicator J that best matches the display physical model to update the parameters of the twin model performance, ultimately making the system's real output y t+1 The best estimator x of the twin system t+1 They can match each other to achieve the purpose of model tracking.

[0098] Step (2.2) constructs the quaternion of the digital twin system;

[0099] In the step (2.2), the industrial robot model is described by a quaternion (S, A, J, M), including the following sub-steps:

[0100] In step (2.2.1), the state set S in the quaternary is composed of the input, output and state elements of the real physical system. For an industrial robot, the state set S is:

[0101] S={I i ,U i ,P i ,x t};

[0102] Where I represents the real-time current signal, U represents the real-time voltage signal, and P represents the real-time power. The industrial robot is a six-axis robot. The subscript i represents the information corresponding to different joints. t is the state variable, including position, velocity and temperature information;

[0103] In step (2.2.2), A in the quadruple is the action set, including the twin system needs to monitor the motion state of the industrial robot in real time,

[0104]

[0105] Where J(θ) is the Jacobian matrix, which contains the angular velocity and linear velocity information of each axis at each moment; R(t)p(t) are the rotation matrix and translation vector respectively; is the linear acceleration, is the angular acceleration, the force f on each link i , torque n i and joint torque τ i ;

[0106] In step (2.2.3), J in the quadruple is the cost function used to evaluate the tracking effect, expressed as:

[0107]

[0108] Where L(x t ,u t ) is the stage cost at time t, F(x N ) is the terminal cost at the end of the forecast horizon;

[0109] In step (2.2.4), M in the quaternary is the mathematical model of system behavior. For the state data monitoring of the industrial robot digital twin system, it is expressed as

[0110] x t+1 =f(x t ,u t ,θ t )

[0111] y t+1 =g(x t ,u t ,θ t )

[0112] where x t is the state variable, which is the state of the system at time t; u t is the system input, the control input of the system at time t, and f is a function that describes the dynamics of the system.

[0113] The third step is to update the parameters of the DQ-SAC algorithm. On the basis of the SAC algorithm, a Q value is added to form a dual-Q network. The process is as follows:

[0114] Step (3.1) Actor is a strategy network, which takes the current state as input and outputs the probability distribution of actions or continuous action values. The Critic network then evaluates the quality of the action and adjusts the strategy accordingly.

[0115] In the step (3.1), Actor is a strategy network, which takes the current state as input and outputs the probability distribution of the action or the continuous action value. The Critic network then evaluates the quality of the action and adjusts the strategy. The formula is as follows:

[0116]

[0117] H(π(·|s t))=E[-logπ(·|s t )]

[0118] Where J(π) is the objective function of the DQ-SAC algorithm, V(s t ) is the state value function, Q(s t ,a t ) is the value function, the minimum value of the Soft-Q value is taken as two Q value functions parameterized by θ1 and θ2, which helps to avoid overestimating inappropriate Q values, and H(·) represents the entropy value. T represents the total number of time steps of interaction between the agent and the environment, ρ π Indicates that under strategy π (s t ,a t ), π is the current strategy, α is a hyperparameter that controls the randomness of the optimal strategy and the importance of weighting entropy for rewards, γ∈[0,1) is the discount factor, Calculate the information entropy distribution of state transition under the current strategy.

[0119] Step (3.2) adds a Q value to the SAC network to form a double Q network, and finally forms a DQ-SAC network;

[0120] In step (3.2), the Q value is a function used in reinforcement learning to measure the expected return that can be obtained after taking action a in a certain state s. The Q value function Q(s,a) gives the expected value of the accumulated reward that can be obtained after selecting action a in state s. In the dual Q network, the update target selects the smaller one of the two Q value networks to reduce the risk of overestimation.

[0121]

[0122]

[0123] in and are the two Q values ​​of the dual Q network, y is the update target with the minimum Q value, J Q (θ i ) is the loss function of the Q-value network.

[0124] The dual Q network of this embodiment can solve the overestimation problem in the traditional SAC algorithm. Through two independent Q value networks, the quality and convergence performance of the strategy caused by the overestimation of the Q value can be avoided, thereby improving the performance and stability of the learning algorithm.

[0125] Reference Figure 1In this embodiment, the digital twin model is used as the training environment to iteratively train the intelligent agent, and at the same time, the Actor and Critic networks are updated synchronously, and finally the converged optimal control strategy is obtained. The currently obtained optimal control strategy is applied when the digital twin is deployed; according to the real-time input of the real industrial robot system to the digital twin system, the real-time state parameter θ of the twin system is adjusted. t The performance index J is modified so that the digital twin system can respond to various information of the actual industrial robot system in real time and accurately.

[0126] This embodiment uses a fully connected neural network as the Actor network and Critic network in the reinforcement learning framework. Each layer consists of 256 neurons, ReLU is used as the activation function, Adam is used as the optimizer, and the network hyperparameter settings are shown in Table 3.

[0127] This embodiment verifies the effectiveness of the DQ-SAC algorithm of the present invention for self-updating the digital twin system through experiments. The DQ-SAC algorithm proposed in the present invention uses two Q value networks, and in the dual Q network, the update target selects the smaller one of the two Q value networks to reduce the risk of overestimation. In order to verify the effectiveness of DQ-SAC, a verification experiment was carried out based on the collected data set. The detailed information of the experimental setting is shown in Table 3, and the implementation process is as follows:

[0128] Step 1: Get the experimental data set.

[0129] The experimental data used in the present invention are derived from real-time operation data of a six-axis industrial robot in a sensor remotely acquired using the TCP / IP communication protocol.

[0130] Among them, the corresponding labels of industrial robot fault samples are shown in Table 2. There is a large category imbalance problem between the "axis i position out of limit" fault caused by the actual position exceeding the preset range and the sample collection of several other types of faults generated during the operation of the system. If this data is directly used for training, the resulting prediction model will inevitably have serious bias, causing the evaluation results to shift to categories with a larger number. In order to solve this problem, the present invention randomly selects a part of the samples with a similar number of other faults from the "axis i position out of limit" fault by undersampling, reconstructs the training data set, and obtains the final experimental sample data distribution as shown in Table 1.

[0131] The fault simulated in this experiment is "axis 2 position out of limit", with the input u given by the real system t Under the condition, the input u is collected at a frequency of 250Hz. t The angle change parameter θ of axis 2 from the start of the action to 4 seconds after the fault occurs 2t. A total of 200 data items, including 10 sets of normal motion data of real systems and 190 sets of simulated fault data, are used as the training set for the reinforcement learning DQ-SAC algorithm. When the industrial robot is operating normally, the working state is to continuously cycle the preset action in advance. In this experiment, the single data collection is set to cycle 200 preset action instructions, and the current action is stopped immediately when a fault occurs.

[0132] Step 2: Define evaluation indicators.

[0133] The network parameter settings are shown in Table 3. The evaluation criteria for learning effect are as follows:

[0134]

[0135] Among them, R is the criterion for judging the learning effect. When the value is closer to 1, it means that the twin model has a better tracking effect on the real system. r is the true parameter value, x a is the algorithm tracking value, and the experimental results are as follows Figure 2 shown.

[0136] Step 3: Experimental verification

[0137] (1) Verify the training effect of the DQ-SAC algorithm;

[0138] (2) Conduct a comparative experiment between the responses of different parameters under the control of the DQ-SAC algorithm and the actual response of the system;

[0139] Step 4: Result analysis

[0140] After the iterative training process, the model optimal control strategy is saved. Figure 2 and Figure 3 The performance of the current strategy on the entire training set and the specific effect on randomly selected data tracks are given. Figure 2 The twin model tracking training results show that on the entire training set, the algorithm control parameters are R ≥ 0.95 after 150 iterations, and the algorithm tracking value remains close to the true value. Figure 3 It can be seen that the specific performance of the verification on the randomly selected fault data is good. Under the current control strategy, the above-mentioned evaluation criterion R is close to 1, indicating that the median of the tracking parameter is close to the true value, and the upper and lower quartiles are also within a reasonable range. However, during the verification process, some outliers still appeared for each fault state. The reason may be that the training sample size of normal operation is large, the training effect is good, and the very individual data deviation is large, which may be abnormal data. There is one parameter that affects the "fault 1" state, which is the angle; there are two parameters that affect the "fault 2" state, which are the angle and speed; there are two parameters that affect the "fault 3" state, which are acceleration and torque. When there are many parameters affecting a certain state, the tracking effect is more challenging.

[0141] The system working cycle input and sensor measurement output are used as the input of the current optimal strategy, and the health characterization parameters of the twin model are continuously updated to complete the model self-update application verification. The experimental results are as follows Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 As shown in the figure, during the deployment phase, under the action of the optimal control strategy, the expected output of the twin model is close to the real system response, and the mirror image consistency of the mapping of the physical world is basically maintained.

[0142] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A self-updating method for a digital twin model of an industrial robot based on a reinforcement learning algorithm, characterized in that: The method comprises the following steps: The first step is to collect training sets and parameters. The process is as follows: Step (1.1) collects various real-time motion parameters of the industrial robot through sensors as the training set of the digital twin model, and obtains the optimal control strategy after an iterative training process; Step (1.2) classifies the different behaviors of industrial robots and balances the data collected; The second step is model predictive control modeling. The process is as follows: Step (2.1) constructing a digital twin model of an industrial robot; Step (2.2) constructs the quaternion of the digital twin system; The third step is to update the parameters of the DQ-SAC algorithm. On the basis of the SAC algorithm, a Q value is added to form a dual-Q network. The process is as follows: Step (3.1) Actor is a strategy network, which takes the current state as input and outputs the probability distribution of actions or continuous action values. The Critic network then evaluates the quality of the action and adjusts the strategy accordingly. Step (3.2) adds a Q value to the SAC network to form a double Q network, and finally forms a DQ-SAC network.

2. The method for self-updating the digital twin model of an industrial robot based on a reinforcement learning algorithm as claimed in claim 1, characterized in that: The step (1.2) includes the following sub-steps: Step (1.2.1) classifies different behaviors of real industrial robots; For the normal operation data collection of industrial robots, one sample is a cycle of the set action movement; for the fault data collection, one sample is the data of 4 seconds before and after the fault moment; different faults are classified; Step (1.2.2) balances the data during the collection process; Through undersampling, a portion of samples with a similar number to other faults are randomly selected from the faults with the highest frequency, and the training data set is reconstructed to obtain the final experimental samples; After obtaining the experimental samples, the fault data features of different categories are extracted and learned, and then the corresponding fault data are labeled, and the DQ-SAC algorithm is used to self-update the digital twin.

3. The industrial robot digital twin model self-updating method based on reinforcement learning algorithm according to claim 1 or 2, characterized in that: In the step (2.1), reinforcement learning is used to realize the real-time working condition feedback of the twin model to the real system. In the process of simulating the behavior of the real physical system, the industrial robot digital twin model DTM inputs u t , the state variable x t , state parameter θ t Under the condition of performance index J, the twin system state variable x is obtained according to the state equation t The best estimate of It also provides the real output of the system y t Best estimate with twin system Therefore, the model tracking problem is described as follows, i i =π(x t ,y t ,y t+1 ,u t+1 ); The reinforcement learning framework models the tracking problem through the model predictive control process (MPC), and applies the maximum entropy deep reinforcement learning algorithm to find the performance indicator J that best matches the display physical model to update the parameters of the twin model performance, ultimately making the system's real output y t+1 The best estimator x of the twin system t+1 Can match each other to achieve the purpose of model tracking; In the step (2.2), the industrial robot model is described by a quaternion (S, A, J, M), including the following sub-steps: In step (2.2.1), the state set S in the quaternary is composed of the input, output and state elements of the real physical system. For an industrial robot, the state set S is: S={I i ,U i ,P i ,x t }; Where I represents the real-time current signal, U represents the real-time voltage signal, and P represents the real-time power. The industrial robot is a six-axis robot. The subscript i represents the information corresponding to different joints. t is the state variable, including position, velocity and temperature information; In step (2.2.2), A in the quadruple is the action set, including the twin system for industrial robots that need to monitor their motion state in real time, Where J(θ) is the Jacobian matrix, which contains the angular velocity and linear velocity information of each axis at each moment; R(t)p(t) are the rotation matrix and translation vector respectively; is the linear acceleration, is the angular acceleration, the force f on each link i , torque n i and joint torque τ i ; In step (2.2.3), J in the quadruple is the cost function used to evaluate the tracking effect, expressed as: Where L(x t ,u t ) is the stage cost at time t, F(x N ) is the terminal cost at the end of the forecast horizon; In step (2.2.4), M in the quaternary is the mathematical model of system behavior. For the state data monitoring of the industrial robot digital twin system, it is expressed as x t+1 =f(x t ,u t ,θ t ) y t+1 =g(x t ,u t ,θ t ) where x t is the state variable, which is the state of the system at time t; u t is the system input, the control input of the system at time t, and f is a function that describes the dynamics of the system.

4. The industrial robot digital twin model self-updating method based on reinforcement learning algorithm according to claim 1 or 2, characterized in that: In the step (3.2), Actor is a strategy network, which takes the current state as input and outputs the probability distribution of the action or the continuous action value. The Critic network then evaluates the quality of the action and adjusts the strategy. The formula is as follows: H(π(·s t ))=E[-logπ(·s t )] Where J(π) is the objective function of the DQ-SAC algorithm, V(s t ) is the state value function, Q(s t ,a t ) is the value function, the minimum of the Soft-Q value is the two Q-value functions parameterized by θ1 and θ2, which helps to avoid overestimating inappropriate Q-values, H(·) represents the entropy value, T represents the total number of time steps of interaction between the agent and the environment, ρ π Indicates that under strategy π (s t ,a t ), π is the current strategy, α is a hyperparameter that controls the randomness of the optimal strategy and the importance of weighting entropy for rewards, γ∈[0,1) is the discount factor, Calculate the information entropy distribution of state transition under the current strategy.

5. The method for self-updating the digital twin model of an industrial robot based on a reinforcement learning algorithm as claimed in claim 4, characterized in that: In the step (3.3), the Q value is a function value used in reinforcement learning to measure the expected return after taking action a in a certain state s. The Q value function Q(s,a) gives the expected value of the accumulated reward that can be obtained after selecting action a in state s. In the dual Q network, the update target selects the smaller one of the two Q value networks to reduce the risk of overestimation. in and are the two Q values ​​of the dual Q network, y is the update target with the minimum Q value, J Q (θ i ) is the loss function of the Q-value network.

Images