Demonstrator learning method, device, medium, product and system

Abstract

The invention provides a demonstrator learning method, device, medium, product and system, and the method combines an artificial intelligence technology (namely a robot) with a demonstrator, can effectively reduce the technical threshold of a demonstrator learner, improves the accuracy of an operator using the demonstrator, reduces the safety risk of a working place, and improves the working efficiency. A data source used by machine learning is clarified by adopting a machine learning model, the model is trained by adopting operation data of a demonstrator operator and action data of a robot, and a final preset model is applied to a control process of the robot, so that the robot can cope with different scenes, and the control efficiency of the robot is improved. Therefore, the problem that in an existing scheme, an operator can only judge the distance between the robot and a contact object by naked eyes in most cases when using a demonstrator, and in the complex task execution process, the difference between the action of the robot and the expectation is large, so that the robot cannot complete the expectation task is solved.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a learning method, device, medium, product and system for a teaching pendant. Background Art

[0002] A teach pendant is a tool or device that is usually used in robotic systems to manually guide or program the robot to perform specific tasks. The teach pendant allows the human operator to directly control the robot and record these operating steps, which the robot can then repeat. In the foreseeable future, with the continuous decline in the labor force and the continuous increase in labor costs, more industrial robots (such as six-axis robots and scara robots) will inevitably be used, and therefore the accompanying teach pendants will also be used more.

[0003] The production environment where the teach pendant is used in industrial robots is often complex, and the operator has full authority over the robot and must make independent judgments to deal with various situations. Under the interference of multiple factors, the operator's operating accuracy and safety cannot be guaranteed.

[0004] In existing solutions, operators using a teach pendant can only rely on their naked eyes to judge the distance between the robot and the contact object in most cases. During the execution of complex tasks, the robot's movements are far from expectations, resulting in the robot being unable to complete the expected task. Summary of the invention

[0005] The main purpose of the present application is to provide a teaching pendant learning method, device, medium, product and system, so as to at least solve the problem in existing solutions that in most cases the operator can only rely on the naked eye to judge the distance between the robot and the contact object when using the teaching pendant. During the execution of complex tasks, the robot's movements are far from expectations, which makes the robot unable to complete the expected tasks.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a learning method for a teach pendant is provided, the method comprising: obtaining initial data, the initial data comprising operation data of a teach pendant operator and motion data of a robot, the motion data comprising position data, velocity data and acceleration data, the motion sequence, execution time and execution effect during the operation of the operation data; training a preset model based on a training set in the initial data to obtain a trained preset model, the type of the preset model being a machine learning model; using a verification set in the initial data to verify and optimize the trained preset model to obtain a final preset model, and applying the final preset model to the control process of the robot.

[0007] Optionally, training a preset model based on a training set in the initial data to obtain a trained preset model includes:

[0008] In the case where the preset model is a DQN algorithm model, the Q value is determined according to Q(s,a)=\mathbb{E}[R_t|s_t=s,a_t=a], where Q(s,a) is the Q value when the state is s and the action is a, R_t is the cumulative value of the immediate and future rewards obtained from time t to the end of the task, s_t is the state at time t, and a_t is the action at time t;

[0009] Extracting robot control experience data from the initial data, and storing the robot control experience data in an experience replay pool, wherein the experience replay pool provides randomly selected robot control experience data, and the robot control experience data represents corresponding solution strategies when the robot encounters different situations;

[0010] During each training, target experience data is extracted from the experience replay pool, and the DQN algorithm model is trained using the target experience data and the maximum value of the Q value predicted by the DQN algorithm model to obtain the training preset model to update the parameters in the DQN algorithm model, wherein the target experience data is at least part of the robot control experience data in the experience replay pool.

[0011] Optionally, training a preset model based on a training set in the initial data to obtain a trained preset model includes:

[0012] When the preset model is a policy gradient algorithm model, the objective function is constructed as:

[0013] J(\theta)=\mathbb{E}{\tau\sim\pi\theta}\left[G_t\right];

[0014] Where \mathbb{E} represents the expected value, \tau is a complete time series, \sim represents sampling from the target distribution, \pi is a strategy that depends on the network parameters (\theta), J(\theta) is the cost value, and \left[G_t\right] is the future return starting from time step t;

[0015] The objective function is gradually optimized until the objective function is minimized to obtain the training preset model.

[0016] Optionally, in the process of training the preset model based on the training set in the initial data to obtain the trained preset model, the method also includes: optimizing the preset model using the training set in the initial data based on gradually increasing the initial preset learning rate to obtain the rate of change of the loss value; when the rate of change of the loss value is less than a first preset rate, continuing to increase the learning rate; when the rate of change of the loss value is greater than or equal to the first preset rate and the rate of change of the loss value is less than a second preset rate, keeping the learning rate unchanged; when the rate of change of the loss value is greater than or equal to the second preset rate, reducing the learning rate.

[0017] Optionally, the validation set in the initial data is used to verify and optimize the training preset model to obtain a final preset model, including: using Python's scikit-learn to calculate the accuracy of the training preset model in predicting the validation set; when the accuracy of the training preset model is less than an accuracy threshold, using the validation set to optimize the training preset model until the accuracy of the training preset model is greater than or equal to the accuracy threshold; when the accuracy of the training preset model is greater than or equal to the accuracy threshold, determining that the final preset model is the current training preset model.

[0018] Optionally, the validation set in the initial data is used to validate and optimize the training preset model to obtain a final preset model, including: based on Python's scikit-learn and K-fold cross-validation method, calculating the average cross-validation accuracy of the training preset model for predicting the validation set; when the average cross-validation accuracy of the training preset model is less than an accuracy threshold, using the validation set to optimize the training preset model until the average cross-validation accuracy of the training preset model is greater than or equal to the accuracy threshold; when the average cross-validation accuracy of the training preset model is greater than or equal to the accuracy threshold, determining that the final preset model is the current training preset model.

[0019] According to another aspect of the present application, a learning device for a teach pendant is provided, the device comprising: an acquisition unit for acquiring initial data, the initial data comprising operation data of a teach pendant operator and motion data of a robot, the motion data comprising position data, velocity data and acceleration data, the motion sequence, execution time and execution effect during the operation of the operation data; a first processing unit for training a preset model based on a training set in the initial data to obtain a trained preset model, the type of the preset model being a machine learning model; a second processing unit for verifying and optimizing the trained preset model using a verification set in the initial data to obtain a final preset model, and applying the final preset model to the control process of the robot.

[0020] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0021] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the methods described are implemented.

[0022] According to another aspect of the present application, a learning system of a teach pendant is provided, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the methods described.

[0023] By applying the technical solution of the present application, artificial intelligence technology (i.e., a robot) is combined with a teach pendant, which can effectively lower the technical threshold for teach pendant learners, improve the accuracy of operators in using the teach pendant, and reduce safety risks in the workplace. A machine learning model is used to clarify the source of data used for machine learning, and the model is trained using the operating data of the teach pendant operator and the motion data of the robot. The final preset model is applied to the control process of the robot to enable the robot to cope with different scenarios, thereby solving the problem in the existing solution that in most cases the operator can only rely on the naked eye to judge the distance between the robot and the contact object when using the teach pendant. During the execution of complex tasks, the robot's movements are greatly different from expectations, which leads to the robot's inability to complete the expected tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 A schematic diagram of a flow chart of a learning method of a teaching pendant provided according to an embodiment of the present application is shown;

[0026] Figure 2 A schematic diagram of a learning method of a teaching pendant provided according to an embodiment of the present application is shown;

[0027] Figure 3 A structural block diagram of a learning device of a teaching pendant provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0032] scikit-learn is a Python library for machine learning that provides a variety of machine learning algorithms and tools, including classification, regression, clustering, dimensionality reduction, and model selection. scikit-learn is an open source project that aims to provide a simple and effective tool to help users build and deploy machine learning models. By using scikit-learn, users can quickly implement various machine learning tasks and perform model evaluation and optimization.

[0033] K-fold cross validation is a commonly used model evaluation method, which is usually used to evaluate the performance of machine learning algorithms. In K-fold cross validation, the data set is divided into K subsets of similar size, and then K model training and evaluation are performed, each time using K-1 subsets as training sets and the remaining subset as test sets. Finally, the average of the K evaluation results is used as the performance indicator of the model.

[0034] The DQN algorithm model (Deep Q-Network) is a method based on deep learning and reinforcement learning, which is used to solve the reinforcement learning problem in discrete action space. The algorithm approximates the Q value function through a neural network and selects the optimal action by maximizing the Q value. The DQN algorithm uses experience replay and target network methods to stabilize the training process during training.

[0035] The policy gradient algorithm model is a type of reinforcement learning method that directly optimizes the policy. It directly learns the policy function instead of the value function. Common policy gradient algorithms include the REINFORCE algorithm and the PPO algorithm. These algorithms update the policy parameters by maximizing the objective function, so that the policy can obtain higher rewards in the environment. The policy gradient algorithm is usually applicable to reinforcement learning problems in continuous action spaces.

[0036] The scara robot is a robot used in industrial production and automation applications. It has a horizontal rotating joint, a vertical rotating joint and a telescopic arm. This robot is often used for assembly, packaging, handling and other tasks that require precise and fast operations. Its design enables it to perform complex movements in a limited space while maintaining high precision and stability. The scara robot is often used in the electronics, automotive, medical and food industries.

[0037] Q-learning is a reinforcement learning algorithm for learning to make optimal decisions in a trial-and-error environment. In Q-learning, the agent continuously updates its Q-value function, which represents the expected reward value for a given state and action, by interacting with the environment.

[0038] As introduced in the background technology, the production site where the teach pendant is used in industrial robots is often in a complex environment, and the operator has full authority over the robot. He must make independent judgments to deal with various situations. The operator's operating accuracy and safety cannot be guaranteed under the interference of various factors. In order to solve the problem that in the existing solutions, the operator can only rely on the naked eye to judge the distance between the robot and the contact object when using the teach pendant, and in the process of executing complex tasks, the robot's movements are far from expectations, which makes the robot unable to complete the expected tasks. The embodiments of the present application provide a learning method, device, medium, product and system for a teach pendant.

[0039] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0040] In this embodiment, a learning method for a teach pendant is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] Figure 1 1 is a flow chart of a teaching pendant learning method provided according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0042] Step S101, obtaining initial data, the initial data including the operation data of the teach pendant operator and the motion data of the robot, the motion data including position data, velocity data and acceleration data, the motion sequence, execution time and execution effect during the operation of the operation data;

[0043] Specifically, based on the operation data of the teach pendant operator and the motion data of the robot, it is easy to determine which operation strategy should be used to control the robot in different situations.

[0044] Step S102, training a preset model based on the training set in the initial data to obtain a trained preset model, where the type of the preset model is a machine learning model;

[0045] In one embodiment of the present application, the step S102 of training the preset model based on the training set in the initial data to obtain the trained preset model includes:

[0046] In the case where the above preset model is the DQN algorithm model, the Q value is determined according to Q(s,a)=\mathbb{E}[R_t|s_t=s,a_t=a], where Q(s,a) is the Q value when the state is s and the action is a, R_t is the cumulative value of the immediate and future rewards obtained from time t to the end of the task, s_t is the state at time t, and a_t is the action at time t;

[0047] Extract robot control experience data from the initial data, and store the robot control experience data in an experience replay pool, wherein the experience replay pool provides randomly selected robot control experience data, and the robot control experience data represents corresponding solution strategies when the robot encounters different situations;

[0048] During each training, target experience data is extracted from the above-mentioned experience replay pool, and the above-mentioned DQN algorithm model is trained using the above-mentioned target experience data and the maximum value of the Q value predicted by the above-mentioned DQN algorithm model to obtain the above-mentioned training preset model to update the parameters in the above-mentioned DQN algorithm model, and the above-mentioned target experience data is at least part of the above-mentioned robot control experience data in the above-mentioned experience replay pool.

[0049] Specifically, the relative position of objects, the current position and posture of the robot, sensor readings (such as force, temperature or light intensity), environmental changes (such as the appearance of new objects or the movement of obstacles) are the parameters that need to be defined first; define the robot's executable action set, such as moving to specific coordinates, grabbing and placing objects, and performing detection tasks. These will be part of the strategy of the DQN algorithm model.

[0050] Establish a DQN environment, including an online network and a target network. The online network is used to predict the Q value of possible actions, while the target network is used to determine the future rewards of these actions. The Q value evaluates an action (a) in a certain state (s), which represents the expected sum of future rewards that can be obtained from the current time point until the end of the task after the action is executed in this state. Here, the reward is usually assumed to be a value discounted at a certain discount rate at a future time point.

[0051] The initial data is converted into training samples, each of which contains the current state (data collected from the teach pendant), the action taken (the demonstrated action), the immediate reward obtained (such as the inverse of goal completion, efficiency, or energy consumption), and the next state.

[0052] Extract robot control experience data from the initial data, and store the robot control experience data in an experience replay pool, wherein the experience replay pool provides randomly selected robot control experience data, and the robot control experience data represents corresponding solution strategies when the robot encounters different situations;

[0053] During each training, target experience data is extracted from the above-mentioned experience replay pool, and the above-mentioned DQN algorithm model is trained using the above-mentioned target experience data and the maximum value of the Q value predicted by the above-mentioned DQN algorithm model to obtain the above-mentioned training preset model to update the parameters in the above-mentioned DQN algorithm model, and the above-mentioned target experience data is at least part of the above-mentioned robot control experience data in the above-mentioned experience replay pool.

[0054] This achieves the goal of applying the DQN algorithm model to the model training scenario of the teach pendant-controlled robot.

[0055] In one embodiment of the present application, the step S102 of training the preset model based on the training set in the initial data to obtain the trained preset model includes:

[0056] When the above preset model is a policy gradient algorithm model, the objective function is constructed as:

[0057] J(\theta)=\mathbb{E}{\tau\sim\pi\theta}\left[G_t\right];

[0058] Where \mathbb{E} represents the expected value, \tau is a complete time series, \sim represents sampling from the target distribution, \pi is a strategy that depends on the network parameters (\theta), J(\theta) is the cost value, and \left[G_t\right] is the future return (i.e., discounted return) starting from time step t;

[0059] \tau\sim\pi\theta is an overall symbolic representation used to describe the trajectory τ from the policy π θ Generated.

[0060] \tau, or τ, is a complete time series, indicating that starting from the initial state s0, according to the strategy π θ Select action a t , and receive a reward t+1 The whole process, that is, τ=(s0,a0,r1,s1,a1,r2,…,s T ,a T ,r T+1 ).

[0061] \sim, that is, ~, means sampling from the target distribution.

[0062] \pi\theta represents π θ is a strategy that depends on the network parameters θ, which represents the probability distribution of selecting action a in a given state s, i.e., π θ(s,a) represents the probability of selecting action a in state s.

[0063] The objective function is gradually optimized until it is minimized to obtain the preset training model.

[0064] Specifically, define a policy function π, which is a mapping from state to action probability, π(a|s) represents the probability of selecting action a in state s. This policy can be a neural network, the input of the network is the state of the environment (such as object position, robot position and posture, etc.), and the output is the action probability of each possible action.

[0065] Using the gradient ascent (or descent) algorithm, the policy parameters can be updated by calculating the gradient of J(\theta). Since the optimization is performed directly on the policy, the gradient update directly reflects the step close to the maximum expected reward.

[0066] In one embodiment of the present application, in the process of training the preset model based on the training set in the initial data to obtain the trained preset model, the method further includes:

[0067] Based on gradually increasing the initial preset learning rate, the preset model is optimized using the training set in the initial data to obtain the rate of change of the loss value;

[0068] When the change rate of the loss value is less than the first preset rate, continue to increase the learning rate;

[0069] When the change rate of the loss value is greater than or equal to the first preset rate and the change rate of the loss value is less than the second preset rate, keeping the learning rate unchanged;

[0070] When the rate of change of the loss value is greater than or equal to the second preset rate, the learning rate is reduced.

[0071] Specifically, the learning rate controls the magnitude of the model change after each data input, affecting the convergence speed and stability of the model during training. A learning rate that is too large may cause the model to oscillate or fail to converge, while a learning rate that is too small may cause the training time to be too long or fall into a local optimum.

[0072] Learning rate More specifically, a moderate initial learning rate is usually selected when the model is initially trained, such as 0.001 or 0.01. During the training process, monitor the changes in the model's loss or other performance indicators on the training set. If the training loss value does not decrease or decreases too slowly, the learning rate can be increased. If the training loss fluctuates greatly or the training becomes unstable after increasing the learning rate, it may be necessary to gradually reduce the learning rate or adopt a dynamic learning rate adjustment strategy. Common dynamic learning rate adjustment methods include: fixed adjustment: manually adjust the learning rate after a certain number of iterations; learning rate decay: gradually reduce the learning rate as the number of training rounds increases, such as reducing a fixed ratio for each epoch; performance-based adjustment: dynamically adjust the learning rate based on the performance of the model on the validation set, such as reducing the learning rate when the performance no longer improves.

[0073] Step S103, using the verification set in the above initial data to verify and optimize the above training preset model to obtain a final preset model, and applying the above final preset model to the control process of the above robot.

[0074] In the above steps, combining artificial intelligence technology (i.e., robot) with the teach pendant can effectively lower the technical threshold for teach pendant learners, improve the accuracy of operators using the teach pendant, and reduce safety risks in the workplace. The machine learning model is used to clarify the data source used for machine learning, and the model is trained using the teach pendant operator's operating data and the robot's motion data. The above-mentioned final preset model is applied to the control process of the above-mentioned robot to enable the robot to cope with different scenarios, thereby solving the problem in the existing solution that in most cases the operator can only rely on the naked eye to judge the distance between the robot and the contact object when using the teach pendant. During the execution of complex tasks, the robot's actions are greatly different from expectations, which leads to the robot's inability to complete the expected tasks.

[0075] In one embodiment of the present application, the step S103 uses the verification set in the initial data to verify and optimize the training preset model to obtain the final preset model, including:

[0076] Use Python's scikit-learn to calculate the accuracy of the above training preset model in predicting the above validation set;

[0077] In the case where the accuracy of the above-mentioned training preset model is less than the accuracy threshold, the above-mentioned training preset model is optimized by using the above-mentioned validation set until the accuracy of the above-mentioned training preset model is greater than or equal to the above-mentioned accuracy threshold;

[0078] When the accuracy of the above-mentioned training preset model is greater than or equal to the above-mentioned accuracy threshold, the above-mentioned final preset model is determined to be the current training preset model.

[0079] Specifically, the accuracy threshold can be set to different specific values ​​according to different application scenarios, which will not be elaborated here. Using scikit-learn can accurately determine the accuracy of the prediction of the preset training model, which is convenient for optimizing and adjusting the model.

[0080] In one embodiment of the present application, the step S103 uses the verification set in the initial data to verify and optimize the training preset model to obtain the final preset model, including:

[0081] Based on Python's scikit-learn and K-fold cross-validation method, calculate the average cross-validation accuracy of the above training preset model predicting the above validation set;

[0082] When the average value of the cross-validation accuracy of the above-mentioned preset training model is less than the accuracy threshold, the above-mentioned validation set is used to optimize the above-mentioned preset training model until the average value of the cross-validation accuracy of the above-mentioned preset training model is greater than or equal to the above-mentioned accuracy threshold;

[0083] When the average value of the cross-validation accuracy of the above-mentioned training preset model is greater than or equal to the above-mentioned accuracy threshold, the above-mentioned final preset model is determined to be the current training preset model.

[0084] Specifically, the dataset is divided into multiple subsets, the model is trained on different training sets, and then the model performance is evaluated on the remaining subsets. This can evaluate the performance of the model on different data distributions. The original dataset is divided into K equal or almost equal subsets. In K iterations, one of the subsets is selected as the test set each time, and the remaining K-1 subsets are combined as the training set. In this way, each subset will be used as test data once to ensure that all data is tested fairly. In this way, the randomness and contingency that may be caused by the one-time division of the training set and the test set are avoided.

[0085] The "dead point" can be used as an example to specifically illustrate how to test accuracy, learning speed, and generalization ability. A six-axis robot may encounter the so-called "dead point" problem during movement. This problem usually refers to the robot being unable to move continuously or achieve the desired posture due to joint angle limitations or mechanical structure design in a specific position or posture. The teach pendant can be used to plan paths to avoid dead points in robot movement. Use the teach pendant to collect the operator's action data and path selection when avoiding dead points. This data can include the robot's joint angles, the position of the end effector, and posture information.

[0086] Laser ranging and cameras can be used to collect the operating data of the teach pendant operator. If conditions are insufficient, a sensor that senses position changes can be installed in the teach pendant to collect the operating data of the teach pendant operator. Because the teach pendant is held in the operator's left hand when it is in use, and the left hand fingers press the pressure-sensing button, the motion data of the teach pendant can be roughly regarded as the operator's operating data (or motion data) for training the model.

[0087] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the learning method of the teaching pendant of the present application will be described in detail below in combination with specific embodiments.

[0088] The principle diagram of the teaching pendant's learning method is as follows: Figure 2 As shown, this embodiment relates to a specific teaching pendant learning method, including:

[0089] Data collection and preprocessing:

[0090] The data generated by the teach pendant operator when operating the teach pendant is collected. The teach pendant collects and temporarily stores a large amount of non-image data such as the operator's operation records and the scara robot's feedback data (such as sensor data). The supervision system can use the initial data including the teach pendant operator's operation data and the robot's motion data. The above motion data includes position data, speed data and acceleration data, and the above operation data includes the action sequence, execution time and execution effect during the operation process.

[0091] The collected raw data will be summarized and transmitted to the cloud or local server. Preprocessing the collected data is a very important step, which may include data cleaning (processing abnormal data and noise), data normalization or standardization (ensuring that the data is on the same scale), and feature extraction (extracting useful features from the raw data).

[0092] Select a learning model:

[0093] Select the appropriate machine learning model based on the specific tasks and characteristics of the data (such as classification, regression, and time series prediction). In the teaching pendant path and action learning, commonly used models include reinforcement learning algorithms (such as Q-learning, DQN algorithm models), supervised learning (such as classifiers or regression models), and unsupervised learning (such as clustering algorithms). When selecting a model, consider the model's ability to handle action sequences and environmental states. Use historical operation data to train machine learning models, which can learn the operator's normal operating mode and the relationship between operation and environment.

[0094] Model training:

[0095] The selected model is trained using the prepared data set. During the training process, the model will learn how to make decisions and predictions based on the input state and target (such as the best path or action). The training process may involve parameter adjustment and cross-validation techniques. Model training is done in the cloud or on a local server because the teach pendant is usually designed to be lightweight, portable, and easy to use. The computing resources of the teach pendant are relatively insufficient and are almost all used in image applications and control modules.

[0096] Model evaluation and optimization:

[0097] After training is complete, the performance of the model needs to be evaluated. This can be done by comparing it with an existing dataset or by performing a simulation test in an actual teach pendant. The indicators used to evaluate the model include accuracy, learning speed, generalization ability, etc. If the model is not ideal, it can be optimized by adjusting the model architecture, increasing the amount of data, improving feature extraction, etc.

[0098] Deployment and application:

[0099] When the model performs well and reaches the expected performance indicators, it can be deployed to the actual teach pendant. During the deployment process, it is necessary to ensure the integration of the model with the teach pendant system to achieve real-time path planning and action decision-making. After the model is deployed, the teach pendant system can perform adaptive learning and optimization based on the user's habits and real-time needs, predict possible user operation errors, and provide timely correction suggestions;

[0100] Continuous Improvement:

[0101] By monitoring performance and feedback in actual applications, collecting new data and periodically retraining the model to adapt to changing environments and needs, the operational efficiency and safety of the teach pendant are continuously improved.

[0102] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0103] The embodiment of the present application also provides a learning device for a teaching pendant. It should be noted that the learning device for the teaching pendant of the embodiment of the present application can be used to execute the learning method for the teaching pendant provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0104] The following is an introduction to the learning device of the teaching pendant provided in the embodiment of the present application.

[0105] Figure 3 1 is a structural block diagram of a learning device of a teaching pendant provided according to an embodiment of the present application. Figure 3 As shown, the device comprises:

[0106] An acquisition unit 31 is used to acquire initial data, wherein the initial data includes operation data of the teach pendant operator and motion data of the robot, wherein the motion data includes position data, velocity data and acceleration data, and the motion sequence, execution time and execution effect during the operation of the operation data;

[0107] A first processing unit 32 is used to train a preset model based on the training set in the initial data to obtain a trained preset model, where the preset model is a machine learning model;

[0108] The second processing unit 33 is used to verify and optimize the above-mentioned training preset model using the verification set in the above-mentioned initial data to obtain the final preset model, and apply the above-mentioned final preset model to the control process of the above-mentioned robot.

[0109] In the above-mentioned device, the combination of artificial intelligence technology (i.e., robot) and teach pendant can effectively lower the technical threshold for teach pendant learners, improve the accuracy of operators using teach pendants, and reduce safety risks in the workplace. The machine learning model is used to clarify the data source used for machine learning, and the model is trained using the operating data of the teach pendant operator and the motion data of the robot. The above-mentioned final preset model is applied to the control process of the above-mentioned robot to enable the robot to cope with different scenarios, thereby solving the problem in the existing solution that in most cases the operator can only rely on the naked eye to judge the distance between the robot and the contact object when using the teach pendant. During the execution of complex tasks, the robot's actions are greatly different from expectations, which leads to the robot's inability to complete the expected tasks.

[0110] In one embodiment of the present application, the first processing unit includes a first processing module, a second processing module and a third processing module.

[0111] The first processing module is used to determine the Q value according to Q(s,a)=\mathbb{E}[R_t|s_t=s,a_t=a] when the above preset model is a DQN algorithm model, where Q(s,a) is the Q value when the state is s and the action is a, R_t is the cumulative value of the immediate and future rewards obtained from time t to the end of the task, s_t is the state at time t, and a_t is the action at time t;

[0112] The second processing module is used to extract robot control experience data from the initial data, and store the robot control experience data in an experience replay pool, wherein the experience replay pool provides randomly selected robot control experience data, and the robot control experience data represents the corresponding solution strategies when the robot encounters different situations;

[0113] The third processing module is used to extract target experience data from the above-mentioned experience replay pool during each training, and use the above-mentioned target experience data and the maximum value of the Q value predicted by the above-mentioned DQN algorithm model to train the above-mentioned DQN algorithm model to obtain the above-mentioned training preset model to update the parameters in the above-mentioned DQN algorithm model, and the above-mentioned target experience data is at least part of the above-mentioned robot control experience data in the above-mentioned experience replay pool.

[0114] In one embodiment of the present application, the first processing unit includes a building module and a fourth processing module.

[0115] The construction module is used to construct the objective function when the above preset model is a policy gradient algorithm model:

[0116] J(\theta)=\mathbb{E}{\tau\sim\pi\theta}\left[G_t\right];

[0117] Among them, \mathbb{E} represents the expected value, \tau is a complete time series, \sim represents sampling from the target distribution, \pi is a strategy that depends on the network parameters (\theta), J(\theta) is the cost value, and \left[G_t\right] is the future return starting from time step t;

[0118] The fourth processing module is used to gradually optimize the above objective function until the above objective function is minimized to obtain the above training preset model.

[0119] In one embodiment of the present application, the first processing unit includes a fifth processing module, a sixth processing module, a seventh processing module and an eighth processing module. In the process of training the preset model based on the training set in the above initial data to obtain the trained preset model, the above method also includes:

[0120] The fifth processing module is used to optimize the preset model by using the training set in the initial data based on gradually increasing the initial preset learning rate to obtain the rate of change of the loss value;

[0121] The sixth processing module is used for continuing to increase the learning rate when the change rate of the loss value is less than the first preset rate;

[0122] The seventh processing module is used to keep the learning rate unchanged when the change rate of the loss value is greater than or equal to the first preset rate and the change rate of the loss value is less than the second preset rate;

[0123] The eighth processing module is used to reduce the learning rate when the change rate of the loss value is greater than or equal to the second preset rate.

[0124] In one embodiment of the present application, the second processing unit includes a ninth processing module, a tenth processing module and an eleventh processing module.

[0125] The ninth processing module is used to calculate the accuracy of the prediction of the above-mentioned validation set by the above-mentioned training preset model using Python's scikit-learn;

[0126] The tenth processing module is used for optimizing the preset training model by using the validation set when the accuracy of the preset training model is less than the accuracy threshold, until the accuracy of the preset training model is greater than or equal to the accuracy threshold;

[0127] The eleventh processing module is used to determine that the final preset model is the current training preset model when the accuracy of the training preset model is greater than or equal to the accuracy threshold.

[0128] In one embodiment of the present application, the second processing unit includes a twelfth processing module, a thirteenth processing module and a fourteenth processing module.

[0129] The twelfth processing module is used to calculate the average cross-validation accuracy of the above-mentioned training preset model predicting the above-mentioned validation set based on Python's scikit-learn and K-fold cross-validation method;

[0130] The thirteenth processing module is used for optimizing the above-mentioned preset training model by using the above-mentioned validation set when the average value of the cross-validation accuracy of the above-mentioned preset training model is less than the accuracy threshold, until the average value of the above-mentioned cross-validation accuracy of the above-mentioned preset training model is greater than or equal to the above-mentioned accuracy threshold;

[0131] The fourteenth processing module is used to determine that the final preset model is the current training preset model when the average value of the cross-validation accuracy of the training preset model is greater than or equal to the accuracy threshold.

[0132] The learning device of the teaching pendant includes a processor and a memory. The acquisition unit, the first processing unit, the second processing unit, etc. are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0133] The processor contains a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that in most cases, the operator can only rely on the naked eye to judge the distance between the robot and the contact object when using the teach pendant. In the process of executing complex tasks, the robot's movements are far from the expected ones, which makes the robot unable to complete the expected tasks.

[0134] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0135] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the learning method of the teaching pendant.

[0136] An embodiment of the present invention provides a processor, which is used to run a program, wherein the learning method of the teaching pendant is executed when the program is running.

[0137] An embodiment of the present invention provides a device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements at least the following steps when executing the program: obtaining initial data, the initial data includes the operation data of the teach pendant operator and the motion data of the robot, the motion data includes position data, velocity data and acceleration data, the motion sequence, execution time and execution effect during the operation of the operation data; training a preset model based on the training set in the initial data to obtain a training preset model, the type of the preset model is a machine learning model; using the verification set in the initial data to verify and optimize the training preset model to obtain a final preset model, and applying the final preset model to the control process of the robot. The device in this article can be a server, PC, PAD, mobile phone, etc.

[0138] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps: obtaining initial data, the initial data including the operation data of the teach pendant operator and the motion data of the robot, the motion data including position data, velocity data and acceleration data, the motion sequence, execution time and execution effect during the operation of the operation data; training a preset model based on the training set in the initial data to obtain a trained preset model, the type of the preset model being a machine learning model; using the verification set in the initial data to verify and optimize the trained preset model to obtain a final preset model, and applying the final preset model to the control process of the robot.

[0139] The present application also provides a learning system for a teach pendant, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any of the above methods. Combining artificial intelligence technology (i.e., a robot) with a teach pendant can effectively reduce the technical threshold of teach pendant learners, improve the accuracy of operators using the teach pendant, reduce safety risks in the workplace, use a machine learning model to clarify the data source used for machine learning, use the teach pendant operator's operating data and the robot's motion data to train the model, and apply the above final preset model to the control process of the robot, so that the robot can cope with different scenarios, thereby solving the problem that in most cases of the existing solution, the operator can only rely on the naked eye to judge the distance between the robot and the contact object when using the teach pendant. In the process of complex task execution, the robot's action is far from the expected, which leads to the problem that the robot cannot complete the expected task.

[0140] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0142] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0145] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0146] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0147] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0148] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0149] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0150] 1) The learning method of the teach pendant of the present application combines artificial intelligence technology (i.e., a robot) with a teach pendant, which can effectively lower the technical threshold for teach pendant learners, improve the accuracy of operators using the teach pendant, and reduce safety risks in the workplace. The machine learning model is used to clarify the data source used for machine learning, and the operating data of the teach pendant operator and the robot's motion data are used to train the model. The above-mentioned final preset model is applied to the control process of the above-mentioned robot to enable the robot to cope with different scenarios, thereby solving the problem that in most cases in the existing solution, the operator can only rely on the naked eye to judge the distance between the robot and the contact object when using the teach pendant. During the execution of complex tasks, the robot's actions are greatly different from expectations, which leads to the robot's inability to complete the expected tasks.

[0151] 2) The learning device of the teach pendant of the present application combines artificial intelligence technology (i.e., a robot) with a teach pendant, which can effectively lower the technical threshold for teach pendant learners, improve the accuracy of operators using the teach pendant, and reduce safety risks in the workplace. The machine learning model is used to clarify the data source used for machine learning, and the operating data of the teach pendant operator and the robot's motion data are used to train the model. The above-mentioned final preset model is applied to the control process of the above-mentioned robot to enable the robot to cope with different scenarios, thereby solving the problem in the existing solution that in most cases the operator can only rely on the naked eye to judge the distance between the robot and the contact object when using the teach pendant. During the execution of complex tasks, the robot's actions are greatly different from expectations, which leads to the robot's inability to complete the expected tasks.

[0152] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A teaching pendant learning method, characterized in that: include: Acquire initial data, the initial data including operation data of the teach pendant operator and motion data of the robot, the motion data including position data, velocity data and acceleration data, and the motion sequence, execution time and execution effect during the operation of the operation data; Training a preset model based on a training set in the initial data to obtain a trained preset model, where the type of the preset model is a machine learning model; The training preset model is verified and optimized using a verification set in the initial data to obtain a final preset model, and the final preset model is applied to the control process of the robot.

2. The method according to claim 1, characterized in that The preset model is trained based on the training set in the initial data to obtain a trained preset model, including: In the case where the preset model is a DQN algorithm model, the Q value is determined according to Q(s,a)=\mathbb{E}[R_t|s_t=s,a_t=a], where Q(s,a) is the Q value when the state is s and the action is a, R_t is the cumulative value of the immediate and future rewards obtained from time t to the end of the task, s_t is the state at time t, and a_t is the action at time t; Extracting robot control experience data from the initial data, and storing the robot control experience data in an experience replay pool, wherein the experience replay pool provides randomly selected robot control experience data, and the robot control experience data represents corresponding solution strategies when the robot encounters different situations; During each training, target experience data is extracted from the experience replay pool, and the DQN algorithm model is trained using the target experience data and the maximum value of the Q value predicted by the DQN algorithm model to obtain the training preset model to update the parameters in the DQN algorithm model, wherein the target experience data is at least part of the robot control experience data in the experience replay pool.

3. The method according to claim 1, characterized in that The preset model is trained based on the training set in the initial data to obtain a trained preset model, including: When the preset model is a policy gradient algorithm model, the objective function is constructed as: J(\theta)=\mathbb{E}{\tau\sim\pi\theta}\left[G_t\right]; Among them, \mathbb{E} represents the expected value, \tau is a complete time series, \sim represents sampling from the target distribution, \pi is a strategy that depends on the network parameters (\theta), J(\theta) is the cost value, and \left[G_t\right] is the future return starting from time step t; The objective function is gradually optimized until the objective function is minimized to obtain the training preset model.

4. The method according to claim 1, characterized in that In the process of training the preset model based on the training set in the initial data to obtain the trained preset model, the method further includes: Based on gradually increasing the initial preset learning rate, the preset model is optimized using the training set in the initial data to obtain the rate of change of the loss value; When the rate of change of the loss value is less than the first preset rate, continue to increase the learning rate; When the change rate of the loss value is greater than or equal to the first preset rate and the change rate of the loss value is less than the second preset rate, keeping the learning rate unchanged; When the rate of change of the loss value is greater than or equal to the second preset rate, reducing the learning rate.

5. The method according to claim 1, characterized in that The training preset model is verified and optimized using the verification set in the initial data to obtain a final preset model, including: Using Python's scikit-learn to calculate the accuracy of the training preset model in predicting the validation set; When the accuracy of the training preset model is less than the accuracy threshold, optimizing the training preset model using the validation set until the accuracy of the training preset model is greater than or equal to the accuracy threshold; When the accuracy of the training preset model is greater than or equal to the accuracy threshold, the final preset model is determined to be the current training preset model.

6. The method according to claim 1, characterized in that The training preset model is verified and optimized using the verification set in the initial data to obtain a final preset model, including: Based on Python's scikit-learn and K-fold cross-validation method, calculate the average cross-validation accuracy of the training preset model predicting the validation set; In the case where the average value of the cross-validation accuracy of the training preset model is less than the accuracy threshold, optimizing the training preset model using the validation set until the average value of the cross-validation accuracy of the training preset model is greater than or equal to the accuracy threshold; When the average value of the cross-validation accuracy of the training preset model is greater than or equal to the accuracy threshold, the final preset model is determined to be the current training preset model.

7. A teaching pendant learning device, characterized in that: include: An acquisition unit, used to acquire initial data, the initial data including operation data of the teach pendant operator and motion data of the robot, the motion data including position data, velocity data and acceleration data, and the motion sequence, execution time and execution effect during the operation of the operation data; A first processing unit, configured to train a preset model based on a training set in the initial data to obtain a trained preset model, wherein the preset model is a machine learning model; The second processing unit is used to verify and optimize the training preset model using the verification set in the initial data to obtain a final preset model, and apply the final preset model to the control process of the robot.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A teaching pendant learning system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 6.

Images