Multi-type shaft hole assembly mechanical arm control method and related device

Abstract

The invention belongs to the technical field of automatic assembly, and discloses a multi-type shaft hole assembly mechanical arm control method and related device.The method comprises the steps that the task type of an assembly task is obtained, and model parameters of a preset control strategy model are determined according to the task type of the assembly task; state data of the mechanical arm are obtained, and according to the state data of the mechanical arm, an initial control strategy of the mechanical arm is obtained through a control strategy model determined through the model parameters; the control strategy model is obtained through the following training modes that an incremental reinforcement learning model used for predicting a control strategy of the mechanical arm is obtained; and performing incremental training on the incremental reinforcement learning model based on the simulation environment of the training assembly tasks of the plurality of task types, and reserving model parameters of the plurality of task types to obtain a control strategy model. Therefore, the control strategy model can quickly adapt to an environment in which task types change continuously, skill knowledge of assembly tasks of different task types is kept, training time is shortened, and the assembly success rate is increased.

Description

Technical Field

[0001] The invention belongs to the technical field of automatic assembly, and relates to a multi-type shaft hole assembly mechanical arm control method and related devices. Background Art

[0002] With the continuous development of industry, robots are playing an increasingly important role in it. Among them, shaft hole assembly is a common task in production and life. In practical applications, the objects to be assembled are often diverse. For example, shaft sleeve assembly and gear assembly in industrial production, as well as plug insertion and USB insertion in life scenes. For different assembly objects, the assembly skills that the robot arm needs to master are also different.

[0003] Reinforcement learning, as a common machine learning method, is very suitable for robot arm skill learning. However, traditional reinforcement learning can only target a single skill learning scenario and is less applicable to the continuous learning of multiple skills. In addition, due to the limitations of neural networks themselves, commonly used deep reinforcement learning algorithms often forget old skills to a certain extent after learning new skills.

[0004] Based on the above analysis, the current control method of assembly robot based on reinforcement learning model can often only adapt to a single assembly task. When facing a new assembly task, the reinforcement learning model needs to be retrained, which takes a long time and the assembly accuracy is difficult to effectively control. In addition, after incremental training for new assembly tasks, the adaptability of old assembly tasks will inevitably be reduced, which will lead to a decrease in assembly control performance. Summary of the invention

[0005] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide a multi-type shaft hole assembly robot arm control method and related devices.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: According to a first aspect of the present invention, a method for controlling a multi-type shaft hole assembly robot arm is provided, comprising: obtaining a task type of an assembly task, and determining model parameters of a preset control strategy model according to the task type of the assembly task; obtaining state data of the robot arm, and determining a control strategy model through model parameters according to the state data of the robot arm to obtain an initial control strategy of the robot arm; wherein the control strategy model is obtained by the following training method: obtaining an incremental reinforcement learning model for predicting the control strategy of the robot arm; performing incremental training on the incremental reinforcement learning model in a simulation environment for training assembly tasks of several task types, and retaining the model parameters of several task types to obtain a control strategy model.

[0007] Optionally, the obtaining of the task type of the assembly task includes: obtaining type influence data of the assembly task; and according to the type influence data of the assembly task, obtaining a task similarity index value of the assembly task through a preset task type determination model, and obtaining the task type of the assembly task in combination with the preset task similarity index values ​​of each task type; wherein the task type determination model is obtained by: obtaining a pre-processing neural network model for predicting task similarity index values; training a pre-processing neural network model based on the type influence data of the trained assembly tasks of several task types to obtain a task type determination model; the obtaining of the state data of the robotic arm, and obtaining an initial control strategy of the robotic arm according to a control strategy model determined by model parameters based on the state data of the robotic arm includes: obtaining a normalization threshold of the assembly task according to the task type of the assembly task, normalizing the state data of the robotic arm according to the normalized state data of the robotic arm, and obtaining an initial control strategy of the robotic arm according to a control strategy model determined by model parameters based on the state data of the robotic arm.

[0008] Optionally, the type influencing data includes one or more of the following: the number of shaft holes to be assembled, the shape of the shaft holes to be assembled, the length of the shaft to be assembled, and the matching type of shaft hole assembly; the pre-processing neural network model includes three fully connected layers connected in sequence, and the activation function of the fully connected layer adopts the ReLU function.

[0009] Optionally, the state quantity of the incremental reinforcement learning model for: ,in, , and are the forces at the end of the robot in the x, y and z directions respectively, is the insertion depth of the sleeve; the action amount of the incremental reinforcement learning model for: ,in, are the displacements of the end of the robot in the x, y and z directions respectively; the reward function of the incremental reinforcement learning model for: ,in, For safe returns, is the step efficiency return, is the step efficiency return, and are all preset discount factors; the initial policy network of the incremental reinforcement learning model is based on the Actor-Critic structure, and the Actor network and the Critic network of the Actor-Critic structure both include four fully connected layers connected in sequence, and the activation functions between the first three fully connected layers all use the ReLU function, the last fully connected layer of the Actor network uses the tanh activation function, and the last fully connected layer of the Critic network does not use an activation function.

[0010] Optionally, when performing incremental training on the incremental reinforcement learning model, the parameters of the initial policy network of the incremental reinforcement learning model are updated based on the SAC algorithm, and the replay pool technology is used for training.

[0011] Optionally, after obtaining the initial control strategy of the robotic arm, it also includes: determining model parameters of a preset strategy supplement model according to the task type of the assembly task; obtaining the residual control strategy of the robotic arm according to the initial control strategy of the robotic arm through the strategy supplement model determined by the model parameters; obtaining the robotic arm control strategy according to the initial control strategy of the robotic arm and the residual control strategy of the robotic arm; wherein the post-processing neural network model is obtained by the following training method: obtaining training data of training assembly tasks of several task types and a post-processing neural network model for predicting the residual control strategy; training a preset post-processing neural network model according to the training data of training assembly tasks of several task types and the initial control strategy of the robotic arm, and retaining the model parameters of several task types to obtain a strategy supplement model.

[0012] Optionally, the post-processing neural network model is constructed based on the Actor-Critic structure, and the Actor network and the Critic network of the Actor-Critic structure both include four fully connected layers connected in sequence, and the activation functions between the first three fully connected layers all use ReLU functions, the last fully connected layer of the Actor network uses a tanh activation function, and the last fully connected layer of the Critic network does not use an activation function.

[0013] According to a second aspect of the present invention, a multi-type shaft hole assembly robot control system is provided, comprising: a parameter determination module, used to obtain the task type of the assembly task, and determine the model parameters of a preset control strategy model according to the task type of the assembly task; a strategy prediction module, used to obtain the state data of the robot arm, and determine the control strategy model through the model parameters according to the state data of the robot arm to obtain the initial control strategy of the robot arm; wherein the control strategy model is obtained by the following training method: obtaining an incremental reinforcement learning model for predicting the control strategy of the robot arm; incrementally training the incremental reinforcement learning model in a simulation environment for training assembly tasks of several task types, and retaining the model parameters of several task types to obtain the control strategy model.

[0014] According to a third aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned multi-type shaft hole assembly robot arm control method when executing the computer program.

[0015] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned multi-type shaft hole assembly robot arm control method are implemented.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The control method of the multi-type shaft hole assembly robot arm of the present invention first determines the model parameters of the preset control strategy model according to the task type of the assembly task, and then determines the control strategy model through the model parameters according to the state data of the robot arm to obtain the control strategy of the robot arm to realize the control of the robot arm. Among them, the control strategy model is based on the simulation environment of the training assembly tasks of several task types, and the incremental reinforcement learning model is incrementally trained, and the model parameters of each task type are retained. By designing the model parameters of the control strategy model in the incremental training as the model parameters of each task type, the control strategy model can quickly adapt to the changing environment of the task type, and realize the retention of the skill knowledge of assembly tasks of different task types, reduce the training time and improve the assembly success rate. At the same time, while continuously learning the model parameters of the task type, the model parameters of the old task type can also be kept from being forgotten, ensuring the high adaptability of the control strategy model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of a method for controlling a multi-type shaft hole assembly robot according to an embodiment of the present invention.

[0018] Figure 2 Schematic diagram of the control strategy model training principle of an embodiment of the present invention.

[0019] Figure 3 This is a structural block diagram of a multi-type shaft hole assembly robot control system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] The present invention is further described in detail below in conjunction with the accompanying drawings: See also Figure 1 In one embodiment of the present invention, a method for controlling a multi-type shaft hole assembly robot arm is provided, which can maintain the assembly success rate after continuously learning multiple assembly skills.

[0023] Specifically, the multi-type shaft hole assembly robot arm control method of the present invention comprises the following steps: S1: Obtaining a task type of an assembly task, and determining model parameters of a preset control strategy model according to the task type of the assembly task.

[0024] S2: Acquire state data of the robot arm, and determine the control strategy model through model parameters according to the state data of the robot arm to obtain an initial control strategy of the robot arm.

[0025] Among them, the control strategy model is obtained through the following training method: obtaining an incremental reinforcement learning model for predicting the control strategy of the robotic arm; incrementally training the incremental reinforcement learning model based on a simulation environment of training assembly tasks of several task types, and retaining model parameters of several task types to obtain the control strategy model.

[0026] The control method of the multi-type shaft hole assembly robot arm of the present invention first determines the model parameters of the preset control strategy model according to the task type of the assembly task, and then determines the control strategy model through the model parameters according to the state data of the robot arm to obtain the control strategy of the robot arm to realize the control of the robot arm. Among them, the control strategy model is based on the simulation environment of the training assembly tasks of several task types, and the incremental reinforcement learning model is incrementally trained, and the model parameters of each task type are retained. By designing the model parameters of the control strategy model in the incremental training as the model parameters of each task type, the control strategy model can quickly adapt to the changing environment of the task type, and realize the retention of the skill knowledge of assembly tasks of different task types, reduce the training time and improve the assembly success rate. At the same time, while continuously learning the model parameters of the task type, the model parameters of the old task type can also be kept from being forgotten, ensuring the high adaptability of the control strategy model.

[0027] In the problem of multi-type shaft hole assembly of the present invention, the intelligent agent is a robotic arm, and the assembly objects are mainly single, double and triple shaft hole kits and two-pin and three-pin flat plugs. The incremental reinforcement learning model can quickly adapt to the changing environment by continuously learning new assembly skills. Among them, the state data of the robotic arm includes the force of the end of the robotic arm in the x, y and z directions and the insertion depth of the sleeve, and the control strategy of the robotic arm includes the displacement of the end of the robotic arm in the x, y and z directions.

[0028] In a possible implementation, obtaining the task type of the assembly task includes: obtaining type influence data of the assembly task; and according to the type influence data of the assembly task, determining a model through a preset task type to obtain a task similarity index value of the assembly task, and combining the preset task similarity index values ​​of each task type to obtain the task type of the assembly task.

[0029] The task type determination model is obtained by: obtaining a pre-processing neural network model for predicting task similarity index values; training a pre-processing neural network model based on type influence data of training assembly tasks of several task types to obtain a task type determination model.

[0030] Specifically, when performing assembly control of different assembly tasks, it is necessary to quantify and classify the assembly tasks through the pre-trained pre-processing neural network model to determine the task type of the current assembly task. According to the task similarity index value of the assembly task and the preset task similarity index value of each task type, the task type corresponding to the preset task similarity index value closest to the task similarity index value of the assembly task is used as the task type of the current assembly task.

[0031] The pre-processing neural network model is used to process the prior parameters of the assembly task to be assembled, that is, to obtain the task similarity index value of the assembly task according to the type of assembly task influencing data.

[0032] Optionally, the type influencing data includes one or more of the following: the number of shaft holes to be assembled, the shape of the shaft holes to be assembled, the length of the shaft to be assembled, and the matching type of shaft hole assembly.

[0033] In this implementation, the following types of influence data are used: :

[0034] in, Indicates the number of shaft holes to be assembled, such as single, double and triple shaft holes; Represents the shape of the shaft hole to be assembled, such as round, rectangular, triangular and irregular shapes; Represents the length of the shaft to be assembled; Represents the fit type of shaft-hole assembly, such as interference fit, overfit, and clearance fit.

[0035] Optionally, the pre-processing neural network model includes three fully connected layers connected in sequence, and the activation function of the fully connected layer adopts the ReLU function. The number of hidden layer neuron nodes of the three fully connected layers of the pre-processing neural network model are 50, 30 and 1 respectively.

[0036] The task type determination model is obtained by pre-training the pre-processing neural network model. The task type determination model can classify the assembly tasks to be processed and output the task similarity index value.

[0037] The main steps for pre-training the pre-processing neural network model include: 1. Construct a training data set for the pre-processing neural network model.

[0038] 2. Select the Adam algorithm as the learning algorithm for training, and the loss function Select the following mean squared error loss function: ,in, is the desired task similarity index value, is the task similarity index value actually obtained by the pre-processing network.

[0039] 3. Input the training data set into the pre-processing neural network model for training until the loss function reaches the minimum and the pre-processing neural network model converges, and obtain the task type determination model and test it.

[0040] During the pre-training process of the pre-processing neural network model, the training assembly tasks of similar task types will obtain relatively close task similarity index values. Therefore, the closer the task similarity index values ​​are, the higher the similarity between the assembly tasks is, and they can be considered to be similar assembly tasks.

[0041] In a possible implementation, the step of acquiring the state data of the robotic arm and obtaining an initial control strategy for the robotic arm based on a control strategy model determined by model parameters according to the state data of the robotic arm includes: obtaining a normalization threshold of the assembly task according to a task type of the assembly task, normalizing the state data of the robotic arm according to the normalization threshold, and obtaining an initial control strategy for the robotic arm based on the control strategy model determined by model parameters according to the state data of the robotic arm according to the normalized state data of the robotic arm.

[0042] Specifically, when performing state normalization, the normalization threshold of the assembly task is determined according to the task type of the assembly task, that is, assembly tasks of the same task type can use the same normalization threshold for normalization and denormalization operations, and finally input into the control strategy model for subsequent processing.

[0043] In one possible implementation, the problem of predicting the control strategy of the robotic arm can be modeled as a Markov decision process, where each assembly task can correspond to the distribution of a Markov decision process. At this moment, the system can observe the current state quantity And perform the action accordingly After execution, the state of the system is converted to the next state quantity , and receive corresponding rewards , to perform the next step. Therefore, the assembly task Can be represented by a tuple ( ) indicates that, is the discount factor.

[0044] In this embodiment, the state quantity is used to obtain an incremental reinforcement learning model for predicting the control strategy of the robot arm. Designed for: ,in, , and are the forces at the end of the robot in the x, y and z directions respectively, is the insertion depth of the sleeve.

[0045] Amount of action Designed for: ,in, are the displacements of the end of the robot in the x, y, and z directions respectively.

[0046] Reward Function It is mainly divided into three parts: safety reward, step length efficiency reward and step number efficiency reward. Specifically, the reward function Designed for: ,in, For safe returns, is the step efficiency return, is the step efficiency return, and are all preset discount factors. In this implementation, and Both are 0.5, and can be designed according to different needs.

[0047] Among them, safety returns :

[0048] in, is the radial force of the current assembly, is the set radial force threshold, i.e. the maximum assembly radial force. If the radial force in this round is closer to the radial force threshold, the safety is lower, so the safety return value will also be smaller.

[0049] Step length efficiency return :

[0050] in, is the current insertion step size, A safe return for the previous step. The function is: if the radial contact force of the previous step is larger, the step is guided to be inserted with a smaller step length. On the contrary, if the radial contact force of the previous step is relatively small, that is, the safety factor is higher, the step can be inserted with a larger step length, thereby improving assembly efficiency.

[0051] Step efficiency return :

[0052] in, is the number of assembly steps in the previous round, The maximum number of assembly steps is set. It means that the smaller the number of assembly steps in the previous round, the greater the reward, which improves the assembly efficiency.

[0053] In summary, the reward function As shown below:

[0054] The initial policy network of the incremental reinforcement learning model is designed based on the Actor-Critic structure. The Actor network and the Critic network of the Actor-Critic structure both include four fully connected layers connected in sequence, and the activation functions between the first three fully connected layers all use the ReLU function. The last fully connected layer of the Actor network uses the tanh activation function, and the last fully connected layer of the Critic network does not use an activation function.

[0055] In this implementation, the number of neuron nodes in the four layers of the Actor network of the initial policy network is 4, 300, 100 and 3 respectively, and the number of neuron nodes in the four layers of the Critic network is 7, 300, 100 and 1 respectively. In this implementation, an Actor is designed as the policy network; four Critic are designed to calculate the Q value, which are Q network , And the target Q network , .

[0056] In a possible implementation, when the incremental reinforcement learning model is incrementally trained, the parameters of the initial policy network of the incremental reinforcement learning model are updated based on the SAC algorithm, and the replay pool technology is used for training.

[0057] Specifically, the use of replay pool technology for training can make effective use of past experience. The replay pool technology is as follows: historical data is generated through the interaction between the incremental reinforcement learning model and the environment, stored in the replay pool, and then sampled and input into the incremental reinforcement learning model to update the parameters.

[0058] In a possible implementation manner, after obtaining the initial control strategy of the robotic arm, the method further includes: Determine the model parameters of the preset strategy supplement model according to the task type of the assembly task; obtain the residual control strategy of the robot arm through the strategy supplement model determined by the model parameters according to the initial control strategy of the robot arm; obtain the robot arm control strategy according to the initial control strategy of the robot arm and the residual control strategy of the robot arm; wherein the post-processing neural network model is obtained by the following training method: obtain training data of training assembly tasks of several task types and a post-processing neural network model for predicting the residual control strategy; train the preset post-processing neural network model according to the training data of training assembly tasks of several task types and the initial control strategy of the robot arm, and retain the model parameters of several task types to obtain the strategy supplement model.

[0059] Specifically, the post-processing neural network model is mainly trained to obtain a strategy supplement model, which is used to generate the residual control strategy of the robot arm. The residual control strategy of the robot arm is an effective correction to the initial control strategy of the robot arm. The initial control strategy of the robot arm is supplemented and corrected by the residual control strategy of the robot arm to make the robot arm control strategy closer to the actual assembly process.

[0060] Among them, according to the initial control strategy of the robot arm and the remaining control strategy of the robot arm, the control strategy of the robot arm is obtained, that is, the remaining control strategy of the robot arm is superimposed by the following formula: and the initial control strategy of the robot arm Get the robot control strategy : ;in, Represents the action obtained by the output of the robot control strategy in the current state, Represents the action obtained by the initial control strategy output of the robot arm in the current state, Represents the action obtained by the output of the remaining control strategy of the robot arm in the current state.

[0061] The main training steps of the post-processing neural network model include: 1. According to the task type of the training assembly task obtained by the pre-processing neural network, determine the corresponding post-processing neural network input channel (model parameter). If there is no corresponding channel, add the corresponding channel.

[0062] 2. First, update the Q network by minimizing the following loss function:

[0063] in, Represents Q Network Parameters Replay Pool represent Q value represent The target Q value is k =1,2; Represents the regularization coefficient Represents the initial strategy are the parameters of the policy network.

[0064] 3. Secondly, update the policy network parameters by minimizing the following loss function:

[0065] in, Pick and The minimum value in:

[0066] in, and represent and Q value.

[0067] 4. Again, update the regularization coefficient by minimizing the following loss function :

[0068] in, Represents the target entropy value.

[0069] 5. Finally, update the Q network by exponential moving average:

[0070] in, Target Q Network Parameters, k =1,2; is the exponential moving average coefficient.

[0071] 6. Repeat steps 2 to 5 until the post-processing neural network reaches convergence.

[0072] In one possible implementation, the post-processing neural network model is constructed based on the Actor-Critic structure, and the Actor network and the Critic network of the Actor-Critic structure both include four fully connected layers connected in sequence, and the activation functions between the first three fully connected layers all use the ReLU function, the last fully connected layer of the Actor network uses the tanh activation function, and the last fully connected layer of the Critic network does not use an activation function. In this implementation, the number of neuron nodes in the four layers of the Actor network of the post-processing neural network model is 4, 300, 100 and 3 respectively, and the number of neuron nodes in the four layers of the Critic network is 7, 300, 100 and 1 respectively. In this implementation, an Actor is designed as a policy network; four Critic are designed to calculate Q values, which are Q network , And the target Q network , .

[0073] For details, see Figure 2 , the training process of the control strategy model is as follows: First, the pre-processing neural network model is pre-trained to obtain a task type determination model, and then the task type of the training assembly task is obtained through the task type determination model. Based on the task type of the training assembly task and the state data of the robot generated by the simulation environment, the corresponding normalization threshold is selected for state normalization and action denormalization. Then, the incremental reinforcement learning model is incrementally trained, and the historical data is stored in the replay pool for subsequent learning. The incremental training is continuously performed until it is completed and the reward of the incremental reinforcement learning model reaches a convergence state, and the strategy supplement model is obtained through the post-processing neural network model. Then, the residual control strategy of the robot generated by the strategy supplement model further supplements and corrects the initial control strategy of the robot generated by the control strategy model to obtain the final control strategy. Then, in the real assembly environment, exploratory trial assembly is carried out, and the final output action is further fine-tuned to make the action more in line with the requirements of the real assembly task.

[0074] In a possible implementation, in order to verify the effectiveness of the multi-type shaft hole assembly robot control method of the present invention, the assembly success rate of the skill retention experiment after the incremental reinforcement learning model continuously learned the assembly skills of five types of assembly tasks was tested, and the results are shown in Table 1.

[0075] Table 1

[0076] It can be seen from Table 1 that after learning five skills, the success rates of the control strategy model in assembling single-axis hole kits, double-axis hole kits, three-axis hole kits, two-pin flat plugs and three-pin flat plugs are 90%, 90%, 85%, 80% and 90% respectively. It can be seen that after learning multiple skills, the control strategy model still has not forgotten the old skills, has a good maintenance of assembly safety and efficiency, and the learning effect has not declined significantly.

[0077] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0078] See also Figure 3 In another embodiment of the present invention, a multi-type shaft hole assembly robot arm control system is provided, which can be used to implement the above-mentioned multi-type shaft hole assembly robot arm control method. Specifically, the multi-type shaft hole assembly robot arm control system includes a parameter determination module and a strategy prediction module.

[0079] Among them, the parameter determination module is used to obtain the task type of the assembly task, and determine the model parameters of the preset control strategy model according to the task type of the assembly task; the strategy prediction module is used to obtain the state data of the robotic arm, and determine the control strategy model through the model parameters according to the state data of the robotic arm to obtain the initial control strategy of the robotic arm; wherein, the control strategy model is obtained by the following training method: obtaining an incremental reinforcement learning model for predicting the control strategy of the robotic arm; incrementally training the incremental reinforcement learning model based on a simulation environment for training assembly tasks of several task types, and retaining the model parameters of several task types to obtain the control strategy model.

[0080] All relevant contents of each step involved in the aforementioned embodiment of the multi-type shaft hole assembly robot arm control method can be referred to the functional description of the functional modules corresponding to the multi-type shaft hole assembly robot arm control system in the embodiment of the present invention, and will not be repeated here.

[0081] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0082] In another embodiment of the present invention, a computer device is provided, the computer device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the multi-type shaft hole assembly robot control method.

[0083] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the multi-type shaft hole assembly robot control method in the above embodiment.

[0084] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multi-type shaft hole assembly robot arm control method, characterized in that: include: Obtaining a task type of the assembly task, and determining model parameters of a preset control strategy model according to the task type of the assembly task; Acquire state data of the robot arm, and determine the control strategy model through model parameters according to the state data of the robot arm to obtain an initial control strategy of the robot arm; Among them, the control strategy model is obtained through the following training method: obtaining an incremental reinforcement learning model for predicting the control strategy of the robotic arm; incrementally training the incremental reinforcement learning model based on a simulation environment of training assembly tasks of several task types, and retaining model parameters of several task types to obtain the control strategy model.

2. The multi-type shaft hole assembly robot arm control method according to claim 1 is characterized in that: The task types for obtaining the assembly task include: Acquire type influence data of the assembly task; and determine the model according to the type influence data of the assembly task by using the preset task type to obtain the task similarity index value of the assembly task, and obtain the task type of the assembly task by combining the preset task similarity index values ​​of each task type; The task type determination model is obtained by: obtaining a pre-processing neural network model for predicting task similarity index values; training a pre-processing neural network model based on type influence data of training assembly tasks of several task types to obtain a task type determination model; The step of obtaining the state data of the robot arm and determining the control strategy model through model parameters according to the state data of the robot arm to obtain the initial control strategy of the robot arm includes: A normalization threshold of the assembly task is obtained according to the task type of the assembly task, the state data of the robot arm is normalized according to the normalized state data of the robot arm, and an initial control strategy of the robot arm is obtained according to a control strategy model determined by model parameters based on the normalized state data of the robot arm.

3. The multi-type shaft hole assembly robot arm control method according to claim 2, characterized in that: The type influencing data includes one or more of the following: the number of shaft holes to be assembled, the shape of the shaft holes to be assembled, the shaft length to be assembled, and the matching type of shaft hole assembly; the pre-processing neural network model includes three fully connected layers connected in sequence, and the activation function of the fully connected layer adopts the ReLU function.

4. The multi-type shaft hole assembly robot arm control method according to claim 1, characterized in that: The state quantity of the incremental reinforcement learning model for: ,in, , and are the forces at the end of the robot in the x, y and z directions respectively, is the insertion depth of the sleeve; The action volume of the incremental reinforcement learning model for: ,in, are the displacements of the end of the robot arm in the x, y and z directions respectively; The reward function of the incremental reinforcement learning model for: ,in, For safe returns, is the step efficiency return, is the step efficiency return, and All are preset discount factors; The initial policy network of the incremental reinforcement learning model is based on the Actor-Critic structure. The Actor network and the Critic network of the Actor-Critic structure both include four fully connected layers connected in sequence, and the activation functions between the first three fully connected layers all use the ReLU function, the last fully connected layer of the Actor network uses the tanh activation function, and the last fully connected layer of the Critic network does not use an activation function.

5. The multi-type shaft hole assembly robot arm control method according to claim 1, characterized in that: When the incremental reinforcement learning model is incrementally trained, the parameters of the initial policy network of the incremental reinforcement learning model are updated based on the SAC algorithm, and the replay pool technology is used for training.

6. The multi-type shaft hole assembly robot arm control method according to claim 1, characterized in that: After obtaining the initial control strategy of the robotic arm, the method further includes: Determining model parameters of a preset strategy supplement model according to the task type of the assembly task; According to the initial control strategy of the robot arm, the remaining control strategy of the robot arm is obtained by supplementing the model with the strategy determined by the model parameters; According to the initial control strategy of the robot arm and the remaining control strategy of the robot arm, a control strategy of the robot arm is obtained; Among them, the post-processing neural network model is obtained through the following training method: obtaining training data of training assembly tasks of several task types and a post-processing neural network model for predicting the remaining control strategy; training a preset post-processing neural network model according to the training data of training assembly tasks of several task types and the initial control strategy of the robotic arm, and retaining the model parameters of several task types to obtain a strategy supplement model.

7. The multi-type shaft hole assembly robot arm control method according to claim 6, characterized in that: The post-processing neural network model is constructed based on the Actor-Critic structure. The Actor network and the Critic network of the Actor-Critic structure both include four fully connected layers connected in sequence, and the activation functions between the first three fully connected layers all use the ReLU function, the last fully connected layer of the Actor network uses the tanh activation function, and the last fully connected layer of the Critic network does not use an activation function.

8. A multi-type shaft hole assembly robot arm control system, characterized in that: include: A parameter determination module, used to obtain the task type of the assembly task and determine the model parameters of the preset control strategy model according to the task type of the assembly task; A strategy prediction module is used to obtain the state data of the robot arm and obtain the initial control strategy of the robot arm according to the control strategy model determined by the model parameters based on the state data of the robot arm; Among them, the control strategy model is obtained through the following training method: obtaining an incremental reinforcement learning model for predicting the control strategy of the robotic arm; incrementally training the incremental reinforcement learning model based on a simulation environment of training assembly tasks of several task types, and retaining model parameters of several task types to obtain the control strategy model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multi-type shaft hole assembly robot arm control method as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the multi-type shaft hole assembly robot control method according to any one of claims 1 to 7 are implemented.

Images