Multi-type shaft hole assembly mechanical arm control method and related device

Abstract

The present application belongs to the technical field of automatic assembly, and discloses a multi-type shaft hole assembly mechanical arm control method and related device, which comprises obtaining the task type of an assembly task, and determining the model parameters of a preset control strategy model according to the task type of the assembly task; obtaining the state data of a mechanical arm and determining the initial control strategy of the mechanical arm through the control strategy model determined by the model parameters; wherein the control strategy model is obtained through the following training method: obtaining an incremental reinforcement learning model for predicting the control strategy of the mechanical arm; incrementally training the incremental reinforcement learning model based on the simulation environment of the training assembly task of several task types, and retaining the model parameters of several task types to obtain the control strategy model. The control strategy model can quickly adapt to the environment with constantly changing task types, and can realize the maintenance of the skill knowledge of assembly tasks of different task types, reduce the training time and improve the assembly success rate.

Description

Technical Field

[0001] This invention belongs to the field of automatic assembly technology, and relates to a control method and related device for a multi-type shaft and hole assembly robotic arm. Background Technology

[0002] With the continuous development of industry, robots are playing an increasingly important role. Shaft and hole assembly is a common task in both production and daily life. In practical applications, the objects to be assembled are often diverse. Examples include shaft and gear assembly in industrial production, and plug and USB insertion in everyday life. The assembly skills required by the robotic arm vary depending on the object being assembled.

[0003] Reinforcement learning, a common machine learning method, is well-suited for skill learning in robotic arms. However, traditional reinforcement learning often only addresses single skill learning scenarios and is poorly applicable to the continuous learning of multiple skills. Furthermore, due to the inherent limitations of neural networks, commonly used deep reinforcement learning algorithms often experience some degree of forgetting of older skills after learning new ones.

[0004] Based on the above analysis, current assembly robot control methods based on reinforcement learning models are often only adaptable to a single assembly task. When faced with a new assembly task, the reinforcement learning model needs to be retrained, which takes a long time and makes it difficult to effectively control assembly accuracy. In addition, incremental training for new assembly tasks inevitably reduces the adaptability to old assembly tasks, leading to a decline in assembly control performance. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a control method and related device for assembling multi-type shaft holes.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] In a first aspect, the present invention provides a control method for a multi-type shaft and hole assembly robotic arm, comprising: acquiring the task type of the assembly task, and determining the model parameters of a preset control strategy model based on the task type of the assembly task; acquiring the state data of the robotic arm, and obtaining an initial control strategy for the robotic arm based on the state data of the robotic arm and the control strategy model determined by the model parameters; wherein the control strategy model is obtained through the following training method: acquiring an incremental reinforcement learning model for predicting the control strategy of the robotic arm; incrementally training the incremental reinforcement learning model based on a simulation environment of the assembly task for several task types, and retaining the model parameters of several task types to obtain the control strategy model.

[0008] Optionally, obtaining the task type of the assembly task includes: obtaining the type influence data of the assembly task; and obtaining the task similarity index value of the assembly task through a preset task type determination model based on the type influence data of the assembly task, and obtaining the task type of the assembly task by combining the preset task similarity index value of each task type; wherein, the task type determination model is obtained by: obtaining a preprocessing neural network model for predicting the task similarity index value; training the preprocessing neural network model based on the type influence data of the assembly task for several task types to obtain the task type determination model; obtaining the state data of the robotic arm and obtaining the initial control strategy of the robotic arm based on the state data of the robotic arm through the control strategy model determined by the model parameters includes: obtaining the normalization threshold of the assembly task according to the task type of the assembly task, normalizing the state data of the robotic arm according to the normalization threshold, and obtaining the initial control strategy of the robotic arm based on the normalized state data of the robotic arm through the control strategy model determined by the model parameters.

[0009] Optionally, the type of data influencing the data includes one or more of the following: the number of shaft holes to be assembled, the shape of the shaft holes to be assembled, the length of the shaft to be assembled, and the fit type of the shaft hole assembly; the preprocessing neural network model includes three fully connected layers connected in sequence, and the activation function of the fully connected layers adopts the ReLU function.

[0010] Optionally, the state variables of the incremental reinforcement learning model for: ,in, , and These represent the forces at the end effector of the robotic arm in the x, y, and z directions, respectively. The insertion depth of the bushing; the action amount of the incremental reinforcement learning model. for: ,in, These represent the displacements of the robotic arm's end effector in the x, y, and z directions, respectively; the reward function of the incremental reinforcement learning model. for: ,in, For safety and return, For step size efficiency returns, For step efficiency rewards, and All are preset discount factors; the initial policy network of the incremental reinforcement learning model is based on the Actor-Critic structure. The Actor network and the Critic network of the Actor-Critic structure each include four fully connected layers connected in sequence. The activation function between the first three fully connected layers is the ReLU function. The last fully connected layer of the Actor network uses the tanh activation function, and the last fully connected layer of the Critic network does not use an activation function.

[0011] Optionally, when incrementally training the incremental reinforcement learning model, the parameters of the initial policy network of the incremental reinforcement learning model are updated based on the SAC algorithm, and a replay pooling technique is used for training.

[0012] Optionally, after obtaining the initial control strategy of the robotic arm, the method further includes: determining the model parameters of a preset strategy supplementary model based on the task type of the assembly task; obtaining the remaining control strategy of the robotic arm through the strategy supplementary model determined by the model parameters based on the initial control strategy of the robotic arm; and obtaining the robotic arm control strategy based on the initial control strategy and the remaining control strategy of the robotic arm. The post-processing neural network model is obtained through the following training method: acquiring training data for several task types of training assembly tasks and a post-processing neural network model for predicting the remaining control strategy; training the preset post-processing neural network model based on the training data for several task types of training assembly tasks and the initial control strategy of the robotic arm, and retaining the model parameters for several task types to obtain the strategy supplementary model.

[0013] Optionally, the post-processing neural network model is constructed based on the Actor-Critic structure. Both the Actor network and the Critic network in the Actor-Critic structure include four fully connected layers connected in sequence. The activation function between the first three fully connected layers is the ReLU function. The last fully connected layer of the Actor network uses the tanh activation function, while the last fully connected layer of the Critic network does not use an activation function.

[0014] In a second aspect, the present invention provides a multi-type shaft and hole assembly robotic arm control system, comprising: a parameter determination module, used to acquire the task type of the assembly task and determine the model parameters of a preset control strategy model according to the task type of the assembly task; and a strategy prediction module, used to acquire the state data of the robotic arm and obtain the initial control strategy of the robotic arm according to the state data of the robotic arm and the control strategy model determined by the model parameters; wherein the control strategy model is obtained through the following training method: acquiring an incremental reinforcement learning model for predicting the control strategy of the robotic arm; incrementally training the incremental reinforcement learning model based on a simulation environment of the assembly task with several task types, and retaining the model parameters of several task types to obtain the control strategy model.

[0015] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described multi-type shaft hole assembly robotic arm control method.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described multi-type shaft hole assembly robotic arm control method.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] This invention discloses a multi-type shaft and hole assembly robotic arm control method. First, it determines the model parameters of a preset control strategy model based on the task type of the assembly task. Then, based on the robotic arm's state data, it obtains the control strategy for the robotic arm through the control strategy model determined by the model parameters, thereby controlling the robotic arm. Specifically, the control strategy model is obtained by incrementally training an incremental reinforcement learning model on a simulation environment of several task types of assembly tasks, retaining the model parameters for each task type. By designing the model parameters of the control strategy model in incremental training to be the model parameters for each task type, the control strategy model can quickly adapt to the constantly changing environment of task types and maintain skill knowledge for assembly tasks of different types, reducing training time and improving assembly success rate. Simultaneously, while continuously learning the model parameters for task types, it also ensures that the model parameters for old task types are not forgotten, guaranteeing the high adaptability of the control strategy model. Attached Figure Description

[0019] Figure 1 This is a flowchart of the multi-type shaft hole assembly robotic arm control method according to an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram illustrating the training principle of the control strategy model in an embodiment of the present invention.

[0021] Figure 3 This is a block diagram of the multi-type shaft hole assembly robotic arm control system according to an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] The present invention will now be described in further detail with reference to the accompanying drawings:

[0025] See Figure 1 In one embodiment of the present invention, a control method for a multi-type shaft hole assembly robotic arm is provided, which can maintain the assembly success rate after continuously learning multiple assembly skills.

[0026] Specifically, the multi-type shaft hole assembly robotic arm control method of the present invention includes the following steps:

[0027] S1: Obtain the task type of the assembly task, and determine the model parameters of the preset control strategy model based on the task type of the assembly task.

[0028] S2: Obtain the state data of the robotic arm, and based on the state data of the robotic arm, obtain the initial control strategy of the robotic arm through the control strategy model determined by the model parameters.

[0029] The control strategy model is obtained through the following training method: an incremental reinforcement learning model for predicting the control strategy of the robotic arm is obtained; the incremental reinforcement learning model is incrementally trained based on the simulation environment of the assembly task for several task types, and the model parameters of several task types are retained to obtain the control strategy model.

[0030] This invention discloses a multi-type shaft and hole assembly robotic arm control method. First, it determines the model parameters of a preset control strategy model based on the task type of the assembly task. Then, based on the robotic arm's state data, it obtains the control strategy for the robotic arm through the control strategy model determined by the model parameters, thereby controlling the robotic arm. Specifically, the control strategy model is obtained by incrementally training an incremental reinforcement learning model on a simulation environment of several task types of assembly tasks, retaining the model parameters for each task type. By designing the model parameters of the control strategy model in incremental training to be the model parameters for each task type, the control strategy model can quickly adapt to the constantly changing environment of task types and maintain skill knowledge for assembly tasks of different types, reducing training time and improving assembly success rate. Simultaneously, while continuously learning the model parameters for task types, it also ensures that the model parameters for old task types are not forgotten, guaranteeing the high adaptability of the control strategy model.

[0031] In the multi-type shaft hole assembly problem of this invention, the intelligent agent is a robotic arm, and the assembly objects are mainly single, double, and triple shaft hole kits, as well as two-pin and three-pin flat plugs. The incremental reinforcement learning model can quickly adapt to changing environments by continuously learning new assembly skills. The robotic arm's state data includes the forces at its end effector in the x, y, and z directions, and the insertion depth of the bushing; the robotic arm's control strategy includes the displacement of its end effector in the x, y, and z directions.

[0032] In one possible implementation, obtaining the task type of the assembly task includes: obtaining the type influence data of the assembly task; and obtaining the task similarity index value of the assembly task by using a preset task type determination model based on the type influence data of the assembly task, and obtaining the task type of the assembly task by combining the preset task similarity index value of each task type.

[0033] The task type determination model is obtained by: acquiring a preprocessing neural network model for predicting task similarity index values; training the preprocessing neural network model based on task type influence data for training several task types to obtain the task type determination model.

[0034] Specifically, when controlling assembly for different assembly tasks, a pre-trained pre-processing neural network model is needed to quantify and classify the assembly tasks to determine the task type of the current assembly task. Specifically, based on the task similarity index value of the assembly task and combined with preset task similarity index values ​​for each task type, the task type corresponding to the preset task similarity index value that is closest to the assembly task's task similarity index value is taken as the task type of the current assembly task.

[0035] The preprocessing neural network model is used to process the prior parameters of the task to be assembled, that is, to obtain the task similarity index value of the assembly task based on the influence data of the type of assembly task.

[0036] Optionally, the type-affected data includes one or more of the following: the number of shaft holes to be assembled, the shape of the shaft holes to be assembled, the length of the shaft to be assembled, and the fit type of the shaft hole assembly.

[0037] In this embodiment, the following types of influence data are used. :

[0038]

[0039] in, Indicates the number of shaft holes to be assembled, such as single, double, and triple shaft holes; This represents the shape of the shaft hole to be assembled, such as round, rectangular, triangular, and irregular shapes. Represents the length of the shaft to be assembled; This represents the type of fit in the shaft-hole assembly, such as interference fit, transition fit, and clearance fit.

[0040] Optionally, the preprocessing neural network model includes three fully connected layers connected in sequence, and the activation function of the fully connected layers is the ReLU function. The number of hidden layer neurons in the three fully connected layers of the preprocessing neural network model are 50, 30, and 1, respectively.

[0041] By pre-training the pre-processing neural network model, a task type determination model is obtained. This model can classify the assembly tasks to be processed and output task similarity index values.

[0042] The main steps in pre-training a pre-processing neural network model include:

[0043] 1. Construct the training dataset for the preprocessing neural network model.

[0044] 2. The Adam algorithm was chosen as the training algorithm, and the loss function was... Choose one of the following mean squared error loss functions: ,in, The desired task similarity metric value is... This represents the actual task similarity index value obtained from the preprocessing network.

[0045] 3. Input the training dataset into the preprocessing neural network model for training until the loss function reaches its minimum and the preprocessing neural network model converges, thus obtaining the task type determination model and testing it.

[0046] During the pre-training process of the pre-processing neural network model, training assembly tasks of similar task types will yield relatively close task similarity index values. Therefore, the closer the task similarity index values ​​are, the higher the similarity between the assembly tasks, and thus they can be considered similar assembly tasks.

[0047] In one possible implementation, the step of acquiring the state data of the robotic arm and obtaining the initial control strategy of the robotic arm based on the control strategy model determined by the model parameters according to the state data of the robotic arm includes: obtaining the normalization threshold of the assembly task according to the task type of the assembly task; normalizing the state data of the robotic arm according to the normalization threshold; and obtaining the initial control strategy of the robotic arm based on the normalized state data of the robotic arm and the control strategy model determined by the model parameters according to the state data of the robotic arm.

[0048] Specifically, during state normalization, the normalization threshold for assembly tasks is determined based on the task type of the assembly task. That is, assembly tasks of the same task type can use the same normalization threshold for normalization and denormalization operations, and finally input them into the control strategy model for subsequent processing.

[0049] In one possible implementation, the problem of predicting the control strategy for the robotic arm can be modeled as a Markov decision process for each assembly task. Each of these can correspond to a distribution of a Markov decision process. At any given moment, the system can observe the current state variable. And perform the corresponding amount of action. After execution, the system's state is transformed into the next state variable. and receive corresponding rewards. This is to proceed to the next step. Therefore, the assembly task... It can be composed of a tuple ( ) indicates that, This is the discount factor.

[0050] In this embodiment, regarding the problem of obtaining an incremental reinforcement learning model for predicting the control strategy of the robotic arm, the state variable... Designed as follows: ,in, , and These represent the forces at the end effector of the robotic arm in the x, y, and z directions, respectively. This represents the insertion depth of the bushing.

[0051] Motion volume Designed as follows: ,in, These represent the displacements of the robotic arm's end effector in the x, y, and z directions, respectively.

[0052] Reward function It is mainly divided into three parts: safety reward, step size efficiency reward, and step count efficiency reward. Specifically, the reward function... Designed as follows: ,in, For safety and return, For step size efficiency returns, For step efficiency rewards, and All of these are preset discount factors. In this embodiment, and Both are 0.5, and can be adapted to different needs.

[0053] Among them, security returns :

[0054]

[0055] in, The radial force of the current assembly. This is the set radial force threshold, i.e., the maximum assembly radial force. The closer the radial force in this round is to the radial force threshold, the lower the safety, and therefore the smaller the safety reward value will be.

[0056] Step size efficiency return :

[0057]

[0058] in, The current insertion step size, This is a safety reward for the previous step. The purpose of this step is to guide the insertion of the next step with a smaller step size if the radial contact force of the previous step is greater. Conversely, if the radial contact force of the previous step is smaller, i.e., the safety factor is higher, the next step can be inserted with a larger step size, thereby improving assembly efficiency.

[0059] Step efficiency reward :

[0060]

[0061] in, This represents the number of assembly steps in the previous round. This is the maximum number of assembly steps set. This means that the smaller the number of assembly steps in the previous round, the greater the reward, thus improving assembly efficiency.

[0062] In summary, the reward function As shown below:

[0063]

[0064] The initial policy network of the incremental reinforcement learning model is designed based on the Actor-Critic structure. Both the Actor network and the Critic network in the Actor-Critic structure contain four fully connected layers connected in sequence. The activation function between the first three fully connected layers is the ReLU function. The last fully connected layer of the Actor network uses the tanh activation function, while the last fully connected layer of the Critic network does not use an activation function.

[0065] In this embodiment, the initial policy network has four layers of neurons with 4, 300, 100, and 3 nodes in the Actor network, and four layers of neurons with 7, 300, 100, and 1 node in the Critic network. In this embodiment, one Actor network is designed as the policy network; four Critic networks are designed to calculate the Q-value, forming the Q-network. , and the target Q network , .

[0066] In one possible implementation, when incrementally training the incremental reinforcement learning model, the parameters of the initial policy network of the incremental reinforcement learning model are updated based on the SAC algorithm, and the replay pooling technique is used for training.

[0067] Specifically, training with a replay pool technique allows past experience to be effectively utilized. The replay pool technique involves generating historical data through the interaction between the incremental reinforcement learning model and the environment, storing it in the replay pool, and then sampling it before inputting it into the incremental reinforcement learning model to update the parameters.

[0068] In one possible implementation, after obtaining the initial control strategy for the robotic arm, the method further includes:

[0069] The model parameters of the preset strategy supplement model are determined according to the task type of the assembly task; the residual control strategy of the robotic arm is obtained through the strategy supplement model determined by the model parameters according to the initial control strategy of the robotic arm; the control strategy of the robotic arm is obtained according to the initial control strategy and the residual control strategy of the robotic arm; the post-processing neural network model is obtained through the following training method: training data of several task types of training assembly tasks and post-processing neural network models for predicting the residual control strategy are obtained; the preset post-processing neural network model is trained according to the training data of several task types of training assembly tasks and the initial control strategy of the robotic arm, and the model parameters of several task types are retained to obtain the strategy supplement model.

[0070] Specifically, the post-processing neural network model mainly obtains a policy supplement model through training. The policy supplement model is used to generate the residual control strategy of the robotic arm. The residual control strategy of the robotic arm is an effective correction to the initial control strategy of the robotic arm. The initial control strategy of the robotic arm is supplemented and corrected by the residual control strategy of the robotic arm so that the control strategy of the robotic arm is closer to the actual assembly process.

[0071] The robotic arm control strategy is derived from its initial control strategy and its residual control strategy, which is achieved by superimposing the residual control strategy of the robotic arm using the following formula. Initial control strategy for robotic arms Obtain the control strategy for the robotic arm : ;in, This represents the action output by the robotic arm's control strategy in the current state. This represents the action output by the robotic arm's initial control strategy in the current state. This represents the action output by the remaining control strategy of the robotic arm in the current state.

[0072] The main training steps for post-processing neural network models include:

[0073] 1. Based on the task type of the training assembly task obtained from the preprocessing neural network, determine the corresponding input channel (model parameter) of the postprocessing neural network. If there is no corresponding channel, add the corresponding channel.

[0074] 2. First, update the Q-network by minimizing the following loss function:

[0075]

[0076] in, Representing Q Network parameters Replay pool represent Q value represent The target Q value, k =1,2; Represents the regularization coefficient Represents the initial strategy These are the parameters of the policy network.

[0077] 3. Next, update the policy network parameters by minimizing the following loss function:

[0078]

[0079] in, Pick and Minimum value in:

[0080]

[0081] in, and represent and The Q value.

[0082] 4. Next, update the regularization coefficients by minimizing the following loss function. :

[0083]

[0084] in, This represents the target entropy value.

[0085] 5. Finally, update the Q network using an exponential moving average:

[0086]

[0087] in, Target Q network The parameters, k =1,2; It is the coefficient of the exponential moving average.

[0088] 6. Repeat steps 2-5 until the post-processing neural network converges.

[0089] In one possible implementation, the post-processing neural network model is constructed based on an Actor-Critic structure. Both the Actor and Critic networks in the Actor-Critic structure include four sequentially connected fully connected layers. The activation function between the first three fully connected layers is the ReLU function, the last fully connected layer of the Actor network uses the tanh activation function, and the last fully connected layer of the Critic network does not use an activation function. In this implementation, the number of neurons in the four layers of the Actor network in the post-processing neural network model are 4, 300, 100, and 3, respectively, and the number of neurons in the four layers of the Critic network are 7, 300, 100, and 1, respectively. In this implementation, one Actor is designed as the policy network; four Critic networks are designed to calculate the Q-value, forming the Q-network. , and the target Q network , .

[0090] For details, see Figure 2 The training process of the control policy model is as follows:

[0091] First, a pre-trained pre-processing neural network model is used to obtain a task type determination model. Then, the task type for the training assembly task is determined using this model. Based on the task type and the state data of the robotic arm generated from the simulation environment, corresponding normalization thresholds are selected for state normalization and action de-normalization. Next, an incremental reinforcement learning model is incrementally trained, with historical data stored in a replay pool for subsequent learning. This incremental training continues until completion and the incremental reinforcement learning model's reward reaches convergence. A policy supplement model is then obtained through a post-processing neural network model. The remaining control strategy of the robotic arm generated by the policy supplement model further supplements and corrects the initial control strategy generated by the control strategy model, resulting in the final control strategy. Finally, exploratory trial assembly is performed in a real assembly environment, and the final output actions are further fine-tuned to better match the requirements of the real assembly task.

[0092] In one possible implementation, to verify the effectiveness of the multi-type shaft hole assembly robotic arm control method of the present invention, the assembly success rate of the incremental reinforcement learning model in the skill retention experiment after continuously learning the assembly skills of five types of assembly tasks was tested. The results are shown in Table 1.

[0093] Table 1

[0094]

[0095] As shown in Table 1, after learning five skills, the success rates of the control strategy model in assembling single-axis hole kits, dual-axis hole kits, tri-axis hole kits, two-pin flat plugs, and three-pin flat plugs were 90%, 90%, 85%, 80%, and 90%, respectively. This indicates that after learning multiple skills, the control strategy model did not forget the old skills and maintained good assembly safety and efficiency, with no significant decline in learning effectiveness.

[0096] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0097] See Figure 3 In another embodiment of the present invention, a multi-type shaft hole assembly robotic arm control system is provided, which can be used to implement the above-mentioned multi-type shaft hole assembly robotic arm control method. Specifically, the multi-type shaft hole assembly robotic arm control system includes a parameter determination module and a strategy prediction module.

[0098] The parameter determination module is used to obtain the task type of the assembly task and determine the model parameters of the preset control strategy model based on the task type. The strategy prediction module is used to obtain the state data of the robotic arm and obtain the initial control strategy of the robotic arm based on the state data and the control strategy model determined by the model parameters. The control strategy model is obtained through the following training method: obtaining an incremental reinforcement learning model for predicting the control strategy of the robotic arm; incrementally training the incremental reinforcement learning model based on the simulation environment of the assembly task with several task types, and retaining the model parameters of several task types to obtain the control strategy model.

[0099] All relevant content of each step involved in the aforementioned embodiments of the multi-type shaft hole assembly robotic arm control method can be referenced to the functional description of the corresponding functional module of the multi-type shaft hole assembly robotic arm control system in the embodiments of the present invention, and will not be repeated here.

[0100] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0101] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a multi-type shaft-hole assembly robotic arm control method.

[0102] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the multi-type shaft hole assembly robotic arm control method in the above embodiments.

[0103] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0105] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0106] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multi-type shaft-hole assembly robot control method, characterized by, The method comprises the following steps: obtaining the task type of the assembly task, and determining the model parameters of the preset control strategy model according to the task type of the assembly task; obtaining the state data of the mechanical arm, and obtaining the initial control strategy of the mechanical arm through the control strategy model determined by the model parameters according to the state data of the mechanical arm; wherein the control strategy model is obtained through the following training method: obtaining an incremental reinforcement learning model for predicting the control strategy of the mechanical arm; incrementally training the incremental reinforcement learning model based on the simulation environment of the training assembly task of the plurality of task types, and retaining the model parameters of the plurality of task types to obtain the control strategy model; Specifically, the training process of the control strategy model is as follows: first, pre-training the preprocessing neural network model to obtain a task type determination model, then obtaining the task type of the training assembly task through the task type determination model, selecting the corresponding normalization threshold for state normalization and action denormalization based on the task type of the training assembly task and the state data of the mechanical arm generated through the simulation environment, then incrementally training the incremental reinforcement learning model, and storing the historical data into a replay pool for subsequent learning, continuously performing incremental training until completion and the return of the incremental reinforcement learning model reaches a convergence state, and obtaining a strategy supplement model through the post-processing neural network model, and further supplementing and correcting the initial control strategy of the mechanical arm generated through the control strategy model based on the residual control strategy of the mechanical arm generated by the strategy supplement model to obtain the final control strategy; Then, exploratory and trial assembly is performed in a real assembly environment, and the final output action is further fine-tuned to make the action more in line with the requirements of the real assembly task.

2. The multi-type shaft-hole assembly robot control method according to claim 1, wherein The task type of the assembly task is obtained by: obtaining type influence data of the assembly task; and obtaining a task similarity index value of the assembly task through a preset task type determination model based on the type influence data of the assembly task, and combining the task similarity index value preset for each task type to obtain the task type of the assembly task; wherein the task type determination model is obtained through the following method: obtaining a preprocessing neural network model for predicting the task similarity index value; training the preprocessing neural network model based on the type influence data of the training assembly task of the plurality of task types to obtain the task type determination model; The state data of the mechanical arm is obtained, and the initial control strategy of the mechanical arm is obtained through the control strategy model determined by the model parameters according to the state data of the mechanical arm, which comprises: obtaining the normalization threshold of the assembly task according to the task type of the assembly task, normalizing the state data of the mechanical arm according to the normalization threshold, and obtaining the initial control strategy of the mechanical arm through the control strategy model determined by the model parameters according to the state data of the mechanical arm according to the normalized state data of the mechanical arm.

3. The multi-type shaft-hole assembly robot control method according to claim 2, wherein The type influence data comprises one or more of the following: the number of shaft holes to be assembled, the shape of the shaft hole to be assembled, the length of the shaft to be assembled, and the fitting type of the shaft hole assembly; the preprocessing neural network model comprises three fully connected layers connected in sequence, and the activation function of the fully connected layer adopts the ReLU function.

4. The multi-type shaft-hole assembly robot control method of claim 1, wherein State quantity of the incremental reinforcement learning model For: Wherein, , And F, F and F are the forces of the end of the mechanical arm in the x, y and z directions respectively, Is the insertion depth of the shaft sleeve; action amount of the incremental reinforcement learning model is: wherein, respectively, are displacement amounts of the end of the robot arm in the x, y, and z directions. a reward function of the incremental reinforcement learning model is: wherein, is a safety reward, is a step efficiency reward, is a step number efficiency reward, and are preset discount factors; The initial policy network of the incremental reinforcement learning model is based on an Actor-Critic structure, the internal part of the Actor network and the Critic network of the Actor-Critic structure each comprises four fully connected layers connected in sequence, the activation function between the first three fully connected layers is a ReLU function, the last fully connected layer of the Actor network adopts a tanh activation function, and the last fully connected layer of the Critic network does not use an activation function.

5. The multi-type shaft-hole assembly robot control method of claim 1, wherein When the incremental reinforcement learning model is incrementally trained, the parameters of the initial policy network of the incremental reinforcement learning model are updated based on the SAC algorithm, and the training is performed by using the replay pool technology.

6. The multi-type shaft-hole assembly robot control method of claim 1, wherein After the initial control strategy of the mechanical arm is obtained, the method further comprises: determining the model parameters of the preset policy supplement model according to the task type of the assembly task; obtaining the residual control strategy of the mechanical arm by the policy supplement model determined by the model parameters according to the initial control strategy of the mechanical arm; obtaining the control strategy of the mechanical arm according to the initial control strategy of the mechanical arm and the residual control strategy of the mechanical arm; The post-processing neural network model is obtained by the following training method: obtaining training data of training assembly tasks of a plurality of task types and a post-processing neural network model for predicting a residual control strategy; training the preset post-processing neural network model according to the training data of the training assembly tasks of the plurality of task types and the initial control strategy of the mechanical arm, and retaining model parameters of the plurality of task types to obtain a policy supplement model.

7. The multi-type shaft-hole assembly robot control method according to claim 6, wherein The post-processing neural network model is based on an Actor-Critic structure, the internal part of the Actor network and the Critic network of the Actor-Critic structure each comprises four fully connected layers connected in sequence, the activation function between the first three fully connected layers is a ReLU function, the last fully connected layer of the Actor network adopts a tanh activation function, and the last fully connected layer of the Critic network does not use an activation function.

8. A multi-type shaft hole assembly robot control system characterized by, Comprise: The parameter determination module is configured to obtain the task type of the assembly task, and determine the model parameters of the preset control strategy model according to the task type of the assembly task. The strategy prediction module is configured to obtain the state data of the mechanical arm, and obtain the initial control strategy of the mechanical arm by the control strategy model determined by the model parameters according to the state data of the mechanical arm. The control strategy model is obtained by the following training method: obtaining an incremental reinforcement learning model for predicting the control strategy of the mechanical arm; incrementally training the incremental reinforcement learning model based on a simulation environment of training assembly tasks of a plurality of task types, and retaining model parameters of the plurality of task types to obtain the control strategy model; Specifically, the training process of the control strategy model is as follows: first, the pre-processing neural network model is pre-trained to obtain a task type determination model, then the task type of the training assembly task is obtained through the task type determination model, based on the task type of the training assembly task and the state data of the robot arm generated through the simulation environment, a corresponding normalization threshold is selected for state normalization and action denormalization, then the incremental reinforcement learning model is incrementally trained, and the historical data is stored in the replay pool for subsequent learning, the incremental training is continuously performed until it is completed and the return of the incremental reinforcement learning model reaches a convergence state, and the post-processing neural network model is used to obtain a strategy supplement model, and then the remaining control strategy of the robot arm generated based on the strategy supplement model is used to further supplement and correct the initial control strategy of the robot arm generated through the control strategy model, to obtain the final control strategy. Then, exploratory and trial assembly is performed in a real assembly environment, and the final output action is further fine-tuned to make the action more meet the requirements of the real assembly task.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the multi-type shaft hole assembly robot arm control method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the multi-type shaft hole assembly robot arm control method according to any one of claims 1 to 7.

Images