Assembly model training and assembling method, device and system

Abstract

The embodiment of the invention belongs to the technical field of robots, and relates to an assembly model training method, which comprises the following steps: acquiring a tactile signal training sample; taking the relative pose of the assembly body and the to-be-assembled body as a label of a tactile signal training sample; and training the assembly model based on the tactile signal training sample to obtain a trained assembly model. The invention further provides an assembly method, a related device, a related system and the like. According to the technical scheme, the assembly control precision can be improved.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular, to an assembly model training, assembly method, device, and system. Background Art

[0002] For existing automated assembly tasks performed by robots, such as pin insertion assembly, mainly vision combined with pure position control, or control combined with 6D force / tactile sensor information at the end of the robot is adopted. There are the following main defects and deficiencies:

[0003] 1. Vision is prone to occlusion or low vision positioning accuracy, resulting in low control accuracy.

[0004] 2. Taking the pin insertion assembly task as an example, the robot faces the following problems during the assembly process: First, the pose of the pin cannot be directly obtained through the end effector of the robot or the 6D force / tactile sensor. Second, during the assembly process, the information provided by the 6D force / tactile sensor at the end of the robot is limited and difficult to provide detailed information about the state of the pin relative to the hole during assembly. In addition, usually, the 6D force / tactile sensor is installed between the end effector and the end of the robotic arm, and it is impossible to accurately obtain the contact force and direction between the pin and the hole. Summary of the Invention

[0005] The purpose of the embodiments of this application is to propose an assembly model training, assembly method, device, and system to improve the accuracy of assembly control.

[0006] In a first aspect, the embodiments of this application provide an assembly model training method, including the following technical solutions:

[0007] An assembly model training method, the method includes the following steps:

[0008] Obtain tactile signal training samples; use the relative pose of the assembled body and the body to be assembled as the label of the tactile signal training samples;

[0009] Train an assembly model based on the tactile signal training samples to obtain a trained assembly model.

[0010] Further, in one embodiment, before obtaining the tactile signal training samples, the following method steps are included:

[0011] Obtain the pose information of the body to be assembled;

[0012] Generate an assembly instruction based on the pose information of the body to be assembled, so as to instruct the robot to drive the assembled body to repeatedly attempt to assemble with the body to be assembled through the assembly instruction;

[0013] Obtain the tactile signal during each attempted assembly, and correspondingly obtain the relative pose of the assembled body and the body to be assembled;

[0014] Save the tactile signal as the tactile signal training sample, and the corresponding relative pose as the label.

[0015] Further, in one embodiment, the obtaining of the relative pose of the assembled body and the body to be assembled includes the following steps:

[0016] Obtain an image of the assembled body;

[0017] Identify the pose information of the assembled body based on the image of the assembled body;

[0018] Based on the pose information of the body to be assembled and the pose information of the assembled body, obtain the relative pose of the assembled body and the body to be assembled.

[0019] Further, in one embodiment, training the assembly model based on the tactile signal training sample includes the following steps:

[0020] Use the tactile signal training sample as the input of the assembly model, and output the predicted relative pose;

[0021] Calculate the loss between the relative pose and the label, and update the parameters of the assembly model based on the loss;

[0022] Repeat the above steps until the preset termination condition is met.

[0023] In a second aspect, there is provided an assembly method for an assembly model obtained by the above-mentioned assembly model training method, and the method includes the following steps:

[0024] Obtain an image of the body to be assembled;

[0025] Obtain the initial pose of the body to be assembled based on the image of the body to be assembled;

[0026] Generate an initial assembly instruction based on the initial pose to instruct the robot to drive the assembled body to attempt an assembly operation;

[0027] Obtain the tactile information under the current assembly operation;

[0028] Use the tactile information as the input of the assembly model, and output the current relative pose of the assembled body and the body to be assembled;

[0029] Based on the current relative pose, iteratively execute the current assembly operation until the assembly is completed.

[0030] In a third aspect, there is provided an assembly model training device, and the device includes:

[0031] A sample acquisition module, configured to acquire tactile signal training samples; using the relative pose between the assembly and the to-be-assembled body as the label of the tactile signal training samples.

[0032] A model training module, configured to train an assembly model based on the tactile signal training samples to obtain a trained assembly model.

[0033] In a fourth aspect, there is provided an assembly device based on the assembly model obtained from the above-mentioned assembly model training device, characterized in that the device includes:

[0034] An image acquisition module, configured to acquire an image of the to-be-assembled body;

[0035] A pose preliminary determination module, configured to obtain the preliminary pose of the to-be-assembled body based on the image of the to-be-assembled body;

[0036] An instruction generation module, configured to generate a preliminary assembly instruction based on the preliminary pose to instruct the robot to drive the assembly body to attempt an assembly operation;

[0037] A tactile acquisition module, configured to acquire tactile information under the current assembly operation;

[0038] A pose generation module, configured to use the tactile information as the input of the assembly model and output the current relative pose between the assembly body and the to-be-assembled body;

[0039] An assembly execution module, based on the current relative pose, iteratively executes the current assembly operation until the assembly is completed.

[0040] In a fifth aspect, there is provided an assembly system, the system includes: a robot, a force / tactile sensor and a controller; the robot includes: a robotic arm and an end effector provided at the grasping execution end of the robotic arm; the force / tactile sensor is provided at the end effector;

[0041] The controller is respectively communicatively connected to the robot and the force / tactile sensor;

[0042] The controller is configured to implement the steps of the above-mentioned assembly model training method and / or the above-mentioned assembly method.

[0043] In a sixth aspect, there is provided a controller, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, it implements the steps of the above-mentioned assembly model training method and / or the above-mentioned assembly method.

[0044] In a seventh aspect, there is provided a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-described assembly model training method and / or the above-described assembly method are implemented.

[0045] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects:

[0046] In the embodiments of the present application, by obtaining tactile signal training samples; using the relative pose of the assembled body and the body to be assembled as the label of the tactile signal training samples, and training the assembly model based on the tactile signal training samples, by setting a force / tactile sensor at the end effector, it is possible to more accurately sense the change in force between the assembled body and the body to be assembled, and at the same time, it is also possible to more accurately obtain the contact point force, so as to more accurately estimate the relative pose of the assembled body and the body to be assembled, and improve the accuracy and success rate of assembly control. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the solutions in the present application, the following will briefly introduce the drawings required for the description of the embodiments of the present application. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0048] Figure 1 is a system architecture diagram of an embodiment of the application system of the present application;

[0049] Figure 2 is a flowchart of an embodiment of the assembly model training method of the present application;

[0050] Figure 3 is a flowchart of an embodiment of the assembly method of the present application;

[0051] Figure 4 is a structural block diagram of an embodiment of the assembly model training device of the present application;

[0052] Figure 5 is a structural block diagram of an embodiment of the assembly device of the present application;

[0053] Figure 6 is a structural diagram of an embodiment of the computer device of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects and not to describe a specific order.

[0055] Reference to "embodiment" herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0056] To enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0057] As Figure 1 shown, Figure 1 is a system architecture diagram of an embodiment of the application system of this application.

[0058] In one embodiment, this application provides an assembly system, which includes: a robot 110, a force / tactile sensor 120, an image sensor 130, and a controller 140.

[0059] Robot

[0060] Specifically, the robot 110 can be but is not limited to: a humanoid robot (the arms of a humanoid robot can be regarded as robotic arms); or robotic arms connected in series or in parallel (such as: a Delta robot, a four-axis robot, or a six-axis robot).

[0061] The robot includes: a robotic arm and an end effector 111 disposed at the grasping execution end of the robotic arm (such as: a dexterous hand or a gripper); the force / tactile sensor 120 is disposed on the end effector 111.

[0062] Exemplarily, taking the robot as a humanoid robot as an example, the ends of the two robotic arms of the robot are equipped with dexterous hands, and force / tactile sensors are distributed on each finger and palm of the dexterous hands; or as Figure 1As shown, a gripper 111 is provided at the end of the robotic arm of the robot 110, and force / tactile sensors are distributed on the contact surface between the gripper 111 and the object.

[0063] Force / Tactile Sensor

[0064] Among them, the above-mentioned force / tactile sensors refer to: force sensors and / or tactile sensors.

[0065] Specifically, the force sensor can be, but is not limited to, a two-dimensional or multi-dimensional force sensor for measuring two-dimensional or multi-dimensional force data.

[0066] Specifically, the tactile sensor is a type of sensing device that can be placed at the end of actuators such as robotic arms and / or robots, etc., and is used to measure tactile information on the premise of cooperating with the end effector to achieve the grasping function.

[0067] To grasp objects of different shapes and softness, the contact surface between the tactile sensor and the object is usually flexible and has good resilience. The tactile information includes, but is not limited to: array multi-dimensional force information, surface deformation information, temperature information, texture information, etc.

[0068] The implementation of the tactile sensor includes a flexible contact surface, a sensing circuit, a computing device, and a tactile signal parsing algorithm.

[0069] It should be noted that in the embodiments of the present application, the signals measured and output by the force / tactile sensors are collectively referred to as tactile signals. In addition, for the convenience of understanding, the embodiments of the present application mainly take the tactile sensor as an example for description below.

[0070] Specifically, the force / tactile sensors are pre-calibrated, so that the pose transformation relationship between the force / tactile sensors and the end effector can be obtained.

[0071] Image Sensor

[0072] An image sensor, which is used to collect various image data.

[0073] It should be noted that the image sensor in the embodiments of the present application can be a monocular or multi-camera (such as: RGB camera, depth camera, or point cloud camera, etc.), a camera, or a device such as a mobile phone, tablet computer, notebook computer, desktop computer, etc. that includes a camera and a camera.

[0074] The image sensor can be set at the end of the robotic arm or at any required position outside the robotic arm.

[0075] Specifically, the image sensor and the robotic arm are pre-calibrated through hand-eye calibration, so that the pose transformation relationship between the image sensor and the robotic arm can be obtained, and further the pose transformation relationship between the image sensor and the force / tactile sensors can be obtained.

[0076] Controller

[0077] The controller 140 is communicatively connected to the robot 110, the force / tactile sensor 120, and the image sensor 130 respectively in a wired or wireless manner. For the definition of the controller, refer to the description of the assembly model training and / or the assembly method in the following embodiments.

[0078] It should be noted that the above wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other currently known or future-developed wireless connection methods.

[0079] The controller in the embodiment of the present application may be, but is not limited to: a computer terminal (Personal Computer, PC); an industrial control computer terminal (Industrial Personal Computer, IPC); a mobile terminal; a server; a system including a terminal and a server, and implemented through the interaction between the terminal and the server; a programmable logic controller (Programmable Logic Controller, PLC); a field programmable gate array (Field-Programmable Gate Array, FPGA); a digital signal processor (Digital Signal Processer, DSP) or a microcontroller unit (Microcontroller unit, MCU). The controller generates program instructions according to a pre-fixed program in combination with the data output by the robot, the force / tactile sensor, and / or the image sensor, etc. Exemplarily, it can be applied to a computer device as Figure 6 shown.

[0080] The controller 140 described in the embodiment of the present application may be an independent controller, or may be fully or partially integrated in the robot, the force / tactile sensor, and / or the image sensor, etc., and the present application does not make any limitations.

[0081] It should be noted that the assembly model training method provided in the embodiment of the present application is generally executed by the controller 140 of the assembly system described in the above embodiments. Correspondingly, the training device for the assembly model is generally arranged in the controller 140 of the assembly system. Figure 1 As

[0082] shown Figure 2 in Figure 2It is a schematic flowchart of an embodiment of the assembly model training method of the present application. An embodiment of the present application provides an assembly model training method, which may include the following method steps:

[0083] Step 210: Obtain tactile signal training samples; use the relative pose of the assembled body and the body to be assembled as the label of the tactile signal training samples.

[0084] It should be noted that the assembled body and the body to be assembled described in the embodiments of the present application can be any corresponding objects as needed. For example, in the peg in hole task, the assembled body is a peg, and the body to be assembled is a hole. Or in the plug and unplug task, the assembled body is a plug, and the body to be assembled is a socket. For ease of understanding, the following mainly takes the assembled body as a peg and the body to be assembled as a hole as an example for illustration.

[0085] In one embodiment, before the above step 210 of obtaining tactile signal training samples, the following method steps may be included to generate tactile signal training samples.

[0086] Step 230: Obtain the pose information of the body to be assembled.

[0087] In one embodiment, the controller can obtain the pre-generated and accurate pose information of the hole from the memory or the server according to the preset address. We can first let the robotic arm hold the peg and insert it into the hole, so that the pose of the hole relative to the robotic arm can be obtained through the forward kinematics of the robotic arm. Thereafter, the position of the hole is fixed and the pose information is stored, so that the controller can directly obtain the accurate pose information of the hole.

[0088] Step 240: Generate an assembly instruction based on the pose information of the body to be assembled, so as to instruct the robot to drive the assembled body to attempt to assemble with the body to be assembled multiple times through the assembly instruction.

[0089] In one embodiment, the controller controls the robotic arm to grip a peg, uses the pose of the accurate hole or a point near it (such as a point directly above the point) as the target pose, and continuously generates motion instructions based on the target pose to instruct the robot to drive the peg to attempt to insert into the hole multiple times.

[0090] Step 250: Obtain the tactile signal during each attempt to assemble, and calculate the relative pose of the assembled body and the body to be assembled correspondingly.

[0091] Based on the above embodiments, during the continuous attempt of assembly, the tactile signal at the current moment and the pose of the peg relative to the hole are recorded simultaneously (that is, each time an attempt is made to assemble, the tactile signal output by the force / tactile sensor is synchronously acquired, and the relative pose of the peg to the hole to be assembled is obtained by synchronously calculating the joint movement amount and other information based on the output of the robot's encoder), and this relative pose is used as the ground truth. After the data collection is completed, a set of tactile signals and the corresponding ground truth poses of the peg relative to the hole are obtained.

[0092] In the embodiments of the present application, during each execution of the assembly attempt, each time the peg and the hole cause a change in the measurement data of the force / tactile sensor, the multi-dimensional tactile data measured by the force / tactile sensor is acquired. This tactile data may be a certain value or multi-dimensional tactile data with time series.

[0093] Step 260 saves the tactile signal as a tactile signal training sample, and the corresponding relative pose as a label.

[0094] In one embodiment, the controller saves the tactile signal training sample and the relative pose label of the assembled body and the body to be assembled at the corresponding preset address.

[0095] Through the above method steps, the embodiments of the present application obtain the tactile signal training sample and the corresponding relative pose label based on the force / tactile sensor provided at the end effector of the robot. Compared with methods such as setting a 6D force / tactile sensor at the end of the robotic arm, this method can more accurately estimate the relative pose of the assembled body and the body to be assembled, and improve the accuracy and success rate of assembly control.

[0096] In one embodiment, in step 250, "obtaining the relative pose of the assembled body and the body to be assembled" may include the following method steps:

[0097] Step 251 acquires an image of the assembled body.

[0098] In one embodiment, during each execution of the assembly attempt, the controller can acquire the image of the assembled body collected by the image sensor from the memory or the server according to the preset address.

[0099] Step 252 identifies the pose information of the assembled body based on the image of the assembled body.

[0100] In one embodiment, the controller can identify the pose information of the assembled body based on the preset pose recognition model or traditional image processing methods and other methods based on the image of the assembled body.

[0101] Step 253 calculates the relative pose between the assembled body and the body to be assembled based on the pose information of the body to be assembled and the pose information of the assembled body.

[0102] In one embodiment, the controller can calculate the relative pose between the assembled body and the body to be assembled based on the pose information of the body to be assembled and the pose information of the assembled body.

[0103] It should be noted that in addition to the above method steps, any existing or future-developed method steps for generating training samples can also be used as needed.

[0104] Step 220 trains the assembly model based on the haptic signal training samples to obtain the trained assembly model.

[0105] Specifically, the above assembly model can be various existing or future-developed models capable of recognizing the relative pose of assembly, such as: Feed-Forward Networks, RNN, LSTM, Transformer, GNN, GAN, AE, Convolutional Neural Network (CNN). Common CNN models can include but are not limited to: LeNet, AlexNet, ZFNet, VGG, GoogLeNet, Residual Net, DenseNet, R-CNN, SPP-NET, Fast-RCNN, Faster-RCNN, FCN, Mask-RCNN, YOLO, SSD, GCN, GMM, GP.

[0106] In one embodiment, step 220 may include the following method steps:

[0107] Step 221 uses the haptic signal training samples as the input of the assembly model and outputs the predicted relative pose;

[0108] Step 222 calculates the loss between the relative pose and the label, and updates the parameters of the assembly model based on the loss;

[0109] Step 223 repeats the above steps until the preset termination condition is met (such as: meeting the preset number of iterations or the difference meeting the preset threshold).

[0110] In one embodiment, based on the previous embodiments, during the process of training the network model, the method of supervised learning can be adopted. The haptic signal is used as the input of the assembly model, and the assembly model will output the predicted relative pose. Then, the loss between this relative pose and the ground truth is calculated, and the network parameters are iteratively updated using this loss.

[0111] In the embodiment of the present application, a tactile signal training sample is obtained; the relative pose of the assembly and the object to be assembled is used as the label of the tactile signal training sample, and an assembly model is trained based on the tactile signal training sample. By setting a force / tactile sensor at the end effector, the change in force between the assembly and the object to be assembled can be felt more precisely, and at the same time, the magnitude and direction of the contact point force can be obtained more precisely. Compared with methods such as setting a 6D force / tactile sensor at the end of the robotic arm, this method can more accurately estimate the relative pose of the assembly and the object to be assembled, improving the accuracy and success rate of assembly control.

[0112] As Figure 3 shown, Figure 3 is a schematic flow chart of an embodiment of the assembly method of the present application.

[0113] Based on the assembly model obtained by the assembly training method described in the above embodiment, the embodiment of the present application further provides an assembly method, which may include the following method steps:

[0114] It should be noted that the assembly method provided by the embodiment of the present application is generally executed by the controller 140 of the system described in the above embodiment Figure 1 correspondingly, the device for assembly is generally set in the controller 140.

[0115] Step 310: Obtain an image of the object to be assembled.

[0116] In one embodiment, the controller may obtain the image of the object to be assembled collected by the image sensor from the memory or the server according to a preset address.

[0117] Step 320: Calculate the initial pose of the object to be assembled based on the image of the object to be assembled.

[0118] In one embodiment, the controller may identify the initial pose of the object to be assembled based on the image of the object to be assembled based on a preset pose recognition model or a traditional image processing method, etc.

[0119] Step 330: Generate an assembly instruction based on the initial pose to instruct the robot to drive the assembly to attempt an assembly operation.

[0120] In one embodiment, the controller may use the initial pose (or other poses obtained based on the initial pose) as the target pose for the robot to move. Based on this target pose, combined with the robot model and the inverse kinematics equation, etc., an assembly instruction for the motion information of each motion joint of the robot (such as: motion amount, speed / angular velocity, acceleration / angular acceleration, etc.) is generated, so as to instruct the robot to drive the peg to attempt an assembly operation through this instruction.

[0121] Step 340: Obtain the tactile information under the current assembly operation.

[0122] It should be noted that the current assembly operation can refer to an attempted assembly operation performed based on the initial positioning pose, or any assembly operation before the completion of assembly.

[0123] In one embodiment, the controller can obtain the tactile information output by the force / tactile sensor under the attempted assembly operation from the memory or the server according to a preset address.

[0124] Step 350 takes the tactile information as the input of the assembly model and outputs the current relative pose between the assembled body and the body to be assembled.

[0125] In one embodiment, the controller can take the tactile information as the input of the assembly model and output the relative pose between the assembled body and the body to be assembled.

[0126] Step 360 iteratively performs the current assembly operation based on the current relative pose until the assembly is completed.

[0127] In one embodiment, based on the output of the assembly model, the pose of the hole relative to the peg, pegHhole, can be obtained. Assuming that the peg is tightly grasped by the robotic arm, the end pose of the peg relative to the robotic arm is invariant, that is, eeHpeg is known. Our goal is to control the end of the robotic arm so that the poses of the hole and the peg coincide, that is, to make eeHpeg = eeHhole. This can be achieved by various existing or future control methods, such as: Cartesian Position Control or Cartesian Velocity Control of the robot, etc.

[0128] In one embodiment, after performing the current assembly operation, it can be determined whether the assembly is successful. If not, the method steps from step 340 to step 360 are repeatedly executed until the assembly operation is completed. If successful, the current assembly terminates.

[0129] Specifically, any existing or future-developed method can be adopted as needed to determine whether the assembly operation is completed.

[0130] In one embodiment, the length of the assembled body (such as: peg) located outside the body to be assembled (such as: hole) can be recognized based on the visual image collected by the image sensor. When the length located outside the hole meets the preset requirements, it is regarded as the completion of the assembly.

[0131] In the embodiments of the present application, by combining a trained assembly model, the relative pose of the assembled body and the body to be assembled is obtained based on tactile information, thereby improving the accuracy of assembly control. By setting a force / tactile sensor at the end effector, the change in force between the assembled body and the body to be assembled can be felt more precisely, and at the same time, the magnitude and direction of the contact point force can be obtained more precisely. Compared with methods such as setting a 6D force / tactile sensor at the end of the robotic arm, this method can more accurately estimate the relative pose of the assembled body and the body to be assembled, improving the accuracy and success rate of assembly control.

[0132] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0133] It should be understood that although the steps in the flowchart of the accompanying drawings are displayed sequentially according to the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. Their execution order is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0134] Further reference Figure 4 to Figure 2 As an implementation of the method shown above, an embodiment of a training device for an assembly model is provided in the present application. This device embodiment corresponds to the embodiment of the training method for the assembly model shown in Figure 2 and can be specifically applied to various controllers.

[0135] As Figure 4 shown, the training device 200 for the assembly model in this embodiment includes:

[0136] A sample acquisition module 210, configured to acquire tactile signal training samples; using the relative pose of the assembled body and the body to be assembled as the label of the tactile signal training samples.

[0137] A model training module 220, configured to train an assembly model based on tactile signal training samples to obtain a trained assembly model.

[0138] In one embodiment, the training device 200 of the assembly model further includes:

[0139] A pose acquisition module, configured to acquire the pose information of the object to be assembled.

[0140] An instruction generation module, configured to generate an assembly instruction based on the pose information of the object to be assembled, so as to instruct the robot to drive the assembly object to attempt to assemble with the object to be assembled multiple times through the assembly instruction.

[0141] A data calculation module, configured to acquire the tactile signal during each attempt of assembly, and calculate the relative pose between the assembly object and the object to be assembled accordingly.

[0142] A data storage module, configured to store the tactile signal as a tactile signal training sample and the corresponding relative pose as a label.

[0143] In one embodiment, the data calculation module includes:

[0144] An image acquisition sub-module, configured to acquire an image of the assembly object.

[0145] A pose recognition sub-module, configured to recognize the pose information of the assembly object based on the image of the assembly object.

[0146] A pose calculation sub-module, configured to calculate the relative pose between the assembly object and the object to be assembled based on the pose information of the object to be assembled and the pose information of the assembly object.

[0147] In one embodiment, the model training module 220 includes:

[0148] A pose generation sub-module, configured to use the tactile signal training sample as the input of the assembly model and output a predicted relative pose.

[0149] A parameter update sub-module, configured to calculate the loss between the relative pose and the label, and update the parameters of the assembly model based on the loss.

[0150] A step repetition sub-module, configured to repeat the above steps until a preset termination condition is met.

[0151] Further referring to Figure 5 As an implementation of the method shown above, Figure 3 This application provides an embodiment of an assembly device. This device embodiment corresponds to the assembly method embodiment shown in Figure 2 and can be specifically applied to various controllers.

[0152] As Figure 5As shown in the figure, the assembly device 300 of this embodiment includes:

[0153] An image acquisition module 310, configured to acquire an image of the object to be assembled.

[0154] An initial pose determination module 320, configured to obtain the initial pose of the object to be assembled based on the image of the object to be assembled.

[0155] An instruction generation module 330, configured to generate an initial assembly instruction based on the initial pose to instruct the robot to drive the assembly to attempt an assembly operation.

[0156] A tactile acquisition module 340, configured to acquire tactile information under the current assembly operation.

[0157] A pose generation module 350, configured to use the tactile information as an input to the assembly model and output the current relative pose between the assembly and the object to be assembled.

[0158] An assembly execution module 360, based on the current relative pose, iteratively executes the current assembly operation until the assembly is completed.

[0159] Specifically, please refer to Figure 6 , to solve the above technical problems, the embodiment of the present application further provides a controller (taking the computer device 6 as an example).

[0160] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are communicatively connected to each other through a system bus. It should be noted that only the computer device 6 with components 61-63 is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, those skilled in the art of the present technology can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0161] The computer device may be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer device can perform human-computer interaction with the user through a keyboard, a mouse, a remote control, a touchpad or a voice control device and other means.

[0162] The memory 61 includes at least one type of readable storage medium, which includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disc, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, FlashCard, etc. equipped on the computer device 6. Of course, the memory 61 may also include both the internal storage unit and the external storage device of the computer device 6. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as the program code for training the assembly model and / or the assembly method. In addition, the memory 61 may also be used to temporarily store various types of data that have been output or will be output.

[0163] In some embodiments, the processor 62 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the program code stored in the memory 61 or process data, such as running the program code for training the assembly model and / or the assembly method.

[0164] The network interface 63 may include a wireless network interface or a wired network interface, and this network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0165] This application also provides another implementation manner, that is, to provide a computer-readable storage medium storing a training and / or assembly program of an assembly model, and the training and / or assembly program of the assembly model can be executed by at least one processor, so that the at least one processor executes the steps of the training and / or assembly method of the assembly model as described above.

[0166] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0167] Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The preferred embodiments of the present application are given in the drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosed content of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions recorded in the foregoing specific embodiments, or perform equivalent replacements on some of the technical features. Any equivalent structure directly or indirectly using the content of the specification and drawings of the present application in other related technical fields is similarly within the scope of the patent protection of the present application.

Claims

1. A method for training an assembly model, characterized in that, The method includes the following steps: Obtain a haptic signal training sample; use the relative pose of the assembly and the to-be-assembled body as the label of the haptic signal training sample; Train an assembly model based on the haptic signal training sample to obtain a trained assembly model.

2. The assembly model training method according to claim 1, wherein Before obtaining the haptic signal training sample, the following method steps are included: Obtain the pose information of the to-be-assembled body; Generate an assembly instruction based on the pose information of the to-be-assembled body, so as to instruct the robot to drive the assembly to attempt to assemble with the to-be-assembled body multiple times through the assembly instruction; Obtain the haptic signal during each attempt at assembly, and correspondingly obtain the relative pose of the assembly and the to-be-assembled body; Save the haptic signal as the haptic signal training sample, and the corresponding relative pose as the label.

3. The assembly model training method according to claim 2, wherein The obtaining of the relative pose of the assembly and the to-be-assembled body includes the following steps: Obtain an image of the assembly; Identify the pose information of the assembly based on the image of the assembly; Calculate the relative pose of the assembly and the to-be-assembled body based on the pose information of the to-be-assembled body and the pose information of the assembly.

4. The assembly model training method according to any one of claims 1 to 3, characterized in that Training the assembly model based on the haptic signal training sample includes the following steps: Use the haptic signal training sample as the input of the assembly model, and output to obtain a predicted relative pose; Calculate the loss between the relative pose and the label, and update the parameters of the assembly model based on the loss; Repeat the above steps until a preset termination condition is met.

5. An assembly method for an assembly model obtained by the assembly model training method according to any one of claims 1 to 4, characterized in that, The method includes the following steps: Obtain an image of the to-be-assembled body; Calculate the initial pose of the to-be-assembled body based on the image of the to-be-assembled body; Generate an initial assembly instruction based on the initial pose to instruct the robot to drive the assembly to attempt an assembly operation; Obtain the haptic information under the current assembly operation; Use the haptic information as the input of the assembly model, and output to obtain the current relative pose of the assembly and the to-be-assembled body; Iteratively execute the current assembly operation based on the current relative pose until the assembly is completed.

6. An assembly model training device, characterized in that The device includes: A sample acquisition module, used to obtain a haptic signal training sample; use the relative pose of the assembly and the to-be-assembled body as the label of the haptic signal training sample; A model training module, used to train an assembly model based on the haptic signal training sample to obtain a trained assembly model.

7. An assembly device for an assembly model obtained by the assembly model training device according to claim 6, characterized in that The device includes: An image acquisition module, used to obtain an image of the to-be-assembled body; A pose initial determination module, used to calculate the initial pose of the to-be-assembled body based on the image of the to-be-assembled body; An instruction generation module, used to generate an initial assembly instruction based on the initial pose to instruct the robot to drive the assembly to attempt an assembly operation; A haptic acquisition module, used to obtain the haptic information under the current assembly operation; A pose generation module, used to use the haptic information as the input of the assembly model, and output to obtain the current relative pose of the assembly and the to-be-assembled body; An assembly execution module, based on the current relative pose, iteratively executes the current assembly operation until the assembly is completed.

8. An assembly system, characterized in that, The system includes: a robot, a force / tactile sensor, and a controller; the robot includes: a robotic arm and an end effector disposed at a grasping execution end of the robotic arm; the force / tactile sensor is disposed at the end effector; The controller is communicatively connected to the robot and the force / tactile sensor respectively; The controller is configured to implement the steps of the assembly model training method according to any one of claims 1 to 4 and / or the assembly method according to claim 5.

9. A controller, characterized in that, It includes a memory and a processor, and a computer program is stored in the memory. When the processor executes the computer program, the steps of the assembly model training method according to any one of claims 1 to 4 and / or the assembly method according to claim 5 are implemented.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the assembly model training method according to any one of claims 1 to 4 and / or the assembly method according to claim 5 are implemented.

Images