A multi-robot flexible object servo method based on deep learning
Through the multi-robot flexible object servo method based on deep learning, the DeformerNet network and multi-machine coordinated operation is used to solve the complexity problem of 3D deformable object shape control, and the precise servo of objects of different shapes and Young's modulus is achieved, reducing the number of objects with different shapes and Young's modulus. Cost and expand robot application scenarios.
Patent Information
- Application Number
- CN202411133433.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-19
AI Technical Summary
The prior art is difficult to effectively solve the complexity of 3D deformable object shape control, especially in the high degree of freedom and nonlinear deformation models for perceived deformation. The traditional method is expensive, noise-sensitive and unable to generalize to various objects. .
Using a multi-robot flexible object servo method based on deep learning, the deformation control module is trained using the DeformerNet network, combining multi-machine collaborative operation and point cloud feature extraction to achieve shape servo of flexible objects.
It reduces costs, improves task expansion and robustness, realizes accurate shape servo for objects of different shapes and Young's modulus, reduces labor and consumable costs, and enhances the application potential of robots in the field of aviation manufacturing.
Smart Images

Figure CN118650634B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of multi-machine collaboration technology, and in particular to a multi-robot flexible object servo method based on deep learning. Background Art
[0002] With the continuous advancement of industrial automation, more tasks in the field of industrial manufacturing require robots to complete autonomously. In the field of aviation manufacturing, rigid parts are the main components, but there are also some parts with certain deformable properties, such as cables and long stringers. To achieve the operation of three-dimensional deformable objects, robots can complete more assembly tasks independently or in collaboration with people in aviation manufacturing scenarios. For example, when assembling aircraft wall panels, industrial robots grab long stringer parts and assemble them on the skin; robots in cable assembly in aircraft cabins should quickly and accurately grab cables and servo them into a certain shape for assembly; it also includes the draping of composite materials on some parts of the aircraft, which is completed through the collaboration of multiple robots and people to improve process accuracy and draping speed.
[0003] However, there are still many challenges in deformable object manipulation (DOM). Rigid bodies only need 6 degrees of freedom (DOF) to be represented, while the shape of deformable objects requires potentially infinite degrees of freedom. Therefore, at the technical level, we need to solve three challenges: the complexity of perceiving deformation, the high degrees of freedom of soft bodies, and the complexity of nonlinearity in deformation models. To address the above problems, we will extract the point cloud image features of deformable objects, use neural networks to train the controller of 3D deformable object shape servo, and use multi-machine collaboration to achieve the task of shape servo control.
[0004] In the field of industrial manufacturing, a variety of solutions have been proposed for tasks related to deformable objects, such as cable shape recognition and composite material laying. There are methods that use shape tracking, using monocular cameras or 3D cameras, but the application objects of these methods are relatively limited. Some scholars have also proposed model-based shape servo methods. Among model-based shape servo, the finite element method (FEM) is the most widely used. A control method based on open-loop simulation is proposed, in which the desired deformation is directly mapped to the joint angle command. FEM is used in the open-loop control method of handheld soft object manipulation. However, the mechanical model is a reliable solution, but it is costly and depends on the inherent mechanical properties of the object. Therefore, some people have proposed model-free shape servo methods, one of which is sensor-based deformation Jacobian, such as online sensor measurements are used to estimate the deformation Jacobian used in shape servo. However, the algorithm calculation of this type of method is complex and sensitive to noise. In recent years, the control of deformable objects has increasingly tended to adopt learning-based methods. Among them, reinforcement learning (RL) is the most widely used method. Although these methods are relatively successful, they also have certain limitations.
[0005] Solutions for shape control of 3D deformable objects can be divided into two categories: learning-based and non-learning-based methods. In the non-learning-based methods, a series of studies define geometric features on the object as state representation. Researchers use this representation to perform adaptive linear controlled visual servoing. However, these methods are only applicable to known objects with obvious textures and cannot be generalized to a variety of objects. This representation only controls the displacement of individual points and cannot fully reflect the 3D shape of the object. In order to achieve precise control, a large number of feature points must be used, making the control highly sensitive to noise and occlusion. On the other hand, other non-learning-based methods use 2D image contours to represent the shape of the object, but this severely limits the controllable 3D deformation space. Summary of the invention
[0006] The present invention provides a multi-robot flexible object servo method based on deep learning to solve the technical problems mentioned in the background technology.
[0007] To achieve the above object, the technical solution of the present invention is achieved as follows:
[0008] The present invention provides a multi-robot flexible object servo method based on deep learning, comprising the following steps:
[0009] S1. Build robot simulation models and flexible object simulation models, import simulation platforms, and build flexible object assembly scenarios with multi-machine collaborative operation;
[0010] S2, confirm the position and grasping point of the flexible object based on the external camera information, and control the dual robots to collaboratively grasp the flexible object;
[0011] S3, the two robots cooperate with the flexible object to servo into multiple groups of shapes, and record the point cloud information of the flexible object and the position information of the robot end to obtain multiple groups of model training data;
[0012] S4, inputting the point cloud information in the multiple sets of model training data into the point cloud feature extractor, extracting the feature vector of the point cloud information, inputting the extracted feature vector and the position information of the end of the robot arm into the DeformerNet network, and using the DeformerNet network to train the deformation control module;
[0013] S5. Collect a set of randomly generated shape point cloud information in advance, import the point cloud information into the trained deformation control module, and the deformation control module outputs motion instructions to control the dual robots to collaboratively grasp the flexible object and servo the object shape to the target shape.
[0014] Furthermore, the S1 specifically includes the following steps:
[0015] S11. According to the robot's joint information and connecting rod parameters, the robot's URDF file is written in a programming language, the robot's connecting rod position relationship, joint type, and collision volume of each module component are set, and a robot simulation model is built;
[0016] S12, setting the volume parameters of the flexible object, generating a quadrilateral mesh file of the flexible object by using the FTetWild algorithm, writing a URDF file of the flexible object by using a programming language in combination with finite element parameters, setting the Young's modulus, collision volume, and volume density of the flexible object, and constructing a simulation model of the flexible object;
[0017] S13. Import the URDF files of the robot simulation model and the flexible object simulation model into the simulation platform to build a flexible object assembly scene with multi-machine collaborative operation.
[0018] Furthermore, the S2 specifically includes the following steps:
[0019] S21, using the point cloud image information collected by the camera through the RANSAC point cloud segmentation algorithm in PCL to obtain separate point cloud image information of the flexible object;
[0020] S22, using a heuristic algorithm to determine whether the flexible object has a top surface according to the shape point cloud information of the flexible object and the coordinate information of the center point of the flexible object, generating two symmetrical robot end grasping points with the center point as the midpoint on the surface of the flexible object according to the coordinate information of the center point, and transmitting the coordinate information of the grasping points to the robot;
[0021] S23. The robot moves to the position of the grasping point through the built-in end position controller, closes the gripper, and completes the grasping.
[0022] Furthermore, the S3 specifically includes the following steps:
[0023] S31. Pre-generation The robot will move along the random trajectory, and each flexible object will generate Random shapes;
[0024] S32, during the process of the robot operating the flexible object, the robot is controlled by using a master-slave control algorithm;
[0025] Specifically, for dual robots, we set one robot as the master and the other as the slave. We set the end position of the master and joint angles Synchronize to the end position of the slave and joint angles , realize the task of synchronous grasping of the host and the slave to complete the shape servo of the flexible object;
[0026] S33. During the robot's operation of an object, each trajectory is set Time checkpoint t, the camera in the simulation platform records the point cloud of the current flexible object shape at the set time checkpoint and the pose information of the robot end effector ;
[0027] S34, check the current point cloud at the input time point t and target shape point cloud Compose supervision data input-output pairs; and calculate the displacement between the robot's end position at the end of the trajectory and the robot's end position at each time checkpoint t ,get Set model training data.
[0028] Furthermore, the S4 specifically includes the following steps:
[0029] S41, reducing the number of point clouds in the model training data obtained in S34 by a downsampling method, and performing lightweight processing on the point cloud information;
[0030] S42, passing the light-weighted point cloud information through a feature extractor consisting of three consecutive PointConv convolutional layers, and continuously outputting points of a first set dimension and a feature vector of a second set dimension;
[0031] S43. Set the feature extractor to , the shape feature vector output after the lightweight point cloud is taken as input is defined as , the current point cloud and target shape point cloud Input into the feature extractor of the DeformerNet network to generate feature vectors and , where the target shape point cloud This is the position information of the end of the robot arm; then take the difference between the two to get the current point cloud and target shape point cloud The characteristic displacement of
[0032] ;
[0033] S44, define the deformation control function as , the characteristic displacement and deformation control function obtained by S43 Construct deformation control module;
[0034] S45, passing the feature vector of the second set dimension extracted in S42 to a deformation control module, where the deformation control module is composed of a series of fully connected layers, and maps the 3D displacement required by the end of the robotic arm through the fully connected output layer;
[0035] S46. Construct a loss function, calculate the loss between the mapped 3D displacement required by the end of the robotic arm and the actual 3D displacement required, loop S42 to S46, update the parameters of the deformation control module in the DeformerNet network through the back propagation algorithm, gradually reduce the value of the loss function until convergence, and obtain the trained deformation control module.
[0036] Furthermore, the deformation control module in S44 is expressed by a formula, which is as follows:
[0037] ;
[0038] in, Represents the deformation control module.
[0039] Furthermore, the series of fully connected layers in S45 contain 128, 64 and 32 neural units respectively.
[0040] Furthermore, the S5 specifically includes the following steps:
[0041] S51, randomly generate a group A of trajectories, drive the robot to operate the flexible object to servo out a group A of random target shapes, and collect the target shape point cloud of the group A of flexible objects through the camera , obtain the test set, and input the data in the test set into the trained deformation control module;
[0042] S52. Obtaining point cloud information of the current flexible object through the camera and position information, and calculate the grasping point on the surface of the flexible object through a heuristic algorithm;
[0043] S53, according to the position of the grasping point, the position controller drives the robot to move to the grasping point, and the gripper closes to grasp the target object;
[0044] S54, deformation control module according to the current object shape point cloud and target shape point cloud Output the robot's motion command Z. The robot completes the shape servo task according to the motion command Z output by the deformation control module, and records the shape point cloud image information of the flexible object at the end of the trajectory. ;
[0045] S55, calculate the target shape point cloud and shape point cloud image information The chamfer distance between the two objects is determined to determine whether the calculated chamfer distance is less than a preset value. If so, it indicates that the flexible object has been servoed to the target shape required by us and the shape servo task has been completed. If not, the process proceeds to S56.
[0046] S56, input the point cloud to the trained deformation control module, and collect the real point cloud of the part after moving to the target point;
[0047] S57, calculating the chamfer distance between the current point cloud and the target point cloud;
[0048] S58. Determine whether the calculated chamfer distance is less than the preset value. If so, it means that the flexible object has been servoed to the target shape we require and the shape servo task has been completed. If not, loop S56 to S58 until the calculated chamfer distance is less than the preset value.
[0049] Furthermore, the target shape point cloud is calculated in S55 and shape point cloud image information The chamfer distance between is expressed by the formula as follows:
[0050] ;
[0051] in, Is the shape point cloud information currently collected ; is the current target shape point cloud .
[0052] Furthermore, the preset value in S55 and S58 is 0.5.
[0053] Beneficial effects of the present invention:
[0054] (1)Cost saving:
[0055] At present, the traditional method requires manual repeated debugging, and the ability to collect information randomly and autonomously is poor. Some weak rigid parts cannot be used repeatedly, and the labor cost is high, the consumables cost is high, and it takes a long time. This method generates multiple sets of random trajectories in a simulation environment, repeatedly operates on objects with different Young's moduli, and collects point cloud image information of the operated objects through the built-in depth camera in the environment, reducing labor costs and consumables costs.
[0056] (2) High task expandability:
[0057] At present, the main objects in the field of industrial assembly are still rigid objects, and most of them are operated by robots. The present invention uses dual robots to work together, and the operating objects they face are flexible workpieces. It can greatly expand the richness of robot assembly tasks, enable robots to further participate in aviation assembly scenarios, further promote robots to cover human operations, and realize unmanned factories and assembly automation.
[0058] (3) High robustness:
[0059] Compared with non-learning methods, this method adopts a neural network to train the deformation control module. By collecting a large amount of point cloud information of the operating object and inputting it into the fully connected neural network to train the deformation control module, it can learn the positions of corresponding operating points under more shapes, and can complete the corresponding shape servo tasks for operating objects with different shapes and different Young's moduli.
[0060] (4) High accuracy:
[0061] The present invention can more accurately describe the physical characteristics of the operating object through the representation method of the finite cloud model in the simulation platform; and adopts a fully connected neural network to train the deformation control module. Each layer of the network contains multiple neural units, which can better learn the image characteristics of the point cloud itself. In terms of multi-machine collaboration, the master-slave control method is adopted to complete more accurate shape servo tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 is a flow chart of the present invention;
[0063] Figure 2 A schematic diagram of building a multi-machine collaborative flexible object assembly scene in S1 of the present invention;
[0064] Figure 3 A schematic diagram of multiple groups of model training data collected in S2 and S3 of the present invention;
[0065] Figure 4 Schematic diagram of the network structure of the DeformerNet network in the present invention;
[0066] Figure 5 is a schematic diagram of performing a shape servo task in S5 of the present invention;
[0067] Figure 6 Schematic diagram of the shape servo task execution process in S5 of the present invention;
[0068] Figure 7 This is a flow chart of the execution of the shape servo task in S5 of the present invention. DETAILED DESCRIPTION
[0069] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many other different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.
[0070] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0071] Reference Figure 1 The embodiment of the present application provides a multi-robot flexible object servo method based on deep learning, comprising the following steps:
[0072] S1. Build robot simulation models and flexible object simulation models, import the Isaac Gym simulation platform, and build a flexible object assembly scene with multi-machine collaborative operation;
[0073] S2, confirm the position and grasping point of the flexible object based on the external camera information, and control the dual robots to collaboratively grasp the flexible object;
[0074] S3, the two robots cooperate with the flexible object to servo into multiple groups of shapes, and record the point cloud information of the flexible object and the position information of the robot end to obtain multiple groups of model training data;
[0075] S4, inputting the point cloud information in the multiple sets of model training data into the point cloud feature extractor, extracting the feature vector of the point cloud information, inputting the extracted feature vector and the position information of the end of the robot arm into the DeformerNet network, and using the DeformerNet network to train the deformation control module;
[0076] S5. Collect a set of randomly generated shape point cloud information in advance, import the point cloud information into the trained deformation control module, and the deformation control module outputs motion instructions to control the dual robots to collaboratively grasp the flexible object and servo the object shape to the target shape.
[0077] In some embodiments, the S1 specifically includes the following steps:
[0078] S11. According to the robot's joint information and connecting rod parameters, the robot's URDF file is written in XML programming language, the robot's connecting rod position relationship, joint type, and collision volume of each module component are set, and a robot simulation model is built;
[0079] S12, setting the volume parameters of the flexible object, generating a quadrilateral mesh file of the flexible object by using the FTetWild algorithm, writing the URDF file of the flexible object by using the XML programming language in combination with the finite element parameters, setting the Young's modulus, collision volume, and volume density of the flexible object, and constructing a simulation model of the flexible object;
[0080] S13. Import the URDF files of the robot simulation model and the flexible object simulation model into the Isaac Gym simulation platform to build a multi-machine collaborative operation flexible object assembly scene.
[0081] In some embodiments, S2 specifically includes the following steps:
[0082] S21, using the point cloud image information collected by the camera through the RANSAC point cloud segmentation algorithm in PCL to obtain separate point cloud image information of the flexible object;
[0083] S22, using a heuristic algorithm to determine whether the flexible object has a top surface according to the shape point cloud information of the flexible object and the coordinate information of the center point of the flexible object, generating two symmetrical robot end grasping points with the center point as the midpoint on the surface of the flexible object according to the coordinate information of the center point, and transmitting the coordinate information of the grasping points to the robot;
[0084] S23. The robot moves to the position of the grasping point through the built-in end position controller, closes the gripper, and completes the grasping.
[0085] In some embodiments, S3 specifically includes the following steps:
[0086] S31, pre-generate 300 random trajectories, the robot will move along the random trajectories, and each flexible object will generate 10 random shapes;
[0087] S32, during the process of the robot operating the flexible object, the robot is controlled by using a master-slave control algorithm;
[0088] Specifically, for dual robots, we set one robot as the master and the other as the slave. We set the end position of the master and joint angles Synchronize to the end position of the slave and joint angles , realize the task of synchronous grasping of the host and the slave to complete the shape servo of the flexible object;
[0089] S33. During the robot's operation of the object, 10 time checkpoints t are set for each trajectory, and the camera in the IsaacGym simulation platform records the point cloud of the current flexible object shape at the set time checkpoints. and the pose information of the robot end effector ;
[0090] S34, check the current point cloud at the input time point t and target shape point cloud Compose supervision data input-output pairs; and calculate the displacement between the robot's end position at the end of the trajectory and the robot's end position at each time checkpoint t , and obtain 10,000 sets of model training data.
[0091] In some embodiments, the S4 specifically includes the following steps:
[0092] S41, reducing the number of point clouds in the model training data obtained in S34 by a downsampling method, and performing lightweight processing on the point cloud information;
[0093] S42, passes the light-weighted point cloud information through a feature extractor consisting of three consecutive PointConv convolutional layers, and continuously outputs points with dimensions of (64,512) and (128,256) and a 256-dimensional feature vector;
[0094] S43. Set the feature extractor to , the shape feature vector output after the lightweight point cloud is taken as input is defined as , the current point cloud and target shape point cloud Input into the feature extractor of the DeformerNet network to generate feature vectors and , where the target shape point cloud This is the position information of the end of the robot arm; then take the difference between the two to get the current point cloud and target shape point cloud The characteristic displacement of
[0095] ;
[0096] S44, define the deformation control function as , the characteristic displacement and deformation control function obtained by S43 Construct deformation control module;
[0097] S45, passing the extracted 256-dimensional point cloud feature vector to the deformation control module, which is composed of a series of fully connected layers, and maps the 3D displacement required by the end of the robotic arm through the fully connected output layer;
[0098] S46. Construct a loss function, calculate the loss between the mapped 3D displacement required by the end of the robotic arm and the actual 3D displacement required, loop S42 to S46, update the parameters of the deformation control module in the DeformerNet network through the back propagation algorithm, gradually reduce the value of the loss function until convergence, and obtain the trained deformation control module.
[0099] Specifically, the 256-dimensional point cloud feature vector extracted by the feature extractor is used as input. We use the standard mean square error as the loss function to train DeformerNet, and adopt the Adam optimizer and the decay learning rate method. The learning rate is from Initially, the loss function is reduced by 1 / 10 every 140 epochs. The network parameters are updated through the back-propagation algorithm, gradually reducing the value of the loss function until convergence;
[0100] In some embodiments, the deformation control module in S44 is expressed by a formula, which is as follows:
[0101] ;
[0102] in, Represents the deformation control module.
[0103] In some embodiments, the series of fully connected layers in S45 contain 128, 64, and 32 neural units, respectively.
[0104] In some embodiments, S5 specifically includes the following steps:
[0105] S51, randomly generate a group A of trajectories, drive the robot to operate the flexible object to servo out a group A of random target shapes, and collect the target shape point cloud of the group A of flexible objects through the camera , obtain the test set, and input the data in the test set into the trained deformation control module;
[0106] S52. Obtaining point cloud information of the current flexible object through the camera and position information, and calculate the grasping point on the surface of the flexible object through a heuristic algorithm;
[0107] S53, according to the position of the grasping point, the position controller drives the robot to move to the grasping point, and the gripper closes to grasp the target object;
[0108] S54, deformation control module according to the current object shape point cloud and target shape point cloud Output the robot's motion command Z. The robot completes the shape servo task according to the motion command Z output by the deformation control module, and records the shape point cloud image information of the flexible object at the end of the trajectory. ;
[0109] S55, calculate the target shape point cloud and shape point cloud image information The chamfer distance between the two is used to determine whether the calculated chamfer distance is less than 0.5. If so, it means that the flexible object has been servoed to the target shape required by us and the shape servo task has been completed; if not, enter S56;
[0110] S56, input the point cloud to the trained deformation control module, and collect the real point cloud of the part after moving to the target point;
[0111] S57, calculating the chamfer distance between the current point cloud and the target point cloud;
[0112] S58. Determine whether the calculated chamfer distance is less than 0.5. If so, it means that the flexible object has been servoed to the target shape we require and the shape servo task has been completed. If not, loop S56 to S58 until the calculated chamfer distance is less than the preset value.
[0113] In some embodiments, the target shape point cloud is calculated in S55. and shape point cloud image information The chamfer distance between is expressed by the formula as follows:
[0114] ;
[0115] in, Is the shape point cloud information currently collected ; is the current target shape point cloud .
[0116] Chamfer distance, in the field of computer vision, image processing and pattern recognition, is a distance metric used to compare the similarity between two point clouds or images. It is obtained by calculating the distance between each point and the nearest point of the target shape, and accumulating and averaging these distances. It quantifies the minimum cost required to transform one shape into another. Therefore, the chamfer distance between two point clouds is used as a criterion for judging the performance of the controller.
[0117] Compared with the prior art, the advantages of the present invention are:
[0118] (1)Cost saving:
[0119] At present, the traditional method requires manual repeated debugging, and the ability to collect information randomly and autonomously is poor. Some weak rigid parts cannot be used repeatedly, and the labor cost is high, the consumables cost is high, and it takes a long time. This method generates multiple sets of random trajectories in a simulation environment, repeatedly operates on objects with different Young's moduli, and collects point cloud image information of the operated objects through the built-in depth camera in the environment, reducing labor costs and consumables costs.
[0120] (2) High task expandability:
[0121] At present, the main objects in the field of industrial assembly are still rigid objects, and most of them are operated by robots. The present invention uses dual robots to work together, and the operating objects they face are flexible workpieces. It can greatly expand the richness of robot assembly tasks, enable robots to further participate in aviation assembly scenarios, further promote robots to cover human operations, and realize unmanned factories and assembly automation.
[0122] (3) High robustness:
[0123] Compared with non-learning methods, this method adopts a neural network to train the deformation control module. By collecting a large amount of point cloud information of the operating object and inputting it into the fully connected neural network to train the deformation control module, it can learn the positions of corresponding operating points under more shapes, and can complete the corresponding shape servo tasks for operating objects with different shapes and different Young's moduli.
[0124] (4) High accuracy:
[0125] The present invention can more accurately describe the physical characteristics of the operating object through the representation method of the finite cloud model in the simulation platform; and adopts a fully connected neural network to train the deformation control module. Each layer of the network contains multiple neural units, which can better learn the image characteristics of the point cloud itself. In terms of multi-machine collaboration, the master-slave control method is adopted to complete more accurate shape servo tasks.
[0126] The present invention provides a multi-robot deformable object servo method based on deep learning. Aiming at the shape servo operation tasks of multiple robots on deformable objects, a framework based on deep learning, flexible object shape servo and multi-machine collaborative control is designed, which improves the generalization of the robot's shape servo for deformable parts of different materials and enhances the complexity and accuracy of the flexible component shape servo operation tasks. The method uses Isaac Gym simulation software as a test platform for the algorithm; combines the quadrilateral mesh file of the flexible object, the finite element model method and the CAD model, uses XML language to write the URDF files of the deformable object, the robot and the aircraft panel, and imports them into Isaac Gym to build a multi-machine collaborative operation flexible object assembly scene; uses an external depth camera to obtain 3D image information of the current environment; uses the RANSAC algorithm to segment the point cloud of the target object; uses a heuristic algorithm and combines the position information of the center point of the object to infer the position of the robot's grasping point on the surface of the object; by generating random paths, the robot operates the deformable object to servo into multiple groups of shapes, and uses an external depth camera to record the shape point cloud of the object, and trains the deformation control module through deep learning; by inputting a large amount of data, the control performance of the shape controller is improved; by collecting data on objects of different materials and shapes and training the controller, the generalization performance of the shape controller for objects of different materials and shapes is improved.
[0127] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention. In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A multi-robot flexible object servo method based on deep learning, It is characterized in that The steps include: S1. Build robot simulation models and flexible object simulation models, import simulation platforms, and build flexible object assembly scenarios with multi-machine collaborative operation; S2, confirm the position and grasping point of the flexible object based on the external camera information, and control the dual robots to collaboratively grasp the flexible object; S3, the two robots cooperate with the flexible object to servo into multiple groups of shapes, and record the point cloud information of the flexible object and the position information of the robot end to obtain multiple groups of model training data; The S3 specifically includes the following steps: S31. Pre-generation The robot will move along the random trajectory, and each flexible object will generate Random shapes; S32, during the process of the robot operating the flexible object, the robot is controlled by using a master-slave control algorithm; Specifically, for dual robots, we set one robot as the master and the other as the slave. We set the end position of the master and joint angles Synchronize to the end position of the slave and joint angles , realize the task of synchronous grasping of the host and the slave to complete the shape servo of the flexible object; S33. During the robot's operation of an object, each trajectory is set Time checkpoint t, the camera in the simulation platform records the point cloud of the current flexible object shape at the set time checkpoint and the pose information of the robot end effector ; S34, check the current point cloud at the input time point t and target shape point cloud Compose supervision data input-output pairs; and calculate the displacement between the robot's end position at the end of the trajectory and the robot's end position at each time checkpoint t ,get Set model training data; S4, inputting the point cloud information in the multiple sets of model training data into the point cloud feature extractor, extracting the feature vector of the point cloud information, inputting the extracted feature vector and the position information of the end of the robot arm into the DeformerNet network, and using the DeformerNet network to train the deformation control module; S5. Collect a set of randomly generated shape point cloud information in advance, import the point cloud information into the trained deformation control module, and the deformation control module outputs motion instructions to control the dual robots to collaboratively grasp the flexible object and servo the object shape to the target shape.
2. The multi-robot flexible object servo method according to claim 1, It is characterized in that The S1 specifically includes the following steps: S11. According to the robot's joint information and connecting rod parameters, the robot's URDF file is written in a programming language, the robot's connecting rod position relationship, joint type, and collision volume of each module component are set, and a robot simulation model is built; S12, setting the volume parameters of the flexible object, generating a quadrilateral mesh file of the flexible object by using the FTetWild algorithm, writing a URDF file of the flexible object by using a programming language in combination with finite element parameters, setting the Young's modulus, collision volume, and volume density of the flexible object, and constructing a simulation model of the flexible object; S13. Import the URDF files of the robot simulation model and the flexible object simulation model into the simulation platform to build a flexible object assembly scene with multi-machine collaborative operation.
3. The multi-robot flexible object servo method according to claim 2, It is characterized in that The S2 specifically includes the following steps: S21, using the point cloud image information collected by the camera through the RANSAC point cloud segmentation algorithm in PCL to obtain separate point cloud image information of the flexible object; S22, using a heuristic algorithm to determine whether the flexible object has a top surface according to the shape point cloud information of the flexible object and the coordinate information of the center point of the flexible object, generating two symmetrical robot end grasping points with the center point as the midpoint on the surface of the flexible object according to the coordinate information of the center point, and transmitting the coordinate information of the grasping points to the robot; S23. The robot moves to the position of the grasping point through the built-in end position controller, closes the gripper, and completes the grasping.
4. The multi-robot flexible object servo method according to claim 1, It is characterized in that The S4 specifically includes the following steps: S41, reducing the number of point clouds in the model training data obtained in S34 by a downsampling method, and performing lightweight processing on the point cloud information; S42, passing the light-weighted point cloud through a feature extractor consisting of three consecutive PointConv convolutional layers, and continuously outputting points of a first set dimension and a feature vector of a second set dimension; S43. Set the feature extractor to , the shape feature vector output after the lightweight point cloud is taken as input is defined as , the current point cloud and target shape point cloud Input into the feature extractor of the DeformerNet network to generate feature vectors and , where the target shape point cloud This is the position information of the end of the robot arm; then take the difference between the two to get the current point cloud and target shape point cloud The characteristic displacement of ; S44, define the deformation control function as , the characteristic displacement and deformation control function obtained by S43 Construct deformation control module; S45, passing the feature vector of the second set dimension extracted in S42 to a deformation control module, where the deformation control module is composed of a series of fully connected layers, and maps the 3D displacement required by the end of the robotic arm through the fully connected output layer; S46. Construct a loss function, calculate the loss between the mapped 3D displacement required by the end of the robotic arm and the actual 3D displacement required, loop S42 to S46, update the parameters of the deformation control module in the DeformerNet network through the back propagation algorithm, gradually reduce the value of the loss function until convergence, and obtain the trained deformation control module.
5. The multi-robot flexible object servo method according to claim 4, It is characterized in that The deformation control module in S44 is expressed by a formula, which is as follows: ; in, Represents the deformation control module.
6. The multi-robot flexible object servo method according to claim 5, It is characterized in that The series of fully connected layers in S45 contain 128, 64 and 32 neural units respectively.
7. The multi-robot flexible object servo method according to claim 6, It is characterized in that The S5 specifically includes the following steps: S51, randomly generate a group A of trajectories, drive the robot to operate the flexible object to servo out a group A of random target shapes, and collect the target shape point cloud of the group A of flexible objects through the camera , obtain the test set, and input the data in the test set into the trained deformation control module; S52. Obtaining point cloud information of the current flexible object through the camera and position information, and calculate the grasping point on the surface of the flexible object through a heuristic algorithm; S53, according to the position of the grasping point, the position controller drives the robot to move to the grasping point, and the gripper closes to grasp the target object; S54, deformation control module according to the current object shape point cloud and target shape point cloud Output the robot's motion command Z. The robot completes the shape servo task according to the motion command Z output by the deformation control module, and records the shape point cloud image information of the flexible object at the end of the trajectory. ; S55, calculate the target shape point cloud and shape point cloud image information The chamfer distance between the two objects is determined to determine whether the calculated chamfer distance is less than a preset value. If so, it indicates that the flexible object has been servoed to the target shape required by us and the shape servo task has been completed. If not, the process proceeds to S56. S56, input the point cloud to the trained deformation control module, and collect the real point cloud of the part after moving to the target point; S57, calculating the chamfer distance between the current point cloud and the target point cloud; S58. Determine whether the calculated chamfer distance is less than the preset value. If so, it means that the flexible object has been servoed to the target shape we require and the shape servo task has been completed. If not, loop S56 to S58 until the calculated chamfer distance is less than the preset value.
8. The multi-robot flexible object servo method according to claim 7, It is characterized in that The target shape point cloud is calculated in S55 and shape point cloud image information The chamfer distance between is expressed by the formula as follows: ; in, Is the shape point cloud information currently collected ; is the current target shape point cloud .
9. The multi-robot flexible object servo method according to claim 8, It is characterized in that The preset value in S55 and S58 is 0.5.
Citation Information
Cited By
Robot operation-oriented branch linear flexible body dynamics modeling method
CN121061909A
A branch linear flexible body dynamics modeling method for robot operation
CN121061909B
