A method for autonomous intelligent assembly of spatial superstructures based on virtual teaching

Abstract

The present invention discloses an autonomous intelligent assembly method of a spatial superstructure based on virtual teaching, which belongs to the field of robot intelligent control technology. The method first reconstructs the object to be assembled in three dimensions, transmits the three-dimensional information to the ground virtual simulation system, and the ground virtual simulation system reproduces the virtual information of the object to be assembled; secondly, based on the ground virtual simulation system, the assembly operation of the virtual object to be assembled is performed multiple times, and multi-source data in each assembly process is obtained to form and train a control data set, with the shortest time and minimum energy consumption as the optimization goals, and obstacle avoidance and on-orbit device capability parameters as constraints to form the optimal control method; finally, based on the optimal control method, an autonomous control method of the on-orbit device is generated, uploaded to the on-orbit device, and the on-orbit device assembles the object to be assembled. The present invention can effectively reduce the control cost and the damage rate of the operation object, and improve the success rate of on-orbit assembly.

Description

Technical Field

[0001] The invention relates to an autonomous intelligent assembly method of a space superstructure based on virtual teaching, and belongs to the technical field of robot intelligent control. Background Art

[0002] Space superstructures, such as large space truss structures, are an important component of the supporting framework of space-based infrastructure such as space stations, large solar arrays, and large reflector antennas.

[0003] There are currently two main ways to achieve the above-mentioned large space structures: one is on-orbit deployment, where the structure is folded before launch and unfolded after being sent into orbit; the other is on-orbit assembly (assembly), where parts / components are manufactured on the ground and assembled by a space robotic arm after being sent into orbit. The former has a size bottleneck due to launch volume / weight restrictions; the latter has no size restrictions. The larger the size of the space superstructure, the better the performance of the space-based infrastructure such as the solar array / antenna it constitutes, so on-orbit assembly is the development trend of future space-based infrastructure construction.

[0004] At present, on-orbit assembly methods based on space robots are generally targeted at predetermined structures or tasks, such as auxiliary docking of spacecraft cabins, on-orbit assembly / replacement of ground-manufactured components, etc. It is difficult to adapt to changes in multiple space scenarios, such as changes in the size, shape, and performance of the components to be assembled, changes or interference in light during assembly, and complex assembly sequences or the need for reconstruction. There is an urgent need to study an intelligent assembly method with strong versatility to complete more complex on-orbit assembly tasks. Summary of the invention

[0005] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art, propose an autonomous intelligent assembly method of a spatial superstructure based on virtual teaching, adopt an on-orbit three-dimensional reconstruction method to evolve real operations into virtual operations, generate an optimal control method through virtual operations, and solve the problem that the existing on-orbit assembly control methods of space robots are not universal.

[0006] The technical solution of the present invention is:

[0007] A method for autonomous intelligent assembly of a spatial superstructure based on virtual teaching, comprising:

[0008] Step 1, using an on-orbit device to perform three-dimensional reconstruction of the object to be assembled;

[0009] Step 2, transmitting the three-dimensional information of the object to be assembled to the ground virtual simulation system, and the ground virtual simulation system reproduces the virtual information of the object to be assembled;

[0010] Step 3: Based on the ground virtual simulation system, assembling operations on the virtual object to be assembled are performed multiple times to obtain multi-source data in each assembly process to form a manipulation data set;

[0011] Step 4: Train the control data set, take the shortest time and minimum energy consumption as optimization goals, and take obstacle avoidance and on-orbit device capability parameters as constraints to form the optimal control method;

[0012] Step 5: According to the optimal control method, using the mapping relationship between the virtual object to be assembled and the on-orbit object to be assembled, an autonomous control method for the on-orbit device is generated, uploaded to the on-orbit device, and the on-orbit device assembles the object to be assembled.

[0013] Furthermore, in step 1, the on-orbit device includes a space robot, a manipulator, a mechanical sensor, a tactile sensor, and a visual sensor;

[0014] The mechanical sensor is installed between the manipulator and the space robot, the tactile sensor is installed on the clamping surface of the manipulator, and the visual sensor is installed on the top of the space robot or the manipulator;

[0015] The space robot is provided with an autonomous control algorithm for autonomous operation and can also be remotely controlled by a ground virtual simulation system.

[0016] Furthermore, in step 1, the ground virtual simulation system remotely controls the space robot so that the visual sensor has a maximum field of view for the object to be assembled, obtains the shape, contour, position and posture information of the object to be assembled within the field of view, and the space robot scene camera obtains environmental information;

[0017] The ground virtual simulation system remotely controls the space robot to accurately grasp the target object, and uses mechanical sensors and tactile sensors to obtain the size, hardness, and temperature of the object to be assembled, and uses visual sensors to obtain the shape and contour of the object to be assembled within the obstructed field of view;

[0018] The space robot reconstructs the three-dimensional model and material properties of the object to be assembled based on multi-source information from mechanical sensors, tactile sensors, and visual sensors, and synchronously transmits the above information and environmental information to the ground virtual simulation system.

[0019] Further, in step 2, the ground virtual simulation system includes VR / AR glasses, data gloves, and a data glove positioning system;

[0020] The data glove includes a posture sensor, a tactile feedback driver, a force feedback driver, and a glove; the posture sensor obtains the position and posture of the virtual hand in the VR / AR glasses, so that the position and posture of the virtual hand are consistent with the data glove;

[0021] The tactile feedback driver is used to feedback the normal force and tangential force information when the virtual hand clamps the virtual object to be assembled, and the force feedback driver is used to feedback the three-dimensional force and three-dimensional torque when the virtual hand manipulates the virtual object to be assembled.

[0022] Furthermore, in step 2, the operator wears VR / AR glasses and data gloves, and sees the virtual interface through the VR / AR glasses. The virtual interface includes a virtual environment, a virtual target, and a virtual hand. According to the information transmitted in step 1, a corresponding virtual environment and a virtual target are established to reproduce the virtual information of the object to be assembled.

[0023] Furthermore, in step 3, the operator wears gloves, the virtual hand is synchronized with the hand, and the virtual object to be assembled is assembled through human-computer interaction, and multi-source data collected during the assembly process is obtained, specifically including:

[0024] When the virtual hand contacts and clamps the virtual object to be assembled, the tactile driver provides real-time feedback of virtual tactile information, including tactile normal force and tangential force;

[0025] When the virtual hand moves while holding the virtual object to be assembled, the virtual object to be assembled and the virtual hand remain relatively still;

[0026] When the virtual hand holds the virtual object to be assembled for collision and assembly operations, the force feedback driver provides real-time feedback of the three-dimensional force and three-dimensional torque, and finally completes the assembly task of the virtual object to be assembled.

[0027] The multi-source data of the entire assembly process is obtained, including the position and posture of the virtual object to be assembled, the position and posture of the virtual hand, and the tactile normal force, tangential force, three-dimensional control force, and three-dimensional control torque between the two in the above-mentioned time periods.

[0028] Furthermore, in step 2, the virtual hand is synchronized with the real end to reduce the difference, that is, the virtual hand includes a virtual robotic arm and a virtual manipulator, which are mapped to the space robot and the manipulator respectively, so that the sensor information of the virtual robotic arm and the virtual manipulator can be obtained during the virtual assembly process in step 3, thereby improving the data accuracy of the operation data set.

[0029] Furthermore, in step 5, while the on-orbit device is assembling the object to be assembled, the ground operator wears data gloves and monitors the assembly in real time under the space robot's scene camera image. If an error occurs, the space robot is controlled to perform moving, clamping or assembly tasks through ground-to-ground teleoperation to achieve successful assembly.

[0030] The advantages of the present invention compared with the prior art are:

[0031] (1) The method for autonomous intelligent assembly of spatial superstructures based on virtual teaching proposed in the present invention adopts an on-orbit three-dimensional reconstruction method. The on-orbit assembly object is a part, component or assembly in the actual state on-orbit, which can be a part, component or assembly manufactured on the ground, or manufactured on the ground but changed due to the on-orbit environment, or a part, component or assembly unknown in advance (such as a non-cooperative target or an on-orbit manufactured component).

[0032] (2) The autonomous intelligent assembly method of space superstructure based on virtual teaching proposed in the present invention adopts the "virtual teaching + autonomous intelligent learning" method, which uses the space robot to manipulate the assembly object on orbit less times, and transforms the real operation into virtual operation, effectively reducing the manipulation cost and the damage rate of the operation object while ensuring sufficient big data.

[0033] (3) The autonomous intelligent assembly method of space superstructure based on virtual teaching proposed in the present invention adopts the method of "virtual teaching + autonomous intelligent learning" to generate the autonomous control algorithm of the space robot, and at the same time assists in correcting the deficiencies of the autonomous algorithm operation process in real time by means of "ground-to-ground remote operation", further supplementing the database of virtual teaching, and greatly improving the success rate of on-orbit assembly. In addition, compared with the on-orbit assembly method that relies solely on ground-to-ground remote operation, the operation risk is lower, and the success rate and efficiency are higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0035] Figure 1 Schematic diagram of a method for autonomous intelligent assembly of a spatial superstructure based on virtual teaching according to an embodiment of the present invention;

[0036] Figure 2 It is a schematic diagram of the technical route of autonomous intelligent assembly of spatial superstructures based on virtual teaching in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0038] The present invention proposes a spatial superstructure autonomous intelligent assembly method based on virtual teaching, such as Figure 1 , Figure 2As shown, it is composed of a virtual teaching autonomous intelligent assembly method and a ground-to-earth teleoperation on-orbit assembly method, and specifically includes the following steps:

[0039] Step 1, three-dimensionally reconstruct the object to be assembled on-orbit. The object to be assembled can be a large range of different types of components manufactured on the ground, manufactured on the ground but changed due to the on-orbit environment, or unknown in advance (such as non-cooperative targets, on-orbit manufactured components). In this embodiment, the object to be assembled is a space superstructure component, also called a target object.

[0040] The on-orbit 3D reconstruction system includes a space robot, a flexible manipulator and a 3D reconstruction component. Among them, the 3D reconstruction component includes a mechanical sensor, a tactile sensor and a visual sensor. The mechanical sensor is installed between the flexible manipulator and the space robot, the tactile sensor is installed on the clamping surface of the flexible manipulator, and the visual sensor is arranged on the space robot or the flexible manipulator. The ground operator remotely controls the space robot to obtain a good view of the target object from the visual sensor, and uses the visual sensor and its 3D reconstruction algorithm to obtain the shape, contour, position and posture information of the target object within the field of view; the ground operator remotely controls the space robot to accurately clamp the target object, on the one hand, using the force / tactile sensor to obtain the force / tactile information such as the size, hardness, and temperature of the target object, and on the other hand, using the visual sensor to obtain the shape, contour and other information of the original target object within the field of view that is blocked. Furthermore, through the fusion of multi-source information of force tactile sensors, tactile sensors and visual sensors, deep learning models such as convolutional neural networks and adversarial generative networks are used to perform information fusion through shared embedding representation, accurately reconstruct the 3D model, material properties and other information of the target object, and synchronously transmit these information of the target object to the ground.

[0041] Step 2: The information of the object to be assembled is transmitted to the ground through the ground-ground data, and the virtual information of the object to be assembled is reproduced through VR or AR glasses.

[0042] VR / AR scene reproduction involves operators, VR / AR glasses, data gloves, and corresponding positioning systems. Data gloves include posture sensors, tactile / force feedback drivers, and gloves. Among them, the position and posture of the virtual hand are obtained by the posture sensor of the data glove, so the position and posture of the virtual hand are consistent with the data glove. The tactile feedback driver is used to feedback the tactile information such as the normal force and tangential force when the virtual hand clamps the virtual target. The force feedback driver is used to feedback the three-dimensional force, three-dimensional torque and other force information when the virtual hand manipulates the virtual target. The operator wears VR / AR glasses and data gloves to see the virtual interface, which includes the virtual environment, virtual target, and virtual hand. Among them, the virtual environment is obtained by the spatial three-dimensional reconstruction part. The material properties, three-dimensional model, initial position and posture information of the virtual target are obtained by the spatial three-dimensional reconstruction part.

[0043] In step 3, the operator uses data gloves and VR / AR glasses to manipulate virtual objects, such as clamping and assembling, and simultaneously generates big data such as clamping force, manipulation torque, and virtual object posture information.

[0044] The operator moves the hand (data glove) and the virtual hand is synchronized with it. When the virtual hand contacts and clamps the virtual target, the data glove uses the tactile driver to feedback virtual tactile information in real time, including tactile normal force, tangential force and other information; when the virtual hand clamps the virtual target and moves, the virtual target and the virtual hand are relatively stationary; when the virtual hand clamps the virtual target for collision, assembly and other operations, the data glove uses the force feedback driver to feedback the three-dimensional force and three-dimensional torque in real time, and finally completes the assembly task of the virtual target. The above is a single teaching operation, and multi-source data of the entire teaching process can be obtained, including the virtual target position / posture, virtual hand position / posture and tactile normal force / tangential force / control force / torque between the two in each time period. The above data sets can be obtained through multiple teaching operations.

[0045] Step 4: Use machine learning methods to train the big data generated in step 3 to form the optimal control algorithm.

[0046] Taking the shortest time and minimum energy consumption as optimization goals, and obstacle avoidance and space robot component capability parameters as constraints, the deep deterministic policy gradient (DDPG) algorithm in deep reinforcement learning is used, assisted by the YOLO algorithm for visual perception, as well as imitation learning and hindsight experience replay (HER) technology to improve learning efficiency and performance, so as to obtain the optimal control algorithm, that is, the optimal virtual target position / posture, virtual hand position / posture and tactile normal force / tangential force / control force / torque and other information control algorithm in each time period.

[0047] Step 5: Based on the optimal control algorithm of step 4, the mapping relationship between virtual objects and real objects is used to generate an autonomous control algorithm for the space robot, and finally complete the on-orbit assembly of the space superstructure.

[0048] The above optimal manipulation algorithm is further optimized through robot control parameter debugging according to the characteristics of space robot components, and finally sent to the space robot for autonomous execution. In addition, in order to accurately apply the above optimal manipulation algorithm to the space robot, the virtual hand can be synchronized with the real end to reduce the difference, that is, the virtual hand is a virtual robot arm and manipulator, and the virtual robot arm and virtual manipulator are mapped to the space robot and flexible manipulator respectively. The virtual teaching process can accurately obtain the sensor information of the virtual robot arm and virtual manipulator, supplement the data set of step 3 above, and effectively reduce the manipulation cost and the damage rate of the operation object under the condition of ensuring sufficient big data.

[0049] Step 6, the ground-to-earth remote-controlled on-orbit assembly method, is to perform on-orbit assembly by directly remotely controlling the space robotic arm by the ground operator. It is also a real-time correction and data supplement to the virtual teaching autonomous intelligent assembly method, which is used to improve the success rate of operations.

[0050] The ground operator wears data gloves and controls the space robot to perform various tasks such as moving, clamping, and assembly in real time (or with a short delay) under the space robot scene camera image; or, the ground operator wears VR / AR glasses and data gloves to control the virtual robotic arm in the virtual interface. In this process, the space robot components are mapped to it in real time, which plays a role in expanding the field of vision. In the autonomous operation and assembly process of the space robot in step 5, the ground-to-earth teleoperation can be used for real-time monitoring. At the same time, once an error or poor error occurs, the space robot operation task can be suspended, and the space robot action can be quickly corrected to achieve the purpose of successful operation and assembly. This type of operation data will be used for retraining of the data set in step 3 to further improve the versatility of the algorithm.

[0051] The above-described embodiments are only preferred specific implementations of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for autonomous intelligent assembly of a spatial superstructure based on virtual teaching, characterized in that: include: Step 1, using an on-orbit device to perform three-dimensional reconstruction of the object to be assembled; Step 2, transmitting the three-dimensional information of the object to be assembled to the ground virtual simulation system, and the ground virtual simulation system reproduces the virtual information of the object to be assembled; Step 3: Based on the ground virtual simulation system, assembling operations on the virtual object to be assembled are performed multiple times to obtain multi-source data in each assembly process to form a manipulation data set; Step 4: Train the control data set, take the shortest time and minimum energy consumption as optimization goals, and take obstacle avoidance and on-orbit device capability parameters as constraints to form the optimal control method; Step 5: According to the optimal control method, using the mapping relationship between the virtual object to be assembled and the on-orbit object to be assembled, an autonomous control method for the on-orbit device is generated, uploaded to the on-orbit device, and the on-orbit device assembles the object to be assembled.

2. The autonomous intelligent assembly method of a spatial superstructure based on virtual teaching according to claim 1 is characterized in that: In step 1, the on-orbit device includes a space robot, a manipulator, a mechanical sensor, a tactile sensor, and a visual sensor; The mechanical sensor is installed between the manipulator and the space robot, the tactile sensor is installed on the clamping surface of the manipulator, and the visual sensor is installed on the top of the space robot or the manipulator; The space robot is provided with an autonomous control algorithm for autonomous operation and can also be remotely controlled by a ground virtual simulation system.

3. The autonomous intelligent assembly method of spatial superstructure based on virtual teaching according to claim 2 is characterized in that: In step 1, the ground virtual simulation system remotely controls the space robot so that the visual sensor has the maximum field of view for the object to be assembled, obtains the shape, contour, position and posture information of the object to be assembled within the field of view, and the space robot scene camera obtains environmental information; The ground virtual simulation system remotely controls the space robot to accurately grasp the target object, and uses mechanical sensors and tactile sensors to obtain the size, hardness, and temperature of the object to be assembled, and uses visual sensors to obtain the shape and contour of the object to be assembled within the obstructed field of view; The space robot reconstructs the three-dimensional model and material properties of the object to be assembled based on multi-source information from mechanical sensors, tactile sensors, and visual sensors, and synchronously transmits the above information and environmental information to the ground virtual simulation system.

4. The method for autonomous intelligent assembly of a spatial superstructure based on virtual teaching according to claim 1 is characterized in that: In step 2, the ground virtual simulation system includes VR / AR glasses, data gloves, and a data glove positioning system; The data glove includes a posture sensor, a tactile feedback driver, a force feedback driver, and a glove; the posture sensor obtains the position and posture of the virtual hand in the VR / AR glasses, so that the position and posture of the virtual hand are consistent with the data glove; The tactile feedback driver is used to feedback the normal force and tangential force information when the virtual hand clamps the virtual object to be assembled, and the force feedback driver is used to feedback the three-dimensional force and three-dimensional torque when the virtual hand manipulates the virtual object to be assembled.

5. The autonomous intelligent assembly method of spatial superstructure based on virtual teaching according to claim 1 is characterized in that: In step 2, the operator wears VR / AR glasses and data gloves and sees the virtual interface through the VR / AR glasses. The virtual interface includes a virtual environment, a virtual target, and a virtual hand. According to the information transmitted in step 1, the corresponding virtual environment and virtual target are established to reproduce the virtual information of the object to be assembled.

6. The autonomous intelligent assembly method of a spatial superstructure based on virtual teaching according to claim 4 is characterized in that: In step 3, the operator wears gloves, the virtual hand is synchronized with the hand, and the virtual object to be assembled is assembled through human-computer interaction, and multi-source data collected during the assembly process is obtained, including: When the virtual hand contacts and clamps the virtual object to be assembled, the tactile driver provides real-time feedback of virtual tactile information, including tactile normal force and tangential force; When the virtual hand moves while holding the virtual object to be assembled, the virtual object to be assembled and the virtual hand remain relatively still; When the virtual hand holds the virtual object to be assembled for collision and assembly operations, the force feedback driver provides real-time feedback of the three-dimensional force and three-dimensional torque, and finally completes the assembly task of the virtual object to be assembled. The multi-source data of the entire assembly process is obtained, including the position and posture of the virtual object to be assembled, the position and posture of the virtual hand, and the tactile normal force, tangential force, three-dimensional control force, and three-dimensional control torque between the two in the above-mentioned time periods.

7. The method for autonomous intelligent assembly of a spatial superstructure based on virtual teaching according to claim 2 is characterized in that: In step 2, the virtual hand is synchronized with the real end to reduce the difference, that is, the virtual hand includes a virtual robotic arm and a virtual manipulator, which are mapped to the space robot and the manipulator respectively, so that the sensor information of the virtual robotic arm and the virtual manipulator can be obtained during the virtual assembly process in step 3, thereby improving the data accuracy of the operation data set.

8. The method for autonomous intelligent assembly of a spatial superstructure based on virtual teaching according to claim 1 is characterized in that: In step 5, while the on-orbit device is assembling the object to be assembled, the ground operator wears data gloves and monitors the assembly in real time under the space robot's scene camera image. If an error occurs, the space robot is controlled to move, clamp or assemble tasks through ground-to-earth teleoperation to achieve successful assembly.

Images