Robot configuration system and method for simulating the operation of a real robotic arm
Patent Information
- Application Number
- TW114126548
- Authority / Receiving Office
- TW · TW
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2025-07-14
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2045-07-13
Smart Images

Figure TWG2TB001920692_001 
Figure TWG2TB001920692_002 
Figure TWG2TB001920692_003
Abstract
Claims
1. A method for simulating the operation of at least one real robotic arm, the method being implemented by a robot configuration system, the robot configuration system comprising a user interface, a control module, a scene simulation module, and a virtual sensing module, the user interface being operable by a user to set a set of robotic arm configuration parameters related to the at least one real robotic arm, a set of scene configuration parameters related to a real scene, and a set of sensing module configuration parameters related to a real sensing module, the control module having an object recognition model, the method comprising the following steps: (A) the scene simulation module, based on the set of robotic arm configuration parameters and the set of scene configuration parameters, simulates and renders a virtual scene corresponding to the real scene, the virtual scene being configured with a plurality of virtual objects, a target area, and a virtual robotic arm corresponding to the at least one real robotic arm; (B) the virtual sensing module, based on the set of sensing module configuration parameters, generates object data in the virtual scene in real time and transmits the object data to the control module; (C) The control module uses the object recognition model to select one of the virtual objects as a target object based on the object data and further determines the position and orientation of the target object; (D) The control module generates a motion plan for multiple consecutive actions of the virtual robotic arm based on the position and orientation of the target object, the set of robotic arm configuration parameters, and the set of scene configuration parameters. The actions of the virtual robotic arm include moving to the position, gripping the target object, moving to the target area, and releasing the target object; (E) The scene simulation module performs real-time simulation and rendering based on the motion plan to generate a simulation animation of the virtual robotic arm performing the actions in the virtual scene; (F) After the virtual robotic arm releases the target object, steps (B) to (E) are repeated until all the virtual objects in the virtual scene of the simulated animation generated in step (E) have been moved to the target area; and (G) After step (F), the control module outputs the set of robotic arm configuration parameters and the set of sensing module configuration parameters, wherein the set of robotic arm configuration parameters and the set of sensing module configuration parameters can be used to configure the at least one real robotic arm and the real sensing module in the real scene.
2. The method described in claim 1, wherein the robot configuration system further includes a training module, wherein step (B) further includes transmitting the object data to the training module, and the method further includes the following steps: (H) the training module uses the object data received in step (B) to train the object recognition model and transmits a set of model parameters of the trained object recognition model to the control module, and the control module updates the object recognition model in the control module according to the received set of model parameters.
3. The method as described in claim 1, wherein step (G) further comprises: The control module calculates the operation time required to execute steps (B) to (F); The control module determines whether the operation time is less than a preset time; and when the operation time is less than the preset time, the control module outputs the configuration parameters of the robotic arm and the configuration parameters of the sensing module.
4. The method as described in claim 2, wherein step (H) is to train the object recognition model using the Solomon SolVision software program.
5. The method as described in claim 2, wherein step (B) comprises: The virtual sensing module receives image data of virtual objects in the virtual scene from the scene simulation module based on a general scene description data framework; the virtual sensing module generates 3D point cloud data and RGB images of the virtual objects based on the image data; and the virtual sensing module transmits the 3D point cloud and the RGB image as the object data to the control module and the training module.
6. The method of claim 1, wherein the set of robotic arm configuration parameters is represented in a uniform robot description format and relates to the control of the posture of the at least one real robotic arm and the position of at least one joint of the at least one real robotic arm.
7. The method as described in claim 1, wherein step (D) is performed using a motion planning algorithm based on a robot operating system, hardware acceleration technology, and optimization technology.
8. The method as described in claim 7, wherein step (D) is to use the Movet software development kit and the NVIDIA cuMotion library in the robot operation system to generate the motion plan.
9. The method as described in claim 1, wherein step (E) is to generate the simulated animation based on a data framework of a general scene description.
10. The method of claim 1, wherein the object recognition model is a deep learning model that uses multimodal inputs to determine the position and orientation of the target object.
11. The method as described in claim 10, wherein the object identification model is the FoundationPose model.
12. The method as described in claim 1, wherein the user interface and the control module are integrated into an automated control system.
13. The method as described in claim 12, wherein the automated control system is implemented using the Solomon AccuPick system.
14. The method of claim 1, wherein step (E) is performed using a platform that supports a robotic operating system.
15. The method as described in claim 14, wherein the platform is the NVIDIA Omniverse platform.
16. The method of claim 15, wherein step (E) is to generate the simulation animation using the NVIDIA Isaac Sim platform provided by the NVIDIA Omniverse platform.
17. The method of claim 1, wherein step (B) is to use an application interface to generate 3D point cloud data and RGB images of the virtual objects as object data.
18. The method of claim 17, wherein step (B) is to generate the object data using the NVIDIA Omniverse Sensor RTX application interface.
19. A robot configuration system, comprising: a user interface for a user to set a set of robot arm configuration parameters related to at least one real robot arm, a set of scene configuration parameters related to a real scene, and a set of sensing module configuration parameters related to a real sensing module; and a scene simulation module connected to the user interface, the scene simulation module being configured to simulate and render a virtual scene corresponding to the real scene based on the set of robot arm configuration parameters and the set of scene configuration parameters, the virtual scene having configured a plurality of virtual objects, a target area, and a virtual robot arm corresponding to at least one real robot arm; A virtual sensing module, connected to the user interface and the scene simulation module, is used to generate object data in the virtual scene in real time according to the configuration parameters of the sensing module. A control module, connected to the user interface, the scene simulation module, and the virtual sensing module, has an object recognition model. The control module uses the object recognition model to select one of the virtual objects as a target object based on the object data and further determines the position and orientation of the target object. The control module is also used to generate a motion plan for multiple consecutive actions of the virtual robotic arm based on the position and orientation of the target object, the robotic arm configuration parameters, and the scene configuration parameters. These actions of the virtual robotic arm include moving to the position, gripping the target object, moving to the target area, and releasing the target object. The virtual sensing module also transmits the object data to the control module. The scene simulation module is also used to perform real-time simulation and rendering based on the motion plan to generate a simulated animation of the virtual robotic arm performing the actions in the virtual scene. When all the virtual objects in the virtual scene of the simulated animation have been moved to the target area, the control module is also used to output the set of robotic arm configuration parameters and the set of sensing module configuration parameters. The set of robotic arm configuration parameters and the set of sensing module configuration parameters can be used to configure at least one real robotic arm and the real sensing module in the real scene.
20. The robot configuration system of claim 19 further comprises: a training module connected to the virtual sensing module and the control module, and configured to receive object data from the virtual sensing module, train an object recognition model using the received object data, and transmit a set of model parameters of the trained object recognition model to the control module, wherein the control module is further configured to update the object recognition model in the control module according to the received set of model parameters.
Citation Information
Patent Citations
Robot simulation device
TW202247961A
Method for generating an operating instruction for a robotic arm
TW202521313A
Surgical robotic arm control system and control method thereof
TWI880123B
Simulation Apparatus And Robot Control Apparatus
US20180297202A1
Robot simulation device
US20240123611A1
