Robot configuration system and method for simulating the operation of a real robotic arm

TWI945940BActive Publication Date: 2026-11-11SOLOMON TECHNOLOGY CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
TW114126548
Authority / Receiving Office
TW · TW
Patent Type
Patents
Current Assignee / Owner
Filing Date
2025-07-14
Publication Date
2026-11-11
Estimated Expiration
2045-07-13

Smart Images

  • Figure TWG2TB001920692_001
    Figure TWG2TB001920692_001
  • Figure TWG2TB001920692_002
    Figure TWG2TB001920692_002
  • Figure TWG2TB001920692_003
    Figure TWG2TB001920692_003
Patent Text Reader

Abstract

A method for simulating the operation of a real robotic arm includes: (A) simulating and rendering a virtual scene corresponding to a real scene based on a set of robotic arm configuration parameters and a set of scene configuration parameters, wherein the virtual scene is configured with a plurality of virtual objects, a target area, and a virtual robotic arm corresponding to the real robotic arm; (B) generating object data in the virtual scene in real time; (C) using an object recognition model to select one of the virtual objects as a target object based on the object data and further determining the position and orientation of the target object; (D) generating a motion plan for the virtual robotic arm based on the position and orientation of the target object, the set of robotic arm configuration parameters, and the set of scene configuration parameters; (E) performing real-time simulation and rendering based on the motion plan to generate a simulated animation; repeating steps (B) to (E) until all the virtual objects have been moved to the target area; and outputting the set of robotic arm configuration parameters and the set of sensing module configuration parameters.
Need to check novelty before this filing date? Find Prior Art

Claims

1. A method for simulating the operation of at least one real robotic arm, the method being implemented by a robot configuration system, the robot configuration system comprising a user interface, a control module, a scene simulation module, and a virtual sensing module, the user interface being operable by a user to set a set of robotic arm configuration parameters related to the at least one real robotic arm, a set of scene configuration parameters related to a real scene, and a set of sensing module configuration parameters related to a real sensing module, the control module having an object recognition model, the method comprising the following steps: (A) the scene simulation module, based on the set of robotic arm configuration parameters and the set of scene configuration parameters, simulates and renders a virtual scene corresponding to the real scene, the virtual scene being configured with a plurality of virtual objects, a target area, and a virtual robotic arm corresponding to the at least one real robotic arm; (B) the virtual sensing module, based on the set of sensing module configuration parameters, generates object data in the virtual scene in real time and transmits the object data to the control module; (C) The control module uses the object recognition model to select one of the virtual objects as a target object based on the object data and further determines the position and orientation of the target object; (D) The control module generates a motion plan for multiple consecutive actions of the virtual robotic arm based on the position and orientation of the target object, the set of robotic arm configuration parameters, and the set of scene configuration parameters. The actions of the virtual robotic arm include moving to the position, gripping the target object, moving to the target area, and releasing the target object; (E) The scene simulation module performs real-time simulation and rendering based on the motion plan to generate a simulation animation of the virtual robotic arm performing the actions in the virtual scene; (F) After the virtual robotic arm releases the target object, steps (B) to (E) are repeated until all the virtual objects in the virtual scene of the simulated animation generated in step (E) have been moved to the target area; and (G) After step (F), the control module outputs the set of robotic arm configuration parameters and the set of sensing module configuration parameters, wherein the set of robotic arm configuration parameters and the set of sensing module configuration parameters can be used to configure the at least one real robotic arm and the real sensing module in the real scene.

2. The method described in claim 1, wherein the robot configuration system further includes a training module, wherein step (B) further includes transmitting the object data to the training module, and the method further includes the following steps: (H) the training module uses the object data received in step (B) to train the object recognition model and transmits a set of model parameters of the trained object recognition model to the control module, and the control module updates the object recognition model in the control module according to the received set of model parameters.

3. The method as described in claim 1, wherein step (G) further comprises: The control module calculates the operation time required to execute steps (B) to (F); The control module determines whether the operation time is less than a preset time; and when the operation time is less than the preset time, the control module outputs the configuration parameters of the robotic arm and the configuration parameters of the sensing module.

4. The method as described in claim 2, wherein step (H) is to train the object recognition model using the Solomon SolVision software program.

5. The method as described in claim 2, wherein step (B) comprises: The virtual sensing module receives image data of virtual objects in the virtual scene from the scene simulation module based on a general scene description data framework; the virtual sensing module generates 3D point cloud data and RGB images of the virtual objects based on the image data; and the virtual sensing module transmits the 3D point cloud and the RGB image as the object data to the control module and the training module.

6. The method of claim 1, wherein the set of robotic arm configuration parameters is represented in a uniform robot description format and relates to the control of the posture of the at least one real robotic arm and the position of at least one joint of the at least one real robotic arm.

7. The method as described in claim 1, wherein step (D) is performed using a motion planning algorithm based on a robot operating system, hardware acceleration technology, and optimization technology.

8. The method as described in claim 7, wherein step (D) is to use the Movet software development kit and the NVIDIA cuMotion library in the robot operation system to generate the motion plan.

9. The method as described in claim 1, wherein step (E) is to generate the simulated animation based on a data framework of a general scene description.

10. The method of claim 1, wherein the object recognition model is a deep learning model that uses multimodal inputs to determine the position and orientation of the target object.

11. The method as described in claim 10, wherein the object identification model is the FoundationPose model.

12. The method as described in claim 1, wherein the user interface and the control module are integrated into an automated control system.

13. The method as described in claim 12, wherein the automated control system is implemented using the Solomon AccuPick system.

14. The method of claim 1, wherein step (E) is performed using a platform that supports a robotic operating system.

15. The method as described in claim 14, wherein the platform is the NVIDIA Omniverse platform.

16. The method of claim 15, wherein step (E) is to generate the simulation animation using the NVIDIA Isaac Sim platform provided by the NVIDIA Omniverse platform.

17. The method of claim 1, wherein step (B) is to use an application interface to generate 3D point cloud data and RGB images of the virtual objects as object data.

18. The method of claim 17, wherein step (B) is to generate the object data using the NVIDIA Omniverse Sensor RTX application interface.

19. A robot configuration system, comprising: a user interface for a user to set a set of robot arm configuration parameters related to at least one real robot arm, a set of scene configuration parameters related to a real scene, and a set of sensing module configuration parameters related to a real sensing module; and a scene simulation module connected to the user interface, the scene simulation module being configured to simulate and render a virtual scene corresponding to the real scene based on the set of robot arm configuration parameters and the set of scene configuration parameters, the virtual scene having configured a plurality of virtual objects, a target area, and a virtual robot arm corresponding to at least one real robot arm; A virtual sensing module, connected to the user interface and the scene simulation module, is used to generate object data in the virtual scene in real time according to the configuration parameters of the sensing module. A control module, connected to the user interface, the scene simulation module, and the virtual sensing module, has an object recognition model. The control module uses the object recognition model to select one of the virtual objects as a target object based on the object data and further determines the position and orientation of the target object. The control module is also used to generate a motion plan for multiple consecutive actions of the virtual robotic arm based on the position and orientation of the target object, the robotic arm configuration parameters, and the scene configuration parameters. These actions of the virtual robotic arm include moving to the position, gripping the target object, moving to the target area, and releasing the target object. The virtual sensing module also transmits the object data to the control module. The scene simulation module is also used to perform real-time simulation and rendering based on the motion plan to generate a simulated animation of the virtual robotic arm performing the actions in the virtual scene. When all the virtual objects in the virtual scene of the simulated animation have been moved to the target area, the control module is also used to output the set of robotic arm configuration parameters and the set of sensing module configuration parameters. The set of robotic arm configuration parameters and the set of sensing module configuration parameters can be used to configure at least one real robotic arm and the real sensing module in the real scene.

20. The robot configuration system of claim 19 further comprises: a training module connected to the virtual sensing module and the control module, and configured to receive object data from the virtual sensing module, train an object recognition model using the received object data, and transmit a set of model parameters of the trained object recognition model to the control module, wherein the control module is further configured to update the object recognition model in the control module according to the received set of model parameters.

Citation Information

Patent Citations

  • Robot simulation device

    TW202247961A

  • Method for generating an operating instruction for a robotic arm

    TW202521313A

  • Surgical robotic arm control system and control method thereof

    TWI880123B

  • Simulation Apparatus And Robot Control Apparatus

    US20180297202A1

  • Robot simulation device

    US20240123611A1