Simulation system for household robot tasks and data collection

Abstract

A simulation system for household robot tasks and data collection, comprising: a data collection module, a Unity module, a Python module, a communication module, a VR module and a resource file module. The present invention can support a variety of object attributes, an interactive simulation system for household robot tasks, and the system can support data acquisition for deep learning training, thereby reducing the cost of data annotation.

Description

Technical Field

[0001] The present invention relates to a technology in the field of automatic control, in particular to a simulation system based on Unity3D for household robot tasks and data collection. Background Art

[0002] Simulation environment plays an important role in robotics. Robots need to simulate and perform tasks in simulation. The design and verification of an algorithm will rely on the simulation environment. The existing simulation environment cannot fully support tasks in various household scenarios. The main difficulty lies in the comprehensive simulation of properties such as rigid bodies, joint bodies, flexible bodies, fluids, and transparent bodies. Summary of the invention

[0003] In view of the fact that the prior art only supports objects with rigid body attributes and the trained models cannot be directly migrated to real robots, the present invention greatly reduces the scalability deficiency of the environment and proposes a simulation system for household robot tasks and data collection, which can support a variety of object attributes and is an interactive simulation system for household robot tasks. At the same time, the system can support data acquisition for deep learning training, thereby reducing the cost of data annotation.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a simulation system for household robot tasks and data collection based on Unity3D, comprising: a data collection module, a Unity module, a Python module, a communication module, a VR module and a resource file module, wherein: the data collection module uses a camera and a laser rangefinder to obtain a point cloud, and uses a key point matching algorithm between different image frames to align the point clouds under different viewing angles, thereby converting a real object into a model file and outputting it to the resource file module; the Python module calls the model file in the resource file module, and simultaneously obtains resource information data such as the number and object attributes corresponding to the model file from a database of the resource file module, and outputs it to the Unity module through the communication module; the Unity module uses the camera and the model file to obtain resource information data such as the number and object attributes corresponding to the model file, and outputs it to the Unity module through the communication module; the Unity module uses the camera and the model file to obtain the resource information data such as the number and object attributes corresponding to the model file. The position parameters of the file are used, and the real-time rendering technology of Unity3D software is used to render the model file in the three-dimensional scene; the VR module is connected to the Unity module, and the hand model information of the VR module is fed back to the Unity module in real time, and the real-time rendering technology of Unity3D software is used to reflect the hand model in the VR module in real time in the three-dimensional scene, and can interact with objects in the scene; at the same time, the Unity module will use the physical engine to simulate different types of model files, and output the scene information to the Python module through the communication module; the Python module inputs the received results into the reinforcement learning algorithm, obtains the data instructions for controlling the robot, and outputs them to the Unity module through the communication module to control the robot, thereby interacting.

[0006] Technical Effects

[0007] The present invention can support serialization protocols and deserialization protocols for lossless conversion of raw data of any quantity, order or type. The Unity module and the Python module can realize the intercommunication of any information under any conditions, so that all functions provided by the Unity module can also be directly called by the Python module. At the same time, the algorithm implemented by the Python module can be directly applied to the robot control in the Unity module. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a schematic diagram of the system of the present invention;

[0009] Figure 2 It is a schematic diagram of the data acquisition module;

[0010] Figure 3 It is a schematic diagram of the communication module;

[0011] Figure 4 It is a schematic diagram of the embodiment flow chart;

[0012] Figure 5 The graph is a result of the embodiment. DETAILED DESCRIPTION

[0013] like Figure 1 As shown, this embodiment includes: a data acquisition module, a Unity module, a Python module, a communication module, a VR module and a resource file module, wherein: the data acquisition module uses a camera and a laser rangefinder to obtain a point cloud, and uses a key point matching algorithm between different image frames to align the point clouds under different perspectives, thereby converting a real object into a model file and outputting it to the resource file module; the Python module calls the model file in the resource file module, and at the same time obtains resource information data such as the number and object attributes corresponding to the model file from the database of the resource file module, and outputs it to the Unity module through the communication module; the Unity module uses Unity according to the position parameters of the camera and the model file The real-time rendering technology of 3D software renders the model file in the three-dimensional scene; the VR module is connected to the Unity module, and the hand model information of the VR module is fed back to the Unity module in real time, and the real-time rendering technology of Unity3D software is used to reflect the hand model in the VR module in real time in the three-dimensional scene, and can interact with objects in the scene; at the same time, the Unity module will use the physical engine to simulate different types of model files, and output the scene information to the Python module through the communication module; the Python module inputs the received results into the reinforcement learning algorithm, obtains the data instructions for controlling the robot, and outputs them to the Unity module through the communication module to control the robot, thereby interacting.

[0014] like Figure 2 As shown, the data acquisition module includes: a scanner, a scanning platform and annotation software, wherein: the scanner is a handheld 3D scanner, the scanning platform is an ordinary turntable with a certain number of tracking points attached to it, so that the scanner can match the current frame image with the previous frame image, and the user can fill in the semantic classification, quality and attributes of the object with the help of the annotation software.

[0015] like Figure 3 As shown, the communication module includes: a serialization unit, a deserialization unit, a gRPC transmission unit, a Unity management unit and a Python management unit, wherein: the serialization unit serializes the original data according to the serialization protocol to obtain a serialized byte stream.

[0016] The serialization protocol is as follows: the protocol unit first traverses the unserialized data, obtains the data type of each data monomer in the data, writes the data type into a string in order, and bytecodes the string as the frame header of the serialized data, and the remaining data is bytecode and written after the frame header. The serialization unit outputs the serialized byte stream to the gRPC transmission unit. The gRPC transmission unit uses gRPC technology to output the byte stream to the Unity management unit or the Python management unit. After receiving the byte stream, the Unity management unit or the Python management unit outputs the serialized byte stream to the deserialization unit, and waits for the deserialization unit to restore the data. The deserialization unit deserializes the serialized byte stream according to the deserialization protocol to restore the original data.

[0017] The deserialization protocol is as follows: the protocol unit first obtains the serialized data frame header, translates the frame header bytes into a string and obtains the data type of each data unit in the serialized data, and then determines how many bytes in the serialized data to read and translate them into data of the corresponding type according to the data type, thereby realizing the deserialization of the data. The protocol can ensure that any amount, order or type of data can be losslessly serialized and deserialized. The deserialized raw data will be returned to the Unity management unit or the Python management unit, and then returned to the Unity module or the Python module, and finally a communication task is completed.

[0018] The resource information data includes: hash code of resource file, location of model, posture of model, attributes of model, master-slave relationship of model in the scene.

[0019] The VR module includes: VR equipment and VR program, wherein: the VR equipment includes VR glasses that use gravity sensors to obtain the user's posture in the VR scene and display the scene picture at the current viewing angle for the user, and VR gloves that use magnetic sensors to capture the changes in the user's hand posture. The user posture and head posture obtained by the VR device will be fed back to the VR program in real time. The VR program will filter and smooth the above data, and output the processed data to the Unity module through the serial port function of Unity3D. At the same time, the VR program will wait for the Unity module to return a screenshot of the simulation result in the current user posture, and output the screenshot to the VR glasses for the user to watch.

[0020] The physical engine includes: PhysX, Flex and Obi, wherein: PhysX is used for the simulation of rigid bodies and articulated bodies, Flex is used for the simulation of fluids, and Obi is used for the simulation of cloth and soft bodies.

[0021] The scene information includes: the encoding, semantic classification, position, posture, speed, mass, center of mass position, position of each particle, 3D detection frame, camera parameters under the current viewing angle, RGB image, depth image, normal image, instance segmentation mask, and optical flow image of each object in the scene.

[0022] The position, posture, and three-dimensional detection frame are important parameters for the robot to perceive objects.

[0023] The speed, mass, center of mass position, and position of each particle are extremely important physical properties of the object itself.

[0024] The data instruction includes: instruction name, object number and instruction parameters, wherein the instruction name is a predefined name, such as force, translation, rotation, etc.; the instruction parameters are the parameters required for the execution of the instruction, and correspond one-to-one with the instruction name.

[0025] The data acquisition module includes: a scanner, a scanning platform and annotation software, wherein the scanner is a handheld 3D scanner, and the scanning platform is an ordinary turntable with a certain number of tracking points attached to it, so that the scanner can match the current frame image with the previous frame image. The user uses the annotation software to fill in the semantic classification, mass, center of mass position and attributes of the object.

[0026] The Unity module includes: a data queue unit and a simulation program unit, wherein: the data queue unit is responsible for receiving data from the communication module and storing it in a queue; the simulation program unit obtains the target rotation angle position of each joint of the current robotic arm from the data queue unit, and applies the result to the robotic arm in the simulation environment to simulate the state of each object in the scene after the robotic arm moves, and then sends the scene information to the communication module; the simulation program unit also receives and simulates the information sent by the VR module through the serial port, and the corresponding simulation result will be returned to the VR module.

[0027] The Python module includes: a data queue unit and a data processing unit, wherein: the data queue unit is responsible for receiving the data transmitted by the communication module and storing it in the queue; the data processing unit obtains the current simulation result from the data queue unit, uses the reinforcement learning algorithm to make predictions, obtains the target angular position of each joint of the robotic arm, and sends the result to the communication module.

[0028] The resource file module stores model files, and the types of model files supported include OBJ, FBX, DAE and STL. The resource file module includes: model files and model databases, wherein: the model files are AssetBundle files supported by Unity3D that contain models of all scanned objects to ensure that they can be dynamically loaded into the scene when the Unity module is running; the model database is an SQL database with the hash code of the model file as the primary key, and the database records the semantic type, mass, center of mass position, and attributes of each model.

[0029] This embodiment uses the Linux platform to design corresponding programs. A scenario is built using the Franka robot arm grasping an object as an example. The robot's actions are predicted through a reinforcement learning algorithm, so that the robot can complete the grasping task.

[0030] like Figure 4 As shown, this embodiment relates to a simulation method based on the above system, comprising the following steps:

[0031] Step 1. The user places the object to be grasped by the robot on the scanning table and turns on the scanner for scanning. After the scanner is turned on, the system will automatically pop up the annotation software. During the scanning process, the user needs to slowly rotate the turntable for one circle, and the rotation time must not be less than 1 minute. After the scan is completed, the 3D model of the object will appear in the annotation software, and the user needs to annotate it with semantic classification, name, and quality. After the annotation is completed, the model is automatically added to the resource file module.

[0032] Step 2: By running the Python module command, the user can call up the Unity module interface. In the interface, the user can choose to add the Franka robot arm to any position in the scene and add the objects just scanned to the scene to build the basic environment.

[0033] Step 3: The user chooses whether to use the VR module to intervene in the robot arm for control. If the VR module is chosen, the user needs to wear VR equipment, and the current scene can be presented in the VR glasses. The user can control the end gripper of the Franka robot arm to grab the object by changing the position and posture of the right hand and the opening and closing of the five fingers until the object is successfully grabbed.

[0034] Step 4: If the user does not choose to use the VR module, the reinforcement learning stage will begin. The user can enter the number of reinforcement learning trainings in the Unity module interface. During the training process, the upper left corner of the interface will display the current training status, success rate, estimated remaining time and other information in real time. After waiting for the training to be completed, the interface will directly start using the trained model to display the training results to the user. If the user is not satisfied with the training results, he can return to step 2 to adjust the position of the robot arm and the block in the scene; otherwise, it ends.

[0035] This embodiment can record the data of the user controlling the robotic arm through the VR module to facilitate data support for other academic research, and can also directly migrate the reinforcement learning training model to the real robotic arm for further algorithm evaluation.

[0036] In this embodiment, the Franka robot is built into the resource file module. In step 2, the user can directly see the thumbnail of the robot in the Unity module interface and can directly drag it to any position.

[0037] In this embodiment, the object position, posture, speed, robot arm gripper position and speed in the scene information are observations passed into the reinforcement learning algorithm, and the model in the algorithm will calculate the position of the gripper at the next moment relative to the current motion position. The Python module will calculate the actual position of the robot arm from the current gripper position and the predicted value of the model, and use the inverse kinematics formula to calculate the rotation angle of each axis of each robot arm, which will be output to the Unity module as data instructions. Through continuous iteration of the above data, the reinforcement learning algorithm makes the model eventually converge. The trained model can be migrated to the real robot arm for further verification and optimization.

[0038] After specific actual experiments, in the environment setting of using Franka robot arm, the initial position of the object is randomly generated in the 30cm*30cm plane, and the target position is randomly generated in the 30cm*30cm*20cm space, the SAC+HER reinforcement learning algorithm is used to train the above tasks for 1,000,000 steps with 5 different random seeds. The final average success rate is 91.2%. The training process curve is shown in Figure 5 shown.

[0039] Compared with the existing technology, all simulated objects in this method are based on real object scanning, without the need for time-consuming CAD modeling, and can obtain more realistic rendering and simulation effects, and can support any robot and object to interact and complete tasks.

[0040] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A simulation system for household robot tasks and data collection based on Unity3D, characterized in that: include: Data acquisition module, Unity module, Python module, communication module, VR module and resource file module, among which: the data acquisition module uses camera and laser ranging to obtain point cloud, and uses key point matching algorithm between different image frames to align point clouds under different perspectives, so as to convert real objects into model files and output them to resource file module; Python module calls the model file in resource file module, and obtains resource information data corresponding to the model file from the database of resource file module, and outputs it to Unity module through communication module; Unity module uses real-time rendering of Unity3D software according to the position parameters of camera and model file The technology renders the model file in the three-dimensional scene; the VR module is connected to the Unity module, and the hand model information of the VR module is fed back to the Unity module in real time, and the real-time rendering technology of the Unity3D software is used to reflect the hand model in the VR module in real time in the three-dimensional scene, and can interact with the objects in the scene; at the same time, the Unity module will use the physical engine to simulate different types of model files, and output the scene information to the Python module through the communication module; the Python module inputs the received results into the reinforcement learning algorithm, obtains the data instructions for controlling the robot, and outputs them to the Unity module through the communication module to control the robot, thereby interacting; The resource information data includes: hash code of resource file, location of model, posture of model, attributes of model, master-slave relationship of model in the scene; The physical engines include: PhysX, Flex and Obi, wherein: PhysX is used for simulation of rigid bodies and joint bodies, Flex is used for simulation of fluids, and Obi is used for simulation of cloth and soft bodies; The scene information includes: the encoding, semantic classification, position, posture, speed, mass, center of mass position, position of each particle, three-dimensional detection frame, camera parameters under the current viewing angle, RGB image, depth image, normal image, instance segmentation mask, and optical flow image of each object in the scene; The data instruction includes: instruction name, object number and instruction parameter, wherein the instruction name is a predefined name; the instruction parameter is the parameter required when the instruction is executed, and corresponds to the instruction name one by one; The communication module includes: a serialization unit, a deserialization unit, a gRPC transmission unit, a Unity management unit and a Python management unit, wherein: the serialization unit will serialize the original data according to the serialization protocol to obtain a serialized byte stream; the serialization unit will output the serialized byte stream to the gRPC transmission unit; the gRPC transmission unit will output the byte stream to the Unity management unit or the Python management unit using the gRPC technology; after receiving the byte stream, the Unity management unit or the Python management unit will output the serialized byte stream to the deserialization unit, and wait for the deserialization unit to recover the data; the deserialization unit will deserialize the serialized byte stream according to the deserialization protocol to recover the original data; the deserialized original data will be returned to the Unity management unit or the Python management unit, and then returned to the Unity module or the Python module, and finally a communication task is completed; The serialization protocol is as follows: the protocol unit first traverses the unserialized data, obtains the data type of each data monomer in the data, writes the data type into a string in sequence, and byte-codes the string as the frame header of the serialized data, and the remaining data is byte-coded in sequence and written after the frame header; The deserialization protocol is as follows: the protocol unit first obtains the serialized data frame header, translates the frame header bytes into a string and obtains the data type of each data unit in the serialized data, and then determines how many bytes in the serialized data to read and translate them into data of the corresponding type according to the data type, thereby realizing data deserialization.

2. The Unity3D-based simulation system for household robot tasks and data collection according to claim 1, characterized in that: The VR module includes: a VR device and a VR program, wherein: the VR device includes VR glasses that use a gravity sensor to obtain the user's posture in the VR scene and display the scene picture at the current viewing angle for the user, and VR gloves that use a magnetic sensor to capture the user's hand posture changes. The user posture and head posture obtained by the VR device will be fed back to the VR program in real time; the VR program will filter and smooth the above data, and output the processed data to the Unity module through the serial port function of Unity3D. At the same time, the VR program will wait for the Unity module to return a screenshot of the simulation result at the current user posture, and output the screenshot to the VR glasses for the user to watch.

3. The simulation system for household robot tasks and data collection based on Unity3D according to claim 1, characterized in that: The data acquisition module includes: a scanner, a scanning platform and annotation software, wherein the scanner is a handheld 3D scanner, and the scanning platform is an ordinary turntable with a certain number of tracking points attached to it, so that the scanner can match the current frame image with the previous frame image. The user uses the annotation software to fill in the semantic classification, mass, center of mass position and attributes of the object.

4. The simulation system for household robot tasks and data collection based on Unity3D according to claim 1, characterized in that: The Unity module includes: a data queue unit and a simulation program unit, wherein: the data queue unit is responsible for receiving data from the communication module and storing it in a queue; the simulation program unit obtains the target rotation angle position of each joint of the current robotic arm from the data queue unit, and applies the result to the robotic arm in the simulation environment to simulate the state of each object in the scene after the robotic arm moves, and then sends the scene information to the communication module; the simulation program unit also receives and simulates the information sent by the VR module through the serial port, and the corresponding simulation result will be returned to the VR module.

5. The simulation system for household robot tasks and data collection based on Unity3D according to claim 1, characterized in that: The Python module includes: a data queue unit and a data processing unit, wherein: the data queue unit is responsible for receiving the data transmitted by the communication module and storing it in the queue; the data processing unit obtains the current simulation result from the data queue unit, uses the reinforcement learning algorithm to make predictions, obtains the target angular position of each joint of the robotic arm, and sends the result to the communication module.

6. The Unity3D-based simulation system for household robot tasks and data collection according to claim 1, characterized in that: The resource file module stores model files, and the types of model files supported include OBJ, FBX, DAE and STL. The resource file module includes: model files and model databases, wherein: the model files are AssetBundle files supported by Unity3D that contain models of all scanned objects to ensure that they are dynamically loaded into the scene when the Unity module is running; the model database is an SQL database with the hash code of the model file as the primary key, and the database records the semantic type, mass, center of mass position, and attributes of each model.

7. A simulation method for a household robot task and data acquisition-oriented simulation system based on Unity3D as described in any one of claims 1 to 6, comprising the following steps: Step 1. The user places the object to be grasped by the robot on the scanning table and turns on the scanner for scanning. After the scanner is turned on, the system will automatically pop up the annotation software; during the scanning process, the user needs to slowly rotate the turntable for one circle, and the rotation time must not be less than 1 minute; after the scanning is completed, the 3D model of the object will appear in the annotation software, and the user needs to annotate the semantic classification, name and quality for it; after the annotation is completed, the model is automatically added to the resource file module; Step 2: The user can call up the Unity module interface by running the Python module command. In the interface, the user chooses to add the Franka robot arm to any position in the scene and add the objects just scanned to the scene to build the basic environment. Step 3: The user chooses whether to use the VR module to control the robot arm. If the VR module is used, the user needs to wear VR equipment, and the current scene can be presented in the VR glasses. The user controls the gripper at the end of the Franka robot arm to grab the object by changing the position and posture of the right hand and the opening and closing of the five fingers until the object is successfully grabbed. Step 4: If the user does not choose to use the VR module, the reinforcement learning stage will be entered; the user enters the number of reinforcement learning training times in the Unity module interface; During the training process, the current training status, success rate, and estimated remaining time will be displayed in real time in the upper left corner of the interface. After waiting for the training to be completed, the interface will directly start using the trained model to display the training results to the user. If the user is not satisfied with the training results, he can return to step 2 to adjust the position of the robot arm and the object in the scene; otherwise, the process ends.

Images