A scene editing and virtual reality based robot arm control method and system

Abstract

The application discloses a kind of mechanical arm control method and system based on scene editing and virtual reality, by virtual reality technology, operator first operates virtual object on virtual placement surface in virtual scene, simulates free fall motion of virtual object after operation is finished, predicts the target pose of object falling on placement surface.Then by the initial pose of each virtual object, target pose and the order of being manipulated, automatically control mechanical arm to carry out corresponding operation to real object.In the present application, operator only needs to focus on the modification and editing of virtual scene and virtual object by oneself, without concerning the motion control of mechanical arm, so the work burden of operator can be effectively reduced, and the success rate and efficiency of task execution are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of teleoperation, and particularly relates to a mechanical arm control method and system based on scene editing and virtual reality. BACKGROUND

[0002] Nowadays, robots are widely applied to industrial production, intelligent medical treatment, family service and the like. With the development of artificial intelligence, sensor and the like, the intelligent degree of robots is gradually improved, and the robots have certain autonomous decision and action ability. However, in the face of unknown or complex environment, the current robots are still difficult to independently complete the predetermined task, and need to be artificially controlled by teleoperation technology.

[0003] A teleoperation system includes an operation end and a remote robot end, and the two ends are usually connected by means of a computer network. People can receive information collected by robot sensors and make decisions in a place far away from the robot, and then issue instructions to control the robot to perform actions. Teleoperation technology allows people and robots to be in the same space, and is widely used in space exploration, remote medical treatment, dangerous space operation and the like.

[0004] The existing teleoperation system mainly uses a robot-centered direct control strategy. For example, the system uses a camera on the robot as an environment sensor, and the operator controls the pose of the end effector of the mechanical arm or sets the way points of the trajectory to control the mechanical arm according to the environment information displayed by the VR. This kind of direct control strategy still needs the operator to manually determine each action of the robot when executed, which is very tedious and challenging, resulting in a large workload and high professional requirements for the operator in actual operation.

[0005] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a mechanical arm control method and system based on scene editing and virtual reality, aiming at solving the problem that the existing teleoperation system adopts a direct control strategy, and the operator needs to manually determine each action of the robot when executed, resulting in high labor cost.

[0007] The technical scheme adopted by the present application to solve the problem is as follows:

[0008] In a first aspect, the present application provides a mechanical arm control method based on scene editing and virtual reality, which comprises:

[0009] previously generating a virtual placement surface corresponding to the placement surface in the virtual scene;

[0010] Obtain pose data corresponding to each of a plurality of objects on the object placing surface, and generate a plurality of opaque models on the virtual object placing surface according to the pose data of each of the objects, wherein each of the opaque models is used to reflect an initial pose of a virtual object corresponding to one of the objects;

[0011] If the current mode is the human-computer interaction control mode, for each of the opaque models, a transparent model overlapping the opaque model is grasped to a target position and released by using hand motion data of an operator, and after the release, a target pose of the transparent model on the virtual object placing surface is obtained by simulating free fall motion;

[0012] Generate operation data corresponding to the robot arm according to the initial pose, the target pose, and the manipulation sequence of each of the virtual objects.

[0013] In an embodiment, the virtual object placing surface corresponding to the object placing surface is generated in the virtual scene in advance, including:

[0014] Obtain environment point cloud data by using a depth camera, wherein the environment point cloud data is used to reflect a surrounding environment of the robot arm, including the object placing surface;

[0015] Obtain a plurality of environment planes by using a preset segmentation algorithm and the environment point cloud data, and extract a maximum environment plane from each of the environment planes;

[0016] Determine boundary data according to the maximum environment plane, generate an object placing surface model in the virtual scene according to the boundary data, and obtain the virtual object placing surface.

[0017] In an embodiment, the determination of the boundary data according to the maximum environment plane includes:

[0018] Remove discrete points in the maximum environment plane, and uniformly sample the maximum environment plane;

[0019] Calculate the boundary data according to the uniform sampling result.

[0020] In an embodiment, the obtaining of the pose data corresponding to each of the plurality of objects on the object placing surface includes:

[0021] For each of the objects, obtain pose data of the object relative to the depth camera by using an object pose estimation algorithm;

[0022] Obtain a transformation matrix between a coordinate system of the depth camera and a coordinate system of the robot arm;

[0023] Obtain pose data of the object relative to the robot arm according to the transformation matrix and the pose data of the object relative to the depth camera.

[0024] In an embodiment, the method further comprises:

[0025] pre-generating a virtual robot arm corresponding to the robot arm in the virtual scene, wherein an end effector of the virtual robot arm contains a virtual polyhedron;

[0026] if the current mode is the end effector control mode, dragging the virtual polyhedron through the hand motion data of the operator;

[0027] generating operation data of the robot arm through the pose data of the virtual polyhedron to realize synchronous movement of the robot arm and the virtual polyhedron.

[0028] In an embodiment, the method further comprises:

[0029] if the hand motion data of the operator is preset target motion, displaying a virtual menu in the virtual scene, wherein different regions in the virtual menu correspond to different control services respectively;

[0030] obtaining click data generated by the operator based on the virtual menu, and calling the control service of the corresponding region according to the click data.

[0031] In an embodiment, the calling the control service of the corresponding region according to the click data comprises:

[0032] if the called control service is release, controlling the gripper of the robot arm to open to a preset position;

[0033] if the called control service is grasp, controlling the gripper of the robot arm to close at a preset speed, and adjusting the motor speed to zero when monitoring that the motor current data exceeds a preset current value.

[0034] In a second aspect, the embodiments of the present application further provide a robot arm control system based on scene editing and virtual reality, which comprises:

[0035] a plane detection module, configured to pre-generate a virtual placement surface corresponding to a placement surface in a virtual scene;

[0036] a pose estimation module, configured to obtain pose data corresponding to a plurality of objects on the placement surface respectively, and generate a plurality of opaque models on the virtual placement surface according to the pose data of each object, wherein each opaque model is used to reflect the initial pose of a virtual object corresponding to one object;

[0037] a virtual reality module, configured to, if the current mode is the human-computer interaction control mode, for each of the opaque models, grasp the transparent model overlapping the opaque model to a target position by the hand action data of the operator, and release the transparent model; and after the release, obtain the target pose of the transparent model falling on the virtual placement surface by simulating the free fall motion.

[0038] a robot arm control module, configured to generate the operation data of the robot arm according to the initial pose, the target pose and the manipulated sequence of each of the virtual objects.

[0039] In a third aspect, an embodiment of the present application further provides a terminal, which comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the scene editing and virtual reality based robot arm control method; and the processor is configured to execute the programs.

[0040] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores a plurality of instructions, and the instructions are adapted to be loaded and executed by a processor to implement the steps of the scene editing and virtual reality based robot arm control method.

[0041] The present application has the following advantages: the embodiment of the present application uses the virtual reality technology, and the operator first operates the virtual objects on the virtual placement surface in the virtual scene, and then simulates the free fall motion of the virtual objects to predict the target pose of the objects falling on the placement surface. Then, the initial pose, the target pose and the manipulated sequence of each of the virtual objects are used to automatically control the robot arm to perform corresponding operations on the real objects. In the present application, the operator only needs to focus on the modification and editing of the virtual scene and the virtual objects, and does not need to care about the motion control of the robot arm, so that the working burden of the operator can be effectively reduced, and the success rate and efficiency of task execution can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0043] Figure 1 is a flowchart of the scene editing and virtual reality based robot arm control method provided by the embodiment of the present application.

[0044] Figure 2 is a schematic diagram of the MR scene design provided by the embodiment of the present application.

[0045] Figure 3 is a schematic diagram of different control modes in the MR interface provided by the embodiment of the application.

[0046] Figure 4 is a schematic diagram of the module of the mechanical arm control system based on scene editing and virtual reality provided by the embodiment of the application.

[0047] Figure 5 is a schematic diagram of the architecture and workflow of the mechanical arm control system based on scene editing and virtual reality provided by the embodiment of the application.

[0048] Figure 6 is a schematic diagram of the principle of the terminal provided by the embodiment of the application. DETAILED DESCRIPTION

[0049] The application discloses a mechanical arm control method and system based on scene editing and virtual reality, in order to make the purpose, technical scheme and effect of the application more clear and definite, the application is further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and not to limit the application.

[0050] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the phrase "comprising" used in the specification of the application means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0051] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood as having meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0052] In view of the above defects of the prior art, the application provides a mechanical arm control method based on scene editing and virtual reality, as Figure 1As shown, the method specifically includes:

[0053] Step S100, a virtual object placement surface corresponding to the object placement surface is generated in the virtual scene in advance.

[0054] Specifically, in order to realize the modification and editing of the virtual scene and the object by the operator, a virtual reality interface (MR interface) is provided in advance, through which the operator can see a virtual mechanical arm and a virtual object placement surface where the virtual mechanical arm is located (for example Figure 2 As shown). The virtual mechanical arm and the virtual object placement surface are respectively the images of the real mechanical arm and the real object placement surface in the virtual scene, and the movement of the real mechanical arm can be guided and controlled through the modification and editing of the virtual scene and the object by the operator.

[0055] In an implementation manner, the object placement surface can be a desktop, or a ground, or other flat surface on which objects can be placed; and the mechanical arm can be an independent mechanical arm or a component in a robot execution mechanism.

[0056] In an implementation manner, the virtual object placement surface is generated in the virtual scene in advance, including:

[0057] Obtaining environment point cloud data through a depth camera, wherein the environment point cloud data is used to reflect the surrounding environment of the mechanical arm, including the object placement surface;

[0058] Obtaining a plurality of environment planes through a preset segmentation algorithm and the environment point cloud data, and extracting a maximum environment plane from each of the environment planes;

[0059] Determining boundary data according to the maximum environment plane, generating an object placement surface model in the virtual scene according to the boundary data, and obtaining the virtual object placement surface.

[0060] Specifically, the mixed reality technology is adopted to reproduce the environment, which can better provide the operator with the cognition of the three-dimensional information of the space, thereby solving the problem that the traditional visual feedback-based teleoperation system is difficult to intuitively feedback the three-dimensional environment information around the robot. For example Figure 2As shown, the visualization elements in the virtual scene also include a screen, which can display pictures captured by a depth camera (e.g., a L515 depth camera). The environment point cloud data is extracted from the depth pictures, and then the environment point cloud data is processed by a preset segmentation algorithm to extract a plurality of environment planes. The largest one of the environment planes is filtered out, i.e., the largest environment plane is obtained. Then, the boundary data of the largest environment plane is calculated, and a corresponding model is constructed in the virtual scene by using the boundary data, i.e., a placement surface model is obtained. The placement surface model can reflect the image of the placement surface in the virtual scene, i.e., a virtual placement surface is obtained. In an implementation manner, the placement surface model can be obtained by calling a DesktopDetection service as one of the visualization elements in the virtual scene.

[0061] In an implementation manner, the boundary data is determined according to the largest environment plane, including:

[0062] The discrete points in the largest environment plane are removed, and the largest environment plane is uniformly sampled;

[0063] The boundary data is calculated according to the uniform sampling result.

[0064] Specifically, in order to accurately generate the virtual placement surface, the embodiment needs to remove the discrete points in the plane after extracting the largest environment plane, for example, a clustering algorithm can be used to remove the discrete points. Then, the boundary is calculated after the plane is uniformly sampled, so that the boundary data of the placement surface is accurately obtained. The virtual placement surface can be accurately generated by using the boundary data in combination with virtual reality technology.

[0065] For example, the specific generation process of the virtual placement surface is as shown below:

[0066] The collision model of the placement surface can be obtained by plane detection. First, the depth pictures are captured by a depth camera, and the environment point cloud data obtained from the depth pictures is processed by using a PCL library (Point Cloud Library). A model-based segmentation algorithm is used to obtain all the planes in the environment, i.e., a plurality of environment planes. The largest plane is extracted, i.e., the largest environment plane is obtained. Then, a clustering algorithm is used to remove the discrete points in the plane, the plane is uniformly sampled, and the boundary box is calculated. Finally, the boundary coordinates of the plane are calculated and a grid message is generated. In an actual application scenario, the plane detection function can be integrated into the system by using a DesktopDetection ROS service. When the service is called, the grid representation of the placement surface used in virtual reality can be provided.

[0067] In step S200, pose data corresponding to each of the objects on the object placement surface is acquired, and a plurality of opaque models are generated on the virtual object placement surface according to the pose data of each of the objects, wherein each of the opaque models is used to reflect an initial pose of a virtual object corresponding to one of the objects.

[0068] The embodiment can synchronize the pose of a virtual object in a virtual reality interface with the pose of an object in a real world through a pose estimation algorithm. Specifically, as shown in Figure 2 For each object in the real world, the visualization element in the virtual scene further includes a virtual object corresponding to the object, and the virtual object has an image of an opaque model, and the pose of the opaque model is consistent with the pose of the corresponding object in the real world.

[0069] In an implementation manner, the acquiring of the pose data corresponding to each of the objects on the object placement surface comprises:

[0070] For each of the objects, the pose data of the object relative to the depth camera is acquired through an object pose estimation algorithm;

[0071] The transformation matrix between the coordinate system of the depth camera and the coordinate system of the robot arm is acquired;

[0072] According to the transformation matrix and the pose data of the object relative to the depth camera, the pose data of the object relative to the robot arm is obtained.

[0073] Specifically, taking one object as an example, the pose data of the object relative to the depth camera can be obtained by using an object pose estimation algorithm DOPE based on a neural network. In order to obtain the coordinates of the object relative to the robot arm, a transformation matrix from the camera coordinate system to the robot arm base coordinate system can be obtained through hand-eye calibration. Finally, the coordinates of the object relative to the robot arm base coordinate system can be obtained by using the transformation matrix for coordinate transformation, so as to obtain the pose data of the object relative to the robot arm.

[0074] In an implementation manner, the object pose calculation corresponding module can be integrated into the system through a ROS node, and the detected object id and the corresponding 6-degree-of-freedom pose are published in real time through an object_pose topic. In the virtual scene, the object models in the HOPE dataset are pre-imported and hidden by default. After receiving the object_poses message, the corresponding opaque model is displayed according to the id and the corresponding pose.

[0075] In step S300, if the current mode is the human-computer interaction control mode, for each of the opaque models, the transparent model overlapping with the opaque model is grasped to a target position and released by the hand action data of the operator; after the release, a target pose of the transparent model falling on the virtual placement surface is obtained by simulating free fall motion.

[0076] Specifically, as shown in Figure 3 The control interface in this embodiment includes a human scene (Human Scene Interaction Control, HSI-Control) control mode. In the HSI control mode, the operator directly interacts with the virtual environment using virtual reality technology, and controls the robot arm by moving virtual objects. As shown in Figure 2 For each object in reality, the visualization element in the virtual scene also includes a transparent model corresponding to the object. The transparent model can be moved according to the operation of the operator, and is used to reflect the target state of the object that the operator wants to move. By default, the opaque model and the transparent model of each object are coincident. In addition, due to possible operation errors, the object may not be in a completely stable pose when the operator places the virtual object. Therefore, after the operator places the object, a stable pose of the object falling on the placement surface can be calculated by gravity simulation and collision simulation, i.e., a target pose is obtained.

[0077] For example, in the HSI control mode, the operator can start the control process by grasping the virtual object that he or she wants to move. At this time, the opaque model of the virtual object being moved remains stationary, and the corresponding transparent model follows the movement of the operator's hand. The operator can move the transparent model to a target position and release the hand. After the hand is released, the gravity attribute of the transparent model is activated. By simulating free fall motion through the Unity physics engine, the transparent model will fall and collide with the table. Finally, the transparent model will stably fall on the table, and a final target pose is calculated. By combining the physical simulation technology, the embodiment can improve the prediction accuracy of the final pose of the object being manipulated, thereby improving the success rate of remote operation.

[0078] In an implementation manner, after the target pose of the transparent model falling on the virtual placement surface is obtained by simulating free fall motion, the method further includes:

[0079] determining whether the target pose is qualified;

[0080] If not, the step of grasping the transparent model overlapping with the opaque model to a target position and releasing the transparent model by the hand action data of the operator is re-executed until the target pose is qualified.

[0081] In step S400, operation data corresponding to the robot arm is generated according to the initial pose, the target pose and the operation sequence of each virtual object.

[0082] Specifically, the initial pose, the target pose and the operation sequence of the virtual object can be used to guide the pick-and-place operation performed by the robot arm in the real space, including the coordinates before and after the object is moved and the sequence in which the object is moved. For each object, the robot arm will first move above the object, perform a top-down pick-up, then move the object to the target position, and finally release the gripper and drop the object. By interacting with the virtual scene, the operator can control the movement of the robot arm in the real world, which can save operation time and reduce the burden and learning difficulty of the operator.

[0083] For example, when the operator finishes operating the virtual object, the operator can click the "Execute" button on the virtual menu in the virtual scene. The button will call the ExecuteTask service, and the service request will contain the ID, initial pose and target pose of each object in the order in which the operator operates, and command the robot arm to perform the corresponding task. The control node of the present embodiment is developed based on the official driver of the Kinova Gen3 robot arm, and when the ExecuteTask service is called from the HSI control mode, the robot arm will operate each object in turn to complete the task.

[0084] In an implementation manner, the method further includes:

[0085] The virtual robot arm corresponding to the robot arm is generated in the virtual scene in advance, wherein a virtual polyhedron is contained on an end effector of the virtual robot arm;

[0086] If the current mode is the end effector control mode, the virtual polyhedron is dragged by the hand action data of the operator;

[0087] Operation data of the robot arm is generated by pose data of the virtual polyhedron, so as to realize synchronous movement of the robot arm and the virtual polyhedron.

[0088] Specifically, as shown in Figure 2 The visualization elements of the virtual scene further include a virtual robot arm, and a virtual polyhedron, for example, an icosahedron, is arranged on an end effector of the virtual robot arm. The pose of the virtual polyhedron is bound to the tool coordinate system of the end effector of the robot arm by default. After the operator holds the virtual polyhedron, the pose is unbound, so that the operator can freely move or rotate the virtual polyhedron. Figure 3As shown, the control interface in this embodiment also includes a traditional end-effector control mode (End-Effector Control, EE-Control). In the EE control mode, the operator can directly control the robot arm by dragging the virtual polyhedron.

[0089] For example, the virtual robot arm can be obtained by importing a virtual Kinova Gen3 robot arm model through URDF Importe, and synchronized with the real robot arm by receiving the joint_states topic. The pose of the virtual polyhedron is published to the control node through the target_pose topic, so as to control the robot arm to move synchronously with the virtual polyhedron. In the EE control mode, the node will put the target pose into a fixed-length queue when receiving the target_pose message. The main program will loop to get the target pose from the queue and control the robot arm to move to the corresponding pose.

[0090] In an implementation manner, the method further includes:

[0091] If the hand action data of the operator is a preset target action, a virtual menu is displayed in the virtual scene, wherein different regions in the virtual menu correspond to different control services respectively;

[0092] Click data generated by the operator based on the virtual menu is obtained, and a control service corresponding to a region is called according to the click data.

[0093] Specifically, as shown in Figure 2 The visualization elements in the virtual scene also include a virtual menu, which appears when the operator performs a specific hand action, and the operator can call a specific control service / command of the region by clicking different regions in the virtual menu.

[0094] For example, the virtual menu can be a hand menu, which appears when the operator extends the left hand. There are several buttons on the hand menu, each of which corresponds to a different command. The operator can call a specific command by clicking the corresponding button. For example, the "grasp" button is used to control the closing of the mechanical gripper, and the "release" button is used to control the opening of the mechanical gripper. The two buttons are clicked to call GraspService and ReleaseService services respectively.

[0095] In an implementation manner, the calling of the control service corresponding to the region according to the click data includes:

[0096] If the called control service is release, the gripper of the robot arm is controlled to open to a preset position;

[0097] If the control service is grasping, the gripper of the robot arm is controlled to close at a preset speed, and when it is monitored that the motor current data exceeds a preset current value, the motor speed is adjusted to zero.

[0098] Specifically, since some types of mechanical grippers lack force sensors (e.g. Robotiq 2f-85 mechanical gripper), the embodiment provides two control modes to operate the gripper. For the control service of release, the embodiment adopts a position control mode: when ReleaseService is called, the program controls the gripper to a fully open position. For the control service of grasping, the embodiment adopts a grasping control mode based on motor current feedback: when GraspService is called, the gripper closes at a preset speed (e.g. 0.1 m / s), and at the same time the program monitors the motor current. If the current exceeds a preset value (e.g. 0.12 A), it is considered that the object is grasped, and then the program sets the motor speed to zero.

[0099] Based on the above embodiment, the application also provides a robot arm control system based on scene editing and virtual reality, as shown in Figure 4 The system comprises:

[0100] a plane detection module 01 for generating a virtual placement surface corresponding to a placement surface in a virtual scene in advance;

[0101] a pose estimation module 02 for obtaining pose data corresponding to a plurality of objects on the placement surface respectively, and generating a plurality of opaque models on the virtual placement surface according to the pose data of each object, wherein each opaque model is used to reflect the initial pose of a virtual object corresponding to the object;

[0102] a virtual reality module 03 for, if the current mode is a human-computer interaction control mode, grasping a transparent model overlapping the opaque model to a target position and releasing it for each opaque model through the hand action data of the operator; after release, the target pose of the transparent model falling on the virtual placement surface is obtained through simulation of free fall motion;

[0103] a robot arm control module 04 for generating operation data corresponding to the robot arm according to the initial pose, the target pose and the manipulated sequence of each virtual object respectively.

[0104] It should be noted that the foregoing explanation and description of the embodiment of the robot arm control method based on scene editing and virtual reality also applies to the embodiment of the robot arm control system based on scene editing and virtual reality, which will not be described here again.

[0105] Figure 5The architecture and workflow of the system of the embodiment are shown. First, the operator can see the virtual placement surface through the MR interface, and the virtual objects on the virtual placement surface and the objects on the real placement surface will be pose-synchronized. The operator first operates the virtual objects in the virtual scene, and after the operation is completed, simulates the free fall motion of the virtual objects, and predicts the target pose of the objects falling on the placement surface. Then, through the initial pose, target pose and manipulated sequence of each virtual object, the mechanical arm is automatically controlled to perform corresponding operations on the real objects.

[0106] Based on the above embodiment, the application further provides a terminal, and a principle block diagram thereof can be as shown in the figure. Figure 6 The terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. The processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the terminal is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement the scene editing and virtual reality based mechanical arm control method. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0107] Those skilled in the art can understand that, Figure 6 The principle block diagram shown in the figure is only a block diagram of part of the structure related to the application scheme, and does not constitute a limitation on the terminal to which the application scheme is applied. The specific terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0108] In an implementation manner, the memory of the terminal stores more than one program, and is configured to execute the more than one program by more than one processor, which contains instructions for performing the scene editing and virtual reality based mechanical arm control method.

[0109] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0110] In summary, the present application discloses a mechanical arm control method and system based on scene editing and virtual reality. The method makes the operator operate the virtual objects on the virtual placement surface in the virtual scene through virtual reality technology, simulates the free fall motion of the virtual objects after the operation is completed, and predicts the target pose of the objects falling on the placement surface. Then, through the initial pose, target pose and manipulated sequence of each virtual object, the mechanical arm is automatically controlled to perform corresponding operation on the real objects. In the present application, the operator only needs to focus on the modification and editing of the virtual scene and virtual objects, without worrying about the motion control of the mechanical arm, so that the working burden of the operator can be effectively reduced, and the success rate and efficiency of task execution can be improved.

[0111] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes shall belong to the protection scope of the claims of the present application.

Claims

1. A scene editing and virtual reality-based robot arm manipulation method, characterized by, The method comprises: pre-generating a virtual placement surface corresponding to a placement surface in a virtual scene; acquiring pose data corresponding to each of a plurality of objects on the placement surface, and generating a plurality of opaque models on the virtual placement surface according to the pose data of each object, wherein each opaque model is used to reflect an initial pose of a virtual object corresponding to one of the objects; if the current mode is a human-computer interaction control mode, then for each opaque model, a transparent model overlapping the opaque model is grabbed to a target position and released by using hand action data of an operator; after the release, a target pose of the transparent model falling on the virtual placement surface is obtained by simulating free fall motion; generating operation data corresponding to a robot arm according to the initial pose, the target pose and a manipulated sequence of each virtual object. 2.The scene editing and virtual reality based robot arm manipulation method of claim 1, wherein, The pre-generating of the virtual placement surface corresponding to the placement surface in the virtual scene comprises: acquiring environment point cloud data by using a depth camera, wherein the environment point cloud data is used to reflect a surrounding environment of the robot arm, including the placement surface; obtaining a plurality of environment planes by using a preset segmentation algorithm and the environment point cloud data, and extracting a maximum environment plane from each of the environment planes; determining boundary data according to the maximum environment plane, generating a placement surface model in the virtual scene according to the boundary data, and obtaining the virtual placement surface. 3.The scene editing and virtual reality based robot arm manipulation method of claim 2, wherein, The determining of the boundary data according to the maximum environment plane comprises: removing discrete points in the maximum environment plane, and uniformly sampling the maximum environment plane; calculating the boundary data according to the uniformly sampled result. 4.The scene editing and virtual reality based robot arm manipulation method of claim 2, wherein, The acquiring of the pose data corresponding to each of the plurality of objects on the placement surface comprises: for each object, acquiring pose data of the object relative to the depth camera by using an object pose estimation algorithm; acquiring a transformation matrix between a coordinate system of the depth camera and a coordinate system of the robot arm; acquiring pose data of the object relative to the robot arm according to the transformation matrix and the pose data of the object relative to the depth camera. 5.The scene editing and virtual reality based robot arm manipulation method of claim 1, wherein, The method further comprises: pre-generating a virtual robot arm corresponding to the robot arm in the virtual scene, wherein a virtual polyhedron is contained on an end effector of the virtual robot arm; if the current mode is an end effector control mode, then the virtual polyhedron is dragged by using the hand action data of the operator; generating operation data of the robot arm by using pose data of the virtual polyhedron, so as to realize synchronous motion of the robot arm and the virtual polyhedron. 6.The scene editing and virtual reality based robot arm manipulation method of claim 1, wherein, The method further comprises: if the hand action data of the operator is a preset target action, then a virtual menu is displayed in the virtual scene, wherein different regions in the virtual menu correspond to different control services respectively; acquiring click data generated by the operator based on the virtual menu, and calling a control service of a corresponding region according to the click data.

7. The scene editing and virtual reality based robot arm manipulation method of claim 6, wherein, The calling of the control service of the corresponding region according to the click data comprises: if the called control service is release, then the gripper of the robot arm is controlled to open to a preset position; If the control service of the call is grabbing, the gripper of the robot arm is controlled to close at a preset speed, and when it is monitored that the motor current data exceeds a preset current value, the motor speed is adjusted to zero.

8. A scene editing and virtual reality-based robot arm manipulation system, characterized by, The system comprises: A plane detection module is configured to generate a virtual placement plane corresponding to a placement plane in a virtual scene in advance; A pose estimation module is configured to acquire pose data corresponding to a plurality of objects on the placement plane, and generate a plurality of opaque models on the virtual placement plane according to the pose data of each object, wherein each opaque model is used to reflect the initial pose of a virtual object corresponding to one object; A virtual reality module is configured to, if the current mode is a human-computer interaction control mode, for each opaque model, capture a transparent model overlapping with the opaque model to a target position and release it through hand action data of an operator; after the release, the target pose of the transparent model falling on the virtual placement plane is obtained through simulation of free fall motion; A robot arm control module is configured to generate operation data corresponding to a robot arm according to the initial pose, the target pose and the manipulated sequence of each virtual object.

9. A terminal, characterized by comprising: The terminal comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the scene editing and virtual reality based robot arm control method of any one of claims 1-7; the processor is used to execute the program.

10. A computer readable storage medium having stored thereon a plurality of instructions, the plurality of instructions comprising: The instructions are suitable for being loaded and executed by the processor to implement the steps of the scene editing and virtual reality based robot arm control method of any one of claims 1-7.

Images