Mechanical arm control method and system based on scene editing and virtual reality

Abstract

The invention discloses a mechanical arm control method and system based on scene editing and virtual reality, and the method comprises the steps: enabling an operator to operate a virtual object on a virtual object placement surface in a virtual scene through a virtual reality technology, simulating the free falling motion of the virtual object after the operation is completed, and predicting the target pose of the object falling on the object placement surface. And then according to the initial poses, the target poses and the manipulated sequence of the virtual objects, the mechanical arm is automatically controlled to conduct corresponding operation on the real objects. According to the method, an operator only needs to focus on modifying and editing the virtual scene and the virtual object and does not need to concern motion control of the mechanical arm, so that the workload of the operator can be effectively reduced, and the task execution success rate and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote operation, and in particular to a method and system for controlling a robotic arm based on scene editing and virtual reality. Background Art

[0002] Nowadays, robots are widely used in industrial production, smart medical care, home services and other fields. With the development of artificial intelligence, sensors and other technologies, the intelligence of robots is gradually improving, and they have a certain degree of autonomous decision-making and action capabilities. However, when facing unknown or complex environments, current robots still find it difficult to complete the scheduled tasks independently, and need to use teleoperation technology to manually control the actions of robots.

[0003] A teleoperation system consists of an operator and a remote robot, which are usually connected by a computer network. People can receive information collected by sensors on the robot and make decisions far away from the robot, and then issue instructions to control the robot to perform actions. Teleoperation technology allows people and robots to no longer be in the same space, so it is widely used in space exploration, telemedicine, dangerous space operations and other tasks.

[0004] Existing teleoperation systems mainly use robot-centric direct control strategies. For example, the system uses the camera on the robot as an environmental sensor. The operator uses the VR handle to control the position of the end effector of the robot or set the path points of the trajectory to control the robot according to the environmental information displayed by the VR. When executing this type of direct control strategy, the operator still needs to manually determine each movement of the robot, which is very cumbersome and challenging, resulting in a large workload for the operator during actual operation and high professional requirements.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0006] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a method and system for controlling a robotic arm based on scene editing and virtual reality is provided, aiming to solve the problem that the existing remote operation system adopts a direct control strategy, which requires the operator to manually determine each action of the robot during execution, resulting in excessively high labor costs.

[0007] The technical solution adopted by the present invention to solve the problem is as follows:

[0008] In a first aspect, an embodiment of the present invention provides a method for manipulating a robotic arm based on scene editing and virtual reality, the method comprising:

[0009] Pre-generating a virtual placement surface corresponding to the placement surface in the virtual scene;

[0010] Acquire posture data corresponding to a plurality of objects on the placement surface, and generate a plurality of opaque models on the virtual placement surface according to the posture data of each of the objects, wherein each of the opaque models is used to reflect an initial posture of a virtual object corresponding to one of the objects;

[0011] If the current mode is the human-computer interaction control mode, for each of the opaque models, the transparent model overlapping with the opaque model is grabbed to the target position and released according to the operator's hand motion data; after the release, the target position of the transparent model falling on the virtual placement surface is obtained by simulating free fall motion;

[0012] The operation data corresponding to the robot arm are generated according to the initial posture, the target posture and the manipulation sequence corresponding to each virtual object.

[0013] In one embodiment, the step of pre-generating a virtual placement surface corresponding to the placement surface in the virtual scene includes:

[0014] Acquiring environmental point cloud data through a depth camera, wherein the environmental point cloud data is used to reflect the surrounding environment of the robotic arm, including the placement surface;

[0015] Obtaining a plurality of environmental planes by using a preset segmentation algorithm and environmental point cloud data, and extracting a maximum environmental plane from each of the environmental planes;

[0016] Boundary data is determined according to the maximum environment plane, and a placement surface model is generated in the virtual scene according to the boundary data to obtain the virtual placement surface.

[0017] In one implementation, determining boundary data according to the maximum environment plane includes:

[0018] removing discrete points in the maximum environment plane and uniformly sampling the maximum environment plane;

[0019] The boundary data is calculated according to the uniform sampling result.

[0020] In one embodiment, the obtaining of the position and posture data corresponding to the plurality of objects on the placement surface includes:

[0021] For each of the objects, obtaining the pose data of the object relative to the depth camera by using an object pose estimation algorithm;

[0022] Obtaining a transformation matrix between the coordinate system of the depth camera and the coordinate system of the robotic arm;

[0023] The position data of the object relative to the robotic arm is obtained according to the transformation matrix and the position data of the object relative to the depth camera.

[0024] In one embodiment, the method further comprises:

[0025] Pre-generating a virtual robotic arm corresponding to the robotic arm in the virtual scene, wherein an end effector of the virtual robotic arm includes a virtual polyhedron;

[0026] If the current mode is the end effector control mode, dragging the virtual polyhedron according to the hand motion data of the operator;

[0027] The operation data of the robot arm is generated by the position and posture data of the virtual polyhedron, so as to realize the synchronous movement of the robot arm and the virtual polyhedron.

[0028] In one embodiment, the method further comprises:

[0029] If the operator's hand motion data is a preset target motion, a virtual menu is displayed in the virtual scene, wherein different areas in the virtual menu correspond to different control services;

[0030] The click data generated by the operator based on the virtual menu is acquired, and the control service of the corresponding area is called according to the click data.

[0031] In one implementation, calling a control service of a corresponding area according to the click data includes:

[0032] If the called control service is release, the gripper of the robot arm is controlled to open to a preset position;

[0033] If the called control service is grasping, the gripper of the robot arm is controlled to close at a preset speed, and when the motor current data is detected to exceed the preset current value, the motor speed is adjusted to zero.

[0034] In a second aspect, an embodiment of the present invention further provides a robotic arm control system based on scene editing and virtual reality, the system comprising:

[0035] A plane detection module, used to generate a virtual placement surface corresponding to the placement surface in the virtual scene in advance;

[0036] A posture estimation module, used to obtain posture data corresponding to a plurality of objects on the placement surface, and generate a plurality of opaque models on the virtual placement surface according to the posture data of each of the objects, wherein each of the opaque models is used to reflect an initial posture of a virtual object corresponding to one of the objects;

[0037] A virtual reality module, for grabbing a transparent model overlapping with the opaque model to a target position and releasing it according to the operator's hand motion data if the current mode is a human-computer interaction control mode; after the release, obtaining a target position of the transparent model falling on the virtual placement surface by simulating free fall motion;

[0038] The robot arm control module is used to generate operation data corresponding to the robot arm according to the initial posture, the target posture and the manipulation sequence corresponding to each virtual object.

[0039] In a third aspect, an embodiment of the present invention further provides a terminal, comprising a memory and one or more processors; the memory stores one or more programs; the program comprises instructions for executing any of the above-mentioned scene editing and virtual reality based robotic arm control methods; and the processor is used to execute the program.

[0040] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to implement any of the steps of the above-mentioned method for manipulating a robotic arm based on scene editing and virtual reality.

[0041] Beneficial effects of the present invention: The embodiment of the present invention uses virtual reality technology to allow the operator to first operate the virtual object on the virtual placement surface in the virtual scene, and then simulate the free fall motion of the virtual object after the operation is completed, and predict the target position of the object falling on the placement surface. Then, through the initial position, target position and manipulation sequence of each virtual object, the robotic arm is automatically controlled to perform corresponding operations on the real object. In the present invention, the operator only needs to focus on modifying and editing the virtual scene and virtual objects, and does not need to worry about the motion control of the robotic arm. Therefore, the operator's workload can be effectively reduced, and the success rate and efficiency of task execution can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a flow chart of a method for manipulating a robotic arm based on scene editing and virtual reality provided in an embodiment of the present invention.

[0044] Figure 2 It is a schematic diagram of MR scene design provided by an embodiment of the present invention.

[0045] Figure 3 It is a schematic diagram of the principles of different control modes in the MR interface provided by an embodiment of the present invention.

[0046] Figure 4 It is a module schematic diagram of a robotic arm control system based on scene editing and virtual reality provided by an embodiment of the present invention.

[0047] Figure 5 It is a schematic diagram of the architecture and workflow of a robotic arm control system based on scene editing and virtual reality provided by an embodiment of the present invention.

[0048] Figure 6 It is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention discloses a method and system for manipulating a robotic arm based on scene editing and virtual reality. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0051] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0052] In view of the above-mentioned defects of the prior art, the present invention provides a robot arm control method based on scene editing and virtual reality, such as Figure 1As shown, the method specifically includes:

[0053] Step S100: Generate a virtual placement surface corresponding to the placement surface in the virtual scene in advance.

[0054] Specifically, in order to enable the operator to modify and edit the virtual scene and objects, this embodiment pre-sets a virtual reality interface (MR interface), through which the operator can see a virtual robotic arm and the virtual placement surface (such as Figure 2 The virtual robot arm and the virtual object surface are the images of the real robot arm and the real object surface in the virtual scene, respectively. The operator can guide and control the movement of the real robot arm by modifying and editing the virtual scene and objects.

[0055] In one implementation, the placement surface may be a desktop, or the ground, or other planes on which objects may be placed; the robotic arm may be an independently operating robotic arm, or a component in a robot actuator.

[0056] In one implementation, the step of pre-generating a virtual placement surface corresponding to the placement surface in the virtual scene includes:

[0057] Acquiring environmental point cloud data through a depth camera, wherein the environmental point cloud data is used to reflect the surrounding environment of the robotic arm, including the placement surface;

[0058] Obtaining a plurality of environmental planes by using a preset segmentation algorithm and environmental point cloud data, and extracting a maximum environmental plane from each of the environmental planes;

[0059] Boundary data is determined according to the maximum environment plane, and a placement surface model is generated in the virtual scene according to the boundary data to obtain the virtual placement surface.

[0060] Specifically, this embodiment uses mixed reality technology to reproduce the environment, which can better provide the operator with a better understanding of the three-dimensional spatial information, thereby solving the problem that the traditional teleoperation system based on visual feedback is difficult to intuitively feedback the three-dimensional environmental information around the robot. Figure 2As shown, the visualization elements in the virtual scene also include a screen that can display pictures captured by a depth camera (such as an L515 depth camera). Environmental point cloud data is extracted through the depth picture, and then the environmental point cloud data is processed by a preset segmentation algorithm to extract multiple environmental planes. Among these environmental planes, the largest plane is screened out to obtain the largest environmental plane. Then, the boundary data of the largest environmental plane is calculated, and the corresponding model is constructed in the virtual scene through the boundary data to obtain the placement surface model. The placement surface model can reflect the image of the placement surface in the virtual scene, that is, a virtual placement surface is obtained. In one implementation, the placement surface model, as one of the visualization elements in the virtual scene, can be obtained by calling the DesktopDetection service.

[0061] In one implementation, determining boundary data according to the maximum environment plane includes:

[0062] removing discrete points in the maximum environment plane and uniformly sampling the maximum environment plane;

[0063] The boundary data is calculated according to the uniform sampling result.

[0064] Specifically, in order to accurately generate a virtual placement surface, this embodiment needs to remove discrete points in the plane after extracting the maximum environment plane, for example, a clustering algorithm can be used to remove discrete points. Then, the plane is uniformly sampled and then the boundary calculation is performed to accurately obtain the boundary data of the placement surface. The virtual placement surface can be accurately generated by combining the boundary data with virtual reality technology.

[0065] For example, the specific generation process of the virtual placement surface is as follows:

[0066] The collision model of the placement surface can be obtained through plane detection. First, a depth image is collected through a depth camera, and the PCL library (Point Cloud Library) is used to process the environmental point cloud data obtained from the depth image. A model-based segmentation algorithm is used to obtain all the planes in the environment, that is, several environmental planes. The largest plane is extracted, that is, the largest environmental plane. Then a clustering algorithm is used to remove discrete points in the plane, the plane is uniformly sampled, and its bounding box is calculated. Finally, the boundary coordinates of the plane are calculated and a mesh message is generated. In actual application scenarios, the plane detection function can be integrated into the system through the DesktopDetection ROS service. When the service is called, a mesh representation of the placement surface used in virtual reality can be provided.

[0067] Step S200: acquiring posture data corresponding to a plurality of objects on the placement surface, and generating a plurality of opaque models on the virtual placement surface according to the posture data of each of the objects, wherein each of the opaque models is used to reflect an initial posture of a virtual object corresponding to one of the objects.

[0068] This embodiment can synchronize the posture of the virtual object in the virtual reality interface with the posture of the object in the real world through the posture estimation algorithm. Figure 2 As shown, for each object in reality, the visualization elements in the virtual scene also include a virtual object corresponding to the object. The image of the virtual object is an opaque model, and the posture of the opaque model is consistent with the posture of the corresponding object in reality.

[0069] In one implementation, the step of obtaining the position and posture data corresponding to the plurality of objects on the placement surface includes:

[0070] For each of the objects, obtaining the pose data of the object relative to the depth camera by using an object pose estimation algorithm;

[0071] Obtaining a transformation matrix between the coordinate system of the depth camera and the coordinate system of the robotic arm;

[0072] The position data of the object relative to the robotic arm is obtained according to the transformation matrix and the position data of the object relative to the depth camera.

[0073] Specifically, taking an object as an example, the object pose estimation algorithm DOPE based on a neural network can be used to obtain the pose data of the object relative to the depth camera. In order to obtain the coordinates of the object relative to the robotic arm, the transformation matrix from the camera coordinate system to the robotic arm base coordinate system can be obtained through hand-eye calibration. Finally, the coordinates of the object relative to the robotic arm base coordinate system can be obtained by using the transformation matrix for coordinate transformation, thereby obtaining the pose data of the object relative to the robotic arm.

[0074] In one implementation, the module corresponding to the object pose calculation can be integrated into the system through the ROS node, and the detected object id and its corresponding 6-DOF pose are published in real time through the object_pose topic. In the virtual scene, the object models in the HOPE dataset are pre-imported and hidden by default. After receiving the object_poses message, the corresponding opaque model is displayed according to its id and corresponding pose.

[0075] Step S300: If the current mode is the human-computer interaction control mode, for each of the opaque models, the transparent model overlapping with the opaque model is grabbed to the target position and released through the operator's hand motion data; after the release, the target position of the transparent model falling on the virtual placement surface is obtained by simulating free fall motion.

[0076] Specifically, Figure 3 As shown, the control interface in this embodiment includes a human-machine interaction control mode (Human Scene Interaction Control, HSI-Control). In the HSI control mode, the operator uses virtual reality technology to directly interact with the virtual environment and control the robot arm by moving virtual objects. Figure 2 As shown, for each object in reality, the visualization elements in the virtual scene also include a transparent model corresponding to the object. The transparent model can move with the operator's operation to reflect the target state of the object that the operator wants to move. By default, the opaque model and transparent model of each object overlap. In addition, due to possible operational errors, the object may not be in a completely stable posture when the operator places the virtual object. Therefore, after the operator puts down the object, the stable posture of the object falling on the placement surface can be calculated through gravity simulation and collision simulation, that is, the target posture is obtained.

[0077] For example, in the HSI control mode, the operator can start the control process by grabbing the virtual object he wants to move. At this time, the opaque model of the virtual object being moved remains stationary, and the corresponding transparent model moves with the operator's hand. The operator can move the transparent model to the target position and let go. After letting go, the gravity property of the transparent model will be activated. By simulating free fall motion through the Unity physics engine, the transparent model will fall and collide with the desktop. In the end, the transparent model will fall smoothly on the desktop, and the final target posture will be calculated. By combining physical simulation technology, this embodiment can improve the prediction accuracy of the final posture of the manipulated object, thereby improving the success rate of remote operation.

[0078] In one implementation, after the release, simulating free fall motion to obtain the target position of the transparent model falling on the virtual placement surface further includes:

[0079] Determining whether the target posture meets the standard;

[0080] If the standard is not met, the step of grabbing the transparent model overlapping with the opaque model to the target position and releasing it through the operator's hand motion data is executed again until the target posture meets the standard.

[0081] Step S400: Generate operation data corresponding to the robot arm according to the initial posture, the target posture and the manipulation sequence corresponding to each virtual object.

[0082] Specifically, the initial position, target position, and manipulation order of the virtual object can be used to guide the grasping and releasing operation performed by the robot arm in the real space, including indicating the coordinates before and after the object is moved and the order in which it is moved. For each object, the robot arm will first move above the object, perform a top-down grasp, then move the object to the target position, and finally release the gripper and put down the object. This embodiment controls the movement of the robot arm in reality through the interaction between the operator and the virtual scene, which can save operation time and reduce the burden and learning difficulty of the operator.

[0083] For example, when the operator completes the operation of the virtual object, he can click the "Execute" button on the virtual menu in the virtual scene. This button will call the ExecuteTask service. The service request contains the ID, initial posture and target posture of each object in the order of the operator's operation, and commands the robot to perform the corresponding task. The control node of this embodiment is developed based on the official driver of the Kinova Gen3 robot. When the ExecuteTask service is called from the HSI control mode, the robot will operate each object in turn to complete the task.

[0084] In one implementation, the method further includes:

[0085] Pre-generating a virtual robotic arm corresponding to the robotic arm in the virtual scene, wherein an end effector of the virtual robotic arm includes a virtual polyhedron;

[0086] If the current mode is the end effector control mode, dragging the virtual polyhedron according to the hand motion data of the operator;

[0087] The operation data of the robot arm is generated through the position and posture data of the virtual polyhedron, so as to realize the synchronous movement of the robot arm and the virtual polyhedron.

[0088] Specifically, Figure 2 As shown, the visualization elements of the virtual scene also include a virtual robot arm, and a virtual polyhedron, such as an icosahedron, is set on the end effector of the virtual robot arm. The pose of the virtual polyhedron is bound to the tool coordinate system of the end effector of the robot arm by default. After the operator grabs the virtual polyhedron, the pose binding is released, so that it can be freely moved or rotated. Figure 3As shown, the control interface in this embodiment also includes a traditional end-effector control mode (EE-Control). In the EE control mode, the operator can directly control the robot arm by dragging the virtual polyhedron.

[0089] For example, the virtual robot arm can be obtained by importing the virtual Kinova Gen3 robot arm model through URDF Importe, and synchronized with the real robot arm by receiving the joint_states topic. The pose of the virtual polyhedron will be published to the control node through the target_pose topic, so as to control the robot arm to move synchronously with the virtual polyhedron. In EE control mode, when the node receives the target_pose message, it will put the target pose into a fixed-length queue. The main program will loop to obtain the target pose from the queue and control the robot arm to move to the corresponding pose.

[0090] In one implementation, the method further includes:

[0091] If the operator's hand motion data is a preset target motion, a virtual menu is displayed in the virtual scene, wherein different areas in the virtual menu correspond to different control services;

[0092] The click data generated by the operator based on the virtual menu is acquired, and the control service of the corresponding area is called according to the click data.

[0093] Specifically, Figure 2 As shown, the visualization elements in the virtual scene also include a virtual menu, which appears when the operator performs a specific hand movement. The operator can call specific control services / commands in different areas of the virtual menu by clicking on the area.

[0094] For example, the virtual menu can be a hand menu, which appears when the operator extends his left hand. There are several buttons on the hand menu, each corresponding to a different command. The operator can call a specific command by clicking the corresponding button. For example, the "Grasp" button is used to control the closing of the mechanical gripper, and the "Release" button is used to control the opening of the mechanical gripper. When the two buttons are clicked, the GraspService and ReleaseService services can be called respectively.

[0095] In one implementation, calling a control service of a corresponding area according to the click data includes:

[0096] If the called control service is release, the gripper of the robot arm is controlled to open to a preset position;

[0097] If the called control service is grasping, the gripper of the robot arm is controlled to close at a preset speed, and when the motor current data is detected to exceed the preset current value, the motor speed is adjusted to zero.

[0098] Specifically, since some types of robotic grippers lack force sensors (such as the Robotiq 2f-85 robotic gripper), this embodiment provides two control modes to operate the gripper. For the release control service, this embodiment adopts a position control mode: when the ReleaseService is called, the program will control the gripper to a fully open position. For the grasping control service, this embodiment adopts a grasping control mode based on motor current feedback: when the GraspService is called, the gripper will close at a preset speed (for example, 0.1 m / s), and the program will monitor the motor current. If the current exceeds a preset value (for example, 0.12 A), it is considered to have grasped the object, and then the program will set the motor speed to zero.

[0099] Based on the above embodiments, the present invention also provides a robot arm control system based on scene editing and virtual reality, such as Figure 4 As shown, the system comprises:

[0100] The plane detection module 01 is used to generate a virtual placement surface corresponding to the placement surface in the virtual scene in advance;

[0101] A posture estimation module 02 is used to obtain posture data corresponding to a plurality of objects on the placement surface, and generate a plurality of opaque models on the virtual placement surface according to the posture data of each of the objects, wherein each of the opaque models is used to reflect an initial posture of a virtual object corresponding to one of the objects;

[0102] The virtual reality module 03 is used for, if the current mode is the human-computer interaction control mode, for each of the opaque models, grabbing the transparent model overlapping with the opaque model to the target position and releasing it according to the operator's hand motion data; after the release, obtaining the target position of the transparent model falling on the virtual placement surface by simulating free fall motion;

[0103] The robot arm control module 04 is used to generate operation data corresponding to the robot arm according to the initial posture, the target posture and the manipulation sequence corresponding to each virtual object.

[0104] It should be noted that the above explanation of the embodiment of the robotic arm control method based on scene editing and virtual reality is also applicable to the embodiment of the robotic arm control system based on scene editing and virtual reality, and will not be repeated here.

[0105] Figure 5The architecture and workflow of the system of this embodiment are demonstrated. First, the operator can see the virtual placement surface through the MR interface, and the virtual objects on the virtual placement surface will be synchronized with the objects on the real placement surface. The operator first operates the virtual object in the virtual scene, and after the operation is completed, the free fall motion of the virtual object is simulated to predict the target posture of the object falling on the placement surface. Then, through the initial posture, target posture and operation sequence of each virtual object, the robotic arm is automatically controlled to perform corresponding operations on the real object.

[0106] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 6 As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a robotic arm control method based on scene editing and virtual reality is implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0107] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0108] In one implementation, the terminal has one or more programs stored in its memory, and is configured to be executed by one or more processors. The one or more programs include instructions for performing a robotic arm manipulation method based on scene editing and virtual reality.

[0109] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0110] In summary, the present invention discloses a method and system for manipulating a robotic arm based on scene editing and virtual reality. The method uses virtual reality technology to enable an operator to first operate a virtual object on a virtual placement surface in a virtual scene. After the operation is completed, the free fall motion of the virtual object is simulated to predict the target position of the object falling on the placement surface. Then, through the initial position, target position and manipulation sequence of each virtual object, the robotic arm is automatically controlled to perform corresponding operations on the real object. In the present invention, the operator only needs to focus on modifying and editing the virtual scene and virtual objects, and does not need to worry about the motion control of the robotic arm. Therefore, the operator's workload can be effectively reduced, and the success rate and efficiency of task execution can be improved.

[0111] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for manipulating a robotic arm based on scene editing and virtual reality, characterized in that: The method comprises: Pre-generating a virtual placement surface corresponding to the placement surface in the virtual scene; Acquire posture data corresponding to a plurality of objects on the placement surface, and generate a plurality of opaque models on the virtual placement surface according to the posture data of each of the objects, wherein each of the opaque models is used to reflect an initial posture of a virtual object corresponding to one of the objects; If the current mode is the human-computer interaction control mode, for each of the opaque models, the transparent model overlapping with the opaque model is grabbed to the target position and released according to the operator's hand motion data; after the release, the target position of the transparent model falling on the virtual placement surface is obtained by simulating free fall motion; The operation data corresponding to the robot arm are generated according to the initial posture, the target posture and the manipulation sequence corresponding to each virtual object.

2. The method for manipulating a robot arm based on scene editing and virtual reality according to claim 1, characterized in that: The step of pre-generating a virtual placement surface corresponding to the placement surface in the virtual scene includes: Acquiring environmental point cloud data through a depth camera, wherein the environmental point cloud data is used to reflect the surrounding environment of the robotic arm, including the placement surface; Obtaining a plurality of environmental planes by using a preset segmentation algorithm and environmental point cloud data, and extracting a maximum environmental plane from each of the environmental planes; Boundary data is determined according to the maximum environment plane, and a placement surface model is generated in the virtual scene according to the boundary data to obtain the virtual placement surface.

3. The method for manipulating a robot arm based on scene editing and virtual reality according to claim 2, characterized in that: The step of determining boundary data according to the maximum environment plane comprises: removing discrete points in the maximum environment plane and uniformly sampling the maximum environment plane; The boundary data is calculated according to the uniform sampling result.

4. The method for manipulating a robot arm based on scene editing and virtual reality according to claim 2, characterized in that: The obtaining of the position and posture data corresponding to the plurality of objects on the placement surface includes: For each of the objects, obtaining the pose data of the object relative to the depth camera by using an object pose estimation algorithm; Obtaining a transformation matrix between the coordinate system of the depth camera and the coordinate system of the robotic arm; The position data of the object relative to the robotic arm is obtained according to the transformation matrix and the position data of the object relative to the depth camera.

5. The method for manipulating a robot arm based on scene editing and virtual reality according to claim 1, characterized in that: The method further comprises: Pre-generating a virtual robotic arm corresponding to the robotic arm in the virtual scene, wherein an end effector of the virtual robotic arm includes a virtual polyhedron; If the current mode is the end effector control mode, dragging the virtual polyhedron according to the hand motion data of the operator; The operation data of the robot arm is generated by the position and posture data of the virtual polyhedron, so as to realize the synchronous movement of the robot arm and the virtual polyhedron.

6. The method for manipulating a robot arm based on scene editing and virtual reality according to claim 1, characterized in that: The method further comprises: If the operator's hand motion data is a preset target motion, a virtual menu is displayed in the virtual scene, wherein different areas in the virtual menu correspond to different control services; The click data generated by the operator based on the virtual menu is acquired, and the control service of the corresponding area is called according to the click data.

7. The method for manipulating a robot arm based on scene editing and virtual reality according to claim 6, characterized in that: The calling of the control service of the corresponding area according to the click data includes: If the called control service is release, the gripper of the robot arm is controlled to open to a preset position; If the called control service is grasping, the gripper of the robot arm is controlled to close at a preset speed, and when the motor current data is detected to exceed the preset current value, the motor speed is adjusted to zero.

8. A robotic arm control system based on scene editing and virtual reality, characterized in that: The system comprises: A plane detection module, used to generate a virtual placement surface corresponding to the placement surface in the virtual scene in advance; A posture estimation module, used to obtain posture data corresponding to a plurality of objects on the placement surface, and generate a plurality of opaque models on the virtual placement surface according to the posture data of each of the objects, wherein each of the opaque models is used to reflect an initial posture of a virtual object corresponding to one of the objects; A virtual reality module, for grabbing a transparent model overlapping with the opaque model to a target position and releasing it according to the operator's hand motion data if the current mode is a human-computer interaction control mode; after the release, obtaining a target position of the transparent model falling on the virtual placement surface by simulating free fall motion; The robot arm control module is used to generate operation data corresponding to the robot arm according to the initial posture, the target posture and the manipulation sequence corresponding to each virtual object.

9. A terminal, characterized in that: The terminal includes a memory and one or more processors; the memory stores one or more programs; the program contains instructions for executing the robotic arm control method based on scene editing and virtual reality as described in any one of claims 1-7; and the processor is used to execute the program.

10. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the robot arm control method based on scene editing and virtual reality as described in any one of claims 1 to 7 above.

Images