Teleoperation method and device of robot, terminal equipment and storage medium

Abstract

The invention is suitable for the technical field of robot control, and provides a teleoperation method and device of a robot, terminal equipment and a storage medium, and the method comprises the steps: collecting first position data of key points on a user body through VR equipment; and then coordinate conversion is carried out on the first position data to obtain second position data of the first position data in a robot coordinate system, joint control parameters of the robot are determined according to the second position data, and finally the robot is controlled according to the joint control parameters. The position data of the key points of the body is acquired by using the VR equipment, so that the acquired body data is more accurate and more flexible; according to the technical scheme, the body data collected by the VR equipment are converted into the robot coordinate system, then the purpose of controlling the robot according to the body data collected by the VR equipment is achieved, teleoperation of the robot can be completed without specific teleoperation equipment, and teleoperation of the robot is more convenient and simpler.

Description

Technical Field

[0001] The present application belongs to the field of robot control technology, and in particular, relates to a remote operation method, device, terminal equipment and storage medium of a robot. Background Art

[0002] Teleoperation is usually remote operation, through which human actions can be mapped to the robot so that the robot can act according to the human actions. At present, in order to realize the remote operation of the robot, a specific teleoperation device is often required. Only through the specific teleoperation device can the remote operation of the robot be realized. If there is no teleoperation device, the remote operation of the robot cannot be realized, which limits the remote operation of the robot and brings difficulties to the remote operation of the robot. Summary of the invention

[0003] The embodiments of the present application provide a remote operation method, apparatus, terminal device and storage medium for a robot, which can solve the problem that the remote operation of the robot is limited and the remote operation of the robot is difficult.

[0004] In a first aspect, an embodiment of the present application provides a remote operation method of a robot, comprising:

[0005] Acquire first position data of a key point on the user's body collected by a VR device, wherein the first position data is position data of the key point in a VR coordinate system in the VR device;

[0006] Performing coordinate transformation on the first position data to obtain second position data corresponding to the first position data, wherein the second position data is position data of the key point in a robot coordinate system, and the robot coordinate system is a coordinate system in which the robot to be controlled is located;

[0007] Based on the second position data, determining joint control parameters when the robot reaches the current state of the user;

[0008] The robot is controlled to move based on the joint control parameters.

[0009] In a possible implementation manner of the first aspect, performing coordinate conversion on the first position data to obtain second position data corresponding to the first position data includes:

[0010] If the first position data is the first position data of the first frame, a target coordinate system is created according to the first position data of the first frame, the direction of the coordinate axis of the target coordinate system is consistent with the direction of the coordinate axis of the robot coordinate system, and the first position data of the first frame is the first position data obtained after the user acts according to the preset requirement, and the preset requirement is determined based on the creation rule of the robot coordinate system;

[0011] Based on the position of the first position data of the first frame in the target coordinate system, obtain the second position data corresponding to the first position data of the first frame, and the coordinate transformation relationship between the target coordinate system and the VR coordinate system;

[0012] If the first position data is the first position data of the i-th frame, i≥2, coordinate transformation is performed on the first position data of the i-th frame based on the coordinate transformation relationship to obtain second position data corresponding to the first position data of the i-th frame.

[0013] In a possible implementation manner of the first aspect, the first position data includes position data of a user's hand, the hand position data includes position data of multiple key points of a palm and position data of key points of each finger, and the hand position data in the first position data of the first frame is position data obtained after the user places his hand according to the preset requirement;

[0014] The step of creating a target coordinate system according to the first position data of the first frame includes:

[0015] Searching for first target data and second target data from the first position data of the first frame, wherein the first target data is position data of at least three key points of a palm, and the second target data is position data of multiple key points on a target finger, and the target finger is a finger determined according to the preset requirement and the creation rule;

[0016] Constructing an xy plane of the target coordinate system according to the first target data;

[0017] Determining the direction of the x-axis of the target coordinate system on the xy plane according to the second target data;

[0018] Finding a key point on the x-axis of the target coordinate system in the first position data of the first frame;

[0019] The target coordinate system is created by taking the target key point as the origin of the target coordinate system, wherein the target key point is a key point located on the x-axis of the target coordinate system.

[0020] In a possible implementation manner of the first aspect, determining, based on the second position data, a joint control parameter when the robot reaches the current state of the user includes:

[0021] After acquiring the second position data corresponding to the first position data of the j-th frame, saving the second position data corresponding to the first position data of the j-th frame, j≥1;

[0022] Before obtaining the second position data corresponding to the first position data of the j+1th frame, determining the joint control parameters of the robot according to the second position data corresponding to the first position data of the jmth frame according to the control frequency, 0≤m<j;

[0023] After acquiring the second position data corresponding to the first position data of the j+1th frame, saving the second position data corresponding to the first position data of the j+1th frame;

[0024] The joint control parameters of the robot are determined according to the second position data corresponding to the first position data of the j-m+1th frame according to the control frequency.

[0025] In a possible implementation manner of the first aspect, determining, based on the second position data, a joint control parameter when the robot reaches the current state of the user includes:

[0026] Based on the second position data, determining deviation data between the current state of the robot and the second position data;

[0027] Based on the deviation data and the second position data, joint control parameters when the robot reaches the current state of the user are determined.

[0028] In a possible implementation manner of the first aspect, the method further includes:

[0029] Acquire image data of a user's body collected through a VR device;

[0030] extracting body movements of the user from the image data;

[0031] If the body movement matches the preset movement, determining the operation mode corresponding to the body movement;

[0032] The robot is switched to the operating mode.

[0033] In a possible implementation manner of the first aspect, the body movement is a gesture movement;

[0034] If the body motion matches the preset motion, determining the operation mode corresponding to the body motion includes:

[0035] If the gesture action matches the preset action, the operation mode corresponding to the gesture action is determined.

[0036] In a second aspect, an embodiment of the present application provides a remote operation device for a robot, comprising:

[0037] A data acquisition module, used to acquire first position data of a key point on the user's body collected by a VR device, wherein the first position data is position data of the key point in a VR coordinate system in the VR device;

[0038] a coordinate conversion module, configured to perform coordinate conversion on the first position data to obtain second position data corresponding to the first position data, wherein the second position data is position data of the key point in a robot coordinate system, and the robot coordinate system is a coordinate system in which the robot to be controlled is located;

[0039] a parameter determination module, configured to determine, based on the second position data, joint control parameters when the robot reaches the current state of the user;

[0040] A control module is used to control the robot to move based on the joint control parameters.

[0041] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the remote operation method of the robot described in any one of the first aspects above is implemented.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the remote operation method of the robot described in any one of the first aspects above is implemented.

[0043] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the remote operation method of a robot as described in any one of the above-mentioned first aspects.

[0044] Compared with the prior art, the beneficial effects of the first aspect of the present application are as follows: the present application first obtains the first position data of the key points on the user's body collected by the VR device; then performs coordinate conversion on the first position data to obtain the second position data of the first position data in the robot coordinate system, determines the joint control parameters of the robot according to the second position data, and finally controls the robot according to the joint control parameters. The present application uses the VR device to collect the position data of the key points of the body, which can make the collected body data more accurate and flexible; converts the body data collected by the VR device to the robot coordinate system, thereby achieving the purpose of controlling the robot according to the body data collected by the VR device. The present application can complete the remote operation of the robot without a specific remote operation device, making the remote operation of the robot more convenient and simpler.

[0045] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 is a flowchart of a remote operation method of a robot provided in one embodiment of the present application;

[0048] Figure 2 It is a flowchart of a method for creating a target coordinate system provided in an embodiment of the present application;

[0049] Figure 3 It is a flowchart of a method for switching the operation mode of a robot provided in one embodiment of the present application;

[0050] Figure 4 is a flowchart of a remote operation method of a robot provided in another embodiment of the present application;

[0051] Figure 5 is a schematic structural diagram of a remote operation device of a robot provided in one embodiment of the present application;

[0052] Figure 6 It is a structural diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0053] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0054] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0055] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0056] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0057] Teleoperation is the operation of a machine or device by remote control, particularly where robots are involved, where the operator can operate the machine from a distance.

[0058] Teleoperation can map human actions to robots, making them perform the same actions as humans, thus realizing complex movements of robots. At present, when teleoperating robots, specific teleoperation equipment is often required to realize teleoperation of robots. For example, a small robot or robotic arm is needed to control the small robot or robotic arm to perform actions, so as to realize teleoperation of the robot to be controlled. If there is no specific teleoperation equipment, the robot cannot be teleoperated, so that the teleoperation of the robot is limited by the equipment.

[0059] Based on the above problems, the present application proposes a remote operation method of a robot. The present application collects the position data of key points of the user's body, for example, the position data of key points of the hands, through a VR device, and then controls the robot to move according to the collected position data, so that the robot can move in accordance with the user's movements; the present application can realize remote operation of the robot through a VR device, without the need to use specific remote operation equipment, making remote operation of the robot simpler and more convenient.

[0060] The following combination Figure 1 The remote operation method of the robot in the embodiment of the present application is described in detail.

[0061] Figure 1 A schematic flow chart of the remote operation method of the robot provided by the present application is shown, referring to Figure 1 , the method is described in detail as follows:

[0062] S101, obtaining first position data of a key point on a user's body collected by a VR device, wherein the first position data is position data of the key point in a VR coordinate system in the VR device.

[0063] In this embodiment, a camera is provided on the VR device, and the camera on the VR device collects images of the user, and determines the position coordinates of key points on the user's body in the VR coordinate system through the collected images. The position coordinates of the key points on the user's body determined in this application are marked as first position data. The VR coordinate system is a preset coordinate system.

[0064] As an example, if it is necessary to collect the first position data of the key points of the user's hand, VR glasses can be used for collection. Specifically, the user wears VR glasses, and the VR glasses collect the user's hand image, and then determine the first position data of the key points of the hand based on the hand image.

[0065] In this embodiment, the selection of key points can be set as needed. For example, for the user's hand, multiple position points on the palm and multiple position points on each finger can be selected as the key points of the hand. The key points on the palm can include the palm and multiple points around the palm.

[0066] In this embodiment, after the remote operation starts, the VR device can collect the first position data according to a preset collection frequency. For example, the preset collection frequency can be 5ms or 6ms.

[0067] S102, performing coordinate transformation on the first position data to obtain second position data corresponding to the first position data, wherein the second position data is the position data of the key point in a robot coordinate system, and the robot coordinate system is a coordinate system in which the robot to be controlled is located.

[0068] In this embodiment, since the first position data includes position data of multiple key points, it is necessary to perform coordinate transformation on the position data of each key point to obtain the position coordinates of each key point in the robot coordinate system. The position coordinates of all key points in the robot coordinate system are marked as second position data.

[0069] In this embodiment, the robot coordinate system may be a known coordinate system.

[0070] If the robot coordinate system is a known coordinate system, the conversion relationship between the VR coordinate system and the robot coordinate system can be preset. After obtaining the first position data, the first position data is converted according to the conversion relationship between the VR coordinate system and the robot coordinate system to obtain the second position data.

[0071] In this embodiment, the robot coordinate system may be an unknown coordinate system.

[0072] If the coordinate system of the robot is an unknown coordinate system, a coordinate system identical to the robot coordinate system can be created according to the first position data according to the creation rules of the robot coordinate system. In this application, the coordinate system created according to the first position data is recorded as the target coordinate system.

[0073] If a target coordinate system identical to the robot coordinate system is created based on the first position data, the conversion relationship between the VR coordinate system and the robot coordinate system can be determined based on the position coordinates of the first position data in the robot coordinate system (or target coordinate system); after determining the conversion relationship between the VR coordinate system and the robot coordinate system, the first position data received thereafter can be directly converted according to the conversion relationship between the VR coordinate system and the robot coordinate system to obtain the second position data.

[0074] S103: Determine, based on the second position data, joint control parameters of the robot when it reaches the current state of the user.

[0075] In this embodiment, the expected posture of the robot can be determined based on the second position data. For example, if the second position data includes the position data of the key points of the user's hand, the expected posture of the robot's hand can be determined based on the second position data; the joint control parameters of each joint in the robot can be determined based on the expected posture of the robot and the current posture of the robot, for example, the joint control parameters of the hand joints.

[0076] In this embodiment, if the fingers of the robot need to be controlled, the finger joint angles of the robot may be determined according to the second position data, and the joint control parameters include the finger joint angles.

[0077] S104, controlling the robot to move based on the joint control parameters.

[0078] In this embodiment, the joints in the robot are matched with control motors, and the joint control parameters are sent to the control motors to control the robot to move, so that the robot reaches a desired posture, so that the robot can move following the user's movements.

[0079] As an example, if the hand of the robot needs to be controlled, the desired posture of the hand of the robot can be determined and sent to the control motor of the hand joint of the robot to control the movement of the hand of the robot.

[0080] It should be noted that the real robot and the simulated robot can be controlled simultaneously based on the joint control parameters.

[0081] In this application, the first position data of the key points on the user's body collected by the VR device is first obtained; then the first position data is converted into coordinates to obtain the second position data of the first position data in the robot coordinate system, the joint control parameters of the robot are determined according to the second position data, and finally the robot is controlled according to the joint control parameters. This application uses VR equipment to collect the position data of the key points of the body, which can make the collected body data more accurate and flexible; the body data collected by the VR device is converted to the robot coordinate system, thereby achieving the purpose of controlling the robot according to the body data collected by the VR device. This application can complete the remote operation of the robot without a specific remote operation device, making the remote operation of the robot more convenient and simpler.

[0082] In this application, if the robot coordinate system is unknown, a target coordinate system identical to the robot coordinate system can be created based on the first position data. Mapping the first position data collected at different times in the target coordinate system is equivalent to converting the first position data into the robot coordinate system, and the obtained second position data can be directly used to control the robot.

[0083] Specifically, in order to make the created target coordinate system the same as the robot coordinate system, the user is required to perform actions according to preset requirements after the remote operation begins, so that the VR device can collect the first position data of the user's key points under the action, that is, the first position data of the first frame, and then create the target coordinate system based on the first position data of the first frame. After the target coordinate system is created, no matter how the user moves, the collected first position data can be mapped to the target coordinate system for coordinate conversion to obtain the second position data.

[0084] Therefore, the specific process of step S102 may include:

[0085] S1021, if the first position data is the first position data of the first frame, create a target coordinate system based on the first position data of the first frame, the direction of the coordinate axis of the target coordinate system is consistent with the direction of the coordinate axis of the robot coordinate system, and the first position data of the first frame is the first position data obtained after the user acts according to preset requirements, and the preset requirements are determined based on the creation rules of the robot coordinate system.

[0086] In this embodiment, after the remote operation begins, the user performs actions according to the preset requirements. For example, the user can place the hand parallel to the ground and point the middle finger directly in front of the user according to the preset requirements. The first position data of the first frame collected by the VR device is the position data of each key point obtained after the user performs actions according to the preset requirements.

[0087] Specifically, after the remote operation begins, the VR device can display action prompts that the user needs to perform according to preset requirements. The action prompts can be text prompts and / or image prompts, etc., so that the user can perform actions according to the action prompts.

[0088] In this embodiment, the first position data is input into the neural network model, and the target coordinate system is created through the neural network model.

[0089] In this embodiment, a target coordinate system is created according to preset creation requirements and the first position data of the first frame. The preset creation requirements specify the requirements for the xy plane of the target coordinate system and the direction of each coordinate axis. For example, the plane where the palm is located is selected as the xy plane, and the direction pointed by the middle finger is selected as the direction of the x-axis, etc.

[0090] S1022: Based on the position of the first position data of the first frame in the target coordinate system, obtain second position data corresponding to the first position data of the first frame, and a coordinate conversion relationship between the target coordinate system and the VR coordinate system.

[0091] In this embodiment, since the target coordinate system is created based on the first position data of the first frame, the coordinate conversion relationship between the target coordinate system and the VR coordinate system can be obtained based on the position coordinates of the first position data of the first frame in the target coordinate system. For example, the center point of the palm is taken as the origin of the target coordinate system, and the first position data includes the coordinates of the center point of the palm in the VR coordinate system. Therefore, the coordinate conversion relationship between the target coordinate system and the VR coordinate system can be obtained based on the coordinates of the center point of the palm in the VR coordinate system and the coordinates of the center point of the palm in the target coordinate system.

[0092] S1023: If the first position data is the first position data of the i-th frame, i≥2, coordinate transformation is performed on the first position data of the i-th frame based on the coordinate transformation relationship to obtain second position data corresponding to the first position data of the i-th frame.

[0093] In this embodiment, since the coordinate transformation relationship between the target coordinate system and the VR coordinate system has been determined based on the first position data of the first frame, the first position data received after the first position data of the first frame can be directly transformed according to the coordinate transformation relationship, thereby reducing the processing time of the first position data of the i-th frame.

[0094] In the present application, a target coordinate system is created based on the first position data obtained after the user acts according to preset requirements. The created target coordinate system can be made the same as the robot coordinate system. Mapping the first position data in the target coordinate system is equivalent to mapping the first position data in the robot coordinate system, which reduces the coordinate calibration process and saves data processing.

[0095] In a possible implementation, the first position data includes position data of the user's hand, and the hand position data includes position data of multiple key points of the palm and position data of key points of each finger. For example, the palm may have 5 key points, each finger may have 3 key points, and so on.

[0096] After the teleoperation begins, the user can place the hand according to the preset requirements so that the first position data of the first frame collected includes the position data of the hand, so as to create a target coordinate system according to the position data of the hand.

[0097] like Figure 2 Specifically, the specific implementation process of step S1021 may include:

[0098] S201, searching for first target data and second target data from the first position data of the first frame, wherein the first target data is position data of at least three key points of the palm, and the second target data is position data of multiple key points on a target finger, and the target finger is a finger specified in the creation rule.

[0099] In this embodiment, the position data of each key point in the first position data can be marked with the identification information of the key point. For example, the identification information on the position data of the key point on the palm can be the identification A corresponding to the palm. According to the identification A, the position data can be determined as the position data of the key point on the palm.

[0100] Since three points can determine a plane, the position data of at least three key points of the palm can be found, so as to determine the xy plane of the target coordinate system through the position data of the three key points of the palm.

[0101] In this embodiment, the target finger may be the middle finger, thumb or index finger, etc. For example, if the preset requirement is that the hand is parallel to the ground and the middle finger points to the front of the body, and the creation rule is that the front of the robot body is the direction of the x-axis of the robot coordinate system, since the direction of the middle finger is the same as the direction of the x-axis, in order to make the direction of the x-axis of the target coordinate system the same as the direction of the x-axis of the robot coordinate system, the middle finger is selected as the target finger.

[0102] S202: construct an xy plane of the target coordinate system according to the first target data.

[0103] S203: Determine the direction of the x-axis of the target coordinate system on the xy plane according to the second target data.

[0104] In this embodiment, the first position data of the first frame includes position data of the center point of the palm, and the direction of the second target data relative to the center point of the palm is determined as the direction of the x-axis of the target coordinate system.

[0105] Alternatively, a target finger is constructed according to the second target data, and the direction pointed by the target finger is determined as the direction of the x-axis of the target coordinate system.

[0106] S204: Searching for a key point on the x-axis of the target coordinate system in the first position data of the first frame.

[0107] S205: Create the target coordinate system by taking the target key point as the origin of the target coordinate system, wherein the target key point is a key point on the x-axis of the target coordinate system.

[0108] In this embodiment, since the origin of the target coordinate system must be on the x-axis of the target coordinate system, it is sufficient to search for key points on the x-axis and select one key point from the key points on the x-axis as the origin.

[0109] As an example, if the target finger is the middle finger, the center point of the palm can be selected as the origin of the target coordinate system.

[0110] In this embodiment, after the x-axis of the target coordinate system is determined, the direction passing through the origin and perpendicular to the xy plane can be determined as the direction of the z-axis of the target coordinate system; after determining the x-axis and z-axis, the y-axis of the target coordinate system can be determined according to the right-hand rule. At this point, the target coordinate system is obtained after the xy plane, x-axis, y-axis and z-axis are determined.

[0111] In a possible implementation, the VR device collects the first position data at a data collection frequency, for example, the data collection frequency may be 5 ms. When controlling the robot according to the second position data, the second position data is obtained at a preset control frequency to control the robot, and the control frequency may be 1 ms, etc.

[0112] If the data collection frequency is the same as the control frequency of the robot, after the VR device collects a first position data, the robot can be controlled according to the second position data corresponding to the latest collected first position data.

[0113] If the data collection frequency is greater than the control frequency of the robot, the robot may also be controlled according to the second position data corresponding to the most recently obtained first position data.

[0114] If the data collection frequency is lower than the control frequency of the robot, in order to ensure real-time control of the robot, the robot may be controlled using the second position data obtained the last time before obtaining the latest first position data.

[0115] Therefore, in order to meet the smoothness of robot control under different circumstances, the second position data can be stored in the form of a shared queue, and each time the second position data is obtained, it is saved in the shared queue. After a new frame of second position data is obtained, the header data of the shared queue is deleted. When the robot is controlled according to the control frequency, the header data of the shared queue is read each time, and the joint control parameters of the robot are determined according to the header data. This design can control the robot according to the header data even if the VR device has not collected the first position data for a long time, which ensures the smoothness of the robot control and the safety of the robot operation.

[0116] Specifically, the implementation process of the above step S103 may include:

[0117] S1031, after acquiring the second position data corresponding to the first position data of the j-th frame, save the second position data corresponding to the first position data of the j-th frame, where j≥1.

[0118] In this embodiment, the first position data collected by the VR device each time is subjected to coordinate conversion to obtain the second position data corresponding to the first position data collected each time, and the second position data is saved in the order of collection time.

[0119] S1032, before obtaining the second position data corresponding to the first position data of the j+1th frame, determine the joint control parameters of the robot according to the second position data corresponding to the first position data of the jmth frame according to the control frequency, 0≤m<j.

[0120] In this embodiment, after obtaining the second position data corresponding to the first position data of the jth frame, the second position data before the second position data of the jmth frame can be deleted to reduce the amount of data storage. For example, if jm=3, the second position data of the first frame and the second position data of the second frame can be deleted.

[0121] S1033: After acquiring the second position data corresponding to the first position data of the j+1th frame, save the second position data corresponding to the first position data of the j+1th frame.

[0122] S1034, determining the joint control parameters of the robot according to the second position data corresponding to the first position data of the j-m+1th frame according to the control frequency.

[0123] In this embodiment, after the second position data corresponding to the first position data of the j+1th frame is acquired, the second position data before the second position data of the j-m+1th frame may be deleted.

[0124] As an example, when m=0, j=3, after obtaining the second position data corresponding to the first position data of the third frame, the second position data corresponding to the first position data of the third frame is saved; before obtaining the second position data corresponding to the first position data of the fourth frame, the joint control parameters of the robot are determined according to the second position data corresponding to the first position data of the third frame according to the control frequency; after obtaining the second position data corresponding to the first position data of the fourth frame, the second position data corresponding to the first position data of the fourth frame is saved; and the joint control parameters of the robot are determined according to the second position data corresponding to the first position data of the fourth frame according to the control frequency.

[0125] In the application, before obtaining the next frame of second position data, the previously collected frame of second position data is used to determine the joint control parameters of the robot, thereby ensuring the smoothness of the robot control, uninterrupted control of the robot, and ensuring the safety of the robot control.

[0126] In a possible implementation, the implementation process of step S103 may further include:

[0127] S301, based on the second position data, determining deviation data between the current state of the robot and the second position data.

[0128] In this embodiment, since the second position data represents the expected position data of the robot, the current state of the robot can represent the current position data of each part of the robot. By comparing the current state of the robot with the second position data, the deviation data between the current state of the robot and the second position data can be obtained.

[0129] S302: Determine joint control parameters of the robot based on the deviation data.

[0130] In this embodiment, the deviation data is input into a proportional-derivative controller (PD controller) to obtain joint control parameters.

[0131] Alternatively, the desired angles of the robot's joints are calculated based on the deviation data and the terminal Jacobian matrix, and the joint control parameters are determined based on the desired angles. The joint control parameters include velocity parameters, acceleration parameters, angular velocity, etc. of the joints.

[0132] In one possible implementation, in order to control the robot's mode conversion, the VR device can also collect image data of the user's body, determine body movements based on the image data, and determine the robot's operating mode based on the user's body movements.

[0133] like Figure 3Specifically, the above method may further include:

[0134] S401, obtaining image data of the user's body collected by a VR device.

[0135] In this embodiment, a camera installed on the VR device collects image data of the user's body at a preset frequency.

[0136] S402: Extracting the body movements of the user from the image data.

[0137] In this embodiment, the body movement may include gesture movement, leg movement or head movement, etc.

[0138] Specifically, image recognition is performed on the image data, images of key parts are extracted, and body movements are determined based on the images of key parts. For example, images of hands are extracted from the image data, and gestures are determined based on the images of hands.

[0139] S403: If the body movement matches a preset movement, determine an operation mode corresponding to the body movement.

[0140] In this embodiment, the body movement is compared with the preset movement to determine the degree of match between the preset movement and the body movement. If the degree of match is greater than a preset threshold (e.g., 80%), it is determined that the preset movement matches the body movement; if the degree of match is less than or equal to the preset threshold, it is determined that the preset movement does not match the body movement.

[0141] In this embodiment, an operation mode corresponding to each preset action is preset, and the operation mode may include: stop remote operation mode, start remote operation mode, and the like.

[0142] In one implementation, the operation mode is controlled by a gesture action. If the gesture action matches a preset action, the operation mode corresponding to the gesture action is determined.

[0143] For example, the preset action is: pinch the thumb and the little finger and hold for 2 seconds; the operation mode corresponding to the preset action is the stop remote operation mode.

[0144] S404, switching the robot to the operation mode.

[0145] In the present application, the operation mode of the robot is controlled according to body movements, and the user can easily switch the operation mode of the robot through movements, thereby realizing the control of the operation mode through remote operation and improving the performance of remote operation of the robot.

[0146] In a possible implementation, the method of the present application can realize the control of the hand of the robot.

[0147] Specifically, the above method may further include:

[0148] Acquire first position data of a key point of a user's hand collected by a VR device, wherein the first position data is position data of the key point in a VR coordinate system in the VR device.

[0149] The first position data is subjected to coordinate transformation to obtain second position data corresponding to the first position data, wherein the second position data is the position data of the key point in a robot coordinate system, and the robot coordinate system is a coordinate system in which the robot to be controlled is located.

[0150] Based on the second position data, joint control parameters of the hand of the robot are determined.

[0151] The robot's hand is controlled to move based on the joint control parameters.

[0152] It should be noted that the method of the present application can be implemented in electronic devices (for example, mobile phones, tablet computers, etc.) and can also be implemented in robots.

[0153] If the present application is implemented through an electronic device, the implementation process of the above step S104 may include: sending joint control parameters to the robot, and the robot takes action according to the joint control parameters after receiving the joint control parameters.

[0154] In a possible implementation, the method of the present application can also be implemented in both an electronic device and a robot.

[0155] like Figure 4 As shown, specifically, taking the control of the robot's hand as an example, the method of the present application may also include:

[0156] S11, collecting first position data of key points of the user's hand through a VR device, and sending the first position data to an electronic device, wherein the first position data is the position data of the key points in a VR coordinate system in the VR device.

[0157] S12, the electronic device determines whether the first position data is the first position data of the first frame.

[0158] S13: If the first position data is the first position data of the first frame, the electronic device creates a target coordinate system according to the first position data of the first frame.

[0159] S14, the electronic device obtains the second position data corresponding to the first position data of the first frame and the coordinate conversion relationship between the target coordinate system and the VR coordinate system based on the position of the first position data of the first frame in the target coordinate system, and sends the second position data to the robot.

[0160] S15, if the first position data is the first position data of the i-th frame, i≥2, the electronic device performs coordinate transformation on the first position data of the i-th frame based on the coordinate transformation relationship to obtain the second position data corresponding to the first position data of the i-th frame, and the electronic device sends the second position data to the robot.

[0161] S16, after the robot receives the second position data sent by the electronic device, the robot stores the second position data in a shared queue in chronological order.

[0162] S17, the robot reads the head data in the shared queue according to the control frequency, and calculates the joint control parameters of the robot according to the head data.

[0163] In this embodiment, each time the robot reads the header data in the shared queue, after a new frame of second position data is obtained, the previously stored header data is deleted so that the header data of the shared queue is updated.

[0164] S18, the robot moves according to the joint control parameters.

[0165] In one implementation, the VR device and the electronic device communicate using sockets, for example, ZeroMQ (ZMQ) communication. In this communication mode, the VR device and the electronic device can communicate as long as they are in the same local area network.

[0166] In addition, asynchronous communication can be used between the VR device and the electronic device. After the VR device collects data, the VR device sends the collected data to the electronic device, and the electronic device starts after detecting the data sent by the VR device.

[0167] In actual use, the VR device in this application can send the collected first position data to the local area network, and all devices in the local area network (such as electronic devices and robots) can receive the first position data. Since all devices in the local area network can receive the first position data sent by the VR device, this application can remotely operate multiple robots at the same time.

[0168] In a possible implementation, during the teleoperation process, first motion data of the user at each moment and second motion data of the robot at each moment are recorded, the first motion data including a first motion image, and the second motion data including a second motion image.

[0169] In one method, after the operation is completed, the first action image at different times and the second action image corresponding to the first action image are selected. The first action image and the second action image matching therewith are compared to determine the accuracy of the teleoperation. For example, the first angle between the upper arm and the lower arm of the user is determined by the first action image; the second angle between the upper arm and the lower arm of the robot is determined by the second action image; and the accuracy of the teleoperation can be determined by comparing the first angle and the second angle.

[0170] The average of all the accuracies obtained is calculated to obtain the mean accuracy.

[0171] If the mean value of the accuracy of the teleoperation determined according to the first motion image and the second motion image is less than a preset threshold, it is determined that the accuracy of the teleoperation is poor and the parameters of the teleoperation need to be corrected.

[0172] Specifically, the correction of the teleoperation parameters mainly includes correction of the coordinate conversion parameters when the first position data is converted to the second position data and the parameters used when the joint control parameters are determined according to the second position data.

[0173] In another embodiment, the method of determining joint control parameters according to the first position data of the present invention is implemented in a robot control model. The first motion image and the second motion image are input into the robot control model to train the robot control model and update the parameters in the robot control model. The parameters in the robot control model include coordinate conversion parameters when the first position data is converted to the second position data and parameters used when determining joint control parameters according to the second position data.

[0174] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0175] Corresponding to the remote operation method of the robot described in the above embodiment, Figure 5 A structural block diagram of a remote operation device of a robot provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0176] Reference Figure 5 The device 500 may include: a data acquisition module 510, a coordinate conversion module 520, a parameter determination module 530 and a control module 540.

[0177] The data acquisition module 510 is used to acquire first position data of a key point on the user's body collected by a VR device, wherein the first position data is position data of the key point in a VR coordinate system in the VR device;

[0178] A coordinate conversion module 520 is used to perform coordinate conversion on the first position data to obtain second position data corresponding to the first position data, wherein the second position data is the position data of the key point in a robot coordinate system, and the robot coordinate system is the coordinate system in which the robot to be controlled is located;

[0179] A parameter determination module 530, for determining, based on the second position data, joint control parameters when the robot reaches the current state of the user;

[0180] The control module 540 is used to control the robot to move based on the joint control parameters.

[0181] In a possible implementation, the coordinate conversion module 520 may be specifically used for:

[0182] If the first position data is the first position data of the first frame, a target coordinate system is created according to the first position data of the first frame, the direction of the coordinate axis of the target coordinate system is consistent with the direction of the coordinate axis of the robot coordinate system, and the first position data of the first frame is the first position data obtained after the user acts according to the preset requirement, and the preset requirement is determined based on the creation rule of the robot coordinate system;

[0183] Based on the position of the first position data of the first frame in the target coordinate system, obtain the second position data corresponding to the first position data of the first frame, and the coordinate transformation relationship between the target coordinate system and the VR coordinate system;

[0184] If the first position data is the first position data of the i-th frame, i≥2, coordinate transformation is performed on the first position data of the i-th frame based on the coordinate transformation relationship to obtain second position data corresponding to the first position data of the i-th frame.

[0185] In a possible implementation, the first position data includes position data of a user's hand, the hand position data includes position data of multiple key points of a palm and position data of key points of each finger, and the hand position data in the first position data of the first frame is position data obtained after the user places his hand according to the preset requirement;

[0186] The coordinate conversion module 520 can be specifically used for:

[0187] Searching for first target data and second target data from the first position data of the first frame, wherein the first target data is position data of at least three key points of a palm, and the second target data is position data of multiple key points on a target finger, and the target finger is a finger determined according to the preset requirement and the creation rule;

[0188] Constructing an xy plane of the target coordinate system according to the first target data;

[0189] Determining the direction of the x-axis of the target coordinate system on the xy plane according to the direction of the second target data relative to the key point of the palm;

[0190] Finding a key point on the x-axis of the target coordinate system in the first position data of the first frame;

[0191] The target coordinate system is created by taking the target key point as the origin of the target coordinate system, wherein the target key point is a key point located on the x-axis of the target coordinate system.

[0192] In a possible implementation, the parameter determination module 530 may be specifically used to:

[0193] After acquiring the second position data corresponding to the first position data of the j-th frame, saving the second position data corresponding to the first position data of the j-th frame, j≥1;

[0194] Before obtaining the second position data corresponding to the first position data of the j+1th frame, determining the joint control parameters of the robot according to the second position data corresponding to the first position data of the jmth frame according to the control frequency, 0≤m<j;

[0195] After acquiring the second position data corresponding to the first position data of the j+1th frame, saving the second position data corresponding to the first position data of the j+1th frame;

[0196] The joint control parameters of the robot are determined according to the second position data corresponding to the first position data of the j-m+1th frame according to the control frequency.

[0197] In a possible implementation, the parameter determination module 530 may be specifically used to:

[0198] Based on the second position data, determining deviation data between the current state of the robot and the second position data;

[0199] Based on the deviation data, joint control parameters of the robot are determined.

[0200] In a possible implementation, the device 500 further includes:

[0201] An image acquisition module, used to acquire image data of a user's body collected by a VR device;

[0202] An image recognition module, used to extract the body movements of the user from the image data;

[0203] a mode determination module, configured to determine an operation mode corresponding to the body movement if the body movement matches a preset movement;

[0204] A mode switching module is used to switch the robot to the operation mode.

[0205] In a possible implementation, the body movement is a gesture movement;

[0206] The mode determination module can be specifically used for:

[0207] If the gesture action matches the preset action, the operation mode corresponding to the gesture action is determined.

[0208] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0209] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0210] The present application also provides a terminal device, see Figure 6 The terminal device 600 may include: at least one processor 610, a memory 620, and a computer program stored in the memory 620 and executable on the at least one processor 610. When the processor 610 executes the computer program, the steps in any of the above-mentioned method embodiments are implemented, for example Figure 1 Steps S101 to S104 in the illustrated embodiment. Alternatively, when the processor 610 executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented, for example Figure 5 The functions of the data acquisition module 510 to the control module 540 are shown.

[0211] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 620, and executed by the processor 610 to complete the present application. The one or more modules / units may be a series of computer program segments capable of completing specific functions, and the program segments are used to describe the execution process of the computer program in the terminal device 600.

[0212] Those skilled in the art will understand that Figure 6 It is only an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as input and output devices, network access devices, buses, etc.

[0213] The processor 610 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0214] The memory 620 may be an internal storage unit of the terminal device, or an external storage device of the terminal device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. The memory 620 is used to store the computer program and other programs and data required by the terminal device. The memory 620 may also be used to temporarily store data that has been output or is to be output.

[0215] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0216] The robot remote operation method provided in the embodiment of the present application can be applied to terminal devices such as computers, tablet computers, laptops, netbooks, personal digital assistants (PDAs), etc. The embodiment of the present application does not impose any restrictions on the specific type of terminal devices.

[0217] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0218] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0219] In the embodiments provided in the present application, it should be understood that the disclosed terminal equipment, devices and methods can be implemented in other ways. For example, the terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0220] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0221] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0222] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by one or more processors, the steps of each of the above-mentioned method embodiments can be implemented.

[0223] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by one or more processors, the steps of each of the above-mentioned method embodiments can be implemented.

[0224] Similarly, as a computer program product, when the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0225] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0226] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A remote operation method of a robot, characterized in that: include: Acquire first position data of a key point on the user's body collected by a VR device, wherein the first position data is position data of the key point in a VR coordinate system in the VR device; Performing coordinate transformation on the first position data to obtain second position data corresponding to the first position data, wherein the second position data is position data of the key point in a robot coordinate system, and the robot coordinate system is a coordinate system in which the robot to be controlled is located; Based on the second position data, determining joint control parameters when the robot reaches the current state of the user; The robot is controlled to move based on the joint control parameters.

2. The remote operation method of a robot according to claim 1, characterized in that: The performing coordinate conversion on the first position data to obtain second position data corresponding to the first position data includes: If the first position data is the first position data of the first frame, a target coordinate system is created according to the first position data of the first frame, the direction of the coordinate axis of the target coordinate system is consistent with the direction of the coordinate axis of the robot coordinate system, and the first position data of the first frame is the first position data obtained after the user acts according to the preset requirement, and the preset requirement is determined based on the creation rule of the robot coordinate system; Based on the position of the first position data of the first frame in the target coordinate system, obtain the second position data corresponding to the first position data of the first frame, and the coordinate transformation relationship between the target coordinate system and the VR coordinate system; If the first position data is the first position data of the i-th frame, i≥2, coordinate transformation is performed on the first position data of the i-th frame based on the coordinate transformation relationship to obtain second position data corresponding to the first position data of the i-th frame.

3. The remote operation method of a robot according to claim 2, characterized in that: The first position data includes position data of the user's hand, and the hand position data includes position data of multiple key points of the palm and position data of key points of each finger. The hand position data in the first position data of the first frame is the position data obtained after the user places the hand according to the preset requirements; The step of creating a target coordinate system according to the first position data of the first frame includes: Searching for first target data and second target data from the first position data of the first frame, wherein the first target data is position data of at least three key points of a palm, and the second target data is position data of multiple key points on a target finger, and the target finger is a finger determined according to the preset requirement and the creation rule; Constructing an xy plane of the target coordinate system according to the first target data; Determining the direction of the x-axis of the target coordinate system on the xy plane according to the second target data; Finding a key point on the x-axis of the target coordinate system in the first position data of the first frame; The target coordinate system is created by taking the target key point as the origin of the target coordinate system, wherein the target key point is a key point located on the x-axis of the target coordinate system.

4. The remote operation method of a robot according to claim 1, characterized in that: The determining, based on the second position data, the joint control parameters when the robot reaches the current state of the user comprises: After acquiring the second position data corresponding to the first position data of the j-th frame, saving the second position data corresponding to the first position data of the j-th frame, j≥1; Before obtaining the second position data corresponding to the first position data of the j+1th frame, determining the joint control parameters of the robot according to the second position data corresponding to the first position data of the jmth frame according to the control frequency, 0≤m<j; After acquiring the second position data corresponding to the first position data of the j+1th frame, saving the second position data corresponding to the first position data of the j+1th frame; The joint control parameters of the robot are determined according to the second position data corresponding to the first position data of the j-m+1th frame according to the control frequency.

5. The robot teleoperation method according to claim 1, characterized in that: The determining, based on the second position data, the joint control parameters when the robot reaches the current state of the user comprises: Based on the second position data, determining deviation data between the current state of the robot and the second position data; Based on the deviation data, joint control parameters of the robot when reaching the current state of the user are determined.

6. The remote operation method of a robot according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire image data of a user's body collected through a VR device; extracting body movements of the user from the image data; If the body movement matches the preset movement, determining the operation mode corresponding to the body movement; The robot is switched to the operating mode.

7. The robot teleoperation method according to claim 6, characterized in that: The body movement is a gesture; If the body motion matches the preset motion, determining the operation mode corresponding to the body motion includes: If the gesture action matches the preset action, the operation mode corresponding to the gesture action is determined.

8. A remote control device for a robot, characterized in that: include: A data acquisition module, used to acquire first position data of a key point on the user's body collected by a VR device, wherein the first position data is position data of the key point in a VR coordinate system in the VR device; a coordinate conversion module, configured to perform coordinate conversion on the first position data to obtain second position data corresponding to the first position data, wherein the second position data is position data of the key point in a robot coordinate system, and the robot coordinate system is a coordinate system in which the robot to be controlled is located; a parameter determination module, configured to determine, based on the second position data, joint control parameters when the robot reaches the current state of the user; A control module is used to control the robot to move based on the joint control parameters.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the remote operation method of the robot according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the remote operation method of the robot according to any one of claims 1 to 7 is implemented.

Images