Operation data acquisition method and device of robot, electronic equipment and storage medium

Abstract

The invention provides a robot operation data acquisition method and device, electronic equipment and a storage medium, and belongs to the technical field of robot control. The method comprises the steps of collecting a first pose track of a robot base for executing a target task in an operation space through a first camera module, and collecting a second pose track of an operator for executing the target task in the operation space through a first tracker; and determining first operation data of the robot base corresponding to the target task according to the first pose track and the second pose track. Therefore, according to the scheme, data acquisition can be carried out on the motion trail of the robot base, and the comprehensiveness of data acquisition is improved. Data acquisition is carried out through the data acquisition device, and the data acquisition device can be carried to various scenes to collect data, so that the data diversity can be expanded.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of robot control, and particularly relates to a robot operation data collection method and device, electronic equipment and storage medium. BACKGROUND

[0002] At present, the operation data collection method of mobile robots such as humanoid robots, quadruped plus mechanical arms, wheeled plus upper limbs of humanoids, etc. mainly adopts the traditional method of recognition, planning and control. The operation process is manually designed, and a real mechanical arm is controlled to complete various operation tasks, and data in the operation process is recorded.

[0003] The remote operation mode is also widely adopted. In this method, the collection of robot operation data depends on human remote operation teaching. In the teaching process, multi-dimensional data such as gripper action, mechanical arm joint state change and operation process video are collected. SUMMARY

[0004] The present disclosure provides a robot operation data collection method, device, electronic equipment and computer readable storage medium to at least solve the problem that the motion trajectory data of a robot base cannot be obtained in the related art. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a robot operation data collection method is provided, comprising: collecting, by a first camera module, a first pose trajectory of a robot base performing a target task in an operation space, and collecting, by a first tracker, a second pose trajectory of an operator performing the target task in the operation space; and determining first operation data of the robot base corresponding to the target task according to the first pose trajectory and the second pose trajectory.

[0006] According to a second aspect of an embodiment of the present disclosure, a robot operation data collection device is provided, comprising: a first collection module configured to collect, by a first camera module, a first pose trajectory of a robot base performing a target task in an operation space; a second collection module configured to collect, by a first tracker, a second pose trajectory of an operator performing the target task in the operation space; and a determination module configured to determine first operation data of the robot base corresponding to the target task according to the first pose trajectory and the second pose trajectory.

[0007] According to a third aspect of an embodiment of the present disclosure, an electronic equipment is provided, comprising a processor, a memory for storing processor-executable instructions, wherein the processor is configured to implement the steps of the method according to the first aspect of the present disclosure.

[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, and the computer readable storage medium stores computer program instructions. The computer program instructions are executed by a processor to implement the steps of the method in the first aspect of the embodiments of the present disclosure.

[0009] According to a fifth aspect of the embodiments of the present disclosure, an operation data acquisition system of a robot is provided, comprising: a base tracking tool and a processor, the base tracking tool comprising a first camera module and a first tracker; the first camera module is configured to acquire a first pose trajectory of a robot base performing a target task in an operation space; the first tracker is configured to acquire a second pose trajectory of an operator performing the target task in the operation space; and the processor is configured to determine first operation data of the robot base corresponding to the target task according to the first pose trajectory and the second pose trajectory.

[0010] The embodiments of the present disclosure provide at least the following beneficial effects: the first camera module of the base tracking tool acquires a first pose trajectory of a robot base performing a target task in an operation space, and the first tracker acquires a second pose trajectory of an operator performing the target task in the operation space, so that the first operation data of the robot base can be determined according to the first pose trajectory and the second pose trajectory. Thus, the present solution can realize data acquisition of the motion trajectory of the robot base, and improve the comprehensiveness of data acquisition. The data acquisition by the data acquisition device is convenient to carry to various scenes to collect data, so as to expand the diversity of data.

[0011] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure and do not limit the present disclosure.

[0013] Figure 1 is a schematic diagram of a base tracking tool according to an exemplary embodiment;

[0014] Figure 2 is a flowchart of an operation data acquisition method of a robot according to an exemplary embodiment;

[0015] Figure 3 is a schematic diagram of a coordinate system definition according to an exemplary embodiment;

[0016] Figure 4 is a schematic diagram of a portable display device according to an exemplary embodiment;

[0017] Figure 5 is a flow chart of a robot operation data collection method according to another example embodiment;

[0018] Figure 6 is a schematic diagram of a gripping tool according to an example embodiment;

[0019] Figure 7 is a schematic diagram of a gripping tool according to an example embodiment;

[0020] Figure 8 is a flow chart of a robot operation data collection method according to another example embodiment;

[0021] Figure 9 is a flow chart of a robot operation data collection method according to another example embodiment;

[0022] Figure 10 is a flow chart of a determination process of a first transformation matrix in a robot operation data collection method according to an example embodiment;

[0023] Figure 11 is a flow chart of a robot operation data collection according to an example embodiment;

[0024] Figure 12 is a block diagram of a robot operation data collection system according to an example embodiment;

[0025] Figure 13 is a block diagram of a robot operation data collection apparatus according to an example embodiment;

[0026] Figure 14 is a block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0027] In order to make the ordinary person skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following example embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0029] The acquisition, storage, use, processing, etc. of data in the technical solutions of the present disclosure comply with the relevant legal regulations.

[0030] The operation data acquisition method and device of the robot are described below with reference to the accompanying drawings.

[0031] The operation data acquisition method of the robot of the present disclosure can be applied to various robot mobile operation application scenarios such as wheeled robots, humanoid robots, quadruped robots, and robots with mechanical arms.

[0032] In the present disclosure, the operation data of the robot can be acquired by a data acquisition device. Optionally, the data acquisition device includes a base tracking tool and a holding tool. The base tracking tool and the holding tool are worn by an operator to acquire data of the operator performing a target task.

[0033] Optionally, the data acquired by the base tracking tool is used as first operation data of a robot base, and the data acquired by the holding tool is used as second operation data of a robot arm joint. The robot arm is installed on the robot base, and movement of the robot base can drive the robot arm to move.

[0034] The base tracking tool of the present disclosure includes a first tracker and a first camera module.

[0035] Figure 1 The base tracking tool is shown in the schematic diagram. Figure 1 The base tracking tool is composed of a plurality of fixed structures and a plurality of sensors. The fixed structures include a sensor base plate, a strap perforation, and a camera angle adjustment hinge. The sensors include a first tracker and a first camera module.

[0036] Optionally, the sensor base plate is used to fix the sensors such as the first tracker and the first camera module.

[0037] Optionally, the strap perforation is fixed on the sensor base plate, and a strap can be used to pass through the strap perforation to fix the base tracking tool at different positions.

[0038] Optionally, the camera angle adjustment hinge is used to adjust the included angle between the first camera module and the ground.

[0039] Optionally, the first tracker is an optical positioning device fixed on the sensor base plate. It can be used to acquire the movement process of the operator. The optical positioning device also includes a plurality of base stations fixed in the operation space.

[0040] Optionally, the first camera module can be a wide-angle color camera module, including a color wide-angle camera module and a binocular infrared camera module, which are fixed on the camera angle adjustment hinge. It can be used to collect environmental information, such as collecting color images and depth image information of the environment.

[0041] It should be noted that the base tracking tool in the embodiment of the present disclosure can be made by 3D printing technology, which has simple structure and small volume, and the 3D printing production cost is low. The operation data of the robot can be collected by using low cost and simple structure.

[0042] Figure 2 is a flow chart of a robot operation data collection method according to an exemplary embodiment, as shown in Figure 2 The robot operation data collection method of the present disclosure comprises the following steps:

[0043] S201, collecting a first pose trajectory of a robot base in an operation space performing a target task by a first camera module, and collecting a second pose trajectory of an operator in the operation space performing the target task by a first tracker.

[0044] In some embodiments, the robot can include a movable robot base and a mechanical arm of the robot. The mechanical arm is installed on the robot base, and the movement of the robot base can drive the movement of the mechanical arm.

[0045] The operation space refers to the physical place where the robot performs the target task. Optionally, the space where the robot is located can be regarded as the operation space.

[0046] In some embodiments, the operator can collect the pose trajectory of the operator in the operation space performing the target task by the first camera module in the base tracking tool, as the first pose trajectory of the robot base. Optionally, the pose trajectory can be a set of positions and attitudes.

[0047] For example, the pose trajectory of the operator can be determined based on Figure 1 the strap hole and strap of the base tracking tool in the , the base tracking tool is fixed at the operator's torso, and the video data of the operator performing the target task is collected by the first camera module, and the pose trajectory of the operator is determined based on the video data as the first pose trajectory of the robot base.

[0048] In some embodiments, the pose information of the robot base in the world coordinate system can be obtained by coordinate transformation according to the video data collected by the first camera module, and the pose information can be recorded in time sequence, and the first pose trajectory can be obtained. That is, the first pose trajectory is in the world coordinate system. Optionally, the pose information of the robot base can be determined from the video data based on a computer vision algorithm.

[0049] In some embodiments, the positioning code pattern can be set in the operation space, and the coordinate system in which the positioning code pattern is located is taken as the world coordinate system. That is, the first camera module determines the first pose trajectory in the world coordinate system from the video data containing the positioning code pattern according to the transformation matrix between the camera coordinate system and the world coordinate system. The positioning code pattern can be a binary coded rectangular two-dimensional code.

[0050] As shown in the schematic diagram of the coordinate system definition, the camera coordinate system in which the first camera module is located is the base camera coordinate system, the coordinate origin of which is the center point of the first camera module, and the coordinate system direction is the pre-defined coordinate axis direction. The world coordinate system is the world coordinate system. Figure 3

[0051] In some embodiments, at least one base station can be pre-arranged in the operation space, the base station is used to emit optical signals outwardly, and the first tracker determines the pose trajectory by receiving the optical signals. It should be noted that the position of the base station should be kept unchanged during the operation data collection process.

[0052] That is, the first tracker determines the second pose trajectory by receiving the optical signals emitted by the base station when the operator performs the target task.

[0053] In some embodiments, the base coordinate system of the robot can be calibrated in advance, and the pose information of the human body of the operator in the base coordinate system can be calculated through coordinate system conversion, and the pose information can be recorded in time sequence, so as to obtain the second pose trajectory. That is, the second pose trajectory is in the base coordinate system, which is a pre-defined coordinate system used to represent the offset between the robot base and the first tracker, and the coordinate origin of which is the center point of the robot base.

[0054] It can be understood that when the first tracker collects data, the second pose trajectory is determined by receiving the optical signals of the base station, and the coordinate systems involved in the determination process include the coordinate system in which the base station is located, the coordinate system in which the first tracker is located, and the base coordinate system. The second pose trajectory in the base coordinate system can be obtained through coordinate system conversion according to the transformation matrix between the above-mentioned multiple coordinate systems.

[0055] S202, determining the first operation data of the robot base corresponding to the target task according to the first pose trajectory and the second pose trajectory.

[0056] ​It should be noted that the second pose trajectory collected by the first tracker is a pose trajectory of a human body pose corresponding to the operator, and the human body pose will affect the pose of the robot base. For example, the human body pose can be understood as a pose change amount of the robot base, and in subsequent control of the robot base, the target pose state of the robot base can be finally determined based on the current pose of the robot base and in combination with the pose change amount.

[0057] In some embodiments, the first pose trajectory and the second pose trajectory can be associated to obtain a target pose trajectory, and the target pose trajectory is taken as first operation data of the robot base corresponding to the target task, so that the first operation data contains the pose of the robot base and the pose change amount of the robot base, thereby improving the accuracy of the robot in executing the target task according to the first operation data.

[0058] In some embodiments, since the first pose trajectory and the second pose trajectory are in different coordinate systems, the first pose trajectory and the second pose trajectory can be converted to the same coordinate system, and the first pose trajectory and the second pose trajectory in the same coordinate system are associated to obtain the first operation data.

[0059] Optionally, the first pose trajectory and the second pose trajectory in the same coordinate system can be associated based on the timestamps of the pose trajectories.

[0060] It can be understood that the first pose trajectory is in the world coordinate system, and the second pose trajectory is in the base coordinate system. By obtaining the transformation matrix between the base coordinate system and the world coordinate system, the second pose trajectory can be converted to the world coordinate system.

[0061] In some embodiments, there are multiple conversion relationships between the base coordinate system and the world coordinate system, and their corresponding transformation matrices. For example, taking the world coordinate system as a first spatial coordinate system, the base coordinate system as a second spatial coordinate system, and the coordinate system in which the first tracker is located as a first local coordinate system, the second pose trajectory can be converted from the base coordinate system to the first local coordinate system, further converted from the first local coordinate system to the second spatial coordinate system, and further converted from the second spatial coordinate system to the first spatial coordinate system.

[0062] For example, matrix A is set as a transformation matrix for converting from the base coordinate system to the first local coordinate system, matrix B is a transformation matrix for converting from the first local coordinate system to the second spatial coordinate system, and matrix C is a transformation matrix for converting from the second spatial coordinate system to the first spatial coordinate system.

[0063] The second pose trajectory can be converted from the base coordinate system to the first space coordinate system according to the matrix A, the matrix B and the matrix C respectively; and the matrix D between the base coordinate system and the first space coordinate system can be calculated according to the matrix A, the matrix B and the matrix C, so that the second pose trajectory is converted from the base coordinate system to the first space coordinate system according to the matrix D.

[0064] In some embodiments, in order to improve the accuracy of the operator collecting the first operation data, the operator can be prompted during the collection process based on the portable display device. As shown in Figure 4 The portable display device is a head-mounted device, and the prompt information can be projected on the lens to prompt the operator. Figure 4

[0065] In some embodiments, after the first operation data is determined, the first operation data can be stored in an external storage device. Optionally, the external storage device can be connected to the base tracking tool through a cable, so that the first operation data collected by the base tracking tool can be received.

[0066] The robot operation data collection method provided by the embodiments of the present disclosure can collect the first pose trajectory of the robot base in the operation space performing the target task through the first camera module of the base tracking tool, and collect the second pose trajectory of the operator in the operation space performing the target task through the first tracker, so that the first operation data of the robot base can be determined according to the first pose trajectory and the second pose trajectory. Therefore, the present scheme can realize data collection of the motion trajectory of the robot base, and improve the comprehensiveness of data collection. The data is collected by the data acquisition device, which is convenient to carry to various scenes to collect data, so as to expand the diversity of data.

[0067] Figure 5 is a flowchart of a robot operation data collection method according to an exemplary embodiment, as shown in Figure 5 The robot operation data collection method provided by the embodiments of the present disclosure includes the following steps:

[0068] S501, collecting the first pose trajectory of the robot base in the operation space performing the target task through the first camera module.

[0069] S502, collecting the second pose trajectory of the operator in the operation space performing the target task through the first tracker.

[0070] For related content of steps S501-S502, refer to the above embodiments, which will not be repeated here.

[0071] S503, performing coordinate system conversion on the second pose trajectory to determine the third pose trajectory in the first space coordinate system.​

[0072] In some embodiments, the second pose trajectory is in the base coordinate system, and the third pose trajectory in the first space coordinate system can be determined by performing coordinate system conversion on the second pose trajectory based on a transformation matrix between the base coordinate system and the first space coordinate system. The first space coordinate system is a coordinate system defined for representing real-world positions.

[0073] In some embodiments, the first space coordinate system can be determined based on a first space coordinate system definition file. Figure 3 Some coordinate system definitions related to the embodiments of the present disclosure are explained as follows:

[0074] 1. First space coordinate system

[0075] Refers to the world coordinate system, Figure 3 The world coordinate system can be determined by setting a positioning code map in the operation space and taking the coordinate system of the positioning code map as the first space coordinate system.

[0076] 2. Second space coordinate system

[0077] Refers to the coordinate system of the base station, Figure 3 The tracker_world coordinate system, the origin of the second space coordinate system can be the center point of the base station, and the coordinate system direction is the pre-defined coordinate axis direction.

[0078] 3. First local coordinate system

[0079] Refers to the coordinate system of the first tracker, Figure 3 The base_tracker coordinate system, the origin of the first local coordinate system can be the center point of the first tracker, and the coordinate system direction is the pre-defined coordinate axis direction.

[0080] In some embodiments, the transformation matrix between the base coordinate system and the first space coordinate system can be determined by pre-determining the transformation matrices between different coordinate systems and performing matrix operations, so that the coordinate system conversion of the second pose trajectory can be performed based on the transformation matrix to obtain the third pose trajectory in the first space coordinate system.

[0081] In some embodiments, the fourth transformation matrix between the base coordinate system and the first space coordinate system can be determined based on the first transformation matrix between the first space coordinate system and the second space coordinate system The second transformation matrix between the first local coordinate system of the first tracker and the second space coordinate system And the third transformation matrix between the first local coordinate system and the base coordinate system

[0082] ​Optionally, the formula for determining the fourth transformation matrix is as follows:

[0083]

[0084] Further, the fourth transformation matrix can be determined according to the second transformation matrix and the third transformation matrix. The second pose trajectory is mapped to the first spatial coordinate system to obtain a third pose trajectory. That is, each point in the second pose trajectory can be directly mapped to the corresponding position in the first spatial coordinate system through the fourth transformation matrix, thereby obtaining the third pose trajectory.

[0085] S504, determining the first operation data according to the first pose trajectory and the third pose trajectory.

[0086] In some embodiments, the first pose trajectory and the third pose trajectory can be associated according to the timestamps of the first pose trajectory and the third pose trajectory, thereby obtaining the first operation data. It can be understood that the first camera module and the first tracker synchronously collect the pose trajectory, and then the poses corresponding to the same timestamps in the first pose trajectory and the third pose trajectory can be directly associated.

[0087] For example, the timestamps of the first pose trajectory and the third pose trajectory are timestamp 1, timestamp 2 and timestamp 3, the positions in the first pose trajectory are pose A, pose B and pose C, and the positions in the third pose trajectory are pose 1, pose 2 and pose 3, then pose A is associated with pose 1 according to timestamp 1, pose B is associated with pose 2 according to timestamp 2, and pose C is associated with pose 3 according to timestamp 3, thereby obtaining the first operation data.

[0088] The operation data acquisition method of the robot provided by the embodiments of the present disclosure can obtain the third pose trajectory in the first spatial coordinate system by performing coordinate system conversion on the second pose trajectory, and then determine the first operation data according to the first pose trajectory and the third pose trajectory. Therefore, the present scheme can realize unified acquisition of the pose trajectory, thereby eliminating the coordinate system difference, improving the consistency of the data, realizing fusion of various sensor data, and further improving the reliability of the data.

[0089] In some embodiments, the data acquisition device further comprises a clamping tool, and the clamping tool comprises a second tracker, such as Figure 6 and Figure 7 The schematic diagram of the clamping tool is shown, and the clamping tool is composed of a grabbing structure and a sensor. As shown in Figure 6 The clamping tool comprises a second tracker, a second camera module, a radio receiver, a transmission device, a sliding block, a positioning code slot, and a clamping jaw. As shown in Figure 7 The clamping tool further comprises a six-dimensional force sensor, an operation instruction button, an operation handle and a film pressure sensor.

[0090] Wherein, the operation handle, transmission device, slider and gripper belong to the grabbing structure of the clamping tool; the second camera module, second tracker, thin film pressure sensor, six-dimensional force sensor, sound collector and operation instruction button belong to the sensor of the clamping tool.

[0091] Wherein, the operation handle is connected with the input part of the transmission device, and when the operator needs to close the gripper to collect data, presses the operation handle to generate movement, and transmits the handle movement to the slider through the transmission device to form horizontal symmetrical movement from both sides to the center.

[0092] The transmission device: internally contains a series of transmission mechanisms. Input operation handle movement, output slider horizontal symmetrical movement from both sides to the center.

[0093] The slider: fixed on the horizontal guide rail inside the transmission mechanism, connected with the output part of the transmission device. When the slider moves, it drives the gripper tip to form a symmetrical clamping action. The slider is engraved with a fixed-length positioning code pattern slot for installing the positioning code pattern.

[0094] The gripper: made of flexible material, fixed on the slider. When clamping operation is performed, it can produce certain deformation. The inner surface of the gripper is pasted with anti-slip soft rubber to provide gripping friction.

[0095] The second camera module: includes a color wide-angle camera module and a binocular infrared camera module in the module. The module is fixed on the transmission device shell, and the camera optical axis is parallel to the gripper symmetry axis. During operation, the camera module is used to collect image information of the operated object and the surrounding environment. The binocular depth camera is used to generate a depth image.

[0096] The second tracker: an optical positioning device, including a tracker and a base station. The tracker is fixed on the transmission device shell. Multiple base stations are fixedly arranged in the working space. During operation, it is used to collect the motion state of the device with high precision.

[0097] The thin film pressure sensor: pasted on the inner surface of the gripper. When clamping operation is performed, the thin film pressure sensor deforms with the flexible gripper and converts into a digital signal which is recorded.

[0098] The six-dimensional force sensor: fixed between the handle and the transmission mechanism shell. When the gripper contacts with the outside world, the six-dimensional force sensor collects the force and torque of the gripper and records them.

[0099] The sound collector: fixed beside the second camera module, used to collect environmental sound and voice instructions during operation.

[0100] The operation instruction button: fixed on the operation handle, used to send start collecting and stop collecting signals to the external storage device.

[0101] It should be noted that the clamping tool in the embodiments of the present disclosure can be made by 3D printing technology, which has simple structure and small volume, and the 3D printing production cost is low. The operation data of the robot can be collected by using low cost and simple structure.

[0102] Figure 8 is a flow chart of a robot operation data collection method according to an exemplary embodiment, as Figure 8 shown, the robot operation data collection method of the present disclosure comprises the following steps:

[0103] S801, the fourth pose trajectory of the clamping tool performing the target task in the operation space is collected by the second tracker of the clamping tool.

[0104] In some embodiments, the second tracker can collect the pose of the clamping tool performing the target task in the operation space. Optionally, during the movement of the clamping tool, the optical signal emitted by the base station can be received, the coordinate system conversion can be performed based on the optical signal, and the fourth pose trajectory can be recorded in time sequence.

[0105] It can be understood that the clamping tool performing the target task in the operation space uses the clamping jaw in the clamping tool, and the corresponding coordinate system is used to describe the coordinate system of the end posture of the robot during movement. The pose information collected by the second tracker is in the coordinate system of the second tracker, and then the pose information can be converted to the end coordinate system of the robot to obtain the fourth pose trajectory.

[0106] As Figure 3 shown, Figure 3 contains two clamping tools, namely left (L, Left) clamping tool and right (R, Right) clamping tool. Taking the L clamping tool as an example, the L_gripper_tracker coordinate system is the coordinate system of the second tracker, the coordinate origin is the center point of the second tracker, and the coordinate system direction is the pre-defined coordinate axis direction. The L_tip coordinate system is the end coordinate system of the robot, the coordinate origin is the center point of the two clamping jaws, and the coordinate system direction is the pre-defined coordinate axis direction.

[0107] That is, the pose information collected by the second tracker of the L clamping tool can be converted from the L_gripper_tracker coordinate system to the L_tip coordinate system, and recorded in time sequence to obtain the left pose trajectory. Similarly, the pose information collected by the second tracker of the R clamping tool can be converted from the R_gripper_tracker coordinate system to the R_tip coordinate system, and recorded in time sequence to obtain the right pose trajectory. By merging the left pose trajectory and the right pose trajectory, the fourth pose trajectory can be obtained.

[0108] S802, according to the fourth pose trajectory, driving the robot model to perform trajectory restoration in the simulation space of the operation space.

[0109] In some embodiments, the robot model is in the base coordinate system, by converting the fourth pose trajectory from the end coordinate system to the base coordinate system, and according to the fourth pose trajectory in the base coordinate system, driving the robot model to perform trajectory restoration in the simulation space of the operation space, realizing the restoration of the motion trajectory of the end coordinate system in the base coordinate system point by point.

[0110] In some embodiments, the fourth pose trajectory can be converted to the world coordinate system, that is, the first space coordinate system, and then converted from the first space coordinate system to the base coordinate system. The transformation matrix between the end coordinate system and the base coordinate system can also be determined to convert the fourth pose trajectory to the base coordinate system.

[0111] In some embodiments, by inputting the fourth pose trajectory in the base coordinate system into the simulation space, the robot model can move the end of the robot arm according to the pose trajectory to drive the joint movement of the robot arm, and restore the motion trajectory of the end of the robot arm in the base coordinate system point by point.

[0112] S803, collecting the position and motion data of the robot arm joint when the robot model restores the trajectory point as the second operation data of the robot arm joint corresponding to the target task.

[0113] In some embodiments, by collecting the position of the robot arm joint corresponding to each trajectory point and the motion data of the joint as the second operation data of the robot arm joint corresponding to the target task. The motion data includes the motion speed of the joint.

[0114] Optionally, the position and motion data of the robot arm joint corresponding to the trajectory point can be collected according to the timestamp of the trajectory point, and the position and motion data of the robot arm joint are sorted according to the timestamp to obtain the second operation data of the robot arm joint corresponding to the target task.

[0115] It should be noted that the first operation data and the second operation data are collected synchronously, and after the collection of the first operation data and the second operation data is completed, they can be stored in the external storage device. Optionally, the external storage device can be connected with the base tracking tool and the clamping tool through a cable, so as to receive the collected first operation data and second operation data.

[0116] In some embodiments, the collection of data can be started by controlling the operation instruction button of the gripping tool, the operator performs a target task in the operation space, the base can synchronously collect the pose trajectory of the operator according to the tool and the gripping tool, and the pose trajectory is transmitted to an external storage device, and then the collected pose trajectory is calculated by the external storage device to obtain first operation data and second operation data.

[0117] Optionally, after the end of the execution of the target task, the collection of data can be ended by controlling the operation instruction button of the gripping tool again.

[0118] The operation data collection method of the robot provided by the embodiments of the present disclosure collects the fourth pose trajectory of the gripping tool performing the target task in the operation space through the second tracker of the gripping tool, and simulates according to the fourth pose trajectory to obtain the second operation data of the joints of the mechanical arm corresponding to the target task. Thus, the present scheme can realize the collection of the joint motion data of the mechanical arm, and the mixed collection in the simulation environment and the real environment can improve the reliability of the data. The data collected by the second camera module of the gripping tool is more real and accurate, and the real mechanical arm operation is not needed to obtain the data, thereby improving the convenience of data collection.

[0119] Figure 9 is a flowchart of an operation data collection method of a robot according to an exemplary embodiment, as shown in Figure 9 The operation data collection method of the robot of the embodiments of the present disclosure includes the following steps:

[0120] S901, collecting, by a second tracker of a gripping tool, a fourth pose trajectory of the gripping tool performing a target task in an operation space.

[0121] For details of step S901, reference can be made to the above embodiments, which will not be repeated here.

[0122] S902, performing first coordinate conversion on the fourth pose trajectory to determine a fifth pose trajectory of the gripping tool in a first space coordinate system.

[0123] In some embodiments, the first coordinate conversion can be combined with Figure 3 The partial coordinate system definitions related to the embodiments of the present disclosure are explained as follows:

[0124] 1. Second local coordinate system

[0125] Refers to the coordinate system of the second tracker, Figure 3 The origin of the second local coordinate system can be the center point of the second tracker, and the coordinate system direction is the pre-defined coordinate axis direction.

[0126] 2. end-of-arm coordinate system

[0127] refers to a coordinate system describing the pose of the end of the robot arm, Figure 3 comprises the R tip coordinate system and the L tip coordinate system, and the coordinate origin of the R tip coordinate system or the L tip coordinate system is the center point of the two gripper ends, and the coordinate system direction is a pre-defined coordinate axis direction.

[0128] In some embodiments, by pre-determining the transformation matrix between different coordinate systems, and by matrix operation, the transformation matrix between the end-of-arm coordinate system and the first spatial coordinate system can be obtained, so that the coordinate system conversion of the fourth pose trajectory can be performed based on the transformation matrix to obtain the fifth pose trajectory of the gripping tool in the first spatial coordinate system.

[0129] In some embodiments, the first transformation matrix between the first spatial coordinate system and the second spatial coordinate system can be determined based on the first local coordinate system of the first tracker and the second local coordinate system of the second tracker. the fifth transformation matrix between the second local coordinate system of the second tracker and the second spatial coordinate system and the sixth transformation matrix between the second local coordinate system and the end-of-arm coordinate system determining the seventh transformation matrix between the end-of-arm coordinate system and the first spatial coordinate

[0130] Optionally, the formula for determining the seventh transformation matrix between the end-of-arm coordinate system and the first spatial coordinate is as follows:

[0131]

[0132] Further, the seventh transformation matrix can be determined according to the first transformation matrix between the first spatial coordinate system and the base coordinate system of the robot, the fifth transformation matrix between the second local coordinate system of the second tracker and the second spatial coordinate system, and the sixth transformation matrix between the second local coordinate system and the end-of-arm coordinate system. mapping the fourth pose trajectory to the first spatial coordinate system to obtain the fifth pose trajectory. That is, each point in the fourth pose trajectory can be directly mapped to the corresponding position in the first spatial coordinate system through the seventh transformation matrix, thereby obtaining the fifth pose trajectory.

[0133] S903, performing second coordinate conversion on the fifth pose trajectory to determine the sixth pose trajectory of the gripping tool in the base coordinate system of the robot.

[0134] In some embodiments, by performing second coordinate conversion on the fifth pose trajectory, the fifth pose trajectory can be converted from the first spatial coordinate system to the base coordinate system, that is, the second coordinate conversion can be performed on the fifth pose trajectory according to the transformation matrix between the first spatial coordinate system and the base coordinate system.

[0135] In some embodiments, a ninth transformation matrix between the end coordinate system and the base coordinate system can also be determined according to the seventh transformation matrix and an eighth transformation matrix between the base coordinate system and the first space coordinate system, and the fifth pose trajectory can be converted into a sixth pose trajectory of the clamping tool in the base coordinate system of the robot according to the ninth transformation matrix.

[0136] In some embodiments, the seventh transformation matrix can be determined based on the first pose trajectory and the second pose trajectory. and the eighth transformation matrix between the base coordinate system and the first space coordinate system determine the ninth transformation matrix between the end coordinate system and the base coordinate system

[0137] Optionally, the formula for determining the ninth transformation matrix between the end coordinate system and the base coordinate system is as follows:

[0138]

[0139] Further, the fifth pose trajectory can be mapped to the base coordinate system to obtain the sixth pose trajectory according to the ninth transformation matrix . That is, each point in the fifth pose trajectory can be directly mapped to the corresponding position in the base coordinate system through the ninth transformation matrix, thereby obtaining the sixth pose trajectory.

[0140] S904, according to the sixth pose trajectory, driving the robot model to perform trajectory restoration in the simulation space of the operation space.

[0141] In some embodiments, by inputting the sixth pose trajectory in the base coordinate system into the simulation space, the robot model can move the end of the robot arm according to the sixth pose trajectory to drive the movement of the joints of the robot arm, and restore the motion trajectory of the end of the robot arm in the base coordinate system point by point.

[0142] S905, collect the position and motion data of the joints of the robot arm when the robot model restores the trajectory points as the second operation data of the joints of the robot arm corresponding to the target task.

[0143] For details of step S905, please refer to the above embodiments, which will not be repeated here.

[0144] The operation data collection method of the robot provided by the embodiment of the present disclosure can obtain the sixth pose trajectory of the clamping tool in the base coordinate system of the robot through twice coordinate conversion of the fourth pose trajectory, so as to drive the robot model to perform trajectory restoration in the simulation space of the operation space according to the sixth pose trajectory. Therefore, the present scheme can realize the conversion of the pose trajectory from the end coordinate system of the robot arm to the base coordinate system, and drive the robot simulation using the pose trajectory in the base coordinate system, so that the position and motion data of the robot arm joint in the global space can be determined, thereby improving the convenience and accuracy of data collection.

[0145] On the basis of the above-mentioned embodiments, the determination process of the first transformation matrix in the embodiment of the present disclosure can be explained and described as shown in the following table: Figure 10 The determination process of the first transformation matrix in the embodiment of the present disclosure includes the following steps:

[0146] S1001, determining the tenth transformation matrix between the second local coordinate system and the second space coordinate system according to the seventh pose trajectory of the operator pre-acquired by the second tracker

[0147] It can be understood that the base station and the position encoding map can be arranged in the operation space before the operation data of the robot is collected, and the base tracking tool can be fixed at the operator's torso, and the operator can hold the clamping tool to collect data.

[0148] In some embodiments, the seventh pose trajectory of the operator moving in the operation space can be pre-acquired by the second tracker, and the transformation matrix between the second local coordinate system and the second space coordinate system can be calculated as the tenth transformation matrix according to the seventh pose trajectory.

[0149] In some embodiments, the second local coordinate system relative to the second space coordinate system in the seventh pose trajectory can be determined according to the seventh pose trajectory, and the position vector and the rotation matrix of the second local coordinate system relative to the second space coordinate system can be determined, so as to combine the rotation matrix and the position vector to obtain the tenth transformation matrix.

[0150] S1002, determining the eleventh transformation matrix between the second local coordinate system and the camera coordinate system according to the three-dimensional models of the second camera module in the second tracker and the clamping tool

[0151] It should be noted that a coordinate system can be defined for the corresponding three-dimensional model of the clamping tool when the clamping tool is manufactured, for describing the geometric features and dimensions of the clamping tool. By obtaining the three-dimensional model of the second tracker and the three-dimensional model of the second camera module in the clamping tool, the calculation of the transformation matrix can be performed based on the three-dimensional model, so as to determine the eleventh transformation matrix between the second local coordinate system and the camera coordinate system.

[0152] In some embodiments, the second tracker and the second camera module can be modeled using three-dimensional modeling software to obtain the three-dimensional model. The three-dimensional model of the second tracker and the second camera module can also be generated using artificial intelligence to generate the three-dimensional model of the second tracker and the second camera module. The manner of obtaining the three-dimensional model is not limited in the present disclosure.

[0153] Alternatively, the CAD model of the second tracker and the second camera module can be obtained as the three-dimensional model, so as to determine the relative position between the origins of the coordinate systems in which the second tracker and the second camera module are located, and determine the relative direction between the two coordinate systems, so that the eleventh transformation matrix can be calculated according to the relative position and the relative direction.

[0154] S1003, determining a twelfth transformation matrix between the camera coordinate system and the first spatial coordinate system

[0155] In some embodiments, by video capturing the positioning encoder in the operation space, the transformation matrix between the camera coordinate system and the world coordinate system can be determined according to the captured video data. Alternatively, the operation instruction button on the clamping tool can be operated to start recording, and the video data containing the positioning code image can be obtained by moving in the operation space.

[0156] That is, the coordinate system of the positioning code image can be defined as the world coordinate system, i.e., the first spatial coordinate system, so as to calculate the transformation matrix between the camera coordinate system and the first spatial coordinate system as the twelfth transformation matrix according to the captured video data by Augmented Reality University of Cordoba (Aruco).

[0157] S1004, determining a first transformation matrix according to the tenth transformation matrix, the eleventh transformation matrix and the twelfth transformation matrix

[0158] In some embodiments, by performing matrix operation on the tenth transformation matrix, the eleventh transformation matrix and the twelfth transformation matrix, the first transformation matrix between the first spatial coordinate system and the second spatial coordinate system can be calculated.

[0159] Optionally, the formula for calculating the first transformation matrix is as shown below:

[0160]

[0161] In some embodiments, in order to improve the accuracy of the first transformation matrix and reduce the error of the coordinate system conversion, a plurality of seventh pose trajectories can be collected, and a plurality of candidate first transformation matrices corresponding to the plurality of seventh pose trajectories respectively can be determined, so that the candidate first transformation matrix with the highest accuracy can be determined from the plurality of candidate first transformation matrices as the first transformation matrix between the first spatial coordinate system and the second spatial coordinate system.

[0162] Optionally, the plurality of candidate first transformation matrices can be statistically analyzed to determine the final first transformation matrix from the plurality of candidate first transformation matrices based on the statistical analysis result. For example, the candidate first transformation matrices with a difference from the geometric median greater than a set value can be filtered out, and the mean matrix of the remaining candidate first transformation matrices can be calculated, and the difference between the candidate first transformation matrices and the mean matrix can be calculated to obtain the candidate first transformation matrix with the smallest difference as the final first transformation matrix.

[0163] The robot operation data collection method provided by the embodiments of the present disclosure can determine the first transformation matrix according to the seventh pose trajectory collected by the second tracker in advance, which can complete complex data calculation before data collection, and can be beneficial to the calculation of coordinate system conversion in the process of collecting the pose trajectory, and improve the calculation speed.

[0164] On the basis of the above-mentioned embodiments, the operation data further includes at least one of the following data: an eighth pose trajectory of the end of the robot arm collected by the second camera module in the clamping tool, a target opening and closing trajectory between the first clamping finger and the second clamping finger in the clamping tool, clamping force data of the clamping part in the clamping tool, stress data of the clamping part, audio data and image data in the process of executing the target task.

[0165] In some embodiments, the first transformation matrix can be determined by combining Figure 6 and Figure 7The clamping tool shown is used to explain the eighth pose trajectory. The operator can hold the clamping tool to perform a target task, from starting data collection to the end of the target task, which is a target task process. During the process, the second camera module can collect video stream data of the target task process, the film pressure sensor on the clamping part of the clamping tool can collect clamping force data in the clamping process, and the clamping force data represents the change of the clamping force in the clamping process. The six-dimensional force sensor on the clamping tool can detect the force data of the clamping part in the target task process, which can include force / torque data at the sensor position. The microphone on the clamping tool can collect audio data of the clamping task in real time, which can include user voice instructions and environmental sound data.

[0166] In some embodiments, the eighth pose trajectory of the end of the robot arm in the first spatial coordinate system can be determined according to the video stream data. Optionally, the pose trajectory in the camera coordinate system is determined based on the video stream data, and the pose trajectory is converted between coordinate systems to obtain the eighth pose trajectory in the first spatial coordinate system. By mapping the pose trajectory in the camera coordinate system to the coordinate system of the end of the robot arm, and then based on the ninth transformation matrix Mapping the pose trajectory to the first spatial coordinate system obtains the eighth pose trajectory.

[0167] In some embodiments, the clamping part of the clamping tool in the video stream data can be tracked and positioned to determine the thirteenth transformation matrix between the camera coordinate system and the first positioning encoding map The fourteenth transformation matrix between the camera coordinate system and the second positioning encoding map coordinate system Further, the seventh transformation matrix and the eighth transformation matrix can be used for homogeneous transformation to obtain the fifteenth transformation matrix between the first positioning encoding map coordinate system and the second positioning encoding map coordinate system Thus, the target opening and closing trajectory can be determined based on the fifteenth transformation matrix.

[0168] In some embodiments, the first spatial coordinate system is the coordinate system of the end of the robot arm, and the second spatial coordinate system is the coordinate system of the end of the robot arm. Figure 3With the L-shaped clamping tool as an example, the first and second position encoding diagrams on each frame of the video stream data can be extracted by using the landmark detection algorithm, and the motion trajectories of the first and second position encoding diagrams in the camera coordinate system are detected. According to the positional relationship between the camera coordinate system (L camera coordinate system) and the first position encoding diagram coordinate system (L_R_finger coordinate system), the thirteenth transformation matrix between the camera coordinate system and the first position encoding diagram coordinate system is obtained. Similarly, according to the positional relationship between the camera coordinate system (L camera coordinate system) and the second position encoding diagram coordinate system (L_L_finger coordinate system), the fourteenth transformation matrix between the camera coordinate system and the second position encoding diagram coordinate system is obtained.

[0169] Alternatively, the formula for determining the fifteenth transformation matrix between the first position encoding diagram coordinate system and the second position encoding diagram coordinate system is as follows:

[0170]

[0171] By substituting the thirteenth transformation matrix and the fourteenth transformation matrix into the aforementioned formula (5), the fifteenth transformation matrix between the first position encoding diagram coordinate system and the second position encoding diagram coordinate system can be calculated.

[0172] It can be understood that the fifteenth transformation matrix represents the relative distance change between the first and second clamping fingers in the clamping tool, so that the fifteenth transformation matrix can reflect the trajectory change of the opening and closing degree of the clamping part during the execution of the clamping task, that is, the target opening and closing trajectory is determined.

[0173] That is, based on the above embodiment, the operation data of the target task includes:

[0174] The first operation data represents the positional change of the robot base during the execution of the target task.

[0175] The second operation data represents the positional change and motion change of the robot joint during the execution of the target task.

[0176] The video stream data represents the image video during the execution of the target task.

[0177] The clamping force data represents the clamping force change of the clamping part during the execution of the target task.

[0178] The force data represents the force and / or torque change of the clamping part during the execution of the target task.

[0179] The audio data represents the voice instruction or environmental audio collected during the execution of the target task.

[0180] The eighth pose trajectory represents the position change of the center point of the end of the clamping part during the execution of the target task.

[0181] The target opening and closing trajectory represents the change of the opening and closing width of the clamping part during the execution of the target task.

[0182] In some embodiments, the above-mentioned data can be packaged and stored after being time-stamped, and named according to the starting time of collection. Then the above-mentioned data collection process is repeated, and the collection data of each target task is packaged and stored in turn.

[0183] In some embodiments, the video stream data collected by the second camera module during the execution of the target task by the clamping tool may be affected by factors such as light and obstruction, resulting in data in the target opening and closing trajectory calculated that exceeds the range of the opening and closing width. The maximum and minimum values of the opening degree of the clamping jaw can be collected before the execution of the target task, so that the movement of the clamping jaw can be calibrated according to the maximum and minimum values.

[0184] Optionally, the fifteenth transformation matrix between the first positioning code map coordinate system and the second positioning code map coordinate system can be determined according to formula (5), and the opening degree between the clamping jaws during the execution of the target task can be calculated according to the fifteenth transformation matrix, and mapped between the maximum and minimum values to achieve calibration.

[0185] In some embodiments, by slotting the first positioning code map and the second positioning code map on the positioning code map of the clamping tool, by controlling the clamping jaw to slowly open and close, and by collecting image data covering the maximum and minimum states of the movement range of the clamping jaw, the thirteenth transformation matrix between the camera coordinate system and the first positioning code map, and the fourteenth transformation matrix between the camera coordinate system and the second positioning code map coordinate system can be determined according to the image data. By substituting the thirteenth transformation matrix and the fourteenth transformation matrix into the aforementioned formula (5), the fifteenth transformation matrix can be obtained to calculate the opening degree of the clamping jaw according to the fifteenth transformation matrix, so as to determine the maximum and minimum values.

[0186] Figure 11 A flowchart of the operation data collection of the robot is shown. Figure 11 It includes a pre-collection preparation stage, a data collection stage, a pose trajectory extraction stage, and a data set formation stage.

[0187] Pre-collection preparation stage: by arranging a base station in the operation space and making the operator wear a data acquisition device, i.e. a base tracking tool and a clamping tool; by recording the environment of the operation space to determine the first transformation matrix between the first space coordinate system and the second space coordinate system according to the recorded data; by recording the clamping jaw opening calibration data to calibrate the target opening and closing trajectory between the first clamping finger and the second clamping finger according to the recorded data.

[0188] Data collection stage: using the base tracking tool and the gripping tool to record operation data, including but not limited to: using the first camera module of the base tracking tool, the first tracker to record operation data, and using the second camera module of the gripping tool, the second tracker to record operation data.

[0189] Pose trajectory extraction stage: according to the operation data recorded by the base tracking tool, the pose trajectory of the robot base can be extracted, that is, the first operation data of the embodiment of the present disclosure; according to the operation data recorded by the gripping tool, the pose trajectory of the joint of the mechanical arm can be extracted, that is, the second operation data of the embodiment of the present disclosure; from the operation data recorded by the gripping tool and the environmental data of the operation space, the pose trajectory of the end of the mechanical arm can be extracted, that is, the eighth pose trajectory of the present disclosure; from the target opening and closing trajectory between the first clamping finger and the second clamping finger; according to the clamping jaw opening degree calibration data and the operation data recorded by the gripping tool, the clamping jaw opening degree trajectory can be extracted, that is, the target opening and closing trajectory between the first clamping finger and the second clamping finger in the present disclosure.

[0190] Data set construction stage: through the pose trajectory extracted by the pose trajectory extraction stage and the collected video data and other information, the first operation data, the second operation data, the image data, the gripping force data, the force data, the audio data, the eighth pose trajectory, and the target opening and closing trajectory can be obtained, so as to constitute the data set of the robot operation data. In the embodiment of the present disclosure, the way of determining the data in the data set can refer to the above-mentioned embodiments, which will not be described here.

[0191] Figure 12 A block diagram of the robot operation data collection system is shown. As Figure 12 shown, the robot operation data collection system 1200 of the embodiment of the present disclosure includes a base tracking tool 1201 and a processor 1202.

[0192] In some embodiments, the base tracking tool 1201 includes a first camera module 1211 and a first tracker 1212.

[0193] Optionally, the first camera module 1211 is configured to collect a first pose trajectory of a robot base performing a target task in an operation space.

[0194] Optionally, the first tracker 1212 is configured to collect a second pose trajectory of an operator performing a target task in an operation space.

[0195] Optionally, the processor 1202 is configured to determine first operation data of a robot base corresponding to a target task according to the first pose trajectory and the second pose trajectory.

[0196] It should be noted that the specific implementation of collecting the first pose trajectory, the second pose trajectory, and determining the first operation data according to the first pose trajectory and the second pose trajectory can refer to the above-mentioned embodiments, which will not be repeated here.

[0197] In some embodiments, the operation data collection system 1200 of the robot further includes a clamping tool 1203.

[0198] The clamping tool 1203 includes a second tracker 1231.

[0199] In some embodiments, the second tracker 1231 is configured to collect a fourth pose trajectory of the clamping tool performing a target task in the operation space.

[0200] In some embodiments, the processor 1202 is further configured to drive the robot model to perform trajectory restoration in the simulation space of the operation space according to the fourth pose trajectory, and collect the position and motion data of the joints of the robot model at the restored trajectory points as the second operation data of the joints of the robot corresponding to the target task.

[0201] It should be noted that the specific implementation of collecting the fourth pose trajectory and determining the second operation data according to the fourth pose trajectory can refer to the above-mentioned embodiments, which will not be repeated here.

[0202] The operation data collection system of the robot provided by the embodiments of the present disclosure can collect the first pose trajectory of the robot base performing a target task in the operation space through the first camera module of the base tracking tool, and collect the second pose trajectory of the operator performing the target task in the operation space through the first tracker, so as to determine the first operation data of the robot base according to the first pose trajectory and the second pose trajectory. Further, the fourth pose trajectory of the clamping tool performing the target task in the operation space can be collected through the second tracker of the clamping tool, so as to determine the second operation data of the joints of the robot corresponding to the target task according to the fourth pose trajectory. Therefore, the present scheme can realize data collection of the motion trajectory of the robot base, and improve the comprehensiveness of data collection. The motion data of the joints of the robot can also be collected, and the reliability of the data can be improved through mixed collection in the simulation environment and the real environment. The data is collected through the data acquisition device, which is convenient to carry to various scenes to collect data, so as to expand the diversity of the data.

[0203] Figure 13 is a block diagram of a robot operation data collection device according to an example embodiment. Referring to Figure 13 , the robot operation data collection device 1300 of the embodiments of the present disclosure includes:

[0204] The first acquisition module 1301 is configured to acquire, by a first camera module, a first pose trajectory of a robot base performing a target task in an operation space.

[0205] The second acquisition module 1302 is configured to acquire, by a first tracker, a second pose trajectory of an operator performing the target task in the operation space.

[0206] The determination module 1303 is configured to determine first operation data of the robot base corresponding to the target task according to the first pose trajectory and the second pose trajectory.

[0207] In an embodiment of the present disclosure, the determination module 1303 is further configured to: perform coordinate system conversion on the second pose trajectory to determine a third pose trajectory in a first space coordinate system; and determine the first operation data according to the first pose trajectory and the third pose trajectory.

[0208] In an embodiment of the present disclosure, the determination module 1303 is further configured to: determine a first transformation matrix between the first space coordinate system and a second space coordinate system based on the first space coordinate system and the second space coordinate system a second transformation matrix between a first local coordinate system of the first tracker and the second space coordinate system and a third transformation matrix between the first local coordinate system and a base coordinate system determine a fourth transformation matrix between the base coordinate system and the first space coordinate system map the second pose trajectory to the first space coordinate system according to the fourth transformation matrix to obtain the third pose trajectory.

[0209] In an embodiment of the present disclosure, the apparatus further includes: a third acquisition module 1304 configured to acquire, by a second tracker of the clamping tool, a fourth pose trajectory of the clamping tool performing the target task in the operation space; a simulation module 1305 configured to drive a robot model to perform trajectory restoration in a simulation space of the operation space according to the fourth pose trajectory; and a fourth acquisition module 1306 configured to acquire position and motion data of a robot arm joint when the robot model restores a trajectory point, as second operation data of the robot arm joint corresponding to the target task.

[0210] In an embodiment of the present disclosure, the simulation module 1305 is further configured to: perform first coordinate conversion on the fourth pose trajectory to determine a fifth pose trajectory of the clamping tool in the first space coordinate system; perform second coordinate conversion on the fifth pose trajectory to determine a sixth pose trajectory of the clamping tool in a base coordinate system of the robot; and drive the robot model to perform trajectory restoration in the simulation space of the operation space according to the sixth pose trajectory.

[0211] In an embodiment of the present disclosure, the simulation module 1305 is further configured to: determine a first transformation matrix between the first space coordinate system and a second space coordinate system based on the first space coordinate system and the second space coordinate system a fifth transformation matrix between the second local coordinate system of the second tracker and the second spatial coordinate system and a sixth transformation matrix between the second local coordinate system and the end coordinate system of the robot arm determine a seventh transformation matrix between the end coordinate system and the first spatial coordinate system according to the seventh transformation matrix map the fourth pose trajectory to the first spatial coordinate system to obtain a fifth pose trajectory.

[0212] In an embodiment of the present disclosure, the simulation module 1305 is further configured to: determine a tenth transformation matrix between the second local coordinate system and the second spatial coordinate system according to the seventh pose trajectory of the operator pre-acquired by the second tracker and an eighth transformation matrix between the base coordinate system and the first spatial coordinate system determine a ninth transformation matrix between the end coordinate system and the base coordinate system according to the ninth transformation matrix map the fifth pose trajectory to the base coordinate system to obtain a sixth pose trajectory.

[0213] In an embodiment of the present disclosure, the simulation module 1305 is further configured to: determine a tenth transformation matrix between the second local coordinate system and the second spatial coordinate system according to the seventh pose trajectory of the operator pre-acquired by the second tracker determine an eleventh transformation matrix between the second local coordinate system and the camera coordinate system according to the three-dimensional models of the second camera module in the clamping tool and the second tracker determine a twelfth transformation matrix between the camera coordinate system and the first spatial coordinate system determine the first transformation matrix according to the tenth transformation matrix, the eleventh transformation matrix and the twelfth transformation matrix

[0214] In an embodiment of the present disclosure, the device further comprises: acquiring a plurality of seventh pose trajectories; determining a plurality of candidate first transformation matrices corresponding to the plurality of seventh pose trajectories respectively; determining a final first transformation matrix from the plurality of candidate first transformation matrices based on a statistical analysis result.

[0215] In an embodiment of the present disclosure, the operation data further comprises at least one of the following data: an eighth pose trajectory of the end of the robot arm acquired by the second camera module in the clamping tool; a target opening and closing trajectory between the first clamping finger and the second clamping finger in the clamping tool; clamping force data of the clamping part in the clamping tool; stress data of the clamping part; audio data and image data in the process of executing the target task.

[0216] The operation data acquisition device of the robot provided by the embodiment of the present disclosure can track the first camera module of the tool to acquire the first pose trajectory of the robot base in the operation space when performing the target task, and can track the first tracker to acquire the second pose trajectory of the operator in the operation space when performing the target task, so that the first operation data of the robot base can be determined according to the first pose trajectory and the second pose trajectory. Therefore, the present scheme can realize data acquisition of the motion trajectory of the robot base, and improve the comprehensiveness of data acquisition. The data acquisition device is convenient to carry to various scenes to collect data, so that the diversity of data can be improved.

[0217] As to the device in the above embodiment, the specific manner in which the various modules perform operations has been described in detail in the embodiments relating to the method, and will not be described in detail here.

[0218] Figure 14 is a block diagram of an electronic device according to an example embodiment.

[0219] Optionally, the electronic device can include a server, a user terminal and other smart devices. Optionally, the user terminal includes but is not limited to a mobile phone, a computer, a smart voice interaction device and the like. Optionally, the server includes but is not limited to a network server, an application server, and can also be a server of a distributed system, or a server combined with a blockchain, etc.

[0220] As shown in Figure 14 , the above electronic device 1400 includes:

[0221] The memory 1401 and the processor 1402, the bus 1403 connecting different components (including the memory 1401 and the processor 1402), the memory 1401 stores a computer program, and the processor 1402 executes the program to realize the operation data acquisition method of the robot in the embodiment of the present disclosure.

[0222] The bus 1403 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to industry standard architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video electronics standards association (VESA) local bus, and peripheral component interconnect (PCI) bus.

[0223] The electronic device 1400 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 1400, including volatile and non-volatile media, removable and non-removable media.

[0224] The storage 1401 also can include a computer-readable storage medium in the form of a volatile memory, such as random access memory (RAM) 1404 and / or cache memory 1405. The electronic device 1400 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 1406 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Figure 14 Although not shown, a magnetic disk drive can also be utilized in some embodiments to read from and write to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive can be utilized in some embodiments to read from and write to a removable, non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media). In these instances, each drive can be connected to the bus 1403 by one or more data media interfaces. The storage 1401 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the disclosure. Figure 14

[0225] Program / utility 1408, having a set (at least one) of program modules 1407, can be stored in, for example, storage by way of example, and not limitation, an operating system, one or more application programs, other program modules, and program data, each of which

[0226] The electronic device 1400 can also communicate with one or more external devices 1409 such as a keyboard or a pointing device, displays 1491, etc.; other devices Figure 14 such as a printer; or a scanner. Communication with one or more external devices 1409 can occur, for example, through an input / output (I / O) interface 1492. Still yet, the electronic device 1400 can communicate with one or more networks, such as one or more local area networks (LANs), wide area networks (WANs), and / or the Internet, through a network adapter 1493. As illustrated, the network adapter 1493 can communicate with the other components of the electronic device 1400 through the bus 1403. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with the electronic device 1400. Such components can include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0227] The processor 1402 performs various function applications and data processing by running programs stored in the memory 1401.

[0228] It should be noted that the implementation process and technical principles of the electronic device of the embodiment are referred to the above description of the robot operation data acquisition method of the embodiment of the disclosure, and will not be repeated here.

[0229] In order to realize the above-mentioned embodiment, the disclosure further provides a computer readable storage medium, which stores computer program instructions, and the program instructions are executed by a processor to realize the steps of the robot operation data acquisition method provided by the disclosure.

[0230] Optionally, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0231] It should be noted that the terms of interface circuit, interface, transceiver pin, and transceiver can be replaced with each other.

[0232] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. The disclosure is intended to cover any variations, uses or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure that come within known

[0233] It should be understood that the disclosure is not limited to the precise structures described and shown in the drawings, and that various modifications and changes can be made without departing from its scope. The scope of the disclosure is limited only by the claims that follow.

Claims

1. A method of collecting operation data of a robot, characterized by, Comprising: acquiring, by a first camera module, a first pose trajectory of a robot base performing a target task in an operation space, and acquiring, by a first tracker, a second pose trajectory of an operator performing the target task in the operation space; determining first operation data of the robot base corresponding to the target task according to the first pose trajectory and the second pose trajectory.

2. The method of claim 1, wherein, The determining first operation data of the robot base corresponding to the target task according to the first pose trajectory and the second pose trajectory comprises: performing coordinate system conversion on the second pose trajectory to determine a third pose trajectory in a first space coordinate system; determining the first operation data according to the first pose trajectory and the third pose trajectory.

3. The method of claim 2, wherein, The performing coordinate system conversion on the second pose trajectory to determine a third pose trajectory in a first space coordinate system comprises: a first transformation matrix between the first spatial coordinate system and the second spatial coordinate system a second transformation matrix between a first local coordinate system of the first tracker and the second spatial coordinate system a third transformation matrix between the first local coordinate system and a base coordinate system determining a fourth transformation matrix between the base coordinate system and the first spatial coordinate system According to the fourth transformation matrix Mapping the second pose trajectory to the first spatial coordinate system to obtain the third pose trajectory.

4. The method of claim 1, wherein, The method further comprises: acquiring, by a second tracker of a clamping tool, a fourth pose trajectory of the clamping tool performing the target task in the operation space; driving a robot model to perform trajectory restoration in a simulation space of the operation space according to the fourth pose trajectory; acquiring position and motion data of a robot arm joint when the robot model restores the trajectory point as second operation data of the robot arm joint corresponding to the target task.

5. The method of claim 4, wherein, The driving a robot model to perform trajectory restoration in a simulation space of the operation space according to the fourth pose trajectory comprises: performing first coordinate conversion on the fourth pose trajectory to determine a fifth pose trajectory of the clamping tool in the first space coordinate system; performing second coordinate conversion on the fifth pose trajectory to determine a sixth pose trajectory of the clamping tool in a base coordinate system of the robot; driving a robot model to perform trajectory restoration in a simulation space of the operation space according to the sixth pose trajectory.

6. The method of claim 5, wherein, The performing first coordinate conversion on the fourth pose trajectory to determine a fifth pose trajectory of the clamping tool in the first space coordinate system comprises: a first transformation matrix between the first local coordinate system of the first tracker and the second spatial coordinate system a fifth transformation matrix between a second local coordinate system of the second tracker and the second spatial coordinate system and a sixth transformation matrix between the second local coordinate system and a coordinate system of an end of a robot arm determining a seventh transformation matrix between the coordinate system of the end of the robot arm and the first spatial coordinate According to the seventh transformation matrix Mapping the fourth pose trajectory to the first spatial coordinate system to obtain the fifth pose trajectory.

7. The method of claim 5, wherein, The performing second coordinate conversion on the fifth pose trajectory to determine a sixth pose trajectory of the clamping tool in a base coordinate system of the robot comprises: based on the seventh transformation matrix and an eighth transformation matrix between the base coordinate system and the first spatial coordinate system determining a ninth transformation matrix between the end coordinate system and the base coordinate system According to the ninth conversion matrix The fifth pose trajectory is mapped to the base coordinate system to obtain the sixth pose trajectory.

8. The method according to any one of claims 4-7, characterized in that, The first transformation matrix The determination comprises: determining a seventh pose trajectory of the operator in advance according to the second tracking device determining an eleventh transformation matrix between the second local coordinate system and a camera coordinate system of a second camera module of the second tracker and the clamping tool, respectively determining a twelfth transformation matrix between the camera coordinate system and the first spatial coordinate system determining the first transformation matrix according to the tenth transformation matrix, the eleventh transformation matrix and the twelfth transformation matrix 9. The method of claim 8, wherein, The method further comprises: acquiring a plurality of seventh pose trajectories; determining a plurality of candidate first transformation matrices corresponding to the plurality of seventh pose trajectories respectively; determining a final first transformation matrix from the plurality of candidate first transformation matrices based on a statistical analysis result.

10. The method of any one of claims 4-8, wherein, The operation data further comprises at least one of the following data: an eighth pose trajectory of the robot arm end acquired by a second camera module in the clamping tool; a target opening and closing trajectory between a first clamping finger and a second clamping finger in the clamping tool; clamping force data of a clamping part in the clamping tool; stress data of the clamping part; audio data and image data in the process of performing the target task.

11. An operation data collection device of a robot, characterized by comprising: The device comprises: a first acquisition module configured to acquire, by a first camera module, a first pose trajectory of a robot base performing a target task in an operation space; a second acquisition module configured to acquire, by a first tracker, a second pose trajectory of an operator performing the target task in the operation space; The determining module is configured to determine first operation data of a robot base corresponding to the target task according to the first pose trajectory and the second pose trajectory.

12. The apparatus of claim 11, wherein, The determining module is further configured to: perform coordinate system conversion on the second pose trajectory to determine a third pose trajectory in a first spatial coordinate system; determine the first operation data according to the first pose trajectory and the third pose trajectory.

13. The apparatus of claim 11, wherein, The apparatus further includes: a third acquisition module configured to acquire, by a second tracker of a clamping tool, a fourth pose trajectory of the clamping tool in an operation space performing a target task; a simulation module configured to drive a robot model to perform trajectory restoration in a simulation space of the operation space according to the fourth pose trajectory; a fourth acquisition module configured to acquire position and motion data of a robot arm joint of the robot model when restoring a trajectory point as second operation data of the robot arm joint corresponding to the target task.

14. An electronic device, comprising: The apparatus includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: implement steps of the method of any one of claims 1-10.

15. A computer-readable storage medium having stored thereon computer program instructions, wherein, The program instructions, when executed by the processor, implement steps of the method of any one of claims 1-10.

16. An operation data collection system of a robot characterized by comprising: The apparatus includes: a base tracking tool and a processor, the base tracking tool including a first camera module and a first tracker; the first camera module is configured to acquire a first pose trajectory of a robot base in an operation space performing a target task; the first tracker is configured to acquire a second pose trajectory of an operator in the operation space performing the target task; the processor is configured to determine first operation data of the robot base corresponding to the target task according to the first pose trajectory and the second pose trajectory.

17. The system of claim 16, wherein, The apparatus further includes: a clamping tool, the clamping tool including a second tracker, the second tracker is configured to acquire a fourth pose trajectory of the clamping tool in the operation space performing the target task; the processor is configured to drive a robot model to perform trajectory restoration in a simulation space of the operation space according to the fourth pose trajectory, and acquire position and motion data of a robot arm joint of the robot model when restoring a trajectory point as second operation data of the robot arm joint corresponding to the target task.

Images