Method for acquiring operation data of end of robot arm by data acquisition device

Abstract

This specification relates to the field of robot control technology, and specifically provides a method for obtaining operation data of the end of a robot arm through a data acquisition device, the method includes using a clamping tool to perform a clamping task, and collecting various raw data through a camera and various sensors, and then extracting the motion trajectory and opening and closing trajectory of the end of the clamping part from the raw data, and storing them together with the raw data as the operation data of the clamping task. Efficient collection of operation data of the end of the robot arm is achieved through a simple clamping tool, the operation method is simple, the technical threshold and cost are low, and data collection is completed in a real environment, avoiding the problem of differences between simulation data and real environment, and the data reliability is higher.

Description

Technical Field

[0001] The present specification relates to the field of robot control technology, and specifically to a method for acquiring operation data of a robot end through a data acquisition device, a clamping tool, an electronic device, and a storage medium. Background Art

[0002] In recent years, the application of deep neural networks (DNN) has brought revolutionary breakthroughs in the operation and control of robotic arms, realizing end-to-end robotic arm control strategies, showing excellent performance and a wide range of application scenarios. However, in order to fully tap the potential of DNN algorithms, building a large-scale robotic arm operation dataset is an indispensable basic step.

[0003] In related technologies, the collection of robot arm operation data is mainly through planning the actual operation of the real machine, or human remote control of the equipment to complete specific operations and record relevant data. This method requires large and expensive robot arm equipment, and the technical threshold is high, which is not conducive to the collection of large-scale operation data. Summary of the invention

[0004] In order to achieve low-cost, high-efficiency and comprehensive robot arm operation data collection and provide a reliable data basis for control model training, this manual provides a method, a clamping tool and a storage medium for obtaining robot arm end operation data through a data acquisition device.

[0005] In a first aspect, an embodiment of the present specification provides a method for acquiring operation data of a robot end through a data acquisition device, wherein the data acquisition device includes a clamping tool, the clamping tool includes a clamping portion, the clamping portion includes a first clamping finger and a second clamping finger, the first clamping finger is provided with a first positioning coding diagram, and the second clamping finger is provided with a second positioning coding diagram, the method includes:

[0006] When the clamping tool is used to perform a clamping task, a first video stream, clamping force data, force data and audio data are obtained, wherein the first video stream represents an image video during the clamping task, the first inertial data represents a change in inertial data of the clamping tool during the clamping task, the clamping force data represents a change in the clamping force of the clamping part during the clamping task, the force data represents a change in the force and / or torque of the clamping part during the clamping task, and the audio data represents a voice command or ambient audio collected during the clamping task;

[0007] Acquire pre-constructed environmental information, the environmental information including the operating space environment of the clamping tool, perform camera tracking and positioning based on the first video stream and the first inertial data, determine the target motion trajectory of the center point of the end of the clamping part in a pre-defined second space coordinate system, and the clamping tool moves in the operating space;

[0008] Determine the target opening and closing trajectory of the first clamping finger and the second clamping finger according to the first positioning code map and the position change information of the first positioning code map in the first video stream, wherein the target opening and closing trajectory represents the change of the opening and closing width of the clamping part during the execution of the clamping task;

[0009] According to the first video stream, the first inertial data, the clamping force data, the force data, the audio data, the target motion trajectory and the target opening and closing trajectory, the operation data corresponding to the clamping task is obtained, and the target motion trajectory represents the position change of the center point of the end of the clamping part during the execution of the clamping task.

[0010] In one implementation of this specification, the obtaining of pre-built environment information includes:

[0011] Driving the clamping tool to move in the operating space, the operating space is provided with a first positioning mark;

[0012] Acquire a second video stream of the operating space and second inertial data of the clamping tool during movement;

[0013] Based on the second video stream and the second inertial data, the environmental information of the operating space is constructed, and a first transformation matrix between a first spatial coordinate system and a second spatial coordinate system is determined, wherein the first spatial coordinate system represents a camera posture corresponding to a first frame image in the second video stream, the second spatial coordinate system represents a coordinate system of the first positioning mark in the operating space, and the camera is used to collect the video stream.

[0014] In one implementation of this specification, determining a first transformation matrix between the first space coordinate system and the second space coordinate system includes:

[0015] Extract features from the frame images in the second video stream to obtain feature points of each frame image, and estimate the initial pose of the camera based on the second inertial data to obtain an initial pose;

[0016] Perform feature matching on the feature points of each frame of image with the first frame of image, and perform posture estimation based on inertial data corresponding to each frame of image and the initial posture to determine a second transformation matrix between the first space coordinate system and the camera coordinate system;

[0017] Tracking and locating the first positioning mark in the second video stream using a landmark detection algorithm to determine a third transformation matrix between the camera coordinate system and the second space coordinate system;

[0018] The first transformation matrix is ​​obtained according to the second transformation matrix and the third transformation matrix.

[0019] In one embodiment of the present specification, obtaining pre-constructed environmental information, the environmental information including the operating space environment of the clamping tool, performing camera tracking and positioning based on the first video stream and the first inertial data, and determining the target motion trajectory of the center point of the end of the clamping part in the pre-defined second space coordinate system includes:

[0020] Loading the environmental information, performing camera tracking and positioning based on the first video stream and the first inertial data, determining a motion trajectory of the camera in the first space coordinate system in the gripping task, and a fourth transformation matrix between the camera coordinate system and the first space coordinate system;

[0021] Obtaining a fifth transformation matrix between a pre-calibrated tool coordinate system and a camera coordinate system;

[0022] Performing homogeneous transformation based on the fourth transformation matrix, the fifth transformation matrix and the first transformation matrix to obtain a sixth transformation matrix between the tool coordinate system and the second space coordinate system;

[0023] The motion trajectory of the camera in the first space coordinate system is spatially transformed based on the sixth transformation matrix to obtain the target motion trajectory.

[0024] In one embodiment of the present specification, the use of a landmark detection algorithm to track and locate the first positioning code map and the second positioning code map in the first video stream to determine the target opening and closing trajectory of the first gripper finger and the second gripper finger includes:

[0025] Tracking and locating the clamping part in the first video stream, determining a seventh transformation matrix between the camera coordinate system and the first positioning code map coordinate system, and an eighth transformation matrix between the camera coordinate system and the second positioning code map coordinate system;

[0026] Performing homogeneous transformation based on the seventh transformation matrix and the eighth transformation matrix to obtain a ninth transformation matrix between the first positioning coding map coordinate system and the second positioning coding map coordinate system;

[0027] The target opening and closing trajectory is determined based on the ninth transformation matrix.

[0028] In one embodiment of the present specification, it also includes:

[0029] Driving the clamping part of the clamping tool to open and close, and using the camera to collect a third video stream of the opening and closing process of the clamping part;

[0030] Tracking and locating the clamping portion in the third video stream to determine the opening and closing width range of the clamping portion;

[0031] The determining the target opening and closing trajectory based on the ninth transformation matrix includes:

[0032] Determine a first opening and closing trajectory based on a ninth transformation matrix;

[0033] The first opening and closing trajectory is normalized based on the opening and closing width range to obtain the target opening and closing trajectory.

[0034] In a second aspect, the present specification provides a clamping tool, the clamping tool comprising:

[0035] The clamping tool body;

[0036] A clamping portion provided on the body, the clamping portion comprising a first clamping finger and a second clamping finger, the first clamping finger being provided with a first positioning coding pattern, and the second clamping finger being provided with a second positioning coding pattern;

[0037] A driving mechanism, comprising an operating rod and a transmission member, wherein the operating rod is connected to the clamping portion through the transmission member, and when a user applies a force to the operating rod, the operating rod drives the first clamping finger and the second clamping finger to open and close through the transmission member;

[0038] A camera, fixedly mounted on the body, wherein the field of view of the camera at least covers the range of movement of the clamping portion;

[0039] An inertial sensor is provided on the body and is used to detect the inertial data of the movement process of the data acquisition device; a clamping force sensor is provided on the clamping surface of the clamping part and is used to detect the clamping force data when the clamping part clamps the object; a six-dimensional force sensor is provided at the connection position between the operating rod and the body and is used to detect the force data when the clamping part clamps the object; a pickup is provided on the body and is used to pick up audio data during the data acquisition process;

[0040] The clamping tool is used to perform the method shown in the first aspect.

[0041] In the embodiment of the present specification, the clamping part is assembled on the body through a linear guide rail, and the operating rod drives the first clamping finger and the second clamping finger to reciprocate on the linear guide rail through the transmission member to realize the opening and closing of the clamping part;

[0042] The driving mechanism further comprises a reset member, which is arranged between the operating rod and the body. When the user does not apply a force to the operating rod, the reset member drives the operating rod to reset.

[0043] In a third aspect, this specification provides an electronic device, including:

[0044] A processor is communicatively connected to the camera of the clamping tool, and the processor executes the acquisition method shown in the first aspect according to the received data.

[0045] According to a fourth aspect, a storage medium is provided, which stores computer instructions, wherein the computer instructions are used to cause a processor to execute the acquisition method according to the first aspect.

[0046] The operation data acquisition method implemented in this specification uses a clamping tool to perform a clamping task. The robot does not need to actually perform the task. Instead, the original data is collected through a camera and various sensors, and then the motion trajectory and opening and closing trajectory of the end of the clamping part are extracted based on the original data, and the original data are combined and stored together as the operation data of the clamping task. The trajectory of the clamping tool collected by this specification is more real and accurate through the above data. In addition, this specification establishes a first spatial coordinate system through the first frame image of the second video stream collected in the calibration stage, that is, the first frame image of the second video stream corresponds to the first spatial coordinate system, and the first spatial coordinate system is used to determine the posture of the camera during movement. The second video stream is collected before the first video stream, so when calculating the camera posture in the first video stream, it can be directly calculated with reference to the first spatial coordinate system. Compared with the prior art, the initial position of the camera is calibrated multiple times by the first frame of each video stream to perform complex calculations to determine the camera posture. This solution is simpler and more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific implementation methods of this specification or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 Schematic diagram of the structure of a clamping tool in an exemplary embodiment of this specification.

[0049] Figure 2 Schematic diagram of the structure of a clamping tool in an exemplary embodiment of this specification.

[0050] Figure 3 It is a structural schematic diagram of a data acquisition device in an exemplary embodiment of this specification.

[0051] Figure 4 is a schematic diagram of various coordinate systems in an exemplary embodiment of this specification.

[0052] Figure 5 is a flowchart of an acquisition method in an exemplary embodiment of this specification.

[0053] Figure 6 is a flowchart of an acquisition method in another exemplary embodiment of this specification.

[0054] Figure 7 is a flowchart of an acquisition method in another exemplary embodiment of this specification.

[0055] Figure 8 is a flowchart of an acquisition method in another exemplary embodiment of this specification.

[0056] Fig. 9 is a flowchart of an acquisition method in another exemplary embodiment of this specification.

[0057] Fig.10 is a flowchart of an acquisition method in another exemplary embodiment of this specification.

[0058] Fig.11 is a structural block diagram of a control device in an exemplary embodiment of this specification. DETAILED DESCRIPTION

[0059] The technical solution of this specification will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described implementation is a part of the implementation of this specification, not all of the implementation. Based on the implementation in this specification, all other implementations obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification. In addition, the technical features involved in the different implementations of this specification described below can be combined with each other as long as they do not conflict with each other.

[0060] In the related technology, the method of collecting robot arm operation data is to manually design the operation process and control the real robot arm to complete the operation task, and record the various data in the process of executing the operation task to build a data set. This method relies on manually designing the action sequence of the robot arm, which has a high technical threshold and requires professional staff to program the task code. In addition, a real robot arm needs to be provided, and the collection cost is high.

[0061] In order to lower the technical threshold for data collection, remote control operation has been widely adopted. For example, the dual-arm data collection platform includes two sets of equipment: the master robot and the slave robot. In the remote operation mode, the user only needs to drag the master robot controlled by the teaching arm to perform the operation task. The master robot records various data and sends them to the slave robot. The slave robot follows and performs the same action to achieve motion mapping. This method does not require users to perform complex action programming design, and data collection is more efficient, but it requires a bulky and expensive robot and operation platform, which limits large-scale data collection in a wider range of scenarios.

[0062] To further reduce the cost of data collection, some related technologies use simulation software to simulate the physical behavior and environmental interaction of the robot in the real world in a simulation environment to generate an operation data set. This method can generate a large amount of operation data in a virtual environment, and does not require real robot equipment and operation equipment. It is low-cost and has high data collection efficiency. However, there are inevitable differences between objects in the simulation environment and objects in the real world. The data generated by simulation software may not fully reflect the complexity of the real world, resulting in limited generalization ability of these data in real applications, and the control strategy learned by the network model is not adaptable enough in the real world.

[0063] Based on this, the implementation method of this specification provides a method for acquiring the end operation data of the robot arm through a data acquisition device, aiming to realize the efficient collection of the end operation data of the robot arm with a simple handheld clamping tool. The operation method is simple, the technical threshold and cost are low, and the data collection is completed in a real environment, avoiding the problem of differences between simulation data and the real environment, and the data reliability is higher.

[0064] In some embodiments, the present specification provides a clamping tool, which is used to collect data of operation data at the end of a robot arm.

[0065] It can be understood that for a robotic arm or manipulator, its end is generally a clamping structure, such as a mechanical clamping part. The operation data collection of the robotic arm is to collect data on the motion trajectory and motion posture of the clamping mechanism at the end of the robotic arm. When controlling the motion of the robotic arm, the kinematic algorithm can be used to calculate in real time according to the end motion position and posture of each joint of the robotic arm, and the control parameters such as the force / torque, speed / acceleration and the degree of opening and closing of the clamping part, so as to realize the operation control of the robotic arm.

[0066] Therefore, in the implementation manner of this specification, a clamping tool is used to perform the clamping task, and relevant data of the end of the clamping tool is collected to realize the collection of operation data of the end of the robot arm.

[0067] In some embodiments, the clamping tool in this specification may be a handheld clamping tool, that is, the user holds the clamping tool to collect the operation data. Figure 1 and Figure 2 The structure of a handheld clamping tool in an exemplary embodiment of this specification is shown below. Figure 1 and Figure 2 Provide explanation.

[0068] See also Figure 1 and Figure 2 As shown, the handheld clamping tool includes a body 110 , which refers to the main structure of the clamping tool, including a shell and other bearing structures for bearing various mechanical structures and electrical components.

[0069] The main body 110 is provided with a clamping part 120, and the clamping part 120 is composed of two clamping fingers, namely a first clamping finger 121 and a second clamping finger 122. The clamping part 120 is assembled on the main body 110 through a linear slide 130. Specifically, the first clamping finger 121 is fixedly connected to the first slider 131, and the first slider 131 is installed on the linear slide 130, so that the first slider 131 slides back and forth on the linear slide 130, which can drive the first clamping finger 121 to slide back and forth along the x-axis direction. The second clamping finger 122 is fixedly connected to the second slider 132, and the second slider 132 is installed on the linear slide 130, so that the second slider 132 slides back and forth on the linear slide 130, which can drive the second clamping finger 122 to slide back and forth along the x-axis direction.

[0070] It can be understood that as the first clamping finger 121 and the second clamping finger 122 slide in the x-axis direction, the distance between the two clamping fingers will change, and the opening and closing of the clamping part can be achieved by changing the distance. Figure 1 In the direction shown, when the first clamping finger 121 slides toward the +x direction and the second clamping finger 122 slides toward the -x direction, the opening and closing width between the two clamping fingers gradually decreases, and the object can be clamped. Conversely, when the first clamping finger 121 slides toward the -x direction and the second clamping finger 122 slides toward the +x direction, the opening and closing width between the two clamping fingers gradually increases, and the object can be released.

[0071] A camera 100 is fixedly provided above the body 110, and the field of view of the camera 100 needs to at least cover the operating range of the clamping portion 120, that is, the camera 100 can collect images or video streams of the clamping portion 120 when performing the clamping task. In the implementation of this specification, there is no restriction on the type and quantity of the camera 100, and the camera type can be an RGB camera, or a black and white camera, and can further include a depth camera, etc., and can be classified into a wide-angle camera or an ultra-wide-angle camera according to the field of view of the camera, and this is not repeated in the implementation of this specification.

[0072] It is worth noting that in the implementation manner of this specification, a coding diagram for tracking and positioning needs to be fixedly set on the first clamping finger 121 and the second clamping finger 122. Specifically, the coding diagram includes a rectangular two-dimensional code with binary code.

[0073] For example Figure 1 In the implementation example, a first positioning coding pattern slot 141 can be opened on the first slider 131, and a second positioning coding pattern slot 142 can be opened on the second slider 132. The first positioning coding pattern is placed in the first positioning coding pattern slot 141, and the second positioning coding pattern is placed in the second positioning coding pattern slot 142.

[0074] In some embodiments, the first positioning coding diagram and the second positioning coding diagram may both be rectangular QR codes based on binary coding, and the two may have different QR code marks, so that the algorithm end may detect and parse the QR code marks to identify the QR code position.

[0075] See also Figure 2 As shown, the clamping tool also includes a driving mechanism, which refers to a related structure used to drive the clamping part to open and close. For example, in some embodiments, the driving mechanism includes an operating rod and a transmission member, the operating rod includes a fixed rod 162 and a movable rod 161, the fixed rod 162 is fixedly connected to the body 110, and the movable rod 161 is connected to the clamping part 120 through a transmission member (not shown in the drawings).

[0076] It can be understood that the fixed rod 162 can be used as a palm rest, and the movable rod 161 can be used as a handle or trigger. When the user holds the clamping tool, the palm can be placed on the fixed rod 162, and the fingers can hold the movable rod 161. When the user applies force to the movable rod 161, the movable rod 161 moves toward the fixed rod 162, and at the same time drives the transmission member to move. The transmission member can drive the first slider 131 and the second slider 132 to slide on the linear guide rail 130, thereby realizing the opening and closing control of the first clamping finger 121 and the second clamping finger 122.

[0077] In some embodiments, the driving mechanism may further include a reset member, which may be disposed between the movable rod 161 and the main body 110. The reset member may be an elastic part such as a reset spring. When the user releases the movable rod 161 with his palm, the reset member may drive the movable rod 161 to reset through elastic force, thereby driving the first clamping finger 121 and the second clamping finger 122 to return to their initial positions.

[0078] It is worth noting that the specific structure of the transmission member is not shown in the attached drawings. Here, a transmission structure that can be implemented is provided. For example, the movable rod 161 can be connected to the special-shaped gear in an articulated manner, and the special-shaped gear is meshed with the linear rack on the slider, so that when the movable rod 161 moves, the special-shaped gear can be driven to rotate, and the rotation of the special-shaped gear will drive the slider to move linearly, thereby realizing the reciprocating motion of the two clamping fingers along the linear slide rail, and realizing the opening and closing of the clamping part. Based on this description, those skilled in the art can undoubtedly understand and fully realize the deployment and implementation of the transmission member, and this specification will not be repeated.

[0079] The above describes the hardware structure and operating principle of the clamping tool. On this basis, a plurality of sensors and storage and processing units are also provided inside the clamping tool. The electrical part is described below.

[0080] A circuit board may be provided inside the body 110 , and various electrical components may be integrated on the circuit board. The electrical components may include but are not limited to various sensors, communication modules, MCU (Micro Control Unit), storage modules and related circuits thereof.

[0081] For example, in one example, the clamping tool includes an inertial sensor, which is also called an IMU (Inertial Measurement Unit) sensor, which generally includes an accelerometer and a gyroscope, and can detect the movement speed / acceleration and multi-axis posture of the device in real time. These detection parameters are inertial data.

[0082] The clamping tool also includes a clamping force sensor, e.g. Figure 1 As shown, the inner surfaces of the first clamping finger 121 and the second clamping finger 122 are defined as the clamping surface. It can be understood that when clamping the target object, the two clamping fingers approach each other and apply force to the target object to clamp the object, so the inner surface is the clamping surface. In some embodiments, a clamping force sensor can be set on the clamping surface of the two clamping fingers to detect the clamping force applied when clamping the object. The clamping force sensor can be a thin film pressure sensor, for example.

[0083] The clamping tool also includes a six-dimensional force sensor 170, such as Figure 2 As shown, the six-axis force sensor 170 is disposed at the connection position between the fixing rod 162 and the body 110. When the clamping part 120 clamps the target object, the six-axis force sensor 170 can collect the force and torque of the clamping part to obtain force data.

[0084] The clamping tool also includes a pickup 150, such as Figure 1As shown, the microphone 150 can be disposed above the clamping tool body 110 , and can pick up ambient sound by opening a microphone hole, thereby collecting audio data.

[0085] In some embodiments, the clamping tool further includes a processor, which is used to obtain data collected by the above-mentioned sensors and cameras, and execute the method steps described below in this specification according to the data to realize the collection of operation data.

[0086] It is worth noting that in some embodiments, the processor can be integrated inside the clamping tool, that is, the processor used to execute the method of acquiring the end-of-arm operation data through the data acquisition device is the processor of the clamping tool itself. In other embodiments, considering the lightweight and low cost of the clamping tool, the computing power of its own processing unit is relatively limited, so an external processor can be used to realize data processing and operation data collection. Figure 3 Provide explanation.

[0087] See also Figure 3 As shown, the present specification provides a data acquisition device, which includes the above-mentioned clamping tool and control device, and the control device and the clamping tool can establish a data communication connection in a wired or wireless manner. For example, in one example, a wireless communication module can be set in both the clamping tool and the control device, and the two can achieve wireless communication connection through the wireless communication module. The wireless communication module can be, for example, a Bluetooth, WiFi or other module. For example, in another example, the clamping tool and the control device can be connected using a wired communication module, for example, a wired connection can be established using a data cable and a data interface based on a USB (Universal Serial Bus) protocol.

[0088] The control device also includes a processor and a memory. The processor can be any type of processor with one or more processing cores. It can perform single-threaded or multi-threaded operations, and is used to parse instructions to perform operations such as acquiring data, performing logical operations, and sending down operation processing results. The memory may include a non-volatile computer-readable storage medium, such as at least one disk storage device, a flash memory device, a distributed storage device remotely arranged relative to the processor, or other non-volatile solid-state storage devices. The memory may have a program storage area for storing non-volatile software programs, non-volatile computer executable programs, and modules for the processor to call so that the processor executes one or more method steps. The memory may also include a volatile random storage medium, or a storage part such as a hard disk, as a data storage area, for storing the operation processing results and data sent down by the processor.

[0089] In the embodiments of the present specification, the memory of the control device stores computer-readable instructions that can be executed by the processor. When the computer-readable instructions are executed, the processor can implement the method of acquiring the end-of-arm operation data through the data acquisition device in the following embodiments of the present specification. On the other hand, the memory can also cache various sensor data and image data sent by the gripping tool, as well as various processing results in the execution method steps.

[0090] It is worth noting that the clamping tool in this specification can be made using 3D printing technology, which has a simple structure and a small size, and the 3D printing production cost is low. The data collection at the end of the robot arm can be realized with a low cost and a simple structure. In addition, in some embodiments, the clamping surfaces of the first clamping finger 121 and the second clamping finger 122 can be made of a material with a certain deformation ability, such as rubber, plastic, etc. The deformation ability of the clamping finger can improve the clamping stability and ensure that the data collected during frequent clamping is stable and reliable.

[0091] The above describes the structure and principle of the clamping tool and the data acquisition device of the implementation mode of this specification. Based on the above, the data acquisition process of the implementation mode of this specification is described below.

[0092] During data collection, the staff is required to hold the aforementioned clamping tool and use the clamping tool to perform multiple clamping tasks. During each clamping task, the image data and sensor data can be collected through the camera and various sensors on the clamping tool. Combined with the above, it can be understood that the operation data collection for the end of the robot arm is mainly for the collection of the motion trajectory and opening and closing trajectory of the end clamping part. Therefore, it is necessary to track and calculate the motion trajectory and opening and closing trajectory of the end of the clamping part based on the image data and related sensor data, and at the same time combine the original image data and sensor data to construct the operation data corresponding to a clamping task.

[0093] Before collecting the operation data, the space matrix and the opening of the clamping part need to be calibrated first. After obtaining the calibration data, the operation data collection process can be started. When collecting the operation data, the motion trajectory and opening and closing trajectory of the end of the clamping part can be tracked and calculated according to the calibrated space matrix and the opening of the clamping part. In the following implementation of this specification, the method steps of the "calibration stage" and the "data collection stage" will be explained respectively.

[0094] 1) Calibration stage

[0095] There are two goals in the calibration phase: one is to establish a first transformation matrix between the first spatial coordinate system and the second spatial coordinate system, through which any point in the first spatial coordinate system can be mapped to the second spatial coordinate system; the other is to calibrate the opening and closing width range of the clamping part, that is, the maximum width and minimum width that the first clamping finger 121 and the second clamping finger 122 can open and close.

[0096] For ease of understanding and explanation, the following Figure 4 Explanation of some coordinate system definitions involved in this manual:

[0097] 1. Second space coordinate system

[0098] Also called the base coordinate system or base coordinate system, it is a coordinate system defined to represent real-world positions.

[0099] 2. Camera coordinate system

[0100] Also called the camera coordinate system, it refers to a three-dimensional coordinate system relatively fixed to the camera 100 of the clamping tool, used to describe the position and direction of the camera relative to the observed scene. The origin of the camera coordinate system is generally the optical center of the camera.

[0101] 3. The first spatial coordinate system

[0102] The first spatial coordinate system is a reference coordinate system defined by the applicant. In the implementation mode of this specification, the camera posture corresponding to the first frame image in the second video stream collected during the calibration phase is defined as the first spatial coordinate system, whose coordinate origin is the camera position corresponding to the first frame image, and the coordinate axis direction is the three-axis direction of the camera corresponding to the first needle image. In addition, this specification establishes the first spatial coordinate system through the first frame image of the second video stream collected during the calibration phase, that is, the first frame image of the second video stream corresponds to the first spatial coordinate system, and the first spatial coordinate system is used to determine the posture of the camera during movement. The second video stream is collected before the first video stream, so when calculating the camera posture in the first video stream, it can be directly calculated with reference to the first spatial coordinate system. Through the first spatial coordinate system, compared with the prior art that calibrates the initial position of the camera through the first frame of each video stream for multiple times and performs complex calculations to determine the camera posture, this solution is simpler and more efficient.

[0103] 4. Tool coordinate system

[0104] Also called the Tip coordinate system, the Tip coordinate system is a coordinate system used to describe the posture of the end of the robot arm, also called the tool interface coordinate system. In the implementation of this specification, the origin of the Tip coordinate system is the center point of the two gripper ends, and the direction of the Tip coordinate system is the pre-defined coordinate axis direction.

[0105] 5. The first positioning coding map coordinate system and the second positioning coding map coordinate system

[0106] See also Figure 4 As shown, the positioning coding diagram coordinate system refers to the coordinate system where the positioning coding diagram set on the clamp finger is located. The origin of the positioning coding diagram coordinate system can be the center point or corner point of the positioning coding diagram, and the direction of the coordinate system is the predefined coordinate axis direction. In the implementation mode of this specification, the first positioning coding diagram coordinate system is the R_finger coordinate system, which represents the coordinate system of the first positioning coding diagram on the first clamp finger 121, and the second positioning coding diagram coordinate system is the L_finger coordinate system, which represents the coordinate system of the second positioning coding diagram on the second clamp finger 122.

[0107] In the implementation of this specification, the motion trajectory of the clamping part in the real world coordinate system (the second space coordinate system) needs to be calculated. Therefore, it is necessary to pre-calibrate the transformation matrix between the first space coordinate system and the second space coordinate system (the base coordinate system), that is, the first transformation matrix. Figure 5 Provide explanation.

[0108] like Figure 5 As shown, in some implementations, the acquisition method of the implementation of this specification, the process of calibrating the first conversion matrix includes:

[0109] S510, driving the clamping tool to move in an operating space, where a first positioning mark is provided, and acquiring a second video stream of the operating space and second inertial data of the clamping tool during its movement.

[0110] In the implementation manner of this specification, the operation space refers to the environment space used to perform the gripping task. For example, in one example, the operation space can be a desktop or tabletop where the target object can be placed. First, a first positioning mark can be placed on the desktop or tabletop. The first positioning mark is used as a mark for positioning detection, and the first positioning mark can be a binary-coded rectangular QR code.

[0111] It can be understood that the first positioning mark and the aforementioned first positioning coding diagram and the second positioning coding diagram can all be marking QR codes. The difference between the three QR codes is that the markings are different and the setting positions are different. The first positioning mark is fixedly set in the operating space, for example, attached to a certain position on the desktop, so that the camera 100 can collect the first positioning mark. The first positioning coding diagram is set on the first clamping finger 121 to follow the movement of the first clamping finger 121, and the second positioning coding diagram is set on the second clamping finger 122 to follow the movement of the second clamping finger 122. In other words, a total of 3 QR codes are involved in this specification, and their corresponding QR code marks are different.

[0112] After arranging the first positioning mark of the operation space, the staff can hold the clamping tool, move slowly in the operation space, and turn on the video recording of the camera, so that the camera can collect the video stream of the clamping tool in motion, which is the second video stream defined in this manual. At the same time, the inertial sensor on the clamping tool can collect inertial data, which is the second inertial data defined in this manual. The second inertial data can include parameters such as position, attitude, (angular) velocity, (angular) acceleration, etc.

[0113] It can be understood that the second video stream captures an image sequence of the operating space, and at least a portion of the images in the image sequence include the first positioning mark. The purpose of capturing the second video stream is, on the one hand, to use the second inertial data to construct a map of the operating space and track the camera motion trajectory, and on the other hand, to combine the positioning detection technology to map the camera's motion trajectory relative to the first space coordinate system to the second space coordinate system, thereby calibrating and obtaining the first transformation matrix.

[0114] S520: construct environmental information of the operating space based on the second video stream and the second inertial data, and determine a first transformation matrix between the first space coordinate system and the second space coordinate system.

[0115] In this specification, feature extraction can be performed on each frame of the video stream obtained above to obtain feature points included in each frame, and then feature matching can be performed on the feature points of adjacent frames. The purpose of feature point matching is to find the same point represented on adjacent frames, and then establish a corresponding relationship between feature points by continuously matching feature points between adjacent frames, and fuse IMU data to estimate the camera motion posture, optimize the camera trajectory and the position of map points, and realize the real-time generation and construction of environmental information and the tracking and detection of the camera motion trajectory. Exemplarily, the above environmental information can include map information.

[0116] Combined with the following Figure 6 The implementation method specifically describes the process of calibrating the first transformation matrix.

[0117] like Figure 6 As shown, in some implementations, the method exemplified in this specification, the calibration process of the first transformation matrix includes:

[0118] S521. Perform feature extraction on the frame images in the second video stream to obtain feature points of each frame image, and perform initial pose estimation on the camera based on the second inertial data to obtain an initial pose.

[0119] This specification extracts features from each frame of the second video stream to obtain feature points on each frame. For example, in one example, an image feature extraction algorithm can be used to extract feature points on each frame. The extracted feature points are generally corner points, edge points, and other points with obvious features on the image.

[0120] At the same time, according to the above, the second inertial data is the inertial data collected during the movement of the clamping tool in the operating space, and it is expressed synchronously with the frame image in the second video stream through a timestamp. Therefore, for the first frame image in the second video stream, the posture is estimated according to the inertial data corresponding to the first frame image, and the initial posture of the device can be obtained, including the three-axis posture and direction of the device. At this time, the posture and direction represented by the initial posture can be defined as the first space coordinate system.

[0121] S522, performing feature matching with the first frame image according to feature points of each frame image, and performing posture estimation based on inertial data corresponding to each frame image and the initial posture, to determine a second transformation matrix between the first space coordinate system and the camera coordinate system.

[0122] According to the above, the initial camera posture corresponding to the first frame image in the second video stream represents the first spatial coordinate system. Therefore, for the entire movement process, the posture change of each frame image relative to the first frame image can represent the movement trajectory of the camera in the first spatial coordinate system.

[0123] Specifically, the feature points of each frame image can be matched with the first frame image, and the motion trajectory of the camera can be tracked by combining the inertial data of each frame image with the posture change of the initial posture. The motion trajectory represents the motion trajectory of the camera in the first spatial coordinate system, so that the transformation matrix of the camera coordinate system relative to the first spatial coordinate system can be extracted. , the transformation matrix This is the second transformation matrix defined in this specification .

[0124] It can be understood that the second transformation matrix It represents the mapping relationship between the camera coordinate system and the first space coordinate system. In other words, for each point in the camera coordinate system, the second transformation matrix After the space conversion, it can be mapped to the corresponding position in the first space coordinate system.

[0125] S523. Track and locate the first positioning mark in the second video stream, and determine a third transformation matrix between the camera coordinate system and the second space coordinate system.

[0126] It can be understood that the second video stream is a video stream obtained by collecting images of the operation space. A first positioning mark is provided in the operation space, so the frame image in the second video stream contains the first positioning mark.

[0127] In some embodiments, the first positioning mark is a QR code, and the positioning tracking of the first positioning mark can be achieved by identifying the position and posture of the first positioning mark on each frame of the image.

[0128] According to the foregoing, in the implementation of this specification, the coordinate system where the first positioning mark is located is defined as the second space coordinate system, so that the motion trajectory of the camera relative to the second space coordinate system can be tracked through the positioning detection algorithm, and the transformation matrix between the camera coordinate system (camera coordinate system) and the second space coordinate system (base coordinate system) can be extracted. , the conversion matrix is ​​the third transformation matrix defined in this specification .

[0129] It can be understood that the third transformation matrix It represents the mapping relationship from the camera coordinate system to the second space coordinate system. In other words, for each point in the camera coordinate system, the third transformation matrix After the space conversion, it can be mapped to the corresponding position in the second space coordinate system.

[0130] S524. Obtain a first transformation matrix according to the second transformation matrix and the third transformation matrix.

[0131] It can be understood from the homogeneous transformation matrix that when there are multiple transformation matrices in a three-dimensional space, a transformation matrix for composite space transformation can be obtained by multiplying the multiple homogeneous transformation matrices in sequence.

[0132] In the implementation of this specification, the second transformation matrix It represents the homogeneous transformation matrix from the camera coordinate system to the first space coordinate system. The third transformation matrix It represents the homogeneous transformation matrix from the second space coordinate system to the camera coordinate system. Thus, by multiplying the homogeneous transformation matrix represented by the following formula (1), the first transformation matrix from the first space coordinate system to the second space coordinate system can be calculated. :

[0133] (1)

[0134] The first transformation matrix between the first space coordinate system and the second space coordinate system can be determined by the matrix multiplication algorithm of formula (1): , that is, for each point in the first spatial coordinate system, through the first transformation matrix It can be directly mapped to the corresponding position in the second space coordinate system without the need for complex conversion of the camera coordinate system.

[0135] Through the above Figure 5 and Figure 6 In this implementation method, the environment information of the operating space can be constructed, and the first transformation matrix mapped from the first space coordinate system to the second space coordinate system can be obtained by calibration. For the processor side, the processor can combine the environment information and the first transformation matrix Stored in the memory, waiting to be called in real time during subsequent data collection.

[0136] The following is an explanation of the calibration process of the clamping portion opening and closing width range. Figure 7 As shown, in some embodiments, the process of calibrating the opening and closing width range of the clamping portion includes:

[0137] S710, driving the clamping part of the clamping tool to open and close, and using a camera to collect a third video stream of the opening and closing process of the clamping part.

[0138] S720, tracking and locating the clamping part in the third video stream, and determining the opening and closing width range of the clamping part.

[0139] Combination Figure 1 For the clamping tool shown, the staff can turn on the video recording of the camera 100, then hold the clamping tool and slowly control the closing and opening of the clamping part. The opening and closing process can be repeated many times. After the video recording is finished, the video stream captured by the camera 100 can be obtained. This video stream is the third video stream defined in this specification.

[0140] It is worth noting that, since the opening and closing range of the clamping part needs to be calibrated, when controlling the opening and closing of the clamping part, the staff needs to at least control the clamping part to reach the maximum opening and closing width and the minimum opening and closing width.

[0141] It can be understood that the opening and closing width of the clamping portion actually represents the relative movement distance between the first clamping finger 121 and the second clamping finger 122. Therefore, by calculating the conversion matrix between the first positioning coding map coordinate system R_finger and the second positioning coding map coordinate system L_finger, the width change during the opening and closing process can be determined, and the maximum and minimum width values ​​can be selected from them to obtain the opening and closing width range of the clamping portion [ W min ，W max ].

[0142] Specifically, the landmark detection algorithm can be used to extract the position and posture of the first positioning code map and the second positioning code map on each frame of the third video stream, and detect the motion trajectory of the first positioning code map in the camera coordinate system and the motion trajectory of the second positioning code map in the camera coordinate system. Then, according to the positional relationship between the camera coordinate system (camera coordinate system) and the first positioning code map coordinate system (R_finger coordinate system), the seventh transformation matrix between the camera coordinate system and the first positioning code map coordinate system is obtained. Similarly, according to the positional relationship between the camera coordinate system (camera coordinate system) and the second positioning coding map coordinate system L_finger coordinate system), the eighth transformation matrix between the camera coordinate system and the second positioning coding map coordinate system is obtained. .

[0143] Then, according to the aforementioned homogeneous transformation matrix multiplication principle, the ninth transformation matrix between the first positioning coding map coordinate system and the second positioning coding map coordinate system can be calculated by the following formula (2): , expressed as:

[0144] (2)

[0145] Combined with the above, we can see that the ninth transformation matrix It represents the change of the relative distance between the first clamping finger 121 and the second clamping finger 122, so that the ninth transformation matrix can reflect the change of the trajectory of the opening and closing degree of the clamping part. Find the maximum and minimum values, which represent the maximum value of the opening and closing width of the clamping part in the third video stream. W max and minimum value W min , so that the opening and closing width range of the clamping part can be constructed [ W min ，W max ].

[0146] Through the above Figure 7 Embodiment, can calibrate the opening and closing width range of the clamping part of the clamping part tool [ W min ， W max ], for the processor end, the processor can set the opening and closing width range of the clamping part to [ W min ，W max ] is stored in the memory, waiting to be called in real time during subsequent data acquisition.

[0147] 2) Data collection stage

[0148] During the data collection stage, the staff can hold the aforementioned clamping tool and perform n clamping tasks, where n is an integer greater than 0. The n clamping tasks can be clamping different target objects, or clamping the same or different target objects in different postures, or clamping different parts of the same target object. This manual does not limit the specific task type of the clamping task.

[0149] Taking a clamping task as an example, the staff starts the data collection model on the data collection device. At this time, the camera and various sensors on the clamping tool start working. Then the staff uses the clamping tool to clamp the target object until the clamping task is completed. The camera can collect video data of the clamping task process, and the various sensors can collect various data of the clamping task process.

[0150] Then, the gripping tool can send these data to the control device by wired or wireless means, and the control device completes the data collection for this gripping task by executing the method steps below in this specification. By repeating the gripping task n times, the collected data corresponding to the n gripping tasks can be obtained, and the operation data set of the end of the robot arm can be constructed based on the collected data of these n gripping tasks.

[0151] The following is an example of the data operation process of a clamping task. Figure 8 Embodiment The above-mentioned method of this specification is demonstrated.

[0152] like Figure 8 As shown, in some embodiments, the method for obtaining the operation data of the end of the robot arm exemplified in this specification includes:

[0153] S810. When using a clamping tool to perform a clamping task, obtain a first video stream, first inertia data, clamping force data, force data, and audio data.

[0154] Combination Figure 1The clamping tool shown in the figure can be held by the staff to perform a clamping task. The process of a clamping task is from starting data collection to the end of the clamping task. During this process, the camera 100 can collect video stream data of the process of performing the clamping task, and the video stream data is the first video stream defined in this specification. The inertial sensor on the clamping tool can collect inertial data of the process of performing the clamping task. The inertial data may include, for example, data such as three-axis velocity / acceleration, angular velocity / angular acceleration, etc. The inertial data is the first inertial data defined in this specification. The pressure sensor on the clamping part can collect clamping force data during the clamping process, and the clamping force data represents the change of clamping force during the clamping process. The six-dimensional force sensor on the clamping work can detect the force data of the clamping part during the clamping task, for example, it can include data such as force / torque of the sensor position. The microphone on the clamping tool can collect audio data of the clamping task in real time, and the audio data may include, for example, user voice instructions and ambient sound data.

[0155] S820: Acquire pre-constructed environmental information, perform camera tracking and positioning based on the first video stream and the first inertial data, and determine the target motion trajectory of the center point of the end of the clamping part in a pre-defined second space coordinate system.

[0156] It can be understood that one of the goals of collecting the robot arm operation data is to determine the motion trajectory of the center point of the end of the clamping part in the real world coordinate system (the second space coordinate system). In the implementation of this specification, this motion trajectory is defined as the target motion trajectory. Fig. 9 The method process of calculating the target motion trajectory is explained.

[0157] like Fig. 9 As shown, in some embodiments, the method exemplified in this specification, the process of determining the target motion trajectory includes:

[0158] S821. Load environmental information, perform camera tracking and positioning based on the first video stream and the first inertial data, determine the motion trajectory of the camera in the first space coordinate system in the clamping task, and the fourth transformation matrix between the camera coordinate system and the first space coordinate system.

[0159] In the implementation mode of this specification, the environment information of the operating space (such as three-dimensional map information) has been pre-constructed through the aforementioned calibration stage, so that the environment information pre-stored in the memory can be loaded. Then, the algorithm extracts feature points from each frame of the first video stream, and then matches them with the feature points in the environment information, and at the same time combines the first inertial data to estimate the posture of the camera in each frame of the image, so as to achieve tracking and positioning of the camera and obtain the motion trajectory of the camera in the first space coordinate system.

[0160] As can be seen from the above, the first spatial coordinate system represents the initial position and posture of the first frame image of the first video stream. Therefore, the motion trajectory obtained by tracking and locating the position and posture of the camera relative to the first spatial coordinate system is the motion trajectory of the camera in the first spatial coordinate system relative to the initial posture. Then, according to the motion trajectory of the camera in the first spatial coordinate system, the fourth transformation matrix of the camera coordinate system relative to the first spatial coordinate system can be extracted: .

[0161] It can be understood that the fourth transformation matrix here is The calculation principle is the same as above Figure 6 In the implementation method, the second transformation matrix is ​​calculated The principle is the same as that of the present invention, and those skilled in the art can understand and fully implement it in combination with the foregoing, so this specification will not elaborate on it.

[0162] S822. Obtain a fifth transformation matrix between a pre-calibrated tool coordinate system and a camera coordinate system, and perform a homogeneous transformation based on the fourth transformation matrix, the fifth transformation matrix, and the first transformation matrix to obtain a sixth transformation matrix between the tool coordinate system and the second space coordinate system.

[0163] It can be understood that in order to calculate the target motion trajectory of the end of the clamping part in the second space coordinate system, it is necessary to determine the sixth transformation matrix of the tool coordinate system (Tip coordinate system) relative to the second space coordinate system (base coordinate system): , combined with the idea of ​​the aforementioned homogeneous transformation matrix compound, the sixth transformation matrix It can be expressed as:

[0164] (3)

[0165] In formula (3), Represents the fifth transformation matrix between the tool coordinate system and the camera coordinate system, refer to Figure 4 As shown, it can be understood that the working coordinate system represents the coordinate system of the center point of the clamping part, which has a fixed spatial relationship with the camera coordinate system. Therefore, when the structure of the clamping tool is determined, the fifth transformation matrix can be obtained by pre-calibration through calibration means. , which is the fifth transformation matrix is a known quantity.

[0166] In addition, the fourth transformation matrix of the camera coordinate system relative to the first space coordinate system in this clamping task has been calculated through the aforementioned S821 method step: At the same time, the first transformation matrix of the first space coordinate system relative to the second space coordinate system is It has been obtained in the calibration stage, so the first transformation matrix , the fourth transformation matrix And the fifth transformation matrix Substituting into formula (3), we can obtain the sixth transformation matrix of the tool coordinate system (Tip coordinate system) relative to the second space coordinate system (base coordinate system): .

[0167] It can be understood that the sixth transformation matrix It represents the mapping relationship between the tool coordinate system (Tip coordinate system) and the second space coordinate system (base coordinate system). That is, for any point in the tool coordinate system, the sixth transformation matrix It can be mapped to the corresponding position of the second space coordinate system.

[0168] S823: Perform spatial transformation on the motion trajectory of the camera in the first space coordinate system based on the sixth transformation matrix to obtain the target motion trajectory.

[0169] Based on the idea of ​​the aforementioned space transformation, it can be understood that the motion trajectory of the camera in the first space coordinate system determined in the aforementioned S821 can be first mapped to the Tip coordinate system, and then based on the sixth transformation matrix By mapping the motion trajectory into the second space coordinate system, the motion trajectory of the center point of the end of the clamping part in the second space coordinate system can be obtained, that is, the target motion trajectory.

[0170] S830. Determine the target opening and closing trajectories of the first clamping finger and the second clamping finger according to the first positioning code map in the first video stream and the position change information of the first positioning code map.

[0171] This specification uses the above data to collect the trajectory of the clamping tool, which is more realistic and accurate, and does not require real robotic arm operation to obtain data. In addition, this specification establishes a first spatial coordinate system through the first frame image of the second video stream collected during the calibration phase, that is, the first frame image of the second video stream corresponds to the first spatial coordinate system, and the first spatial coordinate system is used to determine the position and posture of the camera during movement. The second video stream is collected before the first video stream, so when calculating the camera posture in the first video stream, it can be directly calculated with reference to the first spatial coordinate system. Through this first spatial coordinate system, compared with the prior art that calibrates the initial position of the camera multiple times through the first frame of each video stream to perform complex calculations to determine the camera posture, this solution is simpler and more efficient.

[0172] In the implementation mode of this specification, the process of determining the target opening and closing trajectory in S830 is the same as that described above. Figure 7 The idea of ​​the calibration stage shown in the figure is the same, both of which are based on the mark detection algorithm to track and locate the first positioning code map and the second positioning code map, and use the relative position change of the first positioning code map and the second positioning code map to determine the opening and closing trajectory of the clamping part. Fig.10 Provide explanation.

[0173] like Fig.10 As shown, in some embodiments, the data collection method exemplified in this specification, the process of determining the target opening and closing trajectory includes:

[0174] S831. Track and locate the clamping part in the first video stream, and determine the seventh transformation matrix between the camera coordinate system and the first positioning coding map coordinate system, and the eighth transformation matrix between the camera coordinate system and the second positioning coding map coordinate system.

[0175] S832. Perform homogeneous transformation based on the seventh transformation matrix and the eighth transformation matrix to obtain a ninth transformation matrix between the first positioning coding map coordinate system and the second positioning coding map coordinate system.

[0176] S833. Determine the target opening and closing trajectory based on the ninth transformation matrix.

[0177] In the implementation mode of this specification, the landmark detection algorithm can be used to extract the position and posture of the first positioning code map and the second positioning code map on each frame of the first video stream, and detect the motion trajectory of the first positioning code map in the camera coordinate system and the motion trajectory of the second positioning code map in the camera coordinate system. Then, according to the positional relationship between the camera coordinate system (camera coordinate system) and the first positioning code map coordinate system (R_finger coordinate system), the seventh transformation matrix between the camera coordinate system and the first positioning code map coordinate system is obtained. Similarly, according to the positional relationship between the camera coordinate system (camera coordinate system) and the second positioning coding map coordinate system L_finger coordinate system), the eighth transformation matrix between the camera coordinate system and the second positioning coding map coordinate system is obtained. .

[0178] Then, the seventh transformation matrix And the eighth transformation matrix Substituting into the above formula (2), the ninth transformation matrix between the first positioning coding map coordinate system and the second positioning coding map coordinate system can be calculated: .

[0179] It can be understood that the ninth transformation matrix It represents the change in the relative distance between the first clamping finger 121 and the second clamping finger 122, so that the ninth transformation matrix can reflect the change in the trajectory of the opening and closing degree of the clamping part during the clamping task, that is, determine the target opening and closing trajectory.

[0180] S840. Establish operation data corresponding to the clamping task according to the first video stream, the first inertia data, the clamping force data, the force data, the audio data, the target motion trajectory, and the target opening and closing trajectory.

[0181] In the implementation of this specification, after determining the target motion trajectory and the target opening and closing trajectory, the operation data corresponding to this clamping task can be established in combination with the aforementioned collected raw data. That is, the operation data of the clamping task includes:

[0182] The first video stream represents the image video during the execution of the gripping task;

[0183] The first inertia data represents the change of inertia data during the gripping task;

[0184] The clamping force data indicates the change in the clamping force of the clamping part during the clamping task;

[0185] Force data, indicating the change of force and / or torque of the clamping part during the clamping task;

[0186] Audio data, which represents the voice commands or ambient audio collected during the gripping task;

[0187] The target motion trajectory represents the position change of the center point of the end of the gripping part during the gripping task;

[0188] The target opening and closing trajectory represents the change in the opening and closing width of the clamping part during the clamping task.

[0189] In some implementations, the above data can be packaged and stored after being synchronized with the timestamp, and named according to the collection start time. Then the above data collection process is repeated, and the collected data of each clamping task is packaged and stored in sequence, so as to construct an operation data set.

[0190] It is worth noting that in traditional data collection methods, generally only the motion trajectory and opening and closing trajectory of the end of the clamping part are collected, or the original image data and inertial data are further packaged and stored as robot arm operation data. However, through research, it is found that in different robot arm operation scenarios, there are many factors that affect the control of the robot arm. In order to achieve better control effects, it is necessary to build a multimodal network model. However, the data set collected by the traditional method has fewer data types, which is difficult to meet the training requirements of the multimodal network model, resulting in a decrease in model accuracy and poor actual control effect on the robot arm.

[0191] In the method described in the implementation mode of this specification, not only basic data such as video data, inertial data, motion trajectory and opening and closing trajectory are collected, but also the clamping force data, clamping force data during the clamping process, environmental audio data, etc. are further integrated to achieve multi-modal model input, which can enable the network model to learn control strategies for more scenarios and improve the end-to-end robot arm control effect. In addition, the first spatial coordinate system is defined based on the first frame in the second video stream, which makes it easier to determine the camera posture.

[0192] In addition, in the implementation manner of this specification, the data acquisition equipment has a simple structure and low cost. It can realize operation data acquisition without a real robotic arm and remote operation equipment, which is more conducive to large-scale data collection and promotion. At the same time, it is not necessary to require staff to understand professional knowledge such as robotic arm control. The operation threshold is lower and data collection is more efficient. Moreover, by collecting data in real working scenarios, the problem of data in simulation environment being incompatible with the real environment can be avoided, which makes it easier to expand data diversity.

[0193] In some embodiments, when the clamping tool is performing a clamping task, the first video stream captured by the camera may be affected by factors such as lighting and occlusion, resulting in the presence of data that exceeds the opening and closing width range in the target opening and closing trajectory calculated in S830. These erroneous data will interfere with model training and affect model convergence.

[0194] Therefore, in some implementations, based on the aforementioned ninth transformation matrix After determining the first opening and closing trajectory during the gripping task, use the aforementioned Figure 7 The opening and closing width range obtained by pre-calibration during the calibration phase [ W min ， W max ]Correct the erroneous data.

[0195] For example, in one example, based on the maximum and minimum values ​​of the opening and closing width range calibration, the data out of the range in the first opening and closing trajectory can be normalized, linearly interpolated, etc., to constrain the opening and closing width to within the opening and closing width range [ W min ， W max After the first opening and closing trajectory is processed by the above method, the processed opening and closing trajectory is used as the target opening and closing trajectory.

[0196] From the above, it can be seen that in the implementation of this specification, the pre-calibrated opening and closing width range is used to perform data correction on the opening and closing trajectory during the data collection process, thereby improving the data collection accuracy and providing a high-precision data basis for model training.

[0197] In some embodiments, the present specification provides a storage medium storing computer instructions, wherein the computer instructions are used to cause a processor to execute the method of any of the aforementioned embodiments.

[0198] Specifically, Fig.11 A schematic diagram of the structure of a control device 600 suitable for implementing the method of the present specification is shown. Fig.11The control device shown can realize the corresponding functions of the above-mentioned control device and storage medium.

[0199] like Fig.11 As shown, the control device 600 includes a processor 601, which can perform various appropriate actions and processes according to the program stored in the memory 602 or the program loaded from the storage part 608 into the memory 602. In the memory 602, various programs and data required for the operation of the electronic device 600 are also stored. The processor 601 and the memory 602 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0200] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0201] In particular, according to an embodiment of the present specification, the above method process can be implemented as a computer software program. For example, an embodiment of the present specification includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the above method. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 609, and / or installed from a removable medium 611.

[0202] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the embodiments. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. The obvious changes or modifications derived therefrom are still within the scope of protection created by this specification.

Claims

1. A method for acquiring operation data of a robot end through a data acquisition device, characterized in that: The data acquisition device comprises a clamping tool, the clamping tool comprises a clamping portion, the clamping portion comprises a first clamping finger and a second clamping finger, the first clamping finger is provided with a first positioning coding diagram, the second clamping finger is provided with a second positioning coding diagram, and the method comprises: When the clamping tool is used to perform a clamping task, a first video stream, first inertial data, clamping force data, force data and audio data are obtained, wherein the first video stream represents an image video during the clamping task, the first inertial data represents a change in inertial data of the clamping tool during the clamping task, the clamping force data represents a change in the clamping force of the clamping part during the clamping task, the force data represents a change in the force and / or torque of the clamping part during the clamping task, and the audio data represents a voice command or ambient audio collected during the clamping task; Acquire pre-constructed environmental information, and perform camera tracking and positioning based on the first video stream and the first inertial data to determine the target motion trajectory of the center point of the end of the clamping part in a pre-defined second spatial coordinate system, wherein the environmental information includes the operating space environment of the clamping tool, the camera is used to collect the video stream, and the clamping tool moves in the operating space; Determine the target opening and closing trajectory of the first clamping finger and the second clamping finger according to the first positioning code map and the position change information of the first positioning code map in the first video stream, wherein the target opening and closing trajectory represents the change of the opening and closing width of the clamping part during the execution of the clamping task; Acquire operation data corresponding to the clamping task according to the first video stream, the first inertial data, the clamping force data, the force data, the audio data, the target motion trajectory, and the target opening and closing trajectory, wherein the target motion trajectory represents the position change of the center point of the end of the clamping part during the execution of the clamping task; The obtaining of pre-built environment information includes: Driving the clamping tool to move in the operating space, the operating space is provided with a first positioning mark; Acquire a second video stream of the operating space and second inertial data of the clamping tool during movement; Based on the second video stream and the second inertial data, construct the environmental information of the operation space, and determine a first transformation matrix between a first space coordinate system and a second space coordinate system, wherein the first space coordinate system represents a camera posture corresponding to a first frame image in the second video stream, and the second space coordinate system represents a coordinate system of the first positioning mark in the operation space; Acquiring pre-built environmental information, performing camera tracking and positioning based on the first video stream and the first inertial data, and determining a target motion trajectory of the center point of the end of the clamping portion in a pre-defined second spatial coordinate system includes: Loading the environmental information, performing camera tracking and positioning based on the first video stream and the first inertial data, determining a motion trajectory of the camera in the first space coordinate system in the gripping task, and a fourth transformation matrix between the camera coordinate system and the first space coordinate system; Obtaining a fifth transformation matrix between a pre-calibrated tool coordinate system and a camera coordinate system; Performing homogeneous transformation based on the fourth transformation matrix, the fifth transformation matrix and the first transformation matrix to obtain a sixth transformation matrix between the tool coordinate system and the second space coordinate system; The motion trajectory of the camera in the first space coordinate system is spatially transformed based on the sixth transformation matrix to obtain the target motion trajectory.

2. The method according to claim 1, characterized in that: Determining a first transformation matrix between the first space coordinate system and the second space coordinate system comprises: Extract features from the frame images in the second video stream to obtain feature points of each frame image, and estimate the initial pose of the camera based on the second inertial data to obtain an initial pose; Perform feature matching on the feature points of each frame of image with the first frame of image, and perform posture estimation based on inertial data corresponding to each frame of image and the initial posture to determine a second transformation matrix between the first space coordinate system and the camera coordinate system; Tracking and locating the first positioning mark in the second video stream to determine a third transformation matrix between the camera coordinate system and the second space coordinate system; The first transformation matrix is ​​obtained according to the second transformation matrix and the third transformation matrix.

3. The method according to claim 1, characterized in that: Determining target opening and closing trajectories of the first gripping finger and the second gripping finger according to the first positioning code map and position change information of the first positioning code map in the first video stream includes: Tracking and locating the clamping part in the first video stream, determining a seventh transformation matrix between the camera coordinate system and the first positioning code map coordinate system, and an eighth transformation matrix between the camera coordinate system and the second positioning code map coordinate system; Performing homogeneous transformation based on the seventh transformation matrix and the eighth transformation matrix to obtain a ninth transformation matrix between the first positioning coding map coordinate system and the second positioning coding map coordinate system; The target opening and closing trajectory is determined based on the ninth transformation matrix.

4. The method according to claim 3, characterized in that: Also includes: Driving the clamping part of the clamping tool to open and close, and using the camera to collect a third video stream of the opening and closing process of the clamping part; Tracking and locating the clamping portion in the third video stream to determine the opening and closing width range of the clamping portion; The determining the target opening and closing trajectory based on the ninth transformation matrix includes: Determine a first opening and closing trajectory based on a ninth transformation matrix; The first opening and closing trajectory is normalized based on the opening and closing width range to obtain the target opening and closing trajectory.

5. A clamping tool, characterized in that: The clamping tool comprises: The clamping tool body; A clamping portion provided on the body, the clamping portion comprising a first clamping finger and a second clamping finger, the first clamping finger being provided with a first positioning coding pattern, and the second clamping finger being provided with a second positioning coding pattern; A driving mechanism, comprising an operating rod and a transmission member, wherein the operating rod is connected to the clamping portion through the transmission member, and when a user applies a force to the operating rod, the operating rod drives the first clamping finger and the second clamping finger to open and close through the transmission member; A camera, fixedly mounted on the body, wherein the field of view of the camera at least covers the range of movement of the clamping portion; An inertial sensor is provided on the body and is used to detect the inertial data of the movement process of the data acquisition device; a clamping force sensor is provided on the clamping surface of the clamping part and is used to detect the clamping force data when the clamping part clamps the object; a six-dimensional force sensor is provided at the connection position between the operating rod and the body and is used to detect the force data when the clamping part clamps the object; a pickup is provided on the body and is used to pick up audio data during the data acquisition process; The clamping tool is used to perform the method according to any one of claims 1 to 4.

6. The clamping tool according to claim 5, characterized in that: The clamping part is assembled on the body through a linear guide rail, and the operating rod drives the first clamping finger and the second clamping finger to reciprocate on the linear guide rail through the transmission member to realize the opening and closing of the clamping part; The driving mechanism further comprises a reset member, which is arranged between the operating rod and the body. When the user does not apply a force to the operating rod, the reset member drives the operating rod to reset.

7. An electronic device, characterized in that: include: A processor is communicatively connected to a camera of the clamping tool, and the processor executes the method according to any one of claims 1 to 4 according to received data.

8. A storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a processor execute the method according to any one of claims 1 to 4.

Images