Human demonstration guided robot factory operation data acquisition method and system

Abstract

The invention relates to a human demonstration guided robot factory operation data acquisition method and system, and the method comprises the steps: collecting human demonstration data including human motion capture, voice data, sensor data, first visual angle data and third visual angle data; the problem that in the prior art, due to the fact that single-mode demonstration data is used, the anti-interference performance is poor is solved, human motion capturing is screened in detail through the first view angle data and the third view angle data, only effective human motion capturing frames are reserved, then the effective human motion capturing frames are redirected to the robot, the quality of human teaching data is optimized, and the human motion capturing efficiency is improved. The problem that the robot training effect is poor due to the fact that teaching data errors are difficult to control in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to humanoid robot technology, and in particular, to a method and system for collecting factory operation data of a robot guided by human demonstration. Background Art

[0002] Currently, factories use offline programming to control robots, which requires engineers to write codes or manually teach, resulting in low efficiency and difficulty in adapting to complex tasks. With the development of technology, factories are gradually realizing intelligentization, with the key lying in aspects such as improving production quality and automated management. High-quality data is an important part of promoting factory intelligentization and automation. However, current factory operation data mostly relies on engineers' coding or repeated debugging, with low efficiency. A notable solution is for humans to act as teachers for demonstration, and robots reduce data debugging by imitating human movement skills and strengthening them, thereby improving the efficiency of factory operation tasks. Factory operations have a certain degree of complexity and dexterity, so it is crucial for how to efficiently conduct human demonstrations. For example, Chinese Patent Application "CN111890357A" discloses a method for an intelligent robot to grasp based on action demonstration teaching. Through this method, using meta-learning algorithms and deep video processing technology, although it solves the problem of dexterity in grasping unknown objects in the prior art, it relies on single-modal human teaching data for robot training, has weak anti-interference ability, and the cost of clear data is high, which is not conducive to robot cross-task migration and skill generalization.

[0003] Therefore, it is a technical problem to be solved to provide a method for processing human teaching data that guides robots to have higher dexterity and stronger skill generalization. Summary of the Invention

[0004] The purpose of the present invention is to overcome the above-mentioned defects existing in the prior art and provide a method and system for collecting factory operation data of a robot guided by human demonstration. By means of human demonstration and operation intention reasoning, robot operation data is collected, and data is collected from different perspectives to solve problems such as weak anti-interference ability of single-modal data and data errors, and improve the ability of scene semantic understanding and cross-task migration, aiming to improve factory intelligent automation.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] According to the first aspect of the present invention, there is provided a method for collecting factory operation data of a robot guided by human demonstration, the method comprising:

[0007] S1. Collect human demonstration data and a complete operation video for a robot task. The human demonstration data includes human motion capture, voice data, sensor data, first-person perspective data, and third-person perspective data. Generate a semantic parsing instruction based on the voice data;

[0008] S2. Obtain the eye movement focus coordinates and combine them with the semantic parsing instruction to generate semantic tags that match the operation actions based on the operation video;

[0009] S3. After performing spatio-temporal alignment on the human demonstration data, filter the valid frames of the human motion capture based on the voice data, first-person perspective data, and third-person perspective data. Convert the valid frames to the SMPL format, and redirect the SMPL-format valid frames to the robot. Obtain the smooth trajectory of the robot performing the operation actions and its corresponding robot poses based on the redirection result;

[0010] S4. Obtain the logical relationships and key data between operation actions based on the human demonstration data, and obtain the safety parameters for the robot task corresponding to the human demonstration data. Construct a robot control instruction based on the logical relationships, key data, safety parameters, smooth trajectory, robot poses, and sensor data;

[0011] S5. Obtain the robot sensor data when the robot executes the control instruction. If the difference between the robot sensor data in the current frame and the robot sensor data in the previous frame is greater than the error threshold or the robot sensor data is a null value, return to S1. Otherwise, align the control instruction and the semantic tags based on the operation actions to obtain the robot factory operation data.

[0012] As a preferred technical solution, the method for generating the semantic parsing instruction includes:

[0013] Use semantic conversion technology to convert the voice data into text data, and its expression is:

[0014]

[0015] where T represents the text data; X represents the voice data;

[0016] Generate a semantic parsing instruction based on the text data. The semantic parsing instruction includes operation actions, objects corresponding to the operation actions, and operation action targets.

[0017] As a preferred technical solution, the redirection includes:

[0018] Construct a unified coordinate reference system for the motion capture coordinate system and the robot coordinate system, and calculate the homogeneous transformation matrix from the motion capture coordinate system to the robot coordinate system for the valid frames and the fixed offset from the tool to the marker position;

[0019] Obtain the tool coordinate system of the tool, align the tool coordinate system with the robot coordinate system, and obtain the pose of the tool in the robot coordinate system. Its expression is:

[0020]

[0021] where represents the fixed offset matrix from the human (world) coordinate system to the robot coordinate system; represents the homogeneous transformation matrix; represents the fixed offset matrix from the tool to the motion capture marker; represents the position of the tool in the robot coordinate system; is the fixed offset; represents the position of the tool in the human (world) coordinate system;

[0022] Based on the minimum error between the pose of the tool in the robot coordinate system and the effective frame as the objective function, solve the robot joint angles based on IK;

[0023] Calculate the geometric Jacobian matrix based on the robot joint angles, and update the robot joint angles based on the geometric Jacobian matrix;

[0024] Compensate for the load difference between the effective frame and the robot using dynamic compensation based on the updated robot joint angles.

[0025] As a preferred technical solution, the expression of the objective function is:

[0026]

[0027] where x human represents the effective frame of human motion capture; f k (q) represents the forward kinematic function of the robot; q represents the robot joint angles; q neutral represents the joint coordinates of a neutral-gender human; represents the derivative of the joint angle with respect to time i.e., the joint velocity.

[0028] As a preferred technical solution, the method for updating the robot joint angles based on the geometric Jacobian matrix includes:

[0029] Calculate the geometric Jacobian matrix, and its expression is: where represents a six-dimensional spatial twist; x represents the end effector pose;

[0030] Use nonlinear optimization to iteratively update the robot joint angles, and its expression is: q′ = JT (JJ T +ζ 2 I) -1 e, where q′ is the updated robot joint angle; J represents the geometric Jacobian matrix; ζ represents the damping coefficient; and I represents the identity matrix.

[0031] As a preferred technical solution, the method for compensating the load difference includes:

[0032] Calculate the gravity compensation torque, and its expression is:

[0033]

[0034] Where, represents the gravity compensation torque; J n represents the Jacobian matrix of the centroid of the nth link of the robot; q′ represents the updated joint angle; m n represents the equivalent mass of the nth link; g represents the acceleration due to gravity; R n represents the rotation matrix from the local coordinate system of the nth link to the robot coordinate system; and z0 represents the unit vector of the Z-axis of the robot coordinate system.

[0035] As a preferred technical solution, obtaining the smooth trajectory of the robot performing the operation action and its corresponding robot pose includes:

[0036] Optimize the redirected result based on the sensor data, and the optimization includes time synchronization, spatial calibration, and data fusion.

[0037] Among them, the time synchronization is: Align the valid frames to the unified time axis through multiple spline interpolations, and its expression is: Where D s (t ′ ) represents the sensor data synthesized by interpolation at the unified time axis t ′ ; D s (t i ) represents the sensor data at time t[[ID=4⑨]] i ; and R i (t ′ ) represents the basis function of the multiple spline interpolation.

[0038] As a preferred technical solution, the method for the spatial calibration is:

[0039] Solve the transformation matrix from the sensor coordinate to the robot coordinate through hand-eye calibration, and its expression is:

[0040]

[0041] Where; Represents the transformation matrix from the sensor coordinate system to the robot coordinate system; Respectively represent the coordinates of the i-th calibration point in the sensor coordinate system and the robot coordinate system; Represents the optimal transformation matrix from the motion capture camera sensor coordinate system to the robot coordinate system; argmin represents minimizing the coordinate conversion error of the calibration points in the two coordinate systems; M represents the number of motion capture reflective spheres, and is not less than three;

[0042] Optimize the redirected result after time synchronization based on the transformation matrix.

[0043] As a preferred technical solution, the method for data fusion includes:

[0044] Allocate the weight of the data fusion through the sensor confidence, and the expression is: where c s Represents the sensor confidence; D s Represents the original value collected by the current sensor;

[0045] Fuse the human teaching data based on the weight to obtain the fused data, and based on the fused data and the redirected result, use zero-phase filtering to eliminate the phase delay, and its expression is:

[0046]

[0047] where q f (t) represents the robot joint pose at time t after filtering; W represents the sum of Gaussian kernel weights, that is, the normalization factor; w(τ) represents the Gaussian kernel weight; τ represents the time interval; σ represents the standard deviation for controlling the smoothing intensity; a k Represents the boundary condition; k represents the polynomial order, and a fifth-degree polynomial can ensure the continuity of the second derivative of the trajectory;

[0048] After eliminating the phase delay, obtain a smooth trajectory using a multi-degree spline curve based on the boundary condition, and obtain the corresponding robot pose.

[0049] According to the second aspect of the present invention, a human demonstration-guided machine factory operation data acquisition system is provided. The system is used to implement the above method and includes:

[0050] Data acquisition module: Collect data through a variety of sensors. The sensors include an eye tracker, a tactile or force feedback glove, and an environmental sensor. The data includes human demonstration data and robot sensor data;

[0051] Data processing module: Used for data processing. The data processing includes semantic parsing instruction generation, semantic label generation, and redirection;

[0052] Instruction generation module: used to generate control instructions based on the data processing results of the data processing module;

[0053] Feedback module: generates a feedback signal based on the human demonstration data and the robot sensor data;

[0054] After the feedback module generates a feedback signal, the system drives the data acquisition module to perform re-acquisition.

[0055] Compared with the prior art, the present invention proposes a human demonstration-guided machine factory operation data acquisition system. The eye movement data obtained by the eye tracker provides the first perspective data, and the depth map obtained by the depth camera provides the third perspective data. Based on the first perspective data and the third perspective data, the obtained human motion capture data is preliminarily screened to obtain effective frames to enhance the quality of the acquired data. The effective frames are redirected to the robot, and the redirection result is optimized based on the multi-modal fusion data. Finally, control instructions are generated based on the optimized redirection result and the collected multi-source human teaching data, and the control instructions and semantic tags are aligned to obtain the machine factory operation data. It not only solves the problems of weak anti-interference ability of data and difficulty in controlling the quality of human demonstration data caused by relying on single-modal data in the prior art, but also solves the problem of morphological difference mapping of existing public data sets, makes up for the limitation of scarce factory operation data, and improves the robot's scene semantic understanding and cross-task migration ability as well as the robot's skill learning efficiency. Description of the Drawings

[0056] Figure 1 It is the flowchart of the method of the present invention;

[0057] Figure 2 It is the schematic diagram of human motion capture of the present invention. Detailed Embodiments

[0058] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0059] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the ordinary meanings understood by those with ordinary skills in the technical field to which this application belongs. The words such as "a", "one", "kind", "the" and the like involved in this application do not indicate a quantity limitation and may represent a singular or plural number. The terms "comprising", "including", "having" and any variations thereof involved in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or units, but may further include unlisted steps or units, or may further include other steps or units inherent to these processes, methods, products or devices. The similar words such as "connected", "linked", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the front and rear associated objects. The terms "first", "second", "third" and the like involved in this application are only used to distinguish similar objects and do not represent a specific sorting of the objects.

[0060] The method based on imitation learning has achieved remarkable results in the field of robot skill learning and has been physically verified on various robotic arms. Among them, the key to imitation lies in how to make each joint of the robot accurately track human movements, which is exactly what this invention focuses on. For this reason, this invention provides a method for collecting operation data of a human demonstration-guided robot factory, and its process is as Figure 1 shown, including:

[0061] S1. Collect human demonstration data and a complete operation video under a robot task, and generate semantic parsing instructions based on the human demonstration data.

[0062] Specifically, arrange the factory operation scenario, including an operation table, parts, an optical motion capture camera and sensors, for collecting human demonstration data. The sensors at least include a motion capture camera, an environmental sensor, an RGB-D camera, and a tactile or force sensor. Therefore, the human demonstration data collected by it at least includes: human motion capture, voice data, sensor data, first-person perspective data, and third-person perspective data. Specifically, it is the joint coordinates, pitch-roll-yaw angles of the joints, angular acceleration, grasping force, etc. in three-dimensional directions.

[0063] Use semantic conversion technology to convert voice data into text data, and its expression is:

[0064]

[0065] Among them, T represents text data; X represents speech data;

[0066] Generate semantic parsing instructions based on the text data. The semantic parsing instructions include an operation action, an object corresponding to the operation action, and an operation action target. For example, Y = {action, object, target}. Assume that the speech data collected at this time is "Align part A with the slot of part B", then the corresponding semantic parsing instruction for this period is: Y = {align, part A, the slot of part B}.

[0067] S2. Obtain the eye movement focus coordinates and combine them with the semantic parsing instructions to generate semantic labels that match the operation actions based on the operation video. These semantic labels are used for training datasets such as robot operation action generation training and task planning.

[0068] S3. After performing spatio-temporal alignment on the human demonstration data, filter the valid frames of human motion capture based on the speech data, first-person perspective data, and third-person perspective data. Convert the valid frames to the SMPL format, and redirect the SMPL-format valid frames to the robot. Based on the redirection result, obtain the smooth trajectory of the robot executing the operation action and its corresponding robot pose.

[0069] If the spatio-temporal alignment is not performed, it may occur that at time t i the result of the motion capture at the moment is that the human posture is bending down 45° while holding a part, but in the third-person perspective data, the current moment is in an upright state, resulting in data fusion errors. Therefore, it is necessary to perform spatio-temporal alignment on the human demonstration data to obtain results as shown in Figure 2 to ensure the data fusion accuracy.

[0070] S31. Redirection:

[0071] S311. Construct a unified coordinate reference system for the motion capture coordinate system and the robot coordinate system, and calculate the homogeneous transformation matrix from the motion capture coordinate system to the robot coordinate system and the fixed offset from the tool to the marker position.

[0072] S312. Obtain the tool coordinate system of the tool, align the tool coordinate system with the robot coordinate system, and obtain the pose of the tool in the robot coordinate system. Its expression is:

[0073]

[0074] Among them, represents the fixed offset matrix from the human (world) coordinate system to the robot coordinate system; represents the homogeneous transformation matrix; represents the fixed offset matrix from the tool to the motion capture marker; represents the position of the tool in the robot coordinate system; That is the fixed offset; Indicates the position of the tool in the human (world) coordinate system.

[0075] S313. Use the minimum error between the pose of the tool in the robot coordinate system and the effective frame as the objective function, and solve for the robot joint angles based on IK.

[0076] Specifically, through kinematic modeling and degree-of-freedom mapping, solve for the robot joint angles q based on IK, and make the pose x of the robot end effector robot approach the human demonstration pose x human , that is, its objective function expression is:

[0077]

[0078] where x human represents the effective frame of human motion capture; f k (q) represents the forward kinematic function of the robot; q represents the robot joint angles; q neutral represents the human joint coordinates of neutral gender; represents the derivative of the joint angle with respect to time that is, the joint velocity

[0079] S314. Calculate the geometric Jacobian matrix based on the robot joint angles, and update the robot joint angles based on the geometric Jacobian matrix.

[0080] In this embodiment, the damping least squares method is used to optimize and calculate the geometric Jacobian matrix, and its expression is:

[0081]

[0082] where represents a six-dimensional spatial twist; x represents the end effector attitude;

[0083] Use nonlinear optimization to iteratively update the robot joint angles, and its expression is: q′ = J T (JJ T + ζ 2 I) -1 e, where q′ is the updated robot joint angle; J represents the geometric Jacobian matrix; ζ represents the damping coefficient to avoid matrix singularity; I represents the identity matrix.

[0084] S315. Based on the updated robot joint angles, use dynamic compensation to compensate for the load difference between the effective frame and the robot.

[0085] Calculate the gravity compensation torque, and its expression is:

[0086]

[0087] Among them, represents the gravity compensation moment; J n represents the Jacobian matrix of the centroid of the nth link of the robot; q′ represents the updated joint angle; m m represents the equivalent mass of the nth link; g represents the acceleration due to gravity; R n represents the rotation matrix from the local coordinate system of the nth link to the robot coordinate system; z0 represents the unit vector of the Z-axis of the robot coordinate system.

[0088] S32. Obtain the smooth trajectory of the robot performing the operation action and its corresponding robot pose:

[0089] Optimize the redirected result based on the sensor data. The optimization includes time synchronization, spatial calibration, and data fusion. Among them,

[0090] The time synchronization is as follows:

[0091] Align the valid frames to the unified time axis through multiple-spline interpolation. Its expression is: Among them, D s (t ′ ) represents the sensor data synthesized by interpolation at the unified time axis t ′ ; D s (t i ) represents the sensor data at time t i ; R i (t ′ ) represents the basis function of the multiple-spline interpolation.

[0092] The method of spatial calibration is as follows:

[0093] Solve the transformation matrix from the sensor coordinate system to the robot coordinate system through hand-eye calibration. Its expression is:

[0094]

[0095] Among them; represents the transformation matrix from the sensor coordinate system to the robot coordinate system; respectively represent the coordinates of the ith calibration point in the sensor coordinate system and the robot coordinate system; represents the optimal transformation matrix from the motion capture camera sensor coordinate system to the robot coordinate system; argmin represents minimizing the coordinate transformation error of the calibration points in the two coordinate systems; M represents the number of motion capture reflective spheres, and it is not less than three because three can form a rigid body;

[0096] Optimize the redirected result after time synchronization based on the transformation matrix.

[0097] The methods of data fusion include:

[0098] The weights of data fusion are assigned based on the sensor confidence, and the expression is: where c s represents the sensor confidence, and represents the sensor noise variance, t delay represents the transmission delay; D s represents the raw value collected by the current sensor;

[0099] The weights of the data fusion are assigned based on the sensor confidence, and the expression is: where c s represents the sensor confidence; D s represents the raw value collected by the current sensor;

[0100] Based on the weights, the human demonstration data are fused to obtain fused data, and based on the fused data and the redirection result, zero-phase filtering is used to eliminate the phase delay, and its expression is:

[0101]

[0102] where q f (t) represents the robot joint pose at time t after filtering; W represents the sum of Gaussian kernel weights, that is, the normalization factor; w(τ) represents the Gaussian kernel weight; τ represents the time interval; σ represents the standard deviation of the control smoothing intensity; a k represents the boundary condition; k represents the polynomial order, and a fifth-order polynomial can ensure the continuity of the second derivative of the trajectory;

[0103] After eliminating the delay, according to the boundary conditions: starting point position, speed or ending point position, speed, etc., a fifth-order spline curve is used to generate a smooth trajectory and obtain the corresponding robot pose, so as to ensure the continuity of the robot pose and speed, etc.

[0104] S4. Based on the human demonstration data, obtain the logical relationship and key data between operation actions, and obtain the safety parameters of the robot task corresponding to the human demonstration data. Based on the logical relationship, key data, safety parameters, smooth trajectory, robot pose and sensor data, construct robot control instructions.

[0105] Specifically, the control instructions at least include: action control instructions, sensor feedback instructions, task logic instructions and fault tolerance instructions. Among them, the action instructions are constructed depending on the smooth trajectory, robot pose and key data, and mainly include the target position and attitude, such as the three-dimensional coordinates and Euler angles of the end effector.

[0106] The sensor feedback instruction is constructed based on sensor data and is used to dynamically correct the robot's actions based on data from environmental sensors. It can include force control parameters for operating actions, such as stiffness, damping, and inertia; motion trajectory parameters, such as trajectory interpolation methods and speeds, and the thresholds are adjusted in real time based on these parameters. Also, when constructing the sensor feedback instruction based on sensor data, spatio-temporal alignment needs to be performed again to confirm that the data is correct.

[0107] The task logic instruction is constructed based on logical relationships, smooth trajectories, and the robot's pose. It mainly includes the operation step decomposition of splitting human demonstration actions into programmable atomic operation sequences and the conditional branch judgment of dynamically selecting operation branches according to the environment. In this embodiment, the atomic operation sequence is mainly basic operation behaviors, including placing, grasping, and moving, etc.; the conditional branch judgment mainly refers to the judgment of the completed operation steps and the current operation steps. For example, if the current task goal is to place the tool scissors into the toolbox, the atomic operation sequence is ""step1":"Pick(scissor)","step2":"Pick(tool box)","step3":Move(move the scissors to the front of the tool box)","step4":"Place(place scissors in the toolbox)"" and the current step is step4. It is necessary to judge whether step1 / step2 / step3 are correctly completed. If step1 is not correctly completed (such as misselecting the tool screwdriver), step1 needs to be re-executed and the following steps need to be executed.

[0108] The detailed encoding of the atomic operation sequence is shown in Table 1:

[0109] Table 1 Atomic operation sequence code

[0110]

[0111]

[0112] The fault tolerance instruction is constructed based on safety parameters and involves boundary constraints of parameters such as the robot's workspace limit, joint protection, and current overload threshold. It also includes the processing protocol for emergency responses when collisions, communication interruptions, or sensor failures occur.

[0113] Taking the PCB_Assembly_001 task as an example, the encoding of its corresponding control instruction is shown in Table 2, including the atomic operation sequence, joint pose, and trajectory of the current task.

[0114]

[0115]

[0116]

[0117]

[0118]

[0119] S5. Obtain the robot sensor data for the robot to execute the control instruction. If the difference between the robot sensor data in the current frame and the robot sensor data in the previous frame is greater than the error threshold or the robot sensor data is a null value, then return to S1. Otherwise, align the control instruction and the semantic label based on the operation action to obtain the robot factory operation data.

[0120] The present invention also provides a human demonstration-guided robot factory operation data acquisition system for implementing the above method, including:

[0121] Data acquisition module: Collect data through a variety of sensors. The sensors include an eye tracker, a tactile or force feedback glove, and an environmental sensor. The data includes human demonstration data and robot sensor data;

[0122] Data processing module: Used for data processing. The data processing includes semantic parsing instruction generation, semantic label generation, and redirection;

[0123] Instruction generation module: Used to generate a control instruction based on the data processing result of the data processing module;

[0124] Feedback module: Generate a feedback signal based on the human demonstration data and the robot sensor data;

[0125] When the feedback module generates a feedback signal, the system drives the data acquisition module to perform re-acquisition.

[0126] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the foregoing method embodiment and will not be elaborated herein.

[0127] In addition, the present invention also provides an electronic device, including a central processing unit (CPU), which can execute various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or computer program instructions loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus.

[0128] Multiple components in the device are connected to the I / O interface, including: an input unit, such as a keyboard, a mouse, etc.; an output unit, such as various types of displays, speakers, etc.; a storage unit, such as a disk, an optical disc, etc.; and a communication unit, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit allows the device to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0129] The processing unit executes the various methods and processes described above, such as methods S1 - S5. For example, in some embodiments, methods S1 - S5 may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device via the ROM and / or the communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 - S5 described above can be executed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 - S5 by any other suitable means (e.g., by means of firmware).

[0130] The functions described above herein can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGA), Application Specific Integrated Circuits (ASIC), Application Specific Standard Products (ASSP), System on Chip (SOC), Complex Programmable Logic Devices (CPLD), and so on.

[0131] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing devices, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0132] In the context of the present invention, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] As described above, the foregoing are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily conceive of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for collecting human demonstration-guided industrial robot operation data, characterized in that The method described above includes: S1. Collect human demonstration data and a complete operation video for a robot task. The human demonstration data includes human motion capture, voice data, sensor data, first-person perspective data, and third-person perspective data. Generate a semantic parsing instruction based on the voice data. S2. Obtain the eye movement focus coordinates and combine them with the semantic parsing instruction. Generate a semantic label that matches the operation action based on the operation video. S3. After performing spatio-temporal alignment on the human demonstration data, filter the valid frames of the human motion capture based on the voice data, first-person perspective data, and third-person perspective data. Convert the valid frames into the SMPL format, and redirect the valid frames in the SMPL format to the robot. Obtain the smooth trajectory of the robot performing the operation action and its corresponding robot pose based on the redirection result. S4. Obtain the logical relationship and key data between operation actions based on the human demonstration data, and obtain the safety parameters for the robot task corresponding to the human demonstration data. Construct a robot control instruction based on the logical relationship, key data, safety parameters, smooth trajectory, robot pose, and sensor data. S5. Obtain the robot sensor data when the robot executes the control instruction. If the difference between the robot sensor data in the current frame and the robot sensor data in the previous frame is greater than the error threshold or the robot sensor data is a null value, return to S1. Otherwise, align the control instruction and the semantic label based on the operation action to obtain the machine factory operation data.

2. The method for collecting human demonstration-guided robot factory operation data according to claim 1, wherein The method for generating the semantic parsing instruction includes: Use semantic conversion technology to convert the voice data into text data, and its expression is: where, T represents the text data; X represents the voice data. Generate a semantic parsing instruction based on the text data. The semantic parsing instruction includes an operation action, an object corresponding to the operation action, and an operation action target.

3. A method for collecting human demonstration-guided robot factory operation data according to claim 1, characterized in that, The redirection described above includes: Construct a unified coordinate reference system for the motion capture coordinate system and the robot coordinate system, and calculate the homogeneous transformation matrix from the motion capture coordinate system to the robot coordinate system for the valid frames and the fixed offset from the tool to the marker position. Obtain the tool coordinate system of the tool, align the tool coordinate system with the robot coordinate system, and obtain the pose of the tool in the robot coordinate system. Its expression is: Among them, represents the fixed offset matrix from the world coordinate system to the robot coordinate system; represents the homogeneous transformation matrix; represents the fixed offset matrix from the tool to the motion capture marker; represents the position of the tool in the robot coordinate system; represents the fixed offset from the human coordinate system to the robot coordinate system; represents the position of the tool in the world coordinate system; Use the minimum error between the pose of the tool in the robot coordinate system and the valid frames as the objective function, and solve for the robot joint angles based on IK. Calculate the geometric Jacobian matrix based on the robot joint angles, and update the robot joint angles based on the geometric Jacobian matrix. Compensate for the load difference between the valid frames and the robot based on the updated robot joint angles using dynamic compensation theory.

4. A method for collecting human demonstration-guided robot factory operation data according to claim 3, characterized in that, The expression of the objective function is: Among them, x human represents the valid frames of human motion capture; f k (q) represents the forward kinematic function of the robot; q represents the joint angles of the robot; q neutral represents the human joint coordinates of the neutral gender; represents the derivative of the joint angle with respect to time i.e., the joint velocity.

5. A method for collecting human demonstration-guided industrial robot operation data according to claim 3, characterized in that The method for updating the robot joint angles based on the geometric Jacobian matrix includes: Calculate the geometric Jacobian matrix, and its expression is: where represents a six-dimensional space spinor; x represents the pose of the end effector; The robot joint angles are updated iteratively using non - linear optimization, and its expression is: q′ = J T (JJ T + ζ 2 I) -1 e, where q′ represents the updated robot joint angles; J represents the geometric Jacobian matrix; ζ represents the damping coefficient; and I represents the identity matrix.

6. A method for collecting human demonstration-guided robot factory operation data according to claim 3, characterized in that The method for compensating for the load difference includes: Calculate the gravity compensation torque, and its expression is: Among them, represents the gravity compensation moment; J n represents the Jacobian matrix of the centroid of the nth link of the robot; q′ represents the updated joint angle; m n represents the equivalent mass of the nth link; g represents the acceleration due to gravity; R n represents the rotation matrix from the local coordinate system of the nth link to the robot coordinate system; z0 represents the unit vector of the Z-axis of the robot coordinate system.

7. A method for collecting human demonstration-guided industrial robot operation data according to claim 1, characterized in that, Obtaining the smooth trajectory of the robot performing the operation action and its corresponding robot pose includes: Optimize the redirected result based on the sensor data described above. The optimization includes time synchronization, spatial calibration, and data fusion. Among them, the time synchronization is: aligning the valid frames to a unified time axis through multiple spline interpolations, and its expression is: Among them, D s (t′) represents the sensor data after interpolation synthesis at the unified time axis t′; D s (t i ) represents the sensor data at time t i ; R i (t′) represents the basis function of multiple spline interpolations.

8. A method for collecting human demonstration-guided industrial robot operation data according to claim 7, characterized in that, The method of the spatial calibration is as follows: Solve the transformation matrix from the sensor coordinate to the robot coordinate through hand-eye calibration. Its expression is: Wherein; represents the transformation matrix from the sensor coordinate system to the robot coordinate system; respectively represent the coordinates of the i-th calibration point in the sensor coordinate system and the robot coordinate system; represents the optimal transformation matrix from the motion capture camera sensor coordinate system to the robot coordinate system; argmin represents minimizing the coordinate conversion error of the calibration points in the two coordinate systems; M represents the number of motion capture reflective spheres, and is not less than three; Optimize the redirected result after time synchronization based on the transformation matrix.

9. A method for collecting human demonstration-guided industrial robot operation data according to claim 7, characterized in that, The method of the data fusion includes: Assign the weight of the data fusion through the sensor confidence, and the expression is: where c s represents the sensor confidence; D s represents the original value collected by the current sensor; Fuse the human teaching data based on the weights to obtain the fused data, and based on the fused data and the redirected result, use zero-phase filtering to eliminate the phase delay. Its expression is: where q f (t) represents the robot joint pose at time t after filtering; W represents the sum of Gaussian kernel weights, that is, the normalization factor; w(τ) represents the Gaussian kernel weight; τ represents the time interval; σ represents the standard deviation of the control smoothing intensity; a k represents the boundary condition; k represents the polynomial order; After eliminating the phase delay, obtain the smooth trajectory and the robot pose by using the multi-spline curve based on the boundary conditions.

10. A human demonstration-guided data acquisition system for robot factory operations, characterized in that, The system is used to implement the method described in claims 1 to 9, and includes: Data acquisition module: Collect data through a variety of sensors. The sensors include an eye tracker, a tactile or force feedback glove, and an environmental sensor. The data includes human demonstration data and robot sensor data. Data processing module: Used for data processing. The data processing includes semantic parsing instruction generation, semantic label generation, and redirection. Instruction generation module: Used to generate control instructions based on the data processing results of the data processing module. Feedback module: Generate a feedback signal based on the human demonstration data and the robot sensor data. After the feedback module generates a feedback signal, the system drives the data acquisition module to perform re-acquisition.

Images