A robot collaborative assembly method based on human hand spatial posture intention perception

Abstract

The present invention relates to the field of robotics and provides a method for collaborative robot assembly based on the perception of human hand spatial posture intentions, comprising: constructing a customized gesture recognition model for tool-holding gestures; identifying a human hand in a target image in real time and extracting the three-dimensional coordinate information of the hand's key points in real time; calculating the hand's spatial rotation angle using a gesture algorithm based on the three-dimensional coordinate information of the hand's key points; mapping the rotation angle data to the range of motion parameters required for controlling a humanoid wrist joint robot; integrating the motion parameters with the gesture information to generate control instructions, and controlling the robot's movements based on the control instructions. The present invention calculates the hand's spatial rotation angle in real time, and based on this spatial rotation angle, the humanoid hand wrist joint robot can recognize the angle information and dynamically respond to the operator's hand posture, thereby realizing a continuous hand-following function and achieving human-machine collaborative assembly.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a robot collaborative assembly method based on human hand spatial posture intention perception. Background Art

[0002] As industrial manufacturing develops towards diversification and intelligence, collaborative assembly between robots and humans faces the dual challenges of understanding operator intentions and improving collaborative efficiency.

[0003] Although the current mainstream vision-guided assembly technology has made significant progress in workpiece identification and positioning, the existing methods still have obvious limitations: on the one hand, the technical path mainly focuses on the collection and analysis of workpiece information, lacks active perception and understanding of the operator's intentions, resulting in the operator having to passively adapt to the robot's preset processes and unable to capture the assembly strategy adjustment intentions in real time, resulting in low efficiency of human-computer interaction; on the other hand, posture perception solutions generally rely on multi-sensor fusion architecture. This technical path not only brings high equipment deployment costs and complex system integration problems, but is also susceptible to environmental interference under specific working conditions, seriously affecting system stability.

[0004] Therefore, a robot collaborative assembly method that can integrate intention perception is needed to improve the performance and stability of the human-robot collaborative assembly process. Summary of the Invention

[0005] In order to improve the performance and stability of the human-machine collaborative assembly process, the present invention provides a robot collaborative assembly method based on human hand spatial posture intention perception. The specific technical solution is as follows:

[0006] A robot collaborative assembly method based on human hand spatial posture intention perception includes:

[0007] Build a customized gesture recognition model for tool holding gestures to identify specific assembly gestures;

[0008] Acquire the target image, identify the human hand in the target image in real time, and extract the three-dimensional coordinate information of the key points of the hand in real time;

[0009] Based on the three-dimensional coordinate information of the key points of the hand, the spatial rotation angle of the hand is calculated through the posture algorithm, and the rotational spatial angle is decomposed into rotation components around the X-axis, Y-axis and Z-axis;

[0010] Mapping the rotation angle data to the motion parameter range required for humanoid wrist joint robot control;

[0011] The motion parameters and gesture information are integrated to generate control instructions, and the robot movements are controlled based on the control instructions to achieve real-time collaborative assembly of human hands and humanoid wrist joint robots.

[0012] The robot collaborative assembly method based on the spatial posture intention perception of the human hand calculates the spatial rotation angle of the hand in real time. Based on the spatial rotation angle, the humanoid hand wrist joint robot can recognize the angle information and dynamically respond to the operator's hand posture, thereby realizing continuous hand following function and achieving human-machine collaborative assembly.

[0013] Preferably, the specific method of mapping the rotation angle data to the motion parameter range required for controlling the humanoid wrist joint robot includes:

[0014] Normalize the rotation angle data;

[0015] The normalized rotation angle data is mapped to the motion parameter range required for the control of the humanoid wrist joint robot through a linear fitting algorithm.

[0016] Preferably, the specific method of controlling the robot motion based on the control instruction includes:

[0017] Transmit control instructions to the motion control card;

[0018] The motion control card transmits the control instructions to the servo drive;

[0019] The servo drive controls the servo motor to perform motion according to control instructions.

[0020] Preferably, the specific method for calculating the spatial rotation angle of the hand includes:

[0021] Extract the wrist key points, index finger tip key points and little finger tip key points from the key nodes of the hand;

[0022] Calculate the vector from the index finger tip key point to the wrist key point and the vector from the pinky fingertip keypoint to the wrist keypoint ;

[0023] By vector and vector Calculate the cross product of the hand's spatial rotation axis and normalize it;

[0024] By vector and vector Calculate the hand's rotation angle in space.

[0025] Preferably, the specific method of decomposing the rotational space angle into rotational components about the X-axis, the Y-axis, and the Z-axis includes:

[0026] According to the unitized spatial rotation axis and spatial rotation angle Calculate the spatial rotation matrix;

[0027] According to the spatial rotation matrix Calculate the actual rotation Euler angle of each axis;

[0028] Among them, the rotation Euler angle includes the Yaw angle 、Pitch angle and Roll Corner .

[0029] Preferably, by the formula Calculate the hand's rotation angle in space.

[0030] Preferably, according to the formula ;in, Represents the identity matrix.

[0031] Preferably, the robot collaborative assembly method further comprises:

[0032] When the system is in a singularity state, the Roll angle is fixed , only solve the Yaw angle and Pitch Angle ;

[0033] According to the Yaw angle 、Pitch angle and Roll Corner Drive the humanoid wrist joint robot to the corresponding target posture.

[0034] Preferably, the specific method of constructing a customized gesture recognition model for tool holding gestures includes:

[0035] For three types of tools, screwdrivers, hammers, and wrenches, corresponding typical gripping gestures are defined respectively;

[0036] Construct tool-gesture mapping relationships and analyze the operator's action intentions in real time based on the tool-gesture mapping relationships.

[0037] Preferably, the vector from the key point of the index finger tip to the key point of the wrist is calculated and the vector from the pinky fingertip keypoint to the wrist keypoint The specific methods include:

[0038] Get wrist key points separately Confidence , key points on the tip of the index finger Confidence And the key point of the little finger Confidence ;

[0039] According to the formula Get Vector , according to the formula Get Vector . BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention can be further understood from the following description in conjunction with the accompanying drawings. The components in the figures are not necessarily drawn to scale, but rather the emphasis is placed on illustrating the principles of the embodiments. In different views, the same reference numerals designate corresponding parts.

[0041] Figure 1 This is a schematic diagram of the overall process of a robot collaborative assembly method based on human hand spatial posture intention perception in one embodiment of the present invention;

[0042] Figure 2 is a flowchart of a specific method for calculating the spatial rotation angle of a hand in one embodiment of the present invention;

[0043] Figure 3 1 is a flow chart of a specific method for calculating the cross product of the spatial rotation axes of the hand and normalizing the cross product in one embodiment of the present invention;

[0044] Figure 4 1 is a flow chart of a specific method for decomposing a rotational space angle into rotational components about the X-axis, the Y-axis, and the Z-axis in one embodiment of the present invention;

[0045] Figure 5 is a flowchart of a specific method for mapping rotation angle data to a range of motion parameters required for controlling a humanoid wrist joint robot in one embodiment of the present invention;

[0046] Figure 6 This is a sample diagram of a human-machine collaborative workbench in one embodiment of the present invention;

[0047] Figure 7 is a schematic diagram of a human-machine collaborative workbench in one embodiment of the present invention;

[0048] Figure 8 1 is a schematic diagram showing the effect of normalizing the calculated actual rotation Euler angles of each axis in one embodiment of the present invention;

[0049] Figure 9 2. It is a schematic diagram of a wrench grasping intention recognition and robot collaboration experiment in one embodiment of the present invention;

[0050] Figure 10 2. It is a schematic diagram of a screwdriver grasping intention recognition and robot collaboration experiment in one embodiment of the present invention;

[0051] Figure 11 Schematic diagram of hammer grasping intention recognition and robot collaboration experiment in one embodiment of the present invention;

[0052] Figure 12 This is an angle feedback diagram of a wrench grasping intention recognition and robot collaboration experiment in one embodiment of the present invention;

[0053] Figure 13 This is an angle feedback diagram of a screwdriver gripping intention recognition and robot collaboration experiment in one embodiment of the present invention;

[0054] Figure 14 This is an angle feedback diagram of a hammer grasping intention recognition and robot collaboration experiment in one embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with its embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0056] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly attached to the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only implementation methods.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0058] The "first" and "second" in the present invention do not represent specific quantities and orders, but are only used to distinguish names.

[0059] Before describing the embodiments of the present invention in detail, a brief introduction to the prior art is given first.

[0060] In recent years, deep learning technology has made significant progress in human-robot collaboration. Researchers have applied deep learning methods to human-robot collaborative tasks, using convolutional neural networks to identify and predict the operator's hand posture, achieving promising results. By integrating visual perception with a custom gesture model, the robot can continuously perceive the operator's intentions. Visual perception provides hand posture information, while the custom gesture model enables the robot to obtain important information such as the operator's intentions. By combining these two approaches, the robot can perceive the operator's work intentions, thereby better cooperating with the human hand and improving the efficiency of human-robot collaborative assembly.

[0061] One drawback of existing technologies for human-robot collaborative assembly is their limited ability to continuously perceive the operator's intentions. This limitation prevents the robot from fully understanding the operator's real-time intentions during the assembly process, hindering the closeness and efficiency of human-robot collaboration.

[0062] Furthermore, existing posture sensing technologies have significant limitations in the field of human-robot collaborative assembly. Existing posture detection methods often rely on a combination of multiple sensors. While these methods can obtain accurate posture information, they result in complex and costly systems. Furthermore, these systems are susceptible to electromagnetic interference under complex working conditions, resulting in cumbersome information processing and high system complexity.

[0063] To improve the performance and stability of human-robot collaborative assembly, this paper proposes a collaborative robot assembly method based on intentional hand spatial posture perception. This method continuously predicts the operator's hand posture through a deep learning model and, in combination with this intentional information, adjusts the robot's posture in real time, thereby improving the efficiency and stability of the collaborative assembly process. Furthermore, this method only requires a camera to acquire hand posture information, eliminating the need for multiple sensors. This simplifies system complexity and hardware architecture, reduces deployment costs, and effectively improves system reliability and practicality.

[0064] like Figure 1 As shown, a robot collaborative assembly method based on human hand spatial posture intention perception in one embodiment of the present invention includes the following steps:

[0065] S1, builds a customized gesture recognition model for tool holding gestures to recognize specific assembly gestures.

[0066] Here, you can use MediaPipe Model Maker to build a customized gesture recognition model for tool holding gestures to ensure that the system can accurately recognize specific assembly gestures.

[0067] As a preferred technical solution, the specific method of building a customized gesture recognition model for tool holding gestures includes:

[0068] S10 defines corresponding typical gripping gestures for three types of tools: screwdrivers, hammers, and wrenches.

[0069] Specifically, the Mediapipe Model Maker is used to customize typical grasping gestures as follows:

[0070] Screwdriver: Grasp the screwdriver handle head with your five fingers naturally bent, press the end with the palm of your hand, and drive the tightening action by rotating your wrist;

[0071] Hammer: Place the palm of your hand parallel to the hammer handle, hold the hammer handle tightly with your four fingers, place your thumb against your index finger, and swing your forearm to simulate the hammering operation;

[0072] Wrench: Wrench handle with four fingers around it, thumb and palm close to the handle surface, use wrist twisting to rotate the tool.

[0073] S11, constructing a tool-gesture mapping relationship, and analyzing the operator's action intention in real time according to the tool-gesture mapping relationship.

[0074] The system accurately maps the tool-gesture relationship, analyzes the operator's action intentions in real time, and triggers the coordinated motion response of the humanoid wrist joint robot operation platform. Specifically, the following methods can be used to improve the efficiency of human-machine collaboration and operational stability:

[0075] 1. By detecting the naturally bent hand posture of the five fingers of the hand skeleton, the system analyzes and matches the operator's intention to use a screwdriver and tighten a screw in real time. The system executes the relevant program based on this intention and accurately controls the humanoid wrist joint robot to complete the following actions:

[0076] 1.1 The humanoid wrist joint robot obtains the operator's intention to use a screwdriver and tighten the screws through the system;

[0077] 1.2 Based on the real-time calculation of the hand's spatial rotation angle, the humanoid wrist joint robot synchronizes its own position in real time, enabling real-time collaborative assembly between the human hand and the humanoid wrist joint robot;

[0078] 1.3 Throughout the screwdriving process, the system continuously senses the difference in posture between the hand and the robot and automatically adjusts the posture to ensure posture accuracy and stability during the screwdriving process.

[0079] 2. By detecting the hand posture of the hand skeleton with four fingers clenched and the thumb touching the index finger, the system interprets and matches the operator's intention to use a hammer and hammer a nail in real time. Based on this intention, the system executes the relevant program and precisely controls the humanoid wrist joint robot to complete the following actions:

[0080] 2.1 The humanoid wrist joint robot obtains the operator's intention to use a hammer and hammer a nail through the system;

[0081] 2.2 Based on the real-time calculation of the hand's spatial rotation angle, the humanoid wrist joint robot adjusts its position in real time, enabling real-time collaborative assembly between the human hand and the humanoid wrist joint robot;

[0082] 2.3 During the entire nailing process, the system detects the matching degree between the hand and the robot's movements in real time and performs real-time error correction to ensure the accuracy and stability of the posture during the nailing process.

[0083] 3. Detecting the hand posture of the four fingers pressed together and the thumb extended outward, the system analyzes and matches the operator's intention to use a wrench and tighten a nut in real time. Based on this intention, the system executes the relevant program and precisely controls the humanoid wrist joint robot to complete the following actions:

[0084] 3.1 The humanoid wrist joint robot obtains the operator's intention to use a wrench and tighten a nut through the system;

[0085] 3.2 Based on the real-time calculation of the hand's spatial rotation angle, the humanoid wrist joint robot adjusts its posture in real time, enabling real-time collaborative assembly between the human hand and the humanoid wrist joint robot;

[0086] 3.3 During the entire nut tightening process, the system monitors the position deviation between the hand posture and the robot posture in real time and automatically corrects the action to ensure the posture accuracy and stability of the nut tightening process.

[0087] S2, obtains the target image, identifies the human hand in the target image in real time, and extracts the three-dimensional coordinate information of the key points of the hand in real time.

[0088] The target image is acquired through a calibrated industrial camera, and the MediaPipe hand detection module is used to identify the human hand in the target image in real time and extract the 3D coordinate information of the key points of the hand in real time.

[0089] S3, based on the three-dimensional coordinate information of the key points of the hand, calculates the spatial rotation angle of the hand through the posture algorithm, and decomposes the rotation space angle into rotation components around the X axis, Y axis and Z axis.

[0090] Specifically, the three-dimensional coordinate information of the key points of the hand is transmitted to the PC control system, the spatial rotation angle of the hand is calculated through the posture algorithm, and the rotational spatial angle is decomposed into rotation components around the X-axis, Y-axis and Z-axis.

[0091] As a preferred technical solution, Figure 2 As shown, in step S3, the specific method for calculating the spatial rotation angle of the hand includes:

[0092] S31, extracting the wrist key point, the index finger tip key point, and the little finger tip key point from the hand key nodes.

[0093] Specifically, the MediaPipe hand detection model is used to extract the wrist key points, index finger tip key points, and little finger tip key points from the 21 key points of the hand.

[0094] S32, calculate the vector from the key point of the index finger tip to the key point of the wrist and the vector from the pinky fingertip keypoint to the wrist keypoint .

[0095] Specifically, the formula Calculate the key points of the index finger tip To the wrist key point Vector and the key point of the little finger To the wrist key point Vector .

[0096] Considering that hand shaking may cause unstable vector calculation results in practical applications, the vector can be weighted according to the confidence of the hand key point detection to improve stability. Assume that each key point has a confidence score c (the value range is 0-1, 1 indicates the highest confidence). As a preferred technical solution, the vector from the key point of the index finger tip to the key point of the wrist is calculated. and the vector from the pinky fingertip keypoint to the wrist keypoint The specific method includes the following steps:

[0097] The first step is to obtain the wrist key points Confidence , key points on the tip of the index finger Confidence And the key point of the little finger Confidence ;

[0098] The second step is based on the formula Get Vector , according to the formula Get Vector .

[0099] For the confidence 、 as well as , can be obtained based on the heat map peak, that is, the key point is located by predicting the heat map of the key point. The value of each pixel in the heat map represents the probability that the position belongs to the corresponding key point. The peak position of the heat map is the key point coordinate, and the peak value is the confidence level. The specific steps include:

[0100] 1. Heatmap generation: The model outputs three heatmaps, each corresponding to a key point (such as Point1, Point2, Point3).

[0101] 2. Peak detection: Perform non-maximum suppression on each heat map and find the local maximum as the key point location.

[0102] 3. Confidence extraction: The normalized value (range 0-1) of the heat map peak is the confidence.

[0103] Of course, object detection models such as YOLOv8 can also output both the classification probability and the position confidence within the detection box. The final confidence is the product of the two. Specifically, the detection box containing the hand key points is first predicted, and then the classification probability of the "key points" within the detection box is obtained. The intersection over union (IoU) between the detection box and the ground truth box is then calculated, and the final overall confidence is obtained = classification probability × IoU. It can be understood that in the object detection model YOLOv8, the confidence is determined by both the probability of the key points within the bounding box and the degree of match (IoU) with the ground truth position.

[0104] In this way, key points with higher confidence will occupy a greater weight in vector calculation, making the vector results more reliable.

[0105] Furthermore, the original confidence can be modified by combining the hand bone length ratio and angle constraints ;in, Indicates the A raw confidence, such as the wrist key point Confidence , key points on the tip of the index finger Confidence And the key point of the little finger Confidence .

[0106] Represents key points The actual bone length. For example, when =1, It can be understood as the Euclidean distance between the wrist key point and the thumb fingertip key point. =2, It can be understood as the Euclidean distance between the key point of the index finger tip and the key point of the index finger metacarpophalangeal. =3, It can be understood as the Euclidean distance between the key point of the little finger tip and the key point of the little finger metacarpophalangeal.

[0107] Represent key points Here, we can obtain the key points of multiple different individuals by Based on the actual bone length data samples, the average and standard deviation of the bone lengths of multiple different individuals are calculated. The standard deviation of bone length reflects the individual differences in bone length or measurement noise.

[0108] In actual applications, in occlusion or low-light scenes, the system may mistakenly identify background noise as key points. Deviation from the mean When the confidence Attenuation is based on Gaussian distribution. For example: if Beyond , the confidence is significantly reduced, suppressing the abnormal detection results.

[0109] In this way, by correcting the confidence, the vector and vectors accuracy.

[0110] S33, through vector and vector Calculate the cross product of the hand's spatial rotation axis and normalize it, such as Figure 3 As shown, the specific steps include:

[0111] S331, calculation vector and vector The cross product of .

[0112] S332, obtain the spatial rotation axis Model .

[0113] S333, get the normalized vector .

[0114] S334, express the normalized vector in the form of an antisymmetric matrix .

[0115] S34, through vector and vector Calculate the spatial rotation angle of the hand and clamp the cosine value of the spatial rotation angle to the range of [-1, 1].

[0116] Specifically, through the formula Calculate the spatial rotation angle of the hand Finally, the spatial rotation angle of the hand .

[0117] As a preferred technical solution, Figure 4 As shown, the specific method of decomposing the rotation space angle into rotation components around the X-axis, Y-axis, and Z-axis includes:

[0118] S35, according to the unitized spatial rotation axis and spatial rotation angle Computes the spatial rotation matrix.

[0119] Specifically, according to the formula Calculate the spatial rotation matrix; where, Represents the identity matrix.

[0120] S36, according to the spatial rotation matrix Calculate the actual rotation Euler angle of each axis; the rotation Euler angle includes the Yaw angle 、Pitch angle and Roll Corner .

[0121] Specifically, the spatial rotation matrix is ​​composed of three Euler angles (Yaw), (Pitch), and (Roll) in the order of ZYX: The expanded matrix form is:

[0122]

[0123] By matrix elements and Calculating the Yaw Angle : .

[0124] By matrix elements 、 and Calculating the Pitch Angle : .

[0125] By matrix elements and Calculating the Roll Angle : .

[0126] S4, maps the rotation angle data to the motion parameter range required for the control of the humanoid wrist joint robot.

[0127] As a preferred technical solution, Figure 5As shown, in step S4, the specific method of mapping the rotation angle data to the motion parameter range required for the control of the humanoid wrist joint robot includes:

[0128] S41, normalizing the rotation angle data;

[0129] S42, mapping the normalized rotation angle data to a motion parameter range required for controlling the humanoid wrist joint robot through a linear fitting algorithm.

[0130] S5 integrates motion parameters and gesture information to generate control instructions, and controls the robot's movements based on the control instructions to achieve real-time collaborative assembly of human hands and humanoid wrist joint robots.

[0131] As a preferred technical solution, the specific method of controlling the robot motion based on the control instructions includes:

[0132] S51, transmitting the control instruction to the motion control card;

[0133] S52, the motion control card transmits the control instruction to the servo drive;

[0134] S53, the servo driver controls the servo motor to execute motion according to the control instruction.

[0135] Figure 6 The following is a sample picture of the human-machine collaborative workbench. Figure 6 In the figure, the industrial camera above is used to capture the pose information of the human hand image, and the humanoid wrist joint robot operating platform below is used to achieve human-machine collaboration through gesture intention and pose information.

[0136] Figure 7 The figure shows the workflow diagram of the human-machine collaboration workbench. Figure 7 In the process, the images processed by MediaPipe (i.e., the hand key point information) and the typical grasping gestures customized by MediaPipe Model Maker are transmitted to the computer for real-time processing, and the processing results such as angle information and posture information are transmitted to the robot operating platform in real time.

[0137] The main innovations of the robot collaborative assembly method based on human hand spatial posture intention perception described in this embodiment include:

[0138] 1. By customizing the grip gestures for different tools using MediaPipe Model Maker, the system can perceive the operator's gesture information in real time, thereby understanding the operator's work intentions and making corresponding movements.

[0139] 2. Using the MediaPipe hand detection model, the system extracts the 3D coordinate information of the key points of the hand and calculates the rotation angle of the hand in 3D space based on the 3D coordinate information, obtaining the real-time position information of the human hand in 3D space. The humanoid wrist joint robot adjusts its movement posture by fusing gesture intention with position information, realizing real-time collaborative assembly between the human hand and the humanoid wrist joint robot.

[0140] like Figure 9 、 Figure 10 as well as Figure 11 As shown, it can be seen that the robot collaborative assembly method according to the present invention can effectively identify the grasping intentions of three types of tools: screwdrivers, hammers, and wrenches.

[0141] like Figure 12 、 Figure 13 as well as Figure 14 As shown, by observing the experimental angle feedback diagrams of different tool grasping intentions and robot collaboration, it can be seen that after conducting the collaborative experiment, the robot collaborative assembly method based on the present invention can accurately infer the operator's work intention and achieve continuous perception, thereby realizing real-time collaborative assembly between human hands and humanoid wrist joint robots.

[0142] The robot collaborative assembly method based on human hand spatial posture intention perception in one embodiment of the present invention further includes the following steps:

[0143] S6, when the system is in a singularity state, the Roll angle is fixed ( ), only the Yaw angle is solved and Pitch Angle .

[0144] Specifically, when (Right now ), the system is in a singularity state.

[0145] S7, according to the Yaw angle 、Pitch angle and Roll Corner Drive the humanoid wrist joint robot to the corresponding target posture.

[0146] Specifically, when the Euler angle of the target posture around the y-axis is detected and calculated (Pitch) and the Euler angle of rotation around the x-axis (Roll), the y-axis motor and x-axis motor of the two-degree-of-freedom humanoid wrist joint robot are controlled to move to the corresponding target angle position.

[0147] like Figure 8As shown, the calculated actual rotation Euler angles of each axis are normalized and linear fitting is performed. The formulas for normalization and linear fitting are as follows:

[0148]

[0149]

[0150] in, Represents the original input data The minimum value in Represents the original input data The maximum value in represents the normalized input data, The slope of the fitted line, represents the intercept of the fitted line, Represents the output data after linear fitting.

[0151] In summary, this invention achieves continuous perception of the operator's work intentions through the deep integration of vision and intention perception technologies, while significantly simplifying the hardware architecture and reducing deployment costs. Compared with traditional human-machine collaboration solutions, this invention has the following core advantages:

[0152] 1. Intent-driven dynamic collaboration: Through deep learning models, the robot can perceive the operator's gesture intentions in real time, enabling it to proactively adjust its posture based on the operator's intentions.

[0153] 2. Lightweight Perception System: Compared to existing solutions using inertial measurement units, data gloves, or surface electromyography sensors, the use of a visual perception solution simplifies the hardware architecture and reduces system complexity. This solution, centered around lightweight design, optimizes perception performance while enabling rapid deployment, effectively reducing costs and improving system reliability.

[0154] 3. Improving Human-Robot Collaboration Efficiency: By combining industrial cameras with the MediaPipe hand detection model, this invention can rapidly perceive and acquire hand poses in real time, thereby shortening the robot's response time to the operator's actions. This real-time optimization reduces system latency, enabling the robot to more quickly adapt to the human operating rhythm, thereby improving the efficiency of human-robot collaboration.

[0155] 4. Improve assembly stability: By continuously sensing and acquiring the hand posture, the robot can dynamically adjust its posture in real time and adapt to changes during the assembly process, improving assembly stability and success rate.

[0156] That is to say, the present invention provides a robot collaborative assembly method based on the intention perception of the spatial posture of the human hand. By comprehensively utilizing vision and intention perception technology, the wrist joint robot operating platform can continuously perceive the operator's intention, realize the real-time collaborative assembly of the human hand and the humanoid wrist joint robot, and improve the stability of the assembly.

[0157] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A robot collaborative assembly method based on human hand spatial posture intention perception, characterized in that: The robot collaborative assembly method comprises: Build a customized gesture recognition model for tool holding gestures to identify specific assembly gestures; Acquire the target image, identify the human hand in the target image in real time, and extract the three-dimensional coordinate information of the key points of the hand in real time; Based on the three-dimensional coordinate information of the key points of the hand, the spatial rotation angle of the hand is calculated through the posture algorithm, and the rotational spatial angle is decomposed into rotation components around the X-axis, Y-axis and Z-axis; Mapping the rotation angle data to the motion parameter range required for humanoid wrist joint robot control; Integrate motion parameters and gesture information to generate control instructions, and control robot movements based on the control instructions to achieve real-time collaborative assembly between the human hand and the humanoid wrist joint robot; The specific methods for calculating the spatial rotation angle of the hand include: Extract the wrist key points, index finger tip key points and little finger tip key points from the key nodes of the hand; Calculate the vector from the index finger tip key point to the wrist key point and the vector from the pinky fingertip keypoint to the wrist keypoint ; By vector and vector Calculate the cross product of the hand's spatial rotation axis and normalize it; By vector and vector Calculate the spatial rotation angle of the hand; The specific methods for building a customized gesture recognition model for tool holding gestures include: For three types of tools, screwdrivers, hammers, and wrenches, corresponding typical gripping gestures are defined respectively; Construct tool-gesture mapping relationships and analyze the operator's action intentions in real time based on the tool-gesture mapping relationships; Calculate the vector from the index finger tip key point to the wrist key point and the vector from the pinky fingertip keypoint to the wrist keypoint The specific methods include: Get wrist key points separately Confidence , key points on the tip of the index finger Confidence And the key point of the little finger Confidence ; According to the formula Get Vector , according to the formula Get Vector .

2. A robot collaborative assembly method based on human hand spatial posture intention perception as claimed in claim 1, characterized in that: The specific methods for mapping rotation angle data to the range of motion parameters required for humanoid wrist joint robot control include: Normalize the rotation angle data; The normalized rotation angle data is mapped to the motion parameter range required for the control of the humanoid wrist joint robot through a linear fitting algorithm.

3. The robot collaborative assembly method based on human hand spatial posture intention perception according to claim 1, characterized in that: The specific methods of controlling robot motion based on control instructions include: Transmit control instructions to the motion control card; The motion control card transmits the control instructions to the servo drive; The servo drive controls the servo motor to perform motion according to control instructions.

4. The robot collaborative assembly method based on human hand spatial posture intention perception according to claim 1, characterized in that: The specific methods for decomposing the rotation space angle into rotation components around the X-axis, Y-axis, and Z-axis include: According to the unitized spatial rotation axis and spatial rotation angle Calculate the spatial rotation matrix; According to the spatial rotation matrix Calculate the actual rotation Euler angle of each axis; Among them, the rotation Euler angle includes the Yaw angle 、Pitch angle and Roll Corner .

5. The robot collaborative assembly method based on human hand spatial posture intention perception as claimed in claim 4, characterized in that: By formula Calculate the hand's rotation angle in space.

6. The robot collaborative assembly method based on human hand spatial posture intention perception according to claim 5, characterized in that: According to the formula Calculate the spatial rotation matrix; where, Represents the identity matrix.

7. A robot collaborative assembly method based on human hand spatial posture intention perception as claimed in claim 6, characterized in that: The robot collaborative assembly method further comprises: When the system is in a singularity state, the Roll angle is fixed , only solve the Yaw angle and Pitch Angle ; According to the Yaw angle 、Pitch angle and Roll Corner Drive the humanoid wrist joint robot to the corresponding target posture.

Images