Fine perception method for shaft-hole operation based on visual-tactile perception

Abstract

This invention relates to a fine-grained perception method for shaft-hole manipulation based on visual-tactile sensing, belonging to the field of robot tactile perception and planning. It includes: extracting the optical flow image from a visual-tactile sensor; analyzing the combined optical flow vectors in various directions based on the optical flow image; obtaining the motion trend of the gripped object relative to the gripper based on the combined optical flow vectors in various directions; obtaining the relative orientation of the current shaft-hole contact point based on the motion trend of the gripped object relative to the gripper; obtaining the current relative contact position of the shaft-hole based on the motion trend of the gripped object relative to the gripper; and designing a closed-loop feedback controller to adjust the relative position of the shaft-hole until alignment is achieved based on the current relative orientation of the shaft-hole contact point and the current relative contact position of the shaft-hole. This invention solves the problem of occlusion in traditional visual positioning and perception during fine manipulation and extreme environment tasks, achieving a higher-precision and faster shaft-hole assembly method.

Description

Technical Field

[0001] This invention belongs to the field of robot tactile perception and planning, and specifically relates to a method for robot shaft hole assembly positioning perception and operation by comprehensively using tactile state information. Background Technology

[0002] In the field of robot environmental perception, current methods primarily rely on visual signals acquired by various cameras for perception and analysis, enabling the robot to know "where it is." For example, in a robot's material grasping task, the robot uses an RGB or RGB-D camera to acquire image information of the task area, extract the outline of the object to be grasped, further analyze the object's pose, and finally control the robotic arm to complete the grasping task. Although the application of robot visual inspection and servo systems in industrial fields is quite mature, the error in visual positioning remains too large when facing unstructured, uncalibrated scenes, making it difficult to complete some delicate operations. Therefore, new perception methods are needed to further reduce the range of perception errors.

[0003] In recent years, robot perception technology has been continuously developing, and tactile perception technology has gradually matured. Representative sensors include the GelSight and Gelslim series from MIT, such as... Figure 1 As shown. Although this tactile sensor still collects visual signals, it uses a deformable colloidal material, which allows for more significant measurement of the deformation on the sensor surface. This enables the estimation of a series of state parameters during the operation process. Furthermore, because the sensors are all small in size, the measurement accuracy is greatly improved.

[0004] Tactile sensors are naturally suited for precision measurement tasks, providing accurate local information and showing great promise for applications in delicate operations. Therefore, this invention applies them to the precision operation of shaft-hole assembly. Summary of the Invention

[0005] The technical problem to be solved by this invention is:

[0006] To address the problem of occlusion in traditional visual positioning and perception during delicate operations and extreme environment tasks, this invention provides a fine perception method for shaft hole operations based on visual-tactile perception. This method uses tactile perception to sense the relative position of shaft holes, enabling faster shaft hole assembly with higher precision.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A fine-sensing method for shaft hole operation based on visual-tactile perception, characterized by employing a two-finger gripper to hold the object for operation, with a visual-tactile sensor installed at the end of each gripper; comprising:

[0009] Extract the optical flow image of the visual-touch sensor, and analyze the optical flow resultant vector of the visual-touch sensor in each direction based on the optical flow image;

[0010] The motion trend of the held object relative to the gripper is obtained based on the optical flow vectors in each direction;

[0011] Based on the motion trend of the clamped object relative to the gripper, the relative orientation of the current shaft hole contact point is obtained according to the relationship between the motion trend of the object when the bottom surface contacts and the orientation of the contact point.

[0012] Based on the motion trend of the clamped object relative to the gripper, the current relative contact position of the shaft hole is obtained by geometric reasoning.

[0013] Based on the current relative orientation of the shaft-hole contact point and the current relative contact position of the shaft-hole, a closed-loop feedback controller is designed to adjust the relative position of the shaft-hole until they are aligned.

[0014] A further technical solution of the present invention: the extraction of the optical flow image of the visual-touch sensor, and the analysis of the optical flow resultant vector of the visual-touch sensor in various directions based on the optical flow image, specifically includes:

[0015] Once the clamping is stable, the RGB camera on the visual-touch sensor reads the image of the current flexible surface of the sensor in real time.

[0016] The received sensor flexible surface image is processed using optical flow method, and the optical flow trend is decomposed into various directions of the sensor coordinate system;

[0017] Calculate the optical flow vector in each direction for each visual-tactile sensor.

[0018] A further technical solution of the present invention: the optical flow vectors in each direction include: X left X right Y left Y right curl left curl right , where X left X right Y left Y right curl left curl right These are the resultant vectors of the optical flow along the X, Y, and Z directions in the left and right finger sensor image coordinate systems, respectively.

[0019] A further technical solution of the present invention: the motion trend of the clamped object relative to the gripper is obtained based on the optical flow vectors in each direction, specifically as follows:

[0020]

[0021] Where r is the thickness at the clamping point; T(X), T(Y), T(Z), T(R) z ) represents the tendency of the clamped object to move relative to the clamp in the clamp coordinate system, and all of them are dimensionless numbers.

[0022] A further technical solution of the present invention: Based on the motion trend of the clamped object relative to the gripper, the relative orientation of the current shaft hole contact point is obtained according to the relationship between the motion trend of the object when the bottom surface contacts and the orientation of the contact point, specifically:

[0023] T(X)=K1*ΔX, T(Y)=K2*ΔY, K1,K2>0

[0024] Where ΔX and ΔY are the distances of the contact points from the origin in the gripper coordinate system. This formula shows that the motion trend of the gripped object is directly proportional to the coordinate distribution of the contact points, and <ΔX,ΔY> is the relative orientation of the contact points.

[0025] A further technical solution of the present invention: the current relative contact position of the shaft hole is obtained based on the movement trend of the clamped object relative to the gripper and through geometric reasoning, specifically as follows:

[0026]

[0027] in, <D x D y >The dimensional information of the contact surface relative to the contact point is obtained by measurement. <T(X) max ,T(Y) max >For size information <D x D y >Corresponding magnitude of the movement trend;<T(X),T(Y)> The magnitude of the movement trend corresponding to the current contact position; <d x ,d y >This represents the calculated location of the contact area.

[0028] A further technical solution of the present invention: the closed-loop feedback controller specifically comprises:

[0029]

[0030] Where δX and δY are the adjustment amounts in the X and Y directions of the gripper coordinate system, P x and P y The adjustment ratio is determined by the following formula:

[0031]

[0032] Where, αx and α y These are the scaling factors for the X and Y directions, respectively.

[0033] A further technical solution of the present invention includes completing the shaft hole assembly operation based on shaft hole alignment, specifically as follows:

[0034] After aligning, twist downwards to insert. Check if insertion is complete using the following method:

[0035]

[0036] Where, ε Z and These are the motion trend thresholds in the Z and Rz directions.

[0037] A computer system is characterized by comprising: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described above.

[0038] A computer-readable storage medium is characterized by storing computer-executable instructions, which, when executed, are used to implement the above-described method.

[0039] The beneficial effects of this invention are as follows:

[0040] This invention provides a fine perception method for shaft hole operation based on visual-tactile perception, which solves the problem of occlusion in traditional visual positioning and perception in fine operation and extreme environment tasks, and realizes a method for high-precision and rapid shaft hole assembly.

[0041] 1. Based on the tactile perception of a two-finger gripper, a method is designed to estimate the motion trend of an object held by a gripper using the optical flow characteristics in a visual-tactile sensor.

[0042] 2. Based on the relative position estimation of the shaft and hole using the tactile perception of a two-finger gripper, a method is designed to estimate the position of the external contact point according to the movement trend of the gripped object, and then estimate the relative position of the shaft and hole.

[0043] 3. Design a closed-loop feedback controller based on the estimated relative position of the shaft and hole to complete the alignment and assembly of the shaft and hole. Attached Figure Description

[0044] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0045] Figure 1 GelSight and Gelslim series sensors.

[0046] Figure 2 Flowchart of a fine-sensing method for shaft hole operation based on visual-tactile perception.

[0047] Figure 3 Explanation of the coordinate system used in the experiment.

[0048] Figure 4 Experimental scenario for shaft and hole assembly.

[0049] Figure 5 A schematic diagram of the optical flow during operation.

[0050] Figure 6 Data curve 1 of the shaft-hole assembly experiment process.

[0051] Figure 7 Data curve 2 of the shaft-hole assembly experiment process.

[0052] Figure 8 Data curve 3 of the shaft-hole assembly experiment process.

[0053] Figure 9 Data curve 4 of the shaft-hole assembly experiment process. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0055] This invention provides a method for precise sensing and manipulation of shaft-hole assembly based on visual-tactile perception. It employs a two-finger gripper to hold the object for manipulation, with a Gelsight visual-tactile sensor mounted at the end of the gripper. Specific scene and coordinate system definitions are as follows: Figure 3 and Figure 4 As shown. The specific steps include:

[0056] Step 1: Extract the optical flow image from the visual-tactile sensor and analyze the combined optical flow vector in each direction.

[0057] Once the gripping is stable, an RGB camera mounted on the Gelsight sensor reads the image of the sensor's flexible surface in real time. Because the relative motion between the gripped object and the two grippers causes deformation of the Gelsight sensor's flexible surface, the received sensor image is processed using optical flow. The optical flow trend is decomposed into six directions in the sensor coordinate system: +X, -X, +Y, -Y, +Rz, -Rz. Here, <+Rz, -Rz> represents the optical flow's rotational tendency around the Z-axis, which appears as a vortex in the image coordinate system. We choose to use the curl quantity to characterize the magnitude of this vortex-like motion trend. The resultant vector of the optical flow in each direction on the left and right grippers is calculated and defined as: X left X right Y left Y right curl left curl right .

[0058] Step 2: Based on obtaining the combined vectors of optical flow in each direction, the motion trend of the clamped object relative to the gripper is obtained.

[0059] In the gripper coordinate system, since the grippers stably hold one end of the object, the motion tendency of the object relative to the grippers in any direction can be decomposed into the following eight independent motion directions: +X, -X, +Y, -Y, +Z, -Z, +Rz, -Rz, where <+Rz, -Rz> is ​​the rotational tendency of the object fingers around the Z-axis. The motion tendency of the two fingers' optical flow and its combination can comprehensively reflect the eight motion tendencies of the gripped object relative to the grippers. The motion tendency of the gripped object relative to the grippers satisfies the following quantitative analysis:

[0060]

[0061] Among them, X left X right Y left Y right T(X), T(Y), T(Z), and T(R) are the resultant vectors of the optical flow along the X and Y directions in the left and right finger sensor image coordinate systems, respectively; r is the thickness at the clamping point; T(X), T(Y), T(Z), and T(R) are the optical flow vectors of the left and right finger sensor image coordinate systems, respectively. z ) represents the tendency of the clamped object to move relative to the clamp in the clamp coordinate system, and all of them are dimensionless numbers.

[0062] Step 3: Based on the motion trend of the clamped object, the relative orientation of the current shaft hole contact point is obtained according to the relationship between the motion trend of the object when the bottom surface contacts and the orientation of the contact point.

[0063] Since the contact occurs on the bottom surface of the clamped object, the tendency of motion and the position of the contact point satisfy the following relationship:

[0064] T(X)=K1*ΔX, T(Y)=K2*ΔY, K1,K2>0 (2)

[0065] Where ΔX and ΔY are the distances of the contact points from the origin in the gripper coordinate system. This formula shows that the motion trend of the gripped object is directly proportional to the coordinate distribution of the contact points. <ΔX,ΔY> are the relative orientations of the contact points. The orientations of the contact points are defined as the positive and negative signs of ΔX and ΔY, which can be divided into four directions: positive X direction, negative X direction, positive Y direction, and negative Y direction.

[0066] Step 4: Based on obtaining the relative orientation of the current shaft hole contact point, the position of the current shaft hole relative contact area is obtained through geometric reasoning.

[0067] Given the geometry and dimensions of the contact surfaces, the location of the current shaft-hole contact area can be estimated by the relative orientation of the contact points.

[0068]

[0069] in, <D x D y >The dimensional information of the contact surface relative to the contact point is obtained by measurement. <T(X) max ,T(Y) max >For size information <D x D y >Corresponding magnitude of the movement trend;<T(X),T(Y)> The magnitude of the movement trend corresponding to the current contact position; <d x ,d y >This represents the calculated location of the contact area.

[0070] Step 5: Based on the current relative contact position of the shaft and hole, design a closed-loop feedback controller to adjust the relative position of the shaft and hole until they are aligned.

[0071] The design of the closed-loop feedback controller is shown below:

[0072]

[0073] Where δX and δY are the adjustment amounts in the X and Y directions of the gripper coordinate system, P x and P y The adjustment ratio is determined by the following formula:

[0074]

[0075] Where, α x and α y These are the scaling factors for the X and Y directions, respectively.

[0076] After the controller is designed, repeat steps one through five until the relative positions of the shaft and hole are aligned. When the relative positions of the shaft and hole are aligned, the following equation is satisfied:

[0077]

[0078] Where, ε X and ε Y The threshold values ​​for the motion trends in the X and Y directions are used. When the motion trends in both the X and Y directions are less than the threshold values, the relative positions of the shaft and the hole can be considered aligned.

[0079] Step 6: After aligning the shaft and holes, complete the shaft and hole assembly operation.

[0080] After aligning, twist downwards to insert. Check if insertion is complete using the following method:

[0081]

[0082] Where, ε Z and The threshold values ​​for motion tendency in the Z and Rz directions are given by this formula. This formula indicates that if the shaft-hole assembly task is completed, there will be a large motion constraint in the Z direction. At the same time, due to the contact between the shaft and the hole wall, the shaft will be subject to a large constraint in the direction of rotation around the Z axis.

[0083] To enable those skilled in the art to better understand the present invention, the present invention will be described in detail below with reference to specific embodiments.

[0084] Example 1:

[0085] To demonstrate the effectiveness of the proposed method for precise sensing and positioning of shaft holes based on visual-tactile perception, we conducted practical verification using a Backyard two-finger gripper, a Gelsight visual-tactile sensor, and a shaft hole assembly task board. The experimental scenario is as follows: Figure 3 and Figure 4 As shown.

[0086] Step 1: Extract the optical flow image from the visual-tactile sensor and analyze the combined optical flow vector in each direction.

[0087] Once the gripping is stable, an RGB camera mounted on the GELSight sensor reads the image of the sensor's flexible surface in real time. Because the relative motion between the gripped object and the two grippers causes deformation of the GELSight sensor's flexible surface, the received sensor flexible surface image is processed using optical flow. The optical flow trend is decomposed into various directions in the sensor coordinate system, resulting in six directions: +X, -X, +Y, -Y, +Rz, -Rz. Here, <+Rz, -Rz> represents the optical flow's rotational tendency around the Z-axis, which appears as a vortex in the image coordinate system. We choose to use the curl quantity to characterize the magnitude of this vortex-like motion trend. The resultant vector of the optical flow in each direction in each sensor is calculated and defined as: X left X right Y left Y right curl left curl right .

[0088] A schematic diagram of optical flow is shown below. Figure 5 As shown, the calculation results of the combined vector are automatically saved into the computer for later use.

[0089] Step 2: Based on obtaining the combined vectors of optical flow in each direction, the motion trend of the clamped object relative to the gripper is obtained.

[0090] In the gripper coordinate system, since the grippers stably hold one end of the object, the motion tendency of the object relative to the grippers in any direction can be decomposed into the following eight independent motion directions: +X, -X, +Y, -Y, +Z, -Z, +Rz, -Rz, where <+Rz, -Rz> is ​​the rotational tendency of the object fingers around the Z-axis. The motion tendency of the two fingers' optical flow and its combination can comprehensively reflect the eight motion tendencies of the gripped object relative to the grippers. The motion tendency of the gripped object relative to the grippers satisfies the following quantitative analysis:

[0091]

[0092] Among them, X left X right Y left Y right T(X), T(Y), T(Z), and T(R) are the resultant vectors of the optical flow along the X and Y directions in the left and right finger sensor image coordinate systems, respectively; r is the thickness at the clamping point; T(X), T(Y), T(Z), and T(R) are the optical flow vectors of the left and right finger sensor image coordinate systems, respectively. z ) represents the tendency of the clamped object to move relative to the clamp in the clamp coordinate system, and all of them are dimensionless numbers.

[0093] The results of the motion trend calculation are stored in the computer.

[0094] Step 3: Based on the motion trend of the clamped object, the relative orientation of the current shaft hole contact point is obtained according to the relationship between the motion trend of the object when the bottom surface contacts and the orientation of the contact point.

[0095] Since the contact occurs on the bottom surface of the clamped object, the tendency of motion and the position of the contact point satisfy the following relationship:

[0096] T(X)=K1*ΔX, T(Y)=K2*ΔY, K1,K2>0 (2)

[0097] Where ΔX and ΔY are the distances of the contact points from the origin in the gripper coordinate system. This formula shows that the motion trend of the gripped object is directly proportional to the coordinate distribution of the contact points. <ΔX,ΔY> represents the relative orientation of the contact points.

[0098] Step 4: Based on obtaining the relative orientation of the current shaft hole contact point, the position of the current shaft hole relative contact area is obtained through geometric reasoning.

[0099] Given the geometry and dimensions of the contact surfaces, the location of the current shaft-hole contact area can be estimated by the relative positions of the contact points.

[0100]

[0101] in, <D x D y >The dimensional information of the contact surface in the relative orientation direction of the contact point is obtained by measurement. <T(X) max ,T(Y) max >For size information <D x D y >Corresponding magnitude of the movement trend;<T(X),T(Y)> The magnitude of the movement trend corresponding to the current contact position; <d x ,d y >This represents the calculated location of the contact area.

[0102] In this example, D x =D y =15mm; T(x) max =0.95, T(y) max =0.048. (Calculated value) <d x ,d y Stored in the computer.

[0103] Step 5: Based on the current relative contact position of the shaft and hole, design a closed-loop feedback controller to adjust the relative position of the shaft and hole until they are aligned.

[0104] The design of the closed-loop feedback controller is shown below:

[0105]

[0106] Where δX and δY are the adjustment amounts in the X and Y directions of the gripper coordinate system, P x and P y The adjustment ratio is determined by the following formula:

[0107]

[0108] Where, α x and α y These are the scaling factors for the X and Y directions, respectively. In this example, the invention specifies α. x =1.5, α y =80.

[0109] After the controller is designed, repeat steps one through five until the relative positions of the shaft and hole are aligned. When the relative positions of the shaft and hole are aligned, the following equation is satisfied:

[0110]

[0111] Where, ε X and ε Y The threshold values ​​for the motion trends in the X and Y directions are used. When the motion trends in both the X and Y directions are less than the threshold values, the relative positions of the shaft and the hole can be considered aligned.

[0112] In this example, the present invention sets ε X =0.1, ε Y =0.005. This invention conducted four sets of experiments, such as... Figures 6 to 9 As shown, the relative positions of the shaft and hole were aligned after 2 to 4 adjustments.

[0113] Step 6: After aligning the shaft and holes, complete the shaft and hole assembly operation.

[0114] After aligning, twist downwards to insert. Check if insertion is complete using the following method:

[0115]

[0116] Where, ε Z and The threshold values ​​for motion tendency in the Z and Rz directions are given by this formula. This formula indicates that if the shaft-hole assembly task is completed, there will be a large motion constraint in the Z direction. At the same time, due to the contact between the shaft and the hole wall, the shaft will be subject to a large constraint in the direction of rotation around the Z axis.

[0117] In this example, the present invention sets ε Z =2, from Figures 6 to 9As can be seen from the data, the final shaft hole assembly meets the conditions for successful insertion, and the number of explorations required for successful insertion ranges from a minimum of 2 to a maximum of 4.

[0118] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in the present invention, and such modifications or substitutions should all be covered within the scope of protection of the present invention.

Claims

1. A fine sensing method for shaft hole operation based on visual-tactile perception, characterized in that, The device uses a two-finger gripper to hold objects for operation, with a visual-tactile sensor installed at the end of each gripper; it includes: Extract the optical flow image from the visual-touch sensor, and analyze the combined optical flow vectors in various directions of the visual-touch sensor based on the optical flow image; specifically: Once the clamping is stable, the RGB camera on the visual-touch sensor reads the image of the current flexible surface of the sensor in real time. The received sensor flexible surface image is processed using optical flow method, and the optical flow trend is decomposed into various directions of the sensor coordinate system; Calculate the optical flow vector in each direction for each visual-touch sensor; the optical flow vector in each direction includes: , , , , , ,in, , , , , These are the resultant vectors of the optical flow along the X, Y, and Z directions in the left and right finger sensor image coordinate systems, respectively. The motion trend of the held object relative to the gripper is obtained based on the optical flow vectors in each direction; specifically: Where r is the thickness of the clamping point; , , , The motion tendency of the clamped object relative to the clamp in the clamp coordinate system is represented by dimensionless numbers. Based on the motion trend of the clamped object relative to the grippers, the relative orientation of the current shaft hole contact point is obtained according to the relationship between the object's motion trend when the bottom surface contacts the contact point; specifically: in, and Let be the distance of the contact point from the origin in the gripper coordinate system. This formula shows that the motion tendency of the gripped object is directly proportional to the coordinate distribution of the contact point. This refers to the relative orientation of the contact point; Based on the motion trend of the held object relative to the grippers, the current relative contact position of the shaft hole is obtained through geometric reasoning; specifically: in, The dimensional information of the contact surface relative to the contact point is obtained by measurement. For size information The corresponding magnitude of the movement trend; The magnitude of the movement trend corresponding to the current contact position; The calculated location of the contact area; Based on the current relative orientation of the shaft-hole contact point and the current relative contact position of the shaft-hole, a closed-loop feedback controller is designed to adjust the relative position of the shaft-hole until alignment is achieved; the closed-loop feedback controller specifically comprises: wherein and are adjustment amounts in the X and Y directions in the gripper coordinate system, and are proportional coefficients of the adjustment, determined by the following equation: wherein, and are the scale factors in the X and Y directions, respectively; This also includes completing the shaft and hole assembly operation based on shaft and hole alignment, specifically... After aligning, twist downwards to insert. Check if insertion is complete using the following method: wherein and are the motion trend thresholds for the Z direction and the Rz direction.

2. A computer system, characterized by include: One or more processors, a computer-readable storage medium for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of claim 1.

3. A computer-readable storage medium, characterized in that The device stores computer-executable instructions, which, when executed, are used to implement the method of claim 1.

Images