Robot assembly method and device based on touch sense and robot

Abstract

The invention provides a touch-based robot assembly method and device and a robot. The method provided by the invention comprises the following steps: when jamming in an insertion process is detected, acquiring a tactile image from a tactile sensor, performing feature extraction on the tactile image, and determining a tactile stream image based on feature maps corresponding to continuous frames of tactile images; according to the displacement variation of each target spot, determining the shearing force applied to the plug-in by each target spot; determining a current target jamming state according to the shearing force and a pre-established stress model; determining a relative posture according to the displacement variation and the shearing force of each target spot; based on a predetermined control strategy and the relative attitude, determining the action amount of each action indicated by the control strategy; and the robot is controlled according to the working sequence indicated by the control strategy and the action quantity to relieve jamming, so that the plug-in is inserted into the jack. The tactile-based robot assembly method and device and the robot are used for ensuring that the plug-in is accurately inserted into the jack.

Description

Technical Field

[0001] This application relates to the field of robot assembly technology, and in particular to a tactile-based robot assembly method, apparatus and robot. Background Technology

[0002] With the rapid development of industrial automation, robotic assembly technology is being used more and more widely in manufacturing. In the field of robotic assembly, the "peg in hole" task is one of the most typical challenges and has been extensively studied for a long time. This type of task requires the robot to precisely insert the plug into the hole, involving complex spatial positioning and force control.

[0003] Current robotic assembly technologies primarily rely on visual feedback. For example, some studies use visual sensors to acquire the positional information of target objects and insert components based on the visual images. However, visual sensors are susceptible to interference from factors such as lighting and occlusion in complex environments, leading to inaccurate positioning. In complex assembly tasks, robots may frequently experience assembly failures or damage to components. Therefore, accurately inserting components into their sockets remains a pressing problem in the field of robotic assembly. Summary of the Invention

[0004] In view of this, this application provides a tactile-based robot assembly method, apparatus, and robot to ensure that plugs are accurately inserted into sockets.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] The first aspect of this application provides a tactile-based robot assembly method, applied to a robot. The robot's gripper is equipped with a tactile sensor for acquiring tactile images of the gripping surface of the gripper. Target points are provided on the gripping surface. The robot is used to insert a plug held by the gripper into a socket. The method includes:

[0007] When obstruction is detected during the insertion process, the tactile image is acquired from the tactile sensor, features are extracted from the tactile image, and a tactile flow image is determined based on the feature maps corresponding to consecutive frames of tactile images; wherein, the tactile flow in the tactile flow image is used to characterize the displacement change of the target point;

[0008] The shear force applied to the plug at each target point is determined based on the displacement change of each target point.

[0009] Based on the shear force applied to the plug at each target point and the pre-established force model corresponding to various jamming states, the current target jamming state is determined.

[0010] The relative orientation of the plug-in to the tactile sensor is determined based on the displacement change of each target point and the shear force applied to the plug-in by each target point.

[0011] Based on the control strategy pre-determined for the target jamming state and the relative attitude, the action amount of each action indicated by the control strategy is determined; wherein, different jamming states correspond to different control strategies, and the control strategy is used to indicate the action sequence of the gripper and the target attitude after each action;

[0012] The robot is controlled according to the working sequence indicated by the control strategy and the amount of action of each action to release the obstruction and allow the plug to be inserted into the socket.

[0013] A second aspect of this application provides a tactile-based robotic assembly device, the device comprising an acquisition module, a determination module, and a control module; wherein,

[0014] The acquisition module is used to acquire the tactile image from the tactile sensor when jamming is detected during the insertion process, extract features from the tactile image, and determine a tactile flow image based on the feature maps corresponding to consecutive frame tactile images; wherein, the tactile flow in the tactile flow image is used to characterize the displacement change of the target point;

[0015] The determining module is used to determine the shear force applied to the plug at each target point based on the displacement change of each target point.

[0016] The determining module is also used to determine the current target jamming state based on the shear force applied to the plug at each target point and the pre-established force model corresponding to various jamming states.

[0017] The determining module is further configured to determine the relative posture of the plug-in with respect to the tactile sensor based on the displacement change of each target point and the shear force applied to the plug-in by each target point.

[0018] The determining module is further configured to determine the amount of action of each action indicated by the control strategy based on the control strategy pre-determined for the target jamming state and the relative posture; wherein, different jamming states correspond to different control strategies, and the control strategy is used to indicate the action sequence of the gripper and the target posture after each action;

[0019] The control module is used to control the robot according to the working sequence indicated by the control strategy and the amount of action of each action, so as to release the jamming and allow the plug to be inserted into the socket.

[0020] A third aspect of this application provides a robot, the robot including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the robot, when executing the program, implements the steps of any of the methods provided in the first aspect of this application.

[0021] The tactile robot assembly method, apparatus, and robot provided in this application first acquire tactile images from a tactile sensor when jamming is detected during the insertion process. Features are extracted from the tactile images, and a tactile flow image is determined based on the feature maps corresponding to consecutive frames of tactile images. Then, the shear force applied to the plug by each target point is determined based on the displacement change of each target point. Furthermore, the current target jamming state is determined based on the shear force applied to the plug by each target point and the pre-established force models corresponding to various jamming states. Next, the relative posture of the plug with respect to the tactile sensor is determined based on the displacement change of each target point and the shear force applied to the plug by each target point. Then, based on the control strategy and relative posture pre-determined for the target jamming state, the action amount of each action indicated by the control strategy is determined. Finally, the robot is controlled according to the working sequence indicated by the control strategy and the action amount of each action to release the jamming and allow the plug to be inserted into the socket. In this way, firstly, by extracting features from tactile images and generating tactile flow images from consecutive frames, the displacement changes of the target points on the clamping surface can be accurately captured. This allows for precise determination of the resistance encountered by the plug-in during insertion, providing crucial data for subsequent attitude adjustment and jamming release. Secondly, by combining the displacement change and shear force information of the target points, the relative attitude of the plug-in with respect to the tactile sensor can be accurately obtained, providing a basis for robot attitude adjustment. Thirdly, by using different control strategies corresponding to different target card states, the most appropriate action can be taken when adjustment is needed, ensuring the accuracy of the plug-in entering the socket. Thus, achieving accurate plug-in control through tactile analysis can improve the robot's adaptability and success rate during assembly. Attached Figure Description

[0022] Figure 1 A flowchart of an embodiment of the tactile-based robot assembly method provided in this application;

[0023] Figure 2 This is a schematic diagram illustrating the clamping surface in an exemplary embodiment of this application;

[0024] Figure 3 A flowchart of Embodiment 2 of the tactile-based robot assembly method provided in this application;

[0025] Figure 4 This is a schematic diagram of a heat map illustrating an exemplary embodiment of this application;

[0026] Figure 5 A hardware structure diagram of the robot in which the tactile robot assembly device provided in this application is located;

[0027] Figure 6 This is a schematic diagram of the structure of a first embodiment of the tactile-based robot assembly device provided in this application. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0029] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0030] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0031] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0032] Figure 1 This is a flowchart of an embodiment of the tactile-based robot assembly method provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0033] S101. When a jam is detected during the insertion process, the tactile image is acquired from the tactile sensor, features are extracted from the tactile image, and a tactile flow image is determined based on the feature map corresponding to the continuous frame tactile images; wherein, the tactile flow in the tactile flow image is used to characterize the displacement change of the target point.

[0034] Before introducing the method provided in this implementation, let's first briefly introduce the application scenario:

[0035] The tactile robot assembly method provided in this embodiment is applied to a jointed robot with two symmetrical grippers at its end. The grippers have target points (tiny protrusions on the gripping surface) on their gripping surfaces. For example, this robot is a 7-DOF collaborative robot with motion and force accuracies of 0.1 mm and 0.05 N, respectively.

[0036] Furthermore, the robot's grippers are equipped with tactile sensors. Specifically, the tactile sensors are located inside the grippers and can be camera devices. These sensors can acquire tactile images of the gripping surfaces of the grippers. In a specific implementation, for example, one embodiment has one tactile sensor in each gripper. This tactile sensor is 20mm to 23mm in size and has a frame rate of 25FPS, providing extremely fine, highly accurate, and repeatable millimeter-level surface detail.

[0037] Furthermore, referring to the preceding description, each gripper's gripping surface is provided with target points. For example, Figure 2 This is a schematic diagram illustrating the clamping surface in an exemplary embodiment of this application. Please refer to... Figure 2 ,exist Figure 2 In the example shown, 63 target points are set on the clamping surface, and these 63 target points are arranged in a 7×9 matrix.

[0038] Furthermore, the specific shapes of the plug and socket are determined according to actual needs, and this embodiment does not limit them. In specific implementation, the plug and socket can have matching shapes, and the gap between the shaft and the hole is 0.5mm. For example, in one embodiment, the plug is cylindrical and the socket is circular.

[0039] It should be noted that during the use of the robot, two grippers are used to pick up the plug-in, and flat-nose pliers are used to fix the plug-in hole on the plane. The robot moves its robotic arm to insert the plug-in into the plug-in hole, thus completing the plug-in insertion and assembly process.

[0040] After introducing the specific application scenarios, the method provided in this application will be described below:

[0041] Specifically, "jamming" refers to an obstruction encountered by the plug-in when it enters the socket, preventing direct insertion. In practice, the robot can detect jamming by monitoring assembly force, speed, or other relevant parameters. Once jamming is detected, the robot immediately enters a tactile adjustment mode (i.e., adjusts the plug-in using the method provided in this application to insert it into the socket). For example, the robot determines jamming upon detecting a reminder command. This reminder command could be a command triggered by a button on the robot when the user notices jamming.

[0042] Specifically, in this step, when a jam is detected during the insertion process, the tactile image is acquired from the tactile sensor, features are extracted from the tactile image, and a tactile flow image is determined based on the feature map corresponding to the continuous frame tactile images.

[0043] The tactile image is a data image containing the physical characteristics of the clamping surface. The tactile information of the clamping surface, such as the distribution of contact force and surface shape, can be determined through the tactile image. In specific implementation, a target point change map of the clamping surface can be captured by a tactile sensor, and an image can be constructed based on the change of the target point to obtain the corresponding tactile image.

[0044] Furthermore, feature extraction is performed on the acquired tactile images to extract useful tactile features. For example, in one embodiment, feature extraction can obtain feature data such as the corners, shape, and texture of the target point. In specific implementation, feature extraction can be performed based on traditional feature extraction methods or neural network-based feature extraction methods; this embodiment does not limit the specific methods used.

[0045] Furthermore, haptic flow images can describe the position of the gripper through changes in the shape and displacement of the target point.

[0046] In a specific implementation, one embodiment can continuously capture tactile images using a tactile sensor, and determine the feature maps corresponding to the extracted continuous frames as a tactile flow to complete the tracking of the target point and record the displacement changes of the target point over time.

[0047] Optionally, in one possible implementation, the process of determining the haptic flow image based on the feature maps corresponding to consecutive frame haptic images may include:

[0048] (1) Select the feature points corresponding to the target points from the feature map.

[0049] In practical implementation, feature points corresponding to the target point can be selected from the feature map based on traditional feature extraction methods, or feature points corresponding to the target point can be selected from the feature map based on neural network extraction methods. For example, in one embodiment, feature points can be extracted based on the following formula:

[0050] ;

[0051] in, These are the feature points corresponding to the target point. Characteristic map.

[0052] (2) The displacement of the feature point between the feature maps corresponding to the continuous frame tactile images is calculated by optical flow method to obtain the displacement vector corresponding to each feature point; wherein the displacement vector represents the direction and speed of movement of each feature point in the image space.

[0053] In practical implementation, the displacement change of each target point can be calculated using optical flow. For example, in one embodiment, tactile flow can be represented as the displacement vector of the target point. and .

[0054] Furthermore, the displacement vector of the target point can be calculated based on the following formula. and :

[0055] ;

[0056] ;

[0057] in, and Let n be the displacement vector of the target point, n be the number of target points, and T represent the tactile flow. and This is an array corresponding to all target points in two adjacent frames.

[0058] (3) Visualize the displacement vector corresponding to each feature point by means of a vector field to obtain the tactile flow image; wherein, in the tactile flow image, the arrow of each feature point indicates the moving direction and speed of the target point corresponding to that feature point.

[0059] Specifically, each target point is visualized using a vector field, assigning each target point to a specific vector field. and The difference between the two values ​​is determined as the displacement change corresponding to that target point.

[0060] In practical implementation, each feature point in the haptic flow image represents a target point, and the movement direction and speed of the target point are represented in the haptic flow image through the haptic flow itself. For example, Figure 4 This is a schematic diagram illustrating a tactile flow image as an exemplary embodiment of this application. Please refer to... Figure 4 The direction and speed of movement of each target point can be characterized by tactile flow images.

[0061] The tactile robot assembly method provided in this embodiment uses optical flow to calculate the displacement between feature points in consecutive frames of tactile images, obtaining the displacement vector corresponding to each feature point. In this way, tactile flow images are generated based on vector fields, which can intuitively show the dynamic changes of the plug-in during the assembly process, providing a foundation for subsequent plug-in insertion into the socket.

[0062] S102. Determine the shear force applied to the plug at each target point based on the displacement change of each target point.

[0063] Specifically, due to the presence of the jamming, the grippers will move relative to each other on the gripping surface. At this time, shearing force will be generated between the target point on the grippers and the plug-in.

[0064] In practical implementation, for each target point, the greater the displacement change of the target point, the greater the shear force applied to the plug-in; the smaller the displacement change of the target point, the smaller the shear force applied to the plug-in. Therefore, in one possible implementation, the shear force applied to the plug-in by each target point can be determined based on the correspondence between displacement change and shear force. For example, in one embodiment, the displacement change of target point A is 0, so the shear force applied to target point A is determined to be 0; the displacement change of target point B is 0.01 mm, so the shear force applied to target point B is determined to be 0.1 N; the displacement change of target point C is 0.03 mm, so the shear force applied to target point C is determined to be 0.3 N.

[0065] Optionally, in one embodiment, the specific implementation process of this step may include:

[0066] Step 1: Determine the shear stress applied to the plug at each target point based on the displacement change of each target point and the shear modulus corresponding to the clamping surface.

[0067] In practice, the shear stress applied to the plug-in at each target point can be calculated based on the following formula:

[0068] ;

[0069] in, It is shear stress. It is the shear modulus. It is the change in displacement of the target point in the vertical plane.

[0070] Step 2: Determine the shear force exerted on the plug-in by each target point based on the shear stress exerted on each target point.

[0071] In practical implementation, the total shear stress on the surface of a single sensor and the magnitude of the shear force at each target point can be characterized by the following formula:

[0072] ;

[0073] Where f is the shear force corresponding to the target point.

[0074] S103. Determine the current target jamming state based on the shear force applied to the plug at each target point and the pre-established force model corresponding to various jamming states.

[0075] Specifically, the jamming states include one-way jamming outside the hole, two-way jamming outside the hole, one-way jamming inside the hole, and two-way jamming inside the hole. One-way jamming outside the hole corresponds to the first force model, two-way jamming outside the hole corresponds to the second force model, one-way jamming inside the hole corresponds to the third force model, and two-way jamming inside the hole corresponds to the fourth force model.

[0076] Specifically, in one embodiment, an assembly force perpendicular to the end face of the plug-in is applied to the plug-in. At that time, reaction forces will be generated at the two contact points. and This causes axial jamming of the plug-in. At this time, the projection of the resultant reaction force F on the horizontal plane always points to the center of the plug-in.

[0077] Furthermore, through force analysis, the force signals of the force sensor in the X and Z axes under radial jamming conditions are... , and torque in the Y-axis direction Size can be represented as:

[0078] ;

[0079] in, , and This is the force sensor signal under bidirectional blocking conditions. The torque in the Y-axis direction, , This represents the force signals of the force sensor in the X and Z axes under radial jamming conditions.

[0080] Furthermore, the force sensor signal under bidirectional jamming state , and Size can be represented as:

[0081] ;

[0082] in, , and This is the force sensor signal under bidirectional blocking conditions. The torque in the Y-axis direction, , This represents the force signals of the force sensor in the X and Z axes under radial jamming conditions.

[0083] Furthermore, The component of the resultant force F generated at the contact point on the bottom surface of the shaft part along the X and Z axes of the coordinate system; The component of the resistance force f generated at the contact point on the side of the shaft part is the force along the X and Z axes of the coordinate system.

[0084] Furthermore, The distance from the contact point on the bottom surface of the shaft component to the origin of the force sensor along the X-axis; The distance from the contact point on the bottom surface of the shaft component to the origin of the force sensor along the X-axis; The distance is the distance from the side contact point of the shaft component to the origin of the force sensor along the X-axis.

[0085] Furthermore, the two contact points A and B on the shaft end face merge into a single point, causing axial single-point jamming of the shaft component. The projection of its reaction force F on the XOY plane still points towards the center of the hole. If the shaft component translates along the direction of the resultant force F, the side of the shaft cylinder will come into contact with the hole component. At this time, the shaft component will be subject to a resistance f, resulting in radial jamming, causing the shaft component to enter a bidirectional jamming state.

[0086] Furthermore, through force analysis, the force sensor signal under radial jamming state... , and Size can be represented as:

[0087] ;

[0088] in, , and This is the force sensor signal under bidirectional blocking conditions. The torque in the Y-axis direction, , This represents the force signals of the force sensor in the X and Z axes under radial jamming conditions.

[0089] Furthermore, the force sensor signal under bidirectional jamming state , and Size can be represented as:

[0090] ;

[0091] in, , and This is the force sensor signal under bidirectional blocking conditions. The torque in the Y-axis direction, , This represents the force signals of the force sensor in the X and Z axes under radial jamming conditions.

[0092] Based on the above analysis of the mechanical models of various contact states of the shaft and hole, when the shaft and hole assembly is in an axial jamming state (whether it is a single-point or multi-point contact), the projection of the reaction resultant force F generated at the contact point on the XOY plane points towards the center of the hole. The shaft part can reach the bidirectional jamming state by moving in the opposite direction of the resultant force.

[0093] Furthermore, different orientations of the plug correspond to different jamming states. By pre-establishing force models corresponding to various jamming states, the correspondence between shear force and different types of jamming states can be determined.

[0094] In specific implementations, for example, in one embodiment, by analyzing the shear force on the plug-in using a force model, it is determined that when the two contact points A and B on the axial end face of the plug-in are combined into a single point (i.e., only one point of force is applied to the bottom of the plug-in), the plug-in's jamming state at this time is defined as an axial single-point jamming state. In this case, the reaction force of the contact point points towards the center of the socket on the horizontal plane. In another embodiment, by analyzing the shear force on the plug-in using a force model, it is determined that after part of the plug-in enters the socket, there are multiple contact points with the socket. The plug-in's jamming state at this time is defined as a bidirectional jamming state within the socket. In this case, the resultant force of the reaction force of the contact points needs to project towards the center of the socket on the horizontal plane.

[0095] S104. Determine the relative orientation of the plug-in with respect to the tactile sensor based on the displacement change of each target point and the shear force applied to the plug-in by each target point.

[0096] Specifically, the relative posture of the plug-in to the tactile sensor can be represented by the spatial coordinates of the plug-in. By mapping the coordinates of the plug-in to the coordinates of the tactile sensor, the relative posture can be obtained.

[0097] In practical implementation, a topographic map of the plug-in can be drawn based on the specific numerical value of the displacement change. The specific orientation of the plug-in can be determined by combining the shear force applied to the plug-in at each target point. The relative attitude of the plug-in and the sensor can be obtained by combining the topographic map and the plug-in orientation. For example, in one embodiment, the target point with a displacement change greater than a displacement threshold is determined as the edge data of the plug-in, the shear force orientation of the target point with a shear force greater than a shear force threshold is determined as the orientation of the plug-in, and the point with the largest displacement change is determined as the center of the plug-in. The relative attitude of the plug-in is obtained by combining the edge data and the orientation data of the plug-in.

[0098] S105. Based on the control strategy pre-determined for the target jamming state and the relative posture, determine the action amount of each action indicated by the control strategy; wherein, different jamming states correspond to different control strategies, and the control strategy is used to indicate the action sequence of the gripper and the target posture after each action.

[0099] Specifically, the control strategy is used to indicate the direction and sequence of the gripper's movement. The control strategy determines how the robot moves, rotates, and places objects. For example, in one embodiment, control strategy a is: the gripper first moves to the left, then backward, and then diagonally downward and backward.

[0100] Furthermore, the relative attitude is adjusted through a control strategy, so that the adjusted plug-in can be inserted into the socket. In other words, the relative attitude of the plug-in can be changed to the target attitude through the control strategy, and the final target attitude can be directly inserted into the socket.

[0101] In specific implementation, for example, in one embodiment, the following methods are used: bidirectional jamming outside the hole: the shaft is raised slightly and then moves downward in a spiral; unidirectional jamming outside the hole: the shaft moves towards the center of the hole and then is inserted in a spiral; bidirectional jamming inside the hole: the shaft is raised to disengage from the jamming and then moves downward in a spiral; unidirectional jamming inside the hole: the shaft is directly inserted in a spiral trajectory.

[0102] S106. Control the robot according to the working sequence indicated by the control strategy and the amount of action of each action to release the jamming and allow the plug to be inserted into the socket.

[0103] In a specific implementation, for example, in one embodiment, when the target jamming state is an axial single-point jamming state, the control strategy is to first move backward and then move left. Then, based on the relative posture at this time, the amount of backward movement is determined to be 3 cm and the amount of left movement is determined to be 2 cm. At this time, by controlling the plug-in to move backward by 3 cm and then move left by 2 cm through the gripper, the plug-in can be released from jamming and inserted downward into the socket.

[0104] The tactile robot assembly method, apparatus, and robot provided in this embodiment, when detecting obstruction during the insertion process, acquire tactile images from a tactile sensor, extract features from the tactile images, and determine a tactile flow image based on the feature maps corresponding to consecutive frames of tactile images. Then, based on the displacement changes of each target point, determine the shear force applied to the plug-in by each target point. Furthermore, based on the shear force applied to the plug-in by each target point and the pre-established force models corresponding to various obstruction states, determine the current target obstruction state. Then, based on the displacement changes of each target point and the shear force applied to the plug-in by each target point, determine the relative posture of the plug-in with respect to the tactile sensor. Then, based on the control strategy and relative posture pre-determined for the target obstruction state, determine the action amount of each action indicated by the control strategy. Finally, control the robot according to the working sequence indicated by the control strategy and the action amount of each action to release the obstruction and allow the plug-in to be inserted into the socket. In this way, firstly, by extracting features from tactile images and generating tactile flow images from consecutive frames, the displacement changes of target points on the clamping surface can be accurately captured. This allows for precise determination of the resistance encountered by the plug-in during insertion, providing crucial data for subsequent attitude adjustment and jamming release. Secondly, by combining the displacement change and shear force information of the target points, the relative attitude of the plug-in with respect to the tactile sensor can be accurately obtained, providing a basis for robot attitude adjustment. Thirdly, by using different control strategies corresponding to different target card states, the most appropriate action can be taken when adjustment is needed, ensuring the accuracy of the plug-in entering the socket. Thus, accurately obtaining the resistance information through tactile flow images provides a foundation for subsequent adjustments. Determining the attitude based on this ensures that the plug-in's position matches the actual position. Combining the obtained plug-in position with the control strategy ensures reasonable plug-in adjustment, improving the robot's adaptability and success rate during assembly.

[0105] Figure 3 This is a flowchart of Embodiment 2 of the tactile-based robot assembly method provided in this application. Please refer to... Figure 3 Based on the above embodiments, the step of determining the relative attitude of the plug-in with respect to the tactile sensor according to the displacement change of each target point and the shear force applied to the plug-in by each target point includes:

[0106] S401. Based on the shear force applied to the plug at each target point, construct a heat map to represent the distribution of the shear force on the clamping surface; wherein, the color intensity of a point on the heat map represents the magnitude of the shear force at that point on the clamping surface.

[0107] Specifically, the shear force corresponding to the target point is visualized to obtain a heat map. Each region on the heat map corresponds to a target point, and its color corresponds to the shear force at the target point.

[0108] In practice, a heatmap drawing tool can be used to convert the two-dimensional array corresponding to the target shear force data into a visual heatmap.

[0109] Figure 4 This is a schematic diagram of a heat map illustrating an exemplary embodiment of this application. Please refer to... Figure 4 Based on the shear force applied at each target point, a thermogram of the shear force on the clamping surface is obtained. Figure 4 The dark red areas represent target regions with higher shear forces, while the light yellow areas represent target regions with lower shear forces.

[0110] Understandably, the plug exerts a downward force on the target under the action of gravity, and the target therefore generates shear force. Depending on the shape of the plug, different target points have different shear forces due to their different contact positions with the plug. The orientation of the plug can be determined based on the heat map corresponding to the shear force.

[0111] S402. Based on the heat map and the displacement changes of each target point, determine the center point of the plug-in on the heat map and the length direction of the plug-in on the heat map.

[0112] In specific implementation, for example, in one embodiment, in conjunction with the above embodiment, the shape of the plug-in is determined according to the heat map, and the length direction of the plug-in is determined according to the extension direction of the dark red line. According to the displacement change law of the target point, the point with the largest displacement change inside the heat map is located as the center point of the plug-in.

[0113] Optionally, in one embodiment, the specific implementation process of this step includes:

[0114] (1) Based on the displacement change of each target point, find the boundary point with the largest displacement change on the boundary line of the clamping surface and the target point with the largest displacement change on the inner surface of the clamping surface other than the boundary line from the heat map.

[0115] Specifically, the boundary lines of the clamping surface are the four outermost edges. Please refer to [the documentation / reference]. Figure 3 The lines corresponding to the four columns of target points in the top, bottom, left, and right rows are the boundary lines of the heat map, and the parts corresponding to the other target points are the internal surfaces.

[0116] In practice, the displacement change of each target point corresponding to the boundary line can be obtained, and the point with the largest displacement change can be determined as the boundary point with the largest displacement change on the boundary line of the heat map. Then, the displacement change of each target point corresponding to the internal surface can be obtained, and the point with the largest displacement change can be determined as the target point with the largest displacement change on the internal surface of the heat map.

[0117] (2) The target point is determined as the center point of the plug-in on the heat map, and the direction of the line connecting the target point and the boundary point is determined as the length direction of the plug-in on the heat map.

[0118] Specifically, the plugin can be approximated as a cylinder, with the center point being the geometric center of the cylinder, and the line connecting the target point and the boundary point representing the length direction of the plugin on the heatmap.

[0119] In a specific implementation, for example, in one embodiment, the boundary point is determined as α, the target point is β, and the line connecting α and β is the length direction of the plug-in on the heatmap.

[0120] Understandably, the diameter and length of the plug-in can be measured, and the plug-in's position can be determined based on its diameter and length, combined with its center point and length direction.

[0121] The tactile robot assembly method provided in this embodiment accurately identifies the target point with the largest displacement change on the clamping surface by analyzing the displacement changes of each target point on the heat map when determining the center point. This point is then determined as the center point of the plug-in on the heat map. This method considers not only the overall displacement of the plug-in but also the deformation information of the plug-in surface under stress, thus enabling accurate and efficient center point location. When determining the length direction of the plug-in, the length direction of the plug-in is accurately obtained by connecting the target point and the boundary point with the largest displacement change. In this way, by analyzing the deformation information of the plug-in surface under stress to identify its position and orientation, the robot can accurately identify and control plug-ins with complex shapes and diverse sizes. This enhances the robot's adaptability to complex assembly scenarios and enables it to handle more types of assembly tasks.

[0122] Furthermore, in another possible implementation, the specific implementation process of this step may include:

[0123] (1) Based on the displacement change of each target point, find the target point with the largest displacement change on the inner surface of the clamping surface other than the boundary line from the heat map.

[0124] In practice, the displacement changes of all target points on the internal surface are compared numerically, and the target point with the largest displacement change is determined as the target point.

[0125] (2) Based on the shear force at each point in the heat map, draw a target region on the heat map and determine the intersection of the target region with the boundary line of the heat map; wherein the shear force at each point in the target region is greater than a preset threshold.

[0126] Furthermore, after obtaining the displacement change of the target point, the shear force at each point in the thermogram can be calculated.

[0127] In practice, this step involves drawing a heat map based on the shear force at each point, and determining the intersection of the target area and the boundary line based on the heat map. For example, in one embodiment, the target area and the boundary line have one intersection point. In another embodiment, the target area and the boundary line may have no intersection point, two intersection points, or three intersection points.

[0128] (3) When the target area does not intersect with the boundary line of the heat map, the target point is taken as the center point of the plug-in, and the moving direction of any target point is taken as the length direction of the plug-in.

[0129] Specifically, the direction of movement of the target point can be determined based on the displacement change of the target point. In practice, the target point is determined as the center point of the plugin, and the direction of movement corresponding to the displacement of any target point is selected as the length direction of the plugin.

[0130] (4) When the target area intersects with a boundary line of the heat map, the first target boundary point with the largest displacement change is found according to the displacement change of each target point on the boundary line, and the target point is used as the center point of the plug-in, and the direction of the line connecting the target point and the first target boundary point is used as the length direction of the plug-in.

[0131] In practice, the displacement changes of each target point on the intersecting boundary are obtained, and the target point with the largest displacement change is determined as the first target boundary point. The length direction is obtained by connecting the first target boundary point and the target point. For example, in one embodiment, the target area intersects with the left boundary of the heat map, and the target point corresponding to the largest displacement change on the left boundary is C. The point D with the largest displacement change on the internal surface is determined, and the line connecting C and D is determined as the length direction of the plug-in.

[0132] (5) When two or three boundary lines of the heat map of the target area intersect, according to the displacement change of each target point on the two or three boundary lines, find the second target boundary point with the largest displacement change on each boundary line, determine the line connecting the two target boundary points with larger displacement changes in the second target boundary point as the length direction of the plug-in, and determine the intersection of the line segment passing through the target point and perpendicular to the line connecting the line connecting the line as the center point of the plug-in on the heat map.

[0133] In practice, the displacement change of each target point on the intersecting boundary is obtained, the target point with the largest displacement change is determined as the second target boundary point, and the two second target boundary points with the largest values ​​are connected, and this connection is determined as the length direction of the plugin.

[0134] Furthermore, starting from the target point, draw a perpendicular line to the line connecting the two second target boundary points, and determine the intersection of this perpendicular line and the line connecting the two second target boundary points as the center point of the plugin.

[0135] In specific implementations, for example, in one embodiment, the target area intersects with the upper and right boundaries of the heat map. The target point corresponding to the maximum displacement change at the upper boundary is identified as E, and the target point corresponding to the maximum displacement change at the right boundary is identified as F. The target point with the largest displacement change on the internal surface is determined as D. The connecting line EF is defined as the length direction of the plug-in. A perpendicular line is drawn from point D to the connecting line EF, and the intersection point is G. Point G is the center point of the plug-in. In another embodiment, the target area intersects with the upper, right, and lower boundaries of the heat map. The target point corresponding to the maximum displacement change at the upper boundary is identified as E, the target point corresponding to the maximum displacement change at the right boundary is identified as F, and the target point corresponding to the maximum displacement change at the lower boundary is identified as H. The two points with the largest displacement changes among points E, F, and H are identified as points F and H. The target point with the largest displacement change on the internal surface is determined as D. The connecting line FH is defined as the length direction of the plug-in. A perpendicular line is drawn from point D to the connecting line FH, and the intersection point is I. Point I is the center point of the plug-in.

[0136] The tactile robot assembly method provided in this embodiment determines the number of intersections between the target area and the boundary of the heat map through a heat map. Since the target area represents the posture of the plug-in, the length direction of the plug-in is determined by the intersections. In this way, the length direction of the plug-in and the center point of the plug-in can be accurately determined by combining the posture of the plug-in, which provides a guarantee for the accurate insertion of the plug-in into the socket.

[0137] S403. Based on the center point, the length direction, the preset plug width, and the preset plug length, determine the position of the plug on the heat map, and determine the position as the relative posture of the plug with respect to the tactile sensor.

[0138] Specifically, the preset plugin width and preset plugin length can be obtained by measuring the specific plugin model.

[0139] In specific implementations, for example, in one embodiment, when the plug-in is cylindrical, the height of the cylinder can be determined as a preset plug-in length, and the diameter of the cylinder can be determined as a preset plug-in width. As another example, in another embodiment, when the plug-in is a cuboid with a square cross-section, the side length of the square can be determined as a preset plug-in width, and the height of the cuboid can be determined as a preset plug-in length.

[0140] Furthermore, combining the preset plug width and preset plug length, the relative posture of the plug with respect to the tactile sensor is determined according to the plug's center point and length direction. That is, the plug is represented by an ellipse, with the aforementioned center point as the center of the ellipse, the aforementioned length direction as the major axis, the preset plug length as the length of the major axis, and the preset plug width as the length of the minor axis. Thus, an ellipse can be drawn on the heatmap, representing the plug, and the position of the ellipse represents the relative posture of the plug with respect to the tactile sensor.

[0141] The haptic-based robot assembly method provided in this embodiment first calculates the shear force applied to the plug-in at each target point and constructs a heat map to represent the distribution of the shear force on the clamping surface. This visually demonstrates the force situation when the plug-in contacts the clamping surface. Then, by analyzing the displacement changes of the target points, the center point and length direction of the plug-in on the heat map are determined. Finally, combined with the preset plug-in width and length, the complete position of the plug-in on the heat map can be accurately determined, reflecting the plug-in's relative posture to the haptic sensor. In this way, by accurately identifying the position and posture of the plug-in, the robot can more accurately control its operation, reducing errors and uncertainties in the assembly process.

[0142] Optionally, based on the above embodiments, determining the current target jamming state according to the shear force applied to the plug-in by each target point and the pre-established force model of the plug-in in various jamming states includes:

[0143] (1) For any jamming state, based on the pre-established force model of the plug-in in the jamming state, determine the ratio range between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction that the plug-in experiences in the jamming state; wherein, the ratio range between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction that the plug-in experiences in different jamming states is different.

[0144] In practice, as described above, different force models are preset for different blocking states. For example, in one embodiment, a force model a1 is preset for card group state a, a force model b1 is preset for card group state b, a force model c1 is preset for card group state c, and a force model d1 is preset for card group state d.

[0145] Furthermore, the force model is used to demonstrate the relationship between the horizontal force on the plug in the X-axis direction and the vertical force on the Z-axis direction, and this relationship is represented by a proportional range. It should be noted that once the plug's jamming state is determined, since the plug's tilt angle is fixed, the proportional range of the forces on the plug in the X-axis and Z-axis directions is also fixed. Therefore, different force models can be set for different jamming states.

[0146] (2) Based on the shear force applied to the plug at each target point, determine the current horizontal force on the plug in the X-axis direction and the current vertical force on the plug in the Z-axis direction.

[0147] Specifically, the shear force corresponding to each target point is analyzed, and the shear force is decomposed into the current horizontal force in the X-axis direction and the current vertical force in the Z-axis direction, based on the magnitude and direction of the shear force.

[0148] (3) Calculate the ratio of the current horizontal force to the current vertical force.

[0149] In practice, the ratio of the current horizontal force to the current vertical force is calculated to obtain the ratio corresponding to the current horizontal force and the current vertical force. For example, in one embodiment, the current horizontal force is 1N and the current vertical force is 2N, and the calculated ratio is 0.5.

[0150] (4) Determine the current target jamming state based on the ratio and the ratio range between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction experienced by the plug in various jamming states.

[0151] Specifically, the ratio of all horizontal forces in the X-axis direction and vertical forces in the Z-axis direction included in the jamming state is calculated, and the resulting numerical range is determined as the proportional range.

[0152] In specific implementation, for example, in one embodiment, combined with the above embodiment, the ratio range corresponding to the blocking state a is 0.01-0.31, the ratio range corresponding to the blocking state b is 0.32-0.61, the ratio range corresponding to the blocking state c is 0.62-0.91, and the ratio range corresponding to the blocking state d is 0.92-1.21. Therefore, when the ratio is 0.5, the target blocking state of the plug-in is determined to be blocking state b.

[0153] The tactile robot assembly method provided in this embodiment pre-establishes force models corresponding to different jamming states of the plug-in and determines the corresponding proportional ranges. This allows the robot to determine the jamming state based on the actual force of the plug-in, rather than relying solely on preset assembly paths or experience. This jamming state determination process based on force models enhances the robot's intelligence and adaptability, ensures the accuracy of jamming state determination, and enables the robot to better cope with complex and ever-changing assembly tasks.

[0154] The tactile robot assembly method provided in this embodiment enables the robot to more accurately determine the insertion direction and depth of plug-ins through tactile assembly. It can also overcome various obstacles in the assembly process, thereby significantly improving the success rate of assembly and ensuring the accuracy and reliability of plug-in insertion into the socket.

[0155] Corresponding to the aforementioned embodiment of a tactile-based robot assembly method, this application also provides an embodiment of a tactile-based robot assembly device.

[0156] An embodiment of a tactile-based robotic assembly device disclosed in this application can be applied to a tactile-based robotic assembly robot. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the tactile-based robotic assembly robot loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5 The diagram shown is a hardware structure diagram of a tactile robot assembly robot, which is part of the tactile robot assembly device described in this application. (Except for...) Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, the tactile-based robotic assembly robot in which the device is located in the embodiment may also include other hardware depending on the actual function of the tactile-based robotic assembly device, which will not be described in detail here.

[0157] Figure 6 This is a schematic diagram of the structure of a first embodiment of the tactile-based robotic assembly device provided in this application. Please refer to... Figure 6 The apparatus provided in this embodiment includes an acquisition module 701, a determination module 702, and a control module 703; wherein,

[0158] The acquisition module 701 is used to acquire the tactile image from the tactile sensor when jamming is detected during the insertion process, extract features from the tactile image, and determine a tactile flow image based on the feature map corresponding to the continuous frame tactile images; wherein, the tactile flow in the tactile flow image is used to characterize the displacement change of the target point;

[0159] The determining module 702 is used to determine the shear force applied to the plug at each target point based on the displacement change of each target point.

[0160] The determining module 702 is also used to determine the current target jamming state based on the shear force applied to the plug at each target point and the force model corresponding to various jamming states that are established in advance.

[0161] The determining module 702 is further configured to determine the relative posture of the plug-in with respect to the tactile sensor based on the displacement change of each target point and the shear force applied to the plug-in by each target point.

[0162] The determining module 702 is further configured to determine the amount of action of each action indicated by the control strategy based on the control strategy pre-determined for the target jamming state and the relative posture; wherein, different jamming states correspond to different control strategies, and the control strategy is used to indicate the action sequence of the gripper and the target posture after each action;

[0163] The control module 703 is used to control the robot according to the working sequence indicated by the control strategy and the amount of action of each action, so as to release the jamming and allow the plug to be inserted into the socket.

[0164] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0165] Please continue to refer to Figure 5 This application also provides a tactile-based robotic assembly robot, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0166] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods provided in this application.

[0167] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0168] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0169] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A tactile-based robot assembly method, characterized in that, The method is applied to a robot, wherein a tactile sensor is provided on the gripper of the robot, the tactile sensor is used to acquire tactile images of the gripping surface of the gripper, and target points are provided on the gripping surface. The robot is used to insert a plug held on the gripper into a socket; the method includes: When obstruction is detected during the insertion process, the tactile image is acquired from the tactile sensor, features are extracted from the tactile image, and a tactile flow image is determined based on the feature maps corresponding to consecutive frames of tactile images; wherein, the tactile flow in the tactile flow image is used to characterize the displacement change of the target point; The shear force applied to the plug at each target point is determined based on the displacement change of each target point. Based on the shear force applied to the plug at each target point and the pre-established force model corresponding to various jamming states, the current target jamming state is determined. The relative orientation of the plug-in to the tactile sensor is determined based on the displacement change of each target point and the shear force applied to the plug-in by each target point. Based on the control strategy pre-determined for the target jamming state and the relative attitude, the action amount of each action indicated by the control strategy is determined; wherein, different jamming states correspond to different control strategies, and the control strategy is used to indicate the action sequence of the gripper and the target attitude after each action; The robot is controlled according to the working sequence indicated by the control strategy and the amount of action of each action to release the obstruction and allow the plug to be inserted into the socket.

2. The method according to claim 1, characterized in that, Determining the relative attitude of the plug-in with respect to the tactile sensor based on the displacement change of each target point and the shear force exerted on the plug-in by each target point includes: Based on the shear force applied to the plug at each target point, a heat map is constructed to represent the distribution of the shear force on the clamping surface; wherein, the color intensity of a point on the heat map represents the magnitude of the shear force at that point on the clamping surface. Based on the heat map and the displacement changes of each target point, determine the center point of the plug-in on the heat map and the length direction of the plug-in on the heat map; Based on the center point, the length direction, the preset plug width, and the preset plug length, the position of the plug on the heat map is determined, and this position is determined as the relative posture of the plug with respect to the tactile sensor.

3. The method according to claim 2, characterized in that, Based on the heat map and the displacement changes of each target point, the center point of the plug-in on the heat map and the length direction of the plug-in on the heat map are determined, including: Based on the displacement change of each target point, find the boundary point with the largest displacement change on the boundary line of the clamping surface and the target point with the largest displacement change on the inner surface of the clamping surface other than the boundary line from the heat map. The target point is determined as the center point of the plug-in on the heat map, and the direction of the line connecting the target point and the boundary point is determined as the length direction of the plug-in on the heat map.

4. The method according to claim 2, characterized in that, Based on the heat map and the displacement changes of each target point, the center point of the plug-in on the heat map and the length direction of the plug-in on the heat map are determined, including: Based on the displacement change of each target point, find the target point with the largest displacement change on the inner surface of the clamping surface other than the boundary line from the heat map; Based on the shear force at each point in the heatmap, a target region is drawn on the heatmap, and the intersection of the target region with the boundary line of the heatmap is determined; wherein, the shear force at each point in the target region is greater than a preset threshold. When the target area does not intersect with the boundary line of the heat map, the target point is taken as the center point of the plug-in, and the movement direction of any target point is taken as the length direction of the plug-in. When the target area intersects with a boundary line of the heat map, the first target boundary point with the largest displacement change is found based on the displacement change of each target point on the boundary line, and the target point is used as the center point of the plug-in, and the direction of the line connecting the target point and the first target boundary point is used as the length direction of the plug-in. When two or three boundary lines of the heat map of the target area intersect, based on the displacement change of each target point on the two or three boundary lines, the second target boundary point with the largest displacement change on each boundary line is found. The line connecting the two target boundary points with the largest displacement change among the second target boundary points is determined as the length direction of the plug-in, and the intersection of the line segment passing through the target point and perpendicular to the line connecting the two target boundary points with the line connecting the two target boundary points is determined as the center point of the plug-in on the heat map.

5. The method according to claim 1, characterized in that, The step of determining the current target jamming state based on the shear force applied to the plug-in at each target point and the pre-established force model of the plug-in in various jamming states includes: For any jamming state, based on the pre-established force model of the plug-in corresponding to this type of jamming state, determine the ratio range between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction that the plug-in experiences under this type of jamming state; wherein, the ratio range between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction that the plug-in experiences under different jamming states is different. Based on the shear force exerted on the plug at each target point, determine the current horizontal force on the plug in the X-axis direction and the current vertical force on the plug in the Z-axis direction. Calculate the ratio of the current horizontal force to the current vertical force; Based on the ratio and the range of proportions between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction experienced by the plug-in under various jamming conditions, the current target jamming state is determined.

6. The method according to claim 1, characterized in that, The step of determining the shear force exerted on the plug at each target point based on the displacement change of each target point includes: The shear stress applied to the plug at each target point is determined based on the displacement change of each target point and the shear modulus corresponding to the clamping surface. The shear force exerted on the plug at each target point is determined based on the shear stress exerted on the plug at each target point.

7. The method according to claim 1, characterized in that, The determination of the haptic flow image based on the feature maps corresponding to consecutive frame haptic images includes: Select the feature points corresponding to the target points from the feature map; The displacement of the feature points between the feature maps corresponding to the haptic images in consecutive frames is calculated by optical flow method to obtain the displacement vector corresponding to each feature point; wherein, the displacement vector represents the direction and speed of movement of each feature point in the image space; The tactile flow image is obtained by visualizing the displacement vector corresponding to each feature point using a vector field method; wherein, in the tactile flow image, the arrow of each feature point indicates the movement direction and speed of the target point corresponding to that feature point.

8. A tactile-based robotic assembly device, characterized in that, The device is applied to a robot, wherein the robot's gripper is equipped with a tactile sensor for acquiring tactile images of the gripping surface of the gripper. Target points are provided on the gripping surface. The robot is used to insert a plug held by the gripper into a socket. The device includes an acquisition module, a determination module, and a control module. The acquisition module is used to acquire the tactile image from the tactile sensor when jamming is detected during the insertion process, extract features from the tactile image, and determine a tactile flow image based on the feature maps corresponding to consecutive frame tactile images; wherein, the tactile flow in the tactile flow image is used to characterize the displacement change of the target point; The determining module is used to determine the shear force applied to the plug at each target point based on the displacement change of each target point. The determining module is also used to determine the current target jamming state based on the shear force applied to the plug at each target point and the pre-established force model corresponding to various jamming states. The determining module is further configured to determine the relative posture of the plug-in with respect to the tactile sensor based on the displacement change of each target point and the shear force applied to the plug-in by each target point. The determining module is further configured to determine the amount of action of each action indicated by the control strategy based on the control strategy pre-determined for the target jamming state and the relative posture; wherein, different jamming states correspond to different control strategies, and the control strategy is used to indicate the action sequence of the gripper and the target posture after each action; The control module is used to control the robot according to the working sequence indicated by the control strategy and the amount of action of each action, so as to release the jamming and allow the plug to be inserted into the socket.

9. A robot, characterized in that, The robot is used to implement the steps of the method according to any one of claims 1-7.

Images