A haptics-based robotic assembly method, apparatus, and robot

Abstract

The application provides a haptics-based robot assembly method, device and robot. The method provided by the application comprises: when it is detected that there is a jam in an insertion process, obtaining a haptics image from a haptics sensor, performing feature extraction on the haptics image, and determining a haptics flow image based on feature maps corresponding to continuous frame haptics images; determining a shear force applied by each target point on the plug-in based on a displacement change amount of each target point; determining a target jam state currently in based on the shear force and a pre-established force model; determining a relative attitude based on the displacement change amount of each target point and the shear force; determining a motion amount of each action indicated by a control strategy based on the pre-determined control strategy and the relative attitude; and controlling the robot according to a working sequence indicated by the control strategy and the motion amount, so as to remove the jam and insert the plug-in into the insertion hole. The haptics-based robot assembly method, device and robot provided by the application are used to ensure that the plug-in is accurately inserted into the insertion hole.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot assembly, and in particular to a robot assembly method and device based on tactile sensation and a robot. BACKGROUND

[0002] With the rapid development of industrial automation, robot assembly technology is increasingly widely used in manufacturing. In the field of robot assembly, the "Peg in hole" task is one of the most typical challenges and has been extensively studied for a long time. This kind of task requires the robot to accurately insert the peg into the hole, involving complex spatial positioning and force control.

[0003] Existing robot assembly technology mainly relies on visual feedback for completion. For example, some studies obtain the position information of the target object through a visual sensor and insert the peg according to the visual image. However, the visual sensor is easily disturbed by factors such as light and occlusion in a complex environment, resulting in inaccurate positioning. In complex assembly tasks, the robot may frequently fail to assemble or damage components. Therefore, how to accurately insert the peg into the hole is still a problem to be solved in the field of robot assembly. SUMMARY

[0004] In view of this, the present application provides a robot assembly method and device based on tactile sensation and a robot to ensure accurate insertion of the peg into the hole.

[0005] Specifically, the present application is implemented through the following technical solutions:

[0006] The first aspect of the present application provides a robot assembly method based on tactile sensation. The method is applied to a robot, a tactile sensor is arranged on a gripper of the robot, the tactile sensor is used to obtain a tactile image of a clamping surface of the gripper, a target point is arranged on the clamping surface, and the robot is used to insert a peg clamped on the gripper into a hole. The method comprises the following steps:

[0007] When it is detected that the insertion process is blocked, the tactile image is obtained from the tactile sensor, feature extraction is performed on the tactile image, and a tactile flow image is determined based on the feature maps corresponding to the continuous frame tactile images; wherein the tactile flow in the tactile flow image is used to represent the displacement change of the target point.

[0008] According to the displacement change of each target point, the shear force exerted by each target point on the peg is determined.

[0009] According to the shear force exerted by each target point on the peg and the force model corresponding to each type of blocking state established in advance, the target blocking state currently in is determined.

[0010] determine a relative pose of the plug with respect to the tactile sensor according to the displacement variation of each target point and the shear force applied on the plug by each target point;

[0011] determine an action amount of each action indicated by the control strategy according to the relative pose and the control strategy determined in advance for the target jamming state; different jamming states correspond to different control strategies, and the control strategy is used to indicate a motion sequence of the gripper and a target pose after each action;

[0012] control the robot according to the working sequence indicated by the control strategy and the action amount of each action to remove the jamming and insert the plug into the insertion hole.

[0013] The second aspect of the application provides a tactile-based robot assembly device, which comprises an acquisition module, a determination module and a control module; wherein,

[0014] The acquisition module is used to acquire the tactile image from the tactile sensor when detecting that the insertion process exists jamming, perform feature extraction on the tactile image, and determine a tactile flow image based on the feature maps corresponding to the continuous frame tactile images; wherein, the tactile flow in the tactile flow image is used to represent the displacement variation of the target point;

[0015] The determination module is used to determine the shear force applied on the plug by each target point according to the displacement variation of each target point;

[0016] The determination module is also used to determine the target jamming state currently in according to the shear force applied on the plug by each target point and the force model corresponding to each type of jamming state established in advance;

[0017] The determination module is also used to determine a relative pose of the plug with respect to the tactile sensor according to the displacement variation of each target point and the shear force applied on the plug by each target point;

[0018] The determination module is also used to determine an action amount of each action indicated by the control strategy according to the relative pose and the control strategy determined in advance for the target jamming state; different jamming states correspond to different control strategies, and the control strategy is used to indicate a motion sequence of the gripper and a target pose after each action;

[0019] The control module is used to control the robot according to the working sequence indicated by the control strategy and the action amount of each action to remove the jamming and insert the plug into the insertion hole.

[0020] The third aspect of the present application provides a robot, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the robot implements the steps of the method according to any one of the first aspect of the present application when executing the program.

[0021] The method, device and robot provided by the present application are characterized in that: when it is detected that the insertion process is blocked, a tactile image is acquired from a tactile sensor, feature extraction is performed on the tactile image, and a tactile flow image is determined based on feature maps corresponding to continuous frame tactile images; then, the shear force applied by each target point on the plug-in is determined according to the displacement change of each target point; then, the target blocking state currently in is determined according to the shear force applied by each target point on the plug-in and the force model corresponding to each type of blocking state established in advance; then, the relative pose of the plug-in relative to the tactile sensor is determined according to the displacement change of each target point and the shear force applied by each target point on the plug-in; then, the action amount of each action indicated by the control strategy is determined based on the control strategy determined for the target blocking state and the relative pose; finally, the robot is controlled according to the working order indicated by the control strategy and the action amount of each action, so as to unblock and insert the plug-in into the insertion hole. In this way, the first aspect: the displacement change of the target point on the clamping surface can be accurately captured by performing feature extraction on the tactile image and generating the tactile flow image from the continuous frame tactile images, and then the resistance encountered by the plug-in in the insertion process can be accurately determined, which provides key data for subsequent pose adjustment and unblocking; the second aspect: the relative pose of the plug-in relative to the tactile sensor can be accurately obtained by combining the displacement change of the target point and the shear force information, which provides a basis for the pose adjustment of the robot; the third aspect: the most appropriate action can be made when adjustment is needed by using different control strategies corresponding to different target blocking states, so as to ensure the accuracy of the plug-in entering the insertion hole. In this way, the plug-in can be accurately controlled by tactile analysis, and the adaptability and success rate of the robot in the assembly process can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The flowchart of the method for robot assembly based on tactile provided by the present application is shown in the first embodiment;

[0023] Figure 2 The schematic diagram of the clamping surface is shown in an exemplary embodiment of the present application;

[0024] Figure 3 The flowchart of the method for robot assembly based on tactile provided by the present application is shown in the second embodiment;

[0025] Figure 4 The schematic diagram of the heat map is shown in an exemplary embodiment of the present application;

[0026] Figure 5 A hardware structure diagram of a robot where a haptics-based robot assembly device provided by the present application is located;

[0027] Figure 6 A structure schematic diagram of a first embodiment of the haptics-based robot assembly device provided by the present application. DETAILED DESCRIPTION

[0028] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, unless otherwise indicated, like numbers in the attached drawings refer to the same or similar elements. The following exemplary embodiments described in the following description are not meant to be limiting of the present application.

[0029] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the present application, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0030] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is to be further understood that the terms "comprise," "comprising," "comprises," and / or "comprising" when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0031] The following specific embodiments are given to introduce the technical solutions of the present application in detail.

[0032] Figure 1 A flowchart of a first embodiment of a haptics-based robot assembly method provided by the present application. Please refer to Figure 1 The method provided by the present embodiment can include:

[0033] S101, when it is detected that the insertion process is blocked, acquiring the haptic image from the haptic sensor, performing feature extraction on the haptic image, and determining a haptic flow image based on the feature maps corresponding to the continuous frame haptic images; wherein the haptic flow in the haptic flow image is used to represent the displacement change of the target point.

[0034] Before introducing the method provided by the present embodiment, a brief introduction to the application scenario is given:

[0035] The robot-based haptics assembly method provided in the embodiment is applied to a robot, which is a joint series robot, and the end of the robot is provided with two symmetrical clamping jaws, and the clamping surface of the clamping jaw is provided with a target point (a tiny convex point on the clamping surface is the target point). For example, the robot is a 7-degree-of-freedom collaborative robot, and the motion and force accuracy is 0:1 mm and 0:05 N.

[0036] Further, the clamping jaw of the robot is provided with a haptic sensor, specifically, the haptic sensor is arranged in the inside of the clamping jaw, and the haptic sensor can be a camera device, and the haptic image of the clamping surface of the clamping jaw can be acquired through the haptic sensor. For example, in an embodiment, one haptic sensor is arranged in each clamping jaw, the size of the haptic sensor is 20 mm to 23 mm, and the frame rate is 25 FPS, and an extremely fine, highly accurate and repeatable millimeter-level surface can be provided.

[0037] Further, referring to the foregoing description, the clamping surface of each clamping jaw is provided with a target point. For example, Figure 2 A schematic view of the clamping surface shown in an exemplary embodiment of the present application. Please refer to Figure 2 In the example shown in Figure 2 In the example shown in

[0038] Further, the specific shape of the plug and the socket is set according to actual needs, which is not limited in the embodiment. For example, in an embodiment, the plug is cylindrical and the socket is circular.

[0039] It should be noted that in the use process of the robot, the two clamping jaws are used to clamp the plug, the flat-nose pliers are used to fix the socket on the plane, and the plug is inserted into the socket by moving the mechanical arm of the robot to complete the assembly process of inserting the plug into the socket.

[0040] After introducing the specific application scenario, the method provided in the present application will be introduced as follows:

[0041] Specifically, the card block refers to the blocking of the plug when inserted into the socket, and the plug cannot be directly inserted into the socket. For example, the robot can detect whether there is a card block phenomenon by monitoring the assembly force, speed or other related parameters, and once it is detected that there is a card block, the robot will immediately enter the haptic adjustment mode (i.e. through the method provided in the present application, the adjustment is performed to insert the plug into the socket). For example, the robot determines that there is a card block when detecting a prompt instruction. The prompt instruction can be an instruction triggered by the user through the button on the robot when the user finds that there is a card block.

[0042] Specifically, in the step, when it is detected that the insertion process is blocked, the tactile image is acquired from the tactile sensor, feature extraction is performed on the tactile image, and a tactile flow image is determined based on the feature maps corresponding to the continuous frame tactile images.

[0043] The tactile image is a data image containing physical characteristics of the clamping surface. The tactile information of the clamping surface, such as contact force distribution and surface shape, can be determined through the tactile image. In a specific implementation, a target point change map of the clamping surface can be captured by a tactile sensor, and an image is constructed according to the change of the target point, and then the tactile image is obtained.

[0044] Further, the acquired tactile image is subjected to feature extraction to extract useful tactile features. For example, in an embodiment, the feature data such as the edge, shape and texture of the target point can be obtained through feature extraction. In a specific implementation, the feature extraction can be performed based on a traditional feature extraction method or a neural network-based feature extraction method, which is not limited in the embodiment.

[0045] Further, the tactile flow image can describe the position of the clamping jaw through the shape change and displacement of the target point.

[0046] In a specific implementation, in an embodiment, the continuous tactile images can be captured by the tactile sensor, and the feature maps corresponding to the continuous frames extracted are determined as the tactile flow to complete the tracking of the target point and record the displacement change of the target point over time.

[0047] Optionally, in a possible implementation, the process of determining the tactile flow image based on the feature maps corresponding to the continuous frame tactile images can include:

[0048] (1) selecting a feature point corresponding to the target point from the feature map.

[0049] In a specific implementation, the feature point corresponding to the target point can be selected from the feature map based on a traditional feature extraction method or a neural network-based extraction method. For example, in an embodiment, the feature point can be extracted based on the following formula:

[0050] ;

[0051] wherein, is the feature point corresponding to the target point, characterizes the feature map.

[0052] (2) calculating the displacement of the feature point between the feature maps corresponding to the continuous frame tactile images by an optical flow method to obtain a displacement vector corresponding to each feature point; wherein the displacement vector represents the moving direction and speed of each feature point in the image space.

[0053] In a specific implementation, the displacement change of each target point can be calculated by using an optical flow method. For example, in an embodiment, the tactile flow can be represented as a displacement vector of a target point and .

[0054] Further, the displacement vector of a target point can be calculated based on the following formula and :

[0055] ;

[0056] ;

[0057] wherein, and is the displacement vector of a target point, n is the number of target points, T represents the tactile flow, and is an array corresponding to all target points of two adjacent frames.

[0058] (3) visualizing the displacement vector corresponding to each feature point in the form of a vector field to obtain the tactile flow image; wherein in the tactile flow image, the arrow of each feature point represents the moving direction and speed of the target point corresponding to the feature point.

[0059] Specifically, each target point is displayed by visualizing the vector field, and the difference between and is determined as the displacement change of the target point corresponding to the target point.

[0060] In a specific implementation, each feature point in the tactile flow image represents a target point, and the moving direction and speed of the target point are reflected in the tactile flow image in the form of a tactile flow. For example, Figure 4 is a schematic diagram of a tactile flow image according to an exemplary embodiment of the present application. Please refer to Figure 4 The moving direction and speed of each target point can be represented by the tactile flow image.

[0061] The robot assembly method based on touch provided in the embodiment calculates the displacement between feature points in continuous frame tactile images by using an optical flow method to obtain the displacement vector corresponding to each feature point. In this way, the tactile flow image is generated based on the vector field, and the dynamic change of the plug-in during the assembly process can be directly observed, thereby providing a basis for subsequent plug-in insertion into the jack.

[0062] S102, determining the shear force applied by each target point on the plug-in according to the displacement change of each target point.

[0063] Specifically, due to the existence of the jam, the clamping jaw will move on the clamping surface, at this time, the target point on the clamping jaw and the insert will generate a shear force.

[0064] In a possible implementation, the shear force applied by each target point on the insert can be determined according to the corresponding relationship between the displacement change and the shear force. For example, in an embodiment, the displacement change of the target point A is 0, and it is determined that the shear force applied by the target point A is 0; the displacement change of the target point B is 0.01 mm, and it is determined that the shear force applied by the target point B is 0.1 N; the displacement change of the target point C is 0.03 mm, and it is determined that the shear force applied by the target point C is 0.3 N.

[0065] Optionally, in an embodiment, the specific implementation process of the step can include the following steps.

[0066] Step one: determining the shear stress applied by each target point on the insert according to the displacement change of each target point and the shear modulus corresponding to the clamping surface.

[0067] In a specific implementation, the shear stress applied by each target point on the insert can be calculated based on the following formula:

[0068] ;

[0069] wherein, is the shear stress, is the shear modulus, is the displacement change of the target point in the vertical plane.

[0070] Step two: determining the shear force applied by each target point on the insert according to the shear stress applied by each target point on the insert.

[0071] In a specific implementation, the total shear stress of the surface of a single sensor and the size of the shear force of each target point can be represented by the following formula:

[0072] ;

[0073] wherein, f is the shear force corresponding to the target point.

[0074] S103, determining the target jam state currently in according to the shear force applied by each target point on the insert and the force model corresponding to each type of jam state established in advance.

[0075] Specifically, the card blocking state includes a hole outside one-way card blocking state, a hole outside two-way card blocking state, a hole inside one-way card blocking state, and a hole inside two-way card blocking state. The hole outside one-way card blocking state corresponds to the first stress model, the hole outside two-way card blocking state corresponds to the second stress model, the hole inside one-way card blocking state corresponds to the third stress model, and the hole inside two-way card blocking state corresponds to the fourth stress model.

[0076] Specifically, in an embodiment, an assembly force is applied to the plug perpendicular to the end face of the plug , and a reaction force is generated at the two contact points and , which generates axial card blocking to the plug. At this time, the projection of the resultant force F of the reaction force on the horizontal plane always points to the center of the jack.

[0077] Further, through stress analysis, the force signals in the X and Z axis directions of the force sensor in the radial card blocking state , and the torque in the Y axis direction can be represented as:

[0078] ;

[0079] wherein, , and are the force sensor signals in the two-way card blocking state, is the torque in the Y axis direction, , and the force signals in the X and Z axis directions of the force sensor in the radial card blocking state.

[0080] Further, the force sensor signals in the two-way card blocking state , and can be represented as:

[0081] ;

[0082] wherein, , and are the force sensor signals in the two-way card blocking state, is the torque in the Y axis direction, , and the force signals in the X and Z axis directions of the force sensor in the radial card blocking state.

[0083] Further, is the component force of the resultant force F generated at the contact point of the bottom surface of the shaft part on the X and Z axes of the coordinate system; The force f generated by the side contact point of the shaft part in the coordinate system X, Z axis.

[0084] Further, The distance from the bottom contact point of the shaft part to the origin of the force sensor in the X axis direction. The distance from the bottom contact point of the shaft part to the origin of the force sensor in the X axis direction. The distance from the side contact point of the shaft part to the origin of the force sensor in the X axis direction.

[0085] Further, the two contact points A and B of the shaft end face are combined into a single point, and the shaft part is in a single-point axial jamming state, and the projection of the reaction force F in the XOY plane still points to the center of the hole. If the shaft part translates along the resultant force F, the shaft cylindrical side will contact the hole part, at which time the shaft part will be subjected to a resistance f, causing radial jamming, so that the shaft part enters a bidirectional jamming state.

[0086] Further, through force analysis, the force sensor signals 、 and in the radial jamming state can be represented as:

[0087] ;

[0088] wherein, , and are the force sensor signals in the bidirectional jamming state, is the torque in the Y axis direction, 、 are the force signals of the force sensor in the X, Z axis directions in the radial jamming state.

[0089] Further, the force sensor signals 、 and in the bidirectional jamming state can be represented as:

[0090] ;

[0091] wherein, , and are the force sensor signals in the bidirectional jamming state, is the torque in the Y axis direction, 、 are the force signals of the force sensor in the X, Z axis directions in the radial jamming state.

[0092] Through the above analysis of the mechanical model of each contact state of the shaft hole, when the shaft hole assembly is in the axial jamming state (whether single-point or multi-point contact), the projection of the reaction force F at the contact point in the XOY plane points to the center of the hole, and the shaft part moves in the opposite direction of the force to reach the bidirectional jamming state.

[0093] Further, different poses of the plug-in correspond to different jamming states, and by pre-establishing the force model corresponding to each jamming state, the corresponding relationship between the shear force and different types of jamming states can be further determined.

[0094] In specific implementation, for example, in an embodiment, the shear force on the plug-in is analyzed through the force model to determine that when the two contact points A and B of the shaft end face of the plug-in are combined into a single point (i.e., only one point on the bottom end of the plug-in is subjected to force), the jamming state of the plug-in at this time is determined as the axial single-point jamming state, and at this time, the reaction force of the contact point points to the center of the plug-in in the horizontal plane. For another example, in another embodiment, the shear force on the plug-in is analyzed through the force model to determine that after the plug-in partially enters the plug-in hole, multiple contact points exist, and the jamming state of the plug-in at this time is determined as the bidirectional jamming state in the hole, and at this time, the projection of the resultant force of the reaction forces of the contact points in the horizontal plane needs to point to the center of the plug-in hole.

[0095] S104, according to the displacement change amount of each target point and the shear force of each target point applied on the plug-in, determine the relative pose of the plug-in relative to the tactile sensor.

[0096] Specifically, the relative pose of the plug-in relative to the tactile sensor can be represented by the spatial position coordinates of the plug-in. After the coordinates of the plug-in are corresponded to the coordinates of the tactile sensor, the relative pose can be obtained.

[0097] In specific implementation, the topography of the plug-in can be drawn according to the specific numerical value of the displacement change amount, the specific orientation of the plug-in can be determined in combination with the shear force of each target point applied on the plug-in, and the relative pose of the plug-in and the sensor can be obtained in combination with the topography and the orientation of the plug-in. For example, in an embodiment, the target points with a displacement change amount greater than a displacement threshold value are determined as edge data of the plug-in, the shear force orientation of the target points with a shear force greater than a shear force threshold value is determined as the orientation of the plug-in, and the point with the largest displacement change amount is determined as the center of the plug-in. In combination with the edge data and the orientation data of the plug-in, the relative pose of the plug-in is obtained.

[0098] S105, based on the control strategy determined in advance for the target jamming state and the relative pose, determine the action amount of each action indicated by the control strategy; wherein different control strategies correspond to different jamming states, and the control strategy is used to indicate the action sequence of the clamping jaw and the target pose after each action.

[0099] Specifically, the control strategy is used to indicate the direction and sequence of the movement of the gripper. Through the control strategy, it can be determined how the robot moves, rotates, places the object, etc. For example, in an embodiment, the control strategy a is: first move left, then move backward, and then move obliquely downward.

[0100] Further, the relative pose is adjusted through the control strategy, so that the plug adjusted in movement can be inserted into the socket. In other words, through the control strategy, the sequence of the plug in the relative pose can be changed to the target pose, and the final target pose can be directly inserted into the socket.

[0101] In specific implementation, for example, in an embodiment, the two-way blocking outside the hole: the shaft is lifted a little bit upward and then performs spiral downward movement; the one-way outside the hole: the shaft is advanced to the center of the hole and then performs spiral insertion; the two-way inside the hole: the shaft is lifted a little bit upward to get rid of the blocking and then performs spiral downward movement; the one-way inside the hole: directly performs spiral trajectory insertion.

[0102] S106, control the robot according to the working sequence indicated by the control strategy and the action amount of each action to remove the blocking and insert the plug into the socket.

[0103] In specific implementation, for example, in an embodiment, when the target blocking state is the single-point axial blocking state, the control strategy is to first move backward and then move left. Then, according to the relative pose at this time, the action amount of the backward movement is determined to be 3 cm, and the action amount of the left movement is determined to be 2 cm. At this time, through the gripper control, the plug is first moved backward by 3 cm and then moved left by 2 cm, so that the plug can be removed from the blocking and inserted downward into the socket.

[0104] The robot assembly method, device and robot based on haptics provided in the embodiment detect that there is a jam in the insertion process, obtain a haptic image from a haptic sensor, perform feature extraction on the haptic image, determine a haptic flow image based on features corresponding to continuous frame haptic images, then determine shear forces applied by each target point on the plug based on displacement changes of the target points, further determine a target jam state based on the shear forces applied by each target point on the plug and a force model corresponding to each type of jam state established in advance, then determine a relative pose of the plug relative to the haptic sensor based on the displacement changes of each target point and the shear forces applied by each target point on the plug, determine action amounts of actions indicated by a control strategy based on the relative pose and the control strategy determined in advance for the target jam state, and finally control the robot according to a working sequence indicated by the control strategy and the action amounts of the actions to remove the jam and insert the plug into the socket. In this way, the first aspect: the displacement changes of the target points on the clamping surface can be accurately captured by performing feature extraction on the haptic image and generating the haptic flow image from continuous frame haptic images, and the resistance encountered by the plug in the insertion process can be accurately determined, thereby providing key data for subsequent pose adjustment and jam removal. The second aspect: the relative pose of the plug relative to the haptic sensor can be accurately obtained by combining the displacement changes of the target points and the shear force information, thereby providing a basis for pose adjustment of the robot. The third aspect: the most appropriate action can be taken when adjustment is needed by using different control strategies corresponding to different target jam states, thereby ensuring the accuracy of the plug entering the socket. In this way, the resistance can be accurately obtained by obtaining the haptic flow image, thereby providing a basis for subsequent adjustment, and the pose can be determined on this basis, thereby ensuring that the position of the plug is the same as the actual position. The plug can be reasonably adjusted by combining the obtained position of the plug and the control strategy, thereby improving the adaptability and success rate of the robot in the assembly process.

[0105] Figure 3 The flowchart of the robot assembly method based on haptics provided in the embodiment is provided. Figure 3 On the basis of the above embodiment, the step of determining the relative pose of the plug relative to the haptic sensor based on the displacement changes of each target point and the shear forces applied by each target point on the plug comprises:

[0106] S401, constructing a heat map for representing distribution of the shear forces on the clamping surface based on the shear forces applied by each target point on the plug; wherein the color of a point on the heat map represents the size of the shear force at the point on the clamping surface.

[0107] Specifically, the shear forces corresponding to the target points are visualized to obtain a heat map, each region on the heat map corresponds to a target point, and the color of the region corresponds to the shear force on the target point.

[0108] In a specific implementation, the two-dimensional array corresponding to the target point shear force data can be converted into a visualized heat map based on a heat map drawing tool.

[0109] Figure 4 A schematic diagram of a heat map is shown for an exemplary embodiment of the present application. Please refer to Figure 4 According to the shear force applied to each target point, a heat map of the shear force on the clamping surface is obtained, in which Figure 4 The deep red part is a target point area with a larger shear force, and the light yellow part is a target point area with a smaller shear force.

[0110] It can be understood that the plug applies a downward force to the target point under the action of gravity, and the target point thus generates a shear force. According to the shape of the plug, different target points have different shear forces due to different contact positions of the plug. According to the heat map corresponding to the shear force, the posture of the plug can be determined.

[0111] S402, according to the heat map and the displacement change of each target point, determining the center point of the plug on the heat map and the length direction of the plug on the heat map.

[0112] In a specific implementation, for example, in an embodiment, the topography of the plug is determined according to the heat map, the length direction of the plug is determined according to the extension direction of the deep red color, and the center point of the plug is determined according to the displacement change of the target point.

[0113] Optionally, in an embodiment, the specific implementation process of this step includes:

[0114] (1) According to the displacement change of each target point, find the boundary point with the largest displacement change on the boundary line of the clamping surface and the target point with the largest displacement change on the internal surface of the clamping surface except the boundary line from the heat map.

[0115] Specifically, the boundary line of the clamping surface is the outermost 4 edges, please continue to refer to Figure 3 The four columns of target points in the top row, the bottom row, the leftmost row and the rightmost row are the boundary lines of the heat map, and the parts corresponding to other target points are the internal surface.

[0116] In a specific implementation, the displacement change of each target point corresponding to the boundary line can be obtained, the point with the largest displacement change is determined as the boundary point with the largest displacement change on the boundary line of the heat map, and the displacement change of each target point corresponding to the internal surface is obtained. The point with the largest displacement change is determined as the target point with the largest displacement change on the internal surface of the heat map.

[0117] (2) Determine the target point as the center point of the plug on the heat map, and determine the direction of the line connecting the target point and the boundary point as the length direction of the plug on the heat map.

[0118] Specifically, the plug can be approximated as a cylinder, and the center point is the geometric center point of the cylinder, and the line connecting the target point and the boundary point is the length direction of the plug on the heat map.

[0119] In a specific implementation, for example, in an embodiment, the boundary point is determined as a, and the target point is determined as β, and the line connecting a and β is the length direction of the plug on the heat map.

[0120] It can be understood that the diameter and length of the plug can be obtained by measurement, and the position of the plug can be determined according to the diameter and length of the plug, in combination with the center point and the length direction of the plug.

[0121] The method for robot assembly based on haptics provided in the embodiment can accurately find the target point with the largest displacement change on the clamping surface by analyzing the displacement change of each target point on the heat map, and determine the target point as the center point of the plug on the heat map, which not only considers the overall displacement of the plug, but also combines the deformation information of the plug surface under stress, so that the center point can be accurately and efficiently found. In the determination of the length direction of the plug, the length direction of the plug is accurately obtained by connecting the target point and the boundary point with the largest displacement change. In this way, the position and posture of the plug are identified by analyzing the deformation information of the plug surface under stress, so that even for plugs with complex shapes and various sizes, the robot can accurately identify and control the plugs through the method, thereby enhancing the adaptability of the robot to complex assembly scenes and enabling the robot to handle more types of assembly tasks.

[0122] Further, in another possible implementation, the specific implementation process of the step can include:

[0123] (1) According to the displacement change of each target point, find the target point with the largest displacement change on the internal surface of the clamping surface except the boundary line from the heat map.

[0124] In a specific implementation, the displacement changes of all target points on the internal surface are compared to determine the target point with the largest displacement change, and the target point is determined.

[0125] (2) According to the shear force of each point in the heat map, draw a target area on the heat map, and determine the intersection of the target area and the boundary line of the heat map; wherein the shear force of each point in the target area is greater than a preset threshold.

[0126] Further, after obtaining the displacement variation of the target points, the shear force of each point in the heat map can be calculated.

[0127] In a specific implementation, in this step, the heat map is drawn according to the shear force of each point, and the intersection between the target region and the boundary line is determined according to the heat map. For example, in an embodiment, the target region and the boundary line have one intersection point. For another example, in another embodiment, the target region and the boundary line can have no intersection point, or have two intersection points, or have three intersection points.

[0128] (3) When the target region and the boundary line of the heat map do not intersect, the target point is taken as the center point of the plug-in, and the moving direction of any target point is taken as the length direction of the plug-in.

[0129] Specifically, the moving direction of the target point can be determined based on the displacement variation of the target point. In a specific implementation, the target point is determined as the center point of the plug-in, and the displacement corresponding to the moving direction of any target point is selected as the length direction of the plug-in.

[0130] (4) When the target region intersects with one boundary line of the heat map, the first target boundary point with the largest displacement variation is found according to the displacement variation of each target point on the boundary line, the target point is taken as the center point of the plug-in, and the direction of the line connecting the target point and the first target boundary point is taken as the length direction of the plug-in.

[0131] In a specific implementation, the displacement variation of each target point on the intersected boundary is obtained, and the target point with the largest displacement variation is determined as the first target boundary point. The line connecting the first target boundary point and the target point is obtained, and the length direction is obtained. For example, in an embodiment, the target region intersects with the left boundary of the heat map, the target point corresponding to the largest displacement variation of the left boundary is C, the point D with the largest displacement variation on the inner surface is determined, and the line connecting C and D is determined as the length direction of the plug-in.

[0132] (5) When the target region intersects with two or three boundary lines of the heat map, the second target boundary point with the largest displacement variation on each boundary line is found according to the displacement variation of each target point on the two or three boundary lines, the line connecting the two target boundary points with larger displacement variation in the second target boundary points is determined as the length direction of the plug-in, and the intersection point of the line segment passing through the target point and being perpendicular to the line and the line is determined as the center point of the plug-in on the heat map.

[0133] In a specific implementation, the displacement variation of each target point on the intersected boundary is obtained, and the target point with the largest displacement variation is determined as the second target boundary point. The two second target boundary points with the largest values in the second target boundary points are connected, and the line connecting the two points is determined as the length direction of the plug-in.

[0134] Further, a perpendicular line is drawn from the target point to the line connecting the two second target boundary points, and the intersection point of the perpendicular line and the line connecting the two second target boundary points is determined as the center point of the plug-in.

[0135] In a specific implementation, for example, in an embodiment, the target region intersects the upper boundary and the right boundary of the heat map, a target point corresponding to the maximum displacement variation of the upper boundary is E, a target point corresponding to the maximum displacement variation of the right boundary is F, a target point with the maximum displacement variation on the internal surface is determined as D, the line EF is determined as the length direction of the plug-in, a perpendicular line is drawn from the point D to the line EF, and the intersection point is G, which is the center point of the plug-in. For another example, in another embodiment, the target region intersects the upper boundary, the right boundary, and the lower boundary of the heat map, a target point corresponding to the maximum displacement variation of the upper boundary is E, a target point corresponding to the maximum displacement variation of the right boundary is F, a target point corresponding to the maximum displacement variation of the lower boundary is H, two points with the maximum displacement variation among the points E, F, and H are determined as the points F and H, a target point with the maximum displacement variation on the internal surface is determined as D, the line FH is determined as the length direction of the plug-in, a perpendicular line is drawn from the point D to the line FH, and the intersection point is I, which is the center point of the plug-in.

[0136] The method for robot assembly based on touch provided in this embodiment can determine the number of intersection points of the target region and the boundary of the heat map through the heat map, since the target region represents the posture of the plug-in, the length direction of the plug-in is determined through the intersection points, so that the length direction of the plug-in and the center point of the plug-in can be accurately determined in combination with the posture of the plug-in, which provides a guarantee for subsequent accurate insertion of the plug-in into the jack.

[0137] S403, determining the position of the plug-in on the heat map according to the center point and the length direction, and preset plug-in width and preset plug-in length, and determining the position as the relative posture of the plug-in relative to the touch sensor.

[0138] Specifically, the preset plug-in width and the preset plug-in length can be measured according to the specific model of the plug-in.

[0139] In a specific implementation, for example, in an embodiment, when the plug-in is a cylinder, the height of the cylinder can be determined as the preset plug-in length, and the diameter of the cylinder can be determined as the preset plug-in width. For another example, in another embodiment, when the plug-in is a cuboid with a square cross section, the side length of the square can be determined as the preset plug-in width, and the height of the cuboid can be determined as the preset plug-in length.

[0140] Further, in combination with the preset plug width and the preset plug length, the relative pose of the plug relative to the tactile sensor is determined according to the center point of the plug and the length direction of the plug. That is, the plug is represented by an ellipse, the center point is taken as the center point of the ellipse, the length direction is taken as the long axis direction, the preset plug length is taken as the length of the long axis of the ellipse, and the preset plug width is taken as the length of the short axis of the ellipse. In this way, an elliptical shape representing the plug can be plotted on the heat map, and the position of the elliptical shape is the relative pose of the plug relative to the tactile sensor.

[0141] The method provided by the embodiment based on touch can intuitively show the force condition of the plug when the plug is in contact with the clamping surface by calculating the shear force of each target point on the plug and constructing a heat map to represent the distribution of the shear force on the clamping surface, and then determining the center point and the length direction of the plug on the heat map by analyzing the displacement change of the target point. Finally, in combination with the preset plug width and length, the complete position of the plug on the heat map can be accurately determined, reflecting the relative pose of the plug relative to the tactile sensor. In this way, by accurately identifying the position and pose of the plug, the robot can more accurately control its operation and reduce errors and uncertainties in the assembly process.

[0142] Optionally, on the basis of the above embodiment, the current target jamming state is determined according to the shear force of each target point on the plug and the pre-established force model corresponding to each type of jamming state of the plug, and the method comprises the following steps of:

[0143] (1) For any type of jamming state, the ratio range between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction of the plug in this type of jamming state is determined according to the pre-established force model corresponding to this type of jamming state. The ratio range between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction of the plug in different jamming states is different.

[0144] In specific implementation, the above description is referred to, and different force models are pre-established for different jamming states. For example, in an embodiment, a force model a1 is pre-established for the card group state a, a force model b1 is pre-established for the card group state b, a force model c1 is pre-established for the card group state c, and a force model d1 is pre-established for the card group state d.

[0145] Further, the force model is used to show the correlation between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction of the plug, and the correlation is represented by the ratio range. It should be noted that when the jamming state of the plug is determined, the ratio range between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction of the plug is fixed due to the fixed inclination angle of the plug. Therefore, different force models can be set for different jamming states.

[0146] (2) determining a current horizontal force in the X-axis direction and a current vertical force in the Z-axis direction that the insert is currently subjected to according to the shear force of each target point on the insert.

[0147] Specifically, the shear force of each target point is subjected to stress analysis, and the shear force is split into a current horizontal force in the X-axis direction and a current vertical force in the Z-axis direction according to the size and direction of the shear force.

[0148] (3) calculating a ratio of the current horizontal force and the current vertical force.

[0149] In specific implementation, the ratio of the current horizontal force and the current vertical force is calculated to obtain the ratio corresponding to the current horizontal force and the current vertical force. For example, in an embodiment, the current horizontal force is 1N, the current vertical force is 2N, and the ratio is calculated to be 0.5.

[0150] (4) determining a current corresponding target jamming state according to the ratio and a proportional range between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction that the insert is subjected to in each type of card jamming state.

[0151] Specifically, the ratio is calculated for all the horizontal forces in the X-axis direction and the vertical forces in the Z-axis direction included in the card jamming state, and the obtained numerical range is determined as the proportional range.

[0152] In specific implementation, for example, in combination with the above embodiment, the proportional range corresponding to the card jamming state a is 0.01-0.31, the proportional range corresponding to the card jamming state b is 0.32-0.61, the proportional range corresponding to the card jamming state c is 0.62-0.91, and the proportional range corresponding to the card jamming state d is 0.92-1.21, so when the ratio is 0.5, it is determined that the target jamming state of the insert at this time is the card jamming state b.

[0153] The method for robot assembly based on haptics provided in this embodiment can make the robot determine the card jamming state according to the actual force of the insert, instead of relying only on the preset assembly path or experience, by pre-establishing the force model corresponding to the insert in different card jamming states and determining the corresponding proportional range, which enhances the intelligence and adaptability of the robot, ensures the accuracy of the card jamming state determination, and enables the robot to better cope with complex and variable assembly tasks.

[0154] The method for robot assembly based on haptics provided in this embodiment can enable the robot to more accurately determine the insertion direction and depth of the insert, and can also overcome various obstacles in the assembly process, thereby significantly improving the success rate of assembly and ensuring the accuracy and reliability of the insertion of the insert into the socket.

[0155] Corresponding to the foregoing embodiment of the robot assembly method based on tactile sensation, the application also provides an embodiment of a robot assembly device based on tactile sensation.

[0156] The embodiment of the robot assembly device based on tactile sensation provided by the application can be applied to a robot assembly machine based on tactile sensation. The device embodiment can be realized by software, or realized by hardware or a combination of software and hardware. Taking software realization as an example, as a logically meaningful device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor of the robot assembly machine based on tactile sensation where the device is located. From the hardware level, as shown in the figure, it is a hardware structure diagram of the robot assembly machine based on tactile sensation where the robot assembly device based on tactile sensation is located. In addition to the processor, the memory, the network interface, and the non-volatile memory shown in the figure, the robot assembly machine based on tactile sensation where the device is located in the embodiment usually includes other hardware according to the actual function of the robot assembly device based on tactile sensation, and this will not be described again. Figure 5 Figure 5

[0157] Figure 6 This is a structural schematic diagram of the embodiment one of the robot assembly device based on tactile sensation provided by the application. Please refer to Figure 6 The device provided by the embodiment includes an acquisition module 701, a determination module 702, and a control module 703; wherein,

[0158] The acquisition module 701 is used for acquiring the tactile image from the tactile sensor when it is detected that the insertion process is blocked, performing feature extraction on the tactile image, and determining the tactile flow image based on the feature maps corresponding to the continuous frame tactile images; wherein the tactile flow in the tactile flow image is used to represent the displacement change of the target point.

[0159] The determination module 702 is used for determining the shear force applied on the plug by each target point according to the displacement change of each target point.

[0160] The determination module 702 is also used for determining the target blocking state currently located according to the shear force applied on the plug by each target point and the force model corresponding to each type of blocking state established in advance.

[0161] The determination module 702 is also used for determining the relative attitude of the plug relative to the tactile sensor according to the displacement change of each target point and the shear force applied on the plug by each target point.

[0162] ​​The determining module 702 is further configured to determine an action amount of each action indicated by the control strategy based on the control strategy determined in advance for the target jamming state and the relative pose; different control strategies correspond to different jamming states, and the control strategy is used to indicate a motion sequence of the gripper and a target pose after each action.

[0163] The control module 703 is configured to control the robot according to the working sequence indicated by the control strategy and the action amount of each action to remove the jamming and insert the plug-in into the socket.

[0164] The device of the embodiment can be used to execute the steps of the method embodiment. Figure 1 The steps of the method embodiment are similar to the specific implementation principles and implementation processes, and thus will not be described here.

[0165] Please continue to refer to Figure 5 The application also provides a haptics-based robot assembly robot, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the steps of any one of the methods provided in the first aspect of the application.

[0166] The application also provides a computer-readable storage medium having a computer program stored thereon, and the program is executed by the processor to implement the steps of any one of the methods provided in the application.

[0167] The implementation processes of the functions and roles of each unit in the above device are specifically described in the implementation processes of the corresponding steps in the above method, and thus will not be described here.

[0168] For the device embodiment, since it basically corresponds to the method embodiment, the related parts are described in the part of the method embodiment. The device embodiments described above are only illustrative, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place or distributed on multiple network units. According to actual needs, some or all of the modules can be selected to achieve the purposes of the application. Those skilled in the art can understand and implement without creative labor.

[0169] The above description is only the preferred embodiment of the application, and is not used to limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application should be included in the protection scope of the application.

Claims

1. A tactile-based robot assembly method, characterized in that, The method is applied to a robot, wherein a tactile sensor is provided on the gripper of the robot, the tactile sensor is used to acquire tactile images of the gripping surface of the gripper, and target points are provided on the gripping surface. The robot is used to insert a plug held on the gripper into a socket; the method includes: When obstruction is detected during the insertion process, the tactile image is acquired from the tactile sensor, features are extracted from the tactile image, and a tactile flow image is determined based on the feature maps corresponding to consecutive frames of tactile images; wherein, the tactile flow in the tactile flow image is used to characterize the displacement change of the target point; The shear force applied to the plug at each target point is determined based on the displacement change of each target point. Based on the shear force applied to the plug at each target point and the pre-established force model corresponding to various jamming states, the current target jamming state is determined. Based on the displacement changes of each target point and the shear force exerted on the plug by each target point, the relative attitude of the plug with respect to the tactile sensor is determined; wherein, based on the shear force exerted on the plug by each target point, a heat map representing the distribution of the shear force on the clamping surface is constructed; wherein, the color intensity of a point on the heat map represents the magnitude of the shear force at that point on the clamping surface; based on the heat map and the displacement changes of each target point, the center point of the plug on the heat map and the length direction of the plug on the heat map are determined, based on the displacement changes of each target point... The method involves identifying the boundary point with the largest displacement change on the boundary line of the clamping surface and the target point with the largest displacement change on the internal surface of the clamping surface other than the boundary line from the heat map. The target point is then determined as the center point of the plug-in on the heat map, and the direction of the line connecting the target point and the boundary point is determined as the length direction of the plug-in on the heat map. Based on the center point, the length direction, and the preset plug-in width and length, the position of the plug-in on the heat map is determined, and this position is defined as the relative posture of the plug-in to the tactile sensor. Based on the control strategy pre-determined for the target jamming state and the relative attitude, the action amount of each action indicated by the control strategy is determined; wherein, different jamming states correspond to different control strategies, and the control strategy is used to indicate the action sequence of the gripper and the target attitude after each action; The robot is controlled according to the working sequence indicated by the control strategy and the amount of action of each action to release the obstruction and allow the plug to be inserted into the socket.

2. The method according to claim 1, characterized in that, Based on the heat map and the displacement changes of each target point, the center point of the plug-in on the heat map and the length direction of the plug-in on the heat map are determined, including: Based on the displacement change of each target point, find the target point with the largest displacement change on the inner surface of the clamping surface other than the boundary line from the heat map; Based on the shear force at each point in the heatmap, a target region is drawn on the heatmap, and the intersection of the target region with the boundary line of the heatmap is determined; wherein, the shear force at each point in the target region is greater than a preset threshold. When the target area does not intersect with the boundary line of the heat map, the target point is taken as the center point of the plug-in, and the movement direction of any target point is taken as the length direction of the plug-in. When the target area intersects with a boundary line of the heat map, the first target boundary point with the largest displacement change is found based on the displacement change of each target point on the boundary line, and the target point is used as the center point of the plug-in, and the direction of the line connecting the target point and the first target boundary point is used as the length direction of the plug-in. When two or three boundary lines of the heat map of the target area intersect, based on the displacement change of each target point on the two or three boundary lines, the second target boundary point with the largest displacement change on each boundary line is found. The line connecting the two target boundary points with the largest displacement change among the second target boundary points is determined as the length direction of the plug-in, and the intersection of the line segment passing through the target point and perpendicular to the line connecting the two target boundary points with the line connecting the two target boundary points is determined as the center point of the plug-in on the heat map.

3. The method according to claim 1, characterized in that, The step of determining the current target jamming state based on the shear force applied to the plug-in at each target point and the pre-established force model of the plug-in in various jamming states includes: For any jamming state, based on the pre-established force model of the plug-in corresponding to this type of jamming state, determine the ratio range between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction that the plug-in experiences under this type of jamming state; wherein, the ratio range between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction that the plug-in experiences under different jamming states is different. Based on the shear force exerted on the plug at each target point, determine the current horizontal force on the plug in the X-axis direction and the current vertical force on the plug in the Z-axis direction. Calculate the ratio of the current horizontal force to the current vertical force; Based on the ratio and the range of proportions between the horizontal force in the X-axis direction and the vertical force in the Z-axis direction experienced by the plug-in under various jamming conditions, the current target jamming state is determined.

4. The method according to claim 1, characterized in that, The step of determining the shear force exerted on the plug at each target point based on the displacement change of each target point includes: The shear stress applied to the plug at each target point is determined based on the displacement change of each target point and the shear modulus corresponding to the clamping surface. The shear force exerted on the plug at each target point is determined based on the shear stress exerted on the plug at each target point.

5. The method according to claim 1, characterized in that, The determination of the haptic flow image based on the feature maps corresponding to consecutive frame haptic images includes: Select the feature points corresponding to the target points from the feature map; The displacement of the feature points between the feature maps corresponding to the haptic images in consecutive frames is calculated by optical flow method to obtain the displacement vector corresponding to each feature point; wherein, the displacement vector represents the direction and speed of movement of each feature point in the image space; The tactile flow image is obtained by visualizing the displacement vector corresponding to each feature point using a vector field method; wherein, in the tactile flow image, the arrow of each feature point indicates the movement direction and speed of the target point corresponding to that feature point.

6. A tactile-based robotic assembly device, characterized in that, The device is applied to a robot, wherein the robot's gripper is equipped with a tactile sensor for acquiring tactile images of the gripping surface of the gripper. Target points are provided on the gripping surface. The robot is used to insert a plug held by the gripper into a socket. The device includes an acquisition module, a determination module, and a control module. The acquisition module is used to acquire the tactile image from the tactile sensor when jamming is detected during the insertion process, extract features from the tactile image, and determine a tactile flow image based on the feature maps corresponding to consecutive frame tactile images; wherein, the tactile flow in the tactile flow image is used to characterize the displacement change of the target point; The determining module is used to determine the shear force applied to the plug at each target point based on the displacement change of each target point. The determining module is also used to determine the current target jamming state based on the shear force applied to the plug at each target point and the pre-established force model corresponding to various jamming states. The determining module is further configured to determine the relative posture of the plug-in with respect to the tactile sensor based on the displacement change of each target point and the shear force exerted on the plug-in by each target point; wherein, based on the shear force exerted on the plug-in by each target point, a heat map representing the distribution of the shear force on the clamping surface is constructed; wherein, the color intensity of a point on the heat map represents the magnitude of the shear force at that point on the clamping surface; based on the heat map and the displacement change of each target point, the center point of the plug-in on the heat map and the length direction of the plug-in on the heat map are determined, based on the displacement change of each target point. The displacement change is determined by finding the boundary point with the largest displacement change on the boundary line of the clamping surface and the target point with the largest displacement change on the inner surface of the clamping surface other than the boundary line on the heat map. The target point is determined as the center point of the plug on the heat map, and the direction of the line connecting the target point and the boundary point is determined as the length direction of the plug on the heat map. Based on the center point, the length direction, and the preset plug width and preset plug length, the position of the plug on the heat map is determined, and this position is determined as the relative posture of the plug with respect to the tactile sensor. The determining module is further configured to determine the amount of action of each action indicated by the control strategy based on the control strategy pre-determined for the target jamming state and the relative posture; wherein, different jamming states correspond to different control strategies, and the control strategy is used to indicate the action sequence of the gripper and the target posture after each action; The control module is used to control the robot according to the working sequence indicated by the control strategy and the amount of action of each action, so as to release the jamming and allow the plug to be inserted into the socket.

7. A robot, characterized in that, The robot is used to implement the steps of the method according to any one of claims 1-5.

Images