A fingertip type visual tactile sensing device

Abstract

The present invention discloses a fingertip type visual tactile sensing device, which belongs to the field of robotics and artificial intelligence, and includes a contact module that directly contacts an object and is used to obtain the physical properties of the object through tactile perception; an imaging lighting module that is used to provide a light source and capture a contact image when the contact module contacts the object; a control module that is used to adaptively control a manipulator according to the physical properties of the object and the contact image; and a mechanical module that is used to integrate the sensing device and the manipulator. The present invention is based on the precise measurement of marker displacement, and integrates a pre-grasp controller and a stability adjustment controller in a unified control framework to enhance the manipulator's recognition of the physical properties of the object. This makes the manipulator grip more accurately.

Description

Technical Field

[0001] The invention belongs to the field of robotics and artificial intelligence, and in particular relates to a fingertip type visual-tactile sensing device. Background Art

[0002] Stable grasping is a common behavior for humans, but it is full of challenges for robots. This is due to the large number of receptors distributed under the skin of the hand, which provides rich and sensitive tactile perception. This high sensitivity is particularly prominent in the perception of tiny slips. For example, when grasping fragile and heavy objects, the hand can obtain contact feedback (roughness, hardness and weight) with the object and adjust the grasping strength in time. At the same time, the hand has dynamic and continuous stability adjustment, just like an athlete continuously exerting force to counter the opponent's strength in a tug-of-war game. This grasping method is defined as adaptive grasping, and its advantage lies in the non-invasive operation of objects. According to the physical properties of the manipulator, it is divided into rigid hands and soft hands. Manipulators without tactile properties implement general grasping with a fixed preset grasping force in specific application scenarios. Considering the versatility of grasping, users will set a larger grasping force in the initial stage. For rigid hands, this method meets the grasping of rigid objects / heavy objects, but is less applicable to the grasping of soft / fragile objects. Researchers introduced a variable stiffness mechanism to improve the maximum grasping force of soft hands, so as to take into account both flexible grasping and rigid grasping. However, without tactile feedback, soft hands cannot avoid the problem of contact deformation of flexible objects caused by grasping. Visual tactile sensors have redefined the grasping method of manipulators, but the field of visual tactile sensing focuses more on the recognition and adjustment of grasping posture under fixed grasping force. The high sensitivity and flexibility of visual tactile sensors have unique application potential in the adaptive grasping of objects with different properties. It can be seen as a combination of soft hands and tactile sensing, while maintaining strong stiffness. Although some researchers have also tried to control the grasping strength on rigid objects, there is a lack of experimental verification on soft and fragile objects. The biggest challenge comes from the precise tracking of micro-deformations and the understanding of the physical properties of objects. In addition, the research community lacks attention to the integration and compatibility of visual tactile sensors with manipulators. It is just an independent sensing device that is temporarily integrated into the manipulator for experimental verification. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a fingertip type visual-tactile sensing device to solve the above problems existing in the prior art.

[0004] To achieve the above-mentioned object, the present invention provides a fingertip-type visual-tactile sensing device, comprising a contact module, an imaging lighting module, a control module and a mechanical module;

[0005] The contact module is in direct contact with the object and is used to obtain the physical properties of the object through tactile perception;

[0006] The imaging lighting module is used to provide a light source and capture a contact image when the contact module contacts an object;

[0007] The control module is used to adaptively control the manipulator according to the physical properties of the object and the contact image;

[0008] The mechanical module is used to integrate the sensor device and the manipulator.

[0009] Preferably, the contact module comprises an elastomer, a marking layer, a coating and an acrylic plate;

[0010] The elastomer is prepared by a lamination process using acrylic tape and adhered to the acrylic plate;

[0011] The material of the coating includes aluminum foil, and the aluminum foil is granulated by a wire drawing process to obtain the coating;

[0012] The marking layer is obtained by printing a marking array on water transfer paper using a laser printer.

[0013] Preferably, the marking array is 9×9.

[0014] Preferably, the imaging lighting module comprises an LED filament and a camera;

[0015] The LED filament surrounds the elastic body to provide global illumination;

[0016] The camera adopts a focus-adjustable micro camera module.

[0017] Preferably, the control module comprises a first control unit and a second control unit;

[0018] The first control unit is used to control the manipulator to perform pre-grasping;

[0019] The second control unit is used to adjust the stability of the manipulator according to the physical properties and contact image of the object obtained by pre-grasping.

[0020] Preferably, the pre-grasping method comprises: performing non-invasive grasping by setting a displacement threshold, and judging whether the manipulator forms sufficient micro-contact with the object by using an average marked displacement.

[0021] Preferably, the calculation expression of the marker displacement variable is:

[0022]

[0023] Where D pg,t is the marker mean displacement, Indicates the x-axis pixel displacement of the marker point in the current frame. Indicates the x-axis pixel displacement of the marker point in the previous frame. Indicates the y-axis pixel displacement of the marker point in the current frame. Indicates the y-axis pixel displacement of the marker point in the previous frame.

[0024] Preferably, the stability adjustment method comprises:

[0025] Take the pre-grasped image and capture the object slippage degree and trend through the marker array in the image. When the slippage degree is greater than the threshold, the manipulator performs stability adjustment.

[0026] When the slip degree is less than or equal to the threshold, the manipulator maintains the original grasping strength and determines whether the slip trend tends to be stable. When it tends to be stable, the stability adjustment is completed.

[0027] Preferably, the stability regulation is controlled by a PD controller.

[0028] Preferably, the mechanical module comprises an integrated camera board, an elastomer fixing groove and a housing.

[0029] Compared with the prior art, the present invention has the following advantages and technical effects:

[0030] The present invention realizes a highly integrated and compatible fingertip-shaped visual-tactile sensor and an adaptive grasping strategy. Based on the precise measurement of marker displacement, the pre-grasping controller and the stability adjustment controller are integrated into a unified control framework to enhance the robot's understanding of the physical properties of the object. The robot arm can grasp more accurately. The brushed coating of the present invention has high ductility and wear resistance, and can withstand different degrees of deformation or high-frequency contact. The present invention improves the versatility of adaptive grasping based on visual-tactile sensing through pre-grasping and stability adjustment, and has high robustness for objects with different physical properties. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0032] Figure 1 is a structural diagram of a device according to an embodiment of the present invention;

[0033] Figure 2 A flow chart for making a contact module according to an embodiment of the present invention;

[0034] Figure 3 This is a crawling strategy diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0035] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0036] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] Embodiment 1

[0038] In this embodiment, a fingertip-type visual-tactile sensing device is provided, including a contact module, an imaging lighting module, a control module and a mechanical module;

[0039] The contact module is in direct contact with the object and is used to obtain the physical properties of the object through tactile perception;

[0040] The imaging lighting module is used to provide a light source and capture a contact image when the contact module contacts the object;

[0041] The control module is used to adaptively control the manipulator according to the physical properties of the object and the contact image;

[0042] The mechanical module is used to integrate the sensor device and the robot.

[0043] Further, the contact module includes an elastomer, a marking layer, a coating and an acrylic plate;

[0044] The elastomer was prepared by a lamination process using acrylic tape and bonded to the acrylic sheet;

[0045] The coating material includes aluminum foil, which is granulated by a wire drawing process to obtain the coating;

[0046] The marking layer is obtained by printing a marking array on water transfer paper using a laser printer.

[0047] Specifically, the production process is as follows: Figure 2 As shown, the elastomer is prepared by lamination process using 2 layers of 2mm thick acrylic tape and bonded to acrylic. A layer of 3 micron thick aluminum foil is attached to the 1mm thick acrylic tape by gold plating process. The foil coating is granulated by wire drawing process to obtain a coating similar to spraying effect, but thinner. The marking layer is printed by laser printer on water transfer paper, where the marking array is 9×9, and its sensing density is similar to human touch density. The marking layer is transferred to the surface of the elastomer by transfer process, and finally a coating layer - acrylic tape is attached.

[0048] Further, the imaging lighting module includes an LED filament and a camera;

[0049] LED filaments surround the elastic body to provide global illumination;

[0050] The camera uses a focus-adjustable micro camera module.

[0051] Specifically, considering compactness, this work uses LED filaments. The width of the filament is 2mm, but the thickness is only 1mm. Warm white light color is used to provide global illumination around the elastomer. The camera uses a focus-adjustable micro camera module with an integrated imaging adjustment system to cope with the deviation of the imaging distance design and the defocusing problem caused by contact deformation. Therefore, inside the fingertip-type visual haptics, there is no longer a preset imaging buffer space (improving compactness).

[0052] Further, the control module includes a first control unit and a second control unit;

[0053] The first control unit is used to control the manipulator to perform pre-grasping;

[0054] The pre-fetch method is as follows:

[0055] At present, adaptive grasping based on visual touch sensing cannot adapt to the physical properties of objects in time. For soft / fragile / light objects, the manipulator has formed a relatively stable contact state with the object under micro-contact. For unknown objects, the manipulator explores the process from no contact to micro-contact with high frequency and short control increments. Set a smaller marker displacement threshold , because the small threshold has a higher sensitivity in micro-deformation judgment, which is conducive to the non-invasive grasping of fragile objects / flexible objects / light objects. The average marker displacement is used as the discrimination index because it represents the state of global deformation and ensures that the manipulator forms sufficient micro-contact with the object.

[0056] Pre-fetching is a continuous operation process, which is defined as a time series in this task. The robot determines micro-contact by dynamically tracking the displacement of the marker. The displacement variables of the marker are as follows:

[0057]

[0058] Where D pg,t is the marker mean displacement, Indicates the x-axis pixel displacement of the marker point in the current frame. Indicates the x-axis pixel displacement of the marker point in the previous frame. Indicates the y-axis pixel displacement of the marker point in the current frame. Indicates the y-axis pixel displacement of the marker point in the previous frame.

[0059] The time series is based on the initial marker image. The setting of the threshold needs to take into account the marker positioning fluctuation caused by image processing. This project designs a marker displacement monitor F m The marker is monitored and the control increment of the manipulator is output based on the above displacement variables. The task definition is as follows:

[0060]

[0061]

[0062] In the formula, G pg,t is the capture time series, d c is the gripper control increment, T pg is the crawl threshold.

[0063] The images of each time series are judged in turn. When the displacement variable is less than the threshold, it is determined that the grasp is in non-contact / insufficient micro-contact, and then the manipulator moves according to the preset control increment. When the displacement variable is greater than the threshold, it is determined to be in sufficient micro-contact.

[0064] The second control unit is used to adjust the stability of the manipulator according to the physical properties and contact image of the object obtained by pre-grasping.

[0065] The stability adjustment method is as follows:

[0066] After being grasped, the object that has reached stable grasping without adjustment is confirmed to have flexible / fragile / light properties. However, the pre-grasp strength is not enough to support heavier objects (mainly rigid objects). The object slips downward relative to the manipulator, so the grasping strength needs to be fine-tuned in time. The degree and trend of the slip can be captured by the marker array. In the pre-grasp stage, the object is in a steady state, so the contact feedback is a static measurement. In the stability adjustment stage, there is relative motion between the object and the manipulator, which is a dynamic measurement process.

[0067] Define the stability regulation as a time series The object is pulled upward and leaves the platform, and the time left for the manipulator to adjust the whole process is limited. Therefore, the stability adjustment uses a smaller displacement threshold to determine whether the slip meets the adjustment conditions.

[0068] The stability adjustment takes the stable pre-captured marker image as the initial frame, and measures the displacement of the marker through the previous and next frame images. Accordingly, a slip monitor F is designed. s The crawling status monitoring is as follows:

[0069]

[0070] When the displacement is greater than the threshold, the manipulator performs stability adjustment. Otherwise, the manipulator maintains the original grasping strength. The manipulator will hover for a while, and when the slip tends to be stable (fluctuating within the threshold range), it is considered that the stability adjustment is over.

[0071] As the force increases, the degree of slippage fluctuates and generally shows a downward trend. Therefore, the magnitude of each adjustment is different. The displacement of the marker in the current frame image is used as the main feedback, and the influence of the displacement change trend on the intensity control is also considered. Therefore, a PD controller is used, in which P (proportional term) adjusts the grasping strength according to the current maximum displacement of the marker, and D (differential term) suppresses the control signal according to the rate of change of the displacement. The output of the PD control strategy is as follows:

[0072] d ci =k p e[G sa，t ]+k d (e[G sa,t ]-e[G sa，t-1 ])

[0073]

[0074] Among them, K p and K d is the coefficient of the proportional term and the differential term, D n is the maximum marker displacement.

[0075] Under PD control, the proportional term continuously enhances the control signal, causing the manipulator to apply greater gripping strength. When the displacement change rate decreases, the differential term will generate a reverse control signal to suppress over-grasping in advance. PD control helps reduce overshoot and oscillation of gripping strength. It enables the manipulator to respond quickly and adapt to different gripping requirements.

[0076] Furthermore, the mechanical module includes an integrated camera board, an elastomer fixing groove and a housing.

[0077] Specifically, considering the convenience of assembly, the camera board, the elastomer fixing groove, and the outer shell are designed as an integrated whole, and the upper cover and the fixings are assembled separately. The overall size of the sensor (including the elastomer) is 38×21×23mm, of which the thickness of the elastomer protruding from the upper cover is 1mm. After testing, it was found that 1mm was not enough to support deep contact with the object, resulting in interruption of continuous monitoring of the contact state. The imaging adjustment system allows the sensor to adjust the contact depth between the elastomer and the object by changing the thickness of the acrylic. Therefore, without changing the structure of the sensor, the acrylic thickness is increased by 2mm. The above design shows a basic version of the visual-tactile fingertip. This embodiment also has different forms of connectors for easy integration with the manipulator.

[0078] The present invention achieves the following positive effects:

[0079] This paper proposes a high-performance coating - wire drawing process. It enhances the robustness of the coating and provides the possibility for the commercialization of visual tactile sensors. The skin has high tensile strength and low elastic modulus to adapt to irregular deformation and transmit the microscopic details of objects. The wire drawing process improves the durability of thin coatings while maintaining high sensitivity and spatial resolution at the micrometer scale. The rich tactile features are highly learnable, and the mapping relationship between tactile information and object properties is constructed in combination with artificial feature extraction or deep learning. In this study, silver / aluminum foil was selected instead of copper foil to obtain more stable coating properties, including high adhesion, adaptive deformation and crack resistance. Due to the low metal bonding strength and hardness, the wire drawing process converts silver foil and aluminum foil into nanoscale particles, similar to spray coatings. Extrusion and wear tests show that the wire drawing coating has high ductility and wear resistance, and can withstand different degrees of deformation or high-frequency contact.

[0080] The present invention realizes a highly integrated and compatible fingertip-shaped visual tactile sensor and an adaptive grasping strategy. The visual tactile fingertip is suitable for mainstream manipulators, such as Franka, Robotiq, and DH Robotics. Secondly, the contact state is dynamically monitored by measuring the displacement of the marker by the tactile sensing module. This depends on the accurate detection, positioning and matching of the marker information. The proposed two-stage grasping strategy, such as Figure 3 As shown, it includes a pre-grasping stage and stability adjustment, taking into account both flexible / rigid grasping, and focusing on improving the robot's understanding of physical properties.

[0081] In the pre-grasping stage, the pre-grasping controller helps the manipulator adapt to the physical properties of the object during tactile exploration, which avoids the manipulator from over-grasping soft / fragile objects (non-invasive protection). In the stability adjustment stage, the stability adjustment controller provides dynamic adjustment for the relative slip between the gripper and the object, especially for rigid heavy objects. Considering the different emphasis of the controller on the contact state, the average displacement and maximum displacement are used as discrimination indicators for monitoring the global contact and local contact states respectively. Tactile feedback is the basis for the manipulator to achieve adaptive grasping, and pre-grasping is the prerequisite for the manipulator to recognize the properties of the object. The two controllers have significant effects on the adjustment of the grasping strength in their respective stages of adaptive grasping. The combination of the two improves the versatility of adaptive grasping based on visual-tactile sensing, and has high robustness for objects with different physical properties.

[0082] Compactness: On the one hand, it simplifies useless internal structures and integrates infrequently replaced parts; on the other hand, it maximizes the utilization of the sensing space and abandons the reserved imaging buffer space.

[0083] High compatibility: The sensor body is used as a standard part, and different connectors are equipped to adapt to different types of manipulator integration. The overall size of the sensor is similar to that of a human finger, providing a hardware foundation for the tactile integration of dexterous hands.

[0084] Robustness: Spraying relies on high pressure to enhance the mechanical connection between the coating material and the elastomer, making the coating itself have a certain wear resistance. Metal foil can be used as a high-performance coating material to obtain high-quality tactile mapping, but the gold plating operation on flexible surfaces has the risk of wrinkling caused by local deformation of the metal foil and easy rupture under large deformation. In order to pursue fine tactile mapping, researchers will abandon the protective layer, which means that the coating needs to take on the role of a protective layer. For metal foil, surface strengthening treatment may be an effective way to solve its robustness defects.

[0085] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A fingertip type visual tactile sensing device, characterized in that: It includes a contact module, an imaging lighting module, a control module and a mechanical module; The contact module is in direct contact with the object and is used to obtain the physical properties of the object through tactile perception; The imaging lighting module is used to provide a light source and capture a contact image when the contact module contacts an object; The control module is used to adaptively control the manipulator according to the physical properties of the object and the contact image; The control module includes a first control unit and a second control unit; The first control unit is used to control the manipulator to perform pre-grasping; The second control unit is used to adjust the stability of the manipulator according to the physical properties and contact image of the object obtained by pre-grasping; The stability adjustment method comprises: Take the pre-grasped image and capture the object slippage degree and trend through the marker array in the image. When the slippage degree is greater than the threshold, the manipulator performs stability adjustment. When the slip degree is less than or equal to the threshold, the manipulator maintains the original grasping strength and determines whether the slip trend tends to be stable. When it tends to be stable, the stability adjustment is completed; The stability regulation is controlled by a PD controller; The mechanical module is used to integrate the sensor device and the manipulator.

2. The fingertip type visual-tactile sensor device according to claim 1, characterized in that: The contact module includes an elastomer, a marking layer, a coating and an acrylic plate; The elastomer is prepared by a lamination process using acrylic tape and adhered to the acrylic plate; The material of the coating includes aluminum foil, and the aluminum foil is granulated by a wire drawing process to obtain the coating; The marking layer is obtained by printing a marking array on water transfer paper using a laser printer.

3. The fingertip type visual-tactile sensor device according to claim 2, characterized in that: The marker array is 9×9.

4. The fingertip type visual-tactile sensor device according to claim 1, characterized in that: The imaging lighting module includes an LED filament and a camera; The LED filament surrounds the elastic body to provide global illumination; The camera adopts a focus-adjustable micro camera module.

5. The fingertip type visual-tactile sensor device according to claim 1, characterized in that: The pre-grasping method includes: performing non-invasive grasping by setting a displacement threshold, and judging whether the manipulator forms sufficient micro-contact with the object by using an average mark displacement.

6. The fingertip type visual-tactile sensor device according to claim 5, characterized in that: The calculation expression of the marker displacement variable is: Where D pg,t is the marker mean displacement, Indicates the x-axis pixel displacement of the marker point in the current frame. Indicates the x-axis pixel displacement of the marker point in the previous frame. Indicates the y-axis pixel displacement of the marker point in the current frame. Indicates the y-axis pixel displacement of the marker point in the previous frame.

7. The fingertip type visual-tactile sensor device according to claim 1, characterized in that: The mechanical module includes an integrated camera board, an elastomer fixing groove and a housing.

Images