Finger-shaped visual tactile sensor, sensing method and dexterous hand

Abstract

The invention provides a finger-shaped visual tactile sensor, a sensing method and a dexterous hand. The finger-shaped visual tactile sensor includes: a transparent elastic body; the transparent supporting body is positioned below the transparent elastic body and is used for supporting the transparent elastic body; the LED lamp group is positioned near the transparent support body; the camera is used for acquiring an image of the transparent elastomer; and the shell is used for mounting the transparent supporting body, the LED lamp group and the camera. The finger-shaped visual tactile sensor can better adapt to a dexterous hand to execute fine operation.

Description

Technical Field

[0001] The present application belongs to the field of sensor technology and relates to a visual-tactile sensor, and in particular to a finger-shaped visual-tactile sensor, a sensing method and a dexterous hand. Background Art

[0002] Fine tactile perception can provide accurate force feedback and contact information, enabling dexterous hands to manipulate and dynamically adjust tiny objects in complex environments. Compared with traditional electromagnetic sensors, visual tactile sensors have a higher density of tactile information and can more accurately characterize the contact state, position, and tiny movements of objects, thereby achieving fine touch-based manipulation. However, current visual tactile sensors still have certain shortcomings. Summary of the invention

[0003] The purpose of the present application is to provide a finger-shaped visual-tactile sensor, a perception method and a dexterous hand, wherein the finger-shaped visual-tactile sensor can better adapt to the dexterous hand to perform fine operations.

[0004] In a first aspect, an embodiment of the present application provides a finger-shaped visual-tactile sensor, which includes: a transparent elastomer; a transparent support body, located near the transparent elastomer and used to support the transparent elastomer; an LED light group, located below the transparent support body; a camera, used to capture an image of the transparent elastomer; and a housing, used to install the transparent support body, the LED light group and the camera.

[0005] In an implementation of the first aspect, the outer surface of the transparent elastomer is covered with a reflective coating.

[0006] In an implementation of the first aspect, the reflective coating is coated with a functional coating on the outside.

[0007] In an implementation of the first aspect, the transparent support body is also used as a light-guiding structure to conduct the light of the LED lamp group so that the light is evenly distributed on the reflective coating.

[0008] In an implementation of the first aspect, the LED light group includes a red light group, a green light group and / or a blue light group welded on a flexible circuit board.

[0009] In an implementation of the first aspect, after installation is completed, the green light group and the blue light group are located on the same side of the red light group, and the angle between the red light group and the green light group and the angle between the red light group and the blue light group are both 90°.

[0010] In an implementation of the first aspect, the housing includes a lamp group mounting groove, and the lamp group mounting groove is used to install the LED lamp group.

[0011] In an implementation of the first aspect, the LED lamp group is pre-packaged into the housing using epoxy resin.

[0012] In a second aspect, an embodiment of the present application provides a perception method, which is applied to the finger-shaped visual-tactile sensor described in any one of the first aspect of the embodiments of the present application, and the method includes: the LED light group emits light to the transparent elastomer; when the transparent elastomer is deformed, the camera collects an image of the transparent elastomer; and the processor processes the image to obtain a perception result.

[0013] In a third aspect, an embodiment of the present application provides a dexterous hand, wherein the dexterous hand comprises the finger-shaped visual-tactile sensor described in any one of the first aspect of the embodiments of the present application.

[0014] The finger-shaped visual-tactile sensor provided in the embodiment of the present application adopts a design close to the shape of a finger. Compared with a spherical or planar structure, this approach can better adapt to the dexterous hand to perform fine operations.

[0015] In the embodiment of the present application, the design of the transparent support body can be optimized by curvature control to make the lighting more uniform.

[0016] In the embodiment of the present application, a three-dimensional optical lighting design can be adopted, and the red LED light group, the blue LED light group and the green LED light group are arranged at a 90° angle. This method can increase the angle difference of the three-color illumination, thereby providing richer color information.

[0017] In the embodiment of the present application, the LED light group can be pre-packaged with epoxy resin, and the transparent support body can be directly placed on the LED light group, so there is no need to set up an additional support structure for the transparent support body, which is conducive to further reducing the overall size of the sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1A Shown is a schematic diagram of the structure of a finger-shaped visual-tactile sensor provided in an embodiment of the present application.

[0019] Figure 1B Shown is a schematic diagram of an exploded view of a finger-shaped visual-tactile sensor provided in an embodiment of the present application.

[0020] Figure 2 Shown is a schematic diagram of the structure of the reflective coating, the functional coating and the transparent elastomer in the embodiment of the present application.

[0021] Figure 3Shown is a schematic diagram of the structure of the LED lamp group and the transparent support body in an embodiment of the present application.

[0022] Figure 4 Shown is a schematic diagram of the structure of the housing in an embodiment of the present application.

[0023] Figure 5 Shown is a schematic diagram of a camera being assembled to a housing in an embodiment of the present application.

[0024] Figure 6 Shown is a flowchart of the perception method provided in an embodiment of the present application.

[0025] Figure 7 Shown is a flowchart of image processing in an embodiment of the present application.

[0026] Figure 8 Shown is another flowchart for performing image processing in an embodiment of the present application.

[0027] Component number description

[0028] 1 Finger-Shaped Visual Tactile Sensor

[0029] 11. Transparent elastomer

[0030] 12. Transparent support

[0031] 13 LED light set

[0032] 131 Red Light Group

[0033] 132 Green Light Group

[0034] 133 Blue Light Group

[0035] 14 Camera

[0036] 15. Housing

[0037] 151, 152, 153 Lighting installation slot

[0038] 154 Line crossing

[0039] 155 Camera fixing slot

[0040] 16 Reflective coating

[0041] 17 Functional coating DETAILED DESCRIPTION

[0042] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0043] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0044] The size of visual-tactile sensors has become one of the key factors restricting their application in dexterous hands. In some technical solutions, claw-shaped sensors are provided to improve their ability to operate in narrow spaces by reducing thickness, but the length problem is not fully considered in the design of these sensors, and the integration of their planar sensing interface with dexterous hands is still difficult. In other technical solutions, visual-tactile sensors designed with fingers as the design orientation are provided. However, the size of such sensors is close to or larger than the human thumb, which will make the dexterous hand too bulky and affect its flexibility. In some other technical solutions, by using optical fiber to achieve lighting and image capture, the sensor size is reduced to less than 15mm, making it possible to apply visual-tactile sensors to dexterous hands. However, when optical fiber is integrated into a dexterous hand as a signal transmission medium, it may be difficult to withstand its working state of multiple bending movements.

[0045] In addition, the appearance design of the visual tactile sensor is also crucial. Some visual tactile sensors use a flat sensing surface or a cylindrical + flat sensing surface design. In contrast, in the design of cylindrical + spherical sensing surface, due to the relatively simple arrangement of the camera field of view, it has certain advantages in size reduction. Although the overall size of this type of sensor is close to that of a human finger, its shape is still different from that of a human fingertip. When imitating human fingertips for fine manipulation of objects, these sensors can usually only rely on the cylindrical side walls for operation, which limits their operational flexibility to a certain extent.

[0046] Furthermore, the miniaturization of visual tactile sensors usually comes at the expense of image quality. In some technical solutions, the miniaturization of visual tactile sensors is achieved by focusing on the optimization of the camera's viewing angle. For example, by adjusting variables such as the camera installation angle and the position of the plane mirror, the optimal combination is calculated to minimize the thickness and volume of the sensor. However, although this type of optimization improves the structural compactness of the sensor, how to maintain the quality of the tactile signal while reducing the size is still one of the technical problems that need to be solved urgently.

[0047] In response to some or all of the above-mentioned problems, embodiments of the present application provide a finger-shaped visual-tactile sensor, a sensing method, and a dexterous hand.

[0048] Figure 1A and Figure 1B The structure diagram of the finger-shaped visual tactile sensor provided by the embodiment of the present application is shown. Figure 1A and 1B As shown, the finger-shaped visual-tactile sensor 1 includes a transparent elastic body 11 , a transparent support body 12 , an LED light group 13 , a camera 14 and a housing 15 .

[0049] The transparent elastic body 11 is a component of the sensing unit of the finger-shaped visual tactile sensor 1 and is used to contact external objects. When an object applies pressure, the transparent elastic body 11 will deform, and the size, depth and range of the deformation directly reflect the strength, position and shape of the touch. In addition, the transparent elastic body 11 can have elasticity and deformation recovery capabilities. After the external force is removed, the transparent elastic body 11 can return to its original state. This recovery performance allows the finger-shaped visual tactile sensor 1 to maintain its sensitivity and accuracy during multiple touches.

[0050] The transparent support body 12 is located below the transparent elastomer 11 and serves as a structural support to carry and fix the transparent elastomer 11, ensuring that the transparent elastomer 11 only produces a slight deformation on the upper surface under the action of external force and preventing the transparent elastomer 11 from being excessively deformed due to contact pressure.

[0051] Exemplarily, the transparent support 12 can be made of a strong and highly transparent material, such as glass, transparent polymer, etc. These materials can provide mechanical support and minimize the attenuation and distortion of light during optical transmission.

[0052] The LED light group 13 is located near the transparent support body 12 and is used for emitting light into the transparent elastic body 11 .

[0053] In some implementations, the LED light group 13 may be located below the transparent support 12 .

[0054] In some implementations, the LED light groups 13 may be distributed around the transparent support 12 .

[0055] It should be noted that the above only lists two configuration methods of the LED light groups 13 by way of example, but the present application is not limited thereto.

[0056] In some implementations, the position, quantity, and light-emitting angle of the LED light group 13 can be adjusted to achieve uniform illumination of the entire transparent elastic body 11 , thereby improving the perception effect of the finger-shaped visual tactile sensor 1 .

[0057] The camera 14 is used to collect images of the transparent elastic body 11. Specifically, when the transparent elastic body 11 is deformed, the deformation area changes the reflection, scattering and refraction characteristics of light, forming unique optical features. These optical features can be captured by the camera 14, thereby providing basic data for image processing. In some implementations, the camera 14 can be a miniature camera.

[0058] The housing 15 is used to mount the transparent support 12 , the LED light group 13 and the camera 14 .

[0059] See also Figure 2 In some implementations, the outer surface of the transparent elastomer 11 is covered with a reflective coating 16. The reflective coating 16 can significantly improve the detectability of contact deformation, and by enhancing the reflection effect of light, the finger-shaped visual tactile sensor 1 can be clearer and more accurate when collecting optical signals in the contact area. The reflective coating 16 can not only effectively amplify the concave-convex and texture features during contact and significantly improve the resolution of small deformations, but also improve the perception of complex surface details by the finger-shaped visual tactile sensor 1, so that the surface microstructure, texture direction and depth information can be more realistically reflected. In addition, the reflective coating 16 can improve the anti-interference ability of the finger-shaped visual tactile sensor 1 under weak light or complex lighting conditions, thereby optimizing the accuracy of 3D surface imaging and ensuring that the finger-shaped visual tactile sensor 1 can still reliably collect accurate tactile and visual data in a variety of environments. Therefore, the presence of the reflective coating 16 enables the finger-shaped visual tactile sensor 1 to perform better in scenarios requiring high-precision surface detection and recognition, and is suitable for a variety of high-precision application scenarios such as precision machinery assembly, grasping and operating small objects.

[0060] In some implementations, the reflective coating 16 is coated with a functional coating 17. The functional coating 17 can be used to enhance the performance of the finger-shaped visual tactile sensor 1, for example, it can be used to enhance the wear resistance, puncture resistance, friction and / or dust resistance of the finger-shaped visual tactile sensor 1.

[0061] Exemplarily, the functional coating 17 may include a black protective coating, which is used to effectively shield the interference of external light on internal imaging, and prevent external ambient light sources such as sunlight, indoor lighting, etc. from interfering with the optical signal collection of the sensor.

[0062] In some implementations, the transparent support 12 is also used as a light-guiding structure to conduct the light emitted by the LED light group 13 so that the light is evenly distributed on the reflective coating 16 .

[0063] In some implementations, the structure of the transparent support 12 can be optimized so that the minimum curvature of the transparent support 12 is greater than the curvature threshold. In this way, the overall curvature distribution of the transparent support 12 can be improved, and the light emitted by the LED lamp group 13 can be prevented from being locally focused, thereby ensuring a uniform lighting effect.

[0064] See also Figure 3 In some implementations, the LED light group 13 includes a red light group 131, a green light group 132 and / or a blue light group 133 soldered on a flexible circuit board.

[0065] In some implementations, after installation, the green light group 132 and the blue light group 133 are located on the same side of the red light group 131, and the angle between the red light group 131 and the green light group 132 and the angle between the red light group 131 and the blue light group 133 are both 90°. Because different colors of light have different refraction, reflection and scattering characteristics when propagating in the transparent elastomer 11. In the embodiment of the present application, the LED light group 13 can provide richer color information by adopting the above-mentioned arrangement, thereby more clearly showing the deformation characteristics of the contact area, such as subtle bumps, edges and texture changes. This can effectively improve the contrast and detail perception ability of the finger-shaped visual tactile sensor 1 in complex tactile scenes, so that the finger-shaped visual tactile sensor 1 can more accurately capture subtle contact changes.

[0066] It should be noted that the green light group 132 and the blue light group 133 are located on the same side of the red light group 131 and the angles between the red light group 131 and the green light group 132 and the blue light group 133 are all 90°, which is only one feasible method of the embodiment of the present application, but the present application is not limited to this.

[0067] See also Figure 4 In some implementations, the housing 15 includes lamp assembly mounting grooves 151 to 153, and the lamp assembly mounting grooves 151 to 153 are used to mount the LED lamp assembly 13. Specifically, the lamp assembly mounting groove 151 can be used to mount the green lamp assembly 132, the lamp assembly mounting groove 152 can be used to mount the red lamp assembly 131, and the lamp assembly mounting groove 153 can be used to mount the blue lamp assembly 133.

[0068] In some implementations, the LED lamp group 13 can be pre-packaged into the lamp group mounting groove of the housing 15 using epoxy resin.

[0069] In some implementations, the housing 15 may further include a wire port 154. The wire port 154 is used to lead out a power line of the LED lamp group 13.

[0070] Exemplarily, pre-packaging the LED light group 13 into the light group installation groove of the housing 15 may include: placing the LED light group 13 welded on the flexible circuit board into the light group installation groove of the housing 15. Pre-blocking the wire opening 154 with silica gel or other temporarily blocked materials to prevent glue from flowing out when pouring glue into the LED light group 13. Fixing the LED light group 13 in the light group installation groove with glue, during this process, it should be ensured that the glue can evenly cover the LED light group 13.

[0071] See also Figure 5 In some implementations, the housing 15 may further include a camera fixing slot 155 . The camera fixing slot 155 is used to install the camera 14 .

[0072] Exemplarily, fixing the camera 14 to the housing 15 includes: placing the camera 14 into the camera fixing groove 155 , using its structure to complete the limiting, and fixing the camera 14 with glue.

[0073] In some implementations, the transparent elastomer 11 can be prepared by a mold, and then the reflective coating 16 and the functional coating 17 are sprayed on the outer surface thereof.

[0074] Exemplarily, a marker point may be marked inside the transparent elastic body 11 .

[0075] For example, the transparent elastic body 11 and the transparent support body 12 may be bonded together to form a module.

[0076] In some implementations, the bottom surface shape of the transparent support 12 is consistent with the shape of the lamp assembly installation groove. During assembly, the relative positioning is accurate through the fixture, and the transparent support 12 and the housing 15 are bonded with transparent glue to complete the assembly.

[0077] The embodiment of the present application also provides a perception method, which is applied to the finger-shaped visual-tactile sensor provided in the embodiment of the present application. Figure 6 The flowchart of the perception method provided by the embodiment of the present application is shown. Figure 6 As shown, the sensing method includes the following steps S61 to S63.

[0078] S61 , the LED light group 13 emits light toward the transparent elastic body 11 .

[0079] Specifically, the light sources of different colors (such as red, green, and blue) of the LED light group 13 adjust the angle and intensity of the emitted light to evenly illuminate the surface of the transparent elastic body 11. The multi-color combination of LED light sources not only provides rich light information, but also enhances the contrast and clarity of the image by adapting to the light reflection characteristics of different scenes and materials.

[0080] S62, when the transparent elastic body 11 is deformed, the camera 14 collects an image of the transparent elastic body.

[0081] Specifically, when the transparent elastic body 11 contacts an external object and deforms, the reflection and refraction characteristics of light inside it will change with the deformation. These changes lead to changes in light and color on the surface, generating unique optical features. The camera 14 can collect and record the optical changes on the surface of the transparent elastic body 11 in real time through high-resolution imaging technology. The image data obtained by the camera 14 will serve as the basis for subsequent processing and analysis, providing reliable visual information for further information extraction and analysis.

[0082] S63: The processor processes the image to obtain a perception result.

[0083] For example, the processor can extract the deformation information of the transparent elastic body 11 by analyzing the light and shadow changes and texture features in the image. Based on the image processing results, the processor can infer the contact position, contact area, pressure and other perceptual information of the external object, thereby realizing the fusion perception of tactile and visual signals.

[0084] See also Figure 7 In some implementations, the processor may perform image processing using a method based on image feature analysis, and the process includes the following steps S71 to S73.

[0085] S71, preprocessing the collected original image. Exemplarily, the preprocessing may include operations such as denoising, brightness equalization, and contrast adjustment. Among them, denoising can reduce the impact of external light interference or image acquisition device noise on subsequent analysis. Brightness equalization and contrast adjustment help to highlight the texture and light and shadow changes on the surface of the transparent elastomer 11, making the image features more obvious.

[0086] S72, edge detection and feature extraction are performed on the preprocessed image. For example, an edge detection algorithm (such as Canny edge detection or Sobel operator) can be used to identify the main contours and edge features in the image, and deformation features such as concave depth, contact area, pressure distribution, etc. can be obtained based on the edge detection results.

[0087] S73, feature matching and pattern recognition. For example, the extracted image features can be matched with a pre-established feature library, and different contact types, force magnitudes and other information can be identified through a pattern recognition algorithm (such as K-means clustering or SVM support vector machine). Based on the matching results, the processor can infer the magnitude, direction and distribution of the external contact force.

[0088] See also Figure 8 In some implementations, the processor may perform image processing using a deep learning-based method, and the process includes the following steps S81 to S83.

[0089] S81, preprocessing the collected original image.

[0090] S82, input the preprocessed image into the deep learning model to obtain the magnitude and direction of the externally applied force and the contact type.

[0091] S83, combining the output of the deep learning model with the actual tactile perception task to generate a final perception result. For example, based on the depression depth and pressure distribution of different contact points, the processor can infer the shape, material characteristics, etc. of the contact object.

[0092] It should be noted that the above only lists two image processing methods by way of example, but the present application is not limited thereto.

[0093] The embodiment of the present application also provides a mechanical device, which includes the finger-shaped visual-tactile sensor provided in the embodiment of the present application.

[0094] Exemplarily, the mechanical device may be a dexterous hand.

[0095] The present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by the processor to implement the perception method provided by the present application. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state hard disk (SSD)), etc.

[0096] The embodiment of the present application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program. The processor is communicatively connected to the memory, and the perception method provided by the embodiment of the present application is implemented when the computer program is called.

[0097] The present application embodiment may also provide a computer program product, the computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer or data center.

[0098] When the computer program product is executed by a computer, the computer executes the method described in the above method embodiment. The computer program product may be a software installation package, and when the above method is required, the computer program product may be downloaded and executed on a computer.

[0099] In summary, the embodiments of the present application solve the problem that it is difficult to reduce the volume of visual tactile sensors, and reduce the size of visual tactile sensors to the size of human fingers. In addition, the embodiments of the present application adopt a miniaturized design, so that the sensor can be more easily integrated into various robot systems, especially dexterous hands, micro-robots and other equipment, improving the operating accuracy and flexibility of the equipment. The visual tactile sensor can be installed in a small space or on a complex surface, and is suitable for a variety of application scenarios. Furthermore, in complex operating systems such as dexterous hands, the miniaturized visual tactile sensor provided by the embodiments of the present application can improve the flexibility of operation, better adapt to the shape and surface changes of complex objects, bring tactile feedback similar to human fingers to the robot, and improve the sensitivity and accuracy of operation. Therefore, the present application effectively overcomes the various defects in the prior art and has a high industrial value.

[0100] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0101] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A finger-shaped visual tactile sensor, characterized in that: The finger-shaped visual-tactile sensor comprises: Transparent elastomer; A transparent support body, located below the transparent elastic body, and used to support the transparent elastic body; An LED light group is located near the transparent support body; A camera, used to collect an image of the transparent elastic body; The housing is used for mounting the transparent support, the LED light group and the camera.

2. The finger-shaped visual-tactile sensor according to claim 1, characterized in that: The outer surface of the transparent elastomer is covered with a reflective coating.

3. The finger-shaped visual-tactile sensor according to claim 2, characterized in that: The reflective coating is coated with a functional coating on the outside.

4. The finger-shaped visual-tactile sensor according to claim 2, characterized in that: The transparent support is also used as a light-guiding structure to conduct the light of the LED light group so that the light is evenly distributed on the reflective coating.

5. The finger-shaped visual-tactile sensor according to claim 1, characterized in that: The LED light group includes a red light group, a green light group and / or a blue light group welded on a flexible circuit board.

6. The finger-shaped visual-tactile sensor according to claim 5, characterized in that: After installation, the green light group and the blue light group are located on the same side of the red light group, and the angle between the red light group and the green light group and the angle between the red light group and the blue light group are both 90°.

7. The finger-shaped visual-tactile sensor according to claim 1, characterized in that: The housing comprises a lamp assembly installation slot, and the lamp assembly installation slot is used for installing the LED lamp assembly.

8. The finger-shaped visual-tactile sensor according to claim 7, characterized in that: The LED lamp group is pre-packaged into the housing with epoxy resin.

9. A sensing method, characterized in that: The finger-shaped visual-tactile sensor applied to any one of claims 1 to 8, the method comprising: The LED light group emits light toward the transparent elastic body; When the transparent elastic body is deformed, the camera collects an image of the transparent elastic body; The image is processed by a processor to obtain a perception result.

10. A dexterous hand, characterized in that: The dexterous hand comprises the finger-shaped visual-tactile sensor according to any one of claims 1 to 9.

Images