Multispectral imaging-based tactile sensor

Abstract

Disclosed in the present invention is a multispectral imaging-based tactile sensor, comprising a light-shielding housing, a sensing skin, a vision system and a lighting system; the vision system comprises a visible light camera and a near-infrared camera which are arranged inside the light-shielding housing; the lighting system comprises a near-infrared light source arranged inside the light-shielding housing; the sensing skin is arranged in a tactile sensing area of the light-shielding housing; the sensing skin comprises an elastic film that is transparent to external visible light but opaque to internal near-infrared light due to a difference in light intensities inside and outside the light-shielding housing; the visible light camera collects the external visible light via the elastic film, so as to perform proximity sensing of an object; the near-infrared camera detects the shape and texture of the sensing skin by means of the near-infrared light emitted by the near-infrared light source onto the elastic film, so as to implement tactile sensing of the object. The multispectral imaging-based tactile sensor of the present invention has a proximity sensing function and can implement tactile sensing of objects.

Description

A tactile sensor based on multispectral imaging Technical Field

[0001] The present invention relates to the technical field of tactile sensors, and in particular to a tactile sensor based on multispectral imaging. Background Art

[0002] Intelligent robot perception technology is a prerequisite for robot control, interaction, and intelligence, and also forms the primary technical foundation for robots to perceive their environment and make intelligent decisions. Tactile perception, along with visual perception, forms the core of robot perception. Visual perception technology is typically based on image information acquired by digital imaging devices such as cameras and camcorders. With advances in image processing, pattern recognition, and deep learning, object perception based on visual information has made significant progress, reaching or even surpassing human visual capabilities in certain applications. However, due to limitations in the fundamental principles of imaging, visual-based object perception often fails in conditions such as poor lighting, image obstructions, drastic environmental changes, and complex surface conditions (color, texture, decoration, material, smoothness). In contrast, tactile perception systems offer flexible and simple construction, low cost, relaxed environmental requirements, and flexible distribution, enabling ultra-close-range perception. They play a crucial role in certain applications that visual perception cannot replace. Furthermore, from a biomimetic perspective, tactile perception can complement visual information, simultaneously perceiving multidimensional information such as the temperature, texture, roughness, and shape of environmental objects. Therefore, research on accurate and efficient tactile perception technology to achieve complex object recognition and attribute determination will help improve the perception dimension and perception performance of robots, and has great application prospects in medical service robots, household service robots, industrial sorting robots and even dexterous warfare robots.

[0003] With the development of various flexible pressure sensors, circuit printing technology, and substrate preparation processes, flexible electronic skin that can be equipped on robots has given robots new tactile perception capabilities, making robots have the potential to reach the level of human tactile perception. According to statistics, there are about 16,500 tactile receptors in the human hand, and the density of neurons can reach 90 units / cm 2 Based on this high-density array of tactile receptors, humans can sense temperature, texture, shape, viscosity, and other aspects. Inspired by human skin, scientists have developed tactile sensing systems based on electronic skin to enhance the perception capabilities of robots. However, due to current limitations in manufacturing processes and costs, achieving multimodal, high-resolution electronic skin that approaches the perception capabilities of human skin remains extremely challenging.

[0004] Currently, research on tactile sensors can be broadly and narrowly defined. The broad definition of tactile sensory perception includes touch, pressure, force, slip, and thermal sensation. The narrow definition of tactile sensory perception encompasses force sensation at the interface between a manipulator and an object. Based on their functional characteristics, tactile sensors can be broadly categorized as contact sensors, force-torque sensors, pressure sensors, and slip sensors.

[0005] The current robotic gripper sensing technologies are shown in Table 1, which mainly include the following types:

[0006] (1) Capacitive tactile array sensor. Its principle is that external force changes the relative displacement between the plates, thereby changing the capacitance. The tactile force is measured by detecting the change in capacitance.

[0007] (2) Inductive tactile sensor. It uses the principle of electromagnetic induction to convert pressure into changes in the self-inductance and mutual inductance of the coil, and then converts the circuit into a voltage or current change output.

[0008] (3) Photoelectric tactile sensor. It is developed based on the principle of total internal reflection and is usually composed of a light source and a photodetector. When the pressure applied to the interface changes, the reflection intensity of the sensor's sensitive element and the frequency of the light source will also change accordingly.

[0009] (4) Piezoresistive tactile sensor. This device is made based on the piezoresistive effect of semiconductor materials. Its substrate can be directly used as a measuring sensor element, and the diffused resistors are connected in the substrate to form a bridge. When the substrate is deformed by external force, the resistance values ​​will change, and the bridge will produce a corresponding unbalanced output.

[0010] (5) Piezoelectric tactile sensor: Under the action of pressure, a potential difference appears between the two end surfaces of the piezoelectric material; conversely, mechanical stress is generated when voltage is applied.

[0011] Table 1

[0012] In recent years, with the rapid development of optical imaging and computer vision technologies, researchers have begun combining optical imaging with tactile perception, giving rise to a new type of tactile perception technology: visual tactile sensing. This technology typically involves skin sensing, an imaging unit, and an illumination system. It uses the visual system to acquire tactile information by sensing skin deformation, texture, color, and other information. It boasts high resolution, a large detection area, low cost, and high stability. However, to date, visual tactile sensors primarily utilize information in the visible light band. While researchers have attempted to combine ultraviolet light with visible light to reduce the influence of markers on object contour detection, this approach only addresses some of the shortcomings of visible light imaging and does not provide the functional improvements that tactile perception provides. Summary of the Invention

[0013] In order to solve the problem that the existing tactile sensing function of the tactile sensor is single, the present invention proposes a tactile sensor based on multispectral imaging.

[0014] The technical problem of the present invention is solved by the following technical solutions:

[0015] A tactile sensor based on multispectral imaging includes a light-shielding shell, a sensing skin, a visual system, and a lighting system. The visual system includes a visible light camera and a near-infrared camera arranged in the light-shielding shell. The lighting system includes a near-infrared light source arranged in the light-shielding shell. The sensing skin is arranged on a tactile sensing area of ​​the light-shielding shell. The sensing skin includes an elastic film that is transparent to external visible light and opaque to internal near-infrared light due to the difference in light intensity inside and outside the light-shielding shell. The visible light camera collects external visible light through the elastic film to sense the proximity of an object. The near-infrared camera uses the near-infrared light irradiated on the elastic film by the near-infrared light source to detect the shape and texture of the sensing skin to achieve tactile perception of the object.

[0016] In some embodiments, the light-shielding housing is a sealed structure so that the intensity of visible light outside is greater than the intensity of visible light inside the light-shielding housing.

[0017] In some embodiments, the near-infrared light source includes a 940 nm near-infrared LED lamp array.

[0018] In some embodiments, the near-infrared camera is a 930-950 nm near-infrared camera.

[0019] In some embodiments, a visible light source is provided on the light-shielding housing for providing light for the proximity sensing in dark conditions.

[0020] In some embodiments, the visible light source is an LED light strip.

[0021] In some embodiments, the visible light camera and the near infrared camera are CCD cameras.

[0022] In some embodiments, the proximity sensing determines the distance between the sensor and the object by using the grayscale value of the object image captured by the visible light camera.

[0023] In some embodiments, the elastic film is an inflatable film having an inflatable cavity.

[0024] In some embodiments, the visual system further includes a mid-infrared camera disposed in the light-shielding housing, and the mid-infrared camera is used to collect temperature information of the sensed skin to obtain a temperature image of the object.

[0025] The beneficial effects of the present invention compared with the prior art include:

[0026] The present invention provides a light-shielding housing with an elastic film and a near-infrared light source within the housing, creating a difference in light intensity between the inside and outside of the housing. This makes the elastic film in the tactile sensing area transparent to external visible light but opaque to internal near-infrared light. A visible light camera within the housing collects external visible light through the elastic film to sense the proximity of an object. Furthermore, the near-infrared camera within the housing uses the near-infrared light irradiated on the elastic film by the near-infrared light source to detect the shape and texture of the skin, thereby achieving tactile perception of the object. Thus, the tactile sensor of the present invention utilizes multispectral imaging to achieve both proximity sensing and tactile perception of an object. Furthermore, the present invention provides a sealed housing so that the intensity of visible light outside is greater than the intensity of visible light inside the housing, allowing visible light to pass through the elastic film. This allows the visible light camera to collect external visible light through the elastic film to sense the proximity of an object.

[0027] In some embodiments, a visible light source is provided on the light-shielding housing to provide light for proximity sensing in dark conditions. The present invention also uses the grayscale value of the object image captured by the visible light camera to determine the distance between the sensor and the object, thereby completing proximity sensing. The present invention also configures the elastic film as an inflatable film with an inflatable cavity, and inflates the inflatable film by inflation to replace the existing acrylic support layer, thereby solving the problem that the existing perception of skin deformation is small and it is difficult to obtain texture and contour information of objects with large surface gradients. In addition, the present invention also provides a mid-infrared camera in the light-shielding housing, and uses the mid-infrared camera to collect and sense the temperature information of the skin to obtain a temperature image of the object, thereby achieving temperature perception.

[0028] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of the classification of light according to wavelength.

[0030] FIG2 a is a schematic structural diagram of a tactile sensor according to an embodiment of the present invention.

[0031] FIG2 b is a schematic diagram of the use of the tactile sensor in an embodiment of the present invention.

[0032] FIG3 shows proximity perception images of four objects at distances of 12 / 9 / 6 / 3 / 0 cm by the tactile sensor according to an embodiment of the present invention.

[0033] FIG4 is a flow chart of proximity sensing of a tactile sensor according to an embodiment of the present invention.

[0034] FIG5 is a contact perception image of the tactile sensor on five water bottles with different textures according to an embodiment of the present invention.

[0035] FIG6 is a temperature sensing image of the tactile sensor according to an embodiment of the present invention.

[0036] The reference numerals are as follows: 21 elastic film, 31 visible light camera, 32 near-infrared camera, 33 mid-infrared camera, 41 near-infrared light source, 42 homogenizing plate, 5 visible light source. DETAILED DESCRIPTION

[0037] The present invention will be further described below with reference to the accompanying drawings and in combination with preferred embodiments. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0038] It should be noted that the directional terms such as left, right, up, down, top, and bottom in this embodiment are merely relative concepts, or are based on the normal use status of the product, and should not be considered as restrictive.

[0039] The visual tactile perception technology used by the tactile sensor proposed in the embodiment of the present invention is an evolution of the photoelectric tactile sensor. It integrates multispectral imaging technology to achieve high-resolution tactile information perception. It directly uses a camera to detect and perceive deformation of the skin surface, and has the advantages of large area, high resolution, low cost, and high stability.

[0040] Current visual tactile sensors mostly use visible light cameras to detect deformation of the elastic membrane surface. However, current visual tactile sensors mainly use information in the visible light band, as shown in Figure 1. In fact, light contains information in multiple frequency bands. Single-band optics only retains visible light information, so it has less information fusion capability. In order to achieve more functions, it is necessary to introduce information from more bands. 。

[0041] The inventors of this invention realized that integrating more wavelengths of light would significantly enhance the functionality of visual-tactile sensing technology. To address the low resolution and limited functionality of traditional tactile sensors, the preferred embodiment of the present invention combines visible light, near-infrared, and mid-infrared detection technologies to propose a multimodal tactile sensor based on multispectral imaging that can achieve high-resolution texture, proximity, deformation, and temperature perception. This tactile sensor can be widely used in fields such as home services and robotic grasping.

[0042] The tactile sensor based on multispectral imaging proposed in the preferred embodiment of the present invention has the following functions:

[0043] 1. The mechanical claw has texture detection capabilities;

[0044] 2. Temperature detection capability;

[0045] 3. Deformation detection capability;

[0046] 4. Ability to perceive proximity information;

[0047] Existing visual-tactile sensors usually consist of a sensing skin, a visual system, and a lighting system. The sensing skin is the part of the sensor that comes into direct contact with the outside world. It will manifest the force, texture, and other information generated by contact with the object in the form of deformation or color. The visual system uses optical imaging to obtain the deformation of the sensing skin. The function of the lighting system is to increase the brightness or contrast of the sensor surface through a special optical path. As shown in Figures 2a and 2b, an embodiment of the present invention uses multispectral imaging technology to propose a high-resolution visual-tactile sensor that can simultaneously obtain contact force information, texture information, contact information, and temperature information. In order to achieve this function, the embodiment of the present invention optimizes the design of the sensor's visual system, lighting system, and sensing skin.

[0048] The tactile sensor based on multispectral imaging proposed in an embodiment of the present invention is shown in Figures 2a and 2b, and includes a light-shielding shell, a sensing skin, a visual system and a lighting system. The visual system includes a visible light camera 31 and a near-infrared camera 32 arranged in the light-shielding shell. The lighting system includes a near-infrared light source 41 arranged in the light-shielding shell. The sensing skin is arranged on the tactile sensing area of ​​the light-shielding shell. The sensing skin includes an elastic film 21 that is transparent to external visible light and opaque to internal near-infrared light due to the difference in light intensity inside and outside the light-shielding shell. The visible light camera 31 collects external visible light through the elastic film 21 to sense the approach of an object. The near-infrared camera 32 uses the near-infrared light irradiated on the elastic film 21 by the near-infrared light source to sense the shape and texture of the skin to achieve tactile perception of the object.

[0049] To detect light of multiple wavelengths, the present invention proposes a visual system that can simultaneously detect visible light, near-infrared light, and mid-infrared light. This system includes a visible light camera 31, a near-infrared camera 32, and a mid-infrared camera 33. The visible light camera 31 is used for proximity sensing, the near-infrared camera 32 is used for skin shape and texture detection, and the mid-infrared camera 33 is used for temperature detection.

[0050] In order to realize approaching perception and sense of touch, the embodiment of the present invention realizes the elastic film 21 of one-way transmittance as perception skin by building the light intensity difference inside and outside the light-shielding shell. The elastic film 21 can become opaque on the side of strong light, and can become transparent on the side of weak light. The specific material of the elastic film 21 is not limited. As long as it is a material with certain light transmittance and elasticity, the selective transmission effect of light can be realized by the light brightness of different wavelengths on both sides of the elastic film 21. In one embodiment, preferably adopt the composite material made by latex and aluminum powder or silver powder. Latex can be processed by natural rubber tree sap. For example, the elastic film can be formed according to the proportioning of 200g latex and 0.5g silver powder.

[0051] The sensing skin is the core of visual-tactile sensors. In terms of the structural design of the sensing skin, most existing visual-tactile sensors use acrylic as a support layer and elastic silicone as a deformation layer. While this structure has a large force sensing range, the deformation is small, making it difficult to acquire texture and contour information on objects with large surface gradients. In this embodiment of the present invention, an elastic film 21 is used as the sensing layer. The sensing layer has a texture. Preferably, the elastic film 21 is an inflatable film with an air chamber, which is inflated by inflation to replace the support layer.

[0052] To achieve multifunctional tactile perception, the present invention proposes a visual system comprising a visible light camera 31, a near-infrared camera 32, and a mid-infrared camera 33. This visual system, housed within a light-shielding housing, can simultaneously acquire optical information from different wavelengths. Specifically, the visible light camera 31 collects visible light from outside the tactile sensor through the elastic film 21 to detect the proximity of an object. The near-infrared camera 32 uses near-infrared light irradiated on the elastic film 21 to detect the shape and texture of the skin, enabling tactile perception of the object. The mid-infrared camera 33 collects temperature information from the skin to obtain a temperature image of the object. To reduce sensor design costs, the present invention employs CCD (charge-coupled device) cameras as the primary imaging device. Specifically, the visible light camera 31, the near-infrared camera 32, and the mid-infrared camera 33 are all CCD cameras. Generally speaking, the wavelength detection range of a CCD camera is 50 nm to 1000 nm, encompassing the ultraviolet band below 400 nm, the visible light band from 400 nm to 700 nm, and the near-infrared band from 700 nm to 1000 nm. Among them, long-term exposure to ultraviolet light below 400nm will not only cause harm to the human body, but also limit its large-scale promotion due to problems such as the high cost of ultraviolet narrow-band filters. Therefore, the embodiment of the present invention sets the spectral range of the tactile sensor to the visible light band of 400nm to 700nm and the near-infrared band of 700nm to 1000nm.

[0053] In addition to visual imaging, light also contains temperature information. In nature, when the temperature of an object is higher than absolute zero, due to the existence of thermal motion inside the object, it will continuously radiate electromagnetic waves to the surrounding area. These electromagnetic waves are mid-infrared rays with a wavelength of 0.75 to 100 μm. The temperature we come into contact with daily is approximately between -40 and 300 degrees Celsius, and temperature detection can be achieved using a wavelength between 5.5 and 14 μm. Based on this principle, the embodiment of the present invention introduces mid-infrared light detection in the visual system to achieve temperature perception. Specifically, a mid-infrared camera 33 is used to collect and sense the temperature information of the skin to obtain a temperature image of the object, thereby achieving temperature perception.

[0054] Since longer wavelength light requires more specialized instruments for detection, which is not only costly but also bulky and difficult to install and deploy on small visual-tactile sensors, the embodiments of the present invention plan to control the wavelength range of light to three bands: visible light, near infrared, and mid-infrared.

[0055] The lighting system of this embodiment of the present invention is improved as follows: Since the transmittance of the elastic film 21 is controlled by light intensity, it is necessary to adjust the intensity of light of different wavelengths to control the transmittance of the elastic film 21 in different wavelength bands. First, to achieve the effect of opacity under infrared light, the infrared intensity inside the tactile sensor must be greater than that outside the tactile sensor, that is, the infrared intensity inside the light-shielding housing is greater than that outside the light-shielding housing. Therefore, this embodiment installs a near-infrared light source 41 inside the light-shielding housing of the tactile sensor. Specifically, the near-infrared light source 41 includes a 940nm near-infrared LED array. Correspondingly, the near-infrared camera 32 uses a 930-950nm near-infrared camera 32. Preferably, a light diffuser 42 is used to ensure uniform near-infrared light inside the tactile sensor. Second, the light-shielding housing can make the visible light intensity outside the tactile sensor greater than that inside the sensor, achieving the effect of transparency of the elastic film 21 under external visible light. Preferably, the entire tactile sensor is a sealed structure, that is, the light-shielding housing is a sealed light-shielding structure, so that the visible light intensity outside the light-shielding housing is greater than the visible light intensity inside the light-shielding housing. Specifically, the tactile sensor has no light sources of other wavelengths except the 940nm near-infrared LED array. Generally, the visible light intensity outside the tactile sensor is greater than that inside the tactile sensor.

[0056] Preferably, in order to achieve proximity perception under dark conditions, this embodiment provides a visible light source 5 on the light-shielding shell of the tactile sensor. The visible light source 5 is specifically an LED light strip installed on the outermost layer of the sensor. The LED light strip is specifically a strip-shaped LED array, which is used to provide light for proximity perception under dark conditions to ensure that the visible light intensity on the outside of the tactile sensor is always greater than that on the inside of the tactile sensor, that is, the visible light intensity on the outside of the light-shielding shell is always greater than that on the inside of the light-shielding shell.

[0057] The working principle of the visual system in the tactile sensor proposed in the embodiment of the present invention is as follows:

[0058] Visible light camera 31: Because the tactile sensor is sealed internally (i.e., the light-shielding housing is sealed), there is no visible light source within it. As a result, the visible light intensity inside the tactile sensor is lower than that outside the sensor. Therefore, the elastic film 21 appears transparent under visible light, enabling proximity sensing. Proximity sensing uses the grayscale value of the object image captured by the visible light camera 31 to determine the distance between the tactile sensor and the object. The tactile sensor proposed in this embodiment of the present invention was used to sense the proximity of four objects. The results are shown in Figure 3. The closer the object is to the tactile sensor, the darker the color. By obtaining the grayscale value of the color in the tactile sensor, the distance of the object from the tactile sensor can be determined. The brightness-based proximity sensing method of this embodiment of the present invention can effectively determine the object's position based on its distance from the sensor. As shown in Figure 4, this method first uses a background subtraction method to subtract the captured image from the background image to obtain a background-removed image, thereby removing the influence of background information on detection. The background-removed image is then filtered and thresholded to obtain pure proximity information. Finally, the maximum grayscale value of the color in the image is used to determine the distance of the object from the tactile sensor.

[0059] Near-infrared camera 32: Because the tactile sensor is equipped with a near-infrared light source 41 (i.e., a 940nm near-infrared LED array), the near-infrared light inside the tactile sensor is brighter than that outside the sensor. Therefore, the elastic film 21 appears opaque to the near-infrared camera 32. Therefore, texture information can be acquired using the near-infrared camera 32. Using the tactile sensor proposed in this embodiment of the present invention to sense contact with five mineral water bottles with different textures, the results shown in Figure 5 were obtained.

[0060] Mid-infrared camera 33: The temperature information of an object can be obtained by using the mid-infrared camera 33. The temperature images obtained by testing five objects with different temperatures are shown in FIG6 .

[0061] The embodiment of the present invention further proposes a tactile perception method based on multispectral imaging, which can realize high-resolution tactile information such as texture, temperature, and proximity information.

[0062] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that several equivalent substitutions or obvious variations can be made without departing from the scope of the present invention, and that any equivalent performance or application should be considered to fall within the scope of protection of the present invention.

Claims

1. A tactile sensor based on multispectral imaging, characterized in that It includes a light-shielding housing, a sensing skin, a vision system and an illumination system. The vision system includes a visible light camera and a near-infrared camera disposed within the light-shielding housing. The illumination system includes a near-infrared light source disposed within the light-shielding housing. The sensing skin is disposed on the tactile sensing area of the light-shielding housing. The sensing skin includes an elastic film that is transparent to external visible light and opaque to internal near-infrared light due to the difference in light intensity inside and outside the light-shielding housing. The visible light camera collects external visible light through the elastic film to perform proximity sensing of an object. The near-infrared camera uses the near-infrared light irradiated on the elastic film by the near-infrared light source to detect the shape and texture of the sensing skin, so as to achieve tactile sensing of an object.

2. The multi-spectral imaging-based tactile sensor according to claim 1, characterized in that The light-shielding housing is a sealed structure so that the intensity of external visible light is greater than the intensity of visible light inside the light-shielding housing.

3. The multi-spectral imaging-based tactile sensor according to claim 1 or 2, characterized in that The near-infrared light source includes a near-infrared LED lamp array of 940 nm.

4. The tactile sensor based on multispectral imaging according to claim 3, wherein The near-infrared camera is a near-infrared camera of 930 - 950 nm.

5. The multi-spectral imaging-based tactile sensor according to any one of claims 1 to 4, characterized in that, A visible light source is disposed on the light-shielding housing for providing light for performing the proximity sensing under dark conditions.

6. The multi-spectral imaging-based tactile sensor according to claim 5, characterized in that, The visible light source is an LED light strip.

7. The multi-spectral imaging-based tactile sensor according to any one of claims 1 to 6, characterized in that, The visible light camera and the near-infrared camera are CCD cameras.

8. The multispectral imaging-based tactile sensor according to any one of claims 1 to 7, characterized in that The proximity sensing determines the distance between the sensor and the object through the gray value of the object image collected by the visible light camera.

9. The multispectral imaging-based tactile sensor according to any one of claims 1 to 8, characterized in that, The elastic film is an inflatable film having an inflatable cavity.

10. The multispectral imaging-based tactile sensor according to any one of claims 1 to 9, characterized in that, The vision system further includes a mid-infrared camera disposed within the light-shielding housing. The mid-infrared camera is used to collect the temperature information of the sensing skin to obtain a temperature image of the object.

Images