Multi-spectrum curved surface visual tactile sensor based on beam splitter prism

Abstract

The invention belongs to the related technical field of visual sensors, and discloses a multi-spectrum curved surface visual tactile sensor based on a beam splitter prism. The sensor comprises a contact probe, a support frame, a beam splitter prism, a light source and an image acquisition device, the contact probe is made of a soft material, and the contact probe deforms when being in contact with a measured object; the interior of the contact probe is in a transparent state; the support frame is divided into an upper half part and a lower half part, light sources with different spectrums are uniformly distributed on the inner surface of the upper half part, the lower half part is provided with a beam splitter prism, and image acquisition devices are arranged at the bottom and on the side surface of the beam splitter prism; the upper half part of the supporting frame is arranged in the contact probe, the contact probe and a measured object deform, light emitted by the light source is reflected by the outer surface of the deformed contact probe, and then the reflected light enters the beam splitter prism and is divided into two beams of light which respectively enter the image acquisition device for imaging. According to the invention, complete and effective tactile feedback information of a complex curved surface can be provided.

Description

Technical Field

[0001] The present invention belongs to the technical field related to visual sensors, and more specifically, relates to a multi-spectral curved surface visual-tactile sensor based on a spectroscopic prism. Background Art

[0002] Tactile sensors can provide information about the physical properties of an object and help enhance grip stability through force control. In the field of robotics, tactile perception can enhance the robot's grasping ability. Currently, many types of sensors can achieve tactile sensing, such as resistive-based tactile sensors and magnetic-based tactile sensors. However, because current methods generally rely on discrete sensing elements, it is difficult to achieve high-resolution tactile perception.

[0003] Visual-tactile sensors are a new type of hybrid sensor. They acquire tactile information by observing the deformation of contact surfaces using a camera. Combined with computer vision techniques, they further interpret tactile images into higher-level tactile features. Their high-resolution multimodal fusion demonstrates their superior performance in robotic manipulation. Recent advances in the design of curved visual-tactile sensors have demonstrated significant advantages over flat visual-tactile sensors, particularly in terms of extended sensing coverage and enhanced humanoid compatibility. The GelStereo Palm, a curved visual-tactile sensor, utilizes a binocular vision system to perceive 3D contact geometry. The GelStereo BioTip, a fingertip-sized curved visual-tactile sensor, exhibits flexible, biomimetic features and can sense 3D surface deformation. However, these curved visual-tactile sensors all have simple shapes and relatively simple illumination, which affects the accuracy of subsequent 3D reconstruction of the contact surface, limiting their sensing capabilities for fine manipulation tasks and their generalization across diverse manipulation tasks. Therefore, a device with high 3D reconstruction accuracy is needed to address these challenges. Summary of the Invention

[0004] In response to the above defects or improvement needs of the prior art, the present invention provides a multi-spectral curved surface visual-tactile sensor based on a spectroscopic prism to solve the problem of low three-dimensional reconstruction accuracy.

[0005] To achieve the above objectives, according to one aspect of the present invention, a multispectral curved surface visual-tactile sensor based on a spectroscopic prism is provided, characterized in that the sensor includes a contact probe, a support frame, a spectroscopic prism, a light source, and an image acquisition device, wherein:

[0006] The contact probe is made of soft material and deforms when in contact with the object being measured; the interior of the contact probe is transparent and the outer surface is coated with diffuse reflection paint;

[0007] The support frame is divided into an upper half and a lower half. The inner surface of the upper half is evenly distributed with light sources of different spectra. The lower half is provided with a spectroscopic prism. The bottom and side of the spectroscopic prism are provided with image acquisition devices.

[0008] The upper half of the support frame is arranged in the contact probe, and the contact probe and the object to be measured are deformed. The light emitted by the light source is reflected by the outer surface of the deformed contact probe, and then the reflected light enters the beam splitter prism and is divided into two beams of light, which enter the image acquisition device respectively for imaging.

[0009] Further preferably, the Shore hardness of the contact probe is 8A-10A.

[0010] Further preferably, the contact probe is made of silicone.

[0011] Further preferably, the contact probe is tapered, and the cross-sectional area of the end thereof in contact with the object to be measured is smaller than the cross-sectional area of the bottom end.

[0012] Further preferably, the light source includes light sources of multiple different colors of light and an infrared light source.

[0013] Further preferably, a filter is provided at the front end of the image acquisition device.

[0014] Further preferably, a transparent partition is provided at the junction of the upper half and the lower half of the support frame.

[0015] Further preferably, the image acquisition device is a color RGB camera and a near infrared NIR camera.

[0016] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0017] 1. The visual sensor provided by the present invention utilizes an image acquisition device to capture images of the deformation of a contact probe made of a soft material, which deforms when in contact with the object being measured. For objects with complex geometric shapes, the soft contact probe can fully sense the shape and contact force of the object through full contact with the contact probe, resulting in high detection accuracy.

[0018] 2. The multispectral curved surface visual-tactile sensor based on a spectroscopic prism of the present invention can use multiple types of cameras, such as cameras that can accept different spectra. Because different types of cameras capture different images, the details of different images can be complemented, with high spatial resolution and accuracy. The three-dimensional reconstruction accuracy in the present invention is higher than 0.01mm.

[0019] 3. The sensor of the present invention has a relatively compact structure by rationally arranging the camera and light source, and can maintain a small volume while achieving a high spatial resolution. It uses a low-cost silicone transparent gel layer and an easily available camera, which has a low manufacturing cost and is convenient for promotion and production.

[0020] 4. The present invention improves the sensor's ability to continuously contact complex surfaces by designing a gel layer with a human finger-like shape. At the same time, it uses multiple light sources and cameras to collect images in different bands to achieve complementary image details. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is an exploded view of the structure of a sensor constructed according to a preferred embodiment of the present invention;

[0022] Figure 2 1 is a schematic structural diagram of a support frame constructed according to a preferred embodiment of the present invention, wherein (a) is a front view of the support frame, and (b) is a side view of the support frame;

[0023] Figure 3 is a schematic structural diagram of the upper half of the support frame constructed according to a preferred embodiment of the present invention, which is nested in the contact probe;

[0024] Figure 4 is a schematic diagram of the principle of image acquisition by an image acquisition device constructed according to a preferred embodiment of the present invention;

[0025] Figure 5 is a schematic structural diagram of a molded contact probe constructed according to a preferred embodiment of the present invention;

[0026] Figure 6 1 is a schematic diagram of a process for forming a contact probe according to a preferred embodiment of the present invention. In all drawings, the same reference numerals are used to represent the same elements or structures, wherein:

[0027] 1-front housing, 2-first camera, 3-first camera panel, 4-support frame, 5-beam splitter prism 5, 6-second camera bracket, 7-second camera, 8-back housing, 9-transparent partition, 10-contact probe, 11-first light source, 12-second light source. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0029] like Figure 1 and 2 As shown, a multispectral curved surface visual tactile sensor based on a spectroscopic prism includes a support frame 4, a light source located on the upper surface of the support frame 4, and a spectroscopic prism 5 located in the middle of the support frame 4. In the direction of the transmission light path of the spectroscopic prism 5, the sensor includes a contact probe 10, a light source, and a reflective coating located at the front end of the spectroscopic prism 5; in the direction of the refraction light path of the spectroscopic prism 5, the sensor includes a light source and an image acquisition device; the refraction light path is perpendicular to the transmission light path.

[0030] In an embodiment of the present invention, the light source is divided into a first light source 11 and a second light source 12, and the image camera device includes a first camera 2 and a second camera 7. The first camera 2 and the second camera 7 respectively receive the light generated by the first light source 11 and the second light source 12. The spectra of the light generated by the first light source 11 and the second light source 12 are different.

[0031] According to one embodiment of the present invention, the first light source 11 includes one red, green, and blue LED lamp, and the second light source 12 includes an infrared LED lamp, wherein the red, green, and blue LED light sources and the infrared light source are arranged at the four corners of the square upper surface of the support frame 4, facing the side of the contact probe 10.

[0032] like Figure 2 As shown, the contact probe 10, the back of the first camera 2, and the back of the second camera 7 are used to seal the support frame 4, and the support frame 4 is used to provide a carrier for installing the contact probe 10, the reflective layer, the dichroic prism 5, the first camera 2, the second camera 7, the first light source 11, and the second light source 12.

[0033] According to one embodiment of the present invention, the contact probe 10 has a complex curved surface structure similar to a human finger and can generate continuous deformation when contacting an object.

[0034] According to one embodiment of the present invention, the contact probe 10 is formed by casting with a mold. During casting, the frame is placed vertically, and a mold opening line is avoided on the contact probe 10 by a horizontal mold opening surface.

[0035] like Figure 3 As shown, according to one embodiment of the present invention, the Shore hardness of the contact probe 10 is 8-10A, which is flexible enough to avoid breakage but can also produce observable deformation.

[0036] According to one embodiment of the present invention, the contact probe 10 is made of food-grade silicone, which has good light transmittance, wear resistance and ductility.

[0037] According to one embodiment of the present invention, the surface of the contact probe 10 is coated with silver-grey flaky paint powder as a diffuse reflection coating. The silver-grey flaky paint powder is used to enhance the diffuse reflection effect to meet the assumption of the photometric stereo method in three-dimensional reconstruction.

[0038] According to one embodiment of the present invention, a transparent partition 9 is provided at the boundary between the upper and lower halves of the support frame 4. On the one hand, light can enter the dichroic prism through the transparent partition, and on the other hand, it can prevent the casting material from entering the lower half of the support frame 4 during the casting and molding process of the contact probe 10. In this embodiment, the transparent partition is made of an acrylic plate.

[0039] According to one embodiment of the present invention, the beam splitter prism 5 is a cubic beam splitter prism 5 having equal reflected light path and transmitted light path, a spectral ratio of 1:1, and a refractive index of 1.5168.

[0040] According to one embodiment of the present invention, the cubic beam splitter prism 5 is composed of a right-angle prism glued to another identical right-angle prism, wherein the inclined surfaces of the right-angle prisms are coated with a film to avoid astigmatism.

[0041] According to one embodiment of the present invention, Figure 4 As shown, the first camera 2 and the second camera 7 are respectively a color RGB camera and a near-infrared (NIR) camera. Both cameras have their auto-exposure and auto-white balance functions disabled, and filters are integrated into their lenses. This allows them to capture image information at different wavelengths, extracting different surface gradient information. Finally, after merging and processing, a spatial resolution exceeding 0.01 mm can be achieved.

[0042] According to one embodiment of the present invention, the support frame 4 is a three-dimensional box-type support frame 4 , which is formed by stereoscopic light-curing 3D printing and has higher assembly accuracy.

[0043] like Figure 1 As shown, it is a schematic diagram of the principle of a multispectral curved surface visual-tactile sensor (hereinafter referred to as the sensor) based on a spectroscopic prism 5 according to an embodiment of the present invention. A second light source 12 is added to the configuration of the first light source 11, and the spectroscopic prism 5 is used to enable the first camera 2 and the second camera 7 to filter the required wavelengths of light using RGB and NIR filters.

[0044] All of the above components are based on the support frame 4; the front shell 1 and the back shell 8 are used to provide positioning of the support frame 4 in the mold; the first camera 2 and the second camera 7 can capture images under light of different wavelengths, and filters are installed on the camera lenses to filter light of different wavelengths. The two cameras are connected to the support frame 4 through their corresponding fixed structures, the first camera panel 3 and the second camera bracket 6; the spectroscopic prism 5 is a neutral beam splitter in the shape of a cube, and is also embedded with an optical diffuser (not shown), which can transmit the physical information of the contact probe 10 to the two cameras in the form of an image, and shares a fixed bracket 6 with the second camera 7 to be connected to the support frame 4; the contact probe 10 is silicone coated with a diffuse reflective paint, and the diffuse reflective paint contains flaky silver paint powder.

[0045] When the sensor of this embodiment contacts an object, the contact probe 10 undergoes geometric deformation and generates different gradient information under illumination from different light sources. The camera captures the gradient information and converts it into tactile information.

[0046] like Figure 5 As shown, the GelSplitter3D sensor casting mold of an embodiment of the present invention adopts the upper and lower opening method to avoid the generation of a parting line on the surface of the contact probe; the upper and lower molds are positioned by positioning pins and connected by bolts; an exhaust duct is provided at the dividing interface to discharge internal air during casting; the upper half of the mold is provided with a pressure block to realize the pressing effect on the sensor support frame 4.

[0047] like Figure 6 FIG. 1 is a schematic diagram of the manufacturing process of the GelSplitter 3D sensor according to an embodiment of the present invention. The specific process includes:

[0048] Step 1: Preparation of mold and related materials.

[0049] The main purpose of this step is to arrange the light source and acrylic plate on the support frame 4 in advance and prepare for casting. The first light source 11 used in this embodiment includes three colors of LED lamps, namely red LED lamp (model NCD0603R1, wavelength range 615~630nm), green LED lamp (model NCD0603W1, wavelength 515~530nm) and blue LED lamp (model NCD0603B1, wavelength 463~475nm). The second light source 12 used includes an infrared LED lamp (model XL-1608IRC940, wavelength 940nm). The package of each LED lamp is 0603 package, and the specific size is 1.6mm×0.8mm. The four LED lamps are fixed at the four corners of the upper surface of the support frame 4 with shadowless adhesive, with the luminous surface facing up. Note that when fixing, it is necessary to ensure that the power cord of the LED lamp does not block the camera window. The transparent partition is also fixed to the upper surface of the support frame 4 with shadowless adhesive.

[0050] Mould preparation mainly includes cleaning the mould and applying the release agent. When applying the release agent, it is necessary to ensure that the surface is smooth after application and the application should be as thin and even as possible.

[0051] Step 2, casting gel and shaping.

[0052] The main task of this step is to prepare the gel and cast it into the mold, waiting for it to set. The gel used in this embodiment is made up of three ingredients, with a mass ratio of A:B:Slacker of 1:1:3. The role of Slacker is to soften the gel and slow down the gel solidification rate to ensure that the gel fully fills the mold cavity. After mixing and preparing, it needs to be vacuumed to remove internal bubbles to prevent them from affecting the imaging. Note that the entire process of preparing the gel and vacuuming should be as clean and dust-free as possible to prevent impurities from entering the gel and affecting the imaging.

[0053] While waiting for the gel to solidify, the gel can be heated in a heating box to accelerate the gel setting. The temperature is usually set at 50℃~55℃ and the time is ≥300min.

[0054] Step 3: Open the mold and clean it.

[0055] The primary purpose of this step is to remove the set gel from the mold and clean any remaining wet gel and other impurities. Be gentle when opening the mold to avoid damaging the gel. Gently wipe the gel surface with a paper towel to remove as much dust and impurities as possible, ensuring a smooth, clean surface for the reflective coating.

[0056] Step 4: Prepare the reflective layer paint and apply the reflective layer.

[0057] The main purpose of this step is to mix the reflective paint and brush a diffuse layer on the surface of the transparent gel.

[0058] The reflective coating ingredients are similar to those for transparent adhesive, but the mass ratio of A:B:Slacker is 1:1:4. Silver flake paint powder is added, and the volume ratio of paint powder to gel is 1:1. Stir thoroughly and vacuum the mixture. This completes the coating.

[0059] When applying the coating, be careful to be gentle to avoid damaging the gel surface. Ensure the coating is thin and even, completely covering the entire contact probe 10. Otherwise, light leakage will affect imaging. After application, allow the coating to air dry and solidify. While waiting for the gel to solidify, heat the coating in a heating oven to accelerate solidification. Typically, set the temperature to 50°C to 55°C for at least 120 minutes.

[0060] Step 5: Debug the sensor.

[0061] The purpose of this step is to test the sensor's imaging. Before debugging, the LED light source needs to be powered. In this example, a 5V USB cable is used for power supply. A resistor is connected between the USB cable and the LED power line to protect the LED. During debugging, three key aspects should be observed: lighting quality, the presence of impurities in the contact probe 10 and reflective layer, and camera imaging quality.

[0062] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multispectral curved surface visual tactile sensor based on a spectroscopic prism, characterized in that: The sensor comprises a contact probe (10), a support frame (4), a beam splitter prism (5), a light source, and an image acquisition device, wherein: The contact probe (10) is made of soft material and deforms when in contact with the object to be measured; the interior of the contact probe is transparent, and the outer surface is coated with a diffuse reflection paint; The support frame (4) is divided into an upper half and a lower half, the inner surface of the upper half is evenly distributed with light sources of different spectra, the lower half is provided with a beam splitter prism (5), and the bottom and side surfaces of the beam splitter prism (5) are provided with image acquisition devices; The upper half of the support frame (4) is arranged in the contact probe (10), and the contact probe (10) and the object to be measured are deformed. Light emitted by the light source is reflected by the outer surface of the deformed contact probe, and then the reflected light enters the beam splitter prism (5) and is split into two beams of light, which respectively enter the image acquisition device to form an image.

2. The multispectral curved surface visual-tactile sensor based on a spectroscopic prism according to claim 1, characterized in that: The Shore hardness of the contact probe (10) is 8A-10A.

3. The multi-spectral curved surface visual-tactile sensor based on a spectroscopic prism according to claim 1 or 2, characterized in that: The material of the contact probe (10) is silica gel.

4. The multi-spectral curved surface visual-tactile sensor based on a spectroscopic prism according to claim 1 or 2, characterized in that: The contact probe (10) simulates the shape of a human finger, and the cross-sectional area of one end of the contact probe that contacts the object to be measured is smaller than the cross-sectional area of the bottom end.

5. The multi-spectral curved surface visual-tactile sensor based on a spectroscopic prism according to claim 1 or 2, characterized in that: The light source includes a plurality of light sources of different colors and an infrared light source.

6. The multi-spectral curved surface visual-tactile sensor based on a spectroscopic prism according to claim 1 or 2, characterized in that: A filter is provided at the front end of the image acquisition device.

7. The multi-spectral curved surface visual-tactile sensor based on a spectroscopic prism according to claim 1 or 2, characterized in that: A transparent partition is provided at the junction of the upper half and the lower half of the support frame (4).

8. The multi-spectral curved surface visual-tactile sensor based on a spectroscopic prism according to claim 1 or 2, characterized in that: The image acquisition device is a color RGB camera and a near infrared NIR camera.

Images