A multi-modal vision-haptics system based on color-changing gel and control method

Abstract

The present application relates to a kind of multi-modal vision-tactile system and control method based on color-changing gel, wherein the system includes rigid shell, binocular camera module, ultraviolet lamp group, light guide plate and RBG-LED lamp module placed in rigid shell, one side of rigid shell is provided with opening, water gel is arranged on opening, water gel is located at the outside of light guide plate, binocular camera module and ultraviolet lamp group are located at the inside of light guide plate, and ultraviolet lamp group is located between the two cameras of binocular camera module.Compared with prior art, the present application has the advantages of simultaneous realization of tactile and visual detection scheme by a single sensor, and sensing through the same structure window, with high precision.

Description

Technical Field

[0001] This invention relates to the field of robotic arm sensors, and in particular to a multimodal vision-touch system and control method based on color-changing gel. Background Technology

[0002] Robotic gripping is a key technology for dexterous robot operation, with high demand not only in automated production workshops in industrial manufacturing but also in everyday life applications such as home service robots. During gripping, the positioning of the object and the control of the gripping force are crucial factors determining the success or failure of the task. For example, the gripping points of the robotic arm's finger-like or claw-like structures on the object being grasped directly determine the stability of the grip. Excessive or insufficient gripping force will lead to failure when grasping fragile, easily broken, or soft, easily deformable objects.

[0003] Currently, object grasping and localization are typically achieved using hand-eye cameras, while grasping force perception is usually achieved through methods such as piezoelectric sensors, including visual positioning methods like feature point localization, edge localization, and grayscale localization. Grasping force is assessed using piezoelectric sensors, magnetic sensing, and barometric pressure sensing. Current research on force sensing algorithms for robotic fingertips primarily focuses on tactile sensing technology, employing algorithms such as binocular vision and marker array recognition. The identified information is used for geometric reconstruction of the contact surface, and further data processing follows. However, directly combining visual and tactile sensing points can lead to inconsistencies, resulting in mismatched visual and tactile data and impacting subsequent calculations. Furthermore, current sensors are mainly either visual or tactile sensors installed separately, making it difficult to simultaneously utilize both types of sensors. Therefore, it is also difficult to integrate the advantages of both into the finger-like or claw-like structures of robotic hands.

[0004] However, the trend towards miniaturization and compactness in robotic dexterity urgently requires multimodal sensing devices with high functional and structural integration. Currently, there is no sensor that can integrate object perception and grasping force. Summary of the Invention

[0005] The purpose of this invention is to provide a sensing system that integrates object perception and grasping force with a compact structure and small size. This system is mainly based on the principle of color-changing gel to achieve efficient switching between vision and touch.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A multimodal vision-touch system based on color-changing gel includes a rigid housing, and a binocular camera module, an ultraviolet lamp assembly, a light guide plate, and an RBG-LED lamp module disposed within the rigid housing. One side of the rigid housing has an opening, and hydrogel is disposed on the opening. The hydrogel is located outside the light guide plate. The binocular camera module and the ultraviolet lamp assembly are located inside the light guide plate, and the ultraviolet lamp assembly is located between the two cameras of the binocular camera module.

[0008] The rigid housing includes a middle frame, a front cover, and a rear cover. The binocular camera module and the ultraviolet lamp assembly are mounted on the rear cover, and the opening is located on the front cover.

[0009] The RBG-LED light module is an RBG-LED light strip.

[0010] The projection of the centers of the two cameras of the binocular camera module onto the cross-section of the hydrogel is located at the center of the hydrogel.

[0011] The RBG-LED light module is arranged around the light guide plate.

[0012] The cross-sectional area of ​​the light guide plate is larger than that of the hydrogel.

[0013] The system is installed at the end of the finger-like or claw-like structure of the robotic arm.

[0014] A control method based on the above-mentioned multimodal vision-touch system based on color-changing gel includes:

[0015] Step S1: Upon receiving the first control command, the RGB-LED module is illuminated to emit white light. The binocular camera module acquires external image information through the light guide plate and hydrogel, determines whether the brightness of the RGB-LED module is appropriate, adjusts the brightness of the RGB-LED module based on the conclusion, and controls the ultraviolet light group to be in the off state.

[0016] Step S2: After the RGB-LED light module is adjusted to a suitable brightness, the binocular camera module collects external image information through the light guide plate and hydrogel and sends it out.

[0017] Step S3: When the second control command is received, the ultraviolet lamp group is lit, and the RBG-LED lamp module is controlled to emit red, green and blue light in different areas, and different colors are presented in sections on the inner surface of the hydrogel through the light guide plate.

[0018] Step S4: Calculate the deformation of the inner surface of the hydrogel using a binocular camera module, and obtain the pressure magnitude of each part based on the collected deformation.

[0019] Step S4 includes:

[0020] Step S41: Acquire images captured by the two cameras in the binocular camera module and calculate the correlation.

[0021] Step S42: Obtain the disparity map by selecting and optimizing the deviation value;

[0022] Step S43: By further analyzing the depth of field through the disparity map, effective pressure data can be obtained.

[0023] The correlation degree is calculated using the NCC algorithm.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) Integrating object positioning and grasping force sensing within the same sensor can be applied to the positioning and sensing capabilities of humanoid robotic fingers, greatly improving dexterity. Existing technologies use separate visual sensors and separate visual-tactile sensors; currently, there is no sensor that integrates both. Directly splicing the two sensors will cause the camera positions to deviate, resulting in inconsistencies between the grasping point sensed by the visual sensor and the pressure point sensed by the visual-tactile pressure sensor. Installing a photochromic hydrogel visual-tactile sensor at the end of the finger-like or claw-like structure of the robotic hand can effectively combine the advantages of visual and tactile sensors while ensuring consistency between the visual and pressure sensing points. When no ultraviolet light is applied to the hydrogel, it remains transparent, thus serving as a window for a binocular camera to observe the outside world. When exposed to ultraviolet light, the optical properties of the hydrogel change, and its transparency decreases. At this time, images can be acquired by a camera, and the shape of the contact surface can be inferred through machine vision algorithms to calculate pressure data and determine whether the current pressure value meets the predetermined parameters, thus making a decision.

[0026] (2) The system achieves object localization and grasping force sensing functions with a compact structure and low cost, making it particularly practical for robotic end effectors. Compared to methods that combine camera sensing with pressure sensing, the system reduces the volume of at least one embedded camera sensor structure, achieving a size similar to a single set of visual-tactile pressure sensors. The hydrogel membrane's optical properties can be altered by ultraviolet light irradiation, thus reducing the number of visual sensors required. Only a pair of miniature binocular cameras is needed to achieve all functions. This characteristic allows for control over the overall structural size, enabling installation on smaller robotic end effectors or freeing up more space for other functions. Furthermore, the reduced number of visual sensors results in lower costs when achieving the same functionality.

[0027] (3) Visual algorithms have an advantage in calculating depth of field for parts with different colors but the same gray level. Currently, the main visual algorithms used in visual tactile pressure sensors are concentrated in RGB-LED light group single camera algorithms, binocular vision algorithms, and marker array algorithms. Among them, binocular vision algorithms are affected to some extent when processing color blocks with different colors but the same gray level, while the RGB-LED light group single camera algorithm is affected by the single camera to varying degrees. Compared with the traditional binocular vision algorithm, the method of processing red, green, and blue channels separately can more effectively calculate depth of field in specific color areas due to the color partitioning illumination of RGB-LED light groups. Combining the results obtained from the three color channels can effectively solve the problem of depth of field calculation for parts with different colors but the same gray level. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the structure of the present invention;

[0029] Figure 2 This is a schematic diagram of the structure after the front cover has been removed;

[0030] Figure 3 A schematic diagram showing the arrangement of the binocular camera module and the ultraviolet lamp assembly;

[0031] Figure 4 This is a schematic diagram showing the arrangement of parts on the back cover;

[0032] Figure 5 This is a schematic diagram of the middle frame;

[0033] Figure 6 This is a schematic diagram of the back cover.

[0034] Figure 7 This is a schematic diagram of the front cover structure;

[0035] Figure 8 This is a schematic diagram showing the arrangement of the hydrogel and the light guide plate;

[0036] Figure 9 This is a schematic diagram illustrating the setup of the hydrogel.

[0037] Figure 10 This is a schematic diagram of the control logic;

[0038] The components include: 1. Rear cover, 2. Binocular camera module, 3. Ultraviolet lamp group, 4. RBG-LED lamp module, 5. Pre-drilled wiring holes, 6. Mid-frame, 7. Light guide plate, 8. Front cover, and 9. Hydrogel. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0040] A multimodal visual-tactile system based on color-changing gel, such as Figures 1 to 9 As shown, the device includes a rigid housing, and a binocular camera module 2, an ultraviolet lamp group 3, a light guide plate 7, and an RBG-LED lamp module 4 placed inside the rigid housing. One side of the rigid housing has an opening, and a hydrogel 9 is placed on the opening. The hydrogel 9 is located on the outside of the light guide plate 7. The binocular camera module 2 and the ultraviolet lamp group 3 are located on the inside of the light guide plate 7. The ultraviolet lamp group 3 is located between the two cameras of the binocular camera module 2.

[0041] The rigid housing includes a middle frame 6, a front cover 8, and a rear cover 1. The binocular camera module 2 and the ultraviolet lamp group 3 are mounted on the rear cover 1, and the opening is located on the front cover 8. In addition, a reserved hole 5 is formed on the side of the middle frame 6 and the rear cover 1. This reserved hole is used for the passage of related power supply lines and signal lines, for powering the RBG-LED lamp module 4, the binocular camera module 2, and the ultraviolet lamp group 3, and for transmitting control signals and data signals.

[0042] Preferably, in some embodiments, the RBG-LED light module 4 is an RBG-LED light strip, which is arranged around the side of the light guide plate 7. Of course, in other embodiments, a split RBG-LED light module 4 design can also be adopted, specifically including individual RBG-LED beads. Each RBG-LED bead is arranged around the side of the light guide plate 7.

[0043] In this embodiment, the projection of the centers of the two cameras of the binocular camera module 2 onto the cross-section of the hydrogel 9 is located at the center of the hydrogel, and the two cameras of the binocular camera module 2 are mounted directly opposite the opening where the rigid shell and the hydrogel 9 are joined, thereby obtaining a better field of view and reducing the distortion of the acquired image.

[0044] Specifically, the hydrogel used is a photochromic material. During its preparation, dodecyl dimethyl betaine, a positively charged surfactant, is used. This surfactant binds to molybdenum through an electric field, utilizing the synergistic effect of chemical cross-linking and hydrophobic association to enhance the mechanical properties of the photochromic hydrogel. The resulting hydrogel exhibits excellent transparency, extensibility, and puncture resistance. When not exposed to ultraviolet light, the hexavalent molybdenum in the hydrogel is colorless. Under ultraviolet light, the hexavalent molybdenum gains electrons and is reduced to pentavalent molybdenum, exhibiting green or blue hues. In the air, oxygen gains electrons from the pentavalent molybdenum, oxidizing it back to hexavalent molybdenum, resulting in a colorless state. During the color transition, molybdenum maintains good stability because its structure remains unchanged. During the hydrogel's preparation, the molybdenum-containing compound should be uniformly mixed within the hydrogel and maintained at a relatively low concentration to ensure uniform changes in optical properties without excessively reducing transparency, thus preventing the acquisition of deformation images of the contact surfaces.

[0045] Utilizing the properties of the photochromic hydrogel described above, when the robotic arm is performing a grasping action but has not yet made contact with an object, the ultraviolet lamp group 3 is not turned on, and the hydrogel remains transparent, allowing light to pass through. This allows the binocular camera to capture images outside the opening of the rigid shell. When the grasping action begins and contact is made with the object, the ultraviolet lamp group 3 is turned on. Under the influence of ultraviolet light, the hydrogel changes its transparency after a response time, decreasing in transparency. At this point, the deformation of the hydrogel can be identified by the binocular camera, and the pressure on different points of the membrane can be determined by machine vision algorithms, serving as further criteria for the robotic arm's motion decisions. After the grasping action ends, the light information is erased, causing the hydrogel to return to a transparent state.

[0046] The cross-sectional area of ​​the light guide plate 7 is larger than the cross-sectional area of ​​the hydrogel 9.

[0047] Generally, the above system is installed as a single sensor at the end of the finger-like or claw-like structure of a robotic arm. When it needs to grasp an object, parameters need to be set according to the object to be grasped. Specifically, a control method based on the above-mentioned multimodal vision-touch system based on color-changing gel is as follows: Figure 10 As shown, it includes:

[0048] Step S1: Upon receiving the first control command, the first control command instructs the device to move to a suitable grasping position, illuminates the RGB-LED module 4 to emit white light, and the binocular camera module collects external image information through the light guide plate and hydrogel. It then determines whether the brightness of the RGB-LED module is appropriate, adjusts the brightness of the RGB-LED module based on the conclusion, and controls the ultraviolet light group 3 to remain off.

[0049] Step S2: After the RGB-LED light module is adjusted to a suitable brightness, the binocular camera module 2 acquires external image information through the light guide plate 7 and hydrogel 9 and sends it outwards. At this time, the machine vision algorithm is executed according to the preset parameters, and the robot arm is driven to adjust its pose to meet the set parameters based on the acquired data. During this process, the binocular camera module 2 acquires images of the outside world and compares them with the set grasping point parameters. Due to the presence of the light guide plate 7 and hydrogel 9, the clarity of the acquired image will be affected to some extent, but the binocular camera can still be used to measure the depth of field and determine the distance to the grasped object. At the same time, a blurred image algorithm can also be used in actual processing. When determining the distance to the grasped object, the distance is calculated by calculating the change ratio of the object's outline and the change in the robot arm's pose. When determining whether it is aligned with the grasping point, it can be calculated by comparing the object's outline and feature points with the set parameters, such as the color block deviation.

[0050] Step S3: Upon receiving the second control command, which instructs the robot to grasp the target, the ultraviolet lamp group 3 is illuminated, and the RBG-LED lamp module 4 is controlled to emit red, green, and blue light in different areas. These colors are then distributed across the inner surface of the hydrogel 9 via the light guide plate 7, initiating the grasping action. Further analysis of the depth of field yields effective pressure data, which is used to adjust the gripping force of the robotic arm.

[0051] Step S4: The deformation of the inner surface of the hydrogel 9 is acquired using the binocular camera module 2, and the pressure magnitude at each location is obtained based on the acquired deformation. Since the binocular camera module 2 captures two images with slight differences, these two images are used to reconstruct the depth of field of the contact surface. First, binocular calibration and binocular correction are performed on the images. Then, correlation calculation is performed on the images, and minimum spanning tree filtering is used to improve the calculation results. Next, a disparity map is obtained through deviation value selection and optimization. Finally, effective pressure data can be obtained through further analysis of the depth of field using the disparity map, and this data is used to drive the grip force adjustment of the robotic arm.

[0052] The correlation calculation in the above process uses the NCC algorithm. Directly using the image's grayscale values ​​for NCC calculations may lead to errors in the results due to differences in image colors but similar grayscale values. To avoid potential errors, an RGB-LED light group is used inside the rigid shell. During shooting, this light group adjusts the colors to distinguish the main hues of the hydrogel in different locations as red, green, and blue. Therefore, when using the NCC algorithm to normalize the correlation of the target to be matched, the grayscale values ​​of the red, green, and blue channels can be extracted separately and then normalized using the NCC algorithm for each channel. The correlation formula for the red channel is as follows:

[0053] in

[0054] Where: C(u,v,d) represents the similarity level, with a value range of [-1,1]; R l R represents the grayscale value of the red channel in the left image. r Let x be the grayscale value of the red channel in the right image, and y be the x-coordinate of the pixel in the left image calculated. In the formula, (2ρ+1) represents the side length of the block, and μ... Rl μ is the average grayscale value of the red channel in the selected area of ​​the left image. Rr σ represents the average grayscale value of the red channel in the selected area of ​​the right image. Rl σ is the standard deviation of the red channel grayscale values ​​of the selected block in the left image. Rr The standard deviation of the red channel grayscale values ​​for the selected block in the right image is given. u and v are the center points of the selected block in the left image, d is the positional deviation between the left and right image blocks, and n is the number of pixels within the block. The above formula is for the red channel; the formulas for the green and blue channels can be derived analogously. Since the three colors only occupy the dominant hue within specific areas of the hydrogel, the resulting disparity map only has a good effect within the corresponding areas. Therefore, a visual algorithm should be used to optimize and stitch the disparity map generated in S3.

[0055] Calculate the pressure data. If the pressure data does not match the preset parameters, adjust the gripping force of the robotic arm. If the pressure data matches the preset parameters, the gripping action is achieved, and the process ends.

[0056] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A multi-modal visual-haptic system based on a color-changing gel, characterized in that, The rigid shell comprises a middle frame (6), a front cover (8) and a rear cover (1), the binocular camera module (2) and the ultraviolet lamp group (3) are arranged on the rear cover (1), and the opening is arranged on the front cover (8).

2. A multi-modal visual-haptic system based on a color-changing gel according to claim 1, characterized in that, The RGB-LED lamp module (4) is an RGB-LED lamp strip.

3. A multi-modal visual-haptic system based on a color-changing gel according to claim 1, characterized in that, The projection of the centers of the two cameras of the binocular camera module (2) on the cross section of the hydrogel (9) is located at the center of the hydrogel.

4. The multi-modal visual-haptic system based on a color-changing gel according to claim 1, wherein, The RGB-LED lamp module (4) is arranged around the light guide plate (7).

5. The multi-modal visual-haptic system based on a color-changing gel according to claim 1, wherein, The cross-sectional area of the light guide plate (7) is larger than that of the hydrogel (9).

6. The multi-modal visual-haptic system based on a color-changing gel according to claim 1, wherein, The system is installed at the end of a mechanical hand finger-shaped structure or a claw-shaped structure.

7. A multi-modal visual-haptic system based on a color-changing gel according to any one of claims 1-6, characterized in that, Comprise:

8. A method of controlling a multi-modal visual-haptic system based on the color-changing gel based multi-modal visual-haptic system of claim 7, characterized in that, Step S1: when receiving the first control instruction, light the RGB-LED lamp module (4) to emit white light, according to the binocular camera module (2) through the light guide plate (7) and the hydrogel (9) to collect the image outside, judge whether the brightness of the RGB-LED lamp module is appropriate, and change the brightness of the RGB-LED lamp module according to the conclusion, and control the ultraviolet lamp group (3) to be in the closed state, Step S2: after the RGB-LED lamp module (4) is adjusted to the appropriate brightness, the binocular camera module (2) collects the image information outside through the light guide plate (7) and the hydrogel (9) and sends it outside; Step S3: when receiving the second control instruction, light the ultraviolet lamp group (3), control the RGB-LED lamp module (4) to emit red, green and blue light in different regions, and present different colors on the inner surface of the hydrogel (9) through the light guide plate (7); Step S4: calculate the deformation of the inner surface of the hydrogel (9) through the binocular camera module (2), and obtain the pressure of each part based on the collected deformation. The step S4 comprises:

9. The control method according to claim 8, characterized by, Step S41: obtain the images collected by the two cameras in the binocular camera module (2), and calculate the correlation degree; Step S42: obtain the disparity map through deviation value selection and optimization; Step S43: through further analysis of the depth of field through the disparity map, effective pressure data can be obtained. The correlation degree calculation uses NCC algorithm.

10. The control method according to claim 9, characterized by The rigid shell comprises a middle frame (6), a front cover (8) and a rear cover (1), the binocular camera module (2) and the ultraviolet lamp group (3) are arranged on the rear cover (1), and the opening is arranged on the front cover (8). The RGB-LED lamp module (4) is an RGB-LED lamp strip. The projection of the centers of the two cameras of the binocular camera module (2) on the cross section of the hydrogel (9) is located at the center of the hydrogel. The RGB-LED lamp module (4) is arranged around the light guide plate (7). The cross-sectional area of the light guide plate (7) is larger than that of the hydrogel (9). The system is installed at the end of a mechanical hand finger-shaped structure or a claw-shaped structure. Comprise: Step S1: when receiving the first control instruction, light the RGB-LED lamp module (4) to emit white light, according to the binocular camera module (2) through the light guide plate (7) and the hydrogel (9) to collect the image outside, judge whether the brightness of the RGB-LED lamp module is appropriate, and change the brightness of the RGB-LED lamp module according to the conclusion, and control the ultraviolet lamp group (3) to be in the closed state, Step S2: after the RGB-LED lamp module (4) is adjusted to the appropriate brightness, the binocular camera module (2) collects the image information outside through the light guide plate (7) and the hydrogel (9) and sends it outside; Step S3: when receiving the second control instruction, light the ultraviolet lamp group (3), control the RGB-LED lamp module (4) to emit red, green and blue light in different regions, and present different colors on the inner surface of the hydrogel (9) through the light guide plate (7); Step S4: calculate the deformation of the inner surface of the hydrogel (9) through the binocular camera module (2), and obtain the pressure of each part based on the collected deformation. The step S4 comprises: Step S41: obtain the images collected by the two cameras in the binocular camera module (2), and calculate the correlation degree; Step S42: obtain the disparity map through deviation value selection and optimization; Step S43: through further analysis of the depth of field through the disparity map, effective pressure data can be obtained. The correlation degree calculation uses NCC algorithm.

Images