Visual tactile sensing system and method based on visual-magnetic fusion

Abstract

The invention discloses a visual tactile sensing system and method based on visual-magnetic fusion, and the method comprises the steps: isolating an external light source through a surface coating, reconstructing the surface texture of a sensed object, and obtaining the texture deformation information; the texture deformation information of the surface coating is received based on a camera, image information is obtained, force measurement is carried out through mark points arranged between the surface coating and the transparent silica gel, and standard force is obtained; magnetic field intensity change information caused when the postures of the marking points change is collected through a Hall sensor; based on the image information, the standard force and the magnetic field intensity change information, force measurement of the sensed object is achieved, and a force measurement result is obtained. The method provided by the invention can measure the high-precision multi-dimensional displacement of the mark point, has faster response or frequency, and realizes the functions of high-precision force measurement, high response and proximity sense of the visual tactile sensor.

Description

Technical Field

[0001] The present invention belongs to the field of visual-tactile sensing, and in particular relates to a visual-tactile sensing system and method based on visual-magnetic fusion. Background Art

[0002] For the visual tactile sensor currently using a monocular camera, the visual tactile sensor uses the deformation layer on the surface of the elastic deformable body to realize the state perception of external objects, and a light source is set inside. Therefore, the surface of the conventional visual tactile sensor has an opaque coating, and the marking points under the surface coating are usually used to realize force measurement and deformation perception of the surface silicone. It is impossible to obtain relevant information before contact with the outside world. This method of not being able to perceive before contact is not conducive to robot operation. At the same time, the image information received by the camera converts the three-dimensional displacement change information of the marking point into 2D information, which leads to information loss. When training the collected data, the accuracy of force measurement will be limited by using incomplete information. The response rate of the visual tactile sensor depends on the camera's receiving frequency, image processing and other processes, which will fundamentally limit the response frequency of the sensor, and then limit the application scenarios of sensor perception. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a visual-tactile sensing system and method based on visual-magnetic fusion. Among them, a visual-tactile sensing system based on visual-magnetic fusion includes:

[0004] The surface coating, transparent silicone, elastomer, transparent support plate, and camera are arranged in sequence from top to bottom;

[0005] The surface coating is used to isolate external light sources and reconstruct the surface texture of the perceived object;

[0006] The camera is used to receive the deformation information of the surface coating to realize information collection.

[0007] Preferably, a marking point is provided between the surface coating and the transparent silica gel, and data is collected through the marking point to obtain an image of displacement change of the marking point.

[0008] Preferably, the camera is also used to receive position change information of the marking point.

[0009] Preferably, the marking point is a rigid small hard magnet.

[0010] Preferably, the system further comprises a Hall sensor, which is arranged at the bottom of the system and is in the same plane as the camera, and is used to collect information on changes in magnetic field intensity caused by changes in the posture of the small magnet.

[0011] Preferably, the camera is a monocular camera.

[0012] The present invention also provides a visual-tactile sensing method based on visual-magnetic fusion, comprising:

[0013] The surface coating is used to isolate the external light source, and the surface texture of the perceived object is reconstructed to obtain texture deformation information;

[0014] Receiving texture deformation information of the surface coating based on a camera to obtain image information;

[0015] Data is collected through the marking points set between the surface coating and the transparent silicone to obtain a displacement change image of the marking points; and information on changes in magnetic field intensity caused by changes in the position of the marking points is collected through the Hall sensor;

[0016] The force measurement network model is trained by marking the change image, the change information of the magnetic field intensity and the standard force;

[0017] Based on the image information, the magnetic field intensity change information and the trained force measurement network model, the force measurement of the sensed object is achieved to obtain the force measurement result.

[0018] Preferably, the force is measured by setting a marking point between the surface coating and the transparent silicone, and the process of obtaining the standard force includes:

[0019] The marking point is a rigid hard small magnet. When the surface coating is deformed, the camera collects 2D information of the position change of the small magnet to obtain the 2D displacement change image of the small magnet caused by the posture change. At the same time, the Hall sensor at the bottom synchronously receives the magnetic field strength change information caused by the posture change of the small magnet as a force information supplement to the 2D displacement change image of the small magnet.

[0020] Compared with the prior art, the present invention has the following advantages and technical effects:

[0021] The method of the present invention is based on the perception mode of visual-tactile sensor combined with magnetic induction, which solves the problems of the current visual-tactile sensor in the case of a monocular camera, such as the lack of receiving 3D position information of the marking point, the limited camera frequency and response rate, and the lack of perception ability of the state information of the operating object not in contact, and realizes the high-precision force measurement, high response and proximity perception functions of the visual-tactile sensor.

[0022] The method of the present invention can measure high-precision multi-dimensional displacement of a marking point and has a faster response or frequency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0025] Figure 2 It is a schematic diagram of structural simulation of an embodiment of the present invention;

[0026] Figure 3 Schematic diagram of a multimodal perception method according to an embodiment of the present invention.

[0027] Figure 4 A schematic diagram of a force measurement network model according to an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of proximity sensing verification according to an embodiment of the present invention;

[0029] Figure 6 This is a diagram of force measurement results of an embodiment of the present invention;

[0030] Among them, 1. a first elastic body; 2. a magnetic particle marking point; 3. a light source; 4. a second elastic body; 5. a support plate; 6. a Hall sensor circuit board; 7. a connector; 8. a base; and 9. a camera. DETAILED DESCRIPTION

[0031] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0032] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] like Figure 1-6 As shown, this embodiment provides a visual-tactile sensing system based on visual-magnetic fusion, including:

[0034] The surface coating, transparent silicone, elastic body 1, transparent support plate 5, and camera 9 are arranged in sequence from top to bottom;

[0035] Surface coating, used to isolate external light sources and reconstruct the surface texture of the perceived object;

[0036] The camera 9 is used to receive deformation information of the surface coating to realize information collection.

[0037] Furthermore, a marking point is provided between the surface coating and the transparent silica gel, and data is collected through the marking point to obtain an image of displacement change of the marking point.

[0038] Furthermore, the camera 9 is also used to receive position change information of the marking point.

[0039] Furthermore, the marking point is a rigid hard small magnet.

[0040] Furthermore, the system also includes a Hall sensor circuit board 6, the Hall sensor of the Hall sensor circuit board 6 is arranged at the bottom of the system and is in the same plane as the camera 9, and is used to collect information on changes in magnetic field strength caused by changes in the position of the small magnet.

[0041] This embodiment uses "visual images" and "magnetic field information" as the basis for measurement, and combines them with each other. The magnetic field information makes up for the missing information of the camera receiving the 3D displacement of the marker, and increases the information dimension of the sensor marking point. At the same time, using magnetic induction, the Hall sensor can quickly receive changes in the magnetic field to achieve fast response and high-frequency measurement. At the same time, for all external objects that will cause changes in the magnetic field, the "proximity perception function" can be realized. Using a visual camera, the missing displacement realizes force measurement, and the surface texture information of the object is also within the perception range. The two different perception methods enhance the perception ability of the sensor and expand the perception range.

[0042] Furthermore, the camera 9 is a monocular camera. By selecting a monocular camera, the sampling frequency of the camera can be increased and the time it takes for the sensor to receive a response can be shortened.

[0043] Furthermore, the structure simulation diagram of the visual-tactile sensing system of this embodiment is as follows: Figure 2 As shown, it includes: a first elastic body 1, a magnetic particle marking point 2, a light source 3, a second elastic body 4, a support plate 5, a Hall sensor circuit board 6, a connector 7, a base 8, and a camera 9.

[0044] This embodiment also provides a visual-tactile sensing method based on visual-magnetic fusion, including:

[0045] The surface coating is used to isolate the external light source, and the surface texture of the perceived object is reconstructed to obtain texture deformation information;

[0046] Receiving texture deformation information of the surface coating based on a camera to obtain image information;

[0047] Data is collected through the marking points set between the surface coating and the transparent silicone to obtain a displacement change image of the marking points; and information on changes in magnetic field intensity caused by changes in the position of the marking points is collected through the Hall sensor;

[0048] The force measurement network model is trained by marking the change image, the change information of the magnetic field intensity and the standard force;

[0049] Based on the image information, the magnetic field intensity change information and the trained force measurement network model, the force measurement of the sensed object is achieved to obtain the force measurement result.

[0050] Furthermore, the force is measured by setting a marking point between the surface coating and the transparent silicone, and the process of obtaining the standard force includes:

[0051] The marking point is a rigid hard small magnet. When the surface coating is deformed, the camera collects 2D information of the position change of the small magnet to obtain the 2D displacement change image of the small magnet caused by the posture change. At the same time, the Hall sensor at the bottom synchronously receives the magnetic field strength change information caused by the posture change of the small magnet as a force information supplement to the 2D displacement change image of the small magnet.

[0052] Furthermore, if Figure 4 The force measurement network model schematic diagram is shown in the figure. This embodiment realizes the fusion of visual and magnetic information by splicing and integrating the RGB three-channel information of the tactile image with the two-dimensional information of the magnetic tactile image. The results of the proximity perception verification for watches, headphones, mobile phones, and mice are shown in the figure. Figure 5 shown.

[0053] Furthermore, the force measurement of the sensed object is realized based on the image information, the standard force and the magnetic field intensity change information, and the force measurement result is obtained as follows: Figure 6 As shown, Figure 6 From top to bottom and from left to right in the figure are the line graphs of magnetic tactile measurement results, visual tactile force measurement results, visual-magnetic fusion measurement results, and visual-magnetic estimation verification results. In the line graph of visual-magnetic estimation verification results, red is the true value and green is the estimated value.

[0054] The method of the present invention is based on the perception mode of visual-tactile sensor combined with magnetic induction, which solves the problems of the current visual-tactile sensor in the case of a monocular camera, such as the lack of receiving 3D position information of the marking point, the limited camera frequency and response rate, and the lack of perception ability of the state information of the operating object not in contact, and realizes the high-precision force measurement, high response and proximity perception functions of the visual-tactile sensor.

[0055] The method of the present invention can measure high-precision multi-dimensional displacement of a marking point and has a faster response or frequency.

[0056] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A visual-tactile sensing system based on visual-magnetic fusion, characterized in that: include: The surface coating, transparent silicone, elastomer, transparent support plate, and camera are arranged in sequence from top to bottom; The surface coating is used to isolate external light sources and reconstruct the surface texture of the perceived object; The camera is used to receive the deformation information of the surface coating to realize information collection.

2. The visual-tactile sensing system based on visual-magnetic fusion according to claim 1, characterized in that: A marking point is arranged between the surface coating and the transparent silica gel, and data is collected through the marking point to obtain a displacement change image of the marking point.

3. The visual-tactile sensing system based on visual-magnetic fusion according to claim 2 is characterized in that: The camera is also used to receive position change information of the marking point.

4. The visual-tactile sensing system based on visual-magnetic fusion according to claim 2, characterized in that: The marking point is a rigid hard small magnet.

5. The visual-tactile sensing system based on visual-magnetic fusion according to claim 1, characterized in that: The system also includes a Hall sensor, which is arranged at the bottom of the system and is in the same plane as the camera, and is used to collect information on changes in magnetic field intensity caused by changes in the position of the small magnet.

6. The visual-tactile sensing system based on visual-magnetic fusion according to claim 1, characterized in that: The camera is a monocular camera.

7. A visual-tactile sensing method based on visual-magnetic fusion, characterized in that: include: The surface coating is used to isolate external light sources, and the surface texture of the perceived object is reconstructed to obtain texture deformation information; Receiving texture deformation information of the surface coating based on a camera to obtain image information; Data is collected through the marking points set between the surface coating and the transparent silicone to obtain an image of the displacement change of the marking points; and collecting information on changes in magnetic field intensity caused by changes in the position of the marking point through a Hall sensor; The force measurement network model is trained by marking the change image, the change information of the magnetic field intensity and the standard force; Based on the image information, the magnetic field intensity change information and the trained force measurement network model, the force measurement of the sensed object is achieved to obtain the force measurement result.

8. The visual-tactile sensing method based on visual-magnetic fusion according to claim 7, characterized in that: The force is measured by setting a mark point between the surface coating and the transparent silicone. The process of obtaining the standard force includes: The marking point is a rigid hard small magnet. When the surface coating is deformed, the camera collects 2D information of the position change of the small magnet to obtain the 2D displacement change image of the small magnet caused by the posture change. At the same time, the Hall sensor at the bottom synchronously receives the magnetic field strength change information caused by the posture change of the small magnet as a force information supplement to the 2D displacement change image of the small magnet.

Images