Visual tactile sensor, manipulator and control method thereof

Abstract

A visual tactile sensor, a manipulator and a control method thereof, the visual tactile sensor comprising: a contact layer, at least a portion of which is transparent; when the contact layer is subjected to external force, at least one part of a plurality of markers in the marker array generates displacement and / or deformation; and the image acquisition unit is used for performing image acquisition on the marker array and performing image acquisition on the outside through the transparent part of the contact layer. The visual tactile sensor provided by the invention not only can capture the external visual information before operation to identify the operation object, but also can provide the tactile information in the operation process, so that the perception capability of an operation system is enhanced, and the operation success rate is improved.

Description

Technical Field

[0001] This application relates to the field of sensor technology, specifically to a visual-tactile sensor, a robotic arm, and a control method thereof. Background Technology

[0002] With the rapid development of technologies such as robotics and human-computer interaction, acquiring high-precision, multi-dimensional tactile information has become one of the important ways for intelligent systems to perceive their environment. As an emerging technology that combines visual and tactile perception capabilities, the visual-tactile sensor has attracted much attention due to its simple structure, ease of integration, and ability to perceive rich information. Summary of the Invention

[0003] One of the technical objectives of this application is to provide an improved visual-tactile sensor.

[0004] In a first aspect, embodiments of this application provide a visual-tactile sensor, comprising: a contact layer, at least a portion of which is transparent; a marker array, wherein when the contact layer is subjected to an external force, at least a portion of a plurality of markers in the marker array undergoes displacement and / or deformation; and an image acquisition unit, configured to acquire images of the marker array and to acquire images of the outside through the transparent portion of the contact layer.

[0005] Optionally, the marker array includes a first marker and a second marker, wherein the colors of the first marker and the second marker are different; or, the marker has a first portion and a second portion, wherein the colors of the first portion and the second portion are different.

[0006] Optionally, the image acquisition unit includes an auto-zoom camera; or, the image acquisition unit includes a first camera and a second camera, wherein the first camera is used to acquire images of the transparent portion and the second camera is used to acquire images of the marker array.

[0007] Optionally, it further includes: a light source for providing illumination for image acquisition performed by the image acquisition unit; wherein the light source has an external mode and an internal mode, the external mode for providing illumination when the image acquisition unit performs image acquisition of the external environment, and the internal mode for providing illumination when the image acquisition unit performs image acquisition of the marker array.

[0008] Optionally, the marker array is detachably disposed on the outer surface of the contact layer; or, the visual-tactile sensor further includes a housing with an opening, to which the contact layer is detachably fixed.

[0009] Optionally, the marker array is a textured structure formed on the outer or inner surface of the contact layer.

[0010] Optionally, it also includes: a processor for analyzing the images acquired by the image acquisition unit to obtain the characterization parameters of the external force and / or the external environmental information.

[0011] Secondly, embodiments of this application provide a robotic hand, including: a gripping part; a visual-tactile sensor provided in the first aspect, disposed on the gripping part; and a robotic hand controller connected to the gripping part and the visual-tactile sensor, wherein the robotic hand controller controls the gripping part according to the output result of the visual-tactile sensor.

[0012] Thirdly, embodiments of this application provide a control method for a robotic arm, the robotic arm including the visual-tactile sensor provided in the first aspect, the method including: acquiring a first acquired image from the image acquisition unit; identifying an image of an external object from the first acquired image; determining attribute information of the external object based on the image of the external object; and determining a grasping strategy of the robotic arm for the external object based on the attribute information of the external object.

[0013] Optionally, the attribute information includes at least one of the following: pose, category, surface texture features, and stiffness estimate.

[0014] Optionally, determining the grasping strategy of the robotic arm on the external object based on the attribute information of the external object includes: inputting the attribute information into a grasping strategy generation model to obtain a grasping strategy output by the grasping strategy generation model; wherein the grasping strategy includes at least one of the following: the grasping position of the object, the path of the robotic arm approaching the object, the grasping posture of the robotic arm, and the initial force of the robotic arm grasping.

[0015] Optionally, the method further includes: acquiring a second acquired image from the image acquisition unit; identifying the image of the marker array from the second acquired image; determining the first current pose of each marker in the marker array based on the image of the marker array; determining whether the external object is in contact with the contact layer based on the first current pose and the reference pose of each marker; and determining the characterization parameters of the external force applied by the external object in response to the object contacting the contact layer.

[0016] Optionally, determining the characterization parameters of the external force applied by the external object includes: acquiring a third acquired image from the image acquisition unit; extracting the second current pose of each marker in the marker array from the third acquired image; determining the displacement and / or deformation of each marker based on the second current pose of each marker in the marker array and the reference pose; and determining the characterization parameters of the external force based on the displacement and / or deformation of each marker in the marker array.

[0017] Optionally, before acquiring the second captured image, the method further includes: in response to the external object contacting the contact layer, sending a mode switching command to the light source, the mode switching command being used to control the light source to switch from an external mode to an internal mode.

[0018] Fourthly, embodiments of this application provide a control device for a robotic arm, the robotic arm including the visual-tactile sensor provided in the first aspect, the device comprising: an acquisition module for acquiring a first acquired image from the image acquisition unit; a recognition module for recognizing an image of an external object from the first acquired image; an attribute recognition module for determining attribute information of the external object based on the image of the external object; and a strategy generation module for determining a grasping strategy of the robotic arm for the external object based on the attribute information of the external object.

[0019] Fifthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is run by a processor, the control method provided in the third aspect is executed.

[0020] The storage medium may include ROM, RAM, disk, or optical disk, etc. The storage medium may also include non-volatile or non-transitory memory, etc.

[0021] In a sixth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the control method provided in the third aspect.

[0022] Compared with the prior art, the technical solution of this application embodiment has the following beneficial effects:

[0023] This application provides a visual-tactile sensor, comprising: a contact layer, at least a portion of which is transparent; a marker array, wherein when the contact layer is subjected to an external force, at least a portion of a plurality of markers in the marker array undergoes displacement and / or deformation; and an image acquisition unit, configured to acquire images of the marker array and to acquire images of the outside through the transparent portion of the contact layer.

[0024] By adopting the above scheme, the contact layer has dual functions of perspective and force sensing. The transparent part of the contact layer allows the image acquisition unit in the visual-touch sensor to directly capture external information (such as environmental information or the object to be grasped) through the contact layer, realizing external visual perception. When the contact layer is subjected to external force, the contact layer transmits the external force to the marker array, causing at least a portion of the markers in the marker array to undergo displacement and / or deformation. The image acquisition unit can then observe the displacement and / or deformation of the marker array, calculate the external force information, and realize tactile perception. That is, in the scheme of this application, the image acquisition unit inside the visual-touch sensor is responsible for both acquiring external visual images and capturing the displacement and / or deformation of the markers to calculate force information. This ensures that internal and external information originate from the same sensor and the same viewpoint, guaranteeing the spatiotemporal synchronization and consistency of perception, and eliminating calibration errors and data delays between different sensors. In addition, compared with the scheme of additionally setting a camera outside the visual-touch sensor, the above scheme also reduces hardware costs and integration difficulty, and simplifies the overall structure. Attached Figure Description

[0025] Figure 1 This is a cross-sectional view of a visual-tactile sensor according to an embodiment of this application;

[0026] Figure 2 This is a top view of a marker array in an embodiment of this application;

[0027] Figure 3 This is a top view of another marker array in an embodiment of this application;

[0028] Figure 4 This is a flowchart illustrating a sensing method in an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of a pre-contact visual perception scenario in an embodiment of this application;

[0030] Figure 6 This is a flowchart illustrating a control method for a robotic arm according to an embodiment of this application;

[0031] Figure 7 This is a flowchart illustrating a tactile sensing method according to an embodiment of this application;

[0032] Figure 8 This is a schematic diagram of the structure of a control device for a robotic arm according to an embodiment of this application;

[0033] Figure 9 This is a schematic diagram of a tactile perception scenario after contact, as described in an embodiment of this application. Detailed Implementation

[0034] A visual-tactile sensor is a device that captures visual information using visual means and calculates tactile information from that visual information. In related technologies, visual-tactile sensors typically use a transparent, flexible material as the force-bearing object, and embed a set of markers on or within the surface of this material. When an external force is applied to the sensor surface, a camera tracks the displacement of these markers, and the sensor perceives and estimates the external force based on the tracked displacement.

[0035] Currently, the design of visual-tactile sensors focuses on acquiring information from within the sensor itself. To prevent external light from interfering with image acquisition by the camera and to ensure high image quality, a completely black silicone layer is typically added to the outer surface of a transparent, flexible material to achieve a light-blocking effect. With this structure, the visual-tactile sensor cannot acquire external information.

[0036] To obtain external information, an additional external camera, independent of the visual-tactile sensor, could be considered. This approach not only adds the camera but also requires additional brackets, cables, and calibration of the external camera, increasing both hardware costs and integration complexity, as well as algorithmic difficulty. Furthermore, the different shooting angles of the internal and external cameras can lead to information asynchrony, causing delays or errors in the robotic arm's grasping decisions.

[0037] In view of this, embodiments of this application provide a visual-tactile sensor, comprising: a contact layer, at least a portion of which is transparent; a marker array, wherein when the contact layer is subjected to an external force, at least a portion of a plurality of markers in the marker array undergoes displacement and / or deformation; and an image acquisition unit, configured to acquire images of the marker array and to acquire images of the outside through the transparent portion of the contact layer.

[0038] The visual-tactile sensor provided in this application can capture external visual information for object recognition before operation and provide tactile information during operation, which helps to enhance the perception ability of the operating system and improve the success rate of operation.

[0039] Specifically, using the above scheme, the contact layer has dual functions of perspective and force sensing. The transparent portion of the contact layer allows the image acquisition unit in the visual-touch sensor to directly capture external information (such as environmental information or the object to be grasped) through the contact layer, realizing external visual perception. When the contact layer is subjected to external force, the contact layer transmits the external force to the marker array, causing at least a portion of the markers in the marker array to undergo displacement and / or deformation. The image acquisition unit can then observe the displacement and / or deformation of the marker array, calculate the external force information, and realize tactile perception. That is, in the scheme of this application, the image acquisition unit inside the visual-touch sensor is responsible for both acquiring external visual images and capturing the displacement and / or deformation of the markers to calculate force information. This ensures that internal and external information originate from the same sensor and the same viewpoint, guaranteeing the spatiotemporal synchronization and consistency of perception, and eliminating calibration errors and data delays between different sensors. In addition, compared to the scheme of additionally setting a camera outside the visual-touch sensor, the above scheme also reduces hardware costs and integration difficulty, and simplifies the overall structure.

[0040] It should be noted that "grabbing" in this article can refer to an operation of a robotic arm, and the object to be grasped can be understood as the object of the operation.

[0041] To make the above-mentioned objectives, features and beneficial effects of this application more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0042] Reference Figure 1 , Figure 1 This is a cross-sectional view of a visual-tactile sensor according to an embodiment of this application. Figure 1 As shown, the visual-tactile sensor 100 may include: a contact layer 10, a marker array, an image acquisition unit 30, and a housing 70.

[0043] The housing 70 provides mechanical support for the various components inside the visual-tactile sensor 100, thereby providing a stable working platform for the visual-tactile sensor 100. In specific implementations, the material of the housing 70 can be engineering plastics, metals, etc., but is not limited to these, and this document does not limit it.

[0044] Furthermore, the housing 70 has an opening, and the contact layer 10 is disposed at the opening of the housing 70, so that the outer surface of the contact layer 10 is directly exposed to the external environment, so that the contact layer 10 can come into contact with an object. For example, the contact layer 10 can be fixed to the opening of the housing 70 by means of bolts or adhesive, but is not limited thereto.

[0045] In one embodiment of this application, the contact layer 10 can be detachably fixed to the opening of the housing 70. Exemplarily, the contact layer 10 is fixed to the opening in a detachable manner such as snap-on, magnetic, or threaded, but is not limited to these methods. This detachable structure facilitates the replacement of the contact layer 10 for maintenance of the visual-tactile sensor 100 (e.g., replacement if the contact layer 10 is worn) or to enable the visual-tactile sensor 100 to meet the sensing needs of different scenarios (e.g., replacing different arrangements of marker arrays).

[0046] In the embodiments of this application, at least a portion of the contact layer 10 is transparent. The transparent portion of the contact layer 10 allows the image acquisition unit 30 to observe the external environment.

[0047] For example, the contact layer 10 may be transparent and elastic. That is, the contact layer 10 may be a transparent elastomer. In specific implementations, the contact layer 10 may be made of a material with high light transmittance and high elasticity, such as transparent silicone, polyurethane elastomer, transparent hydrogel, special transparent rubber, or other optically transparent polymer materials.

[0048] For example, the contact layer 10 can be a multi-layer composite structure. Specifically, the contact layer 10 can include an outer layer portion and an inner layer portion, wherein the materials of the outer layer portion and the inner layer portion are different. Both the outer layer portion and the inner layer portion are transparent. For example, the outer layer material can be a wear-resistant transparent elastic material, and the inner layer portion can be a transparent elastic material that is sensitive to the displacement and / or deformation of the marker or has specific optical properties.

[0049] It should be noted that the shape of the contact layer 10 is not limited in this article. The contact layer 10 can be pointed, planar, curved (such as spherical), tubular, spherical, irregular curved, etc., to adapt to different robot end effectors or meet the needs of application scenarios.

[0050] In practical implementation, the contact layer 10 can have a certain thickness. The thickness direction of the contact layer 10 is perpendicular to the surface on which the contact layer 10 is located. For example... Figure 1 As shown, Figure 1 In the figure, the U direction is the upward direction along the thickness direction, and the D direction is the downward direction along the thickness direction.

[0051] It should also be noted that this article does not limit the thickness of the contact layer 10.

[0052] Furthermore, the contact layer 10 may have an outer surface 101 and an inner surface 102, wherein the outer surface 101 can be used to contact external objects, and the inner surface 102 is a surface that is closer to the image acquisition unit 30 than the outer surface 101, and the inner surface 102 faces the image acquisition unit 30.

[0053] Furthermore, in the embodiments of this application, when the contact layer 10 is subjected to an external force, at least a portion of the plurality of markers 200 in the marker array undergoes displacement and / or deformation. The displacement and / or deformation of each marker 200 can be used to calculate information about the external force.

[0054] The marker array in the embodiments of this application will be described in detail below.

[0055] In a specific implementation, the markers 200 can be arranged regularly in an array on or inside the contact layer 10 to form a marker array. For example, the marker array can be a square array or a hexagonal array, but is not limited to these.

[0056] In practice, the marker array can be a sparse array. This structure avoids excessive occlusion of external objects by the marker array, reducing the accuracy of pre-contact visual perception and preventing a mismatch between the grasping strategy and the external objects. The sparsity of the marker array can be set according to the actual application requirements. For example, a sparse array can refer to a plane in which the area occupied by the markers is no larger than the area of ​​the non-marker regions.

[0057] It should be noted that, in this document, "marker" can refer to an object used for optical tracking that can undergo identifiable displacement and / or deformation when the contact layer 10 is subjected to force. "Marker" can be a pattern, structure, component, etc., on the contact layer 10, and this application is not limited in this regard. For example, marker 200 can be in the form of dots, lines, etc.

[0058] In one example, the marker 200 can be a microfluidic channel containing microfluidic fluid. When the contact layer 10 is subjected to external force, the microfluidic channel deforms, and force sensing is performed by monitoring changes in fluid pressure or channel deformation. The microfluidic fluid and its channel can be transparent to avoid obstructing external visibility.

[0059] In one example, marker 200 could be made of fluorescent material. Using markers made of such material helps to improve the contrast between marker 200 and the external environment, thereby improving the accuracy of subsequent image recognition and segmentation.

[0060] In one example, marker 200 may be made of a material with retroreflective properties, which is beneficial to improving the brightness of marker 200 during imaging.

[0061] In one example, the marker array can be a textured structure formed on the outer or inner surface of the contact layer 10. For example, tiny textured structures can be formed on the surface of the contact layer 10 using processes such as etching, laser ablation, or microimprinting to serve as markers 200. Using this approach, no additional material is required to fabricate the markers 200.

[0062] In one example, marker 200 could be made of quantum dot material, utilizing the narrow emission spectrum and tunable color of quantum dot material to achieve high contrast and multi-channel encoding.

[0063] In one example, marker 200 can be a fine grid line, a randomly scattered micro dot pattern, or a preset pattern capable of producing moiré fringes, etc.

[0064] It should be noted that this article does not limit the position of marker 200 in contact layer 10.

[0065] For example, some or all of the markers 200 in the marker array may be located on the outer surface 101 of the contact layer 10. For instance, the marker array may be disposed on the outer surface 101 of the contact layer 10.

[0066] As another example, some or all of the markers 200 in the marker array may be located on the inner surface 102 of the contact layer 10. For example, the marker array may be disposed on the inner surface 102 of the contact layer 10.

[0067] As another example, some or all of the markers 200 in the marker array may be located inside the contact layer 10. The interior of the contact layer 10 may refer to the portion between the outer surface 101 and the inner surface 102.

[0068] In a specific implementation, the optical properties of the marker 200 may be different from those of the contact layer 10, so that the image acquired by the image acquisition unit 30 can distinguish the marker 200 from the contact layer 10.

[0069] In one possible implementation, each marker 200 may have a first part and a second part, the colors of which are different. It should be noted that this document does not limit the relative positional relationship between the first part and the second part. In other embodiments, each marker 200 may also have three or more colors.

[0070] Reference Figure 2 , Figure 2 This is a top view of a marker array in an embodiment of this application.

[0071] like Figure 2As shown, the center of marker 200 can be white, and the outer part can be black. It should be noted that white and black are merely examples, and this article does not limit the colors of the first part and the second part.

[0072] Reference Figure 3 , Figure 3 This is a top view of another marker array in an embodiment of this application.

[0073] like Figure 3 As shown, the marker array 20 may include a first marker 201 and a second marker 202, wherein the first marker 201 has a different color than the second marker 202. More specifically, the first marker 201 and the second marker 202 may be arranged alternately. For example, each first marker 201 may be adjacent to a second marker 202, and each second marker 202 may be adjacent to a first marker 201. It should be noted that... Figure 3 The fill pattern is only used to distinguish the first marker 201 and the second marker 202. The specific colors of the first marker 201 and the second marker 202 are not limited in this document.

[0074] As can be seen from the above, considering external objects, the solution in this application reduces the possibility of the marker and object colors overlapping by using multiple color combinations, thereby reducing the possibility that the marker or external object cannot be accurately identified and segmented, which is beneficial to improving the accuracy of perception. In addition, the multiple color combinations are also beneficial to improving the accuracy of force perception.

[0075] It should be noted that the circle in this article is merely an example of a sign shape, and this article does not limit the specific shape of the sign. The shapes of the signs in a sign array can be circles, squares, triangles, etc. Alternatively, the signs in a sign array can also be cylinders, frustums, cones, etc. Furthermore, multiple signs in a sign array can have the same or different shapes.

[0076] It should be noted that this article does not impose any restrictions on the preparation methods of the biomarkers.

[0077] In one example, the marker can be applied to the surface of the contact layer 10 by pasting or printing (such as screen printing, laser printing, or inkjet printing). This approach simplifies the manufacturing process, reduces production costs, and improves production efficiency.

[0078] In addition, markers can also be prepared by photolithography, vapor deposition, mold forming and other methods.

[0079] In one possible implementation, the marker array is detachably disposed on the outer surface of the contact layer. This detachable structure facilitates the replacement of the marker array. For example, the marker array can be replaced to maintain the visual-tactile sensor 100 if the markers wear out. Alternatively, the shape and arrangement of the markers in the array can be adjusted according to the sensing requirements of different scenarios.

[0080] Continue to refer to Figure 1 The image acquisition unit 30 in this paper is used to acquire images of the marker array in the contact layer 10 and the transparent part of the contact layer 10. The image acquisition unit 30 can acquire the external environment of the visual-touch sensor 100 through the transparent part.

[0081] The image acquisition unit 30 can be located at the bottom of the outer casing 70. Alternatively, the image acquisition unit 30 can be located in the internal cavity of the outer casing 70. The optical axis of the image acquisition unit 30 is approximately perpendicular to the plane containing the marker array.

[0082] In one possible implementation, the image acquisition unit may include a single camera with a telephoto mode and a close-up mode. The telephoto mode is used to capture images of external objects, and the close-up mode is used to capture images of an array of markers. For example, the camera may be an auto-zoom camera, capable of focusing on both a nearby array of markers and distant external objects.

[0083] In another possible implementation, the image acquisition unit may include a first camera and a second camera. The first camera is used to acquire images of the transparent portion, and the second camera is used to acquire images of the marker array. Specifically, the first camera can clearly focus on distant external objects through the transparent portion, while the second camera can focus on the nearby marker array.

[0084] For example, the first camera can be a standard camera or a wide-angle camera, with its focus located outside the contact layer 10, dedicated to acquiring images of external objects or the external environment. The second camera can be a macro camera, with its focus located on the plane where the marker array is located, dedicated to acquiring images of the marker array.

[0085] In the above implementation, two or more cameras can be used to observe the marker array and the external environment from different angles to obtain depth information of external objects and to perform three-dimensional tracking of the markers.

[0086] In a specific implementation, the image acquisition unit may also include a micro projector, which can project structured light through the contact layer 10 onto the surface of an external object to assist in three-dimensional visual perception.

[0087] In specific implementations, the camera described herein may include an image sensor and a lens. The image sensor may be a CMOS or CCD sensor, but is not limited to these.

[0088] For example, the image acquisition unit described herein may include one or more of the following: an event-based camera, a high dynamic range camera, a multispectral camera, and a hyperspectral camera, but is not limited thereto. Event cameras have the advantage of low power consumption; furthermore, they only output data when pixel brightness changes, enabling them to better capture rapid displacement or deformation of landmarks and rapid changes in the external environment. High dynamic range cameras provide better image quality when external lighting conditions are complex and variable (e.g., the simultaneous presence of very bright and very dark areas). Multispectral or hyperspectral cameras can provide more refined color information.

[0089] Furthermore, the visual tactile sensor 100 may also include a support layer 50, which is used to increase the distance between the contact layer 10 and the image acquisition unit 30, so that the distance between the contact layer 10 and the image acquisition unit 30 is adapted to the focal length of the image sensor 30, thereby ensuring image quality.

[0090] In practice, the support layer 50 can be transparent. For example, the support layer 50 can be a highly transparent acrylic or optical-grade silicon wafer, or a polyethylene terephthalate film, flexible printed circuit board, polyimide film, etc., but is not limited to these.

[0091] Furthermore, such as Figure 1 As shown, the visual-tactile sensor 100 may further include a light source 60, which can be used to create a uniform optical environment for the visual sensor 100 to improve the image quality acquired by the image acquisition unit 30. For example, the light source 60 may be an LED light or an infrared light source. It should be noted that the light source 60 is optional.

[0092] Specifically, the light source 60 provides illumination for image acquisition performed by the image acquisition unit 30. For example, the light source 60 can be used to illuminate the marker array, or to illuminate external objects through the contact layer 10.

[0093] In a specific implementation, the light source 60 can be located at the bottom of the housing 70. Alternatively, the light source 60 can also be located on the inner wall of the housing 70.

[0094] It should be noted that this article does not limit the number of light sources 60. For example, there can be multiple light sources 60, and multiple light sources 60 can be arranged around the image sensor 30.

[0095] The illumination method of the light source 60 is described below as an example.

[0096] For example, the light source 60 can provide illumination using polarized illumination. Specifically, a polarized light source can be used in conjunction with a polarizer in front of the camera. This method can effectively eliminate glare and stray reflections that may occur on and inside the contact layer surface, thus improving image quality.

[0097] For example, light source 60 can provide structured light. For instance, light source 60 can integrate structured light projection (such as dot matrix, stripes, etc.) to assist in acquiring three-dimensional contour information of external objects.

[0098] For example, the light source 60 can provide illumination using a time-division multiplexing method. Specifically, the light source 60 has an external mode and an internal mode. The external mode 60 is used to provide illumination when the image acquisition unit 30 acquires images of the external environment, enabling the external objects to have a better imaging effect in the image. The internal mode is used to provide illumination when the image acquisition unit 30 acquires images of the marker array, enabling the markers to have a better imaging effect in the image. For example, the internal mode can be to integrate light in a specific direction, while the external mode can be to provide uniform illumination in all directions. Also for example, the color of the light from the light source 60 can be different in the internal mode and the external mode.

[0099] For example, the light source 60 can provide illumination using waveguide illumination. Specifically, the light source 60 can be coupled to the edge of the contact layer or a specific inner layer, using the principle of total internal reflection to conduct light within the contact layer to uniformly illuminate the marker while minimizing interference with the external visual path.

[0100] For example, the light source 60 can provide illumination in an adaptive manner. Specifically, the light source 60 can dynamically adjust the brightness and / or direction of the illumination based on the intensity of external ambient light or current task requirements.

[0101] Furthermore, the visual-tactile sensor 100 may also include a data interface ( Figure 1 (Unspecified), the visual-tactile sensor 100 can be connected to an external processor via a data interface. This data interface can be a wired communication interface or a wireless communication interface; this document does not limit the type of data interface.

[0102] In one embodiment of this application, the image sensor 30 can transmit the acquired image to a processor 80 outside the visual-touch sensor 100 for processing.

[0103] In another embodiment of this application, the visual-tactile sensor 100 may include a processor. The processor can process or analyze the images acquired by the image acquisition unit 30 to obtain external force characterization parameters and / or external environmental information, etc. Alternatively, the processor in the visual-tactile sensor 100 can partially process the images acquired by the image acquisition unit 30 to obtain intermediate results, and then send the intermediate results to a processor outside the visual-tactile sensor 100. The processor outside the visual-tactile sensor 100 can then obtain external force characterization parameters and / or external environmental information, etc., based on the intermediate results.

[0104] It should be noted that the processor mentioned in this article can refer to any device or equipment with data processing capabilities. More specifically, a processor can be a microcontroller unit (MCU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a personal computer (PC), etc.

[0105] Therefore, the present application provides a visual-tactile sensor whose contact layer has dual functions of perspective and force perception, integrating external visual capabilities with precise multi-dimensional force perception capabilities, thus solving the shortcomings of existing technologies in terms of system integration, interactive intelligence, real-time perception, and cost-effectiveness.

[0106] Furthermore, this application embodiment also provides a robotic hand, including: a gripping part, a visual-tactile sensor, and a robotic hand controller. The gripping part can be used to grasp external objects, and the visual-tactile sensor can be disposed on the gripping part to perform visual perception before grasping the object in order to determine a grasping strategy, and to perform tactile perception in real time during the robotic hand's grasping of the object. For example, the gripping part can be the finger portion of the robotic hand, and the visual-tactile sensor can be conformally attached to the fingertip.

[0107] Furthermore, the robotic arm controller is connected to both the gripper and the visual-tactile sensor. The controller can control the gripper based on the output of the visual-tactile sensor. Specifically, the robotic arm controller can be directly or indirectly connected to the visual-tactile sensor. More specifically, the controller can be directly or indirectly connected to a processor to obtain the processor's processing results. For example, the processing results may be information such as control parameters or external force parameters, as described below. The processor can be an internal processor of the visual-tactile sensor or an external processor.

[0108] Reference Figure 4 , Figure 4This is a flowchart illustrating a sensing method in an embodiment of this application. Figure 4 The method shown can be applied to processors. Among them, Figure 4 The method illustrated can be executed by a single processor, or by multiple processors working together. For example, different processors can execute different steps of the perception method described herein, or different processors can execute different sub-steps within the same step of the perception method described herein. The multiple processors may include, but are not limited to, sensors both inside and outside the visual-tactile sensor.

[0109] Figure 4 The methods shown may include S41, S42, and S43.

[0110] S41, based on the first acquired image, perform pre-contact vision perception.

[0111] S42, switch modes.

[0112] S43, based on the third acquired image, perform post-contact tactile perception.

[0113] For ease of description, the working mode of pre-contact visual perception will be referred to as the first mode, and the mode of tactile perception after contact will be referred to as the second mode.

[0114] In S41, pre-contact visual perception based on the first acquired image means that the image acquisition unit acquires an image of the external object through the transparent part of the contact layer to obtain the first acquired image, and the processor performs object perception and grasping decision analysis based on the first acquired image.

[0115] Reference Figure 5 , Figure 5 This is a schematic diagram of a pre-contact visual perception scenario in an embodiment of this application.

[0116] like Figure 5 As shown, light reflected or emitted by the external object 300 passes through the air medium, enters and passes through the contact layer 10 of the sensor, and is received by the image acquisition unit 30, forming an image of the external object 300 on its photosensitive element, thereby acquiring the first acquired image.

[0117] Furthermore, the first acquired image acquired by the image acquisition unit is transmitted to the processor 80. The processor 80 can analyze the first acquired image to identify the attribute information of the external object 300. The attribute information can be used for tasks such as positioning, path planning, and pre-adjustment of grasping posture of the robot arm.

[0118] For details regarding the S41, please refer to the following section. Figure 6 Related descriptions.

[0119] Continue to refer to Figure 4 After S41 and before S42, it can be determined whether to switch from the first mode to the second mode. Determining whether to switch from the first mode to the second mode can refer to determining whether the contact layer is in contact with an external object.

[0120] In one possible implementation, a distance sensor can also be provided on the gripper of the robotic arm. The distance sensor can be used to detect the distance between the gripper and an external object. If the distance detected by the distance sensor is less than a preset distance threshold, it can be determined that the contact layer is in contact with the external object, or that a mode switch needs to be performed to switch from a first mode to a second mode.

[0121] Specifically, before switching, the shooting mode of the image acquisition unit can be switched, or the data source can be switched. For example, the first acquired image can be captured by an auto-zoom camera in telephoto mode. After determining that a mode switch is needed, the auto-zoom camera can be triggered to switch from telephoto mode to close-up mode to acquire the image, thus obtaining the third acquired image. As another example, the first acquired image can be captured by a first camera. After determining that a mode switch is needed, the second camera can be activated to acquire the image, thus obtaining the third acquired image.

[0122] In another possible implementation, specifically, during the process of the robotic arm moving towards and grasping an object, images can be continuously acquired from the image acquisition unit of the visual-tactile sensor. Image detection algorithms can be used to detect the pose of the marker array, and when a slight change in pose is detected, a mode switch is determined.

[0123] Specifically, a second acquired image can be obtained from the image acquisition unit. Further, the image of the marker array can be identified from the second acquired image. Further, the first current pose of each marker in the marker array can be determined based on the influence of the marker array. Further, based on the first current pose and reference pose of each marker, it can be determined whether an external object is in contact with the contact layer. For example, if the difference between the first current pose and the reference pose is greater than or equal to a preset threshold, it can be determined that the external object is in contact with the contact layer. That is, it is determined that a mode switch needs to be performed to switch from the first mode to the second mode.

[0124] In one example, the second acquired image can be captured by an auto-zoom camera, which, when a mode switch is determined, can be triggered to switch the camera's focus to the plane containing the marker array. Furthermore, a third acquired image can be obtained from the camera, and tactile perception after contact can be performed based on the third acquired image.

[0125] In another example, the second acquired image can be captured by a second camera. Specifically, as the robotic arm approaches the object, the second camera can be activated, and a second acquired image can be obtained from the second camera. Furthermore, when a mode switch is determined, a third acquired image continues to be acquired from the second camera, and tactile perception after contact is performed based on the third acquired image.

[0126] In another possible implementation, the robot task planner can issue an "execute contact" command to trigger a mode switch.

[0127] In one embodiment of this application, S42 further includes: sending a mode switching command to the visual-tactile sensor, the mode switching command being used to control the light source in the visual-tactile sensor to switch from an external mode to an internal mode.

[0128] In S43, tactile perception based on the third acquired image means that after the robot arm comes into contact with the object, the image acquisition unit focuses on the marker array to obtain the displacement and / or deformation of the markers in the marker array, and analyzes the information of the external force applied by the object based on the displacement and / or deformation of the markers.

[0129] Combine Figure 5 and Figure 9 , Figure 9 This is a schematic diagram of a tactile perception scenario after contact, as described in an embodiment of this application. Figure 5 The diagram shows the contact layer 10 in its natural relaxed state when it is not in contact with any object. At this time, the marker array exhibits its initial, regular distribution pattern. Based on the image captured by the image acquisition unit at this moment, the reference pose of each marker can be obtained. For example... Figure 9 As shown, when the contact layer 10 comes into contact with 300 and is subjected to an external force, the contact layer 10 undergoes elastic deformation. This deformation causes the marker to shift, altering its original distribution pattern. Therefore, after the sensor and the object come into contact, the current pose information of the marker is extracted from the third acquired image by the image acquisition unit. Combined with the reference pose in the non-contact state, the external force information exerted by the object on the sensor can be calculated.

[0130] For details regarding S43, please refer to the following section. Figure 7 Related descriptions.

[0131] From the above, Figure 4The method shown connects pre-contact visual perception and post-contact tactile perception in series to achieve bimodal perception in time, thereby realizing a complete closed loop from pre-grab decision-making to in-contact control.

[0132] In practice, before executing S41, the visual and tactile sensors need to be initialized and calibrated to provide an accurate model and benchmark for subsequent perception and control.

[0133] On the one hand, the image acquisition unit can be calibrated.

[0134] As mentioned above, the image acquisition unit may include a single camera with a telephoto mode and a close-focus mode. The camera intrinsic parameters are calibrated in the telephoto and close-focus modes respectively, and the hand-eye calibration of the robotic arm is completed.

[0135] Alternatively, if the image acquisition unit includes a first camera and a second camera, then the intrinsic parameters of the first camera and the second camera can be calibrated separately, and the transformation relationship between the two and the coordinate system of the robotic arm end effector can be established respectively.

[0136] On the other hand, the reference pose of each marker in the marker array is calibrated. The pose in this paper can include position and / or orientation. Specifically, with the contact layer unloaded, the image acquisition unit takes a picture of the marker array to obtain a reference image, and then obtains the pose of each marker based on the reference image, and uses the obtained pose as the reference pose.

[0137] On the other hand, the required models are obtained. For example, the required models may include at least one of the following: a first network model, a second network model, a crawling strategy generation model, a third network model, a fourth network model, and a force calculation model.

[0138] Optionally, after acquiring the image, the image may be preprocessed. Preprocessing may include at least one of the following: noise reduction, distortion correction, and contrast enhancement, etc. The specific content of the preprocessing in the embodiments of this application is not limited.

[0139] Reference Figure 6 , Figure 6 This is a flowchart illustrating a control method for a robotic arm according to an embodiment of this application. Figure 6 The method shown can be applied to processors. Figure 6 The methods shown may include S61, S62, S63 and S64.

[0140] S61, acquire the first acquired image from the image acquisition unit;

[0141] S62, Identify the image of an external object from the first acquired image;

[0142] S63, Determine the attribute information of the external object based on its image;

[0143] S64, Based on the attribute information of the external object, determine the gripping strategy of the robot arm for the external object.

[0144] Specifically, the first acquired image can be obtained by the image acquisition unit capturing an image of the external object through the transparent portion of the contact layer. This first acquired image is the image captured by the image acquisition unit before the robotic arm moves towards the object. More specifically, the first acquired image can be an image captured in telephoto mode. Alternatively, the first acquired image can be an image captured by the first camera mentioned above. For example, while the first camera is acquiring an image, the second camera can be in standby mode, but this is not a limitation. For instance, the first and second cameras can also simultaneously acquire images and calculate the depth information of the external object based on the acquired images.

[0145] In S42, the image of an external object is identified from the first acquired image.

[0146] Specifically, a first network model can be used to identify external objects from the first acquired image and segment the images of the external objects from the first acquired image. This paper does not limit the type of the first network model; for example, it can be an object detection model (such as YOLO, region-based convolutional neural network model, etc.) or an instance segmentation model (such as masked region convolutional neural network, arbitrary segmentation model, etc.).

[0147] It should be noted that this paper does not limit the type or acquisition method of the first network model. It can be any existing model with object recognition or image segmentation functions, and can be trained using existing model training methods (such as gradient descent).

[0148] In step S43, the attribute information of the external object is determined based on its image. The attribute information may include at least one of the following: color, pose, category, size, shape, fragility, surface texture features, and stiffness estimate.

[0149] Specifically, a second network model can be used to extract image features from the image of an external object, and the attribute information of the external object can be obtained based on the image features. This paper does not restrict the type and acquisition method of the second network model; it can be any existing model with feature extraction function, and can be trained using existing model training methods (such as gradient descent).

[0150] In one example, the attribute information of the external object can be determined based on the image and depth information of the external object obtained in S42. As mentioned above, the depth information can be obtained based on images captured by the first and second cameras, or it can be obtained from a time-of-flight camera; alternatively, the depth information can be estimated based on the first captured image from a single camera.

[0151] In step S44, the gripping strategy of the robotic arm is determined based on the attribute information of the external object. The gripping strategy may include at least one of the following: the gripping position of the object, the path the robotic arm takes towards the object, the gripping posture of the robotic arm, and the initial force applied by the robotic arm. The gripping position can refer to the contact position between the robotic arm and the object when gripping it; the gripping posture can be the posture of the robotic arm when it finally grips the object.

[0152] In one example, the attribute information of an external object can be input into the grasping strategy generation model to obtain the grasping strategy output by the grasping strategy generation model.

[0153] As a variation, the attribute information of an external object can be input into the grasping strategy generation model to obtain the object's grasping position and the robot's grasping posture, which are output by the grasping strategy generation model. Furthermore, based on optimization criteria (such as grasping stability, task requirements, etc.), the path for the robot to approach the object and the initial force for the robot's grasp can be determined.

[0154] It should be noted that this paper does not limit the type or acquisition method of the crawling strategy generation model. It can be any existing model with reasoning ability (such as GraspNet), and it can be trained using existing model training methods (such as gradient descent).

[0155] Furthermore, control parameters can be generated based on the grasping strategy, and the robotic arm can be controlled based on these parameters to move towards the object along the determined path and grasp the object according to the determined grasping posture and position. During the control of the robotic arm, the image acquisition unit can continuously monitor the process.

[0156] From the above, through Figure 5 The proposed solution enables preliminary object recognition, pose estimation, and grasping point planning based on the sensor's own visual input, improving the efficiency and success rate of subsequent interactions. For example, it can predict whether an object is fragile and thus preset an appropriate contact force; or it can classify and grasp objects based on their color. This leads to a richer understanding of the interactive object and significantly enhances pre-contact environmental perception and interaction decision-making capabilities.

[0157] Reference Figure 7 , Figure 7This is a flowchart illustrating a tactile sensing method according to an embodiment of this application. Figure 7 The methods shown may include S71, S72, S73 and S74.

[0158] S71, acquire the third acquired image from the image acquisition unit;

[0159] S72, extract the second current pose of each marker in the marker array from the third acquired image;

[0160] S73, determine the displacement and / or deformation of each marker based on the second current pose and reference pose of each marker in the marker array;

[0161] S74, determine the characterization parameters of the external force based on the displacement and / or deformation of each marker in the marker array.

[0162] It should be noted again that the first acquired image in this paper was acquired before the grasping strategy was obtained; the second acquired image was acquired during the process of controlling the robot arm according to the grasping strategy and before the object came into contact with the sensor; and the third acquired image was acquired after the object came into contact with the sensor.

[0163] Specifically, in S72, the third acquired image can be used to identify and segment markers, thereby segmenting the images of each marker in the marker array. For example, a third network model can be used to identify each marker from the third acquired image and segment the images of each marker from the third acquired image.

[0164] Furthermore, the images of each marker can be input together into the pre-trained fourth network model to obtain the second current pose of each marker output by the fourth network model.

[0165] It should be noted that this paper does not limit the type or acquisition method of the third and fourth network models. The third network model can be any existing model with object recognition or image segmentation functions, and the fourth network model can be any model with specific feature extraction capabilities. The third and fourth network models can be trained using existing model training methods (such as gradient descent).

[0166] In S73, the displacement and / or deformation of each marker is determined based on the second current pose and reference pose of each marker in the marker array.

[0167] In S74, the displacement and / or deformation of multiple markers can be input into a pre-obtained force calculation model to obtain the characterization parameters of the external forces output by the force calculation model. These characterization parameters may include at least one of the following: magnitude, direction, and type (e.g., normal force, shear force, torque, etc.).

[0168] For example, based on the displacement of each marker, a displacement field can be obtained, and the displacement field can be decoded into real-time external force representation parameters through a pre-trained force calculation model.

[0169] The force calculation model can be a model establishing the relationship between elastic body deformation and applied external forces through finite element simulation, and then obtaining the perceived parameters of the external forces through inverse finite element analysis in S74. Alternatively, the force calculation model can be a machine learning model, such as a convolutional network model, Transformer, self-supervised learning model, etc., but is not limited to these. Alternatively, the force calculation model can be a model establishing the relationship between elastic body deformation and applied external forces by combining elasticity theory and experimental data.

[0170] As a variation, in S74, the sensing parameters of external forces can also be directly analyzed based on the optical flow field.

[0171] As another variation, if other types of sensing components (such as piezoresistive, capacitive micro-unit arrays, etc.) are integrated into the contact layer or the visual-tactile sensor, the information collected by these sensing components can be fused with the displacement and / or deformation of the marker obtained through visual perception to improve the accuracy of force sensing.

[0172] As can be seen from the above, in Figure 7 In the illustrated scheme, an array of markers is placed in the contact layer. When an object applies an external force to the contact layer, the markers undergo displacement and / or deformation. The displacement and / or deformation of the markers are captured by an image acquisition unit, and combined with a preset algorithm model, the property information of the external force can be calculated.

[0173] Furthermore, the property information of the external force can be input into the robotic arm controller to achieve precise and compliant adjustment of the gripping force. The robotic arm controller can be an impedance controller or a force / position hybrid controller, but is not limited to these.

[0174] As described above, the solution provided in this application employs a transparent elastomer structure and a marker array to organically combine the sensor's external visual perception capability with its precise contact force perception capability. Before an external object comes into contact with the visual-tactile sensor, the sensor directly observes the external environment to acquire visual information about the object to aid decision-making. Furthermore, it accurately perceives multidimensional contact forces during the contact process, which is beneficial for promoting technological development in areas such as embodied intelligence, dexterous robot operation, and safe human-computer interaction.

[0175] It is understandable that, in specific implementation, the above method can be implemented using software programs that run in the processor integrated inside the chip or chip module; or, the method can be implemented using hardware or a combination of hardware and software, such as using a dedicated chip or chip module, or using a dedicated chip or chip module combined with a software program.

[0176] It should be noted that the visual-tactile sensors and their control and sensing methods provided in this paper can be applied to various scenarios such as dynamic grasping or static pressure detection or static friction detection in industrial robots, flexible electronic skin, robot tactile feedback, flexible wearable sensing, medical rehabilitation, and medical palpation, but are not limited to these applications.

[0177] It should also be noted that the various embodiments, possible implementations, possible methods of implementation, various examples, and solutions in various situations provided in this article can be used individually or in combination to achieve different technical effects.

[0178] Reference Figure 8 , Figure 8 This is a schematic diagram of the structure of a control device for a robotic arm according to an embodiment of this application. Figure 8 As shown, the control device may include:

[0179] Acquisition module 81 is used to acquire a first acquired image from the image acquisition unit;

[0180] The recognition module 82 is used to recognize the image of an external object from the first acquired image;

[0181] The attribute recognition module 83 is used to determine the attribute information of the external object based on the image of the external object;

[0182] The strategy generation module 84 is used to determine the gripping strategy of the robotic arm for the external object based on the attribute information of the external object.

[0183] For more information on the working principle, working method, and beneficial effects of the control device for the robotic arm in this embodiment, please refer to the relevant description of the method above, which will not be repeated here.

[0184] It should be understood that in the embodiments of this application, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0185] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0186] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.

[0187] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or in a combination of hardware and software functional units. For example, for various devices or products applied to or integrated into a chip, each module / unit can be implemented using hardware such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware such as circuits; for various devices or products applied to or integrated into a chip module, each module / unit can be implemented using hardware such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware such as circuits. The components can be implemented using software programs that run on the processor integrated within the chip module. The remaining (if any) modules / units can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into the terminal, each of its components / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or in different components within the terminal. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated within the terminal, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits.

[0189] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0190] In this application's embodiments, "multiple" refers to two or more. The descriptions of "first," "second," etc., appearing in this application's embodiments are merely illustrative and for distinguishing the described objects; they have no order and do not indicate a specific limitation on the number of devices in this application's embodiments, nor do they constitute any limitation on the embodiments of this application.

[0191] While this application discloses the above information, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application shall be determined by the scope defined in the claims.

Claims

1. A visual-tactile sensor, characterized in that, include: A contact layer, at least a portion of which is transparent; A marker array in which at least a portion of a plurality of markers in the marker array undergoes displacement and / or deformation when the contact layer is subjected to an external force; An image acquisition unit is used to acquire images of the marker array and to acquire images of the outside through the transparent portion of the contact layer.

2. The visual-tactile sensor according to claim 1, characterized in that, The marker array includes a first marker and a second marker, wherein the first marker and the second marker have different colors; Alternatively, the marker may have a first part and a second part, wherein the first part and the second part are different colors.

3. The visual-tactile sensor according to claim 1, characterized in that, The image acquisition unit includes an autofocus camera; Alternatively, the image acquisition unit may include a first camera and a second camera, wherein the first camera is used to acquire images of the transparent portion and the second camera is used to acquire images of the marker array.

4. The visual-tactile sensor according to claim 1, characterized in that, Also includes: A light source is used to provide illumination for image acquisition by the image acquisition unit; The light source has an external mode and an internal mode. The external mode is used to provide illumination when the image acquisition unit acquires images of the external environment, and the internal mode is used to provide illumination when the image acquisition unit acquires images of the marker array.

5. The visual-tactile sensor according to claim 1, characterized in that, The marker array is detachably disposed on the outer surface of the contact layer; Alternatively, the visual-tactile sensor may also include a housing with an opening, to which the contact layer is detachably fixed.

6. The visual-tactile sensor according to claim 1, characterized in that, The marker array is a textured structure formed on the outer or inner surface of the contact layer.

7. The visual-tactile sensor according to claim 1, characterized in that, Also includes: The processor is used to analyze the images acquired by the image acquisition unit to obtain the characterization parameters of the external force and / or the external environmental information.

8. A robotic arm, characterized in that, include: Holding part; The visual-tactile sensor according to any one of claims 1 to 7 is disposed on the grip portion; A robotic arm controller is connected to the gripping part and the visual-tactile sensor. The robotic arm controller controls the gripping part based on the output of the visual-tactile sensor.

9. A control method for a robotic arm, characterized in that, The robotic arm includes the visual-tactile sensor according to any one of claims 1 to 7, and the method includes: The first acquired image is obtained from the image acquisition unit; Identify images of external objects from the first acquired image; Based on the image of the external object, determine the attribute information of the external object; Based on the attribute information of the external object, the robotic arm determines the grasping strategy for the external object.

10. The control method according to claim 9, characterized in that, The attribute information includes at least one of the following: color, pose, category, surface texture features, and stiffness estimate.

11. The control method according to claim 9, characterized in that, Based on the attribute information of the external object, the robotic arm determines a grasping strategy for the external object, including: The attribute information is input into the crawling strategy generation model to obtain the crawling strategy output by the crawling strategy generation model; The grasping strategy includes at least one of the following: the grasping position of the object, the path of the robotic arm approaching the object, the grasping posture of the robotic arm, and the initial force of the robotic arm grasping.

12. The control method according to claim 9, characterized in that, The method further includes: The second acquired image is obtained from the image acquisition unit; Identify the image of the marker array from the second acquired image; Based on the image of the marker array, determine the first current pose of each marker in the marker array; Based on the first current pose and reference pose of each marker, it is determined whether the external object is in contact with the contact layer; In response to the contact between the object and the contact layer, characterization parameters of the external force applied by the external object are determined.

13. The control method according to claim 12, characterized in that, Determining the characterization parameters of the external force exerted by the external object includes: A third acquired image is obtained from the image acquisition unit; Extract the second current pose of each marker in the marker array from the third acquired image; The displacement and / or deformation of each marker are determined based on the second current pose of each marker in the marker array and the reference pose. The characterization parameters of the external force are determined based on the displacement and / or deformation of each marker in the marker array.

14. The control method according to claim 13, characterized in that, Before acquiring the third captured image, the method further includes: In response to the contact between the external object and the contact layer, a mode switching command is sent, which is used to control the light source to switch from an external mode to an internal mode.

Images