Object shape and texture perception method and corresponding apparatus for underwater robots

Abstract

The present disclosure provides an object shape and texture perception method for underwater robots and a corresponding device. The underwater tactile sensor comprises: a contact module for contacting the surface of an underwater object to deform, the deformation causing displacement of a multi-color marker point field on the upper surface of the contact module; a binocular camera module for capturing the displacement of the multi-color marker point field to obtain a tactile image, wherein the tactile image comprises a left-eye tactile image and a right-eye tactile image; a waterproof module for realizing the waterproof function of the underwater tactile sensor; wherein the multi-color marker point field comprises: a plurality of marker point groups arranged in sequence, each marker point group comprising j marker points of different colors, the spatial order of the j color marker points in each marker point group being fixed, and j being an integer greater than 1.

Description

Technical Field

[0001] This disclosure generally relates to the field of object perception technology, and more specifically, to a method and apparatus for perceiving the shape and texture of objects for underwater robots. Background Technology

[0002] Tactile perception is an important form of near-field sensing in living organisms, possessing unique advantages. Tactile perception acquires information about the shape and texture of objects through direct contact, and also captures the force exerted during contact, guiding limbs or fingers to perform complex maneuvers. Furthermore, tactile perception can assist other sensory methods in completing perceptual tasks: it serves as an alternative when visual perception fails, as in the analogy of the blind men and the elephant. Tactile perception has significant applications underwater: in turbid underwater environments, underwater tactile sensors can replace underwater cameras to perform sensing tasks; in sonar dead zones (near-field areas undetectable by sonar), underwater tactile sensors can provide high-resolution information about the shape and texture of objects.

[0003] With the continuous development of new materials and sensing technologies, artificial tactile technology has begun to attract the attention of researchers. As a supplement and extension to vision, tactile sensors can be integrated into the end effectors of robotic arms, inherently possessing the potential to guide dexterous operations. In recent years, a series of underwater tactile sensors suitable for underwater operations have emerged. Based on their functions, underwater tactile sensors can be divided into force sensors that sense mechanical physical quantities such as pressure, and biomimetic tactile sensors, also called biomimetic skin, that sense other physical quantities such as texture, temperature, and vibration. Underwater force sensors can be classified according to their principles into piezoresistive, capacitive, photoelectric, electromagnetic, ultrasonic, and resistance strain gauge types. These sensors convert the contact deformation of a soft surface into electrical signals, obtaining tactile sensation through signal processing. However, these underwater tactile sensing technologies face some functional challenges and perform poorly in actual underwater environments. Summary of the Invention

[0004] An exemplary embodiment of this disclosure provides a method and apparatus for perceiving the shape and texture of underwater objects, which can efficiently and accurately perceive the shape and texture of underwater objects based on vision.

[0005] According to a first aspect of the present disclosure, an underwater tactile sensor is provided, comprising: a contact module for contacting the surface of an underwater object to cause deformation, the deformation causing displacement of a multi-color marker dot field on the upper surface of the contact module; a binocular camera module for capturing the displacement of the multi-color marker dot field to obtain a tactile image, wherein the tactile image includes a left-eye tactile image and a right-eye tactile image; and a waterproof module for implementing the waterproof function of the underwater tactile sensor; wherein the multi-color marker dot field comprises: a plurality of marker dot groups arranged sequentially, each marker dot group comprising j marker dots of different colors, the spatial order of the j-color marker dots in each marker dot group being fixed, and j being an integer greater than 1.

[0006] Optionally, it further includes: a pressure balancing chamber, a cavity for contacting the lower surface of the contact module with external water, so as to achieve water pressure balance between the upper and lower surfaces of the contact module.

[0007] Optionally, it also includes: a light chamber module for providing illumination conditions for the binocular camera module to capture the displacement of the multi-color marker field.

[0008] According to a second aspect of the present disclosure, an underwater tactile sensing method is provided, comprising: acquiring multiple frames of tactile images from an underwater tactile sensor as described above; taking each frame of tactile image as a current frame and generating a marker point coordinate matrix corresponding to the current frame, wherein each element in the marker point coordinate matrix corresponds one-to-one with each marker point in the multi-color marker point field, and the content of each element is the pixel coordinate of the marker point corresponding to the element in the current frame; determining the three-dimensional spatial coordinates of each marker point in the multi-color marker point field based on the marker point coordinate matrix corresponding to the current frame, so as to obtain a three-dimensional tactile point cloud corresponding to the current frame; wherein the marker point coordinate matrix corresponding to the current frame includes: a marker point coordinate matrix corresponding to the left eye tactile image and a marker point coordinate matrix corresponding to the right eye tactile image in the current frame.

[0009] Optionally, the step of generating the marker point coordinate matrix corresponding to the current frame includes: dividing the current frame into j monochrome channel images that correspond one-to-one with j color channels; determining the marker point coordinate matrix to be repaired for each monochrome channel image of the current frame based on the marker point coordinate matrix corresponding to the previous frame of the current frame; merging the marker point coordinate matrices to be repaired for each monochrome channel image to obtain the marker point coordinate matrix to be repaired for the current frame; and repairing the marker point coordinate matrix to be repaired for the current frame to obtain the marker point coordinate matrix corresponding to the current frame.

[0010] Optionally, the step of determining the marker point coordinate matrix to be repaired corresponding to each monochrome channel image of the current frame based on the marker point coordinate matrix corresponding to the previous frame includes: determining the marker point coordinate matrix corresponding to each monochrome channel image of the previous frame based on the marker point coordinate matrix corresponding to the previous frame; obtaining the pixel coordinates of the marker points detected from the monochrome channel image of the Mth color of the current frame using a blob detection algorithm; and obtaining the marker point coordinate matrix to be repaired corresponding to the monochrome channel image of the Mth color of the current frame based on the matching result between the detected pixel coordinates of the marker points and the marker point coordinate matrix corresponding to the monochrome channel image of the Mth color of the previous frame; wherein M is an integer greater than 0 and less than or equal to j.

[0011] Optionally, the step of repairing the marker point coordinate matrix corresponding to the current frame to obtain the marker point coordinate matrix corresponding to the current frame includes: for missing marker points in the marker point coordinate matrix corresponding to the current frame, determining the potential pixel coordinates of the missing marker point based on the pixel coordinates of other marker points in the marker point group to which the missing marker point belongs; searching for candidate marker points for the missing marker point in the current frame based on the potential pixel coordinates; selecting pixel coordinates as the pixel coordinates of the missing marker point from the pixel coordinates of the candidate marker points and the potential pixel coordinates; and adding the pixel coordinates of the missing marker point to the marker point coordinate matrix corresponding to the current frame.

[0012] Optionally, it further includes: based on the three-dimensional tactile point cloud corresponding to the multi-frame tactile images, selecting a set of tactile images corresponding to the complete contact process between the contact module and the underwater object in this instance from the multi-frame tactile images; selecting the key frame with the most significant deformation of the corresponding three-dimensional tactile point cloud from the set of tactile images; and integrating the three-dimensional tactile point cloud corresponding to the key frame into the tactile map of the surface of the underwater object to obtain the shape and texture of the underwater object.

[0013] Optionally, the step of selecting the set of tactile images corresponding to the complete contact process between the contact module and the underwater object based on the three-dimensional tactile point cloud corresponding to the multi-frame tactile images includes: for each frame of tactile image, determining the intensity of the tactile stimulus received by the underwater tactile sensor when the frame of tactile image was captured, based on the three-dimensional tactile point cloud corresponding to the frame of tactile image and the three-dimensional tactile point cloud corresponding to the first frame in the multi-frame tactile images, wherein the intensity of the tactile stimulus is positively correlated with the degree of deformation of the contact module; if the intensity of the tactile stimulus corresponding to the frame of tactile image exceeds a preset threshold, then the frame of tactile image is added to the set of tactile images.

[0014] According to a third aspect of the present disclosure, an underwater robot is provided, comprising: an underwater tactile sensor as described above; a robotic arm with the underwater tactile sensor mounted at its end; and a controller.

[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores instructions which, when executed by a processor of an electronic device, enable the electronic device to perform the underwater tactile sensing method as described above.

[0016] According to a fifth aspect of the present disclosure, an electronic device is provided, the electronic device comprising: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the underwater tactile sensing method as described above.

[0017] According to a sixth aspect of the present disclosure, a computer program product is provided, including computer-executable instructions that, when executed by at least one processor, implement the underwater tactile sensing method as described above.

[0018] The object shape and texture perception method and corresponding apparatus for underwater robots according to the exemplary embodiments of the present disclosure can efficiently and accurately perceive the shape and texture of underwater objects based on vision.

[0019] In the following description, some aspects and / or advantages of the general concept of this disclosure will be set forth, and other aspects and / or advantages will become apparent from the following description or from practice of the general concept of this disclosure. Attached Figure Description

[0020] These and / or other aspects and advantages of this application will become clearer and more readily understood from the following detailed description of embodiments of this application taken in conjunction with the accompanying drawings, wherein:

[0021] Figure 1 A structural block diagram of an underwater tactile sensor according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 2 A schematic diagram of the structure of an underwater tactile sensor according to an exemplary embodiment of the present disclosure is shown;

[0023] Figure 3 A schematic diagram of the structure of a pressure balancing chamber according to an exemplary embodiment of the present disclosure is shown;

[0024] Figure 4 A schematic diagram of the structure of a marker field according to an exemplary embodiment of the present disclosure is shown;

[0025] Figure 5A flowchart illustrating an underwater tactile sensing method according to an exemplary embodiment of the present disclosure is shown;

[0026] Figure 6 A flowchart illustrating a method for generating a marker point coordinate matrix corresponding to the current frame according to an exemplary embodiment of the present disclosure;

[0027] Figure 7 A flowchart illustrating a method for determining the coordinate matrix of marker points to be repaired corresponding to each monochrome channel image of the current frame, according to an exemplary embodiment of the present disclosure;

[0028] Figure 8 A flowchart illustrating a method for repairing the coordinate matrix of marker points to be repaired corresponding to the current frame according to an exemplary embodiment of the present disclosure;

[0029] Figure 9 A flowchart illustrating a method for updating a tactile map of an underwater object surface according to an exemplary embodiment of the present disclosure;

[0030] Figure 10 An example of an underwater tactile sensing method according to an exemplary embodiment of the present disclosure is shown;

[0031] Figure 11 A structural block diagram of an underwater robot according to an exemplary embodiment of the present disclosure is shown;

[0032] Figure 12 A schematic diagram of the structure of a teleoperated tactile sensing platform according to an exemplary embodiment of the present disclosure is shown;

[0033] Figure 13 An example of a teleoperable tactile sensing framework according to an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0034] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, examples of which are illustrated in the drawings, wherein the same reference numerals always refer to the same parts. The embodiments will now be described with reference to the accompanying drawings in order to explain this disclosure.

[0035] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0036] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0037] Existing underwater tactile sensing technologies face several functional challenges, exhibiting poor performance in real-world underwater environments. Improvements are needed in sensor manufacturing processes and cost, and commercially available underwater tactile sensors are still lacking. Specifically: underwater tactile sensing struggles to effectively eliminate the influence of water pressure, with outputs potentially changing under varying depths and pressures, and even failing to function properly in deep water; underwater force sensors based on electrical signals are susceptible to underwater impurities and temperature variations, potentially exhibiting self-heating and zero-point drift, making stable operation difficult over extended periods; underwater tactile sensing faces challenges in waterproofing and sealing, hindering sensor miniaturization and complicating manufacturing processes, resulting in high production costs; and array-based underwater waterproof skins, limited by processing technology, cannot achieve high-resolution layouts, resulting in a disadvantage in resolution.

[0038] To address the aforementioned underwater sensing problems, this disclosure proposes, on the one hand, a vision-based underwater tactile sensor. It employs a multi-color marker field structure to achieve high-resolution sensing even with low marker density. The sensor features a waterproof and pressure-balanced design, enabling operation in outdoor underwater environments up to 50 meters deep. Furthermore, the sensor's form factor is miniaturized for easy installation and operation on the end effector of an underwater robotic arm. On the other hand, this disclosure also proposes a method for generating a 3D tactile point cloud of the contacted object surface based on the tactile image output by the underwater tactile sensor. This method incorporates marker tracking and surface reconstruction algorithms, enabling the generation of a high-resolution 3D point cloud of the underwater tactile sensor's contact surface under vibration conditions, thus achieving accurate object surface sensing in such environments. Additionally, this disclosure proposes a teleoperated tactile sensing framework for sensing the shape and texture of underwater target objects. The following will combine... Figures 1 to 13 Please provide a detailed explanation.

[0039] Figure 1 A structural block diagram of an underwater tactile sensor according to an exemplary embodiment of the present disclosure is shown.

[0040] Reference Figure 1The underwater tactile sensor 100 according to an exemplary embodiment of the present disclosure includes: a contact module 101, a binocular camera module 102, and a waterproof module 103.

[0041] Specifically, the contact module 101 is used to contact the surface of an underwater object to cause deformation, which causes displacement of the multi-color marker field on the upper surface of the contact module 101.

[0042] The binocular camera module 102 is used to capture the displacement of the multi-color marked point field to obtain tactile images. The tactile images include a left-eye tactile image and a right-eye tactile image.

[0043] The basic principle of the underwater tactile sensor 100 is to convert the tactile information obtained when the sensor comes into contact with an underwater object into the deformation of the contact module 101. This deformation causes the displacement of the marker points on the upper surface of the contact module 101. The binocular camera module 102 identifies and tracks these marker points to obtain a tactile image. By performing three-dimensional reconstruction on the tactile image, three-dimensional point cloud information of the underwater object's surface can be obtained, thereby realizing the perception of the shape and texture of the underwater object.

[0044] The multi-color marker field includes: multiple groups of markers arranged in sequence, each group of markers including j markers of different colors, and the spatial order of the j markers of different colors in each group of markers is fixed, where j is an integer greater than 1.

[0045] As an exemplary embodiment, j can be 4, and the j colors can include: red, green, blue, and black. It should be understood that the value of j can be set according to the actual situation and specific needs, and the j colors are not limited to the colors mentioned above.

[0046] As an example, such as Figure 4 As shown, Figure 4 Image (a) shows a cross-sectional view of the multi-color marker field. Figure 4 (b) shows a top view of a multi-color marker field, which can be a square array of four-color markers with a diameter of 0.5 mm, and the spacing between adjacent markers can be 1.0 mm. Figure 4 (c) shows a schematic diagram of a group of markers, in which the top left marker is red, the top right marker is green, the bottom left marker is blue, and the bottom right marker is black.

[0047] like Figure 4 As shown in (d), the multi-color marker field is a marker field formed by alternating combinations of red, green, blue, and black marker fields. Since the distribution density of markers in the monochrome marker field is 2 mm × 2 mm, while the perceptual resolution of the multi-color marker field is 1 mm × 1 mm, high-resolution perception capability can be achieved under low-resolution distribution.

[0048] It should be understood that the spatial order of the marker points in the multi-color marker point field (e.g., which row and which column they are located in) is fixed, and the multi-color marker point field may be deformed, rotated, scaled, etc. as the contact module 101 deforms.

[0049] The waterproof module 103 is used to enable the underwater tactile sensor 100 to be waterproof.

[0050] As an exemplary embodiment, the underwater tactile sensor 100 according to an exemplary embodiment of the present disclosure may further include: a light chamber module 104. The light chamber module 104 is used to provide illumination conditions (e.g., uniform white illumination conditions) for the binocular camera module 102 to capture the displacement of the marker point field.

[0051] As an exemplary embodiment, the underwater tactile sensor 100 according to an exemplary embodiment of the present disclosure may further include a pressure balancing chamber 105. The pressure balancing chamber 105 is a cavity for contacting the lower surface of the contact module 101 with the external water body to achieve water pressure balance between the upper and lower surfaces of the contact module 101.

[0052] Figure 2 A schematic diagram of the structure of an underwater tactile sensor according to an exemplary embodiment of the present disclosure is shown.

[0053] Figure 2 Figure (A) shows a cross-sectional view of the underwater tactile sensor 100. Figure 2 Image (B) shows a prototype of the underwater tactile sensor 100. Figure 2 (C) shows an exploded view of the underwater tactile sensor 100.

[0054] Reference Figure 2 The contact module 101 may include a casting layer, an RGBK (red, green, blue, black) marker layer (filled with a multi-color marker field), and a gel layer mounted on top of the underwater tactile sensor 100.

[0055] The light chamber module 104 may include a quartz glass layer, a light diffusion layer, and an LED light strip mounted below the contact module 101. The light chamber module 104 is an air-filled sealed cavity designed to provide stable lighting conditions for the binocular camera module 102.

[0056] The binocular camera module 102 may include a binocular camera and a control circuit board mounted at the bottom center of the light chamber module 104.

[0057] The waterproof module 103 may include an upper cover, a waterproof housing, and a lower cover. As an example, the waterproof housing is connected to the quartz glass layer of the light chamber module 104 via a sealing ring (e.g., a rubber sealing ring), and the waterproof housing is connected to the lower cover via a sealing ring. The lower cover can transmit signals and power via a waterproof aviation connector (e.g., a five-hole waterproof aviation connector) to isolate the underwater tactile sensor 100 from external water and maintain an air-filled internal space. For example, the waterproof aviation connector can connect to an internal control board and LED light strip, providing power to the LED light strip while outputting USB signals. As an example, the waterproof module 103 may be made of aluminum alloy and can withstand 50 meters of water pressure.

[0058] Water resistance is a key factor affecting the performance of the underwater tactile sensor 100, as the binocular camera, LED light strip, and control board require a strictly water-free environment. The underwater tactile sensor 100 achieves waterproofing through an aluminum alloy housing and quartz glass. As an example, the quartz glass layer of the light chamber module 104 is a cylindrical sheet structure, 2 mm thick and 41 mm in diameter, designed to withstand water pressure from its upper surface; the waterproof housing and lower cover are made of 5 mm thick aluminum alloy to withstand water pressure from the sides and bottom of the underwater tactile sensor 100.

[0059] Figure 3 A schematic diagram of the structure of a pressure balancing chamber according to an exemplary embodiment of the present disclosure is shown.

[0060] like Figure 3 As shown, a pressure balancing cavity 105 is provided between the contact module 101 and the optical chamber module 104. As an example, the pressure balancing cavity 105 can be a cavity with a height of 0.2 mm, communicating with external water through the water inlet of the upper cover of the waterproof module 103 to achieve water pressure balance between the upper and lower surfaces of the contact module 101. For example, the upper cover of the underwater tactile sensor 100 can have four water inlets, directly connected to the pressure balancing cavity 105, to ensure that external water enters the pressure balancing cavity 105.

[0061] In the underwater tactile sensor 100, balancing water pressure is another crucial factor that cannot be ignored, as water pressure increases with operating depth. When the underwater tactile sensor 100 is at a depth of 50 meters, it will be subjected to water pressure of approximately 5 atmospheres. Such water pressure is a significant disturbance for the underwater tactile sensor 100, which relies on the deformation of the contact module 101. Therefore, this disclosure proposes that the underwater tactile sensor 100 does not completely seal the gel layer of the contact module 101, but instead designs a cavity (i.e., pressure balancing cavity 105) with a height of 0.2 mm between the gel layer and the quartz glass layer. When external water flows in and fills the pressure balancing cavity 105, the water pressure on the upper and lower surfaces of the contact module 101 is the same, thereby preventing the contact module 101 from deforming due to excessive external water pressure.

[0062] Figure 5 A flowchart illustrating an underwater tactile sensing method according to an exemplary embodiment of the present disclosure is shown.

[0063] Reference Figure 5 In step S501, multiple frames of tactile images are acquired from the underwater tactile sensor 100.

[0064] As an exemplary embodiment, the multi-frame tactile images are a sequence of tactile images obtained during a complete approach-contact-separation process of an underwater target object. Each frame of the tactile image actually includes a left-eye tactile image and a right-eye tactile image.

[0065] In step S502, each frame of tactile image is taken as the current frame, and the coordinate matrix of the marker points corresponding to the current frame is generated.

[0066] Each element in the marker coordinate matrix (hereinafter also called the marker seating table or simply the seating table) corresponds one-to-one with each marker in the multi-color marker field. The content of each element is the pixel coordinates of the marker corresponding to that element in the current frame. Specifically, the spatial order position of each element in the marker coordinate matrix (e.g., its row and column number) is the same as the spatial order position of the marker corresponding to that element in the multi-color marker field. That is, the marker seating table contains not only the pixel coordinates of the markers but also their spatial order position information.

[0067] The marker coordinate matrix corresponding to the current frame includes: the marker coordinate matrix corresponding to the left visual tactile image and the marker coordinate matrix corresponding to the right visual tactile image in the current frame.

[0068] In step S503, based on the coordinate matrix of the marker points corresponding to the current frame, the three-dimensional spatial coordinates of each marker point in the multi-color marker point field are determined to obtain the three-dimensional tactile point cloud corresponding to the current frame.

[0069] As an exemplary embodiment, step S503 may include: determining the correspondence between the marker point coordinate matrix corresponding to the left tactile image and the marker point coordinate matrix corresponding to the right tactile image regarding the same marker point; and then using triangulation to determine the three-dimensional spatial coordinates of each marker point based on the correspondence, the marker point coordinate matrix corresponding to the left tactile image, and the marker point coordinate matrix corresponding to the right tactile image, as the three-dimensional tactile point cloud corresponding to the current frame.

[0070] The underwater tactile sensing method according to an exemplary embodiment of this disclosure tracks marker points based on a multi-color marker point field, detects the positions of marker points from a series of tactile images captured by the underwater tactile sensor 100, and obtains a marker point location table. This enables accurate tactile sensing of the surface texture and shape of a target object in complex underwater environments with vibration, impact, or other disturbances. Existing underwater tactile sensing technologies are mainly conducted in relatively stable laboratory environments, where the sensor makes gentle and slow contact with the object surface to maintain good contact between the sensor and the object. These methods are not well-suited for tactile sensing under vibration and impact environments. Therefore, this disclosure proposes a stable tactile point cloud generation method that not only has high resolution but also enables accurate sensing under disturbances such as vibration.

[0071] Figure 6 A flowchart illustrating a method for generating a marker point coordinate matrix corresponding to the current frame according to an exemplary embodiment of the present disclosure is shown.

[0072] Reference Figure 6 In step S601, the current frame is divided into j monochrome channel images, each corresponding to one of the j color channels. Specifically, the monochrome channel image of the Mth color is an image that only displays the marker points of the Mth color, where M is an integer greater than 0 and less than or equal to j.

[0073] As an exemplary embodiment, the tactile image captured by the underwater tactile sensor 100 can be in RGB format, and a color segmentation algorithm based on HSV format can be used to obtain monochrome channel images corresponding to the red, green, blue, and black channels respectively. As an example, different color markers can be distinguished based on a threshold in the H channel, separating the background from each color marker.

[0074] As an example, a color segmentation algorithm based on the HSV format can be implemented using the following formula:

[0075]

[0076] Among them, F n This represents the haptic image of the nth frame, which is in RGB channel format. Represents the HSV conversion function. This represents the nth frame of the haptic image converted to HSV channels, where m and k represent the row and column numbers of the nth frame of the haptic image. This indicates that the H channel is obtained from the nth frame of the haptic image in the HSV channel. n This represents the H channel of the nth frame of the haptic image. The Mth color is one of red, green, blue, or black, and L... M U M Let represent the lower and upper threshold values ​​for the Mth color segmentation algorithm. This represents the monochrome channel image after being segmented by the Mth color.

[0077] As an example, such as Figure 4 As shown in (d), the multi-color marker field is a marker field formed by alternating combinations of four monochrome marker fields (red, green, blue, and black). Since the distribution density of markers in the monochrome marker field is 2mm × 2mm, the tactile perception range of each marker in the monochrome marker field (which can be defined as the maximum movable area where markers do not overlap) is 2.0 × 2.0mm. 2 Square area (e.g.) Figure 4 As shown by the colored squares in (e) of the diagram, by utilizing the prior structural information of the multi-color marker field (i.e., the original marker distribution information), the spatial order of the markers can be determined, thereby achieving sequential super-resolution perception. Specifically, the multi-color marker field combines the tactile perception ranges of the four colors of markers to obtain a common area of ​​1.0 × 1.0 mm. 2 ,like Figure 4 As shown in the intermediate superimposed region in (e), the perceptual resolution of the multi-color marker field is therefore 1.0 × 1.0 mm. 2 According to this disclosure, by segmenting the marker points according to four color channels, a multi-color marker point field achieves high perceptual resolution under low distribution density.

[0078] In step S602, based on the coordinate matrix of the marker points corresponding to the previous frame of the current frame, the coordinate matrix of the marker points to be repaired corresponding to each monochrome channel image of the current frame is determined.

[0079] The following will combine Figure 7 An exemplary embodiment of step S602 is described, which will not be elaborated here.

[0080] In step S603, the coordinate matrices of the marker points to be repaired corresponding to each monochrome channel image are merged to obtain the coordinate matrix of the marker points to be repaired corresponding to the current frame.

[0081] In step S604, the coordinate matrix of the marker points to be repaired corresponding to the current frame is repaired to obtain the coordinate matrix of the marker points corresponding to the current frame.

[0082] The following will combine Figure 8 An exemplary embodiment of step S604 is described, which will not be elaborated here.

[0083] It should be understood that steps S601-S604 are executed for the left visual tactile image in the current frame to obtain the coordinate matrix of the marker points corresponding to the left visual tactile image, and steps S601-S604 are executed for the right visual tactile image in the current frame to obtain the coordinate matrix of the marker points corresponding to the right visual tactile image.

[0084] Figure 7 A flowchart illustrating a method for determining the coordinate matrix of marker points to be repaired corresponding to each monochrome channel image of the current frame, according to an exemplary embodiment of the present disclosure.

[0085] Reference Figure 7 In step S701, based on the coordinate matrix of the marker points corresponding to the previous frame, the coordinate matrix of the marker points corresponding to each monochrome channel image of the previous frame is determined.

[0086] Here, each element in the coordinate matrix of the marker point corresponding to the monochrome channel image of the Mth color corresponds one-to-one with each marker point in the single-color marker point field of the Mth color, and the content of each element is the pixel coordinate of the marker point corresponding to that element in the monochrome channel image.

[0087] As an exemplary embodiment, since the first frame F1 in the acquired multi-frame tactile images does not have a previous frame, the marker point coordinate matrix (hereinafter also referred to as the predetermined marker point coordinate matrix) corresponding to the tactile image F0 captured during initialization can be used as the marker point coordinate matrix corresponding to the previous frame of the first frame.

[0088] In step S702, the pixel coordinates of the marker points detected from the monochrome channel image of the Mth color of the current frame by the blob detection algorithm are obtained.

[0089] As an exemplary embodiment, the centroid pixel coordinates of the marked point can be obtained as the pixel coordinates of the marked point using a blob detection algorithm. As an example, the blob detection algorithm can be implemented using the following formula:

[0090]

[0091] in, This represents a blob detection algorithm. This represents the detection result of the marker point of the Mth color in the nth frame of the tactile image, where l represents the number of marker points.

[0092] In step S703, based on the matching result between the detected pixel coordinates of the marker points and the marker point coordinate matrix corresponding to the monochrome channel image of the Mth color in the previous frame, the marker point coordinate matrix to be repaired corresponding to the monochrome channel image of the Mth color in the current frame is obtained.

[0093] During a complete approach-contact-separation process, the underwater tactile sensor 100 is initially not in contact with the target object, and the contact module 101 remains in a relaxed state. Therefore, the pixel coordinates of the marker points detected in the first frame are almost identical to the pre-determined marker point coordinate matrix. However, due to noise or structural changes in the underwater tactile sensor 100, the pixel coordinates of some marker points may change.

[0094] As an exemplary embodiment, a structure-based nearest neighbor matching algorithm can be used to match the pixel coordinates of the detected marker points with the marker point coordinate matrix corresponding to the pixel coordinates of the previous frame, so as to obtain the marker point coordinate matrix corresponding to the pixel coordinates of the detected marker points.

[0095] Since the spatial order of the markers detected by the blob detection algorithm is unknown, it is necessary to roughly organize the markers detected in the current frame based on the complete marker position table corresponding to the previous frame to give them spatial order. This disclosure designs a K-nearest neighbor matching algorithm based on the relative position between markers. It assumes that there is a large displacement of the marker point cloud between two consecutive tactile images, but the shape of the point cloud changes very little. Therefore, the relative position between markers in the current frame should be similar to the relative position between corresponding points in the previous frame.

[0096] As an exemplary embodiment, firstly, the centroids of the marker point clouds of the current frame's monochrome channel image and the previous frame's monochrome channel image are calculated and normalized to obtain the relative position of each marker point with respect to the centroid. Next, for each marker point (i.e., the target marker point) in the normalized marker point cloud of the previous frame's monochrome channel image, the K-nearest neighbor matching algorithm is applied to find the marker point in the normalized marker point cloud of the current frame's monochrome channel image that is closest to the target marker point as the matching point. Finally, if the matching point exceeds the tactile perception range of the target marker point, the matching point is deleted; if the matching point does not exceed the tactile perception range of the target marker point, the pixel coordinates before normalization of the matching point are added to the marker point coordinate matrix of the current frame's monochrome channel image at the position corresponding to the target marker point, thereby obtaining the marker point coordinate matrix to be repaired for the current frame's monochrome channel image. Specifically, due to physical factors and algorithm limitations, the coordinate matrix of the marker points corresponding to the monochrome channel image of the current frame may be incomplete (i.e., some locations in the matrix may lack corresponding matching points because the corresponding matching points are outside the range of tactile perception), which means it needs to be repaired.

[0097] As an example, the coordinate matrix of the marker points to be repaired corresponding to the monochrome channel image of the Mth color in the current frame can be obtained by the following formula:

[0098]

[0099] in, This represents the detection result of the marker point of the Mth color in the normalized nth frame tactile image, where M(·) represents the mean function. n This represents the number of marker points in the nth frame of the haptic image. Similarly, This represents the restoration result of the marker point of the Mth color in the normalized (n-1)th frame of the tactile image. The matrix representing the coordinates of the marker points corresponding to the monochrome channel image of the Mth color in the (n-1)th frame of the tactile image, l n-1 This represents the number of marker points in the (n-1)th frame of the tactile image. i represents the index of the corresponding matrix element. This indicates that the K-nearest neighbor matching algorithm only selects the nearest point. This represents the matching point of the Mth color in the nth frame of the tactile image. Let L2(·) represent the coordinate matrix of the marker points to be repaired corresponding to the monochrome channel image of the Mth color in the nth frame of the tactile image, where L2(·) represents the Euclidean distance function, and d represents the length of the tactile perception range. n Let represent the coordinate matrix of the marker points to be repaired corresponding to the nth frame of the tactile image, and h represent the number of marker points of all colors.

[0100] Figure 8 A flowchart illustrating a method for repairing the coordinate matrix of marker points to be repaired corresponding to the current frame according to an exemplary embodiment of the present disclosure.

[0101] Reference Figure 8 In step S801, for the missing marker points in the marker point coordinate matrix to be repaired corresponding to the current frame, the potential pixel coordinates of the missing marker points are determined based on the pixel coordinates of other marker points in the marker point group to which the missing marker points belong.

[0102] Missing markers are the markers corresponding to elements in the marker coordinate matrix that are missing content.

[0103] In step S802, candidate marker points for missing marker points are searched from the current frame based on the potential pixel coordinates.

[0104] In step S803, pixel coordinates are selected from the pixel coordinates of candidate marker points and potential pixel coordinates as the pixel coordinates of missing marker points.

[0105] In step S804, the pixel coordinates of the missing marker points are added to the marker point coordinate matrix to be repaired corresponding to the current frame to obtain a marker point coordinate matrix without missing elements (i.e., the repaired marker point coordinate matrix).

[0106] Due to physical factors and algorithmic limitations, the marker point coordinate matrix obtained in step S603 may be incomplete. On the one hand, due to occlusion or changes in lighting conditions during vibration, some marker points in the multi-color marker point field may not be identified. On the other hand, the K-nearest neighbor matching algorithm only matches the nearest marker point, and the distance between the matched point and the target marker point cannot exceed the tactile perception range of the target marker point. This makes it possible for marker points in rapidly deformed areas to be mistakenly identified as missing points because they are deleted outside the tactile perception range. Furthermore, in the matching step, the matching behavior is based on the assumption that the relative position change within the marker point cloud between adjacent frames is small. However, in tactile perception, contact with some sharp objects may cause significant deformation of the contact module 101, resulting in significant changes in the relative position of marker points in steep locations, making matching impossible. Therefore, this disclosure designs a repair algorithm to complete the missing marker points.

[0107] As an exemplary embodiment, firstly, the parallelogram structure of each marker group in the multi-color marker field can be utilized to calculate the potential pixel coordinates of the missing marker based on markers of other colors in the same marker group. For example, if other color markers in the marker group containing the missing marker are not missing in the current frame, the pixel coordinates of those other color markers in the current frame are directly used for calculation; otherwise, the pixel coordinates of those other color markers in the previous frame are used. Then, the K-nearest neighbor matching algorithm can be used to search for multiple neighboring markers of the potential pixel coordinates in the current frame. Only neighboring markers within the tactile perception range of the potential pixel coordinates are considered candidate markers, with candidate markers closer to the potential pixel coordinates having higher confidence values, thereby obtaining a set of candidate markers for the missing marker. If there are no neighboring markers within the tactile perception range of the potential pixel coordinates, the potential pixel coordinates themselves are considered the only candidate pixel coordinates for the missing marker, and a confidence value is assigned. This value can be set via hyperparameters. If a marker is lost in multiple consecutive frames, the trust value of its candidate marker set will be updated using an update formula based on the trust value of the marker in the previous frame. Finally, the pixel coordinates with the highest trust value are selected from the pixel coordinates of the candidate markers and the potential pixel coordinates as the pixel coordinates of the missing marker and added to the marker coordinate matrix to be repaired in the current frame to complete the filling of the missing marker (i.e., the missing elements in the matrix).

[0108] As an example, the repair calculation process can be implemented using the following formula:

[0109]

[0110] Among them, P miss P represents the potential pixel coordinates of the missing marker point. R ,PG ,P B ,P K P represents the pixel coordinates of the other markers (red, green, blue, and black markers) in the marker group containing the missing marker. M This represents the pixel coordinates of the marker point for the Mth color. This represents the detection results of red, green, blue, and black markers in the nth frame of the tactile image. This represents the matrix of marker coordinates corresponding to the monochrome channel image of the Mth color in the (n-1)th frame of the tactile image. This represents the coordinate matrix of the marker points to be repaired corresponding to the monochrome channel image of the Mth color of the nth frame tactile image. This represents the new matrix element index value obtained after dividing the matrix element index value by 4 and rounding it down.

[0111] As an example, the trust value can be updated using the following formula:

[0112]

[0113] Among them, C cand P represents the trust value of the candidate marker. cand L2(·) represents the pixel coordinates of the candidate marker point, L2(·) represents the Euclidean distance function, and η is an adjustable fixed parameter. This represents the confidence value of candidate markers for consecutively missing markers in the nth frame of the haptic image. This represents the confidence value of a candidate marker in the (n-1)th frame of the tactile image, indicating the number of consecutively missing markers. This represents the maximum trust value of all candidate markers when obtaining consecutively missing markers in the (n-1)th frame of the tactile image.

[0114] The repair algorithm proposed in this disclosure utilizes information from the current frame as much as possible to recover missing marker points, rather than relying entirely on marker point information from the previous frame. Furthermore, this repair process is iteratively enhanced over time. As missing marker points fail to find a match in multiple frames, the confidence value of their candidate marker points gradually decreases, while the confidence value of the calculated potential pixel coordinates remains unchanged. Therefore, the repair algorithm tends to select the calculated potential pixel coordinates. Considering that the missing marker point situation corresponds to a scenario where the contact module 101 undergoes significant deformation in real experiments, this state will not last too long. When the contact module 101 returns to a relatively flat state, the potential pixel coordinates calculated for the missing marker points are more likely to accurately represent the true location of the missing marker points. Therefore, the repair algorithm can effectively solve the problem of missing marker points caused by significant deformation and frame loss.

[0115] Figure 9A flowchart illustrating a method for updating a tactile map of an underwater object surface according to an exemplary embodiment of the present disclosure is shown.

[0116] Reference Figure 9 In step S901, based on the three-dimensional tactile point cloud corresponding to the multi-frame tactile images (i.e., the sequence of tactile images captured during a complete approach-contact-separation process), a set of tactile images corresponding to the current complete contact process between the contact module 101 and the underwater object is selected from the multi-frame tactile images. The set of tactile images corresponding to the current complete contact process can be understood as: the visual images captured from the start of contact with the underwater object (i.e., the target object) to the end of contact with the underwater object.

[0117] As an exemplary embodiment, step S901 may include: for each frame of the multi-frame tactile images, based on the three-dimensional tactile point cloud corresponding to that frame and the three-dimensional tactile point cloud corresponding to the first frame in the multi-frame tactile images, determining the intensity of the tactile stimulus received by the underwater tactile sensor 100 when that frame of tactile images was captured; if the intensity of the tactile stimulus corresponding to that frame of tactile images exceeds a preset threshold, then adding that frame of tactile images to the tactile image set. The intensity of the tactile stimulus is positively correlated with the degree of deformation of the contact module 101.

[0118] To eliminate noise interference from the tactile point cloud and prevent false recognition, this disclosure presents the following contact determination algorithm. Inspired by the human pain intensity model, a tactile stimulus intensity is defined to evaluate the tactile stimulus intensity received by the underwater tactile sensor 100. This intensity is proportional to, but not linearly related to, the deformation of the contact module 101. As an example, the tactile stimulus intensity function of the underwater tactile sensor 100 can be defined as follows:

[0119]

[0120] Among them, I st k represents the intensity of tactile stimulation. st P1 represents the proportional parameter that adjusts stimulus sensitivity, and α represents the exponential parameter used to control the intensity of the tactile stimulus. c This represents the initial point cloud (i.e., the 3D haptic point cloud corresponding to the first frame) and the current point cloud (i.e., the 3D haptic point cloud corresponding to the current frame). S represents the mean of the initial point cloud and the mean of the current point cloud. c ρ represents the deformation of the gel layer. c L2(·) represents the correlation between the current point cloud and the initial point cloud, and L2(·) represents the calculation of the Euclidean distance function.

[0121] As an example, a tactile stimulation threshold (i.e., a preset threshold) can be defined to determine whether contact has occurred, as shown in the following formula:

[0122]

[0123] Among them, E contact Indicates whether contact has occurred, I th This indicates the threshold for tactile stimulation.

[0124] Based on the above formula, the exemplary embodiments of this disclosure tend to treat changes in point cloud shape as contact, while simple jitter of marker points will cause ρ c A score close to 1 results in a low tactile stimulus intensity score.

[0125] In step S902, the key frame with the most significant three-dimensional tactile point cloud deformation is selected from the tactile image set.

[0126] In step S903, the three-dimensional tactile point cloud corresponding to the key frame is integrated into the tactile map of the surface of the underwater object (i.e., the target object) to obtain the shape and texture of the underwater object.

[0127] In each contact between the underwater tactile sensor 100 and the object, the frame with the most significant point cloud deformation is selected as the key frame for updating the tactile map. This selection is based on the fact that the moment with the most significant point cloud deformation represents the moment when the underwater tactile sensor 100 has the most sufficient contact with the object.

[0128] Due to the size limitation of the marker layer of the underwater tactile sensor 100, the 3D tactile point cloud acquired from the keyframes can only reflect a small area of ​​the target object's surface. Therefore, this disclosure also proposes a teleoperated tactile perception framework. A teleoperated underwater robot carries an underwater tactile sensor to touch the target object multiple times. Then, the 3D tactile point clouds corresponding to the keyframes in the tactile image sequence corresponding to each touch process (i.e., a complete approach-contact-separation process) are stitched together to obtain a complete tactile impression of the target object. These stitched 3D tactile point clouds constitute a tactile map, containing the texture and shape information of the target object.

[0129] Figure 10 An example of an underwater tactile sensing method according to an exemplary embodiment of the present disclosure is shown.

[0130] The marker tracking algorithm in the underwater tactile sensing method according to an exemplary embodiment of this disclosure may include four steps: initialization, segmentation, matching, and repair, to achieve the discrimination and tracking of marker points under vibration conditions. Wherein, F n t represents the nth frame in a sequence of tactile images captured by the underwater tactile sensor 100 during a complete approach-contact-separation process against a target object. n This indicates that the nth frame F was captured. n At that moment.

[0131] Initialization: For the tactile image F0 output during the initialization of the underwater tactile sensor 100, the pixel coordinates of each marker point are directly obtained using prior knowledge and numbered as a pre-determined marker point coordinate matrix.

[0132] Segmentation: For the tactile image output by the underwater tactile sensor 100, acquire the current frame F. n The tactile image F is obtained using a color segmentation algorithm based on HSV channels. n The monochrome channel images corresponding to the red, green, blue, and black channels are obtained, and then the centroid pixel coordinates of the marker points on each monochrome channel image are obtained through a speckle detection algorithm, which are used as the pixel coordinates of the detected marker points.

[0133] Matching: For the marker pixel coordinates of the red, green, blue, and black channels obtained in the segmentation step, the current frame F is matched using a K-nearest neighbor matching algorithm. n The pixel coordinates of the marker point are the same as those of the previous frame F. n-1 The pixel coordinates of the marker points are matched, and unmatched marker points are treated as missing marker points to obtain an incomplete seating table U to be repaired. n-1 .

[0134] Fix: For incomplete seating table U obtained during the matching step n-1 Based on the prior structural information of the multi-color marker field, the pixel coordinates of missing markers are filled in using the pixel coordinates of intact markers according to the parallelogram law, so as to obtain the repaired complete seating table T. n .

[0135] By repeating the above steps, you can obtain the seating chart of the marker points for each frame.

[0136] The surface reconstruction algorithm in the underwater tactile perception method according to an exemplary embodiment of this disclosure can employ triangulation to obtain the coordinates of spatial marker points using the correspondence between left and right target marker points, thus forming a three-dimensional tactile point cloud. Specifically, the surface reconstruction algorithm utilizes the seating chart of the left and right targets in the current frame obtained by the marker point tracking algorithm, obtains the correspondence between the left and right target marker points according to the seating chart, and then uses triangulation to obtain the spatial coordinates of the marker points to obtain the three-dimensional tactile point cloud.

[0137] Figure 11 A structural block diagram of an underwater robot according to an exemplary embodiment of the present disclosure is shown.

[0138] Reference Figure 11 An underwater robot 1000 according to an exemplary embodiment of the present disclosure includes: an underwater tactile sensor 100, a robotic arm 200 with the underwater tactile sensor 100 mounted at its end, and a controller 300.

[0139] As an exemplary embodiment, the controller 300 may be configured to perform the underwater tactile sensing method as described above.

[0140] It should be understood that the underwater robot 1000 may also include other devices for achieving its functions, and this disclosure does not limit this.

[0141] Figure 12 A schematic diagram of the structure of a teleoperated tactile sensing platform according to an exemplary embodiment of the present disclosure is shown.

[0142] like Figure 12 As shown, the teleoperated tactile sensing platform according to an exemplary embodiment of the present disclosure includes: an underwater robot 1000 equipped with a six-degree-of-freedom robotic arm, two force-touch interaction devices, a host computer, and an underwater tactile sensor 100 installed at the end of the robotic arm.

[0143] As an exemplary embodiment, the underwater robot 1000 may also be equipped with four cameras to provide a wide field of view for the robotic arm operation.

[0144] As an exemplary embodiment, two electromagnetic adsorption devices can be installed on the bottom of the underwater robot 1000 as attachment devices, or they can be installed on the front bracket to attach to the front plane.

[0145] As an exemplary embodiment, the force-touch interaction device is used to control the movement of the underwater robot 1000 body and its robotic arm, and to provide torque feedback to the operator, enabling them to perceive tactile force during operation. As an example, the force-touch interaction device can be divided into a left-hand device and a right-hand device, used to control the positions of the underwater robot 1000 body and its robotic arm, respectively. For the underwater robot 1000 body, motion control can be achieved by mapping the end effector coordinates of the left-hand force-touch interaction device to the thrust torque of the body; for the underwater robotic arm, a spring-damping system can be used to map the end effector coordinates of the right-hand force-touch interaction device to the coordinates of the robotic arm, while filtering out hand tremors and enhancing the user experience.

[0146] As an exemplary embodiment, the underwater tactile sensor 100 can be connected to the controller 300 of the underwater robot 1000 via USB. The underwater tactile sensor 100 transmits tactile images to the controller 300, and the controller 300 can generate a three-dimensional tactile point cloud in real time based on the received tactile images.

[0147] As an exemplary embodiment, the teleoperated tactile sensing platform can communicate using ROS.

[0148] Figure 13 An example of a teleoperable tactile sensing framework according to an exemplary embodiment of the present disclosure is shown.

[0149] like Figure 13As shown, the teleoperable tactile sensing framework according to an exemplary embodiment of the present disclosure can remotely control an underwater robot 1000 carrying an underwater tactile sensor 100 to sense objects using a force-touch interaction method, so as to realize the use of the underwater robot 1000 to perceive the shape and texture of objects.

[0150] The operator can control the underwater robot 1000, which carries the underwater tactile sensor 100, to contact underwater objects through the remote tactile sensing framework. The robot outputs the three-dimensional point cloud of the contacted object in real time through the three-dimensional tactile point cloud generation algorithm, thus completing the perception of the shape and texture of the underwater object.

[0151] When the underwater tactile sensor 100 comes into contact with an underwater object, the deformation of its contact module 101 causes a change in the distribution of the three-dimensional tactile point cloud. Utilizing these changes in the point cloud, a tactile feedback force is generated through a tactile force neural network and applied to the operator's hand via a force-touch interaction device, allowing the operator to perceive the contact. Simultaneously, the three-dimensional tactile point cloud of the contact area is used to update the entire tactile map. As an example, the three-dimensional tactile point cloud can be mapped to a contact force through a neural network, and then scaled proportionally as the feedback torque. The tactile feedback force can be calculated using the following formula:

[0152] F fb =k fb ·F net

[0153] F net k represents the output of the neural network. fb F represents the coefficient parameter. fb This refers to the tactile feedback force applied to the operator.

[0154] The teleoperable tactile perception framework can remotely control an underwater robot 1000 carrying an underwater tactile sensor 100 to contact a target object through force-touch interaction. The framework stitches together the three-dimensional tactile point cloud of each contact to obtain the complete shape and texture of the underwater object.

[0155] As an exemplary embodiment, this disclosure can stitch together three-dimensional point clouds from different locations to obtain a tactile map based on the coordinates of the end effector in the robot arm coordinate system. As an example, coordinate transformation can be used to integrate the three-dimensional point clouds corresponding to keyframes into the tactile map.

[0156] An exemplary embodiment of the present disclosure also provides an electronic device comprising: at least one memory and at least one processor, wherein the at least one memory stores a set of computer-executable instructions, which, when executed by the at least one processor, perform the underwater tactile sensing method as described in the exemplary embodiment above.

[0157] As an example, the electronic device could be the controller 300 of an underwater robot 1000.

[0158] As an example, an electronic device can be a host computer, such as a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, the electronic device is not necessarily a single device; it can be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The electronic device can also be part of an integrated control system or system manager, or can be configured to interface with a portable electronic device locally or remotely (e.g., via wireless transmission).

[0159] In electronic devices, processors may include central processing units (CPUs), graphics processing units (GPUs), programmable logic devices, dedicated processor systems, microcontrollers, or microprocessors. By way of example and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0160] The processor can execute instructions or code stored in memory, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0161] Memory can be integrated with the processor; for example, RAM or flash memory can be housed within an integrated circuit microprocessor. Alternatively, memory can comprise a separate device, such as an external disk drive, storage array, or other storage device that can be used by any database system. Memory and processor can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor to read files stored in the memory.

[0162] In addition, electronic devices may include video displays (such as liquid crystal displays) and user interaction interfaces (such as keyboards, mice, touch input devices, etc.). All components of the electronic device may be interconnected via buses and / or networks.

[0163] According to exemplary embodiments of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, they cause at least one processor to perform the underwater tactile sensing method as described in the exemplary embodiments above. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0164] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, wherein the instructions in the computer program product are executable by at least one processor to perform the underwater tactile sensing method as described in the exemplary embodiments above.

[0165] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0166] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An underwater tactile sensor, characterized in that, include: A contact module is used to contact the surface of an underwater object to cause deformation, the deformation causing displacement of the multi-color marker field on the upper surface of the contact module; A binocular camera module is used to capture the displacement of the multi-color marker field to obtain a tactile image, wherein the tactile image includes a left-eye tactile image and a right-eye tactile image; A waterproof module is used to enable the underwater tactile sensor to be waterproof. A pressure balancing chamber is a cavity used to bring the lower surface of the contact module into contact with the external water body, so as to achieve water pressure balance between the upper and lower surfaces of the contact module. And / or, a light chamber module, used to provide illumination conditions for the binocular camera module to capture the displacement of the multi-color marker field; The multi-color marker field includes: multiple marker groups arranged sequentially, each marker group including j markers of different colors, the spatial order of the j markers of different colors in each marker group is fixed, and j is an integer greater than 1; The contact module includes a casting layer, a marker layer, and a gel layer installed on top of the underwater tactile sensor, wherein the marker layer is filled with the multi-color marker field; the light chamber module includes a quartz glass layer, a light diffusion layer, and an LED light strip; the pressure balancing chamber is a cavity disposed between the gel layer and the quartz glass layer; the binocular camera module is installed at the bottom center of the light chamber module; the waterproof module includes an upper cover, a waterproof outer shell, and a lower cover, wherein the waterproof outer shell is connected to the quartz glass layer of the light chamber module by a sealing ring, and the waterproof outer shell is connected to the lower cover by a sealing ring; the gel layer passes through the upper cover, and the casting layer and the marker layer are in contact with the external water body; the upper cover is also provided with a water inlet so that the pressure balancing chamber can communicate with the external water body through the water inlet.

2. An underwater tactile sensing method, characterized in that, include: Acquire multiple frames of tactile images from the underwater tactile sensor as described in claim 1; Each frame of tactile image is taken as the current frame, and a marker point coordinate matrix corresponding to the current frame is generated. Each element in the marker point coordinate matrix corresponds one-to-one with each marker point in the multi-color marker point field, and the content of each element is the pixel coordinate of the marker point corresponding to that element in the current frame. Based on the coordinate matrix of the marker points corresponding to the current frame, the three-dimensional spatial coordinates of each marker point in the multi-color marker point field are determined to obtain the three-dimensional tactile point cloud corresponding to the current frame; The marker point coordinate matrix corresponding to the current frame includes: the marker point coordinate matrix corresponding to the left visual tactile image and the marker point coordinate matrix corresponding to the right visual tactile image in the current frame.

3. The underwater tactile sensing method according to claim 2, characterized in that, The step of generating the marker point coordinate matrix corresponding to the current frame includes: The current frame is divided into j monochrome channel images, each corresponding to one of the j color channels; Based on the coordinate matrix of the marker points corresponding to the previous frame of the current frame, determine the coordinate matrix of the marker points to be repaired corresponding to each monochrome channel image of the current frame; The coordinate matrices of the marker points to be repaired corresponding to each monochrome channel image are merged to obtain the coordinate matrix of the marker points to be repaired corresponding to the current frame. The coordinate matrix of the marker points to be repaired corresponding to the current frame is repaired to obtain the coordinate matrix of the marker points corresponding to the current frame.

4. The underwater tactile sensing method according to claim 3, characterized in that, The step of determining the marker point coordinate matrix to be repaired for each monochrome channel image of the current frame based on the marker point coordinate matrix corresponding to the previous frame of the current frame includes: Based on the marker point coordinate matrix corresponding to the previous frame, determine the marker point coordinate matrix corresponding to each monochrome channel image of the previous frame; Obtain the pixel coordinates of the marker points detected from the monochrome channel image of the Mth color of the current frame using a blob detection algorithm; Based on the matching result between the detected pixel coordinates of the marker points and the marker point coordinate matrix corresponding to the monochrome channel image of the Mth color of the previous frame, the marker point coordinate matrix to be repaired corresponding to the monochrome channel image of the Mth color of the current frame is obtained. Where M is an integer greater than 0 and less than or equal to j.

5. The underwater tactile sensing method according to claim 3, characterized in that, The step of repairing the marker point coordinate matrix corresponding to the current frame to obtain the marker point coordinate matrix corresponding to the current frame includes: For the missing marker points in the marker point coordinate matrix to be repaired corresponding to the current frame, the potential pixel coordinates of the missing marker points are determined based on the pixel coordinates of other marker points in the marker point group to which the missing marker point belongs. Based on the potential pixel coordinates, search for candidate marker points for the missing marker points in the current frame; Pixel coordinates are selected from the pixel coordinates of the candidate markers and the potential pixel coordinates as the pixel coordinates of the missing markers; Add the pixel coordinates of the missing marker points to the marker point coordinate matrix to be repaired corresponding to the current frame.

6. The underwater tactile sensing method according to claim 2, characterized in that, Also includes: Based on the three-dimensional tactile point cloud corresponding to the multi-frame tactile images, a set of tactile images corresponding to the complete contact process between the contact module and the underwater object is selected from the multi-frame tactile images; Select the keyframe with the most significant three-dimensional tactile point cloud deformation from the set of tactile images; The 3D tactile point cloud corresponding to the keyframe is integrated into the tactile map of the underwater object surface to obtain the shape and texture of the underwater object.

7. The underwater tactile sensing method according to claim 6, characterized in that, The step of selecting the set of tactile images corresponding to the complete contact process between the contact module and the underwater object from the multi-frame tactile images based on the three-dimensional tactile point cloud corresponding to the multi-frame tactile images includes: For each frame of tactile image, based on the three-dimensional tactile point cloud corresponding to that frame of tactile image and the three-dimensional tactile point cloud corresponding to the first frame in the multi-frame tactile image, the intensity of the tactile stimulus received by the underwater tactile sensor when the frame of tactile image is captured is determined, wherein the intensity of the tactile stimulus is positively correlated with the degree of deformation of the contact module; If the intensity of the tactile stimulus corresponding to the frame of tactile image exceeds a preset threshold, then the frame of tactile image is added to the set of tactile images.

8. An underwater robot, characterized in that, include: The underwater tactile sensor as described in claim 1; A robotic arm with the underwater tactile sensor installed at its end; Controller.

9. A computer-readable storage medium for storing instructions, characterized in that, When the instructions are executed by the processor of the electronic device, the electronic device is enabled to perform the underwater tactile sensing method as described in any one of claims 2 to 7.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; At least one memory that stores computer-executable instructions. The computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the underwater tactile sensing method as described in any one of claims 2 to 7.

11. A computer program product comprising computer-executable instructions, characterized in that, When the computer-executable instructions are executed by at least one processor, they implement the underwater tactile sensing method as described in any one of claims 2 to 7.

Images