Object shape and texture sensing method for underwater robot and corresponding device

Abstract

The invention provides an underwater robot-oriented object shape and texture sensing method and a corresponding device. The underwater tactile sensor comprises a contact module which is used for contacting the surface of an underwater object to deform, and the deformation enables a multi-color mark point field on the upper surface of the contact module to generate displacement; the binocular camera module is used for capturing displacement of the multi-color mark point field to obtain tactile images, and the tactile images comprise a left-eye tactile image and a right-eye tactile image; the waterproof module is used for realizing a waterproof function of the underwater tactile sensor; wherein the multi-color mark point field comprises a plurality of mark point groups arranged in sequence, each mark point group comprises j mark points with different colors, the spatial sequence of the mark points with the j colors in each mark point group is fixed, and j is an integer greater than 1.

Description

Technical Field

[0001] The present disclosure generally relates to the field of object perception technology, and more specifically, to an object shape and texture perception method and corresponding device for an underwater robot. Background Art

[0002] Tactile perception is an important close-range perception method for organisms and has unique advantages. Tactile perception obtains the shape and texture information of an object through direct contact, and can also obtain the force of the contact process, which can guide the limbs or fingers of the organism to perform complex operations. In addition, tactile perception can assist other perception methods to complete perception tasks together: as an alternative perception method when visual perception fails, such as the blind man touching an elephant. Tactile perception has important application value underwater: in turbid underwater environments, underwater tactile sensors can replace underwater cameras to complete perception tasks; in sonar dead zones (close-range areas that sonar cannot perceive), underwater tactile sensors can provide high-resolution object shape and texture information.

[0003] With the continuous development of new material technology and sensor technology, artificial tactile technology has begun to attract the attention of researchers. As a supplement and extension of vision, tactile sensors can be integrated into the end effector of the robotic arm, and they are naturally capable of guiding dexterous operations. In recent years, a series of underwater tactile sensors suitable for underwater operations have emerged. According to different functions, underwater tactile sensors can be divided into force touch sensors that sense mechanical quantities such as pressure, and biomimetic tactile sensors that sense other physical quantities such as texture, temperature, and vibration, also called biomimetic skin. Underwater force touch sensors can be divided into piezoresistive, capacitive, photoelectric, electromagnetic, ultrasonic, resistive strain, etc. according to their principles. These sensors convert the contact deformation of the soft surface into electrical signals and obtain touch through signal processing. However, these underwater tactile sensing technologies face some functional difficulties and perform poorly in actual underwater environments. Summary of the invention

[0004] An exemplary embodiment of the present disclosure is to provide an object shape and texture perception method and corresponding device for an underwater robot, which can efficiently and accurately perceive the shape and texture of underwater objects based on vision.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an underwater tactile sensor, comprising: a contact module, for contacting the surface of an underwater object to deform, wherein the deformation causes a multi-color marker point field on the upper surface of the contact module to be displaced; a binocular camera module, for capturing the displacement of the multi-color marker point field to obtain a tactile image, wherein the tactile image comprises a left-eye tactile image and a right-eye tactile image; and a waterproof module, for realizing a waterproof function of the underwater tactile sensor; wherein the multi-color marker point field comprises: a plurality of marker point groups arranged in sequence, each marker point group comprises j marker points of different colors, the spatial ordering of the marker points of j colors in each marker point group is fixed, and j is an integer greater than 1.

[0006] Optionally, it further comprises: a pressure balance chamber, which is a chamber used to make the lower surface of the contact module contact with external water to achieve water pressure balance between the upper and lower surfaces of the contact module.

[0007] Optionally, it also includes: a light chamber module, used to provide lighting conditions for the binocular camera module to capture the displacement of the multi-color marking point field.

[0008] According to a second aspect of an embodiment of the present disclosure, there is provided an underwater tactile perception method, comprising: acquiring multiple frames of tactile images from the underwater tactile sensor as described above; taking each frame of the tactile image as a current frame, and generating a marker point coordinate matrix corresponding to the current frame, wherein each element in the marker point coordinate matrix corresponds one-to-one to each marker point in the multi-color marker point field, and the content of each element is the pixel coordinate of the marker point corresponding to the element in the current frame; based on the marker point coordinate matrix corresponding to the current frame, determining the three-dimensional spatial coordinates of each marker point in the multi-color marker point field to obtain a three-dimensional tactile point cloud corresponding to the current frame; wherein the marker point coordinate matrix corresponding to the current frame includes: a marker point coordinate matrix corresponding to the left-eye tactile image and a marker point coordinate matrix corresponding to the right-eye tactile image in the current frame.

[0009] Optionally, the step of generating a marker point coordinate matrix corresponding to the current frame includes: dividing the current frame into j monochrome channel images corresponding one by one to j color channels; determining the marker point coordinate matrices to be repaired corresponding to each monochrome channel image of the current frame based on the marker point coordinate matrix corresponding to the previous frame of the current frame; merging the marker point coordinate matrices to be repaired corresponding to each monochrome channel image to obtain the marker point coordinate matrix to be repaired corresponding to the current frame; repairing the marker point coordinate matrix to be repaired corresponding to the current frame to obtain the marker point coordinate matrix corresponding to the current frame.

[0010] Optionally, the step of determining the marker point coordinate matrix to be repaired corresponding to each monochrome channel image of the current frame based on the marker point coordinate matrix corresponding to the previous frame of the current frame includes: determining the marker point coordinate matrix corresponding to each monochrome channel image of the previous frame based on the marker point coordinate matrix corresponding to the previous frame; obtaining the pixel coordinates of the marker points detected from the monochrome channel image of the Mth color of the current frame by the spot detection algorithm; obtaining the marker point coordinate matrix to be repaired corresponding to the monochrome channel image of the Mth color of the current frame based on the matching results of the pixel coordinates of the detected marker points and the marker point coordinate matrix corresponding to the monochrome channel image of the Mth color of the previous frame; wherein M is an integer greater than 0 and less than or equal to j.

[0011] Optionally, the step of repairing the marker point coordinate matrix to be repaired corresponding to the current frame to obtain the marker point coordinate matrix corresponding to the current frame includes: for the missing marker point in the marker point coordinate matrix to be repaired corresponding to the current frame, determining the potential pixel coordinates of the missing marker point based on the pixel coordinates of other marker points in the marker point group where the missing marker point is located; based on the potential pixel coordinates, searching for candidate marker points of the missing marker point from the current frame; selecting pixel coordinates from the pixel coordinates of the candidate marker points and the potential pixel coordinates as the pixel coordinates of the missing marker point; and adding the pixel coordinates of the missing marker point to the marker point coordinate matrix to be repaired corresponding to the current frame.

[0012] Optionally, it also includes: based on the three-dimensional tactile point clouds corresponding to the multiple frames of tactile images, screening out a tactile image set corresponding to the current complete contact process between the contact module and the underwater object from the multiple frames of tactile images; selecting a key frame with the most significant deformation of the corresponding three-dimensional tactile point cloud from the tactile image set; integrating the three-dimensional tactile point cloud corresponding to the key frame into the tactile map of the surface of the underwater object to obtain the shape and texture of the underwater object.

[0013] Optionally, the step of filtering out a tactile image set corresponding to the current complete contact process between the contact module and the underwater object from the multiple frames of tactile images based on the three-dimensional tactile point cloud corresponding to the multiple frames of tactile images includes: for each frame of tactile image, based on the three-dimensional tactile point cloud corresponding to the frame of tactile image and the three-dimensional tactile point cloud corresponding to the first frame of the multiple frames of tactile images, determining the tactile stimulation intensity received by the underwater tactile sensor when the frame of tactile image is captured, wherein the tactile stimulation intensity is positively correlated with the deformation degree of the contact module; if the tactile stimulation intensity corresponding to the frame of tactile image exceeds a preset threshold, adding the frame of tactile image to the tactile image set.

[0014] According to a third aspect of the embodiments of the present disclosure, there is provided an underwater robot, comprising: the underwater tactile sensor as described above; a mechanical arm with the underwater tactile sensor mounted at the end; and a controller.

[0015] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided. When the instructions are executed by a processor of an electronic device, the electronic device is enabled to perform the underwater tactile perception method as described above.

[0016] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to execute the underwater tactile perception method as described above.

[0017] According to a sixth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising computer executable instructions, which, when executed by at least one processor, implement the underwater tactile perception method as described above.

[0018] According to the object shape and texture perception method and corresponding device for an underwater robot of the exemplary embodiment of the present disclosure, the shape and texture of underwater objects can be perceived efficiently and accurately based on vision.

[0019] In the following description, some aspects and / or advantages of the general inventive concept will be set forth, and some other aspects and / or advantages will be known through the following description or the implementation of the general inventive concept. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] These and / or other aspects and advantages of the present application will become clearer and easier to understand from the following detailed description of the embodiments of the present application in conjunction with the accompanying drawings, in which:

[0021] Figure 1 A structural block diagram of an underwater tactile sensor according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 2 A schematic structural diagram of an underwater tactile sensor according to an exemplary embodiment of the present disclosure is shown;

[0023] Figure 3 A schematic structural diagram of a pressure balance chamber according to an exemplary embodiment of the present disclosure is shown;

[0024] Figure 4 A schematic diagram showing the structure of a marking point field according to an exemplary embodiment of the present disclosure is shown;

[0025] Figure 5A flowchart showing an underwater tactile perception method according to an exemplary embodiment of the present disclosure;

[0026] Figure 6 A flowchart showing a method for generating a marker point coordinate matrix corresponding to a current frame according to an exemplary embodiment of the present disclosure is shown;

[0027] Figure 7 A flowchart showing a method for determining a coordinate matrix of a marker point to be repaired corresponding to each monochrome channel image of a current frame according to an exemplary embodiment of the present disclosure is shown;

[0028] Figure 8 A flowchart showing a method for repairing a marker point coordinate matrix to be repaired corresponding to a current frame according to an exemplary embodiment of the present disclosure is shown;

[0029] Fig. 9 A flowchart showing a method for updating a tactile map of a surface of an underwater object according to an exemplary embodiment of the present disclosure;

[0030] Fig.10 An example of an underwater tactile perception method according to an exemplary embodiment of the present disclosure is shown;

[0031] Fig.11 A structural block diagram of an underwater robot according to an exemplary embodiment of the present disclosure is shown;

[0032] Fig.12 A schematic diagram showing the structure of a remote-operated tactile perception platform according to an exemplary embodiment of the present disclosure is shown;

[0033] Fig.13 An example of a teleoperation tactile perception framework according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0034] Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments will be described below with reference to the drawings in order to explain the present disclosure.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0036] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0037] Existing underwater tactile sensing technology faces some functional difficulties. It performs poorly in actual underwater environments, and the sensor manufacturing process and cost need to be improved. There is still a lack of commercial underwater tactile sensors. Specifically, it is difficult to effectively eliminate the influence of water pressure in underwater tactile sensing. Its output may change under water pressure at different depths, and it may not even work properly in deep water areas. Underwater force touch sensors based on electrical signals are easily affected by underwater impurities and temperature changes, and may experience self-heating and zero drift, making it difficult to work stably for a long time. Underwater tactile sensing has the problem of waterproof sealing. The size of the sensor will be affected by waterproofing and is difficult to miniaturize. Its manufacturing process is also relatively complex and cumbersome, and the production cost is high. Due to the limitations of the processing technology, it is difficult to achieve a higher resolution layout for array-based underwater waterproof skin, which makes it disadvantageous in terms of resolution.

[0038] In response to the above-mentioned underwater perception problems, on the one hand, the present disclosure proposes a vision-based underwater tactile sensor, which adopts a multi-color marker point field structure to achieve high-resolution perception under low distribution density of marker points; it adopts a waterproof and water pressure balanced structural design, and can work in an outdoor underwater environment at a depth of 50 meters; in addition, the sensor shape is miniaturized to facilitate installation and operation at the end of an underwater robotic arm. On the other hand, the present disclosure also proposes a method for generating a three-dimensional tactile point cloud of the surface of a contacted object based on the tactile image output by an underwater tactile sensor, and introduces a marker point tracking algorithm and a surface reconstruction algorithm, which can generate a high-resolution three-dimensional point cloud of the contact surface of the underwater tactile sensor in a vibration environment, so as to realize the precise perception function of the object surface in a vibration environment. In addition, the present disclosure also proposes a remote-operated tactile perception framework for sensing the shape and texture of underwater target objects. The following will be combined with Figures 1 to 13 Provide detailed explanation.

[0039] Figure 1 A structural block diagram of an underwater tactile sensor according to an exemplary embodiment of the present disclosure is shown.

[0040] Reference Figure 1According to an exemplary embodiment of the present disclosure, an underwater tactile sensor 100 includes: a contact module 101 , a binocular camera module 102 , and a waterproof module 103 .

[0041] Specifically, the contact module 101 is used to contact the surface of an underwater object to cause deformation, and the deformation causes the multi-color marking point field on the upper surface of the contact module 101 to be displaced.

[0042] The binocular camera module 102 is used to capture the displacement of the multi-color mark point field to obtain a tactile image, which includes a left-eye tactile image and a right-eye tactile image.

[0043] The basic principle of the underwater tactile sensor 100 is to convert the tactile information obtained when the sensor contacts the underwater object into the deformation of the contact module 101. This deformation will cause the displacement of the surface markers on the contact module 101. The binocular camera module 102 identifies and tracks these markers to obtain a tactile image. By performing three-dimensional reconstruction on the tactile image, the three-dimensional point cloud information of the surface of the underwater object can be obtained, thereby realizing the perception of the shape and texture of the underwater object.

[0044] The multi-color marking point field includes: a plurality of marking point groups arranged in sequence, each marking point group includes j marking points of different colors, the spatial ordering of the marking points of j colors in each marking point group is fixed, and j is an integer greater than 1.

[0045] As an exemplary embodiment, j may be 4, and the j colors may include: red, green, blue, and black. It should be understood that the value of j may be set according to actual conditions and specific needs, and the j colors are not limited to the above colors.

[0046] As an example, Figure 4 As shown, Figure 4 (a) in FIG. 1 shows a cross-sectional view of a multi-color marker field. Figure 4 (b) shows a top view of a multi-color marking dot field. The multi-color marking dot field may be a square marking dot array consisting of four-color marking dots with a diameter of 0.5 mm, and the spacing between adjacent marking dots may be 1.0 mm. Figure 4 (c) in FIG. 5 shows a schematic diagram of a marker point group, in which the upper left marker point in each marker point group is red, the upper right marker point is green, the lower left marker point is blue, and the lower right marker point is black.

[0047] like Figure 4 As shown in (d) in the figure, the multi-color marker point field is a marker point field formed by the red marker point field, the green marker point field, the blue marker point field, and the black marker point field. Since the distribution density of the marker points in the monochrome marker point field is 2 mm × 2 mm, and the perception resolution of the multi-color marker point field is 1 mm × 1 mm, high-resolution perception capability under low-resolution distribution can be achieved.

[0048] It should be understood that the spatial ordering position of the marking points in the multi-color marking point field (for example, the row and column) is fixed, and the multi-color marking point field may be deformed, rotated, scaled, etc. as the contact module 101 deforms.

[0049] The waterproof module 103 is used to realize the waterproof function of the underwater tactile sensor 100 .

[0050] As an exemplary embodiment, the underwater tactile sensor 100 according to an exemplary embodiment of the present disclosure may further include an optical chamber module 104. The optical chamber module 104 is used to provide lighting conditions (eg, white uniform lighting conditions) for the binocular camera module 102 to capture the displacement of the marker field.

[0051] As an exemplary embodiment, the underwater tactile sensor 100 according to the exemplary embodiment of the present disclosure may further include a pressure balance chamber 105. The pressure balance chamber 105 is a chamber for making the lower surface of the contact module 101 contact with the external water body to achieve water pressure balance between the upper and lower surfaces of the contact module 101.

[0052] Figure 2 A schematic structural diagram of an underwater tactile sensor according to an exemplary embodiment of the present disclosure is shown.

[0053] Figure 2 (A) shows a cross-sectional view of the structure of the underwater tactile sensor 100. Figure 2 (B) in FIG. 1 shows a prototype of the underwater tactile sensor 100 . Figure 2 (C) in FIG. 1 shows an exploded view of the underwater tactile sensor 100 .

[0054] Reference Figure 2 The contact module 101 may include a casting layer installed on the top of the underwater tactile sensor 100, an RGBK (red, green, blue, black) marking point layer (filled with a multi-color marking point field), and a gel layer.

[0055] The optical chamber module 104 may include a quartz glass layer, a light diffusion layer, and an LED light strip installed under the contact module 101. The optical chamber module 104 is a sealed cavity filled with air, and is intended to provide a stable lighting condition for the binocular camera module 102.

[0056] The binocular camera module 102 may include a binocular camera and a control circuit board installed at the bottom center of the optical chamber module 104 .

[0057] The waterproof module 103 may include an upper cover, a waterproof housing, and a lower cover. As an example, the waterproof housing may be connected to the quartz glass layer of the optical chamber module 104 through a sealing ring (e.g., a rubber sealing ring), the waterproof housing may be connected to the lower cover through a sealing ring, and the lower cover may transmit signals and power through a waterproof aviation plug connector (e.g., a five-hole waterproof aviation plug connector) to isolate the underwater tactile sensor 100 from the external water body and keep the internal space full of air. For example, the waterproof aviation plug connector may be connected to the internal control panel and LED light strip to output USB signals while providing power to the LED light strip. As an example, the waterproof module 103 may be made of aluminum alloy and may withstand 50 meters of water pressure.

[0058] Waterproof capability is a key factor affecting the performance of the underwater tactile sensor 100, because the binocular camera, LED light bar and control board require a strictly water-free environment. The underwater tactile sensor 100 is waterproofed by an aluminum alloy shell and quartz glass. As an example, the quartz glass layer of the optical chamber module 104 is a cylindrical sheet structure with a thickness of 2 mm and a diameter of 41 mm, which is used to withstand the water pressure from its upper surface; the waterproof shell and the lower cover are made of 5 mm thick aluminum alloy to withstand the water pressure on the side and bottom of the underwater tactile sensor 100.

[0059] Figure 3 A schematic structural diagram of a pressure balance chamber according to an exemplary embodiment of the present disclosure is shown.

[0060] like Figure 3 As shown, a pressure balance chamber 105 is provided between the contact module 101 and the optical chamber module 104. As an example, the pressure balance chamber 105 may be a cavity with a height of 0.2 mm, which is connected to the external water body through the water inlet of the upper cover of the waterproof module 103 to achieve water pressure balance on the upper and lower surfaces of the contact module 101. For example, the upper cover of the underwater tactile sensor 100 may have four water inlets, which are directly connected to the pressure balance chamber 105 to ensure that the external water body enters the pressure balance chamber 105.

[0061] In the underwater tactile sensor 100, balancing water pressure is another key factor that cannot be ignored. As the operating water depth increases, the water pressure will also increase. When the underwater tactile sensor 100 is located at a depth of 50 meters, it will be subjected to a water pressure of about 5 atmospheres. Such a water pressure is an interference that cannot be ignored for the underwater tactile sensor 100 that relies on the deformation of the contact module 101. Therefore, the present disclosure proposes that in the underwater tactile sensor 100, there is no gel layer that completely seals the contact module 101, and a cavity with a height of 0.2 mm (i.e., the pressure balance cavity 105) is designed between the gel layer and the quartz glass layer. When the external water flow enters and fills the pressure balance cavity 105, the water pressure on the upper and lower surfaces of the contact module 101 is the same, thereby preventing the contact module 101 from being deformed due to excessive external water pressure.

[0062] Figure 5 A flow chart of an underwater tactile perception method according to an exemplary embodiment of the present disclosure is shown.

[0063] Reference Figure 5 In step S501 , multiple frames of tactile images are acquired from the underwater tactile sensor 100 .

[0064] As an exemplary embodiment, the multiple frames of tactile images are also a sequence of tactile images obtained in a complete approach-contact-separation process of an underwater target object. Each frame of tactile image actually includes a left-eye tactile image and a right-eye tactile image.

[0065] In step S502, each frame of the tactile image is taken as a current frame to generate a marker point coordinate matrix corresponding to the current frame.

[0066] Each element in the marker point coordinate matrix (hereinafter also referred to as the marker point seating table or simply the seating table) corresponds one-to-one to each marker point in the multi-color marker point field, and the content of each element is the pixel coordinate of the marker point corresponding to the element in the current frame. Specifically, the spatial sorting position (for example, the row and column) of each element in the marker point coordinate matrix is ​​the same as the spatial sorting position of the marker point corresponding to the element in the multi-color marker point field. That is, the marker point seating table contains not only the pixel coordinates of the marker points, but also the spatial sorting position information of the marker points.

[0067] The marker point coordinate matrix corresponding to the current frame includes: a marker point coordinate matrix corresponding to the left-eye tactile image and a marker point coordinate matrix corresponding to the right-eye tactile image in the current frame.

[0068] In step S503, based on the marker point coordinate matrix corresponding to the current frame, the three-dimensional spatial coordinates of each marker point in the multi-color marker point field are determined to obtain a three-dimensional tactile point cloud corresponding to the current frame.

[0069] As an exemplary embodiment, step S503 may include: determining the correspondence between the marker point coordinate matrix corresponding to the left-eye tactile image and the marker point coordinate matrix corresponding to the right-eye tactile image regarding the same marker point; and then using triangulation to determine the three-dimensional spatial coordinates of each marker point based on the correspondence, the marker point coordinate matrix corresponding to the left-eye tactile image, and the marker point coordinate matrix corresponding to the right-eye tactile image, as the three-dimensional tactile point cloud corresponding to the current frame.

[0070] According to the underwater tactile perception method of the exemplary embodiment of the present disclosure, marker point tracking is performed based on a multi-color marker point field, the positions of the marker points are detected from a series of tactile images captured by the underwater tactile sensor 100, and the marker point seating chart is obtained, which can achieve accurate tactile perception of the surface texture and shape of the target object in a complex underwater environment with vibration, impact or other interference. Existing underwater tactile perception technology is mainly carried out in a relatively stable laboratory environment. The sensor contacts the surface of the object gently and slowly to maintain good contact between the sensor and the object. It is not well suited for tactile perception in vibration and impact environments. Therefore, the present disclosure proposes a stable tactile point cloud generation method that not only has high resolution, but also can achieve accurate perception under interference such as vibration.

[0071] Figure 6 A flowchart of a method for generating a marker point coordinate matrix corresponding to a current frame according to an exemplary embodiment of the present disclosure is shown.

[0072] Reference Figure 6 In step S601, the current frame is divided into j monochrome channel images corresponding to j color channels. Specifically, the monochrome channel image of the Mth color is an image that only displays the Mth color mark point, where M is an integer greater than 0 and less than or equal to j.

[0073] As an exemplary embodiment, the tactile image captured by the underwater tactile sensor 100 may be in RGB format, and a color segmentation algorithm based on the HSV format may be used to obtain monochrome channel images corresponding to the red, green, blue and black channels. As an example, marker points of different colors may be distinguished according to a threshold in the H channel, and the background may be separated from each color marker point.

[0074] As an example, the color segmentation algorithm based on the HSV format can be implemented as follows:

[0075]

[0076] Among them, F n Represents the tactile image of the nth frame in RGB channel format, represents the HSV conversion function, represents the nth frame of tactile image converted to HSV channel, m, k represents the number of rows and columns of the nth frame of tactile image, Indicates obtaining the H channel from the nth frame tactile image of the HSV channel, H n represents the H channel of the nth frame tactile image. The Mth color is one of red, green, blue, and black. M ,U M Represents the lower threshold and upper threshold of the Mth color segmentation algorithm, Represents the monochrome channel image after the Mth color segmentation.

[0077] As an example, Figure 4 As shown in (d) in the figure, the multi-color marker field is formed by the combination of four single-color marker fields (red marker field, green marker field, blue marker field, and black marker field). Since the distribution density of the markers in the single-color marker field is 2 mm × 2 mm, the tactile perception range of each marker in the single-color marker field (which can be defined as the maximum movable area where the markers do not overlap) is 2.0 × 2.0 mm. 2 The square area (such as Figure 4 By utilizing the prior structure information of the multi-color marker field (i.e., the original marker distribution information), the spatial ordering position of the markers can be determined, thereby achieving sequential super-resolution perception. Specifically, the multi-color marker field combines the tactile perception ranges of the markers of the four colors to obtain a common area of ​​1.0×1.0mm. 2 ,like Figure 4 As shown in the middle superimposed area in (e), the perceived resolution of the multi-color marker field is 1.0×1.0 mm 2 According to the present disclosure, by segmenting the marker points according to four color channels, the multi-color marker point field achieves high perceived resolution at a low distribution density.

[0078] In step S602, based on the marker point coordinate matrix corresponding to the previous frame of the current frame, the marker point coordinate matrix to be repaired corresponding to each monochrome channel image of the current frame is determined.

[0079] The following will be combined Figure 7 An exemplary embodiment of step S602 is described, which will not be expanded here.

[0080] In step S603, the coordinate matrices of the marking points to be repaired corresponding to the monochrome channel images are merged to obtain the coordinate matrix of the marking points to be repaired corresponding to the current frame.

[0081] In step S604, the coordinate matrix of the marking points to be repaired corresponding to the current frame is repaired to obtain the coordinate matrix of the marking points corresponding to the current frame.

[0082] The following will be combined Figure 8 An exemplary embodiment of step S604 is described, which will not be expanded here.

[0083] It should be understood that steps S601-S604 are performed for the left eye tactile image in the current frame to obtain the marker point coordinate matrix corresponding to the left eye tactile image, and steps S601-S604 are performed for the right eye tactile image in the current frame to obtain the marker point coordinate matrix corresponding to the right eye tactile image.

[0084] Figure 7 A flowchart of a method for determining a coordinate matrix of marker points to be repaired corresponding to each monochrome channel image of a current frame according to an exemplary embodiment of the present disclosure is shown.

[0085] Reference Figure 7 In step S701, based on the marker point coordinate matrix corresponding to the previous frame, the marker point coordinate matrix corresponding to each monochrome channel image of the previous frame is determined.

[0086] Here, each element in the marker point coordinate matrix corresponding to the monochrome channel image of the Mth color corresponds one-to-one to each marker point in the single-color marker point field of the Mth color, and the content of each element is the pixel coordinate of the marker point corresponding to the element in the monochrome channel image.

[0087] As an exemplary embodiment, since the first frame F1 in the acquired multi-frame tactile image does not have a previous frame, the marker point coordinate matrix corresponding to the tactile image F0 captured during initialization (hereinafter also referred to as the predetermined marker point coordinate matrix) can be used as the marker point coordinate matrix corresponding to the previous frame of the first frame.

[0088] In step S702, the pixel coordinates of the marking point detected from the monochrome channel image of the Mth color of the current frame by the spot detection algorithm are obtained.

[0089] As an exemplary embodiment, the centroid pixel coordinates of the marker spot can be obtained as the pixel coordinates of the marker point by using a spot detection algorithm. As an example, the spot detection algorithm can be implemented by the following formula:

[0090]

[0091] in, represents the blob detection algorithm, represents the detection result of the marker point of the Mth color in the nth frame tactile image, and l represents the number of marker points.

[0092] In step S703, based on the matching results of the pixel coordinates of the detected marking points and the marking point coordinate matrix corresponding to the monochrome channel image of the Mth color of the previous frame, the marking point coordinate matrix to be repaired corresponding to the monochrome channel image of the Mth color of the current frame is obtained.

[0093] In a complete approach-contact-separation process, the underwater tactile sensor 100 does not contact the target object at the beginning, and the contact module 101 remains in a relaxed state. Therefore, the pixel coordinates of the marker points detected in the first frame are almost consistent with the predetermined marker point coordinate matrix. However, due to the influence of noise or structural changes of the underwater tactile sensor 100, the pixel coordinates of some marker points may change.

[0094] As an exemplary embodiment, a structure-based nearest neighbor matching algorithm may be used to match the pixel coordinates of the detected marker points with the marker point coordinate matrix corresponding to the previous frame to obtain a marker point coordinate matrix corresponding to the pixel coordinates of the detected marker points.

[0095] Since the spatial ordering position of the marker points detected by the spot detection algorithm is unknown, it is necessary to roughly sort the marker points detected in the current frame based on the complete marker point seating table corresponding to the previous frame to make them have spatial order. The present disclosure designs a K nearest neighbor matching algorithm based on the relative position between marker points. It is assumed that there is a large displacement of the marker point cloud between the previous and next two frames of tactile images, but the shape of the point cloud changes very little. Therefore, the relative position between the marker points in the current frame should be similar to the relative position between the corresponding points in the previous frame.

[0096] As an exemplary embodiment, first, the centroid of the marker point cloud of the monochrome channel image of the current frame and the monochrome channel image of the previous frame can be calculated and normalized to obtain the relative position of each marker point relative to the centroid. Next, for each marker point in the normalized marker point cloud of the monochrome channel image of the previous frame (i.e., the target marker point), the K nearest neighbor matching algorithm is applied to find the marker point closest to the target marker point in the normalized marker point cloud of the monochrome channel image of the current frame as the matching point. Finally, if the matching point exceeds the tactile perception range of the target marker point, the matching point is deleted. If the matching point does not exceed the tactile perception range of the target marker point, the pixel coordinates before normalization corresponding to the matching point are added to the position corresponding to the target marker point in the marker point coordinate matrix corresponding to the monochrome channel image of the current frame, thereby obtaining the marker point coordinate matrix to be repaired corresponding to the monochrome channel image of the current frame. Specifically, due to physical factors and algorithm limitations, the marker point coordinate matrix corresponding to the monochrome channel image of the current frame may be incomplete (i.e., some positions of the matrix may lack corresponding matching points because the corresponding matching points are beyond the tactile perception range), which means that it needs to be repaired.

[0097] As an example, the coordinate matrix of the marker points to be repaired corresponding to the monochrome channel image of the Mth color of the current frame can be obtained by the following formula:

[0098]

[0099] in, represents the normalized detection result of the marker point of the Mth color in the nth frame tactile image, M(·) represents the mean function, l n represents the number of marked points in the nth frame of the tactile image. Similarly, represents the normalized restoration result of the marker point of the Mth color in the n-1th frame of the tactile image. Represents the coordinate matrix of the marker points corresponding to the monochrome channel image of the Mth color of the n-1th frame of tactile image, l n-1 Indicates the number of marked points in the n-1th frame of the tactile image. i indicates the element index value of the corresponding matrix. Indicates the K nearest neighbor matching algorithm that only selects the nearest point. Represents the matching point of the Mth color in the nth frame tactile image. U represents the coordinate matrix of the marker points to be repaired corresponding to the monochrome channel image of the Mth color in the nth frame of the tactile image, L2(·) represents the Euclidean distance function, and d represents the length of the tactile perception range. n represents the coordinate matrix of the marker points to be repaired corresponding to the nth frame of the tactile image, and h represents the number of marker points of all colors.

[0100] Figure 8 A flowchart of a method for repairing a marker point coordinate matrix to be repaired corresponding to a current frame according to an exemplary embodiment of the present disclosure is shown.

[0101] Reference Figure 8 In step S801, for the missing marker point in the marker point coordinate matrix to be repaired corresponding to the current frame, the potential pixel coordinates of the missing marker point are determined based on the pixel coordinates of other marker points in the marker point group where the missing marker point is located.

[0102] The missing marking point is the marking point corresponding to the element whose content is missing in the coordinate matrix of the marking point to be repaired.

[0103] In step S802, based on the potential pixel coordinates, candidate marker points of the missing marker points are searched from the current frame.

[0104] In step S803, pixel coordinates are selected from the pixel coordinates of the candidate marker points and the potential pixel coordinates as the pixel coordinates of the missing marker point.

[0105] In step S804, the pixel coordinates of the missing marker points are added to the marker point coordinate matrix to be repaired corresponding to the current frame to obtain a marker point coordinate matrix with no missing elements (ie, a repaired marker point coordinate matrix).

[0106] Due to physical factors and algorithm limitations, the marker point coordinate matrix obtained in step S603 may be incomplete. On the one hand, due to occlusion during vibration or changes in lighting conditions, some marker points in the multi-color marker point field may not be recognized. On the other hand, the K nearest neighbor matching algorithm only matches the nearest marker point, and the distance between the matching point and the target marker point cannot exceed the tactile perception range of the target marker point, which makes the marker points in the rapidly deformed area may be mistaken for lost points due to being deleted outside the tactile perception range. In addition, in the matching step, the matching behavior is made based on the assumption that the relative position changes in the marker point cloud between adjacent frames are small. However, in tactile perception, some sharp object contacts may cause significant deformation of the contact module 101, resulting in significant changes in the relative positions of marker points in steep positions and failure to match. Therefore, the present disclosure designs a repair algorithm to complete the missing marker points.

[0107] As an exemplary embodiment, first, the parallelogram structure of each marker point group in the multi-color marker point field can be used to calculate the potential pixel coordinates of the missing marker point based on the marker points of other colors in the same marker point group. As an example, if the marker points of other colors in the marker point group where the missing marker point is located in the current frame are not missing, the pixel coordinates of the marker points of other colors in the marker point group in the current frame are directly used for calculation, otherwise, the pixel coordinates of the marker points of other colors in the marker point group in the previous frame are used for calculation. Then, the K nearest neighbor matching algorithm can be used to search for multiple neighboring marker points of the potential pixel coordinate in the current frame, and only the neighboring marker points within the tactile perception range of the potential pixel coordinate are regarded as candidate marker points, and the candidate marker points closer to the potential pixel coordinate have a higher trust value, thereby obtaining a candidate marker point set of the missing marker point. If there are no neighboring marker points within the tactile perception range of the potential pixel coordinate, the potential pixel coordinate itself is regarded as the only candidate pixel coordinate of the missing marker point, and a trust value is assigned, which can be set by a hyperparameter. If a marker point is lost in multiple consecutive frames, the trust value in its candidate marker point set will be updated using the update formula based on the trust value corresponding to the marker point in the previous frame. Finally, the pixel coordinates with the highest trust value are selected from the pixel coordinates of the candidate marker points and the potential pixel coordinates as the pixel coordinates of the missing marker point and added to the marker point coordinate matrix to be repaired corresponding to the current frame to complete the filling of the missing marker point (that is, the missing element in the matrix).

[0108] As an example, the repair calculation process can be implemented as follows:

[0109]

[0110] Among them, P miss represents the potential pixel coordinates of the missing markers, P R ,PG ,P B ,P K Indicates the pixel coordinates of other markers (red, green, blue, and black markers) in the marker group where the missing marker is located, P M The pixel coordinates of the marker point representing the Mth color, represents the detection results of red, green, blue, and black markers in the nth frame of the tactile image. The coordinate matrix of the marking points corresponding to the monochrome channel image of the Mth color of the n-1th frame of the tactile image, The coordinate matrix of the marking points to be repaired corresponding to the monochrome channel image of the Mth color of the nth frame tactile image. Represents the new matrix element index value obtained by dividing the matrix element index value by 4 and rounding it to the integer.

[0111] As an example, the trust value can be updated as follows:

[0112]

[0113] Among them, C cand represents the trust value of the candidate marker point, P cand represents the pixel coordinates of the candidate marker point, L2(·) represents the Euclidean distance function, and η is an adjustable fixed parameter. represents the trust value of the candidate marker point of the continuously missing marker point in the tactile image of the nth frame, Indicates the trust value of a candidate marker point in the n-1th frame of the tactile image when the marker points are continuously missing. It represents the maximum value of the trust values ​​of all candidate marker points when obtaining the continuously missing marker points in the n-1th frame tactile image.

[0114] The repair algorithm proposed in the present disclosure utilizes the information in the current frame to restore the missing marker points as much as possible, rather than relying entirely on the marker point information in the previous frame. In addition, this repair process is iteratively enhanced over time. As the missing marker points cannot be matched in multiple frames, the trust values ​​of their candidate marker points gradually decrease, while the trust values ​​of the calculated potential pixel coordinates remain unchanged. Therefore, the repair algorithm tends to select the calculated potential pixel coordinates. Considering that the situation of missing marker points corresponds to the situation where the contact module 101 undergoes significant deformation in a real experiment, this state will not last too long. When the contact module 101 is restored to a relatively flat state, the potential pixel coordinates calculated for the missing marker points are more likely to accurately represent the true position of the missing marker points. Therefore, the repair algorithm can effectively solve the problem of missing marker points caused by significant deformation and frame loss.

[0115] Fig. 9A flow chart showing a method of updating a tactile map of a surface of an underwater object according to an exemplary embodiment of the present disclosure.

[0116] Reference Fig. 9 In step S901, based on the three-dimensional tactile point cloud corresponding to the multiple tactile image frames (i.e., the tactile image sequence captured in a complete approach-contact-separation process), a tactile image set corresponding to the current complete contact process between the contact module 101 and the underwater object is screened out from the multiple tactile image frames. The tactile image set corresponding to the current complete contact process can be understood as: the visual images captured in the entire process from the beginning of the contact with the underwater object (i.e., the target object) to the end of the contact with the underwater object.

[0117] As an exemplary embodiment, step S901 may include: for each tactile image frame in the multiple tactile images, based on the three-dimensional tactile point cloud corresponding to the frame tactile image and the three-dimensional tactile point cloud corresponding to the first frame in the multiple tactile images, determining the tactile stimulation intensity received by the underwater tactile sensor 100 when the frame tactile image is captured, if the tactile stimulation intensity corresponding to the frame tactile image exceeds a preset threshold, adding the frame tactile image to the tactile image set. The tactile stimulation intensity is positively correlated with the deformation degree of the contact module 101.

[0118] In order to eliminate the noise interference of the tactile point cloud and prevent misidentification, the present disclosure designs the following contact judgment algorithm. Inspired by the human body pain intensity model, the tactile stimulation intensity is defined to evaluate the tactile stimulation intensity received by the underwater tactile sensor 100. The intensity is proportional to the deformation degree of the contact module 101, but not linear. As an example, the tactile stimulation intensity function definition of the underwater tactile sensor 100 can be as follows:

[0119]

[0120] Among them, I st represents the intensity of tactile stimulation, k st P1,P represents the proportional parameter for adjusting the stimulus sensitivity, and α represents the exponential parameter for controlling the tactile stimulus intensity. c represents the initial point cloud (i.e., the 3D tactile point cloud corresponding to the first frame) and the current point cloud (i.e., the 3D tactile point cloud corresponding to the current frame), Represents the mean of the initial point cloud and the mean of the current point cloud, S c represents the gel layer deformation, ρ c represents the correlation between the current point cloud and the initial point cloud, and L2(·) represents the Euclidean distance function.

[0121] As an example, a tactile stimulation threshold (ie, a preset threshold) may be defined to determine whether contact has occurred, and the formula is as follows:

[0122]

[0123] Among them, E contact Indicates whether contact occurs, I th represents the tactile stimulation threshold.

[0124] According to the above formula, the exemplary embodiment of the present disclosure is more inclined to regard the change of the point cloud shape as contact, while the simple jitter of the marker point will result in ρ c is close to 1, resulting in a lower score for tactile stimulation intensity.

[0125] In step S902 , a key frame corresponding to the most significant deformation of the three-dimensional tactile point cloud is selected from the tactile image set.

[0126] In step S903 , the three-dimensional tactile point cloud corresponding to the key frame is integrated into the tactile map of the surface of the underwater object (ie, the target object) to obtain the shape and texture of the underwater object.

[0127] In each contact between the underwater tactile sensor 100 and the object, the frame with the most significant point cloud deformation is selected as the key frame for updating the tactile map. This selection is based on the fact that the moment when the point cloud deformation is the most significant represents the moment when the underwater tactile sensor 100 and the object are in the most complete contact.

[0128] Due to the size limitation of the marker layer of the underwater tactile sensor 100, the three-dimensional tactile point cloud obtained from the key frame can only reflect a small area on the surface of the target object. Therefore, the present disclosure also proposes a remote-operated tactile perception framework for a remote-operated underwater robot carrying an underwater tactile sensor to touch the target object multiple times, and then splices the three-dimensional tactile point cloud corresponding to the key frame in the tactile image sequence corresponding to each touch process (i.e., a complete approach-contact-separation process) to obtain a complete tactile impression of the target object. These spliced ​​three-dimensional tactile point cloud sets are tactile maps, which contain texture and shape information of the target object.

[0129] Fig.10 An example of an underwater tactile sensing method according to an exemplary embodiment of the present disclosure is shown.

[0130] According to the exemplary embodiment of the present disclosure, the marker point tracking algorithm in the underwater tactile perception method may include four steps: initialization, segmentation, matching, and repair to achieve the identification and tracking of marker points in a vibration environment. n represents the nth frame in the tactile image sequence captured by the underwater tactile sensor 100 during a complete approach-contact-separation process of the target object, t n Indicates that the nth frame F is captured n moment.

[0131] Initialization: For the tactile image F0 outputted when the underwater tactile sensor 100 is initialized, the pixel coordinates of each marking point therein are directly obtained by using prior knowledge and numbered as a predetermined marking point coordinate matrix.

[0132] Segmentation: For the tactile image output by the underwater tactile sensor 100, obtain the current frame F n , using the color segmentation algorithm based on HSV channel to obtain the tactile image F n The monochrome channel images corresponding to the red, green, blue and black channels are obtained, and then the centroid pixel coordinates of the marked point spots on each monochrome channel image are obtained through the spot detection algorithm as the pixel coordinates of the detected marked points.

[0133] Matching: For the pixel coordinates of the red, green, blue, and black channel markers obtained in the segmentation step, the current frame F is matched based on the K nearest neighbor matching algorithm. n The pixel coordinates of the marker points are the same as those of the previous frame F n-1 The pixel coordinates of the marker points are matched, and the unmatched marker points are regarded as missing marker points to obtain the incomplete seating table U to be repaired. n-1 .

[0134] Fix: Incomplete seating chart obtained for matching step n-1 According to the prior structure information of the multi-color marker field, the pixel coordinates of the missing markers are supplemented by the pixel coordinates of the intact markers according to the parallelogram rule to obtain the restored complete seating table T n .

[0135] Repeat the above steps to get the marked point seating table corresponding to each frame.

[0136] According to the surface reconstruction algorithm in the underwater tactile perception method of the exemplary embodiment of the present disclosure, the triangulation method can be used to obtain the coordinates of the space marker points using the correspondence relationship between the left and right target marker points to form a three-dimensional tactile point cloud. Specifically, the surface reconstruction algorithm uses the left and right target seating table of the current frame obtained by the marker tracking algorithm, obtains the correspondence relationship between the left and right target points according to the seating table, and then uses the triangulation method to obtain the spatial coordinates of the marker points to obtain the three-dimensional tactile point cloud.

[0137] Fig.11 A structural block diagram of an underwater robot according to an exemplary embodiment of the present disclosure is shown.

[0138] Reference Fig.11 According to an exemplary embodiment of the present disclosure, an underwater robot 1000 includes an underwater tactile sensor 100 , a robot arm 200 at the end of which the underwater tactile sensor 100 is installed, and a controller 300 .

[0139] As an exemplary embodiment, the controller 300 may be configured to execute the underwater tactile sensing method as described above.

[0140] It should be understood that the underwater robot 1000 may also include other devices for realizing its functions, and the present disclosure does not limit this.

[0141] Fig.12 A schematic structural diagram of a teleoperation tactile perception platform according to an exemplary embodiment of the present disclosure is shown.

[0142] like Fig.12 As shown, the remote-operated tactile perception platform according to an exemplary embodiment of the present disclosure includes: an underwater robot 1000 equipped with a 6-DOF robotic arm, two force-touch interaction devices, a host computer, and an underwater tactile sensor 100 installed at the end of the robotic arm.

[0143] As an exemplary embodiment, the underwater robot 1000 may also be equipped with four cameras to provide a wide field of view for the robot arm operation.

[0144] As an exemplary embodiment, two electromagnetic adsorption devices may be installed on the bottom of the underwater robot 1000 as attachment devices, and may also be installed on the front bracket to attach to the front plane.

[0145] As an exemplary embodiment, the force touch interaction device is used to control the movement of the underwater robot 1000 body and the robotic arm, and provide torque feedback to the operator so that the operator can sense the tactile force during operation. As an example, the force touch interaction device can be divided into a left-hand device and a right-hand device, which are used to control the position of the underwater robot 1000 body and the robotic arm, respectively. For the underwater robot 1000 body, motion control can be achieved by mapping the end effector coordinates of the left-hand force touch interaction device to the thrust torque of the body; for the underwater robotic arm, a spring-damper system can be used to map the end effector coordinates of the right-hand force touch interaction device to the coordinates of the robotic arm, while filtering hand tremors to enhance the user experience.

[0146] As an exemplary embodiment, the underwater tactile sensor 100 may be connected to the controller 300 of the underwater robot 1000 via a USB, the underwater tactile sensor 100 transmits a tactile image to the controller 300, and the controller 300 may generate a three-dimensional tactile point cloud in real time based on the received tactile image.

[0147] As an exemplary embodiment, a teleoperated tactile sensing platform may communicate using ROS.

[0148] Fig.13 An example of a teleoperation tactile perception framework according to an exemplary embodiment of the present disclosure is shown.

[0149] like Fig.13As shown, the teleoperation tactile perception framework according to the exemplary embodiment of the present disclosure can use force touch interaction to teleoperate and control the underwater robot 1000 carrying the underwater tactile sensor 100 to perceive objects, so as to realize the shape perception and texture perception of the object using the underwater robot 1000.

[0150] The operator can control the underwater robot 1000 carrying the underwater tactile sensor 100 to contact the underwater object through the remote control tactile perception framework, and output the three-dimensional point cloud of the contacted object in real time through the three-dimensional tactile point cloud generation algorithm to complete the perception of the shape and texture of the underwater object.

[0151] When the underwater tactile sensor 100 contacts an underwater object, the deformation of its contact module 101 will cause the distribution of the three-dimensional tactile point cloud to change. Using the changes in these point clouds, the tactile feedback force generated by the tactile force neural network will act on the operator's hand through the force touch interaction device, allowing the operator to sense the contact situation; at the same time, the three-dimensional tactile point cloud of the contact area is used to update the entire tactile map. As an example, the three-dimensional tactile point cloud can be mapped to the contact force through the neural network, and then scaled to a certain ratio as the feedback torque. The tactile feedback force can be calculated by the following formula:

[0152] F fb =k fb ·F net

[0153] F net represents the output of the neural network, k fb represents the coefficient parameter, F fb Represents the tactile feedback force applied to the operator.

[0154] The remote-operated tactile perception framework can remotely control the underwater robot 1000 carrying the underwater tactile sensor 100 to contact the target object through force-touch interaction, and splice the three-dimensional tactile point cloud of each contact to obtain the complete shape and texture of the underwater object.

[0155] As an exemplary embodiment, the present disclosure can stitch together three-dimensional point clouds from different positions according to the coordinates of the end effector in the robot arm coordinate system to obtain a tactile map. As an example, the three-dimensional point cloud corresponding to the key frame can be integrated into the tactile map through coordinate transformation.

[0156] According to an exemplary embodiment of the present disclosure, an electronic device is also provided, which includes: at least one memory and at least one processor, wherein a set of computer executable instructions is stored in the at least one memory, and when the set of computer executable instructions is executed by the at least one processor, the underwater tactile perception method as described in the above exemplary embodiment is executed.

[0157] As an example, the electronic device may be the controller 300 of the underwater robot 1000 .

[0158] As an example, the electronic device can be a host computer, such as a PC, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above-mentioned instruction set. Here, the electronic device is not necessarily a single electronic device, but can also be any device or circuit that can execute the above-mentioned instructions (or instruction sets) individually or jointly. The electronic device can also be part of an integrated control system or a system manager, or can be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0159] In electronic devices, the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0160] The processor may execute instructions or codes stored in the memory, wherein the memory 600 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0161] The memory may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor or the like. In addition, the memory may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the memory.

[0162] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0163] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to perform the underwater tactile perception method as described in the above exemplary embodiment. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0164] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by at least one processor to implement the underwater tactile perception method as described in the above exemplary embodiment.

[0165] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0166] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An underwater tactile sensor, characterized in that: include: A contact module, used for contacting the surface of an underwater object to cause deformation, wherein the deformation causes a multi-color marking point field on the upper surface of the contact module to be displaced; A binocular camera module, used for capturing the displacement of the multi-color marking point field to obtain a tactile image, wherein the tactile image includes a left-eye tactile image and a right-eye tactile image; A waterproof module, used to realize the waterproof function of the underwater tactile sensor; The multi-color marking point field includes: a plurality of marking point groups arranged in sequence, each marking point group includes j marking points of different colors, the spatial ordering of the marking points of j colors in each marking point group is fixed, and j is an integer greater than 1.

2. The underwater tactile sensor according to claim 1, characterized in that: Also includes: A pressure balance chamber, which is a chamber used to make the lower surface of the contact module contact with the external water body, so as to achieve water pressure balance between the upper and lower surfaces of the contact module; And / or, a light chamber module, used to provide lighting conditions for the binocular camera module to capture the displacement of the multi-color marking point field.

3. An underwater tactile perception method, characterized in that: include: Acquire multiple frames of tactile images from the underwater tactile sensor as claimed in claim 1 or 2; Taking each frame of the tactile image as the current frame, respectively, generating a marker point coordinate matrix corresponding to the current frame, wherein each element in the marker point coordinate matrix corresponds one-to-one to each marker point in the multi-color marker point field, and the content of each element is the pixel coordinate of the marker point corresponding to the element in the current frame; Based on the marker point coordinate matrix corresponding to the current frame, determine the three-dimensional spatial coordinates of each marker point in the multi-color marker point field to obtain a three-dimensional tactile point cloud corresponding to the current frame; The marker point coordinate matrix corresponding to the current frame includes: a marker point coordinate matrix corresponding to the left-eye tactile image and a marker point coordinate matrix corresponding to the right-eye tactile image in the current frame.

4. The underwater tactile perception method according to claim 3, characterized in that: The step of generating a marker point coordinate matrix corresponding to the current frame comprises: The current frame is divided into j monochrome channel images corresponding to j color channels one by one; Based on the marker point coordinate matrix corresponding to the previous frame of the current frame, determining the marker point coordinate matrix to be repaired corresponding to each monochrome channel image of the current frame; Merging the coordinate matrices of the marking points to be repaired corresponding to the monochrome channel images to obtain the coordinate matrix of the marking points to be repaired corresponding to the current frame; The coordinate matrix of the marking points to be repaired corresponding to the current frame is repaired to obtain the coordinate matrix of the marking points corresponding to the current frame.

5. The underwater tactile perception method according to claim 4, characterized in that: The step of determining the marker point coordinate matrix to be repaired corresponding to each monochrome channel image of the current frame based on the marker point coordinate matrix corresponding to the previous frame of the current frame comprises: Based on the marker point coordinate matrix corresponding to the previous frame, determine the marker point coordinate matrix corresponding to each monochrome channel image of the previous frame; Obtaining pixel coordinates of a marking point detected from the monochrome channel image of the Mth color of the current frame by a spot detection algorithm; Based on the matching result of the pixel coordinates of the detected marking points and the marking point coordinate matrix corresponding to the monochrome channel image of the Mth color of the previous frame, the marking point coordinate matrix to be repaired corresponding to the monochrome channel image of the Mth color of the current frame is obtained; Wherein, M is an integer greater than 0 and less than or equal to j.

6. The underwater tactile perception method according to claim 4, characterized in that: The step of repairing the coordinate matrix of the marker points to be repaired corresponding to the current frame to obtain the coordinate matrix of the marker points corresponding to the current frame comprises: For a missing marker point in the marker point coordinate matrix to be repaired corresponding to the current frame, determine potential pixel coordinates of the missing marker point based on pixel coordinates of other marker points in the marker point group where the missing marker point is located; Based on the potential pixel coordinates, searching for candidate marker points of the missing marker point from the current frame; Selecting pixel coordinates from the pixel coordinates of the candidate marker points and the potential pixel coordinates as the pixel coordinates of the missing marker point; The pixel coordinates of the missing marker points are added to the coordinate matrix of the marker points to be repaired corresponding to the current frame.

7. The underwater tactile perception method according to claim 3, characterized in that: Also includes: Based on the three-dimensional tactile point clouds corresponding to the multiple frames of tactile images, a set of tactile images corresponding to the current complete contact process between the contact module and the underwater object is screened out from the multiple frames of tactile images; Selecting a key frame with the most significant deformation of the corresponding three-dimensional tactile point cloud from the tactile image set; The three-dimensional tactile point cloud corresponding to the key frame is integrated into the tactile map of the surface of the underwater object to obtain the shape and texture of the underwater object.

8. The underwater tactile perception method according to claim 7, characterized in that: The step of selecting a set of tactile images corresponding to the complete contact process between the contact module and the underwater object from the multiple frames of tactile images based on the three-dimensional tactile point clouds corresponding to the multiple frames of tactile images comprises: For each frame of tactile image, based on the three-dimensional tactile point cloud corresponding to the frame of tactile image and the three-dimensional tactile point cloud corresponding to the first frame of the multiple frames of tactile image, determine the tactile stimulation intensity received by the underwater tactile sensor when the frame of tactile image is captured, wherein the tactile stimulation intensity is positively correlated with the deformation degree of the contact module; If the tactile stimulation intensity corresponding to the frame of tactile image exceeds a preset threshold, the frame of tactile image is added to the tactile image set.

9. An underwater robot, characterized in that: include: The underwater tactile sensor as claimed in claim 1 or 2; A mechanical arm with the underwater tactile sensor installed at the end; Controller.

10. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by a processor of an electronic device, the electronic device is enabled to execute the underwater tactile perception method according to any one of claims 3 to 8.

11. An electronic device, characterized in that: The electronic device comprises: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the underwater tactile perception method according to any one of claims 3 to 8.

12. A computer program product comprising computer executable instructions, characterized in that: When the computer executable instructions are executed by at least one processor, the underwater tactile perception method according to any one of claims 3 to 8 is implemented.

Images