Binocular visual tactile sensor based on color block mark point array and real-time high-precision three-dimensional reconstruction method

Abstract

The invention relates to the technical field of robot tactile perception, aims to solve the technical problems that an existing visual tactile sensor is low in precision, poor in real-time performance, incapable of adapting to large deformation and the like during three-dimensional reconstruction, and particularly relates to a binocular visual tactile sensor based on a color block mark point array and a real-time high-precision three-dimensional reconstruction method. According to the method, high-precision three-dimensional reconstruction is completed based on a binocular visual tactile sensor, image preprocessing, color block mark point identification, center coordinate calculation and color discrimination are included, the position of a color block in a mark point array is determined based on coding information and mark point tracking, matching is completed, and finally point cloud is calculated to achieve three-dimensional reconstruction. According to the method, the time spent on part matching is less than 10ms, 400 pieces / cm < 2 > dense point cloud data can be obtained, the method can adapt to the loss of mark points under the condition of large deformation, and the method has an important value for improving the performance and safety of a robot in various complex tasks.

Description

Technical Field

[0001] The present invention relates to the field of robot tactile perception technology, and in particular to a binocular tactile sensor based on a color block marker array and a real-time high-precision three-dimensional reconstruction method. Background Art

[0002] Touch is one of the key senses for humans to perceive the world and interact with the environment, helping humans complete various delicate tasks. To simulate the tactile mechanism, researchers usually use electronic tactile sensors, such as resistive sensors, capacitive sensors, and fiber sensors. However, these sensors have limitations in tactile feedback and high-resolution tactile information, which limits the perception and manipulation performance of robots.

[0003] Since images contain a large amount of information that is easy to process automatically, vision-based tactile sensors can improve the quality of tactile information. Visual tactile sensors have been developed to perform a variety of functions, including texture recognition, multi-dimensional force detection, friction coefficient measurement, slip detection, hardness measurement, 3D reconstruction, and temperature detection. They are also used in a variety of fields, including robotics, industrial inspection, and human-computer interaction.

[0004] Obtaining accurate and detailed three-dimensional morphology of the contacted object can help the robot adjust the grasping posture and strength when performing the operation task. However, the density of the marker is a key determinant of the reconstruction accuracy of the visual tactile sensor. The marker density of existing visual tactile sensors is usually low, about 90 per square centimeter. This is mainly because when the elastomer with high-density markers undergoes large deformation when in contact with the object, the markers are easily lost. In addition, most current marker matching algorithms usually assume that the number of markers per frame is consistent, which poses a huge challenge to processing high-density and uncoded markers, making it difficult to achieve real-time high-precision three-dimensional reconstruction. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a binocular visual-tactile sensor based on a color block marker point array and a real-time high-precision three-dimensional reconstruction method, which solves the technical problems of low accuracy, poor real-time performance and inability to adapt to large deformation in three-dimensional reconstruction of existing visual-tactile sensors.

[0006] In order to solve the above technical problems, the present invention provides the following technical solution: a real-time high-precision three-dimensional reconstruction method based on a color block marker array, the method comprising the following steps:

[0007] S1. Selecting at least N color blocks with obviously different colors as color block marking points, and constructing a color block marking point array for realizing real-time high-precision three-dimensional reconstruction of a binocular tactile sensor based on the color block marking points;

[0008] S2, using the calibration image to calibrate and correct the binocular vision camera in the image acquisition module;

[0009] S3, calculate the coding information corresponding to each color block at each position in the color block marker array, and store it in the corresponding position of the matrix Cells Codes Ground Truth, the coding information includes 1 string of standard codes and 4 strings of rotary codes;

[0010] S4, obtaining the first left and right images when the binocular tactile sensor is not in contact with the object, and taking the pixel coordinates of the center of each color block in the first left and right images and the position of the corresponding color block in the color block mark point array as the previous frame image of the first frame;

[0011] S5, using a binocular vision camera to obtain the second left and right images of the current frame, and assigning an index to each color block detected in the current frame corresponding to the second left and right images according to the color block marker array;

[0012] S6, using the initial information and the coded information of the color block in the color block marker array in the previous frame image, jointly determine the position of the color block in the color block marker array in the current frame image and add an index, and store it in the statistical matrix statistics;

[0013] S7, traverse each position in the statistics matrix statistics, find the index with the most occurrences in each position as the color block at the current position, and store it in the matrix Cells Codes Cur;

[0014] S8, traverse the matrix Cells Codes Cur corresponding to the second left and right images, and calculate the disparity d according to the two color blocks located at the same position p And the three-dimensional coordinates of the center point of the color block in space are obtained by triangulation to generate point cloud data;

[0015] S9. After preprocessing the generated point cloud data, calculate its normal, and perform Poisson reconstruction based on the normal to generate a smooth three-dimensional surface.

[0016] Furthermore, in step S1, the process of constructing the color block marking point array includes:

[0017] S11. Define the design requirements for the color block marker array, including:

[0018] Choose a color block at random in the rectangular digital array, and take it as the center. The 12 adjacent color blocks will not have the same color as the center color block.

[0019] The Hamming distance between the standard code corresponding to each color block and the four rotary codes of the standard code and the five strings of codes of other color blocks in an expanded window centered on the current color block is greater than or equal to 2. At the same time, the Hamming distance between the standard code corresponding to each color block and the four strings of rotary codes is also greater than or equal to 2.

[0020] S12, defining that each color block is represented by a number, and that a plurality of color blocks together form a rectangular digital array of size n×n;

[0021] S13, randomly select a color block at a non-edge position in the rectangular digital array, and combine the eight most adjacent color blocks around it, starting from the rightmost color block, and write the digital representations of the eight most adjacent color blocks in a counterclockwise direction into a string of 8-bit numbers as the standard code of the color block;

[0022] The standard code is shifted forward by one bit, backward by one bit, forward by two bits, and backward by two bits to form four kinds of rotation codes of the standard code;

[0023] S14, randomly filling the rectangular digital array with the selected N colors;

[0024] S15, optimizing the rectangular digital array filled with random colors to obtain a color block marking point array, specifically:

[0025] Traverse the rectangular digital array from top to bottom and from left to right, calculate the 1-string standard code and 4-string rotation code of each color block at the non-edge position, and store the 5-string code of each color block in the corresponding position of the matrix Cells Codes Ground Truth according to the position of the color block in the rectangular digital array;

[0026] Traverse the positions in the Cells Codes Ground Truth from top to bottom and from left to right. With the current traversal position as the center, calculate the Hamming distance between all standard codes in the 13*13 window and the 5 strings of codes of the current traversal position. If the Hamming distance is less than 2, randomly select one of the eight neighboring positions around the current traversal position as the modification position, change the color of the modified position in the rectangular digital array, and ensure that the changed color is different from the 12 neighboring positions, and update the Cells Codes Ground Truth at the same time; if a color different from the 12 neighboring positions cannot be found, do not change the color.

[0027] After traversing the Cells Codes Ground Truth, traverse the Cells Codes Ground Truth again and count the number of times the Hamming distance is less than 2. If the number is not equal to 0, repeat the optimization step S14.

[0028] Furthermore, in step S14, specifically:

[0029] Fill the first row of a rectangular digital array: Given different colors for the first three color blocks in the first row, generate random colors for subsequent color blocks using uniform distribution. If the generated color block has the same color as one of the first three color blocks, regenerate it until the color is different.

[0030] Fill the first column of a rectangular digital array: Given different colors for the first three blocks in the first column, and the colors of these three blocks are different and different from the first three blocks in the first row, the subsequent blocks are generated with random colors using uniform distribution. If the generated color is the same as one of the first three blocks in the first column, it is regenerated until the color is different;

[0031] Fill the remaining positions in the rectangular digital array: given the color of the color block in the second row and second column, which is different from the colors of the first three color blocks in the first row and first column, traverse each position from top to bottom and from left to right: if it is in the second column, ensure that its color is different from the colors of the six adjacent color blocks: top-top, top-left, top, top-right, left, and bottom-left; other positions must be controlled to be different from the six adjacent color blocks: top-top, top-left, top, top-right, left-left, and left.

[0032] The technical solution also provides a binocular visual tactile sensor, including a supporting structure module with a sleeve as a visual tactile sensor frame, and an information processing module that establishes a communication connection with the binocular visual tactile sensor, wherein the supporting structure module is respectively provided with a contact module and an image acquisition module located at the front and rear ends thereof, and an illumination module used in conjunction with the contact module and the image acquisition module;

[0033] The contact module includes a support plate located on the front end of the sleeve, an elastomer is stacked on the support plate, a marking layer, a reflective layer and a protective layer are stacked in sequence on the elastomer, the elastomer is detachably mounted on the sleeve through a fixed cover plate, and a window is provided on the fixed cover plate to extend the contact part of the elastomer to the outside thereof.

[0034] Furthermore, the lighting module adopts a programmable control LED light bar including brightness and color, and the LED light bar is arranged around the periphery of the elastic body to form a total internal reflection light path.

[0035] Furthermore, the image acquisition module uses two miniature camera modules arranged at a fixed baseline distance to form a binocular vision camera, and also includes a base fixing part detachably connected to the rear end of the sleeve, the miniature camera module is located in an installation slot on the base fixing part, and the miniature camera module is detachably fixed to the base fixing part through a camera fixing plate.

[0036] Furthermore, the elastomer is a composite silicone structure composed of any one or more of silicone, TPU / TPR or latex, and the method for making the elastomer comprises the following steps:

[0037] S01. Put the mold into an ultrasonic cleaning machine and set the cleaning time to 2 to 3 minutes. After taking out the mold, blow it dry with a nitrogen gun, wipe it with a paper towel containing ethanol, and let it stand to dry.

[0038] S02. Prepare the first layer of silica gel: Pour 1.5±0.2g of each of the first silica gel A and B components into a container, and stir in the same direction for 3-5 minutes to fully mix the A and B solutions to obtain the first silica gel solution;

[0039] Place the fully mixed first silicone solution in a vacuum barrel, evacuate to -0.1Mpa, and keep it for a while. After all bubbles are completely burst, take it out and pour it into the cleaned mold. Place the mold horizontally and let it stand until the first silicone solution is naturally leveled. Then put it in a drying oven and heat it at a high temperature of more than 70℃ for 40 minutes.

[0040] S03. Making the second layer of silicone: Take the mold out of the drying oven, let it stand, and gently take out the upper mold when the mold temperature drops;

[0041] Take 7±0.5g of the second silica gel A and B components respectively and pour them into a container. Stir in the same direction for 3-5 minutes to fully mix the A and B solutions to obtain the second silica gel solution.

[0042] Place the fully mixed second silica gel solution in a vacuum barrel, evacuate to -0.1Mpa, and keep it for a while. After all bubbles are completely burst, take it out and pour it on the first layer of silica gel. Let it stand, put it in a drying oven, and heat it at a high temperature of more than 70℃.

[0043] S04. Prepare the third layer of silica gel: Pour 1.5±0.2g of each of the first silica gel A and B components into a container, and stir in the same direction for 3-5 minutes to fully mix the AB solutions to obtain the first silica gel solution;

[0044] Place the fully mixed first silica gel solution in a vacuum barrel, evacuate to -0.1Mpa, and keep it for a while. After all bubbles are completely burst, take it out, pour it on the second layer of silica gel, let it stand, put it in a drying oven, and heat it at a high temperature above 70℃ for 35-40 minutes.

[0045] Furthermore, the marking layer is a silicone layer with a marking pattern printed on the surface of the silicone, and the support plate is a 3 mm thick high-transparent acrylic plate.

[0046] Furthermore, the manufacturing method of the reflective layer is:

[0047] Use silver powder and the first type of silica gel, mix 1.5g of each in a 1:1 ratio, stir thoroughly and evacuate to obtain a silica gel silver powder mixture, place the prepared marking layer on a glue spreader, pour the silica gel silver powder mixture on the marking layer, and start at an initial speed of 700r / min, maintain for 30s, then 1100r / min, maintain for 15s, to obtain a reflective layer with a thickness of 0.3mm.

[0048] Furthermore, the manufacturing method of the protective layer is:

[0049] Use black pigment and the first type of silicone, mix them in a ratio of 1:5, stir thoroughly and evacuate to obtain a silicone black pigment mixture, place the marking layer with a reflective layer on a glue spreader, pour the silicone black pigment mixture on the reflective layer, and start with an initial speed of 700 r / min, maintain for 30 seconds, then 1100 r / min, maintain for 15 seconds, to obtain a protective layer with a thickness of 0.3 mm.

[0050] By means of the above technical solution, the present invention provides a binocular visual tactile sensor based on a color block marker array and a real-time high-precision three-dimensional reconstruction method, which has at least the following beneficial effects:

[0051] 1. The binocular vision-tactile sensor proposed in the present invention can realize real-time high-precision three-dimensional reconstruction to overcome the limitations of low precision and poor real-time performance of the existing technology and improve the performance and safety of robots in various complex tasks.

[0052] 2. The overall size of the color block marking point array designed by the present invention is 3cm*3cm, the color block spacing is only 0.5mm, the number of color blocks is as high as 3600, and the marking point density is as high as 400 / cm 2 , which can achieve high-precision three-dimensional reconstruction.

[0053] 3. The present invention introduces four kinds of rotation codes when matching marker points, which is equivalent to considering the situation where adjacent marker points rotate -90° to 90° around the central marker point, and can adapt to large deformation situations. At the same time, when large deformation occurs, there may be partial color block loss and color judgment errors. Only the adjacent 8 color block codes are used to complete the binocular matching, which will only affect the matching of the color blocks around this color block and will not affect the overall situation. It can realize 3D reconstruction of large contact deformation and has strong robustness. In addition, the tracking results are comprehensively considered during matching, so that the obtained matching results are more accurate and stable.

[0054] 4. The elastomer produced by the present invention has a suitable hardness value and optical transparency. While maintaining sufficient softness, it also eliminates the stickiness of the silicone itself, avoiding the phenomenon of the elastomer sticking to dust and being damaged when contacting an object, so that the elastomer can accurately and finely obtain the three-dimensional morphology of the contacting object.

[0055] 5. In the present invention, silver powder is added to the reflective layer. Under the lighting condition of total internal reflection, a strong brightness difference is formed between the pattern and the background area. The pattern separation can be completed using a traditional image processing algorithm. In addition, the time spent by the matching algorithm is generally about 10ms, so real-time three-dimensional reconstruction can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0057] Figure 1 1 is a structural diagram of a binocular tactile sensor in Embodiment 1 of the present invention;

[0058] Figure 2 This is an exploded structural diagram of the binocular tactile sensor in the first embodiment of the present invention;

[0059] Figure 3 This is a flow chart of making an elastic body in Example 1 of the present invention;

[0060] Figure 4 Schematic diagram of the positions of 12 adjacent color blocks in the second embodiment of the present invention;

[0061] Figure 5 This is a schematic diagram of a 5*5 marking point array with 8 colors in the second embodiment of the present invention;

[0062] Figure 6 This is a schematic diagram of filling the remaining positions in the second embodiment of the present invention;

[0063] Figure 7 A schematic diagram of the types of color blocks and the corresponding numbers in the second embodiment of the present invention;

[0064] Figure 8 This is a schematic diagram of a digital array corresponding to the color block marking point array in the second embodiment of the present invention;

[0065] Fig. 9 Schematic diagram of a pattern array corresponding to the color block marking point array in the second embodiment of the present invention;

[0066] Fig.10 is a flow chart of a three-dimensional reconstruction method in Embodiment 2 of the present invention;

[0067] Fig.11 This is a schematic diagram of the principle of obtaining three-dimensional coordinates using triangulation in the second embodiment of the present invention.

[0068] In the figure: 1. Fixed cover; 2. Protective layer; 3. Optical reflective layer; 4. Marking layer; 5. Elastomer; 6. Support plate; 7. LED light strip; 8. Sleeve; 9. Slotted cylindrical head screw M3×4; 10. Camera fixing plate; 11. Micro camera module; 12. Base fixing piece; 13. Slotted cylindrical head screw M2.5×8. DETAILED DESCRIPTION

[0069] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0070] Embodiment 1

[0071] Please refer to Figure 1-Figure 3 This embodiment proposes a binocular visual tactile sensor, including a support structure module with a sleeve 8 as a visual tactile sensor frame, which is printed by stereoscopic light curing 3D technology, and an information processing module that establishes a communication connection with the binocular visual tactile sensor. The information processing module uses a computer for image processing and data analysis. The support structure module is installed with a contact module and an image acquisition module located at the front and rear ends thereof, respectively, and a lighting module used in conjunction with the contact module and the image acquisition module. The contact module includes a support plate 6 located at the front end of the sleeve 8, an elastomer 5 is stacked on the support plate 6, and a marking layer 4, a reflection layer 3 and a protective layer 2 are stacked on the elastomer 5 in sequence. The elastomer 5 is detachably mounted on the sleeve 8 through a fixed cover plate 1, and the two are connected by four slotted cylindrical head screws M2.5×8, and a window is provided on the fixed cover plate 1 to extend the contact part of the elastomer 5 to the outside thereof.

[0072] In this embodiment, the contact module is the core part of the entire binocular tactile sensor, wherein the contact module comprises five parts: an elastic body 5 , a marking layer 4 , a reflection layer 3 , a protection layer 2 and a support plate 6 .

[0073] At present, the main materials for making the elastomer 5 are silicone, TPU / TPR and latex, and the most widely used material is silicone, which has better optical transparency than the other two materials. Currently, the most widely used silicones are Solaris, Silicones from Smooth-on and XP-565 from Ecoflex GEL, Inc., which are all highly transparent silicones. However, the Shore hardness of these silicones is about 15A, which cannot meet the requirement of being sufficiently soft in this embodiment.

[0074] At present, silicones with relatively low hardness values ​​and high optical transparency on the market will have very sticky characteristics, which will cause the elastomer 5 to easily stick to dust and break. A single layer of silicone is difficult to meet the requirements of making elastomers in this embodiment. This embodiment selects a composite silicone structure to achieve the integration of various silicone properties. Therefore, this embodiment uses two types of silicone. The first is Caster's silicone potting glue HL-1010-2, with a Shore hardness of 5A, high transparency, and a non-sticky surface. The second silicone is Shenzhen Yuezhen Technology Co., Ltd. YZ-9400, a Shore hardness of 0A, high transparency, and a sticky surface. The mold for making composite silicone is made of aluminum alloy, processed by a high-precision milling machine, and then polished to obtain the final product.

[0075] As a further implementation method of this embodiment, Figure 3 As shown, the elastomer 5 is a composite silicone structure composed of two different silicones, and the manufacturing method of the elastomer 5 includes the following steps:

[0076] S01. Place the mold in an ultrasonic cleaning machine and set the cleaning time to 2 to 3 minutes. Take out the mold and blow it dry with a nitrogen gun. Use a paper towel containing ethanol to wipe off stains such as silicone residues and let it dry to ensure that there are no impurities on the surface of the elastomer.

[0077] S02. Make the first layer of silicone: prepare the first silicone solution, take 1.5±0.2g of the first silicone A and B components, pour into a disposable plastic cup, and stir in the same direction with a glass rod for 3 to 5 minutes. At the same time, pay attention to scraping the sides and bottom of the plastic cup to ensure that the AB solutions are fully mixed. Place the fully mixed first silicone solution in a vacuum barrel, evacuate to -0.1Mpa, and keep it for a while. After all bubbles are completely burst, take it out and pour it into the cleaned mold. Place the mold on a horizontal laboratory table and let it stand for a while. After the silicone solution is naturally leveled, put it in a drying oven and heat it at a high temperature of more than 70°C for 40 minutes.

[0078] S03. Make the second layer of silicone: Take the mold out of the drying oven, let it stand, and gently take out the upper mold when the temperature of the mold drops. Prepare the second silicone solution, take 7±0.5g of the second silicone A and B components, pour them into a disposable plastic cup, and stir them in the same direction with a glass rod for 3 to 5 minutes. At the same time, pay attention to scraping the sides and bottom of the plastic cup to ensure that the AB solutions are fully mixed. Place the fully mixed second silicone solution in a vacuum barrel, evacuate to -0.1Mpa, and keep it for a while. After all bubbles are completely burst, take it out, pour it on the first layer of silicone, let it stand, put it in a drying oven, and heat it at a high temperature of more than 70°C for 30 minutes.

[0079] S04. Make the third layer of silica gel: prepare the first silica gel solution, take 1.5±0.2g of the first silica gel A and B components respectively, pour into a disposable plastic cup, stir with a glass rod in the same direction for 3-5 minutes, and scrape the sides and bottom of the plastic cup to ensure that the AB solutions are fully mixed. Place the fully mixed first silica gel solution in a vacuum barrel, evacuate to -0.1Mpa, and keep it for a while, take it out after all bubbles are completely burst, pour it on the second layer of silica gel, let it stand, put it in a drying oven, and heat it at a high temperature of more than 70℃ for 35-40 minutes.

[0080] The marking layer 4 is a silicone layer with a marking pattern printed on the silicone surface. The marking layer 4 needs to use the marking pattern corresponding to the color block marking point array designed in this embodiment. Fig. 9 First, clean the surface of the silicone to remove impurities and grease, then use a special silicone UV printing treatment agent to increase the activity of the surface to improve adhesion, and use a high-precision UV printer to print the designed marking pattern.

[0081] The preparation method of the reflective layer 3 is as follows: use silver powder and the first silica gel, mix 1.5g of each in a ratio of 1:1, stir thoroughly and evacuate to obtain a silica gel silver powder mixture, put the prepared marking layer 4 on a glue spreader, pour the silica gel silver powder mixture on the marking layer 4, the initial speed is 700r / min, keep for 30s, then 1100r / min, keep for 15s, to obtain a reflective layer 3 with a thickness of 0.3mm.

[0082] In this embodiment, silver powder is added to the reflective layer 3. Under the lighting condition of total internal reflection, a strong brightness difference is formed between the pattern and the background area. The pattern separation can be completed using a traditional image processing algorithm. In addition, the time spent by the matching algorithm is generally around 10ms, so real-time three-dimensional reconstruction can be achieved.

[0083] The preparation method of the protective layer 2 is as follows: use black pigment and the first silicone rubber, mix them in a ratio of 1:5, stir them thoroughly and evacuate them to obtain a silicone rubber black pigment mixture, put the marking layer 4 with the reflective layer 3 on a glue spreader, pour the silicone rubber black pigment mixture on the reflective layer 3, and start with an initial speed of 700 r / min, keep it for 30 seconds, then 1100 r / min, keep it for 15 seconds, and a protective layer 2 with a thickness of 0.3 mm can be obtained.

[0084] The support plate 6 is a 3 mm thick highly transparent acrylic plate, which is used to assemble and fix the elastic body 5 and ensure that the image acquisition module can accurately capture the deformation of the contact body. This material has good transparency and optical continuity, which improves the performance and reliability of the binocular tactile sensor.

[0085] The lighting module uses a programmable LED light bar 7 with brightness and color. The LED light bar 7 is arranged around the periphery of the elastic body 5 to form a total internal reflection light path. Among them, the lighting module uses a 4mm wide LED light bar and adopts a total internal reflection light path design. The light bar is arranged along the elastic body 5 to ensure uniform lighting effects and improve the imaging quality of the binocular tactile sensor. This design not only effectively reduces the overall size of the binocular tactile sensor, but also supports flexible adjustment of the brightness and color of the light through programming to meet different operating requirements.

[0086] The image acquisition module uses two miniature camera modules 11 arranged at a fixed baseline distance to form a binocular vision camera, and also includes a base fixing member 12 detachably connected to the rear end of the sleeve 8, and the two are connected by four slotted cylindrical head screws M2.5×8. The miniature camera module 11 is located in the installation slot on the base fixing member 12, and the miniature camera module 11 is detachably fixed to the base fixing member 12 through a camera fixing plate 10, which is connected by two slotted cylindrical head screws M3×4. In this example, the fixed baseline distance is preferably 13mm, where the baseline distance is a variable, which can be adjusted according to the shooting distance and the wide angle of the lens to capture the image of the target object from different angles at the same time.

[0087] Specifically, the image acquisition module adopts a binocular design. Both cameras are 16-megapixel 4K fixed-focus micro camera modules equipped with 75° distortion-free lenses. The cameras are arranged at a fixed baseline distance to capture images of target objects from different angles at the same time. The binocular parallax principle is used to accurately calculate the depth information of the object and achieve high-precision 3D morphological reconstruction. While providing higher spatial resolution, this module also significantly improves the measurement accuracy and reliability of binocular tactile sensors in complex scenes.

[0088] The binocular vision-tactile sensor proposed in the present invention can achieve real-time high-precision three-dimensional reconstruction to overcome the limitations of low precision and poor real-time performance of the prior art and improve the performance and safety of robots in various complex tasks.

[0089] Embodiment 2

[0090] Please refer to Figure 4-Figure 11 This embodiment proposes a real-time high-precision three-dimensional reconstruction method for a binocular tactile sensor based on a color block marker array, and the method includes the following steps:

[0091] S1. Select at least N color blocks with obvious color differences as color block markers, and construct a color block marker array based on the color block markers to achieve real-time high-precision three-dimensional reconstruction of binocular tactile sensors. The design requirements of the color block marker array are:

[0092] The first rule: randomly select a color block in the rectangular digital array, and take the color block as the center. The 12 adjacent color blocks will not have the same color as the center color block, ensuring that the color blocks are easy to separate; Figure 4 As shown, the positions of the 12 adjacent color blocks centered on □ are where △ is located.

[0093] Article 2: The Hamming distance between the standard code corresponding to each color block and the four rotary codes of the standard code and the five strings of codes of other color blocks in an expanded window centered on the current color block is greater than or equal to 2. At the same time, the Hamming distance between the standard code corresponding to each color block and the four strings of rotary codes is also greater than or equal to 2, ensuring the uniqueness and large difference of the standard code and rotary code corresponding to the color block in this expanded window.

[0094] As a further implementation of this embodiment, the process of constructing the color block marking point array includes:

[0095] S11, define that each color block is represented by a number, and a plurality of color blocks together form a rectangular digital array of size n×n;

[0096] S12, randomly select a color block at a non-edge position in the rectangular digital array, and combine the eight most adjacent color blocks around it, starting from the rightmost color block, and write the digital representations of the eight most adjacent color blocks in a counterclockwise direction into a string of 8-bit numbers as the standard code of the color block;

[0097] The standard code is shifted forward by one bit, backward by one bit, forward by two bits, and backward by two bits respectively to form four rotary codes of the standard code; the standard code and the four rotary codes together constitute the coding information of the color block.

[0098] like Figure 5 As shown in the figure, no less than 7 different color blocks are selected as marking points, and then an array is generated using the selected color block marking points. In order to facilitate the design of the color block marking point array, each color block is represented by a number. The non-edge color blocks in the array have 8 neighboring color blocks surrounding them. Starting from the rightmost color block, in a counterclockwise direction, the digital representations of the 8 neighboring color blocks are written as a string of 8-bit numbers as the standard code of the color block, as shown in the figure below. Figure 5 As shown in the figure, the color block 3 corresponding to the third row and the third column has a standard code of 27056051. There are four rotation codes for the standard code of each color block, corresponding to the four cases where the standard code is shifted forward by one bit, backward by one bit, forward by two bits, and backward by two bits, for example Figure 5 The four rotation codes of color block 3 are: 70560512, 12705605, 05605127, and 51270560.

[0099] S13, randomly filling the rectangular digital array with the selected N colors, specifically:

[0100] Fill the first row of a rectangular digital array: Given different colors for the first three blocks in the first row, ensure that the colors of these three blocks are different, and use uniform distribution to generate random colors for subsequent blocks. If the generated block has the same color as one of the first three blocks, regenerate it until the color is different.

[0101] Fill the first column of a rectangular digital array: Given different colors for the first three blocks in the first column, and the colors of these three blocks are different and different from the first three blocks in the first row, the subsequent blocks are generated with random colors using uniform distribution. If the generated color is the same as one of the first three blocks in the first column, it is regenerated until the color is different;

[0102] Fill the remaining positions in a rectangular array of numbers: Figure 6 As shown, given the color of the color block in the second row and second column, and this color is different from the colors of the first three color blocks in the first row and first column, traverse each position from top to bottom and from left to right: if it is in the second column, ensure that its color is different from the colors of the six adjacent color blocks of the upper upper, upper left, upper, upper right, left, and lower left, and other positions need to be controlled to be different from the six adjacent color blocks of the upper upper, upper left, upper, upper right, left left, and left. Step S13 can complete the first requirement of the color block marker array design.

[0103] S14, optimizing the rectangular digital array after random color filling, specifically:

[0104] Traverse the rectangular digital array from top to bottom and from left to right. Since the edge color blocks of the array do not have the nearest 8 color blocks surrounding them, no code can be generated. Calculate 1 string of standard codes and 4 strings of rotation codes for each color block at the non-edge position, and store the 5 strings of codes for each color block in the corresponding position of the matrix Cells Codes Ground Truth according to the position of the color block in the rectangular digital array;

[0105] Traverse the positions in the Cells Codes Ground Truth from top to bottom and from left to right. With the current traversal position as the center, calculate the Hamming distance between all the codes in the 13*13 window (ignoring the code of the current position to be compared) and the 5 strings of codes of the current traversal position. If the Hamming distance is less than 2, it means that the rectangular digital array does not meet the second requirement of the color block marker array design. Then randomly select one of the eight adjacent positions around the current traversal position as the modification position, change the color of the modified position in the rectangular digital array, and ensure that the changed color is different from the 12 adjacent positions, and update the Cells Codes Ground Truth at the same time. If a color different from the 12 adjacent positions cannot be found, the color will not be changed.

[0106] After traversing the Cells Codes Ground Truth, traverse the Cells Codes Ground Truth again and count the number of times the Hamming distance is less than 2. If the number is not equal to 0, repeat the optimization step S14.

[0107] Through step S14, the second requirement of the color block marking point array design can be completed while the first requirement of the color block marking point array design is maintained.

[0108] This embodiment takes 8 distinct color blocks as marking points as an example, and uses numbers 0 to 7 to represent each color block. Figure 7 The size of each color block is 0.22mm, and the interval between adjacent color blocks is 0.5mm. The above marking point array design method can generate a 60*60 color block marking point array, and the overall size of the pattern is 3cm*3cm, as shown in Figure 8 and Fig. 9 shown.

[0109] The color block marking point array designed in this embodiment has an overall size of 3cm*3cm, the color block spacing is only 0.5mm, the number of color blocks is as high as 3600, and the marking point density is as high as 400 / cm 2 , which can achieve high-precision three-dimensional reconstruction.

[0110] S2. Use calibration images to calibrate and correct the binocular vision camera in the image acquisition module. Specifically, use the binocular vision camera to take 15-25 calibration images that meet the requirements, and use the built-in Stereo Camera Calibrator module in Matlab to complete the calibration and correction of the binocular camera. After calibrating the binocular vision camera, the internal parameters, external parameters, distortion parameters, correction parameters and other parameters of the binocular vision camera are obtained, and then the binocular vision camera is calibrated according to the obtained parameters.

[0111] S3. According to the color block marker array, calculate the coding information corresponding to each color block in the array and store it in the corresponding position of the matrix Cells Codes Ground Truth. The coding information includes a string of standard codes and 4 strings of rotation codes.

[0112] S4, obtaining the first left and right images when the binocular tactile sensor is not in contact with the object, and using the pixel coordinates of the center of each color block in the first left and right images and the position of the corresponding color block marker array as the previous frame image of the first frame; this step is essentially to initialize the binocular tactile sensor, and when it is not in contact with the object, the first left and right images obtained are standard images. In this embodiment, since the binocular tactile sensor includes two micro camera modules 11, the images obtained are left and right images, and in this embodiment, left and right images are used as representations.

[0113] S5, using a binocular vision camera to obtain the second left and right images of the current frame, and assigning an index to each color block detected in the current frame corresponding to the second left and right images according to the color block marker array. The specific process includes the following steps:

[0114] S51, preprocessing the second left and right images, including adaptive thresholding, corrosion, and expansion;

[0115] S52, performing contour detection on the preprocessed second left and right images, and calculating the centroid of the contour as the center of the color block;

[0116] S53, calculating the Euclidean distance Distance between the mean value of the pixels in the contour and the defined N colors in the RGB color space;

[0117] S54: The color closest to the N colors is used as the color of the current color block, and each color block detected in the current frame is assigned an index.

[0118] In this embodiment, the obtained second left and right images are preprocessed, including a series of operations such as adaptive threshold, corrosion, and expansion. The processed images are detected using the find contours function of opencv. The centroid of the contour is calculated using the moments function and used as the center of the color block. The Euclidean distance between the mean value of the pixels in the contour and the defined 8 colors in the RGB color space is calculated. Assuming that there are two colors (R1, G1, B1) and (R2, G2, B2), the Euclidean distance formula is:

[0119]

[0120] Then the color closest to the eight colors is used as the color of the current color block, and each color block detected in the current frame is assigned an index, so that each color block has three pieces of information: center coordinates, color, and index.

[0121] S6, using the initial information and coding information of the color block in the color block marker array in the previous frame image, jointly determine the position of the color block in the color block marker array in the current frame image and add an index, and store it in the statistics matrix statistics. The initial information is the pixel coordinates of the center of each color block in the first left and right images and the position of the corresponding color block in the color block marker array. In step S6, the specific process includes:

[0122] Determine the starting position: First, find all the color blocks within the range of 3 color blocks from the center of the current color block in the previous frame image (when not deformed), sort them according to the rule of distance from near to far, and judge the colors of the color blocks in turn. The first color block with the same color found is regarded as the corresponding color block in the previous frame image. If the corresponding color block can be found, get the position of the corresponding color block in the color block marker array, and add the index corresponding to the current color block twice to the corresponding position in the statistics matrix statistics. At the same time, the position of the corresponding color block in the color block marker array is used as the starting position. If it cannot be found, perform linear calculation through the center pixel coordinates of the current color block (subtract the center coordinates of the upper leftmost character block when it is not deformed at the beginning, and divide by the spacing) to determine a starting position.

[0123] Find eight neighboring color blocks to generate coding information: Find the 14 neighboring color blocks closest to the current color block, and use the distance and angle information to filter them, as follows: record the Chebyshev distance (maximum xy increment) and angle (angle with the positive direction of the x-axis, ranging from 0-360) between each neighboring color block and the central color block, sort the neighboring color blocks by Chebyshev distance, filter the sorted color blocks in turn, set the filtering rules, and directly add the two nearest neighboring color blocks; for other color blocks, calculate their angles with the added neighboring color blocks, if the minimum angle is greater than angle_threshold_1 (50°), add them; if the minimum angle is less than angle_threshold_2 (20°), do not add them; if the sum of the two smallest angles is greater than angle_threshold_3 (58°), add them. Pick out 8 adjacent color blocks, select the color block with the smallest angle with the x-axis square as the starting color block, extract the color of each adjacent color block in a counterclockwise direction to generate coding information. If it is not possible to pick out 8 adjacent color blocks that meet the requirements to generate coding, skip the step of using the coding information to find the position of the color block in the color block marker array.

[0124] Use the coded information to find the position of the color block in the color block marker array: perform a spiral search with the starting position as the center, and the search range is matchSearchRaidus(5) circles. Calculate the Hamming distance between the code code and the 5 strings of code at the search position, and consider the case where the coded information finds a wrong adjacent color block. If the Hamming distance is greater than or equal to 2, it means that the position is not the correct position, and continue searching. If the Hamming distance is less than 2, it means that the position is the correct position, and add the indexes corresponding to the central color block and the 8 neighboring color blocks to the corresponding positions in the statistical matrix statistics, and end the search.

[0125] S7. Traverse each position in the statistical matrix statistics, find the index with the most occurrences at each position as the color block at the current position, and store it in the matrix Cells Codes Cur; specifically, after traversing all the color blocks, find the index of the color block with the most occurrences at each position in the statistical matrix statistics, and store the index of the corresponding color block in the corresponding position in the matrix Cells Codes Cur.

[0126] S8, traverse the matrix Cells Codes Cur corresponding to the second left and right images, and calculate the disparity d according to the two color blocks located at the same position p , and obtain the three-dimensional coordinates of the center point of the color block in space through triangulation to generate point cloud data; specifically, traverse the matrix Cells Codes Cur of the left image and the right image at the same time to obtain the index of the left and right color blocks at the same position, such as Fig.11 As shown, the left image is taken from the 30th row and 45th column, and the right image must also be taken from the 30th row and 45th column. The center pixel coordinates of the left and right color blocks are obtained according to the index and are P C1 (u1,v1) and P C2 (u2,v2), then calculate the disparity d p (pixel unit), that is:

[0127] d p =u1-u2

[0128] By Parallax d p The depth Z can be calculated, that is:

[0129]

[0130] Among them, f is the focal length and b is the baseline length.

[0131] According to the pinhole imaging principle and using triangulation, the three-dimensional coordinates of the point can be calculated based on the known depth Z and the camera internal parameters, that is:

[0132]

[0133] Among them, c x and c y is the principal point of the camera. Therefore, the generated three-dimensional coordinate point is (X w ,Y w ,Z w ), and then three-dimensional point cloud data can be generated.

[0134] S9. Calculate the normal of the generated point cloud data after preprocessing, and perform Poisson reconstruction based on the normal to generate a smooth three-dimensional surface. In this step, the generated point cloud data first needs to be subjected to outlier removal. Statistical filtering or radius filtering can be used, and then the abnormal points are identified and removed by calculating the mean and standard deviation of the neighbor distance of each point. Next, calculate the normal of the point cloud data. The commonly used method is to fit the local plane through the nearest neighbor point or use the PCA algorithm to calculate the eigenvector corresponding to the minimum eigenvalue as the normal direction. After the normal calculation is completed, Poisson reconstruction is performed, and the normal field is used as the source term. By solving the Poisson equation, a smooth three-dimensional surface is finally generated.

[0135] The present invention completes 3D reconstruction based on the binocular vision tactile sensor, including image preprocessing, identification of color block markers, calculation of center coordinates, color discrimination, and determination of the position of the color block in the marker array based on the coding information and marker tracking and matching, and finally calculation of the point cloud to achieve 3D reconstruction. The matching part of the present invention takes less than 10ms, and can obtain dense (400 / cm 2 ) point cloud data and can adapt to the loss of markers under large deformation conditions, which is of great value in improving the performance and safety of robots in various complex tasks.

[0136] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, so the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0137] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the above embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0138] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A real-time high-precision three-dimensional reconstruction method based on a color block marker array, using a contact module composed of a protective layer, a reflective layer, a multi-layer elastic body and a color block marker array with a coding rule, combined with an image acquisition module, an illumination module and an information processing module to form a binocular tactile sensor, characterized in that: The 3D reconstruction method includes the following steps: S1. Selecting at least N color blocks with obviously different colors as color block marking points, and constructing a color block marking point array for realizing real-time high-precision three-dimensional reconstruction of a binocular tactile sensor based on the color block marking points; S2, using the calibration image to calibrate and correct the binocular vision camera in the image acquisition module; S3, calculate the coding information corresponding to each color block at each position in the color block marker array, and store it in the corresponding position of the matrix CellsCodes Ground Truth, where the coding information includes 1 string of standard codes and 4 strings of rotary codes; S4, obtaining the first left and right images when the binocular tactile sensor is not in contact with the object, and taking the pixel coordinates of the center of each color block in the first left and right images and the position of the corresponding color block in the color block mark point array as the previous frame image of the first frame; S5, using a binocular vision camera to obtain the second left and right images of the current frame, and assigning an index to each color block detected in the current frame corresponding to the second left and right images according to the color block marker array; S6, using the initial information and the coded information of the color block in the color block marker array in the previous frame image, jointly determine the position of the color block in the color block marker array in the current frame image and add an index, and store it in the statistical matrix statistics; S7, traverse each position in the statistics matrix statistics, find the index with the most occurrences in each position as the color block at the current position, and store it in the matrix Cells Codes Cur; S8, traverse the matrix Cells Codes Cur corresponding to the second left and right images, and calculate the disparity d according to the two color blocks located at the same position p And the three-dimensional coordinates of the center point of the color block in space are obtained by triangulation to generate point cloud data; S9. After preprocessing the generated point cloud data, calculate its normal, and perform Poisson reconstruction based on the normal to generate a smooth three-dimensional surface.

2. The real-time high-precision three-dimensional reconstruction method according to claim 1, characterized in that: In step S1, the process of constructing the color block marker array includes: S11. Define the design requirements of the color block marker array; S12, defining that each color block is represented by a number, and that a plurality of color blocks together form a rectangular digital array of size n×n; S13, randomly select a color block at a non-edge position in the rectangular digital array, and combine the eight most adjacent color blocks around it, starting from the rightmost color block, and write the digital representations of the eight most adjacent color blocks in a counterclockwise direction into a string of 8-bit numbers as the standard code of the color block; The standard code is shifted forward by one bit, backward by one bit, forward by two bits, and backward by two bits to form four kinds of rotation codes of the standard code; S14, randomly filling the rectangular digital array with the selected N colors; S15, optimizing the rectangular digital array filled with random colors to obtain a color block marking point array.

3. The real-time high-precision 3D reconstruction method according to claim 2, characterized in that: In step S11, the design requirements include: Choose a color block at random in the rectangular digital array, and take it as the center. The 12 adjacent color blocks will not have the same color as the center color block. The Hamming distance between the standard code corresponding to each color block and the four rotary codes of the standard code and the five strings of codes of other color blocks in an expanded window centered on the current color block is greater than or equal to 2. At the same time, the Hamming distance between the standard code corresponding to each color block and the four strings of rotary codes is also greater than or equal to 2.

4. The real-time high-precision 3D reconstruction method according to claim 2, characterized in that: In step S14, the rectangular digital array is randomly filled with the selected N colors, specifically: Fill the first row of a rectangular digital array: Given different colors for the first three color blocks in the first row, generate random colors for subsequent color blocks using uniform distribution. If the generated color block has the same color as one of the first three color blocks, regenerate it until the color is different. Fill the first column of a rectangular digital array: Given different colors for the first three blocks in the first column, and the colors of these three blocks are different and different from the first three blocks in the first row, the subsequent blocks are generated with random colors using uniform distribution. If the generated color is the same as one of the first three blocks in the first column, it is regenerated until the color is different; Fill the remaining positions in the rectangular digital array: given the color of the color block in the second row and second column, and this color is different from the colors of the first three color blocks in the first row and first column, traverse each position from top to bottom and from left to right. If it is in the second column, ensure that its color is different from the colors of the six adjacent color blocks: top-top, top-left, top, top-right, left, and bottom-left. For other positions, ensure that their colors are different from the six adjacent color blocks: top-top, top-left, top, top-right, left-left, and left.

5. The real-time high-precision 3D reconstruction method according to claim 2, characterized in that: In step S15, the rectangular digital array filled with random colors is optimized, specifically including: Traverse the rectangular digital array from top to bottom and from left to right, calculate the 1-string standard code and 4-string rotation code of each color block at the non-edge position, and store the 5-string code of each color block in the corresponding position of the matrix Cells Codes Ground Truth according to the position of the color block in the rectangular digital array; Traverse the positions in the Cells Codes Ground Truth from top to bottom and from left to right. With the current traversal position as the center, calculate the Hamming distance between all standard codes in the 13*13 window and the 5 strings of codes of the current traversal position. If the Hamming distance is less than 2, randomly select one of the eight adjacent positions around the current traversal position as the modification position, change the color of the modified position in the rectangular digital array, and ensure that the changed color is different from the 12 adjacent positions, and update the Cells Codes Ground Truth at the same time; if a color different from the 12 adjacent positions cannot be found, do not change the color; After traversing the Cells Codes Ground Truth, traverse the Cells Codes Ground Truth again and count the number of times the Hamming distance is less than 2. If the number is not equal to 0, repeat the optimization step S14.

6. The real-time high-precision three-dimensional reconstruction method according to claim 1, characterized in that: In step S5, the specific process includes the following steps: S51, preprocessing the second left and right images, including adaptive thresholding, corrosion, and expansion; S52, performing contour detection on the preprocessed second left and right images, and calculating the centroid of the contour as the center of the color block; S53, calculating the Euclidean distance Distance between the mean value of the pixels in the contour and the defined N colors in the RGB color space; S54: The color closest to the N colors is used as the color of the current color block, and each color block detected in the current frame is assigned an index.

7. The real-time high-precision 3D reconstruction method according to claim 1, characterized in that: In step S6, the specific process includes: Determine the starting position: In the absence of deformation, find all color blocks within a distance of 3 color blocks from the center of the current color block in the previous frame image, sort them according to the rule of distance from near to far, judge the colors of the color blocks in turn, and regard the first color block with the same color found as the corresponding color block in the previous frame image; If the corresponding color block can be found, obtain the position of the corresponding color block in the color block marker array, and add the index corresponding to the current color block twice to the corresponding position in the statistics matrix statistics, and use the position of the corresponding color block in the color block marker array as the starting position; If it cannot be found, a starting position is determined by linear calculation through the coordinates of the center pixel of the current color block; Determine whether eight adjacent color blocks that meet the requirements are found to generate coding information. If so, proceed to the next step; if not, traverse the next color block and determine the starting position; Use the coded information to find the position of the color block in the color block marker array: perform a spiral search with the starting position as the center, calculate the Hamming distance between the coded information and the 5 strings of code at the search position, if the Hamming distance is greater than or equal to 2, it means that the position is not the correct position, continue searching, if the Hamming distance is less than 2, it means that the position is the correct position, add the index corresponding to the central color block and the 8 neighboring color blocks once at the corresponding position in the statistical matrix statistics, and end the search.

8. The real-time high-precision three-dimensional reconstruction method according to claim 7, characterized in that: The step of determining whether eight adjacent color blocks meeting the requirement are found to generate coding information specifically includes: Find the 14 neighboring color blocks closest to the current color block and filter them using distance and angle information, as follows: Record the Chebyshev distance and angle between each adjacent color block and the central color block, sort the adjacent color blocks by Chebyshev distance, screen the sorted color blocks in turn, and directly add the two nearest adjacent color blocks; For other color blocks, calculate their angles with the adjacent color blocks that have been added. If the minimum angle is greater than angle_threshold_1 (50°), add them. If the minimum angle is less than angle_threshold_2 (20°), it will not be added; If the sum of the two smallest angles is greater than angle_threshold_3 (58°), add it; Then pick out 8 adjacent color blocks, select the color block with the smallest angle with the x-axis square as the starting color block, and extract the color of each adjacent color block in a counterclockwise direction to generate coding information.

Images