Visual tactile sensor based on color block mark array optimization and three-dimensional reconstruction method

Abstract

The invention relates to the technical field of robot sensors, in particular to a visual tactile sensor based on color block mark array optimization and a three-dimensional reconstruction method, and the method comprises the steps: enabling four colors with remarkable visual difference to correspond to two patterns with different sizes to form eight digital identifiers, a binocular vision camera is used for collecting a color block array on the surface of an elastomer, on the basis of completing three-dimensional calibration and image preprocessing, the color block characteristics of mark points and neighborhood distribution of the mark points are used as constraints, displacement distance screening, line positioning and inter-frame movement continuity are combined, rapid matching and dynamic tracking of the mark points are achieved, and the accuracy and the accuracy of three-dimensional calibration are improved. According to the method, the spiral search traversal frequency is remarkably reduced, the system processing time is shortened to about 270 ms from about 320 ms through array structure design, matching strategy optimization and sensor structure miniaturization cooperation, high-precision reconstruction is guaranteed, meanwhile, the real-time performance and robustness are improved, and the method is suitable for complex operation scenes such as industrial robot precision assembly and flexible grabbing.

Description

Technical Field

[0001] This invention relates to the field of robot sensor technology, specifically to a visual-tactile sensor and a three-dimensional reconstruction method based on a color block marker array optimization. Background Technology

[0002] As one of the important foundational technologies in the fields of computer vision, robotics and digital twins, object 3D reconstruction technology recovers the 3D geometry, surface texture and spatial position relationship of objects from perceived data, generating quantifiable and visualized 3D models. It is widely used in key scenarios such as industrial reverse engineering, cultural relic protection, medical surgical planning, robot grasping and assembly. As a data acquisition device that can simultaneously sense multi-dimensional information such as normal force, shear force, relative sliding and object pose, visual tactile sensors are an important hardware foundation for realizing high-precision object 3D reconstruction and contact perception.

[0003] A visual-tactile sensor and 3D reconstruction method based on a color block marker array, disclosed in Chinese Patent Publication No. 202411416843.X, combines a color block marker array with binocular vision, achieving matching and 3D reconstruction through neighborhood color sequence encoding and spiral search. However, this scheme still has the following shortcomings:

[0004] To ensure the number of codes, a large number of marker colors are used, resulting in small intervals between different colors in the RGB space. Adjacent colors are easily compressed into the same interval. In practical applications, color block recognition is easily affected by factors such as uneven lighting, changes in reflection caused by deformation of silicone elastomers, and local shadows, which can lead to color misjudgment and reduce the stability of marker recognition.

[0005] The marked color blocks are usually located on the surface of silicone. Light is refracted, scattered and reflected multiple times inside the silicone. After being superimposed with elastic deformation, the boundaries of adjacent color block areas are easily blurred or even stuck together in the image, which further increases the difficulty of color classification and contour segmentation.

[0006] It relies heavily on the independent coding design of each marker point and performs neighborhood retrieval through a large-scale spiral search covering multiple loop regions. Under complex arrays and large-scale search ranges, the number of traversals of the spiral search often reaches dozens or even hundreds of times, resulting in a complex matching process, high time overhead, and difficulty in meeting the requirements of real-time performance and high-precision reconstruction. To address this, a visual-tactile sensor and 3D reconstruction method based on color block marker array optimization is proposed. Summary of the Invention

[0007] To address the technical problems existing in the prior art, the present invention provides a visual-tactile sensor and a three-dimensional reconstruction method based on color block marker array optimization.

[0008] To address the aforementioned technical problems, this invention provides the following technical solution: a visual-tactile sensor and a three-dimensional reconstruction method based on a color block marker array optimization, wherein the three-dimensional reconstruction method includes the following steps:

[0009] S1, Construct a color block marker array: Select four color blocks with significant color differences as basic marker units. Each color corresponds to two color blocks of different sizes, forming eight distinguishable codes, each representing two different numbers, which together form eight distinguishable number identifiers. Generate a number array based on this, and then generate a color block marker array based on the number array.

[0010] S2, Constructing a matching strategy: After the stereo calibration and image preprocessing of the binocular vision camera are completed, the color block features of a single marker point in the color block marker array and the color block distribution information of the eight neighboring marker points around the marker point are used as the matching basis to establish multi-dimensional positional constraints including color features, neighborhood distribution and spatial geometric relationships, so that the current frame has three complete pieces of information.

[0011] S3, Establish an inter-frame tracking mechanism: Extract the initial position and encoding information of the color block in the previous frame image from the three complete information items, use the initial position and encoding information as the prior for the current frame marker point matching and position prediction, take advantage of the small spatial displacement of the marker point between adjacent frames and the continuous change of RGB values, and construct a dynamic tracking link for the marker point by predicting the motion trajectory of the marker point in the continuous frame image and associating it with historical matching information.

[0012] S4 combines the matching strategy and inter-frame tracking mechanism to perform candidate neighborhood retrieval for each marker point in the current frame, and quickly determine the current position of the color block corresponding to the current marker point in the color block marker point array based on displacement distance filtering and row positioning rules. A unique index is assigned to each successfully matched marker point, and an index matrix matrix_serialnumbe is built according to the pixel coordinates of each color block in the image. The unique index of each color block is stored at the corresponding position. This matrix is ​​used to find 8 neighboring color blocks of the target color block.

[0013] S5. Based on the pixel coordinates of the same marker point in the left and right views acquired by the binocular vision camera, calculate the disparity information of the corresponding marker point, recover the depth data of the contact surface, and obtain the three-dimensional coordinates of the center of each marker point in space through triangulation, thereby generating the original point cloud data.

[0014] S6 preprocesses the original point cloud data and uses the Poisson surface reconstruction algorithm to fit the preprocessed point cloud data into a complete three-dimensional surface model by constructing implicit functions and solving the Poisson equation.

[0015] Preferably, in step S1, the eight distinguishable codes are filled to form a two-dimensional color block marker array according to a pre-set array arrangement rule. Within a 13×13 window centered on any marker, the color and spatial distribution combination of the central color block and its eight surrounding neighboring color blocks is unique within the 13×13 window, thereby uniquely determining the position of each marker in the array through the spatial position correlation of the array.

[0016] In the color block marker array, large color blocks represent odd numbers, and small color blocks represent even numbers; the overall size of the color block marker array is 2cm. The color block is 2cm long, the center-to-center spacing between adjacent color blocks is 0.5mm, the number of color blocks is 1600, and the marker density is 400 markers / cm. 2 .

[0017] Preferably, the color block marker array has a size of 40×40, and the array contains:

[0018] Odd rows are composed of small blocks of the same color. The color of each odd row does not repeat the color combination of the three odd rows preceding it (i.e., the three odd rows before it in order of row number). All odd rows in rows 1-8 use only small blocks of color.

[0019] The color blocks in odd columns of even rows are determined by the color blocks in the corresponding columns of the two odd rows above and below that even row. The colors of adjacent odd columns in the same row are different (i.e., the colors of columns 1 and 3, columns 3 and 5, etc. are all different). The color blocks in odd columns of even rows are small color blocks, and the color blocks in even columns of even rows are large color blocks. Furthermore, within the same even row, the color of any large color block in an even column is not the same as the color of the large color blocks in the three even columns to its left in the row containing that color block.

[0020] Preferably, the color block marker array is formed by cyclically filling an 8-row, 8-column pattern, and the large color blocks in the same even-numbered column are all of the same color;

[0021] Preferably, the local window formed by the marker point at each position in the digital array and the corresponding color block and its eight neighboring color blocks has the following characteristics:

[0022] The color blocks in even rows and even columns are all large color blocks, and the eight adjacent color blocks around them are all small color blocks, containing four colors. There are two groups of three small color blocks of the same color that are consecutively adjacent, and the two groups of consecutive small color blocks are separated by two small color blocks of different colors.

[0023] The color blocks in even rows and odd columns are all small color blocks. There are 6 small color blocks and 2 large color blocks in the eight neighboring color blocks around them, containing four colors. There are two groups of three small color blocks of the same color that are consecutively adjacent. The two groups of color blocks are separated by two different color blocks, and the color block itself will not appear in the eight neighboring color blocks.

[0024] The odd-numbered rows and even-numbered columns are all small color blocks. There are 6 small color blocks and 2 large color blocks in the eight neighboring color blocks around it. There are five kinds of color blocks. There are no three consecutive color blocks of the same color. Moreover, the color block itself appears in the eight neighboring color blocks.

[0025] The odd-numbered rows and columns are all small color blocks. The eight neighboring color blocks around it are 4 small color blocks and 4 large color blocks, containing five colors. There are three consecutive adjacent color blocks with no matching colors, and the color block itself appears in the eight neighboring color blocks.

[0026] The spatial characteristics of each color block and its eight neighboring color blocks in the marker array are recorded in the matching matrix Matching_matrix to prepare for subsequent marker matching.

[0027] Preferably, step S2 specifically includes the following steps:

[0028] S21, when the sensor's elastic body is not in contact with any object, images captured by the left and right camera modules are acquired simultaneously and used as the initial left and right standard images, respectively. The acquired initial left and right standard images are preprocessed and adaptive threshold segmentation, erosion and dilation operations are performed in sequence.

[0029] S22, Perform contour detection on the processed image and calculate the centroid of each contour as the center coordinate of the corresponding color block. By using the pixel coordinates of the center of each color block in the image, establish the positional relationship of the color block marker points and complete the initial calibration of the system.

[0030] S23. Use a binocular vision camera to take pictures of the calibration board from multiple angles, and collect 20 to 30 calibration images that meet the specifications. The calibration solution obtains a complete parameter set consisting of camera intrinsic parameters, extrinsic parameters, distortion coefficients and stereo correction parameters. Based on the complete parameter set, subsequent images are corrected and stereo corrected.

[0031] S24, using the corrected left and right images of the elastic body when it is not in contact with any object as standard images, provides a reference coordinate system in an undeformed state for 3D reconstruction, and performs projection transformation on the acquired subsequent images;

[0032] S25. For each contour region, calculate the Euclidean distance between the average pixel value within the contour and the reference color in RGB color and base color. Determine the color with the smallest distance as the recognition result of the color block corresponding to the current marker point. Assign a unique index to each successfully detected color block in the current frame, so that the current frame has three complete pieces of information: center coordinates, color attributes and index identifier.

[0033] Preferably, step S4 specifically includes the following steps:

[0034] S41, Match each marker point to obtain the statistical matrix index_vote_matrix. In the index matrix matrix_serialnumbe, take the position of the color block corresponding to the current marker point in the index matrix matrix_serialnumbe as the center, and use a spiral search method to retrieve neighboring color blocks. Under the condition of satisfying the preset maximum traversal radius, filter out the 8 neighboring color blocks closest to the current marker point according to the displacement distance, and sort the 8 neighboring color blocks in the range of 0° to 360° according to the polar angle relative to the positive x-axis direction.

[0035] S42, combining the spatial displacement constraints of each marker point in the current frame with the absolute value of the RGB difference with the corresponding marker point in the previous frame and the continuity constraints within the preset threshold range, select candidate positions that satisfy the color mode and geometric relationship matching among the sorted neighboring color blocks.

[0036] S43, when the color block corresponding to the current marker point is a large color block, based on the pattern that all eight neighboring color blocks around it are small color blocks and contain four colors and have two sets of three small color blocks of the same color that are consecutively adjacent, the current marker point is determined to be located in the even-numbered row and even-numbered column. Starting from this row, a spiral search is performed along the row direction until the final precise position of the current marker point is determined in the color block marker point array and the corresponding index is used in the statistical matrix.

[0037] S44, when the color block corresponding to the current marker point is a small color block, a branch determination is made based on whether there is a color block with the same color as the current marker point among its eight neighboring color blocks: if there is a color block with the same color, the current marker point is determined to be located in an even row and an odd column, and the row is determined based on the color pattern and geometric distribution of the neighboring color blocks. A spiral search is used within that row to determine the precise position; if there is no color block with the same color, the current marker point is determined to be located in an odd row, and the row is determined based on the color pattern and geometric distribution of the neighboring color blocks. A spiral search is used within that row to determine the precise position.

[0038] S45. After traversing all the color blocks corresponding to the marked points, perform frequency statistics on the index values ​​recorded at each position in the statistical matrix index_vote_matrix, and determine the index that appears most frequently as the final color block identifier for that position.

[0039] Preferably, in step S3, when some marker points are briefly occluded or slightly shifted in the current frame, the inter-frame tracking mechanism, based on the matching results and motion trends of the previous frame, retrieves color blocks in adjacent previous frames that are close to the current marker point and whose sum of absolute RGB differences is within a preset threshold (the value of which can be determined by calibration or experiment) as corresponding matching color blocks, thereby achieving continuous tracking of the occluded marker points.

[0040] A visual-tactile sensor includes a sleeve, an elastomer, a camera mounting plate, a miniature camera module, and a base fixture. The elastomer is made of highly elastic, low-hysteresis silicone material. A color block marker array is provided on the surface of the elastomer to be contacted. Two miniature camera modules are arranged opposite each other at a 12° tilt angle and assembled on the base fixture. Two camera mounting plates are detachably connected to the base fixture by fasteners for fixing the miniature camera modules. The miniature camera modules are binocular vision cameras used to acquire left and right images of the color block marker array under different contact states.

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0042] 1. This invention reduces the original eight marking colors to four visually distinct colors and adopts an encoding method in which each color corresponds to two color blocks of different sizes. While ensuring the number of distinguishable codes, it increases the color interval, reduces the probability of color misjudgment caused by uneven lighting, silicone deformation, reflection changes, and local shadows, and makes the large color block surrounded by the small color block, forming a clear visual interval between adjacent color blocks. This effectively reduces the blurring and adhesion of color block boundaries under silicone deformation and improves the stability of contour segmentation and color recognition.

[0043] 2. This invention employs two miniature camera modules installed at a preset tilt angle of 12°. While achieving device miniaturization, it increases the imaging area of ​​the marker points, reduces invalid areas, optimizes the viewing angle, filters redundant background information, and improves the image signal-to-noise ratio. Combined with a highly elastic, low-hysteresis silicone elastomer and a high-density color block marker array, the sensor can work stably in a confined space and acquire uniform, high-quality point cloud data.

[0044] 3. In this invention, by replanning the color block array arrangement and neighborhood relationship design, the color distribution pattern of a local window composed of any marker point and its eight neighboring marker points is unique in a 13*13 array centered on the local window. Thus, the spatial position correlation of the array replaces the independent encoding design of each marker point, simplifies the formulation of encoding rules and fault tolerance design, and improves the recognition stability and anti-interference ability in complex usage environments.

[0045] 4. In this invention, the matching strategy combines displacement distance screening, line positioning and inter-frame tracking mechanism. By taking advantage of the small displacement of the marker point between adjacent frames and the continuous RGB change, the candidate neighborhood is quickly searched and matched under constraints, which significantly reduces the number of traversals of the spiral search and reduces the system processing time from about 320ms to about 270ms. This improves the overall real-time performance while ensuring the accuracy of 3D reconstruction.

[0046] 5. This invention improves the robot's ability to perceive the geometry, surface characteristics, and deformation state of the object being manipulated, providing reliable visual-tactile fusion perception support for complex tasks such as precision assembly and flexible grasping of industrial robots, reducing collision risks and operational errors during operation, and has good application value and promotion prospects. Attached Figure Description

[0047] Figure 1 This is a schematic diagram comparing the color block arrangement of the prior art with that of the present invention;

[0048] Figure 2 This is a schematic diagram of the digital array of the present invention;

[0049] Figure 3 This is a schematic diagram of the color block array corresponding to the marker points of the present invention;

[0050] Figure 4 A schematic diagram of the retrieval path for an existing marker matching algorithm;

[0051] Figure 5 This is a schematic diagram of the matching algorithm of the present invention;

[0052] Figure 6 This is a schematic diagram illustrating the color block positioning principle of the present invention;

[0053] Figure 7 This is a schematic diagram of the overall visual-tactile sensor of the present invention;

[0054] Figure 8 This is an exploded view of the overall structure of the visual-tactile sensor of the present invention;

[0055] Figure 9 This is a schematic diagram of the base fixing component and the miniature camera module of the present invention;

[0056] Figure 10 This is a structural schematic diagram of the base fixing component of the present invention;

[0057] Figure 11 (a) is a schematic diagram of the original image and (b) is a schematic diagram of the binarized image, showing the adhesion phenomenon of color blocks before the optimization of the marker array;

[0058] Figure 12(a) is a schematic diagram of the original image and (b) is a schematic diagram of the binarized image, showing the phenomenon of no adhesion of color blocks after the optimization of the marker array;

[0059] Figure 13 (a) is a schematic diagram of the original left image without contact, (b) is a schematic diagram of the original right image without contact, and (c) is a schematic diagram of the point cloud without contact.

[0060] Figure 14 Schematic diagrams showing the contact between a pentagonal prism (a), a small protruding cap (b), a multi-prism (c), a hexagonal prism (d), and a pen cap (e) and a sensor.

[0061] The numbers in the diagram represent:

[0062] 1. Sleeve; 2. Elastomer; 3. Camera mounting plate; 4. Miniature camera module; 5. Base fixing component. Detailed Implementation

[0063] The present invention will be further described below with reference to the accompanying drawings and embodiments, which illustrate the above and other technical features and advantages of the present invention. However, the following embodiments are merely preferred embodiments of the present invention and are not exhaustive.

[0064] Example 1:

[0065] like Figure 1-14 As shown, this invention provides a visual-tactile sensor and a 3D reconstruction method based on a color block marker array optimization. The 3D reconstruction method includes the following steps:

[0066] S1, Construct a color block marker array: Select four color blocks with significant color differences as basic marker units. Each color corresponds to two color blocks of different sizes, forming eight distinguishable codes, each representing two different numbers, which together form eight distinguishable number identifiers. Generate a number array based on this, and then generate a color block marker array based on the number array.

[0067] S2, Constructing a matching strategy: After the stereo calibration and image preprocessing of the binocular vision camera are completed, the color block features of a single marker point in the color block marker array and the color block distribution information of the eight neighboring marker points around the marker point are used as the matching basis to establish multi-dimensional positional constraints including color features, neighborhood distribution and spatial geometric relationships, so that the current frame has three complete pieces of information.

[0068] S3, Establish an inter-frame tracking mechanism: Extract the initial position and encoding information of the color block in the previous frame image from the three complete information items, use the initial position and encoding information as the prior for the current frame marker point matching and position prediction, take advantage of the small spatial displacement of the marker point between adjacent frames and the continuous change of RGB values, and construct a dynamic tracking link for the marker point by predicting the motion trajectory of the marker point in the continuous frame image and associating it with historical matching information.

[0069] S4 combines the matching strategy and the inter-frame tracking mechanism to perform candidate neighborhood retrieval for each marker point in the current frame, and quickly determine the current position of the color block corresponding to the current marker point in the color block marker point array based on displacement distance filtering and row positioning rules, and assign a unique index to each successfully matched marker point;

[0070] S5. Based on the pixel coordinates of the same marker point in the left and right views acquired by the binocular vision camera, calculate the disparity information of the corresponding marker point, recover the depth data of the contact surface, and obtain the three-dimensional coordinates of the center of each marker point in space through triangulation, thereby generating the original point cloud data.

[0071] S6 preprocesses the original point cloud data and uses the Poisson surface reconstruction algorithm to fit the preprocessed point cloud data into a complete three-dimensional surface model by constructing implicit functions and solving the Poisson equation.

[0072] In this embodiment, in step S1, eight distinguishable codes are filled to form a two-dimensional color block marker array according to a pre-set array arrangement rule. Within a 13×13 window centered on any marker, the color and spatial distribution combination of the central color block and its eight surrounding neighboring color blocks is unique within the 13×13 window, thereby uniquely determining the position of each marker in the array through the spatial position correlation of the array.

[0073] In the color block marker array, large color blocks represent odd numbers, and small color blocks represent even numbers; the overall size of the color block marker array is 2cm. The color block is 2cm long, the center-to-center spacing between adjacent color blocks is 0.5mm, the number of color blocks is 1600, and the marker density is 400 markers / cm. 2 .

[0074] In this embodiment, the color block marker array is formed by cyclically filling an 8x8 row pattern. Large color blocks in the same even-numbered column are all of the same color. The color block marker array size is 40×40. Within the array:

[0075] Odd-numbered rows (i.e., rows 1, 3, 5, etc.) are composed of small blocks of the same color. The color of each odd-numbered row does not repeat the color combination of the three odd-numbered rows preceding it (i.e., the three odd-numbered rows before this row in order of row number). All odd-numbered rows from row 1 to row 8 use only small blocks of color.

[0076] The color blocks in odd columns of even rows (i.e., rows 2, 4, 6, etc.) are determined by the color blocks in the corresponding columns of the two odd rows above and below that even row. The colors of adjacent odd columns in the same row are different (i.e., columns 1 and 3, columns 3 and 5, etc., are all different). Odd columns in even rows are small color blocks, and even columns in even rows are large color blocks. Furthermore, within the same even row, the color of any large color block in an even column is not the same as the color of the large color blocks in the three even columns to its left in the same row.

[0077] The above design abandons the independent encoding of individual marker points and instead achieves accurate positioning and matching of marker points through array arrangement rules, spatial position correlation and inter-frame motion continuity.

[0078] In this embodiment, the local window formed by the marker point at each position in the digital array and the corresponding color block and its eight neighboring color blocks has the following characteristics:

[0079] The color blocks in even rows and even columns are all large color blocks, and the eight adjacent color blocks around them are all small color blocks, containing four colors. There are two groups of three small color blocks of the same color that are consecutively adjacent, and the two groups of consecutive small color blocks are separated by two small color blocks of different colors.

[0080] The color blocks in even rows and odd columns are all small color blocks. There are 6 small color blocks and 2 large color blocks in the eight neighboring color blocks around them, containing four colors. There are two groups of three small color blocks of the same color that are consecutively adjacent. The two groups of color blocks are separated by two different color blocks, and the color block itself will not appear in the eight neighboring color blocks.

[0081] The odd-numbered rows and even-numbered columns are all small color blocks. There are 6 small color blocks and 2 large color blocks in the eight neighboring color blocks around it. There are five kinds of color blocks. There are no three consecutive color blocks of the same color. Moreover, the color block itself appears in the eight neighboring color blocks.

[0082] The odd-numbered rows and columns are all small color blocks. The eight neighboring color blocks around it are 4 small color blocks and 4 large color blocks, containing five colors. There are three consecutive adjacent color blocks with no matching colors, and the color block itself appears in the eight neighboring color blocks.

[0083] The spatial characteristics of each color block and its eight neighboring color blocks in the marker array are recorded in the matching matrix Matching_matrix to prepare for subsequent marker matching.

[0084] In this embodiment, step S2 specifically includes the following steps:

[0085] S21, when the sensor's elastic body is not in contact with any object, images captured by the left and right camera modules are acquired simultaneously and used as the initial left and right standard images, respectively. The preprocessing of the acquired initial left and right standard images involves performing adaptive threshold segmentation, erosion and dilation operations in sequence to enhance the color block boundaries and contrast.

[0086] S22, Perform contour detection on the processed image and calculate the centroid of each contour as the center coordinate of the corresponding color block. By using the pixel coordinates of the center of each color block in the image, establish the positional relationship of the color block marker points and complete the initial calibration of the system.

[0087] S23. Use a binocular vision camera to take pictures of the calibration board from multiple angles, and collect 20 to 30 calibration images that meet the specifications. The calibration solution obtains a complete parameter set consisting of camera intrinsic parameters, extrinsic parameters, distortion coefficients and stereo correction parameters. Based on the complete parameter set, subsequent images are corrected and stereo corrected.

[0088] S24, using the corrected left and right images of the elastic body when it is not in contact with any object as standard images, provides a reference coordinate system in an undeformed state for 3D reconstruction, and performs projection transformation on the acquired subsequent images;

[0089] S25, in subsequent frames, for each contour region, calculate the Euclidean distance between the average pixel value within the contour and the reference color in RGB color, determine the color with the smallest distance as the recognition result of the color block corresponding to the current marker point, assign a unique index to each successfully detected color block in the current frame, so that the current frame has three complete pieces of information: center coordinates, color attributes and index identifier.

[0090] In this embodiment, step S4 specifically includes the following steps:

[0091] S41, each marker point has a unique index. When a marker point matches, the indices of the current marker point and the surrounding 8 neighboring marker points are recorded in the corresponding positions of the statistical matrix index_vote_matrix. The current marker point is recorded twice, and the 8 neighboring marker points are recorded once. After the matching is completed, the index with the most index counts at each position of index_vote_matrix is ​​found as the matching result. In the index matrix matrix_serialnumbe, with the pixel position of the color block corresponding to the current marker point in the image as the center, a spiral search method is used to retrieve neighboring color blocks. Under the condition of satisfying the preset maximum traversal radius, the 8 neighboring color blocks closest to the current marker point are selected according to the displacement distance, and the 8 neighboring color blocks are sorted in the range of 0° to 360° according to the polar angle relative to the positive x-axis.

[0092] S42, combining the spatial displacement constraints of each marker point in the current frame with the absolute value of the RGB difference with the corresponding marker point in the previous frame and the continuity constraints within the preset threshold range, select candidate positions that satisfy the color mode and geometric relationship matching among the sorted neighboring color blocks.

[0093] S43, when the color block corresponding to the current marker point is a large color block, based on the pattern that all eight neighboring color blocks around it are small color blocks and contain four colors and have two sets of three small color blocks of the same color that are consecutively adjacent, the current marker point is determined to be located in the even-numbered row and even-numbered column. Starting from this row, a spiral search is performed along the row direction until the final precise position of the current marker point is determined in the color block marker point array and the corresponding index is used in the statistical matrix.

[0094] S44, when the color block corresponding to the current marker point is a small color block, a branch determination is made based on whether there is a color block with the same color as the current marker point among its eight neighboring color blocks: if there is a color block with the same color, the current marker point is determined to be located in an even row and an odd column, and the row is determined based on the color pattern and geometric distribution of the neighboring color blocks. A spiral search is used within that row to determine the precise position; if there is no color block with the same color, the current marker point is determined to be located in an odd row, and the row is determined based on the color pattern and geometric distribution of the neighboring color blocks. A spiral search is used within that row to determine the precise position.

[0095] S45. After traversing all the color blocks corresponding to the marked points, perform frequency statistics on the index values ​​recorded at each position in the statistical matrix index_vote_matrix, and determine the index that appears most frequently as the final color block identifier for that position.

[0096] In this embodiment, in step S3, when some marker points are briefly occluded or slightly shifted in the current frame, the inter-frame tracking mechanism searches for color blocks in the adjacent previous frame that are close to the current marker point and whose sum of absolute RGB differences is within a preset threshold (the value of which can be determined by calibration or experiment) based on the matching results and motion trend of the previous frame, so as to realize continuous tracking of the occluded marker points.

[0097] In this embodiment, in step S41, when matching each marker point, the row and column of the starting position of the current marker point's color block are first calculated. Then, based on the spatial characteristics of the 8 neighboring markers and the current marker point's color block, the calculated row is used to find the row where the current marker point's color block is located by spiral matching in the Matching_matrix. In this row, the calculated column is used to find the column where the current marker point's color block is located by spiral matching in the Matching_matrix. After the marker point is matched, the result is saved to the statistical matrix index_vote_matrix.

[0098] When calculating the starting position, a linear calculation is performed based on the position of the color patch corresponding to the current marker point in the index matrix matrix_serialnumbe, as shown in the following formula:

[0099]

[0100] in,( , () is the starting position for matching, , ) represents the coordinates of the center of the current color block in the index matrix matrix_serialnumber. , ) represents the index position of the first color block in the top-left corner of the index matrix matrix_serialnumber. This represents the average spacing between adjacent color blocks in the index matrix matrix_serialnumber.

[0101] In this embodiment, in steps S5 and S6, the disparity information of the corresponding marker point is calculated based on the pixel coordinates of the same marker point in the left and right views acquired by the binocular camera, such as... Figure 10 As shown, if the left image takes the color block in row 15 and column 25, the right image also takes the color block in the same row and column. Based on the color block index, the center pixel coordinates of the left and right color blocks are obtained respectively. and The disparity value d is calculated using the left and right coordinates. p (Unit is pixels), that is: ;

[0102] Then, by combining triangulation to calculate the coordinates of the color block center in three-dimensional space, point cloud data describing the surface morphology of the object is generated, and the parallax d is used to... p The depth Z can then be calculated, that is: ;in, Focal length Baseline length;

[0103] Based on the principle of pinhole imaging and using triangulation, the three-dimensional coordinates of the point can be calculated given the depth Z and the camera's intrinsic parameters.

[0104] ;

[0105] The depth data of the contact surface is recovered, and the three-dimensional coordinates of the center of each marker point in space are obtained by triangulation, thereby generating the original point cloud data. The original point cloud data is preprocessed by filtering, denoising and hole filling. The Poisson surface reconstruction algorithm is used to fit the preprocessed point cloud data into a complete three-dimensional surface model by constructing implicit functions and solving the Poisson equation.

[0106] In this embodiment, in step S6, the collected raw point cloud data needs to be processed sequentially through three core preprocessing steps: outlier removal, normal estimation, and surface reconstruction, in order to ensure the accuracy and reliability of the 3D model.

[0107] In the outlier removal stage, statistical filtering or radius filtering, two classic and efficient noise reduction methods, are preferred.

[0108] In the normal estimation stage, the local neighborhood point set of each point is obtained based on KNN or fixed radius nearest neighbor search algorithm. Then, the normal direction is solved by two mainstream methods: one is to fit the three-dimensional plane in the local neighborhood using the least squares method; the other is to reduce the dimension of the neighborhood point set by the principal component analysis algorithm.

[0109] In the surface reconstruction stage, using the optimized normal field as a constraint, the Poisson surface reconstruction algorithm is employed. By constructing implicit functions and solving the Poisson equation, the point cloud data is fitted into a complete 3D surface model; for example... Figure 13 As shown, this is a non-contact point cloud image, such as... Figure 14 As shown, a contact diagram of a pentagonal prism and a legendary elastomer is used, which demonstrates that the visual-tactile sensor can clearly reproduce the surface morphology of the contacting object.

[0110] A visual-tactile sensor includes a sleeve 1, an elastomer 2, a camera mounting plate 3, a miniature camera module 4, and a base fixing component 5. The elastomer 2 is made of highly elastic, low-hysteresis silicone material. A color block marker array is provided on the surface of the elastomer 2 to be contacted. Two miniature camera modules 4 are arranged opposite each other at a 12° tilt angle and assembled on the base fixing component 5. Two camera mounting plates 3 are detachably connected to the base fixing component 5 by fasteners to fix the miniature camera modules 4. The miniature camera modules 4 are binocular vision cameras used to acquire left and right images of the color block marker array under different contact states.

[0111] In actual operation, the object under test comes into contact with the surface of the elastic body 2, causing the elastic body to undergo local deformation, which in turn causes the color block marker array to deform in space. The miniature camera module 4 acquires the image of the deformed color block array through the observation window formed between the sleeve and the elastic body, and transmits the image to the processing unit to execute steps S5 and S6 to obtain the three-dimensional morphology of the contact area.

[0112] The above description is merely a preferred embodiment of the present invention and is illustrative rather than restrictive. Those skilled in the art will understand that many changes, modifications, and even equivalents can be made within the spirit and scope defined by the claims of the present invention, all of which will fall within the protection scope of the present invention.

Claims

1. A 3D reconstruction method based on color block marker array optimization, characterized in that, The three-dimensional reconstruction method includes the following steps: S1, Construct a color block marker array: Select four color blocks with significant color differences as basic marker units. Each color corresponds to two color blocks of different sizes, forming eight distinguishable codes, each representing two different numbers, which together form eight distinguishable number identifiers. Generate a number array based on this, and then generate a color block marker array based on the number array. S2, Constructing a matching strategy: After the stereo calibration and image preprocessing of the binocular vision camera are completed, the color block features of a single marker point in the color block marker array and the color block distribution information of the eight neighboring marker points around the marker point are used as the matching basis to establish multi-dimensional positional constraints including color features, neighborhood distribution and spatial geometric relationships, so that the current frame has three complete pieces of information. S3, Establish an inter-frame tracking mechanism: Extract the initial position and encoding information of the color block in the previous frame image from the three complete information items, use the initial position and encoding information as the prior for the current frame marker point matching and position prediction, take advantage of the small spatial displacement of the marker point between adjacent frames and the continuous change of RGB values, and construct a dynamic tracking link for the marker point by predicting the motion trajectory of the marker point in the continuous frame image and associating it with historical matching information. S4 combines the matching strategy and the inter-frame tracking mechanism to perform candidate neighborhood retrieval for each marker point in the current frame, and quickly determine the current position of the color block corresponding to the current marker point in the color block marker point array based on displacement distance filtering and row positioning rules, and assign a unique index to each successfully matched marker point; S5. Based on the pixel coordinates of the same marker point in the left and right views acquired by the binocular vision camera, calculate the disparity information of the corresponding marker point, recover the depth data of the contact surface, and obtain the three-dimensional coordinates of the center of each marker point in space through triangulation, thereby generating the original point cloud data. S6 preprocesses the original point cloud data and uses the Poisson surface reconstruction algorithm to fit the preprocessed point cloud data into a complete three-dimensional surface model by constructing implicit functions and solving the Poisson equation.

2. The 3D reconstruction method based on color block marker array optimization as described in claim 1, characterized in that, In step S1, the eight distinguishable codes are filled to form a two-dimensional color block marker array according to the pre-set array arrangement rules. Within a 13×13 window centered on any marker, the color and spatial distribution combination of the central color block and its eight surrounding neighboring color blocks is unique within the 13×13 window, thereby uniquely determining the position of each marker in the array through the spatial position correlation of the array. In the color block marker array, large color blocks represent odd numbers, and small color blocks represent even numbers; The overall size of the color block marker array is 2cm. The color block is 2cm long, the center-to-center spacing between adjacent color blocks is 0.5mm, the number of color blocks is 1600, and the marker density is 400 markers / cm. 2 .

3. The 3D reconstruction method based on color block marker array optimization as described in claim 1, characterized in that, The color block marker array has a size of 40×40, and the array contains: Odd rows are composed of small blocks of the same color. The color of each odd row does not repeat the color combination of the previous three odd rows. All odd rows in rows 1-8 use only small blocks of color. The color blocks in odd columns of even rows are determined by the color blocks in the corresponding columns of the two odd rows above and below that even row. The colors of the color blocks in adjacent odd columns within the same row are different. The color blocks in odd columns of even rows are small color blocks, and the color blocks in even columns of even rows are large color blocks. Furthermore, within the same even row, the color of any large color block in an even column is not the same as the color of the large color blocks in the three even columns to its left in the row containing that color block.

4. The 3D reconstruction method based on color block marker array optimization as described in claim 1, characterized in that, The color block marker array is formed by cyclically filling an 8-row, 8-column grid, with all large color blocks in the same even-numbered column being the same color.

5. The 3D reconstruction method based on color block marker array optimization as described in claim 1, characterized in that, The local window formed by the marker point at each position in the digital array and the corresponding color block, together with its eight neighboring color blocks, has the following characteristics: The color blocks in even rows and even columns are all large color blocks, and the eight adjacent color blocks around them are all small color blocks, containing four colors. There are two groups of three small color blocks of the same color that are consecutively adjacent, and the two groups of consecutive small color blocks are separated by two small color blocks of different colors. The color blocks in even rows and odd columns are all small color blocks. There are 6 small color blocks and 2 large color blocks in the eight neighboring color blocks around them, containing four colors. There are two groups of three small color blocks of the same color that are consecutively adjacent. The two groups of color blocks are separated by two different color blocks, and the color block itself will not appear in the eight neighboring color blocks. The odd-numbered rows and even-numbered columns are all small color blocks. There are 6 small color blocks and 2 large color blocks in the eight neighboring color blocks around it. There are five kinds of color blocks. There are no three consecutive color blocks of the same color. Moreover, the color block itself appears in the eight neighboring color blocks. The odd-numbered rows and columns are all small color blocks. The eight neighboring color blocks around it are 4 small color blocks and 4 large color blocks, containing five colors. There are three consecutive adjacent color blocks with no matching colors, and the color block itself appears in the eight neighboring color blocks. The spatial characteristics of each color block and its eight neighboring color blocks in the marker array are recorded in the matching matrix Matching_matrix to prepare for subsequent marker matching.

6. The 3D reconstruction method based on color block marker array optimization as described in claim 1, characterized in that, Step S2 specifically includes the following steps: S21, when the sensor's elastic body is not in contact with any object, images captured by the left and right camera modules are acquired simultaneously and used as the initial left and right standard images, respectively. The acquired initial left and right standard images are preprocessed and adaptive threshold segmentation, erosion and dilation operations are performed in sequence. S22, Perform contour detection on the processed image and calculate the centroid of each contour as the center coordinate of the corresponding color block. By using the pixel coordinates of the center of each color block in the image, establish the positional relationship of the color block marker points and complete the initial calibration of the system. S23. Use a binocular vision camera to take pictures of the calibration board from multiple angles, and collect 20 to 30 calibration images that meet the specifications. The calibration solution obtains a complete parameter set consisting of camera intrinsic parameters, extrinsic parameters, distortion coefficients and stereo correction parameters. Based on the complete parameter set, subsequent images are corrected and stereo corrected. S24, using the corrected left and right images of the elastic body when it is not in contact with any object as standard images, provides a reference coordinate system in an undeformed state for 3D reconstruction, and performs projection transformation on the acquired subsequent images; S25. For each contour region, calculate the Euclidean distance between the average pixel value within the contour and the reference color in RGB color and base color. Determine the color with the smallest distance as the recognition result of the color block corresponding to the current marker point. Assign a unique index to each successfully detected color block in the current frame, so that the current frame has three complete pieces of information: center coordinates, color attributes and index identifier.

7. The 3D reconstruction method based on color block marker array optimization as described in claim 1, characterized in that, Step S4 specifically includes the following steps: S41, Match each marker point to obtain the statistical matrix index_vote_matrix. In the statistical matrix index_vote_matrix, take the pixel position of the color block corresponding to the current marker point in the image as the center, and use a spiral search method to retrieve neighboring color blocks. Under the condition of satisfying the preset maximum traversal radius, filter out the 8 neighboring color blocks closest to the current marker point according to the displacement distance, and sort the 8 neighboring color blocks in the range of 0° to 360° according to the polar angle relative to the positive x-axis direction. S42, combining the spatial displacement constraints of each marker point in the current frame with the absolute value of the RGB difference with the corresponding marker point in the previous frame and the continuity constraints within the preset threshold range, select candidate positions that satisfy the color mode and geometric relationship matching among the sorted neighboring color blocks. S43, when the color block corresponding to the current marker point is a large color block, based on the pattern that all eight neighboring color blocks around it are small color blocks and contain four colors and have two sets of three small color blocks of the same color that are consecutively adjacent, the current marker point is determined to be located in the even-numbered row and even-numbered column. Starting from this row, a spiral search is performed along the row direction until the final precise position of the current marker point is determined in the color block marker point array and the corresponding index is used in the statistical matrix. S44, when the color block corresponding to the current marker point is a small color block, a branch determination is made based on whether there is a color block with the same color as the current marker point among its eight neighboring color blocks: if there is a color block with the same color, the current marker point is determined to be located in an even row and an odd column, and the row is determined based on the color pattern and geometric distribution of the neighboring color blocks. A spiral search is used within that row to determine the precise position; if there is no color block with the same color, the current marker point is determined to be located in an odd row, and the row is determined based on the color pattern and geometric distribution of the neighboring color blocks. A spiral search is used within that row to determine the precise position. S45. After traversing all the color blocks corresponding to the marked points, perform frequency statistics on the index values ​​recorded at each position in the statistical matrix index_vote_matrix, and determine the index that appears most frequently as the final color block identifier for that position.

8. The 3D reconstruction method based on color block marker array optimization as described in claim 1, characterized in that, In step S3, when some marker points are briefly occluded or slightly shifted in position in the current frame, the inter-frame tracking mechanism searches for color blocks in the adjacent previous frame that are close to the current marker point and whose sum of absolute RGB differences is within a preset threshold range, based on the matching results and motion trends of the previous frame, to achieve continuous tracking of the occluded marker points.

9. A visual-tactile sensor, used in the three-dimensional reconstruction method based on color block marker array optimization as described in any one of claims 1-8, characterized in that, The device includes a sleeve (1), an elastomer (2), a camera mounting plate (3), a miniature camera module (4), and a base fixing component (5). The elastomer (2) is made of highly elastic, low-hysteresis silicone material. The surface of the elastomer (2) to be contacted is provided with an array of color block markers. The two miniature camera modules (4) are arranged opposite each other at a 12° tilt angle and assembled on the base fixing component (5). The two camera mounting plates (3) are detachably connected to the base fixing component (5) by fasteners to fix the miniature camera modules (4). The miniature camera module (4) is a binocular vision camera used to acquire left and right images of the color block marker array under different contact states.

Images