Specular object detection method and system based on fusion of single-phase measurement deflectometry and tactile sensing

Abstract

A specular object detection method and system based on the fusion of single-phase measurement deflectometry and tactile sensing. A fringe pattern with a fixed phase shift is projected onto the surface of a specular object by means of a mono-PMD system, and fringe deformation in a specular area is captured by means of a camera, so as to acquire relative height information of the surface; a tactile sensor is introduced to directly measure the absolute heights of key points, so as to overcome the problem of there being an ambiguity between the height and the gradient because a mono-PMD can only provide relative heights; and then, on the basis of height data acquired by the tactile sensor, a polynomial fitting method is used to combine the relative heights obtained by means of visual measurement and the absolute heights obtained by means of tactile detection, so as to finally realize three-dimensional topography reconstruction for the whole specular object.

Description

Mirror object detection method and system based on fusion of single-phase measurement deflectometry and tactile perception Technical Field

[0001] The present invention belongs to the technical field of highly reflective surface recognition and detection, and mainly relates to a mirror object detection method and system based on the fusion of single-phase measurement deflectometry and tactile perception. Background Art

[0002] With the rapid development of robotics, robots are increasingly taking on complex, high-precision, and repetitive tasks, such as industrial welding, surgery, and inspection. When performing tasks, robots rely primarily on environmental perception, which is crucial for successful mission completion. Enabling robots to better understand and adapt to their environment by enhancing sensory capabilities (such as vision, touch, and hearing) can help them better understand and adapt to their environment. This is particularly true for three-dimensional shape measurement, where the combination of vision and touch is crucial.

[0003] However, with the advancement of precision manufacturing technology and the increasing demand for product surface quality, highly reflective, mirror-like surfaces, such as mirrors, machined parts, optical reflectors, and polished materials, are increasingly appearing in fields such as industry, optics, medicine, and aerospace. These highly reflective surfaces place higher demands on robotic visual perception. Traditional visual measurement systems are mostly optimized for diffusely reflective surfaces. In these cases, the reflected light is evenly distributed, and the camera or optical detector does not need to be specifically positioned or angled. Although diffusely reflective surfaces may have highly reflective areas, HDR technology can now effectively address this by adjusting the camera's exposure time.

[0004] Typical visual measurement techniques are often based on triangulation, such as binocular vision and structured light technology. These methods have been widely used in industrial scenarios and have high measurement stability and effectiveness. However, for highly reflective mirror objects, since the reflected light is constrained by the surface normal and propagates according to strict optical laws, this poses a great challenge to the visual measurement system, resulting in inaccurate positioning and even incorrect identification. The traditional solution is to spray the mirror surface with an imaging agent to form diffuse reflection. Although this method can improve measurement results, the spraying operation before each measurement is cumbersome and inefficient. Another method is tactile detection. Touch is not affected by the optical properties of the surface, but the point-by-point touch method is inefficient, the data is sparse, and it is impossible to obtain full-field three-dimensional information.

[0005] To address this issue, researchers have proposed various solutions. For example, Petz et al. measured mirror surfaces using parallel grating imaging, Savarese et al. analyzed the relationship between geometric imaging and surface topography in mirrors, and Knauer et al. proposed monocular phase measurement deflectometry (Mono-PMD) for measuring highly reflective free-form surfaces. However, Mono-PMD suffers from the ambiguity between gradient and height. As shown in Figure 1, point O1 on the mirror surface originally reflects point A on the screen to the camera. When the mirror surface only deflects by an angle β, point A on the camera image becomes point C. When the mirror surface only translates by a height h, point A on the camera image becomes point B. The reflected image of the same pixel in the camera, as seen from point A, will be affected similarly by the gradient or height of the mirror surface. Therefore, it is impossible to accurately determine whether the phase change is caused by the height of the highly reflective surface or the surface gradient. Although stereo phase deflectometry (Stereo-PMD) combined with binocular vision overcomes some of these limitations, it places extremely high demands on the robustness of pixel matching and system calibration. Some monocular PMD-based systems achieve 3D measurement of discontinuous mirror surfaces through specialized optical path combinations. While extensive research has demonstrated the effectiveness of PMD technology for measuring highly reflective surfaces, the core issue—the ambiguity between gradient and height—has yet to be fully resolved. Furthermore, the accuracy of PMD systems relies heavily on calibration, which presents a challenge.

[0006] Summary of the Invention

[0007] This invention addresses the limitations of traditional visual measurement systems in detecting highly reflective mirrored objects. It provides a mirrored object detection method and system based on the fusion of single-phase measurement deflectometry and tactile perception. Using a Mono-PMD system, a fringe pattern with a fixed phase shift is projected onto the mirrored object surface. A camera captures the fringe deformation in the mirrored area to obtain relative surface height information. A tactile sensor is introduced to directly measure the absolute height of key points, overcoming the ambiguity between height and gradient that Mono-PMD can only provide. Based on the height data obtained by the tactile sensor, a polynomial fitting method is then used to combine the relative height measured by visual measurement with the absolute height detected by tactile measurement, ultimately achieving 3D reconstruction of the entire mirrored object. This method effectively resolves the ambiguity between height and gradient and improves the accuracy and efficiency of 3D reconstruction through polynomial fitting.

[0008] To achieve the above-mentioned object, the technical solution adopted by the present invention is: a mirror object detection method based on the fusion of single-phase measurement deflectometry and tactile perception, comprising the following steps:

[0009] S1. Mirror area recognition: Use a camera to capture the reflection image of the mirror object and identify the mirror area;

[0010] S2. Phase information acquisition: acquiring phase information of the surface of the mirror object by phase measurement deflectometry (PMD), wherein the phase information includes at least phase gradient data;

[0011] S3, phase unpacking and gradient calculation: using a quality-guided phase unpacking algorithm to unpack the phase information obtained in step S2 and perform gradient calculation;

[0012] S4. Calculate relative height: Recover the relative height of the mirror object based on the phase gradient data using an integration method;

[0013] S5. Tactile detection to obtain absolute height: Use the tactile sensor to measure the absolute height and convert it into the height in the world coordinate system;

[0014] S6, visual-tactile fusion: The relative height data of the visual measurement calculated in step S4 is fused with the absolute height data obtained in step S5, and a polynomial is fitted using the least squares method to obtain the overall three-dimensional morphology of the object surface;

[0015] S7. 3D height data output: Output the fused 3D height data to generate 3D shape information of highly reflective mirror objects and complete the detection of mirror objects.

[0016] As an improvement of the present invention, step S1 specifically includes the following steps:

[0017] S11: adjusting the display screen and the shooting angle of the camera, and shooting the display screen image through the mirror area, wherein the display screen image includes two images: pure black and pure white;

[0018] S12: Perform pixel difference operation on the two images obtained in step S11 to obtain a difference image:

[0019] I diff =|I white -I black |

[0020] Among them, I white and I black Respectively represent the grayscale values ​​of the images captured by the camera when the display shows pure white and pure black images;

[0021] S13: Use the Otsu algorithm (OTSU) to binarize the difference image obtained in step S12 to accurately identify the mirror area.

[0022] As an improvement of the present invention, the phase information acquisition in step S2 is specifically as follows: within the mirror area identified in step S1, a sinusoidal fringe pattern with a fixed phase difference is projected onto the mirror surface using a Mono-PMD system, and a camera captures the reflected image of the fringe on the mirror surface to obtain phase gradient data; the specific form of the fringe pattern on the display screen and the phase calculation formula are specifically as follows:

[0023] I n (x,y)=I0+A(x,y)sin(φ(x,y)+2nπ / N)

[0024] Where I0 represents the basic light intensity of the display, A(x,y) represents the modulated light intensity, φ(x,y) represents the phase value at that location, N represents the total number of phase shift steps, and n represents the current phase shift step.

[0025] As another improvement of the present invention, the gradient calculation of step S3 is specifically as follows: the measured phase of the free mirror is subtracted from the phase result of the reference plane mirror to obtain the phase difference Δφ(x, y), and the gradient g of the phase difference in the x and y directions is calculated respectively. x With g y :

[0026] where φ m (x,y) is the measured phase, φ r (x,y) is the reference phase.

[0027] As another improvement of the present invention, in step S4, the center point of the ROI area is selected as the reference point, and the gradient data in the x-direction and y-direction are integrated once to obtain the relative height on the reference line. The integration result is used as the starting point to integrate the entire area and reconstruct the relative three-dimensional height data of the entire mirror object:

[0028] Among them, Z0(x,y) represents the initial reference point, g x 、g y represent the gradients of the phase difference in the x and y directions, respectively.

[0029] As another improvement of the present invention, in step S5, the method for converting the height into the world coordinate system is:

[0030] Among them, P world =(X world ,Y world ,Z world ), is the coordinate in the world coordinate system; P tool =(X tool ,Y tool ,Ztool ), is the position of the tool coordinate system where the current touch point is located when the measurement unit of the tactile sensor exceeds the threshold; It is the coordinate system transformation relationship of each joint of the current robotic arm.

[0031] As another improvement of the present invention, the polynomial fitting formula in step S6 is:

[0032] Where: k represents the degree of the polynomial, K represents the total number of polynomial terms, A k represents the polynomial coefficients, Z visua l is the relative height obtained by integrating the phase difference.

[0033] In order to achieve the above-mentioned purpose, the present invention also adopts the following technical solution: a mirror object detection system based on the fusion of single-phase measurement deflectometry and tactile perception, including a computer program, which implements the steps of any of the above-mentioned methods when executed by a processor.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) Overcoming the ambiguity problem: The method of the present invention solves the problem of height and gradient ambiguity in the traditional PMD method by combining the fusion technology of phase measurement deflectometry and tactile perception, and realizes accurate three-dimensional measurement of objects with highly reflective surfaces.

[0036] (2) Improved measurement accuracy: The method of the present invention ensures the integrity and accuracy of the measurement data by fusing visual relative height and tactile absolute height, and reduces the error of traditional visual systems on highly reflective surfaces.

[0037] (3) Improved measurement efficiency: The method of the present invention uses PMD technology to quickly obtain phase information, which greatly improves the acquisition speed compared to traditional tactile point-by-point measurement. At the same time, the fusion technology compensates for the problem of sparse tactile data.

[0038] (4) Enhanced system robustness: Since it no longer relies on a complex calibration process and provides an absolute height reference through tactile data, the system of the present invention can maintain high measurement stability and accuracy in a variety of complex environments.

[0039] (5) Wide application: The method and system of the present invention are particularly suitable for high-precision mirror detection scenarios in the fields of industry, optics, aerospace, etc., and can provide accurate three-dimensional morphology data for applications such as robot operation and precision machining. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] FIG1 is a schematic diagram of a Mono-PMD measurement principle in the prior art;

[0041] FIG2 is a diagram showing the measurement principle of obtaining absolute height by tactile detection in step S5 of the method of the present invention;

[0042] FIG3 is a flowchart of step S6 of visual-touch fusion according to the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0044] Example 1

[0045] A mirror object detection method based on the fusion of single-phase measurement deflectometry and tactile perception effectively overcomes the limitations of a single method in detecting highly reflective surfaces by integrating vision and touch. This method not only improves measurement accuracy but also enhances the stability and adaptability of the system. The method specifically includes the following steps:

[0046] Step S1: Mirror area identification:

[0047] Adjust the display screen and camera angles so that the camera can capture the entire or partial display screen through the mirrored area. Display pure black and pure white images on the display screen, respectively, and use the camera to capture the reflected image of the mirrored object. Perform a pixel difference operation on these two images to obtain a differential image, as shown in Equation (1).

[0048] I diff =|I white -I black | (1)

[0049] Among them, I white and I black Respectively represent the grayscale values ​​of the images captured by the camera when the display shows pure white and pure black images.

[0050] Subsequently, the difference image is binarized using the Otsu algorithm (OTSU) to accurately identify the mirror area, providing a basis for subsequent phase extraction and unpacking processing.

[0051] Step S2: Acquisition of phase information:

[0052] Phase measurement deflectometry (PMD) is used to obtain phase information on the surface of a mirrored object. Using a camera and display, a sinusoidal fringe pattern with a fixed phase difference is projected onto the surface of the object. The camera then captures the mirror reflection image to obtain phase gradient data.

[0053] Within the identified mirror area, a Mono-PMD system projects a sinusoidal fringe pattern with a fixed phase difference onto the mirror surface. A camera captures the reflected image of the fringe on the mirror surface, and phase information is calculated from four phase-shifted images with different phases using phase shifting technology. The phase value is calculated using the inverse tangent function, and phase gradient data is obtained. The specific form of the fringe pattern on the display and the phase calculation formula are shown in Equations (2) and (3).

[0054] I n (x,y)=I0+A(x,y)sin(φ(x,y)+2nπ / N) (2)

[0055] Step S3: Phase unpacking and gradient calculation:

[0056] The acquired phase data is unpacked using a quality-guided phase unpacking algorithm to ensure the continuity of the phase data and eliminate the error caused by phase jumps.

[0057] Because the phase calculation result of Equation (3) is wrapped in [-π, π), phase unwrapping is required. A quality-guided phase unwrapping algorithm is used to unwrap the phase data of the region of interest (ROI) to obtain a continuous phase distribution and eliminate phase jumps. The unwrapped phase data is then used to further calculate the relative height of the object.

[0058] The phase difference △φ(x,y) is obtained by subtracting the measured phase of the free mirror from the phase result of the reference plane mirror, and the gradient g of the phase difference in the x and y directions is calculated respectively. x With g y :

[0059] where φ m (x,y) is the measured phase, φ r (x,y) is the reference phase.

[0060] Step S4, calculate relative height:

[0061] The relative height of the mirror object is restored based on the phase gradient data through the integration method.

[0062] Based on the unpacked phase gradient data, the center point of the ROI area is selected as the reference point. The gradient data in the x and y directions are integrated once to obtain the relative height on the reference line. Then, using these integration results as the starting point, the entire area is further integrated to reconstruct the relative 3D height data of the entire mirror object, as shown in formula (5).

[0063] Step S5: Acquire absolute height through tactile detection:

[0064] Absolute height measurement is performed using a tactile sensor. The force sensor on the robotic arm acquires absolute height information when it touches the surface of an object. This information is then converted to height in the world coordinate system through kinematic modeling, as shown in Figure 2.

[0065] Use the tactile sensor on the robot arm to perform tactile detection on mirror objects. The force sensor on the robot arm records the height data of the touch point when it touches the surface of the object as the absolute height information. The specific operation is as follows: Under the observation of the camera, the robot arm is controlled by remote control to touch the highly reflective surface. When the measurement unit of the force sensor on the robot arm exceeds the threshold, the position P of the tool coordinate system where the current touch point is located is recorded. tool =(X tool ,Y tool ,Z tool ) and the coordinate system transformation relationship of each joint of the current robotic arm The position of this point is transformed into the world coordinate system P through kinematic modeling world =(X world ,Y world ,Z world ), the calculation formula is shown in formula (6).

[0066] Step S6: visual-touch fusion:

[0067] The relative height data from visual measurements is fused with the absolute height data from the tactile sensor. Using the least squares method to fit a polynomial, the overall three-dimensional topography of the object surface is obtained by combining the visual relative height and the tactile absolute height, as shown in Figure 3.

[0068] In order to avoid multiple recordings of the same contact point or contact points that are too close, if the distance between the current contact point and the existing contact point does not exceed the set storage threshold, the contact point will not be recorded. After the acquisition is completed, all recorded contact points form a tactile point cloud, and the height value Z is extracted from it. world Using the absolute height points obtained by the tactile sensor, the polynomial coefficients are fitted using the least squares method, and the relative height data measured by vision are fused with the absolute height data detected by tactile detection. Finally, the absolute three-dimensional shape of the entire mirror object in the world coordinate system is calculated using the polynomial fitting formula. The formula is shown in (7).

[0069] Where: n represents the degree of the polynomial, A n Represents the polynomial coefficients.

[0070] Step S7: Output of three-dimensional height data:

[0071] After data fusion, the fused three-dimensional height data is output. This three-dimensional data reflects the actual shape of the mirror object, completes the detection of the mirror object, and can be used in scenarios such as high-precision detection and surface analysis.

[0072] Through the above steps, the present invention can effectively combine visual and tactile information, overcome the limitations of traditional methods in detecting highly reflective surfaces, and improve the accuracy of three-dimensional shape reconstruction of mirror objects.

[0073] In summary, the method and system of the present invention overcome the ambiguity problem of height and gradient in the detection of highly reflective mirror objects in traditional visual measurement systems, and effectively alleviate the problem that the optical complexity of highly reflective surfaces makes it difficult to accurately capture the three-dimensional shape. Although a single tactile detection method can provide absolute height information, the data is sparse and the acquisition efficiency is low, and it is impossible to completely reconstruct the three-dimensional shape of the entire object. The method of the present invention combines the fusion technology of vision and tactile measurement to solve the problem of height and gradient ambiguity, and improves the accuracy and efficiency of three-dimensional reconstruction through polynomial fitting, and also enhances the stability and adaptability of the system. It is suitable for scenarios such as industry, optics, and medicine that require high-precision mirror detection.

[0074] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A mirror object detection method based on the fusion of single-phase measurement deflectometry and tactile perception, characterized in that: The steps include: S1. Mirror area recognition: Use a camera to capture the reflection image of the mirror object and identify the mirror area; S2. Phase information acquisition: acquiring phase information of the surface of the mirror object by phase measurement deflectometry (PMD), wherein the phase information includes at least phase gradient data; S3, phase unpacking and gradient calculation: using a quality-guided phase unpacking algorithm to unpack the phase information obtained in step S2 and perform gradient calculation; S4. Calculate relative height: Recover the relative height of the mirror object based on the phase gradient data using an integration method; S5. Tactile detection to obtain absolute height: Use the tactile sensor to measure the absolute height and convert it into the height in the world coordinate system; S6, visual-tactile fusion: The relative height data of the visual measurement calculated in step S4 is fused with the absolute height data obtained in step S5, and a polynomial is fitted using the least squares method to obtain the overall three-dimensional morphology of the object surface; S7. 3D height data output: Output the fused 3D height data to generate 3D shape information of highly reflective mirror objects and complete the detection of mirror objects.

2. The method for detecting mirror objects based on the fusion of single-phase measurement deflectometry and tactile perception according to claim 1, characterized in that: The step S1 specifically includes the following steps: S11: adjusting the display screen and the shooting angle of the camera, and shooting the display screen image through the mirror area, wherein the display screen image includes two images: pure black and pure white; S12: Perform pixel difference operation on the two images obtained in step S11 to obtain a difference image: I diff =|I white -I black | Among them, I white and I black Respectively represent the grayscale of the image captured by the camera when the display shows pure white and pure black images value; S13: Use the Otsu algorithm (OTSU) to binarize the difference image obtained in step S12 to accurately identify the mirror area.

3. The method for detecting mirror objects based on the fusion of single-phase measurement deflectometry and tactile perception according to claim 2, characterized in that: The phase information acquisition in step S2 is specifically as follows: within the mirror area identified in step S1, a sinusoidal fringe pattern with a fixed phase difference is projected onto the mirror surface using a Mono-PMD system, and a camera captures the reflected image of the fringe on the mirror surface to obtain phase gradient data; the specific form of the fringe pattern on the display screen and the phase calculation formula are specifically as follows: n (x,y)=I0+A(x,y)sin(φ(x,y)+2nπ / N) Where I0 represents the basic light intensity of the display, A(x,y) represents the modulated light intensity, φ(x,y) represents the phase value at that location, N represents the total number of phase shift steps, and n represents the current phase shift step.

4. The method for detecting mirror objects based on the fusion of single-phase measurement deflectometry and tactile perception according to claim 3, characterized in that: The gradient calculation in step S3 is specifically as follows: subtract the measured phase of the free mirror from the phase result of the reference plane mirror to obtain the phase difference Δφ(x,y), and calculate the gradient g of the phase difference in the x and y directions respectively. x With g y : where φ m (x,y) is the measured phase, φ r (x,y) is the reference phase.

5. The method for detecting mirror objects based on the fusion of single-phase measurement deflectometry and tactile perception according to claim 4, characterized in that: In step S4, the center point of the ROI area is selected as the reference point, and the gradient data in the x-direction and the y-direction are integrated once to obtain the relative height on the reference line. The integration result is used as the starting point to integrate the entire area and reconstruct the relative three-dimensional height data of the entire mirror object: Among them, Z0(x,y) represents the initial reference point, g x 、g y represent the gradients of the phase difference in the x and y directions, respectively.

6. The method for detecting mirror objects based on the fusion of single-phase measurement deflectometry and tactile perception according to claim 1, characterized in that: In step S5, the method for converting the height into the world coordinate system is: Among them, P world =(X world ,Y world ,Z world ), is the coordinate in the world coordinate system; P tool =(X tool ,Y tool ,Z tool ), is the position of the tool coordinate system where the current touch point is located when the measurement unit of the tactile sensor exceeds the threshold; It is the coordinate system transformation relationship of each joint of the current robotic arm.

7. The method for detecting mirror objects based on the fusion of single-phase measurement deflectometry and tactile perception according to claim 1, characterized in that: The polynomial fitting formula in step S6 is: Where: k represents the degree of the polynomial, K represents the total number of polynomial terms, A k represents the polynomial coefficients, Z visual is the relative height obtained by integrating the phase difference.

8. A system for detecting mirrored objects based on the fusion of single-phase measurement deflectometry and tactile perception, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of any one of the above methods are implemented.

Images