Mirror surface object detection method and system based on single-phase measurement deflection and tactile perception fusion

Abstract

The application discloses a mirror surface object detection method and system based on single-phase measurement deflection and touch perception fusion, projects a stripe pattern with a fixed phase shift on the surface of a mirror surface object through a Mono-PMD system, captures stripe deformation of the mirror surface area by using a camera, and obtains relative height information of the surface; introduces a touch sensor to directly measure absolute height of key points, overcomes the problem that the Mono-PMD can only provide relative height and has ambiguity of height and gradient; then based on height data obtained by the touch sensor, uses a polynomial fitting method to combine the relative height measured by vision measurement and the absolute height detected by touch detection, and finally realizes three-dimensional appearance reconstruction of the entire mirror surface object. The method effectively solves the problem of ambiguity of height and gradient, and improves the accuracy and efficiency of three-dimensional reconstruction through polynomial fitting.

Description

Technical Field

[0001] The present invention belongs to the technical field of highly reflective surface recognition and detection, and mainly relates to a mirror object detection method and system based on the fusion of single-phase measurement deflectometry and tactile perception. Background Art

[0002] With the rapid development of robotics, robots are increasingly taking on complex, high-precision, and repetitive tasks, such as industrial welding, surgery, and inspection. When performing tasks, robots rely primarily on environmental perception, which is crucial for successful mission completion. Enabling robots to better understand and adapt to their environment by enhancing sensory capabilities (such as vision, touch, and hearing) can help them better understand and adapt to their environment. This is particularly true for three-dimensional shape measurement, where the combination of vision and touch is crucial.

[0003] However, with the advancement of precision manufacturing technology and the increasing demand for product surface quality, highly reflective, mirror-like surfaces, such as mirrors, machined parts, optical reflectors, and polished materials, are increasingly appearing in fields such as industry, optics, medicine, and aerospace. These highly reflective surfaces place higher demands on robotic visual perception. Traditional visual measurement systems are mostly optimized for diffusely reflective surfaces. In these cases, the reflected light is evenly distributed, and the camera or optical detector does not need to be specifically positioned or angled. Although diffusely reflective surfaces may have highly reflective areas, HDR technology can now effectively address this by adjusting the camera's exposure time.

[0004] Typical visual measurement techniques are often based on triangulation, such as binocular vision and structured light technology. These methods have been widely used in industrial scenarios and have high measurement stability and effectiveness. However, for highly reflective mirror objects, since the reflected light is constrained by the surface normal and propagates according to strict optical laws, this poses a great challenge to the visual measurement system, resulting in inaccurate positioning and even incorrect identification. The traditional solution is to spray the mirror surface with an imaging agent to form diffuse reflection. Although this method can improve measurement results, the spraying operation before each measurement is cumbersome and inefficient. Another method is tactile detection. Touch is not affected by the optical properties of the surface, but the point-by-point touch method is inefficient, the data is sparse, and it is impossible to obtain full-field three-dimensional information.

[0005] To address this problem, researchers have proposed a variety of solutions. For example, Petz et al. measured the mirror surface through parallel grating imaging, Savarese et al. analyzed the relationship between geometric imaging and surface topography in the mirror, and Knauer et al. proposed monocular phase measurement deflectometry (Mono-PMD) for measuring highly reflective free-form surfaces. However, Mono-PMD has the ambiguity problem of gradient and height, such asFigure 1 As shown, Figure 1 In the example, point O1 on the mirror originally reflects point A on the screen to the camera. When the mirror only deflects by an angle of β, point A in the camera image becomes point C. When the mirror only translates by a height of h, point A in the camera image becomes point B. The reflected image of the same pixel in the camera, as seen from point A, will experience similar effects due to the mirror's surface gradient or height. Therefore, it is impossible to accurately determine whether the phase change is caused by the height of the highly reflective surface or the surface gradient. Although stereo phase deflectometry (Stereo-PMD) combined with binocular vision overcomes some limitations, it places extremely high demands on robust pixel matching and system calibration. Some monocular PMD-based systems achieve 3D measurement of discontinuous mirrors through specialized optical path combinations. While PMD technology has been proven effective for measuring highly reflective surfaces, its core issue—the ambiguity between gradient and height—has yet to be fully resolved. Furthermore, the accuracy of PMD systems relies heavily on calibration, which is a challenge for PMD systems. Summary of the Invention

[0006] This invention addresses the limitations of traditional visual measurement systems in detecting highly reflective mirrored objects. It provides a mirrored object detection method and system based on the fusion of single-phase measurement deflectometry and tactile perception. Using a Mono-PMD system, a fringe pattern with a fixed phase shift is projected onto the mirrored object surface. A camera captures the fringe deformation in the mirrored area to obtain relative surface height information. A tactile sensor is introduced to directly measure the absolute height of key points, overcoming the ambiguity between height and gradient that Mono-PMD can only provide. Based on the height data obtained by the tactile sensor, a polynomial fitting method is then used to combine the relative height measured by visual measurement with the absolute height detected by tactile measurement, ultimately achieving 3D reconstruction of the entire mirrored object. This method effectively resolves the ambiguity between height and gradient and improves the accuracy and efficiency of 3D reconstruction through polynomial fitting.

[0007] To achieve the above-mentioned object, the technical solution adopted by the present invention is: a mirror object detection method based on the fusion of single-phase measurement deflectometry and tactile perception, comprising the following steps:

[0008] S1. Mirror area recognition: Use a camera to capture the reflection image of the mirror object and identify the mirror area;

[0009] S2. Phase information acquisition: acquiring phase information of the surface of the mirror object by phase measurement deflectometry (PMD), wherein the phase information includes at least phase gradient data;

[0010] S3, phase unpacking and gradient calculation: using a quality-guided phase unpacking algorithm to unpack the phase information obtained in step S2 and perform gradient calculation;

[0011] S4. Calculate relative height: Recover the relative height of the mirror object based on the phase gradient data using an integration method;

[0012] S5. Tactile detection to obtain absolute height: Use the tactile sensor to measure the absolute height and convert it into the height in the world coordinate system;

[0013] S6, visual-tactile fusion: The relative height data of the visual measurement calculated in step S4 is fused with the absolute height data obtained in step S5, and a polynomial is fitted using the least squares method to obtain the overall three-dimensional morphology of the object surface;

[0014] S7. 3D height data output: Output the fused 3D height data to generate 3D shape information of highly reflective mirror objects and complete the detection of mirror objects.

[0015] As an improvement of the present invention, step S1 specifically includes the following steps:

[0016] S11: adjusting the display screen and the shooting angle of the camera, and shooting the display screen image through the mirror area, wherein the display screen image includes two images: pure black and pure white;

[0017] S12: Perform pixel difference operation on the two images obtained in step S11 to obtain a difference image:

[0018] I diff =|I white -I black |

[0019] Among them, I white and I black Respectively represent the grayscale values ​​of the images captured by the camera when the display shows pure white and pure black images;

[0020] S13: Use the Otsu algorithm (OTSU) to binarize the difference image obtained in step S12 to accurately identify the mirror area.

[0021] As an improvement of the present invention, the phase information acquisition in step S2 is specifically as follows: within the mirror area identified in step S1, a sinusoidal fringe pattern with a fixed phase difference is projected onto the mirror surface using a Mono-PMD system, and a camera captures the reflected image of the fringe on the mirror surface to obtain phase gradient data; the specific form of the fringe pattern on the display screen and the phase calculation formula are specifically as follows:

[0022] I n (x,y)=I0+A(x,y)sin(φ(x,y)+2nπ / N)

[0023]

[0024] Where I0 represents the basic light intensity of the display, A(x,y) represents the modulated light intensity, φ(x,y) represents the phase value at that location, N represents the total number of phase shift steps, and n represents the current phase shift step.

[0025] As another improvement of the present invention, the gradient calculation of step S3 is specifically as follows: the measured phase of the free mirror is subtracted from the phase result of the reference plane mirror to obtain the phase difference Δφ(x, y), and the gradient g of the phase difference in the x and y directions is calculated respectively. x With g y :

[0026]

[0027] where φ m (x,y) is the measured phase, φ r (x,y) is the reference phase.

[0028] As another improvement of the present invention, in step S4, the center point of the ROI area is selected as the reference point, and the gradient data in the x-direction and y-direction are integrated once to obtain the relative height on the reference line. The integration result is used as the starting point to integrate the entire area and reconstruct the relative three-dimensional height data of the entire mirror object:

[0029]

[0030] Among them, Z0(x,y) represents the initial reference point, g x 、g y represent the gradients of the phase difference in the x and y directions, respectively.

[0031] As another improvement of the present invention, in step S5, the method for converting the height into the world coordinate system is:

[0032]

[0033] Among them, P world =(X world ,Y world ,Z world ), is the coordinate in the world coordinate system; P tool =(X tool ,Y tool ,Z tool ), is the position of the tool coordinate system where the current touch point is located when the measurement unit of the tactile sensor exceeds the threshold; It is the coordinate system transformation relationship of each joint of the current robotic arm.

[0034] As another improvement of the present invention, the polynomial fitting formula in step S6 is:

[0035]

[0036] Where: k represents the degree of the polynomial, K represents the total number of polynomial terms, A k represents the polynomial coefficients, Z visual is the relative height obtained by integrating the phase difference.

[0037] In order to achieve the above-mentioned purpose, the present invention also adopts the following technical solution: a mirror object detection system based on the fusion of single-phase measurement deflectometry and tactile perception, including a computer program, which implements the steps of any of the above-mentioned methods when executed by a processor.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] (1) Overcoming the ambiguity problem: The method of the present invention solves the problem of height and gradient ambiguity in the traditional PMD method by combining the fusion technology of phase measurement deflectometry and tactile perception, and realizes accurate three-dimensional measurement of objects with highly reflective surfaces.

[0040] (2) Improved measurement accuracy: The method of the present invention ensures the integrity and accuracy of the measurement data by fusing visual relative height and tactile absolute height, and reduces the error of traditional visual systems on highly reflective surfaces.

[0041] (3) Improved measurement efficiency: The method of the present invention uses PMD technology to quickly obtain phase information, which greatly improves the acquisition speed compared to traditional tactile point-by-point measurement. At the same time, the fusion technology compensates for the problem of sparse tactile data.

[0042] (4) Enhanced system robustness: Since it no longer relies on a complex calibration process and provides an absolute height reference through tactile data, the system of the present invention can maintain high measurement stability and accuracy in a variety of complex environments.

[0043] (5) Wide application: The method and system of the present invention are particularly suitable for high-precision mirror detection scenarios in the fields of industry, optics, aerospace, etc., and can provide accurate three-dimensional morphology data for applications such as robot operation and precision machining. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The following is a schematic diagram of the Mono-PMD measurement principle in the prior art;

[0045] Figure 2 Schematic diagram of the measurement principle of obtaining absolute height by tactile detection in step S5 of the method of the present invention;

[0046] Figure 3 This is a flowchart of step S6 of visual-touch fusion of the present invention. DETAILED DESCRIPTION

[0047] The present application will be further clarified by the following examples and figures, which should be understood as merely illustrating the application and not limiting the scope of the application.

[0048] Example 1

[0049] The mirror surface object detection method based on the fusion of single-phase measurement deflection and tactile perception effectively solves the limitations of single method in high-reflective surface detection through the fusion of vision and tactile, improves the measurement accuracy, and enhances the stability and adaptability of the system. The method specifically includes the following steps:

[0050] Step S1, mirror surface area identification:

[0051] The shooting angle of the display screen and the camera is adjusted to realize that the camera can shoot the complete or partial picture of the display screen through the mirror surface area. The display screen displays pure black and pure white images respectively, and the camera shoots the reflection image of the mirror surface object. The pixel difference operation is performed on the two images to obtain a difference image, as shown in formula (1).

[0052] I diff = |I white -I black | (1)

[0053] Wherein, I white and I black respectively represent the gray value of the camera collected image when the display screen displays pure white and pure black images.

[0054] Subsequently, the Otsu algorithm (OTSU) is used to perform binaryzation processing on the difference image, so as to accurately identify the mirror surface area, and provide a basis for subsequent phase extraction and unpacking processing.

[0055] Step S2, phase information acquisition:

[0056] The phase information of the mirror surface object is obtained by phase measurement deflection (PMD). Under the cooperation of the camera and the display screen, the sinusoidal fringe pattern with fixed phase difference is projected to the object surface, and the camera is used to capture the mirror reflection image to obtain the phase gradient data.

[0057] In the identified mirror surface area, the Mono-PMD system is used to project the sinusoidal fringe pattern with fixed phase difference to the mirror surface. The camera captures the reflection image of the fringe on the mirror, and calculates the phase information from four different phase shift images by phase shift technology. The phase value is calculated using the arctangent function, and the phase gradient data is obtained. The specific form of the fringe pattern on the display screen and the phase calculation formula are shown in formulas (2) and (3).

[0058] I n (x,y)=I0+A(x,y)sin(φ(x,y)+2nπ / N) (2)

[0059]

[0060] Step S3: Phase unpacking and gradient calculation:

[0061] The acquired phase data is unpacked using a quality-guided phase unpacking algorithm to ensure the continuity of the phase data and eliminate the error caused by phase jumps.

[0062] Because the phase calculation result of Equation (3) is wrapped in [-π, π), phase unwrapping is required. A quality-guided phase unwrapping algorithm is used to unwrap the phase data of the region of interest (ROI) to obtain a continuous phase distribution and eliminate phase jumps. The unwrapped phase data is then used to further calculate the relative height of the object.

[0063] Subtract the measured phase of the free mirror from the phase of the reference plane mirror to obtain the phase difference △φ(x,y), and calculate the gradient g of the phase difference in the x and y directions respectively. x With g y :

[0064]

[0065] where φ m (x,y) is the measured phase, φ r (x,y) is the reference phase.

[0066] Step S4, calculate relative height:

[0067] The relative height of the mirror object is restored based on the phase gradient data through the integration method.

[0068] Based on the unpacked phase gradient data, the center point of the ROI area is selected as the reference point. The gradient data in the x and y directions are integrated once to obtain the relative height on the reference line. Then, using these integration results as the starting point, the entire area is further integrated to reconstruct the relative 3D height data of the entire mirror object, as shown in formula (5).

[0069]

[0070] Step S5: Acquire absolute height through tactile detection:

[0071] Use tactile sensors to measure absolute height. The force sensor on the robotic arm obtains absolute height information when it touches the surface of the object, and converts this information into height in the world coordinate system through kinematic modeling, such as Figure 2shown.

[0072] Use the tactile sensor on the robot arm to perform tactile detection on mirror objects. The force sensor on the robot arm records the height data of the touch point when it touches the surface of the object as the absolute height information. The specific operation is as follows: Under the observation of the camera, the robot arm is controlled by remote control to touch the highly reflective surface. When the measurement unit of the force sensor on the robot arm exceeds the threshold, the position P of the tool coordinate system where the current touch point is located is recorded. tool =(X tool ,Y tool ,Z tool ) and the coordinate system transformation relationship of each joint of the current robotic arm The position of this point is transformed into the world coordinate system P through kinematic modeling world =(X world ,Y world ,Z world ), the calculation formula is shown in formula (6).

[0073]

[0074] Step S6: visual-touch fusion:

[0075] The relative height data measured by vision is integrated with the absolute height data provided by the tactile sensor. The least squares method is used to fit the polynomial, combining the relative height of vision and the absolute height of tactile to obtain the overall three-dimensional shape of the object surface, such as Figure 3 shown.

[0076] In order to avoid multiple recordings of the same contact point or contact points that are too close, if the distance between the current contact point and the existing contact point does not exceed the set storage threshold, the contact point will not be recorded. After the acquisition is completed, all recorded contact points form a tactile point cloud, and the height value Z is extracted from it. world Using the absolute height points obtained by the tactile sensor, the polynomial coefficients are fitted using the least squares method, and the relative height data measured by vision are fused with the absolute height data detected by tactile detection. Finally, the absolute three-dimensional shape of the entire mirror object in the world coordinate system is calculated using the polynomial fitting formula. The formula is shown in (7).

[0077]

[0078] Where: n represents the degree of the polynomial, A n Represents the polynomial coefficients.

[0079] Step S7: Output of three-dimensional height data:

[0080] After data fusion, the fused three-dimensional height data is output. This three-dimensional data reflects the actual shape of the mirror object, completes the detection of the mirror object, and can be used in scenarios such as high-precision detection and surface analysis.

[0081] Through the above steps, the present invention can effectively combine visual and tactile information, overcome the limitations of traditional methods in detecting highly reflective surfaces, and improve the accuracy of three-dimensional shape reconstruction of mirror objects.

[0082] In summary, the method and system of the present invention overcome the ambiguity problem of height and gradient in the detection of highly reflective mirror objects in traditional visual measurement systems, and effectively alleviate the problem that the optical complexity of highly reflective surfaces makes it difficult to accurately capture the three-dimensional shape. Although a single tactile detection method can provide absolute height information, the data is sparse and the acquisition efficiency is low, and it is impossible to completely reconstruct the three-dimensional shape of the entire object. The method of the present invention combines the fusion technology of vision and tactile measurement to solve the problem of height and gradient ambiguity, and improves the accuracy and efficiency of three-dimensional reconstruction through polynomial fitting, and also enhances the stability and adaptability of the system. It is suitable for scenarios such as industry, optics, and medicine that require high-precision mirror detection.

[0083] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A mirror object detection method based on the fusion of single-phase measurement deflectometry and tactile perception, characterized in that: The steps include: S1. Mirror area recognition: Use a camera to capture the reflection image of the mirror object and identify the mirror area; S2. Phase information acquisition: acquiring phase information of the surface of the mirror object by phase measurement deflectometry (PMD), wherein the phase information includes at least phase gradient data; S3, phase unpacking and gradient calculation: using a quality-guided phase unpacking algorithm to unpack the phase information obtained in step S2 and perform gradient calculation; S4. Calculate relative height: Recover the relative height of the mirror object based on the phase gradient data using an integration method; S5. Tactile detection to obtain absolute height: Use the tactile sensor to measure the absolute height and convert it into the height in the world coordinate system; S6, visual-tactile fusion: The relative height data of the visual measurement calculated in step S4 is fused with the absolute height data obtained in step S5, and a polynomial is fitted using the least squares method to obtain the overall three-dimensional morphology of the object surface; S7. 3D height data output: Output the fused 3D height data to generate 3D shape information of highly reflective mirror objects and complete the detection of mirror objects.

2. The method for detecting mirror objects based on the fusion of single-phase measurement deflectometry and tactile perception according to claim 1, characterized in that: The step S1 specifically includes the following steps: S11: adjusting the display screen and the shooting angle of the camera, and shooting the display screen image through the mirror area, wherein the display screen image includes two images: pure black and pure white; S12: Perform pixel difference operation on the two images obtained in step S11 to obtain a difference image: ; in, and Respectively represent the grayscale values ​​of the images captured by the camera when the display shows pure white and pure black images; S13: Use the Otsu algorithm (OTSU) to binarize the difference image obtained in step S12 to accurately identify the mirror area.

3. The method for detecting mirror objects based on the fusion of single-phase measurement deflectometry and tactile perception according to claim 2, characterized in that: The phase information acquisition in step S2 is specifically as follows: within the mirror area identified in step S1, a sinusoidal fringe pattern with a fixed phase difference is projected onto the mirror surface using a Mono-PMD system, and a camera captures the reflected image of the fringe on the mirror surface to obtain phase gradient data; the specific form of the fringe pattern on the display screen and the phase calculation formula are specifically as follows: ; ; in, Indicates the basic light intensity of the display. represents the modulated light intensity, Indicates the phase value at that location, represents the total number of phase shift steps, Indicates the current phase shift step number.

4. The method for detecting mirror objects based on the fusion of single-phase measurement deflectometry and tactile perception according to claim 3, characterized in that: The gradient calculation of step S3 is specifically as follows: subtract the measured phase of the free mirror from the phase result of the reference plane mirror to obtain the phase difference , calculate the gradient of the phase difference in the x and y directions respectively and : ; in is the measured phase, is the reference phase.

5. The method for detecting mirror objects based on the fusion of single-phase measurement deflectometry and tactile perception according to claim 4, characterized in that: In step S4, the center point of the ROI area is selected as the reference point, and the gradient data in the x-direction and y-direction are integrated once to obtain the relative height on the reference line. The integration result is used as the starting point to integrate the entire area and reconstruct the relative three-dimensional height data of the entire mirror object: ; in, represents the initial reference point, 、 represent the gradients of the phase difference in the x and y directions, respectively.

6. The method for detecting mirror objects based on the fusion of single-phase measurement deflectometry and tactile perception according to claim 1, characterized in that: In step S5, the method for converting the height into the world coordinate system is: ; in, , is the coordinate in the world coordinate system; , is the position of the tool coordinate system where the current touch point is located when the measurement unit of the tactile sensor exceeds the threshold; It is the coordinate system transformation relationship of each joint of the current robotic arm.

7. The method for detecting mirror objects based on the fusion of single-phase measurement deflectometry and tactile perception according to claim 1, characterized in that: The polynomial fitting formula in step S6 is: ; Among them: k represents the degree of the polynomial, K represents the total number of polynomial terms, represents the polynomial coefficients, is the relative height obtained by integrating the phase difference.

8. A system for detecting mirrored objects based on the fusion of single-phase measurement deflectometry and tactile perception, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images