Fusion method and system for three-dimensional imaging and medium

Abstract

The invention provides a three-dimensional imaging fusion method and system and a medium. The three-dimensional imaging fusion method provided by the invention comprises the following steps: acquiring a first fringe pattern and a second fringe pattern reflected by the surface of an object to be measured; performing phase calculation on the first fringe pattern to obtain an initial gradient map; performing phase calculation on the second fringe pattern to obtain an initial height map; calculating the contrast ratio of the central pixel point of the current window in each fringe pattern, and judging the reflection type; respectively calculating a fusion coefficient corresponding to each fringe pattern according to the reflection type, the difference value of the contrast ratio and a set difference value threshold value; constructing an energy function based on the fusion coefficient, the initial gradient map and the initial height map; and solving to obtain a fused final height map by minimizing an energy function, and generating a three-dimensional point cloud to realize three-dimensional reconstruction. According to the three-dimensional imaging fusion method based on the FPP and the PMD, the limitation of a single technology is overcome, and the adaptability and integrity of measurement are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a fusion method, system and medium for three-dimensional imaging. Background Technology

[0002] In recent years, optical measurement of the three-dimensional morphology of object surfaces has become an important research direction in application fields such as precision machining, industrial inspection, and reverse engineering. The reflective properties of object surface materials are mainly diffuse reflection and specular reflection.

[0003] Currently, most three-dimensional measurements of object surfaces are performed on a single property. Structured light fringe projection technology is used to measure the surface of objects with diffuse reflection properties, while phase deflection is used to measure the surface of objects with specular reflection properties. However, in actual production processes, there are many composite surfaces that may have both diffuse and specular reflection properties. Using only one method cannot yield good three-dimensional shape measurement results.

[0004] Chinese patent CN120141345A proposes a device for acquiring images of the surface of an object under test, an image selection method, and a defect detection method. This patent integrates phase deflection technology and fringe projection contour measurement into a single defect detection system. For products with complex materials and varying surface roughness, it enables rapid switching between two three-dimensional defect detection technologies, improving detection efficiency. However, this patent only relates to defect detection and differs from the technical problems, methods, and effects described in this solution.

[0005] Chinese patent CN117685905A proposes a multi-modal fusion method, apparatus, device, and storage medium for three-dimensional imaging. This patent projects an coded pattern onto the surface of an object under test using a projection device and a display device, acquiring reflection data and determining the three-dimensional coordinates. It then calculates the modulation coefficients of all three-dimensional coordinates and sets a threshold to fuse the two sets of coordinates into a single system. This patent differs from our proposed solution in its fusion approach, i.e., in its technical methods.

[0006] Chinese patent CN120953500A discloses a method and system for reconstructing the surface of complex reflective objects based on multi-system point cloud fusion. This patent projects the target object using both the projection grating phase method and the phase deflection method, and acquires the structured light patterns of both methods via a camera. It then performs phase calculations on the acquired structured light patterns and, combined with joint calibration information, solves the absolute phases separately to complete independent 3D reconstruction. Finally, it identifies highly reflective areas using a full-white image, achieving adaptive replacement and fusion at the point cloud level to generate a 3D topography model. This patent differs from our proposed solution in its fusion approach, i.e., in its technical methods.

[0007] Existing technologies for achieving 3D reconstruction results by fusing phase-refraction and fringe projection profilometry involve performing 3D reconstructions using both methods separately, and then fusing the two reconstruction results based on calibration results or point cloud matching to obtain the final reconstruction result. However, the fusion of 3D reconstruction results obtained using different imaging principles may result in stitching defects, leading to lower accuracy and quality in the final reconstructed image. Summary of the Invention

[0008] This invention proposes a fusion method, system, and medium for three-dimensional imaging, which solves at least one of the aforementioned technical problems.

[0009] To achieve the above objectives, the present invention proposes the following technical solution:

[0010] A 3D imaging fusion method, comprising:

[0011] Acquire the first and second fringe patterns of reflection from the same physical region on the surface of the object under test;

[0012] The first fringe pattern is obtained based on the principle of phase deflection imaging; the second fringe pattern is obtained based on the principle of fringe projection imaging.

[0013] The first stripe pattern is used to obtain the initial gradient map through phase calculation; the second stripe pattern is used to obtain the initial height map through phase calculation.

[0014] In each stripe pattern, the ratio of the difference between the maximum and minimum pixel values ​​in each sliding window to the difference between the maximum and minimum pixel values ​​in the current stripe pattern is used as the contrast of the center pixel of the current window.

[0015] Calculate the difference between the first contrast of the first stripe pattern and the second contrast of the second stripe pattern corresponding to the same physical location, determine the reflection type of the current pixel, and calculate the fusion coefficient corresponding to each stripe pattern in combination with the set difference threshold.

[0016] Construct an energy function, using the fusion coefficients corresponding to the initial height map and the second stripe map to constrain the consistency between the final height map and the initial height map, and using the fusion coefficients corresponding to the initial gradient map and the first stripe map to constrain the consistency between the gradient of the final height map and the initial gradient map; solve for the final height map by minimizing the energy function;

[0017] Based on the final height map, a 3D point cloud is generated to achieve 3D reconstruction.

[0018] Furthermore, at least one sinusoidal fringe period is used as the pixel range corresponding to the sliding window, and the period of the sinusoidal fringe is 2π.

[0019] Further, reflection types include:

[0020] If the difference between the first contrast and the second contrast is greater than zero, it indicates that the first contrast is greater than the second contrast, and the current position tends to be of the specular reflection type; if the difference between the first contrast and the second contrast is less than zero, it indicates that the first contrast is less than the second contrast, and the current position tends to be of the diffuse reflection type.

[0021] Furthermore, the difference threshold increases with the increase of the absolute difference between the first and second stripe patterns; the difference threshold a∈[0.5,1];

[0022] The absolute difference is the absolute difference between the pixel mean of the first stripe pattern and the second stripe pattern, or the absolute difference between the pixel median, or the absolute difference between a specified pixel value of the first stripe pattern and a specified pixel value of the second stripe pattern.

[0023] Furthermore, based on the difference Δδ between the first contrast and the second contrast and the difference threshold a, the reflection type of the current pixel is determined:

[0024] If -1 < Δδ ≤ -a, the reflection type of the physical location corresponding to this pixel is diffuse reflection;

[0025] If -a < Δδ ≤ 0, the reflection type of the physical location corresponding to this pixel is a mixed reflection with diffuse reflection as the main component.

[0026] If 0 < Δδ < a, the reflection type of the physical location corresponding to this pixel is a mixed reflection with specular reflection as the main component.

[0027] If Δδ≥a, the reflection type of the physical location corresponding to this pixel is specular reflection.

[0028] Furthermore, the type of reflection is determined based on the difference between the first contrast ratio and the second contrast ratio;

[0029] Reflection thresholds are set in the first and second stripe maps respectively. Based on the relationship between the pixel corresponding to the current physical location and the reflection thresholds of the first and second stripe maps respectively, the thinning reflection type corresponding to the current physical location is determined.

[0030] Furthermore, the fusion coefficient includes:

[0031] In mixed reflection dominated by specular reflection, the fusion coefficient λ1 = Δδ + a corresponding to the first fringe pattern, and the fusion coefficient λ2 = 1 - a - Δδ corresponding to the second fringe pattern;

[0032] In mixed reflection dominated by diffuse reflection, the fusion coefficient λ1 corresponding to the first fringe pattern is 1-a-|Δδ|, and the fusion coefficient λ2 corresponding to the second fringe pattern is a+|Δδ|.

[0033] In specular reflection, the fusion coefficient λ1 = 1 for the first fringe pattern and the fusion coefficient λ2 = 0 for the second fringe pattern.

[0034] In diffuse reflection, the fusion coefficient λ1 = 0 for the first fringe pattern and the fusion coefficient λ2 = 1 for the second fringe pattern.

[0035] Where Δδ is the difference between the first contrast ratio and the second contrast ratio, and a is the difference threshold.

[0036] Furthermore, the energy function also includes a smoothing regularization term to suppress noise.

[0037] Based on the same inventive concept, this application also proposes a three-dimensional imaging fusion system, comprising:

[0038] The stripe pattern generation module projects the stripe pattern onto the surface of the object under test based on the principle of phase deflection imaging and the principle of stripe projection imaging.

[0039] The camera acquires a first fringe pattern and a second fringe pattern of reflection from the same physical region on the surface of the object under test; wherein the first fringe pattern is obtained based on the principle of phase deflection imaging; and the second fringe pattern is obtained based on the principle of fringe projection imaging.

[0040] The acquisition module acquires the first and second fringe patterns of reflection from the same physical region on the surface of the object under test.

[0041] The phase calculation module obtains the initial gradient map from the first fringe pattern through phase calculation, and obtains the initial height map from the second fringe pattern through phase calculation.

[0042] The coefficient calculation module uses the ratio of the difference between the maximum and minimum pixel values ​​in each sliding window to the difference between the maximum and minimum pixel values ​​in the current fringe pattern as the contrast of the center pixel of the current window.

[0043] Calculate the difference between the first contrast of the first stripe pattern and the second contrast of the second stripe pattern corresponding to the same physical location, determine the reflection type of the current pixel, and calculate the fusion coefficient corresponding to each stripe pattern in combination with the set difference threshold.

[0044] The function computation module constructs an energy function, using the fusion coefficients corresponding to the initial height map and the second stripe map to constrain the consistency between the final height map and the initial height map, and using the fusion coefficients corresponding to the initial gradient map and the first stripe map to constrain the consistency between the gradient of the final height map and the initial gradient map; the final height map is solved by minimizing the energy function.

[0045] The 3D reconstruction module generates a 3D point cloud from the final height map, thus achieving 3D reconstruction.

[0046] On the other hand, the present invention also proposes a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the method described above.

[0047] Based on the same inventive concept, this application also proposes an electronic device, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus.

[0048] Memory, used to store computer programs;

[0049] The processor implements the above method when executing programs stored in memory.

[0050] The beneficial effects of the embodiments of the present invention are as follows:

[0051] The proposed 3D imaging fusion method based on FPP and PMD effectively fills the measurement blind spot of FPP in the specular region with the high-quality gradient information obtained by PMD; and the reliable height information provided by FPP supplements the measurement of PMD in the diffuse reflection region. Therefore, this scheme can perform complete 3D reconstruction of complex object surfaces containing diffuse reflection, specular reflection and mixed reflection without blind spots, overcome the limitations of single technology, greatly improve the adaptability and completeness of measurement, and meet the precision measurement needs of industrial inspection, optical manufacturing and other fields.

[0052] Compared to existing fusion schemes, the fusion method proposed in this application can pre-calculate the fusion coefficients corresponding to different reflection types based on the original fringe pattern, serving as adaptive weighting factors for the FPP and PMD data terms in the energy function. During subsequent 3D reconstruction, based on the reflection type determination, in the specular reflection (predominant) region, the fusion coefficient corresponding to the first fringe pattern is increased, with phase-deflection imaging dominating the reconstruction result; in the diffuse reflection (predominant) region, the fusion coefficient corresponding to the second fringe pattern is increased, with fringe projection imaging dominating the reconstruction result; furthermore, a smoothing regularization term is set to suppress noise. This mechanism effectively suppresses the interference of local noise and outliers on the final result, ensuring the stability of the reconstruction process and the reliability of the results, enhancing measurement robustness and noise resistance.

[0053] The energy function proposed in this application uses the height map of the FPP (Focused Proportional to Precision Peening) as a global constraint and the gradient map of the PMD (Profound Gradient Map) as a local detail constraint, effectively combining the measurement advantages of both FPP and PMD. The final reconstructed height map retains the global accuracy of FPP while incorporating the high-resolution details of PMD, generating a 3D model of quality far exceeding that of any single technique, clearly presenting minute features such as scratches, textures, and etchings. Attached Figure Description

[0054] Figure 1 This is a schematic diagram illustrating the principle of phase deflection;

[0055] Figure 2 This is a schematic diagram illustrating the principle of fringe projection;

[0056] Figure 3 This is a flowchart illustrating the fusion method for three-dimensional imaging proposed in this invention;

[0057] Figure 4 This is a schematic diagram of the electronic device proposed in this invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising”, “including”, etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0060] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0061] like Figure 1 As shown, a phase measurement system (PMD) typically consists of a light source, a camera, and the object being measured.

[0062] In general, the display screen acts as a diffuse light source, illuminating the structured light pattern onto the surface of the object under test, while the camera is responsible for capturing the structured light pattern that has been modulated and distorted by the surface of the object under test.

[0063] Structured light is commonly used to add feature patterns to featureless surfaces to mark different locations on the surface. Common structured light patterns include phase-shifting fringes, Gray code, and speckle. Among them, phase-shifting fringes are the most commonly used structured light pattern in phase measurement deflection due to their high resolution and strong noise resistance.

[0064] Assuming the distances between the display screen, the surface of the object under test, and the camera are known, the surface gradient information of the object under test can be obtained using the slope equation.

[0065] Since the light path is reversible, the light path is described in reverse order. Any ray emitted from a pixel in the camera image is reflected by point s on the reference plane to point q on the display screen. If the angle between point s on the surface of the object being measured and the reference plane is θ, then the angle between the light rays sq and so emitted by the camera is 2θ.

[0066] In Δsqo, let ∠osq=α. According to the Law of Sines, there exists

[0067] If the period of the stripes on the display screen is P, then |oq| represents the phase change in the fringe pattern.

[0068] According to the above formula, we have This allows us to derive the gradient tanθ of the surface of the object under test.

[0069] Generally, if α≈90° and θ is very small, the calculation formula can be simplified to:

[0070] The simplified formula shows that the gradient information of the object under test is modulated and converted into the modulation phase of the reflection fringe. After processing with phase extraction and unfolding techniques, the phase offset Δ in the horizontal and vertical directions corresponding to each camera pixel can be accurately obtained. x Δ y This transforms the phase into a gradient distribution in two perpendicular directions, tan2θ. x tan2θ y To obtain three-dimensional height distribution information, an integral algorithm or other spatial domain algorithm is used to calculate the three-dimensional surface shape of the object under test.

[0071] like Figure 2 As shown, a fringe projection profilometry (FPP) measurement system typically consists of a projector, a camera, and the object to be measured.

[0072] Projector imaging features a large field of view and strong adaptability, making it suitable for acquiring the overall shape of objects, and it has high reconstruction stability, especially for diffuse reflection areas.

[0073] The projector projects structured light patterns sequentially onto the surface of the object under test according to a set order. The patterns employ a typical phase encoding scheme. This process corresponds to the projection grating phase method, which utilizes the phase distortion formed by structured light on the surface of the object to recover the three-dimensional contour, making it particularly suitable for diffuse reflective surfaces. Simultaneously, a camera acquires a sequence of deformed image sequences of the patterns on the object's surface, thus providing data for subsequent phase calculation and 3D point cloud generation.

[0074] Generally, phase-coded patterns, including but not limited to step stripe patterns, Gray code stripe patterns, sinusoidal stripe patterns, binary stripe patterns, and phase-shifted stripe patterns, can be used for structured light coding patterns that encode pixels one by one. Typically, four-step or eight-step phase-shifted stripe patterns are chosen as structured light patterns.

[0075] The specific principle is as follows: by projecting phase-shifted fringes onto the surface of the object under test, the fringes will deform due to the change in the height of the object's surface. By acquiring images of these deformed fringes and processing them to obtain the phase representing the object's height information, the three-dimensional contour information of the object can be calculated.

[0076] Based on the above measurement principle, this application proposes a three-dimensional imaging fusion system, comprising:

[0077] The stripe pattern generation module projects the stripe pattern onto the surface of the object under test based on the principle of phase deflection imaging and the principle of stripe projection imaging.

[0078] The camera acquires a first fringe pattern and a second fringe pattern of reflections from the same physical region on the surface of the object under test. The first fringe pattern is obtained based on the principle of phase deflection imaging; the second fringe pattern is obtained based on the principle of fringe projection imaging.

[0079] Preferably, the image acquisition of the first fringe pattern and the second fringe pattern can be regarded as two independent modules. The same camera can be used to acquire the fringe pattern (first fringe pattern) on the surface of the object under test based on the phase deflection imaging principle and the fringe pattern (second fringe pattern) on the surface of the object under test based on the fringe projection principle.

[0080] The stripe pattern generation module corresponding to the first stripe pattern can set the display screen as a diffuse reflection light source to illuminate the stripe pattern onto the surface of the object to be tested; the stripe pattern generation module corresponding to the second stripe pattern can set a projector to project the stripe pattern onto the surface of the object to be tested.

[0081] Alternatively, patent CN120141345B can be referenced, which uses a projector, a diffuser, and a reflector as a stripe pattern generation module. The projector projects the stripe pattern. The diffuser diffuses the stripes projected by the projector onto the surface of the object to be measured. The reflector moves between a first position and a second position relative to the object. In the first position, the reflector blocks the light path from the projector to the diffuser and reflects the projected stripes onto the surface of the object perpendicularly. In the second position, the reflector avoids the light path from the projector to the diffuser; the distance from the projector's stripes to the diffuser is equal to the distance from the projector's stripes to the surface of the object when the reflector is in the first position, thus eliminating the need for focusing during switching.

[0082] The camera is used to receive the signal reflected from the surface of the object after the diffuser is incident on it, in order to obtain a phase deflection technique (PMD) image (first fringe pattern), or to receive the signal diffused from the surface of the object after it is incident on the surface of the object, in order to obtain a fringe projection profilometry (FPP) image (second fringe pattern).

[0083] The acquisition module acquires the first and second fringe patterns of reflection from the same physical region on the surface of the object under test.

[0084] The phase calculation module obtains the initial gradient map from the first fringe map through phase calculation, and obtains the initial height map from the second fringe map through phase calculation.

[0085] The coefficient calculation module sets a sliding window in each stripe pattern and uses the ratio of the difference between the maximum and minimum pixel values ​​in each window to the difference between the maximum and minimum pixel values ​​in the current stripe pattern as the contrast of the center pixel of the current window.

[0086] Calculate the difference between the first contrast of the pixel in the first stripe image and the second contrast of the pixel in the second stripe image corresponding to the same physical location to determine the reflection type of the current pixel; calculate the fusion coefficient corresponding to the pixel in each stripe image based on the reflection type, the difference in contrast and the set difference threshold.

[0087] The function computation module constructs an energy function, using the fusion coefficients corresponding to the initial height map and the second stripe map to constrain the consistency between the final height map and the initial height map, and using the fusion coefficients corresponding to the initial gradient map and the first stripe map to constrain the consistency between the gradient of the final height map and the initial gradient map; by minimizing the energy function, the fused final height map is obtained.

[0088] The 3D reconstruction module generates a 3D point cloud from the final height map (converting it into a point cloud or mesh model) to achieve 3D reconstruction.

[0089] like Figure 3As shown, based on the above system, this embodiment proposes the following method, including:

[0090] Acquire the first and second fringe patterns of reflection from the surface of the object under test;

[0091] The first fringe pattern is an image of the surface of the object under test obtained based on the principle of phase deflection imaging; the second fringe pattern is an image of the surface of the object under test obtained based on the principle of fringe projection imaging; the first fringe pattern and the second fringe pattern correspond to the same physical region.

[0092] The first stripe pattern is used to obtain the initial gradient map through phase calculation; the second stripe pattern is used to obtain the initial height map through phase calculation.

[0093] In each stripe pattern, a sliding window is set, and the ratio of the difference between the maximum and minimum pixel values ​​in each window to the difference between the maximum and minimum pixel values ​​in the current stripe pattern is used as the contrast of the center pixel of the current window.

[0094] The contrast δ (δ∈[-1,1]) of the current window center pixel includes the first contrast δ1 and the second contrast δ2;

[0095] The first contrast ratio is the contrast ratio of the center pixel of the current window in the first stripe image, including: the ratio of the difference between the maximum and minimum pixel values ​​in each window of the first stripe image to the difference between the maximum and minimum pixel values ​​in the first stripe image.

[0096] The second contrast ratio is the contrast ratio of the center pixel of the current window in the second stripe image, including: the ratio of the difference between the maximum and minimum pixel values ​​in each window of the second stripe image to the difference between the maximum and minimum pixel values ​​in the second stripe image.

[0097] Preferably, at least one sinusoidal fringe period is used as the pixel range corresponding to the sliding window.

[0098] In this application, the first and second stripe images are sinusoidal structured light images. The period of the sinusoidal stripes is 2π, and there are several periods in a single (stripe) image. The maximum and minimum gray values ​​(i.e., amplitudes) of each period are slightly different. To further improve the accuracy of the calculation results, a sinusoidal stripe pattern with a high stripe period frequency can be selected to increase the number of sliding windows.

[0099] Calculate the difference Δδ between the first contrast of the pixel in the first fringe image and the second contrast of the pixel in the second fringe image at the same physical location, and determine the reflection type of the current pixel. Δδ∈[-1,1].

[0100] Depending on the application scenario of phase deflection, the reflection type of the object under test surface can be divided into specular reflection, mixed reflection (specular reflection + diffuse reflection), and diffuse reflection. The fusion coefficient is determined based on the determination of the reflection type of the object under test surface. In practical scenarios, mixed reflection is predominant.

[0101] If the difference between the first contrast and the second contrast (Δδ=δ1-δ2) is greater than zero, it indicates that the first contrast is greater than the second contrast, meaning that the light energy at the current position mainly enters the PMD imaging optical path through specular reflection, and the current position tends to be of the specular reflection type. If the specific bias of the mixed reflection is not further determined, it is regarded as a mixed reflection dominated by specular reflection, and fusion calculation is performed.

[0102] If the difference between the first contrast and the second contrast (Δδ=δ1-δ2) is less than zero, it indicates that the first contrast is less than the second contrast, meaning that the light energy at the current position mainly enters the FPP imaging optical path through specular reflection, and the current position tends to be of the diffuse reflection type. If the specific bias of the mixed reflection is not further determined, it is regarded as a mixed reflection dominated by diffuse reflection, and fusion calculation is performed.

[0103] If the current hardware conditions are limited, the type of mixed reflection corresponding to the surface of the object under test can be preliminarily determined based on the difference Δδ between the first contrast and the second contrast.

[0104] The difference threshold increases as the absolute difference between the first and second stripe patterns increases; the difference threshold a∈[0.5,1]. Here, the absolute difference is the absolute difference between the pixel mean values ​​of the first and second stripe patterns, or the absolute difference between the pixel median values, or the absolute difference between a specified pixel value of the first stripe pattern and a specified pixel value of the second stripe pattern.

[0105] If the absolute difference between the pixel mean values ​​of the two fringe patterns is larger, it indicates that the mixed reflection type of the surface of the object under test tends to be an ideal specular reflection or an ideal diffuse reflection surface. In this case, the value of 'a' can be increased, such as to 0.8 or 0.9. If the absolute difference between the pixel mean values ​​of the two fringe patterns is smaller, the value of 'a' can be set to a smaller value, such as to 0.55 or 0.6.

[0106] Based on the difference Δδ between the first contrast and the second contrast and the difference threshold a, determine the reflection type of the current pixel:

[0107] If -1 < Δδ ≤ -a, the reflection type of the physical location corresponding to this pixel is diffuse reflection;

[0108] If -a < Δδ ≤ 0, the reflection type of the physical location corresponding to this pixel is a mixed reflection with diffuse reflection as the main component.

[0109] If 0 < Δδ < a, the reflection type of the physical location corresponding to this pixel is a mixed reflection with specular reflection as the main component.

[0110] If Δδ≥a, the reflection type of the physical location corresponding to this pixel is specular reflection.

[0111] Preferably, in addition to setting a difference threshold to analyze the contrast difference and determine the type of refined reflection, the type of refined reflection can also be determined based on the pixel distribution of the current stripe pattern.

[0112] Specifically, it includes:

[0113] The type of reflection is determined based on the difference between the first contrast ratio and the second contrast ratio;

[0114] Reflection thresholds are set in the first and second stripe maps respectively. Based on the relationship between the pixel corresponding to the current physical location and the reflection thresholds of the first and second stripe maps respectively, the thinning reflection type corresponding to the current physical location is determined.

[0115] To further improve the accuracy of the final fusion calculation, this application sets reflection thresholds (including a first threshold, a second threshold, a third threshold, and a fourth threshold) to determine the mixed reflection type of the current pixel. The first and second thresholds are set based on the pixel values ​​of the first stripe pattern, with the first threshold being greater than the second threshold. The third and fourth thresholds are set based on the pixel values ​​of the second stripe pattern, with the third threshold being greater than the fourth threshold.

[0116] Preferably, the reflection threshold can be set based on experience, or obtained by weighted calculation based on the maximum and minimum pixel values ​​in the current image (the first threshold and the second threshold are calculated by weighting the maximum and minimum pixel values ​​in the first stripe image; the third threshold and the fourth threshold are calculated by weighting the maximum and minimum pixel values ​​in the second stripe image).

[0117] For example, the first and third thresholds can be set to 1.2 times the average pixel value of the current image, and the second and fourth thresholds can be set to 0.7 times the average pixel value of the current image; or the first and third thresholds can be set to 0.75 times the maximum pixel value in the current image, and the second and fourth thresholds can be set to 0.35 times the maximum pixel value in the current image.

[0118] Based on the numerical relationship between the current reflection threshold and the current image pixel value distribution, a judgment strategy corresponding to mixed reflection is set. This embodiment is only for illustrative purposes; the judgment strategy is adjusted according to the threshold setting relationship.

[0119] If, provided that the first contrast ratio is greater than the second contrast ratio, the pixel value corresponding to the current position in the first stripe pattern is greater than the first threshold and the pixel value corresponding to the current position in the second stripe pattern is less than the fourth threshold, then it indicates that the current position is a specular reflection.

[0120] If, given that the first contrast is less than the second contrast, the pixel value corresponding to the current position in the first stripe pattern is less than the second threshold and the pixel value corresponding to the current position in the second stripe pattern is greater than the third threshold, then it indicates that the current position is diffuse reflection.

[0121] If the pixel value corresponding to the current position in the first fringe image is greater than the second threshold and less than the first threshold, and the pixel value corresponding to the current position in the second fringe image is greater than the pixel median (or pixel mean or fourth threshold) of the second fringe image, then it is a mixed reflection dominated by diffuse reflection (provided that the first contrast is less than the second contrast). If the pixel value corresponding to the current position in the second fringe image is less than the fourth threshold, then it is a mixed reflection dominated by specular reflection (provided that the first contrast is greater than the second contrast). The pixel median of the second fringe image should be less than the third threshold and greater than the fourth threshold; ideally, the pixel mean should tend towards the pixel median.

[0122] Similarly, the second fringe pattern can be used as the primary indicator to determine mixed reflection: if the pixel value corresponding to the current position in the second fringe pattern is less than the third threshold and greater than the fourth threshold, and the pixel value corresponding to the current position in the first fringe pattern is greater than the median (or mean, or second threshold) of the pixels in the first fringe pattern, then it is mixed reflection dominated by specular reflection (provided that the first contrast is greater than the second contrast); if the pixel value corresponding to the current position in the first fringe pattern is less than the second threshold, then it is mixed reflection dominated by diffuse reflection (provided that the first contrast is less than the second contrast). The median pixel value in the first fringe pattern should be less than the first threshold and greater than the second threshold; ideally, the mean pixel value should tend towards the median pixel value.

[0123] Preferably, the median or mean pixel value of the current image can be used directly as the reflection threshold.

[0124] Provided that the first contrast ratio is greater than the second contrast ratio, if the pixel value corresponding to the current position in the first fringe image is greater than the pixel median or pixel mean of the first fringe image, and the pixel value corresponding to the current position in the second fringe image is less than the pixel median or pixel mean of the second fringe image, then it is determined to be a mixed reflection with specular reflection as the main component.

[0125] If, assuming the first contrast is less than the second contrast, the pixel value corresponding to the current position in the first fringe pattern is less than the median or average pixel value of the first fringe pattern, and the pixel value corresponding to the current position in the second fringe pattern is greater than the median or average pixel value of the second fringe pattern, then it is determined to be a mixed reflection dominated by diffuse reflection.

[0126] If the region is a specular reflection area, the first fringe pattern is used as the data basis for 3D reconstruction (the fusion coefficient λ1 = 1 for the first fringe pattern and λ2 = 0 for the second fringe pattern); if the region is a diffuse reflection area, the second fringe pattern is used as the data basis for 3D reconstruction (the fusion coefficient λ1 = 0 for the first fringe pattern and λ2 = 1 for the second fringe pattern); if the region is a mixed reflection area, in mixed reflection with specular reflection as the main component, the fusion coefficient λ1 = Δδ + a for the first fringe pattern and λ2 = 1 - a - Δδ for the second fringe pattern; in mixed reflection with diffuse reflection as the main component, the fusion coefficient λ1 = 1 - a - |Δδ| for the first fringe pattern and λ2 = a + |Δδ| for the second fringe pattern.

[0127] Where λ1+λ2=1, λ1∈[0,1], λ2∈[0,1]; in the mixed reflection calculation, if the fusion coefficient is greater than 1, the fusion coefficient is truncated to 1; if the fusion coefficient is less than 0, the fusion coefficient is truncated to 0.

[0128] Where Δδ is the difference between the first contrast and the second contrast, Δδ∈[-1,1], and a is the difference threshold, a∈[0.5,1].

[0129] Using the FPP algorithm, the second stripe pattern is used to obtain the initial height map Z_fpp(x,y) through phase calculation.

[0130] Using the PMD algorithm, the first fringe pattern is solved by phase calculation to obtain the initial gradient map (p). x ,p y ).

[0131] An energy function is constructed, using the fusion coefficients corresponding to the initial heightmap and the second stripe map to constrain the consistency between the final heightmap and the initial heightmap. Similarly, the fusion coefficients corresponding to the initial gradient map and the first stripe map are used to constrain the consistency between the gradient of the final heightmap and the initial gradient map. By minimizing the energy function, the fused final heightmap is obtained, thus achieving 3D reconstruction.

[0132] The energy function is as follows:

[0133]

[0134] In the formula, λ1 is the fusion coefficient corresponding to the first stripe pattern; λ2 is the fusion coefficient corresponding to the second stripe pattern; Z is the final height to be solved; Z_fpp is the initial height map corresponding to the second stripe pattern; (p x ,p y () represents the initial gradient map corresponding to the first stripe pattern; The gradient of Z is constrained.

[0135] Where, λ2*(Z-X_fpp)2 The final height Z is constrained to be as close as possible to the initial height map Z_fpp corresponding to the second stripe pattern to ensure the global shape and avoid deviation of the overall structure after optimization.

[0136] Gradient of constraint Z The initial gradient (p) of the PMD measurement x ,p y Get as close as possible to restore local details (such as edges, textures, etc.).

[0137] This application utilizes PMD to measure the gradient of the specular reflection region more accurately (because the deflection angle of the specular reflection changes more significantly), while traditional FPP (such as Fourier phase profilometry) is prone to height measurement errors in the specular region due to phase loss. Therefore, the gradient of PMD is a key supplement to "local details".

[0138] Traditional phase deflection techniques require integrating the gradient field to obtain the height, which presents two major challenges: first, the integration path is dependent, as different paths may yield different results; second, errors accumulate along the integration path, leading to global distortion. This proposed method fundamentally avoids the integration process by treating gradient information as a constraint on the energy function, rather than directly integrating it. The final height map is obtained in a single step under the guidance of global energy optimization, ensuring global consistency and topological correctness of the reconstruction results.

[0139] The energy function is double-integrated over x and y, which means that the constraint is global. It does not only focus on a single pixel, but also accumulates the "shape difference" and "gradient difference" of the entire image region through integration. This ensures that the final height map Z satisfies both constraints globally, avoids local distortion after optimization, and improves the accuracy of the calculation.

[0140] The optimization problem of the energy function E(Z) is solved using an optimization solver to obtain the final height map Z.

[0141] Preferably, a smoothing regularization term can be added to the energy function described above. To suppress noise.

[0142] The energy function is then expressed as

[0143]

[0144] Preferably, the obtained Z can be smoothed or interpolated for missing regions according to the image requirements to obtain the final height map Z.

[0145] This application proposes a variational optimization method of "prior constraints + gradient constraints", that is, FPP provides "global shape prior" (initial height map) and PMD provides "local detail constraints" (initial gradient). The energy function dynamically balances the two through the fusion coefficient, and finds the optimal height distribution through mathematical optimization, so as to achieve complementary fusion of FPP and PMD data, and finally output a height map that integrates global shape and local detail.

[0146] Based on the final height map, a 3D point cloud is generated (converted into a point cloud or mesh model) to achieve 3D reconstruction.

[0147] This application also provides an electronic device, such as... Figure 4 As shown, it includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0148] Memory 303 is used to store computer programs;

[0149] The processor 301 implements the above method when executing the program stored in the memory 303.

[0150] Based on the same inventive concept, this application also proposes a computer-readable storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement the method described above.

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fusion method for three-dimensional imaging, characterized by, include: Acquire the first and second fringe patterns of reflection from the same physical region on the surface of the object under test; The first fringe pattern was obtained based on the principle of phase deflection imaging; The second fringe pattern is obtained based on the principle of fringe projection imaging; The first stripe pattern is obtained by phase calculation to obtain the initial gradient pattern; The second stripe pattern obtains the initial height map through phase calculation; In each stripe pattern, the ratio of the difference between the maximum and minimum pixel values ​​in each sliding window to the difference between the maximum and minimum pixel values ​​in the current stripe pattern is used as the contrast of the center pixel of the current window. Calculate the difference between the first contrast of the first stripe pattern and the second contrast of the second stripe pattern corresponding to the same physical location, determine the reflection type of the current pixel, and calculate the fusion coefficient corresponding to each stripe pattern in combination with the set difference threshold. Construct an energy function, using the fusion coefficients corresponding to the initial height map and the second stripe map to constrain the consistency between the final height map and the initial height map, and using the fusion coefficients corresponding to the initial gradient map and the first stripe map to constrain the consistency between the gradient of the final height map and the initial gradient map; solve for the final height map by minimizing the energy function; Based on the final height map, a 3D point cloud is generated to achieve 3D reconstruction.

2. The fusion method of claim 1, wherein, include: The difference threshold increases as the absolute difference between the first and second stripes increases; The difference threshold a∈[0.5,1]; The absolute difference is the absolute difference between the pixel mean of the first stripe pattern and the second stripe pattern, or the absolute difference between the pixel median, or the absolute difference between a specified pixel value of the first stripe pattern and a specified pixel value of the second stripe pattern.

3. The fusion method according to claim 1 or 2, characterized in that, Reflection types include: If the difference between the first contrast and the second contrast is greater than zero, it indicates that the first contrast is greater than the second contrast, and the current position tends to be of the specular reflection type; if the difference between the first contrast and the second contrast is less than zero, it indicates that the first contrast is less than the second contrast, and the current position tends to be of the diffuse reflection type.

4. The fusion method according to claim 3, characterized in that, include: Based on the difference Δδ between the first contrast and the second contrast and the difference threshold a, determine the reflection type of the current pixel: If -1 < Δδ ≤ -a, the reflection type of the physical location corresponding to this pixel is diffuse reflection; If -a < Δδ ≤ 0, the reflection type of the physical location corresponding to this pixel is a mixed reflection with diffuse reflection as the main component. If 0 < Δδ < a, the reflection type of the physical location corresponding to this pixel is a mixed reflection with specular reflection as the main component. If Δδ≥a, the reflection type of the physical location corresponding to this pixel is specular reflection.

5. The fusion method according to claim 4, characterized in that, include: The type of reflection is determined based on the difference between the first contrast ratio and the second contrast ratio; Reflection thresholds are set in the first and second stripe maps respectively. Based on the relationship between the pixel corresponding to the current physical location and the reflection thresholds of the first and second stripe maps respectively, the thinning reflection type corresponding to the current physical location is determined.

6. The fusion method according to claim 4 or 5, characterized in that, Fusion coefficient, including: In mixed reflection dominated by specular reflection, the fusion coefficient λ1 = Δδ + a corresponding to the first fringe pattern, and the fusion coefficient λ2 = 1 - a - Δδ corresponding to the second fringe pattern; In mixed reflection dominated by diffuse reflection, the fusion coefficient λ1 corresponding to the first fringe pattern is 1-a-|Δδ|, and the fusion coefficient λ2 corresponding to the second fringe pattern is a+|Δδ|. In specular reflection, the fusion coefficient λ1 = 1 for the first fringe pattern and the fusion coefficient λ2 = 0 for the second fringe pattern. In diffuse reflection, the fusion coefficient λ1 = 0 for the first fringe pattern and the fusion coefficient λ2 = 1 for the second fringe pattern. Where Δδ is the difference between the first contrast ratio and the second contrast ratio, and a is the difference threshold.

7. The fusion method according to claim 1, characterized in that, include: The pixel range corresponding to the sliding window is defined by at least one sinusoidal fringe period, and the period of the sinusoidal fringe is 2π.

8. The fusion method according to claim 1, characterized in that, The energy function also includes a smoothing regularization term, used to suppress noise.

9. A fusion system for three-dimensional imaging, characterized in that, include: The stripe pattern generation module projects the stripe pattern onto the surface of the object under test based on the principle of phase deflection imaging and the principle of stripe projection imaging. The camera acquires a first fringe pattern and a second fringe pattern of reflection from the same physical region on the surface of the object under test; wherein the first fringe pattern is obtained based on the principle of phase deflection imaging; and the second fringe pattern is obtained based on the principle of fringe projection imaging. The acquisition module acquires the first and second fringe patterns of reflection from the same physical region on the surface of the object under test. The phase calculation module obtains the initial gradient map from the first fringe pattern through phase calculation, and obtains the initial height map from the second fringe pattern through phase calculation. The coefficient calculation module uses the ratio of the difference between the maximum and minimum pixel values ​​in each sliding window to the difference between the maximum and minimum pixel values ​​in the current fringe pattern as the contrast of the center pixel of the current window. Calculate the difference between the first contrast of the first stripe pattern and the second contrast of the second stripe pattern corresponding to the same physical location, determine the reflection type of the current pixel, and calculate the fusion coefficient corresponding to each stripe pattern in combination with the set difference threshold. The function computation module constructs an energy function, using the fusion coefficients corresponding to the initial height map and the second stripe map to constrain the consistency between the final height map and the initial height map, and using the fusion coefficients corresponding to the initial gradient map and the first stripe map to constrain the consistency between the gradient of the final height map and the initial gradient map; the final height map is solved by minimizing the energy function. The 3D reconstruction module generates a 3D point cloud from the final height map, thus achieving 3D reconstruction.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the method as described in any one of claims 1-8.

Images