Effective region extraction method based on scene confidence analysis

Abstract

The present application relates to a kind of effective area extraction method based on scene confidence analysis, belong to digitization reverse engineering technical field.Use projector to carry out stripe projection to the region to be measured, and use camera to capture image for operation, obtain image wrapped phase.Carry out image segmentation and wrapped phase calculation based on wrapped phase, and mark.Different regions are carried out confidence analysis, and high-confidence area is retained.Through the calculation of the image of the region to be measured and image wrapped phase, and with high-confidence area corresponding, the corresponding relationship of high-confidence area and camera projector is obtained by the solving mode of corresponding point, and the projection stripe of high-confidence area is obtained, the region to be measured is projected again, and the second region to be measured image is captured by camera.Again three-dimensional reconstruction is carried out, and the final three-dimensional point cloud is obtained.Use unified standard to extract invalid point after scene re-projection, can remove error point while reducing the interference of error area to other positions.

Description

Technical Field

[0001] This invention relates to an effective region extraction method based on scene confidence analysis, belonging to the field of digital reverse engineering technology. Background Technology

[0002] Optical 3D measurement technology has been widely applied in industrial inspection, intelligent manufacturing, reverse engineering, and other fields. Fringe projection profilometry (FPP) has become one of the most popular measurement methods due to its advantages such as low cost, high accuracy, and high speed. A typical FPP system usually consists of a projector and a camera, and follows a projection-reflection imaging model. During the measurement process, the projector projects a pre-designed fringe pattern onto the object being measured, and the camera simultaneously captures the fringe pattern modulated by the object's surface. The corresponding 3D point cloud can be obtained by calculating the absolute phase from the captured image.

[0003] Achieving an ideal projection-reflection model in practical fringe projection profilometry (FPP) measurements is difficult due to numerous interferences in the scene. These interferences can be broadly categorized into two types: interference caused by the positional relationships between objects and interference caused by the material properties of the objects themselves. The former includes shadows and object refraction. Shadows arise from the difference in viewing angle between the camera and projector during actual measurements, resulting in projected shadows in the captured image and causing phase retrieval errors. Object refraction occurs when reflected light from one object's surface affects another, producing incorrect fringes. The latter includes overexposure and non-diffuse surfaces. Overexposure occurs when the measured object's surface has uneven reflectivity; high reflectivity areas saturate the camera sensor, and these overexposed areas cannot be correctly decoded in FPP measurements, leading to incorrect reconstruction results. Translucent materials such as marble, synthetic resins, and biological tissues are difficult to measure accurately using FPP due to subsurface scattering effects. Furthermore, stray light in the scene also affects projection-reflection imaging. These problems degrade the fringe modulation quality in the scene, resulting in points with incorrectly modulated fringes in the captured image. The resulting absolute phase will produce incorrect 3D point cloud results, affecting the integrity and accuracy of the point cloud and seriously interfering with subsequent point cloud processing.

[0004] To remove interference from erroneous points in an image, it is necessary to filter out erroneous regions. Currently, many erroneous point filtering methods have been proposed. Filtering from point clouds in an image often only removes single erroneous points. Furthermore, erroneous point removal from point clouds requires prior knowledge of the positional relationship between the projector and camera, and the calculation process is complex and time-consuming, failing to meet the needs of rapid measurement. In addition, there are two other problems: 1) There is no good evaluation metric for determining the scene's image quality; 2) Both methods involve post-processing on the captured image, lacking prior interactive measurement with the scene.

[0005] Currently, scene-prior-based adaptive projection has been widely applied to solve high dynamic range problems in FPP measurements. These methods pre-capture images of highly reflective object surfaces, analyze the surface light intensity and overexposed point distribution in the captured images, and obtain the most suitable fringe intensity for points at different camera coordinates. These fringes are then transformed onto the projected fringe pattern using a camera-projector correspondence, enabling re-measurement of overexposed areas after reprojection. Compared to traditional multi-exposure projection HDR measurements, these methods effectively reduce the number of projections and improve measurement efficiency. However, they are only applicable to overexposed scenes and lack prior interactive measurement for other interfering factors in FPP. Summary of the Invention

[0006] Purpose of the invention: In view of the above-mentioned existing problems and shortcomings, the purpose of this invention is to provide an effective region extraction method based on scene confidence analysis. Compared with other invalid point removal methods, this method uses a unified standard to extract invalid points and then reprojects the scene, which can reduce the interference of erroneous regions on other locations while removing erroneous points.

[0007] Technical solution: To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] An effective region extraction method based on scene confidence analysis includes the following steps:

[0009] Step 1: Use a projector to project stripes onto the area to be tested, and use a camera to capture an image of the area to be tested;

[0010] Step 2: Perform a wrap phase operation on the image of the region to be tested captured in Step 1 to obtain the image wrap phase;

[0011] Step 3: Perform image segmentation and wrap phase calculation based on the wrap phase of the image obtained in Step 1, and label the segmented image;

[0012] Step 4: Perform sinusoidal confidence analysis on different regions of the segmented image obtained in Step 3 to obtain the confidence of each labeled region and retain the high-confidence regions;

[0013] Step 5: Calculate the phase of the image of the region to be tested in Step 1 and the image in Step 3, and match it with the high confidence region obtained in Step 4. Obtain the correspondence between the high confidence region and the camera projector by solving the corresponding points, and obtain the projected fringes of the high confidence region.

[0014] Step 6: Using the projected fringes of the high-confidence region obtained in Step 5, reproject the region to be tested, and capture a secondary image of the region to be tested by the camera;

[0015] Step 7: Perform 3D reconstruction on the secondary test area image obtained in Step 6 to obtain the final 3D point cloud.

[0016] Furthermore, the specific steps of step 3 are as follows:

[0017] Step 3.1: Divide the image of the area to be tested obtained in Step 1 into three groups according to the order of horizontal and vertical stripes. First, move the first group of stripes to the end to obtain the first new stripe image. Then, move the third group of stripes to the beginning to obtain the second new stripe image. A total of two groups of horizontal and vertical stripes with different order from the original stripes are obtained.

[0018] Step 3.2: Calculate the wrapping phase of the two sets of horizontal and vertical stripes with different order from the original stripes obtained in Step 3.1. Perform gradient solution on the image of the area to be measured and the wrapping phase of the two sets of horizontal and vertical stripes with different order from the original stripes to obtain the upper boundary line and phase jump point images of the three wrapping phase images.

[0019] Step 3.3: Perform a logical AND operation on the boundary lines and phase transition point images of the three wrapped phase images obtained in Step 3.2 to remove the phase transition points and obtain the final boundary lines. Mark the different regions separated by the boundary lines.

[0020] Furthermore, the specific steps of step 4 are as follows:

[0021] Step 4.1: Perform a weighted calculation on the different regions obtained in Step 3, using the following formula:

[0022] S(n)=E1+3 / 4*E2+1 / 2*E3+1 / 4*E4,

[0023] Where E1, E2, E3, and E4 represent different values ​​of Gaussian weighted cosine error EG(x,y) within the region, with values ​​ranging from E1∈(0,0.005], E2∈(0.005,0.01], E3∈(0.01,0.05], E3∈(0.05,0.1], and n represents the image region marked in step 3; Step 4.2: The measurable confidence of the region is evaluated by calculating the mean of the remaining valid points in different regions, as shown in the following formula:

[0024] C n =S(n) / NUM(n),

[0025] Where NUM(n) represents the total number of pixels of the corresponding marked object;

[0026] Step 4.3: Retain points with a confidence level greater than 0.8.

[0027] Furthermore, the projected stripes in step 1 consist of the same number of horizontal and vertical stripes, and the frequency period and pixels of the two sets of stripes are the same.

[0028] Furthermore, in step 4, the confidence calculation fringe frequency is greater than 16, and the projection step size is greater than 5.

[0029] Furthermore, in step 5, the corresponding point is obtained by using horizontal stripes to obtain the row coordinates of the corresponding point and vertical stripes to obtain the ordinates of the corresponding point.

[0030] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0031] (1) The invalid region extraction method based on scene confidence analysis in this invention has quantitative threshold analysis compared with the traditional filtering method, and can be applied to scenarios with multiple superimposed interferences.

[0032] (2) The prior effective area reprojection technology proposed in this invention, compared with the traditional method of post-processing on the original image and other invalid point removal methods, uses a unified standard to extract invalid points and then reprojects the scene, which can reduce the interference of the erroneous area on other locations while removing erroneous points.

[0033] (3) The image segmentation method based on wrapping phase in this invention, compared with the traditional image segmentation method, utilizes the additional optical information brought by active projection to obtain more accurate segmentation lines.

[0034] (4) In this invention, different regions in the image are segmented and marked, and the confidence of different regions is calculated respectively. The high confidence region is retained. By obtaining the correspondence between the coordinates of the projector and the camera, the stripes containing only the high confidence region are projected onto the area to be measured. After interactive selection of projection measurement, interactive measurement of the image is realized. Compared with passive measurement, it can better realize interactive measurement with the scene and improve the reconstruction accuracy. Attached Figure Description

[0035] Figure 1 This is a flowchart of the method steps of the present invention;

[0036] Figure 2 This is a flowchart of the confidence analysis of the present invention;

[0037] Figure 3 This is a reprojection result diagram of an embodiment of the present invention.

[0038] In the picture: Figure 3 (a) — Projecting the first set of object images. Figure 3 (b) — Confidence analysis of the first group of objects. Figure 3 (c) — Second set of images before reprojection Figure 3 (d) — The second group of reprojected images;

[0039] Figure 4This is a schematic diagram of the reprojection results of different objects according to an embodiment of the present invention.

[0040] In the picture: Figure 4 (a) — First set of images before reprojection Figure 4 (b) — First set of reprojected images Figure 4 (c) — Second set of images before reprojection Figure 4 (d) — The second group of reprojected images;

[0041] Figure 5 These are schematic diagrams illustrating other interference effects according to embodiments of the present invention;

[0042] Figure 6 This is a schematic diagram illustrating the calculation of the corresponding points of the projector camera according to an embodiment of the present invention. Detailed Implementation

[0043] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0044] like Figure 1 As shown, an effective region extraction method based on scene confidence analysis is proposed.

[0045] Step 1: Use a 3D acquisition device to perform stripe projection on the area to be measured and capture the image of the area to be measured;

[0046] Step 2: Perform a phase wrapping operation on the captured series of stripe patterns. The formula for calculating the phase wrapping operation is as follows:

[0047]

[0048] Where (x, y) are the coordinates of the corresponding pixel in the image captured by the camera. k (x,y) represents the fringe pattern sequence captured by the camera, k represents the image order, and N is the phase shift step number. The enveloping phase of the image is obtained.

[0049] Step 3: Perform image segmentation based on wrap-around phase on the images captured by the camera, and label the segmented images;

[0050] The method for segmenting an image can be obtained as follows:

[0051] a. Change the stripe order of the acquired image:

[0052] S(I Round(KN / 3) ,I Round(KN / 3)+1 ,...,I Round(KN / 3)-2 ,I Round(KN / 3)-1 )

[0053] This formula represents the order of the stripe pattern image sequence after the order is changed, where round(KN / 3) represents the index of the original stripe image, round means rounding the value, K = 2, 3, and N is the number of stripes.

[0054] b. Obtain the wrapper phase corresponding to each sequence:

[0055]

[0056] c. Perform gradient calculation on the obtained horizontal and vertical phases, and then perform a logical 'AND' operation:

[0057] P x (x,y)=P x1 (x,y)∧P x2 (x,y)∧P x3 (x,y)

[0058] P y (x,y)=P y1 (x,y)∧P y2 (x,y)∧P y3 (x,y)

[0059] Where x and y represent the horizontal and vertical stripes respectively, and P represents the gradient calculation of the image. x and P y The final images obtained by solving the horizontal and vertical stripes respectively;

[0060] d. Perform a logical 'OR' operation on the horizontal and vertical dividing lines:

[0061] L(x,y)=P x (x,y)∨P y (x,y)

[0062] Where ∨ represents the logical OR operation, and L(x,y) represents the final segmentation line of the image.

[0063] e. After completing the dividing lines, mark the image.

[0064] Step 4: Perform sinusoidal confidence analysis on different regions of the segmented image obtained in Step 3 to obtain the confidence score of each labeled region, and retain the high-confidence regions; the process is as follows. Figure 2 As shown,

[0065] (1) Image cosine error calculation: Ideally, the main light received by the camera's field of view comes from the direct light reflected directly from the projector onto different areas to be measured. Ideally, the formula for calculating the camera's light intensity can be transformed into:

[0066]

[0067] Where (x,y) represents the image coordinates. Let k be the wrapping phase of the corresponding pixel, k be the image order of the corresponding stripe, N be the total number of stripes, and I be the total number of stripes. k (x,y) represents the gray level of the image pixel, a(x,y) represents the background intensity of the image pixel, and b(x,y) represents the stripe modulation degree of the image pixel.

[0068] However, the image fringes obtained in actual practice do not truly correspond to the wrapping phase. Therefore, the error between the fringes and the wrapping phase can be used as a measure of the corresponding points. The formula is:

[0069]

[0070]

[0071] Where e(x,y) represents the cosine error between the true and predicted values, k represents the order of the stripe pattern, and abs represents taking the absolute value of the calculation result. E(x,y) is the sum of errors. This represents the average value of the error.

[0072] (2) Error Result Enhancement: When the differences between directly obtained errors are small, it is not advisable to calculate them. In reality, error points in the test area are often clustered. Therefore, a Gaussian filtering-based method can be used to enhance the distinction between errors. In practice, we use the original error and the Gaussian filtered error together, with the following formula:

[0073]

[0074] Where EG(x,y) is the cosine error obtained after Gaussian weighting. To perform Gaussian filtering on the image.

[0075] The weighted calculation is performed on the different regions obtained, as shown in the following formula:

[0076] S(n)=E1+3 / 4*E2+1 / 2*E3+1 / 4*E4,

[0077] Where E1, E2, E3, and E4 represent different values ​​of Gaussian weighted cosine error EG(x,y) within the region, with values ​​of E1∈(0,0.005], E2∈(0.005,0.01], E3∈(0.01,0.05], E3∈(0.05,0.1], and n represents the image region marked in step 3.

[0078] The measurable confidence level of a region is assessed by calculating the mean of the remaining valid points in different regions, as shown in the following formula:

[0079] Cn =S(n) / NUM(n),

[0080] Where NUM(n) represents the total number of pixels of the corresponding marked object.

[0081] (3) Calculation of regional confidence: After error enhancement, the confidence of each region needs to be calculated. This mainly includes: differentiating errors of different values ​​by different weights; summing the weighted errors within the region and calculating the average value; removing the image from regions with low confidence; and retaining points with a confidence greater than 0.8.

[0082] Step 5: Calculate the correspondence between the obtained area and the projector, and reproject the obtained image.

[0083] Reprojection has been useful in solving high dynamic range problems, but it has not been previously applied to effective region extraction. Existing effective region extraction methods mostly extract from the original image; this invention obtains the effective region through reprojection.

[0084] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings:

[0085] The first detection is achieved by projecting a series of horizontal and vertical stripe patterns onto the object under test using an optical projection device. Before detection, both the projector and camera must be in focus to ensure the accuracy of the 3D reconstruction. In this invention, obtaining reprojected stripes does not require prior knowledge of the ensemble parameters between the camera and projector. Since the projected stripes acquire active optical information, gradient extraction based on the wrapping phase is primarily used in the extraction of the object region. Due to the characteristics of the wrapping phase, a large gradient can only be obtained at depth jump points and phase jump points. By using different wrapping phases, only the contours of depth jump points can be obtained. After obtaining different regions, the confidence level of each region can be obtained through confidence analysis using cosine error analysis. The higher the confidence level, the better the region conforms to the projection-reflection model, and the higher the corresponding reconstruction accuracy. Due to the presence of interference, higher reconstruction accuracy is achieved only when the distribution of each stripe on the object more closely matches the calculated cosine distribution. After determining the reprojection region, the corresponding points on the projector are obtained through the absolute phase of the horizontal and vertical stripes. By reprojecting the high-confidence region, a new captured image is obtained, achieving high-precision 3D reconstruction of the effective region.

[0086] Figure 3 To perform confidence analysis on the collected objects. Figure 3 (a) and Figure 3 (c) contains 6 objects. The results of the confidence analysis are as follows: Figure 3 (b) and Figure 3(d) shows that the confidence scores for objects 1, 2, 4, and 6 in the scene are all greater than 0.8, while the confidence scores for objects 3 and 5 are all less than 0.8. This demonstrates that confidence analysis effectively distinguishes whether a region conforms to the projection-reflection model across different objects. Figure 4 As shown in (a) to 4(d), through two sets of object experiments, the results of reprojecting the objects after confidence calculation show that the reconstructed areas with higher accuracy are well preserved compared to the original images. Figure 5 As shown, this invention also has a good effect on removing the interference of refracted or scattered light from beam-splitting objects in a scene with other objects. For example... Figure 6 As shown, the correspondence between the camera and projector points can be obtained through the horizontal and vertical absolute phases, and a new projected fringe image can be generated.

Claims

1. An effective region extraction method based on scene confidence analysis, characterized in that: Includes the following steps: Step 1: Use a projector to project stripes onto the area to be tested, and use a camera to capture an image of the area to be tested; Step 2: Perform a wrap phase operation on the image of the region to be tested captured in Step 1 to obtain the image wrap phase; Step 3: Perform image segmentation and wrap phase calculation based on the wrap phase of the image obtained in Step 1, and label the segmented image; Step 4: Perform sinusoidal confidence analysis on different regions of the segmented image obtained in Step 3 to obtain the confidence of each labeled region and retain the high-confidence regions; Step 5: Calculate the phase of the image of the region to be tested in Step 1 and the image in Step 3, and match it with the high confidence region obtained in Step 4. Obtain the correspondence between the high confidence region and the camera projector by solving the corresponding points, and obtain the projected fringes of the high confidence region. Step 6: Using the projected fringes of the high-confidence region obtained in Step 5, reproject the region to be tested, and capture a secondary image of the region to be tested by the camera; Step 7: Perform 3D reconstruction on the secondary test area image obtained in Step 6 to obtain the final 3D point cloud.

2. The effective region extraction method based on scene confidence analysis according to claim 1, characterized in that: The specific steps of step 3 are as follows: Step 3.1: Divide the image of the area to be tested obtained in Step 1 into three groups according to the order of horizontal and vertical stripes. First, move the first group of stripes to the end to obtain the first new stripe image. Then, move the third group of stripes to the beginning to obtain the second new stripe image. A total of two groups of horizontal and vertical stripes with different order from the original stripes are obtained. Step 3.2: Calculate the wrapping phase of the two sets of horizontal and vertical stripes with different order from the original stripes obtained in Step 3.

1. Perform gradient solution on the image of the area to be measured and the wrapping phase of the two sets of horizontal and vertical stripes with different order from the original stripes to obtain the upper boundary line and phase jump point images of the three wrapping phase images. Step 3.3: Perform a logical AND operation on the boundary lines and phase transition point images of the three wrapped phase images obtained in Step 3.2 to remove the phase transition points and obtain the final boundary lines. Mark the different regions separated by the boundary lines.

3. The effective region extraction method based on scene confidence analysis according to claim 2, characterized in that: The specific steps of step 4 are as follows: Step 4.1: Perform a weighted calculation on the different regions obtained in Step 3, using the following formula: S(n)=E1+3 / 4*E2+1 / 2*E3+1 / 4*E4, Where E1, E2, E3, and E4 represent different values ​​of Gaussian weighted cosine error EG(x,y) within the region, with values ​​ranging from E1∈(0,0.005], E2∈(0.005,0.01], E3∈(0.01,0.05], E3∈(0.05,0.1], and n represents the image region marked in step 3; Step 4.2: The measurable confidence of the region is evaluated by calculating the mean of the remaining valid points in different regions, as shown in the following formula: C n =S(n) / NUM(n), Where NUM(n) represents the total number of pixels of the corresponding marked object; Step 4.3: Retain points with a confidence level greater than 0.

8.

4. The effective region extraction method based on scene confidence analysis according to claim 1, characterized in that: The projected stripes in step 1 consist of the same number of horizontal and vertical stripes, with both sets of stripes having the same frequency period and pixel count.

5. The effective region extraction method based on scene confidence analysis according to claim 3, characterized in that: In step 4, the confidence calculation fringe frequency is greater than 16, and the projection step size is greater than 5.

6. The effective region extraction method based on scene confidence analysis according to claim 1, characterized in that: In step 5, the corresponding point is obtained by using horizontal stripes to obtain the row coordinates of the corresponding point and vertical stripes to obtain the ordinates of the corresponding point.

Images