A method and apparatus for joint processing of 3D reconstruction, semantic segmentation and depth estimation

Abstract

This invention discloses a method and apparatus for joint processing of 3D reconstruction, semantic segmentation, and depth estimation. First, the object is modulated using a sinusoidal grating fringe image, and a deformed fringe image containing the object's geometry and fringe coding information is acquired through imaging. Next, a semantic segmentation network is constructed to perform pixel-level classification of the fringe image, identifying different regions and generating a wrapping phase map; simultaneously, a depth estimation network is constructed to predict the depth map. Subsequently, the depth map is converted into a 3D point cloud using camera and projector parameters, and a coarse unwrapping phase is inversely solved. By comparing the coarse unwrapping phase with the wrapping phase predicted by the semantic segmentation network, a fringe order map is obtained, thus acquiring a precise unwrapping phase. Finally, the unwrapping phase is again converted into a 3D point cloud using parameters, achieving accurate 3D reconstruction of the measured object. This invention combines structured light technology, semantic segmentation, and depth estimation, improving the accuracy and efficiency of 3D reconstruction.

Description

Technical Field

[0001] This invention relates to the field of computer 3D reconstruction, and more particularly to a method and apparatus for joint processing of 3D reconstruction, semantic segmentation and depth estimation. Background Technology

[0002] 3D reconstruction technology, as one of the core research directions in the field of computer vision, occupies an important position and has significant application value, widely used in fields such as medicine, cultural relic restoration, and entertainment. Structured light 3D reconstruction, due to its non-contact nature, high accuracy, and computational simplicity, has been widely applied in practical 3D reconstruction applications. Structured light 3D reconstruction technology can not only efficiently acquire the 3D shape information of objects but also provide a solid data foundation for subsequent visual processing and analysis, further promoting the development of computer vision technology and expanding its application areas.

[0003] In traditional structured light 3D reconstruction technology, the reconstruction speed is closely related to the number of projected images used. Specifically, while increasing the number of projected images improves reconstruction accuracy, it also increases computation time, leading to a decrease in reconstruction speed. Conversely, reducing the number of projected images shortens computation time and increases reconstruction speed, but inevitably sacrifices reconstruction accuracy. Therefore, traditional structured light 3D reconstruction technology often involves a trade-off between speed and accuracy.

[0004] Deep learning can learn from large amounts of sample data, extract key features, and perform tasks such as classification, recognition, and regression on new data, finding widespread application across various fields. In recent years, applying deep learning to structured light 3D reconstruction to achieve high-efficiency and high-precision 3D reconstruction has become a research hotspot in the field. Existing methods use direct prediction of the wrapping phase and fringe order to obtain the unwrapped phase. However, the fringe order is spatially continuous and cannot be accurately predicted, resulting in low accuracy of the reconstructed unwrapped phase and hindering high-precision 3D reconstruction. To address this issue, this invention combines semantic segmentation and depth estimation tasks from deep learning. Depth estimation directly predicts the object's depth, which in turn calculates the precise fringe order. Combined with the wrapping phase obtained from a semantic segmentation network, a high-precision unwrapped phase can be obtained, thus achieving high-precision 3D reconstruction. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method and device for joint processing of 3D reconstruction, semantic segmentation, and depth estimation. This invention predicts the unwrapped phase from a striped image to achieve high-precision and high-efficiency structured light 3D reconstruction.

[0006] This invention is achieved through the following technical solution:

[0007] S1: The sinusoidal grating fringe image is projected onto the object under test by a projector to obtain the deformed fringe image of the object after being modulated by structured light fringe. This fringe image is captured by an industrial camera and contains the geometric information and fringe coding information of the object under test.

[0008] S2: Construct a semantic segmentation network, input a striped image, perform pixel-level classification on the striped image, identify different regions of the object and stripe coding information, and generate a wrapping phase map of the striped image;

[0009] S3: Construct a depth estimation network, input a striped image, perform depth prediction on the striped image, and generate a depth map, which represents the depth value of each pixel in the image;

[0010] S4: Based on the internal and external parameters of the camera and projector, convert the depth map into a 3D point cloud;

[0011] S5: Using the relationship between 3D point cloud and absolute phase, the rough unwrapped phase is solved in reverse.

[0012] S6: Subtract the wrapped phase predicted by the semantic segmentation network from the coarse unwrapped phase to obtain the accurate fringe order map of the wrapped phase;

[0013] S7: Combine the wrapped phase map predicted by the semantic segmentation network with the obtained stripe order map to obtain the accurate unwrapped phase;

[0014] S8: By utilizing the intrinsic and extrinsic parameters of the camera and projector, the unwrapped phase is converted into a three-dimensional point cloud, enabling accurate three-dimensional reconstruction of the measured object.

[0015] Further, the sinusoidal grating fringe image described in step S1 is expressed as follows:

[0016] I(x,y)=A(x,y)+B(x,y)cos(2πfx)

[0017] Where I represents the grayscale value of each pixel in the sinusoidal grating stripe, A and B represent the background light intensity and modulation intensity, respectively, and f represents the frequency of the projected sinusoidal stripe.

[0018] Furthermore, the semantic segmentation network described in step S2 is constructed as two branches, which respectively predict the numerator and denominator in the wrapper phase calculation process. In the traditional N-step phase shift method, the formula for calculating the wrapper phase is as follows:

[0019]

[0020] in, Here, N represents the number of captured fringe images. Because the arctan() function is used in the wrapping phase calculation, ambiguity between 0 and 2π easily arises at the boundary of fringe periods, affecting the determination of the fringe order and causing significant errors. The semantic segmentation network used in this invention predicts both the numerator and denominator of the arctan() function, and uses the prediction results to calculate the wrapping phase, which greatly improves the prediction accuracy.

[0021] Furthermore, in step S3, the depth estimation network uses a UNet neural network to predict the depth map.

[0022] Furthermore, in step S4, the depth map predicted by the depth estimation is converted into a 3D point cloud using the intrinsic and extrinsic parameters of the camera and projector. This specifically includes the following sub-steps:

[0023] S41: Calibrate the camera using Zhang Zhengyou's calibration method, take 20 sets of high-precision images of the origin calibration plate, and calculate the camera's intrinsic parameter matrix K using the MATLAB standard tool library. c and extrinsic parameter matrix [R c |T c ].

[0024] S42: Calibrate the projector, treating it as an inverse camera, and obtain the projector's intrinsic parameter matrix K. p and extrinsic parameter matrix [R p |T p ].

[0025] S43: Based on the relationship between the world coordinate system and the camera / projector pixel coordinate system, and using the intrinsic and extrinsic parameters of the camera and projector, the depth Z can be obtained. W In the camera pixel coordinate system (u c v c The relationship between the depth map and the point cloud is used to convert the depth map into a 3D point cloud.

[0026] Furthermore, using the three-dimensional point cloud obtained in step S4, step S5 can obtain a rough unwrapped phase based on the relationship between the three-dimensional point cloud and the absolute phase. The specific steps are as follows:

[0027] S51: Solve for the x-coordinate u of the corresponding point on the projector pixel plane. p .

[0028] S52: For the same phase value φ, the unwrapping phase of the camera pixel plane and the unwrapping phase of the projector pixel plane have the following relationship:

[0029] φ c (u c v c )=φ p (u p )=φ

[0030] The striped image projected by the projector is composed of uniform stripes, which indicates that:

[0031]

[0032] Where W is the projection width of the projector. The pixel coordinates u of the projector are known. p Given the projector's projection width W, we can obtain (u c v c The rough solution of the phase at point )

[0033] Furthermore, in step S6, the accurate fringe order can be calculated using the coarse unwrapping phase and the wrapped phase predicted by the semantic segmentation network. Since the fringe order is related to the period position and has spatial continuity, if the image does not start from the first period, directly predicting the fringe order is prone to misalignment, leading to errors in the wrapped phase unfolding. Therefore, the accurate fringe order is calculated using the coarse unwrapping phase and the wrapped phase.

[0034] Furthermore, the accurate unwrapping phase, obtained by using the wrapping phase predicted by the semantic segmentation model and the depth estimation, and then performing a series of operations to obtain the accurate fringe order, can be represented as follows:

[0035]

[0036] Furthermore, the present invention proposes a joint processing method for 3D reconstruction, semantic segmentation, and depth estimation based on structured light, comprising the following modules:

[0037] The data acquisition module is used to acquire stripe images containing structured light stripe coding information;

[0038] The semantic segmentation module is used to semantically segment the striped image and generate a wrap-around phase map;

[0039] The depth estimation module is used to estimate the depth of the striped image and generate a depth map;

[0040] The 3D reconstruction module is used to generate a 3D point cloud based on the depth map and to calculate a coarse unwrapping phase using the 3D point cloud and the wrapping phase map.

[0041] The phase processing module is used to subtract the wrapped phase from the coarse unwrapped phase to obtain the fringe order map, and to calculate the unwrapped phase by combining it with the wrapped phase map.

[0042] The display and output module is used to display and output the results of the 3D reconstruction.

[0043] Compared with existing technologies, the joint processing method for 3D reconstruction, semantic segmentation, and depth estimation based on structured light proposed in this invention has the following advantages:

[0044] (1) The reconstruction speed and accuracy of traditional structured light 3D reconstruction are limited by the number of projected images, and there is often a trade-off between speed and accuracy. This invention applies a new deep learning model to structured light 3D reconstruction, which can reduce the number of projected images while ensuring high reconstruction accuracy, thus achieving high-precision and high-efficiency structured light 3D reconstruction.

[0045] (2) This invention applies the depth estimation task to structured light 3D reconstruction, transforms the depth map into a 3D point cloud, and then into a coarse unwrapped phase and a precise fringe order, which can effectively assist in the unfolding of the wrapped phase and improve the accuracy of the prediction results. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the workflow of the joint processing method for 3D reconstruction, semantic segmentation, and depth estimation based on structured light according to the present invention.

[0047] Figure 2 This is a diagram of the overall architecture of the model of the present invention.

[0048] Figure 3 This is a stripe pattern captured according to the present invention.

[0049] Figure 4 This is the depth map described in this invention.

[0050] Figure 5 This is the wrapping phase diagram described in this invention.

[0051] Figure 6 This is the stripe order diagram described in this invention.

[0052] Figure 7 This is the unwrapping phase diagram described in this invention. Detailed Implementation

[0053] The present invention will be further described in more detail below with reference to examples and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0054] As attached Figure 1 As shown, this invention proposes a joint processing method for 3D reconstruction, semantic segmentation, and depth estimation based on structured light. Its main workflow is as follows:

[0055] S1: Project the sinusoidal grating fringe image onto the object being measured using a projector, and obtain the image as shown in the attached figure. Figure 3The image shown is a deformed stripe image of an object modulated by structured light stripes. The stripe image was captured by an industrial camera and contains geometric information and stripe coding information of the object under test.

[0056] S2: Construct attachments Figure 2 The semantic segmentation network model shown employs an improved UNet neural network model, with a U-shaped architecture consisting of symmetrical Encoder and Decoder parts. The improved UNet neural network model adds residual modules to each layer of the Encoder, using skip connections to make the model easier to train and better capture the relationships between contexts, exhibiting powerful feature extraction capabilities. The predicted package phase map is attached. Figure 5 As shown.

[0057] S3: Construct attachments Figure 2 The depth estimation network model shown predicts the depth map of the measured object to assist in the subsequent generation of an accurate fringe order map. An example of the depth map is attached. Figure 4 As shown.

[0058] In steps S2 and S3, the semantic segmentation network and the depth estimation network both use the Adam optimizer during training, with a training batch size of 200 epochs and a batch size of 2.

[0059] S4: Using the intrinsic and extrinsic parameters of the camera and projector, the depth map predicted by depth estimation is converted into a 3D point cloud. This specifically includes the following sub-steps:

[0060] S41: Calibrate the camera using Zhang Zhengyou's calibration method, take 20 sets of high-precision images of the origin calibration plate, and calculate the camera's intrinsic parameter matrix K using the MATLAB standard tool library. c and extrinsic parameter matrix [R c |T c ].

[0061] S42: Projector calibration. Treat the projector as an inverse camera, projecting a fringe image onto the calibration board. Take fringe images using a camera, extract feature points from the camera's imaging plane using a feature point extraction algorithm, calculate the absolute phase using the fringe image, and map the feature points from the camera's imaging plane to the projector's imaging plane using absolute phase mapping. Then, use Zhang Zhengyou's calibration method to calibrate the projector, obtaining its intrinsic parameter matrix K. p and extrinsic parameter matrix [R p |T p ].

[0062] S43: The world coordinate system and the camera / projector pixel coordinate system have the following relationship:

[0063] sc [u c v c ,1] T =K c [R c |T c ][X W Y W Z w ,1] T

[0064] s p [u p v p ,1] T =K p [R p |T p ][X W Y W Z W ,1] T

[0065] Among them, s c s p These represent the scale factors of the camera and projector, respectively. c v c ), (u p v p ), (X W Y W Z W Let Z and F be the corresponding points in the camera, projector pixel coordinate system, and world coordinate system, respectively. Using the above relationship and the intrinsic and extrinsic parameters of the camera and projector, the depth Z can be obtained. W In the camera pixel coordinate system (u c v c The relationship between the depth map and the point cloud is used to convert the depth map into a 3D point cloud.

[0066] S5: Based on the relationship between the 3D point cloud and the absolute phase, a rough unwrapped phase can be obtained. The specific steps are as follows:

[0067] S51: To find the x-coordinate of the corresponding point on the projector pixel plane, use the following formula:

[0068]

[0069] The x-coordinate u of the corresponding point on the projector pixel plane can be obtained. p ,in For the camera projection matrix K c [R c |T c The elements in ] For the projection matrix K of the projector p [R p |Tp The elements in ].

[0070] S52: For the same phase value φ, the unwrapping phase of the camera pixel plane and the unwrapping phase of the projector pixel plane have the following relationship:

[0071] φ c (u c v c )=φ p (u p )=φ

[0072] The striped image projected by the projector is composed of uniform stripes, which indicates that:

[0073]

[0074] Where W is the projection width of the projector. The pixel coordinates u of the projector are known. p Given the projector's projection width W, we can obtain (u c v c The rough solution of the phase at point )

[0075] S6: By combining the coarse unwrapped phase with the wrapped phase predicted by the semantic segmentation network, the accurate fringe order can be calculated. Since the range of the arctan() function is between 0 and 2π, and the range of the obtained wrapped phase is also between 0 and 2π, and it changes periodically according to the frequency of the projected fringe image, to ensure the uniqueness of the phase, the wrapped phase needs to be converted into the unwrapped phase. Clearly, the wrapped phase is unique within the periodic range; therefore, simply adding 2kπ to obtain the unwrapped phase ensures the global uniqueness of the phase, where k represents the periodic order. As the concept of fringe order indicates, the fringe order is an integer; therefore, even if the obtained unwrapped phase is coarse, the accurate fringe order can still be calculated, as shown in the appendix. Figure 6 As shown. The fringe order can be represented as follows:

[0076]

[0077] Here, floor(x) is the down-rounding operation, which takes the largest integer less than or equal to x. It is the package phase predicted by the semantic segmentation network.

[0078] S7: Using the wrapped phase and depth estimation predicted by the semantic segmentation model, and through a series of operations, the accurate fringe order is obtained, resulting in the accurate unwrapped phase, which can be represented as follows:

[0079]

[0080] The final accurate unwrapped phase diagram is shown in the attached figure. Figure 7 As shown.

[0081] S8: Using the relationship described in S51, the final unwrapped phase can be converted into a three-dimensional point cloud, and the resulting point cloud pairs Z W Filtering removes irrelevant areas and areas outside the boundaries to obtain the surface information of the object being measured.

[0082] Furthermore, the present invention also discloses a computer storage medium storing a computer program that can execute computer instructions or programs for a joint processing method of three-dimensional reconstruction, semantic segmentation and depth estimation based on structured light provided by the present invention. When the computer instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.

[0083] The above embodiments are preferred implementations of the present invention, but the implementation of the present invention is not limited to these examples. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the core spirit and principles of the present invention should be considered equivalent alternatives and are also included within the protection scope of the present invention.

Claims

1. A joint processing method for 3D reconstruction, semantic segmentation, and depth estimation based on structured light, characterized in that: Includes the following steps: S1: The sinusoidal grating fringe image is projected onto the object under test by a projector to obtain the deformed fringe image of the object after being modulated by structured light fringe. This fringe image is captured by an industrial camera and contains the geometric information and fringe coding information of the object under test. S2: Construct a semantic segmentation network, input a striped image, perform pixel-level classification on the striped image, identify different regions of the object and stripe coding information, and generate a wrapping phase map of the striped image; S3: Construct a depth estimation network, input a striped image, perform depth prediction on the striped image, and generate a depth map, which represents the depth value of each pixel in the image; S4: Based on the internal and external parameters of the camera and projector, convert the depth map into a 3D point cloud; S5: Using the relationship between 3D point cloud and absolute phase, the rough unwrapped phase is solved in reverse. S6: The coarse unwrapping phase φ obtained through step S5 c The fringe order map of the package phase is obtained by combining the package phase predicted by the semantic segmentation network. S7: Combine the wrapped phase map predicted by the semantic segmentation network with the obtained stripe order map to obtain the accurate unwrapped phase; S8: By utilizing the intrinsic and extrinsic parameters of the camera and projector, the unwrapped phase is converted into a three-dimensional point cloud, enabling accurate three-dimensional reconstruction of the measured object.

2. The joint processing method for 3D reconstruction, semantic segmentation, and depth estimation based on structured light according to claim 1, characterized in that: The projected sinusoidal grating fringes are: I(x,y)=A(x,y)+B(x,y)cos(2πfx) (1) Where I represents the grayscale value of each pixel in the sinusoidal grating stripe, A and B represent the background light intensity and modulation intensity, respectively, and f represents the frequency of the projected sinusoidal stripe.

3. The joint processing method for 3D reconstruction, semantic segmentation, and depth estimation based on structured light according to claim 1, characterized in that: The coarse unwrapping phase φ obtained through step S5 c The accurate fringe order map can be obtained by combining the package phase map predicted by the semantic segmentation network with the fringe order k. The formula for calculating the fringe order k is as follows: Here, floor(x) is the floor function, which rounds down to the nearest integer less than or equal to x. It is the package phase predicted by the semantic segmentation network; By combining the accurate fringe order map with the wrapped phase, an accurate unwrapped phase map can be obtained, and its calculation formula is as follows:

4. A structured light-based three-dimensional reconstruction system for use in the joint processing method according to any one of claims 1-3, characterized in that: The 3D reconstruction system includes a data acquisition module, a semantic segmentation module, a depth estimation module, a 3D reconstruction module, a phase processing module, and a display and output module. The data acquisition module is used to acquire a stripe image containing structured light stripe coding information; The semantic segmentation module is used to perform semantic segmentation on the stripe image and generate a wrapping phase map; The depth estimation module is used to estimate the depth of the stripe image and generate a depth map; The aforementioned 3D reconstruction module is used to generate a 3D point cloud based on the depth map, and to calculate a coarse unwrapping phase using the 3D point cloud and the wrapping phase map. The phase processing module is used to subtract the wrapped phase from the coarse unwrapped phase to obtain a fringe order map, and to calculate the unwrapped phase in combination with the wrapped phase map. The display and output module is used to display and output the results of the 3D reconstruction.

5. An electronic device comprising a processor and a memory, characterized in that: The processor is configured to perform the joint processing method of structured light-based 3D reconstruction, semantic segmentation and depth estimation as described in any one of claims 1-3; The memory is used to store programs that support the processing of the joint processing method of structured light-based 3D reconstruction, semantic segmentation and depth estimation as described in any of claims 1-3.

Images