Three-dimensional reconstruction, semantic segmentation and depth estimation combined processing method and equipment

Abstract

The invention discloses a three-dimensional reconstruction, semantic segmentation and depth estimation combined processing method and equipment. According to the method, firstly, a sinusoidal grating stripe image is used for modulating an object, and a deformed stripe image is obtained through shooting and comprises geometric and stripe coding information of the object; thirdly, constructing a semantic segmentation network to carry out pixel-level classification on the fringe image, identifying different regions and generating a wrapped phase diagram; and meanwhile, constructing a depth estimation network to predict a depth map. Then, converting the depth map into a three-dimensional point cloud by combining parameters of a camera and a projector, and reversely solving a rough unwrapping phase; and comparing the rough unwrapping phase with a wrapping phase predicted by the semantic segmentation network to obtain a fringe order graph, thereby obtaining an accurate unwrapping phase. And finally, the unwrapped phase is converted into the three-dimensional point cloud by using the parameters again, and accurate three-dimensional reconstruction of the measured object is realized. According to the method, the structured light technology, semantic segmentation and depth estimation are combined, and the precision and efficiency of three-dimensional reconstruction are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer three-dimensional reconstruction, and in particular to a method and device for jointly processing three-dimensional reconstruction, semantic segmentation and depth estimation. Background Art

[0002] As one of the core research directions in the field of computer vision, 3D reconstruction technology occupies an important position in computer vision and has important application value. It is widely used in medical treatment, cultural relic restoration, entertainment and other fields. Structured light 3D reconstruction has been widely used in actual 3D reconstruction applications due to its non-contact, high precision and simple calculation. Structured light 3D reconstruction technology can not only efficiently obtain the 3D shape information of objects, but also provide a solid data foundation for subsequent visual processing and analysis, further promoting the development of computer vision technology and the expansion of its application fields.

[0003] In traditional structured light 3D reconstruction technology, the reconstruction speed is closely related to the number of projected images used. Specifically, when the number of projected images increases, although the reconstruction accuracy can be improved, the calculation time will increase accordingly, resulting in a decrease in reconstruction speed. Conversely, reducing the number of projected images can shorten the calculation time and increase the reconstruction speed, but it will inevitably sacrifice the reconstruction accuracy. Therefore, traditional structured light 3D reconstruction technology often has a trade-off between speed and accuracy.

[0004] Deep learning can extract the main features of data by learning a large amount of sample data, and realize tasks such as classification, recognition, and regression of new data. It has been well applied in various fields. In recent years, applying deep learning to structured light 3D reconstruction to achieve high-efficiency and high-precision 3D reconstruction has been one of the hot topics in the field of 3D reconstruction. The existing method uses a method of directly predicting the wrapping phase and fringe order to obtain the unwrapping phase. However, the fringe order is continuous in space and cannot be predicted well. Therefore, the reconstructed unwrapping phase is not accurate, and it is difficult to achieve high-precision 3D reconstruction. To solve this problem, the present invention combines the semantic segmentation and depth estimation tasks in deep learning, directly predicts the depth of the object through depth estimation, and in turn calculates the accurate fringe order. Combined with the wrapping phase obtained by the semantic segmentation network, a high-precision unwrapping phase can be obtained, thereby achieving high-precision 3D reconstruction. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings and deficiencies of the above-mentioned prior art and provide a method and device for joint processing of 3D reconstruction, semantic segmentation and depth estimation. The present invention predicts the unwrapped phase from a stripe image to achieve high-precision and high-efficiency structured light 3D reconstruction.

[0006] The present invention is achieved through the following technical solutions:

[0007] S1: Project the sinusoidal grating fringe image onto the object to be measured through a projector to obtain a deformed fringe image of the object modulated by the structured light fringe. The fringe image is taken by an industrial camera and contains geometric information and fringe coding information of the object to be measured;

[0008] S2: Build a semantic segmentation network, input the stripe image, classify the stripe image at the pixel level, identify different regions of the object and stripe encoding information, and generate a wrapped phase map of the stripe image;

[0009] S3: construct a depth estimation network, input the stripe image, perform depth prediction on the stripe image, and generate a depth map, which represents the depth value of each pixel in the image;

[0010] S4: Convert the depth map into a 3D point cloud based on the internal and external parameters of the camera and projector;

[0011] S5: Using the relationship between the 3D point cloud and the absolute phase, the rough unwrapped phase is solved inversely;

[0012] S6: Subtract the wrapped phase predicted by the semantic segmentation network from the rough unwrapped phase to obtain the precise fringe order map of the wrapped phase;

[0013] S7: Combine the wrapped phase map predicted by the semantic segmentation network with the obtained fringe order map to obtain the accurate unwrapped phase;

[0014] S8: Using the internal and external parameters of the camera and projector, the unwrapped phase is converted into a 3D point cloud to achieve accurate 3D reconstruction of the object being measured.

[0015] Furthermore, the sinusoidal grating fringe image described in step S1 is expressed as:

[0016] I(x,y)=A(x,y)+B(x,y)cos(2πfx)

[0017] Where I represents the grayscale value of each pixel of the sinusoidal grating stripes, A and B represent the background light intensity and modulation intensity respectively, and f represents the frequency of the projected sinusoidal stripes.

[0018] Furthermore, the semantic segmentation network described in step S2 is constructed into two branches, which respectively predict the numerator and denominator in the process of calculating the wrapped phase. In the traditional N-step phase shift method, the calculation formula of the wrapped phase is as follows:

[0019]

[0020] in, is the captured stripe image, and N is the number of stripe images. Since the arctan() function is used in the calculation of the wrapped phase, ambiguity between 0 and 2π is likely to occur at the junction of the stripe period, which affects the judgment of the stripe order and causes a large error. The semantic segmentation network used in the present invention predicts the numerator and denominator of the arctan() function respectively, and uses the predicted results to calculate the wrapped phase, which can greatly improve the prediction accuracy.

[0021] Furthermore, the depth estimation network in step S3 uses a UNet neural network to predict the depth map.

[0022] Furthermore, in step S4, the depth map predicted by the depth estimation is converted into a three-dimensional point cloud by using the internal and external parameters of the camera and the projector, which specifically includes the following sub-steps:

[0023] S41: Use Zhang Zhengyou calibration method to calibrate the camera, take 20 sets of high-precision origin calibration plate pictures, and use MATLAB standard tool library to calculate the camera's intrinsic parameter matrix K c and the external parameter matrix [R c |T c ].

[0024] S42: Calibrate the projector, regard the projector as an inverse camera, and obtain the internal parameter matrix K of the projector p and the external parameter matrix [R p |T p ].

[0025] S43: According to the relationship between the world coordinate system and the camera / projector pixel coordinate system, the depth Z can be obtained by using the internal and external parameters of the camera and projector. W Compared with the camera pixel coordinate system (u c , v c ) to convert the depth map into a three-dimensional point cloud.

[0026] Further, using the three-dimensional point cloud obtained in step S4, step S5 can obtain a rough unwrapped phase according to the relationship between the three-dimensional point cloud and the absolute phase. The specific steps are as follows:

[0027] S51: Solve the horizontal coordinate u of the corresponding point on the projector pixel plane p .

[0028] S52: For the same phase value φ, the unwrapped phase of the camera pixel plane and the unwrapped phase of the projector pixel plane have the following relationship:

[0029] φ c (u c , v c )=φ p (u p )=φ

[0030] The fringe image of the projector is composed of uniform fringe patterns, so:

[0031]

[0032] Where W is the projection width of the projector. p and the projection width W of the projector, we can get (u c , v c ) point.

[0033] Furthermore, in step S6, the rough unwrapped phase and the wrapped phase predicted by the semantic segmentation network can be used to calculate the precise fringe order. Since the fringe order is related to the period position and has continuity in space, if the image does not start from the first period, directly predicting the fringe order is prone to misalignment, resulting in errors in the wrapped phase unfolding. Therefore, the rough unwrapped phase and the wrapped phase are used to calculate the precise fringe order.

[0034] Furthermore, in step S7, the accurate unwrapping phase is obtained by using the semantic segmentation model to predict the wrapped phase and the depth estimation and a series of operations to obtain the accurate fringe order, which can be expressed as follows:

[0035]

[0036] Furthermore, the present invention proposes a joint processing method for 3D reconstruction, semantic segmentation and depth estimation based on structured light, comprising the following modules:

[0037] A data acquisition module, used for acquiring a fringe image containing structured light fringe coding information;

[0038] Semantic segmentation module, used to perform semantic segmentation on the stripe image and generate a wrapped phase map;

[0039] A depth estimation module, used to perform depth estimation on the stripe image and generate a depth map;

[0040] A 3D reconstruction module is used to generate a 3D point cloud based on the depth map and calculate a rough unwrapped phase using the 3D point cloud and the wrapped phase map;

[0041] A phase processing module is used to subtract the wrapped phase from the rough unwrapped phase to obtain a fringe order diagram, and calculate the unwrapped phase in combination with the wrapped phase diagram;

[0042] The display and output module is used to display and output the results of 3D reconstruction.

[0043] Compared with the prior art, the joint processing method of 3D reconstruction, semantic segmentation and depth estimation based on structured light proposed in the present invention has the following advantages:

[0044] (1) The reconstruction speed and accuracy of traditional structured light 3D reconstruction are restricted by the number of projected images, and there is often a trade-off between speed and accuracy. The present invention applies a new deep learning model to structured light 3D reconstruction, which can reduce the number of projected images while ensuring high reconstruction accuracy, thus achieving high-precision and high-efficiency structured light 3D reconstruction.

[0045] (2) The present invention applies the depth estimation task to structured light 3D reconstruction, converts the depth map into a 3D point cloud and then into a rough unwrapped phase and precise fringe order, which can well assist the unfolding of the wrapped phase and improve the accuracy of the prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a workflow diagram of the joint processing method of three-dimensional reconstruction, semantic segmentation and depth estimation based on structured light of the present invention.

[0047] Figure 2 This is the overall architecture diagram of the model of the present invention.

[0048] Figure 3 This is the fringe pattern captured according to the present invention.

[0049] Figure 4 This is the depth map described in the present invention.

[0050] Figure 5 This is the wrapped phase diagram described in the present invention.

[0051] Figure 6 This is the fringe order diagram described in the present invention.

[0052] Figure 7 This is the unwrapped phase diagram described in the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described in more detail below in conjunction with examples and drawings, but the embodiments of the present invention are not limited thereto.

[0054] As attached Figure 1 As shown, the present invention proposes a joint processing method of 3D reconstruction, semantic segmentation and depth estimation based on structured light, and its main working process is:

[0055] S1: Project the sinusoidal grating fringe image onto the object to be measured through a projector and obtain the image as shown in the attached figure. Figure 3The deformed fringe image of the object after being modulated by the structured light fringe is shown. The fringe image is taken by an industrial camera and contains geometric information of the object being measured and fringe coding information.

[0056] S2: Build attachment Figure 2 The semantic segmentation network model shown in the figure adopts the improved UNet neural network model, which has a U-shaped architecture and consists of two symmetrical parts: Encoder and Decoder. Based on the original UNet model, the improved UNet neural network model adds a residual module to each layer of the Encoder. Through skip connections, the model is easier to train and can capture the relationship between contexts well, with powerful feature extraction capabilities. The predicted parcel phase map is shown in the attached figure. Figure 5 shown.

[0057] S3: Build attachment Figure 2 The depth estimation network model shown in FIG. 1 predicts the depth map of the object to be measured so as to assist in the subsequent generation of an accurate fringe order map. The depth map is shown in FIG. Figure 4 shown.

[0058] During the training process of the semantic segmentation network and the depth estimation network in steps S2 and S3, the optimizer used is the Adam optimizer, the training batch epochs is 200, and the batch size is set to 2.

[0059] S4: Convert the depth map predicted by the depth estimation into a three-dimensional point cloud through the internal and external parameters of the camera and the projector, which specifically includes the following sub-steps:

[0060] S41: Use Zhang Zhengyou calibration method to calibrate the camera, take 20 sets of high-precision origin calibration plate pictures, and use MATLAB standard tool library to calculate the camera's intrinsic parameter matrix K c and the external parameter matrix [R c |T c ].

[0061] S42: Calibrate the projector. Consider the projector as an inverse camera, project the fringe image onto the calibration board, take the fringe image with the camera, extract the feature points of the camera imaging plane using the feature point extraction algorithm, calculate the absolute phase through the fringe image, and map the feature points of the camera imaging plane to the projector imaging plane through absolute phase mapping. Then calibrate using Zhang Zhengyou's calibration method to obtain the intrinsic parameter matrix K of the projector. p and the external parameter matrix [R p |T p ].

[0062] S43: The world coordinate system and the camera / projector pixel coordinate system have the following relationship:

[0063] sc [u c , v c , 1] T =K c [R c |T c ][X W , Y W , Z w , 1] T

[0064] s p [u p , v p , 1] T =K p [R p |T p ][X W , Y W , Z W , 1] T

[0065] Among them, s c 、s p Represent the scale factors of the camera and projector respectively, (u c , v c )、(u p , v p )、(X W , Y W , Z W ) are the corresponding points in the camera and projector pixel coordinate systems and the world coordinate system respectively. Using the above relationship and the internal and external parameters of the camera and projector, the depth Z can be obtained W With the camera pixel coordinate system (u c , v c ) to convert the depth map into a three-dimensional point cloud.

[0066] S5: Based on the relationship between the 3D point cloud and the absolute phase, a rough unwrapped phase can be obtained. The specific steps are as follows:

[0067] S51: Solve the horizontal coordinate of the corresponding point on the projector pixel plane, and convert the formula from the following:

[0068]

[0069] The horizontal coordinate u of the corresponding point on the projector pixel plane can be obtained p ,in is the camera projection matrix K c [R c |T c ], is the projector projection matrix K p [R p |Tp ] in the .

[0070] S52: For the same phase value φ, the unwrapped phase of the camera pixel plane and the unwrapped phase of the projector pixel plane have the following relationship:

[0071] φ c (u c , v c )=φ p (u p )=φ

[0072] The fringe image of the projector is composed of uniform fringe patterns, so:

[0073]

[0074] Where W is the projection width of the projector. p and the projection width W of the projector, we can get (u c , v c ) point.

[0075] S6: The precise fringe order can be calculated using the rough unwrapped phase and the wrapped phase predicted by the semantic segmentation network. Since the value range of the arctan() function is between 0 and 2π, the obtained wrapped phase value range is also between 0 and 2π, and changes periodically according to the frequency of the projected fringe image. To ensure the uniqueness of the phase, the wrapped phase needs to be converted into the unwrapped phase. Obviously, the wrapped phase is unique within the periodic range, so it is only necessary to add 2kπ to obtain the unwrapped phase to ensure the global uniqueness of the phase, where k represents the periodic order. From the concept of fringe order, it can be seen that the fringe order is an integer, so even if the obtained unwrapped phase is rough, the precise fringe order can be calculated, as shown in the attached figure. Figure 6 The fringe order can be expressed as follows:

[0076]

[0077] Among them, floor(x) is a downward rounding operation, which means taking the largest integer less than or equal to x. is the parcel phase predicted by the semantic segmentation network.

[0078] S7: The wrapped phase and depth estimation predicted by the semantic segmentation model are used to obtain the accurate fringe order through a series of operations, and the accurate unwrapped phase is obtained, which can be expressed as follows:

[0079]

[0080] The final accurate unwrapped phase diagram is shown in the attached figure. Figure 7 shown.

[0081] S8: Using the relationship described in S51, the final unwrapped phase can be converted into a three-dimensional point cloud. The point cloud obtained is Z W Filter out the areas that are not of concern and beyond the boundaries to obtain the surface information of the object being measured.

[0082] In addition, the present invention also discloses a computer storage medium having a computer program stored thereon, which can execute computer instructions or programs of a joint processing method for three-dimensional reconstruction, semantic segmentation and depth estimation based on structured light provided by the present invention. When the computer instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0083] The above embodiment is a preferred implementation of the present invention, but the implementation of the present invention is not limited to these examples. Any change, modification, replacement, combination or simplification without departing from the core spirit and principle of the present invention shall be regarded as an equivalent alternative and also included in the protection scope of the present invention.

Claims

1. A joint processing method for three-dimensional reconstruction, semantic segmentation and depth estimation based on structured light, characterized by: The following steps are involved: S1: Project the sinusoidal grating fringe image onto the object to be measured through a projector to obtain a deformed fringe image of the object modulated by the structured light fringe. The fringe image is taken by an industrial camera and contains geometric information and fringe coding information of the object to be measured; S2: Build a semantic segmentation network, input the stripe image, classify the stripe image at the pixel level, identify different regions of the object and stripe encoding information, and generate a wrapped phase map of the stripe image; S3: construct a depth estimation network, input the stripe image, perform depth prediction on the stripe image, and generate a depth map, which represents the depth value of each pixel in the image; S4: Convert the depth map into a 3D point cloud based on the internal and external parameters of the camera and projector; S5: Using the relationship between the 3D point cloud and the absolute phase, the rough unwrapped phase is solved inversely; S6: Subtract the wrapped phase predicted by the semantic segmentation network from the rough unwrapped phase to obtain the fringe order map of the wrapped phase; S7: Combine the wrapped phase map predicted by the semantic segmentation network with the obtained fringe order map to obtain the accurate unwrapped phase; S8: Using the internal and external parameters of the camera and projector, the unwrapped phase is converted into a 3D point cloud to achieve accurate 3D reconstruction of the object being measured.

2. The method for joint processing of 3D reconstruction, semantic segmentation and depth estimation based on structured light according to claim 1, characterized in that: The projected sinusoidal grating stripes are: I(x,y)=A(x,y)+B(x,y)cos(2πfx) (1) Where I represents the grayscale value of each pixel of the sinusoidal grating stripes, A and B represent the background light intensity and modulation intensity respectively, and f represents the frequency of the projected sinusoidal stripes.

3. The method for joint processing of 3D reconstruction, semantic segmentation and depth estimation based on structured light according to claim 1, characterized in that: The semantic segmentation network and depth estimation network are constructed using deep learning technology and are trained with a large amount of precise data to optimize their prediction accuracy.

4. The method for joint processing of 3D reconstruction, semantic segmentation and depth estimation based on structured light according to claim 1, characterized in that: The constructed semantic segmentation network is based on the improvement of the UNet neural network. Its architecture is U-shaped and consists of two symmetrical parts: Encoder and Decoder. Its functions are: Encoder: Extracts features of the input image through a series of convolutional layers and downsampling, reduces the size of the feature map, and captures the contextual information of the image; Decoder: The features extracted by the Encoder are upsampled to the original image size for pixel-level prediction. Each layer is merged with the feature map of the corresponding layer in the Encoder through a jump connection to restore the spatial information lost during the encoding process and improve the accuracy of segmentation.

5. The method for joint processing of 3D reconstruction, semantic segmentation and depth estimation based on structured light according to claim 1, characterized in that: The constructed depth estimation network is based on the UNet neural network model, and its parameters are adjusted to adapt to the prediction of depth maps and improve the prediction accuracy.

6. The method for joint processing of 3D reconstruction, semantic segmentation and depth estimation based on structured light according to claim 1, characterized in that: The intrinsic parameters of the camera and projector are represented by the intrinsic parameter matrix K: Among them, f x and f y is the focal length parameter, c x and c y are the coordinates of the origin of the image plane; The external parameters of the camera and projector are represented by the rotation matrix R and the translation matrix T:

7. The method for joint processing of 3D reconstruction, semantic segmentation and depth estimation based on structured light according to claim 1, characterized in that: The self-made data set used by the semantic segmentation network in step S2 adopts a three-frequency twelve-step phase shift method, takes nine hundred sets of fringe images of 30 different objects in different orientations, and calculates an accurate unwrapped phase map; The self-made data set used by the depth estimation network in step S3 utilizes the above-mentioned precise unwrapped phase and the internal and external parameters of the camera and projector to calculate and obtain an accurate three-dimensional point cloud, which is then converted into a depth map.

8. The method for joint processing of 3D reconstruction, semantic segmentation and depth estimation based on structured light according to claim 1, characterized in that: The rough unwrapped phase φ obtained in step S5 c , and the wrapped phase map predicted by the semantic segmentation network, the accurate fringe order map k can be obtained, and its calculation formula is as follows: Among them, floor(x) is a downward rounding operation, which means taking the largest integer less than or equal to x. is the parcel phase predicted by the semantic segmentation network; Combining the accurate fringe order diagram with the wrapped phase, we can get the accurate unwrapped phase diagram, which is calculated as follows:

9. A three-dimensional reconstruction system based on structured light, the system implementing the method according to any one of claims 1 to 8, characterized in that: The system includes data acquisition module, semantic segmentation module, depth estimation module, 3D reconstruction module, phase processing module, and display and output module: The data acquisition module is used to obtain a stripe image containing structured light stripe coding information; The semantic segmentation module is used to perform semantic segmentation on the stripe image and generate a wrapped phase map; The depth estimation module is used to perform depth estimation on the stripe image and generate a depth map; The three-dimensional reconstruction module is used to generate a three-dimensional point cloud according to the depth map, and calculate a rough unwrapped phase using the three-dimensional point cloud and the wrapped phase map; The phase processing module is used to subtract the wrapped phase from the rough unwrapped phase to obtain a fringe order diagram, and calculate the unwrapped phase in combination with the wrapped phase diagram; The display and output module is used to display and output the results of three-dimensional reconstruction.

10. An electronic device comprising a processor and a memory, characterized in that: The processor is configured to be used to perform the joint processing method of three-dimensional reconstruction, semantic segmentation and depth estimation based on structured light according to any one of claims 1 to 9; The memory is used to store a program that supports the joint processing method of three-dimensional reconstruction, semantic segmentation and depth estimation based on structured light as described in any one of claims 1-9.

Images