Phase-shift three-dimensional reconstruction method and system of structured grating based on transfer learning

Abstract

The present invention discloses a grating phase shift three-dimensional reconstruction method and system based on transfer learning. Through an end-to-end method, a certain number of phase shift grating stripes are projected to establish a correspondence between projector pixels and camera pixels, so as to obtain a high-quality three-dimensional model of the surface of an object. Phase unwrapping is performed through a neural network, and two frequency phase stripes are accepted as input. While being able to reconstruct a three-dimensional model of the surface of an object with multiple isolated areas, the number of required collected images is reduced, and the robustness of the three-dimensional reconstruction system is enhanced; a method for generating a data-enhanced simulated data set is proposed for training a network model; at the same time, a transfer learning strategy is used to train the neural network, which can effectively deal with the situation where real data is difficult to obtain and scarce. Experiments show that the proposed method can reconstruct an accurate and complete model.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional reconstruction, and in particular relates to phase structure grating, neural network, camera calibration and object reconstruction, and specifically refers to a structured grating phase shift three-dimensional reconstruction method and system based on transfer learning. Background Art

[0002] In the field of 3D reconstruction of object surfaces based on structured gratings, how to perform high-speed, high-precision measurement and reconstruction is very challenging. Existing reconstruction methods based on phase shift usually require an inverse tangent operation to obtain the precise phase, which will cause the generated wrapped phase to generate a phase shift error between each cycle. The phase jump of the image is called phase unwrapping, which requires the phase unwrapping operation to eliminate the ambiguity in the phase representation and convert the phase into an absolute phase. In the past decade, many phase unwrapping algorithms have been proposed, which can be roughly divided into the following two categories according to the different fields of action: spatial unwrapping and temporal unwrapping. The spatial unwrapping method is based on the assumption of phase continuity. It analyzes the information of pixels around a pixel in a single wrapped phase to determine the relative stripe order of the central pixel. Therefore, the spatial unwrapping method cannot successfully measure discontinuous or isolated object areas, and the training of the network requires the collection of a large amount of data. The corresponding temporal unwrapping method can achieve pixel-level phase unwrapping by sequentially analyzing the wrapped phase of more than one different frequency, and because of the use of multi-frequency wrapped phase, it can effectively deal with truncated and isolated object areas. Generally speaking, among the mainstream temporal unwrapping methods, the multi-frequency heterodyne method is highly reliable and robust to noise, and in order to obtain accurate unwrapped phases, it is usually necessary to use more than three frequencies of wrapped phase calculations, which undoubtedly increases the complexity and time consumption of three-dimensional reconstruction. Summary of the invention

[0003] The present invention is to overcome the above shortcomings and meet the needs of real-time and accuracy of structured grating phase shift 3D reconstruction, and proposes a structured grating phase shift 3D reconstruction method and system based on transfer learning. It can quickly and accurately reconstruct the 3D surface model of objects with truncated surfaces and isolated areas.

[0004] The present invention discloses a method for three-dimensional reconstruction of phase shift of structured grating based on neural network, comprising the following steps:

[0005] Step S1: constructing a neural network for predicting the unwrapped phase, including an encoder composed of a residual module embedded in a channel attention module, a decoder composed of cascaded refinement modules, and a feature fusion module for fusing multi-scale features extracted by the encoder and the decoder to predict the absolute phase, wherein the input of the neural network is the wrapped phase;

[0006] Among them, the refinement module adopts a chained multi-head attention interaction mechanism and uses a learnable interactive parameter modulator to dynamically adjust the attention weight. The chained multi-head attention interaction mechanism uses multiple branches to utilize the input features, uses global pooling and fully connected layers to obtain the weights of each channel feature, uses the feature vector learned by the parameter modulator to adjust and fuse the channel feature weights of each branch, and dynamically adjusts the feature maps of each stage according to the weights.

[0007] Step S2: Simulating data set generation, using random interpolation and adding random noise and masking methods to generate several sets of wrapped phases and unwrapped phases for neural network training as data sets for neural network training;

[0008] Step S3: real data acquisition and data set generation, collecting multiple groups of three-frequency four-step phase shift structured light modulated by the object, obtaining the wrapped phase through the four-step phase shift method, and using the three-frequency heterodyne method to obtain the absolute phase after unwrapping to obtain the real data set;

[0009] Step S4: Neural network training: first use the simulated data set to train the neural network, minimize the L1 regularization loss and residual root mean square loss, and obtain the network parameters. Then perform transfer learning, freeze the encoder network parameters, continue training with the real data set, adjust the decoder network parameters, and obtain a fully trained neural network.

[0010] Step S5: data acquisition and processing, collecting a number of fringe images modulated by the calibration plate and the object;

[0011] Step S6: Obtain the unwrapped phase, obtain the wrapped phase through four-step phase shift, input the wrapped phase into the neural network described in S4, and obtain the unwrapped phase corresponding to the input high-frequency wrapped phase;

[0012] Step S7: System calibration, first use the checkerboard calibration plate to calibrate the camera and obtain the camera parameters; then establish a projector-inverse camera model, use the unwrapped phase modulated by the calibration plate obtained and processed in S6 above to calibrate the projector and obtain the projector parameters; finally, use the stereo calibration method to obtain the system parameters;

[0013] Step S8: Establishing a triangulated model of the projector, camera, and object, using the unwrapped phase modulated by the object obtained and processed in S6 and the system parameters obtained in S7 to calculate the three-dimensional information of the object surface;

[0014] Furthermore, the decoder input feature numbers are all feature numbers of the encoding network at the corresponding level. Except for the deepest layer which only accepts the input of the encoding network at the corresponding level, other layers of the decoding network simultaneously receive the output of the encoding network at the corresponding level and the feature maps output by the previous layer, and finally output the predicted data of the original size with a feature number of 1 through the prediction convolution module.

[0015] Furthermore, the specific process of step S2 is as follows:

[0016] S2-1, first randomly generate small matrices of different sizes, use quadratic interpolation to enlarge the small matrices, and then linearly scale their values ​​to The absolute phase after unwrapping is obtained within the interval, which can be set as needed, and continuity is ensured by interpolation, and randomness is ensured by random parameter selection of the initial matrix;

[0017] S2-2, construct the high-frequency wrapped phase by the generated absolute phase, according to the following formula:

[0018]

[0019]

[0020]

[0021] in is the absolute phase of the unwrapped and is the corresponding coordinate, and denote the real part and the imaginary part respectively, To obtain the wrapped phase, the real and imaginary parts of the absolute phase are obtained, and the high-frequency wrapped phase is obtained by the inverse tangent function;

[0022] S2-3, construct unit wrapped phase through absolute phase, and linearly map the absolute phase value to The interval gets the unit wrapping phase;

[0023] S2-4, adds random Gaussian noise to some of the wrapped phases generated by S2-2 and S2-3; adds random masks to a set of data sets generated by S2-1, S2-2 and S2-3, divides the continuous phase into several isolated random geometric regions, simulates the discontinuity of the surface phase of the object, and finally maps the processed wrapped phases to And generate a grayscale image.

[0024] Furthermore, the specific process of step S3 is as follows:

[0025] Prepare computers, shooting devices and projection devices. The computer controls the shooting devices and projection devices and processes the data collected by the shooting devices.

[0026] In a dark room, a projector is used to sequentially project phase-shifted structured light stripe patterns;

[0027] Collect multiple sets of four-step phase-shifted structured light images at four frequencies after being modulated by the object, and select the frequency , , and , phase difference ,Firstly, the phase principal value of the wrapped phase of the grating fringe pattern with the same frequency is solved by four-step phase shift, and then the corresponding high-frequency absolute phase is obtained by three-frequency heterodyne synthesis and expansion. The high-frequency absolute phase, the corresponding high-frequency wrapped phase and the unit frequency phase are used to form a real data set.

[0028] The present invention establishes a projector-inverse camera model through the data acquisition device, and uses a checkerboard calibration plate to calibrate the data acquisition device; captures images in a dark environment, and obtains the correspondence between the projection pixels of the digital grating projector and the pixels of the captured image through the captured images; and uses a four-step phase shift method to obtain the pre-encoded sinusoidal structured light stripes projected by the projector based on the captured image data.

[0029] Furthermore, step S6 adopts a four-step phase shift method to obtain the wrapping phase.

[0030] Furthermore, the hardware for data collection in step S5 also includes a calibration board. For the data used for calibration, an image of a solid color background projected by a projector is collected for camera calibration.

[0031] Furthermore, the calibration in step S7 specifically includes:

[0032] First, the camera is calibrated. With the help of a checkerboard calibration plate, the Zhang Zhengyou calibration method is used to obtain the camera's intrinsic parameters.

[0033] Then, a projector-inverse camera model is established. The correspondence between the projector light and the camera light is obtained through the absolute phase. The projector imaging process is regarded as the inverse process of camera imaging. The projector is calibrated using the Zhang Zhengyou calibration method to obtain the internal parameters of the projector.

[0034] Finally, the binocular stereo calibration of the camera and projector is performed using the binocular calibration method to obtain the external parameters of the system.

[0035] Based on the same inventive concept, the present invention also designs a three-dimensional reconstruction system of a structured grating phase shift three-dimensional reconstruction method based on a neural network, including a structured light image acquisition device, a predictive unwrapping phase neural network and a three-dimensional reconstruction module, wherein:

[0036] The structured light image acquisition device is used to acquire real data of structured light images used for training, as well as structured light image data to be reconstructed in three dimensions;

[0037] A neural network for predicting an unwrapped phase, the network comprising an encoder composed of a residual module embedded in a channel attention module, a decoder composed of cascaded refinement modules, and a feature fusion module for fusing multi-scale features extracted by the encoder and the decoder to predict the absolute phase, wherein the input of the neural network is the wrapped phase;

[0038] The wrapped phase was generated using the random matrix expansion and filling method, and then a simulated data set was generated after adding random Gaussian noise and a random number of graphic masks;

[0039] Collect structured light image data, and obtain the high-frequency absolute phase, the corresponding high-frequency package phase, and the unit frequency phase of the structured light image data, and use the above phase data as a real data set;

[0040] First, use the simulated data set to train the neural network, minimize the L1 regularization loss and the residual root mean square loss, obtain the network parameters, and then perform transfer learning, freeze the encoder network parameters, continue training with the real data set, adjust the decoder network parameters, and obtain the trained neural network;

[0041] The three-dimensional reconstruction module first collects the structured light image; obtains the wrapped phase, and uses the trained neural network to obtain the unwrapped phase; uses the unwrapped phase to calibrate the projector and obtain the projector parameters; finally, uses the stereo calibration method to obtain the system parameters; establishes a triangulated model of the projector, camera, and object, and uses the unwrapped phase and the system parameters to calculate the three-dimensional information of the object surface.

[0042] Based on the same inventive concept, the present invention also discloses an electronic device, comprising:

[0043] one or more processors;

[0044] A storage device for storing one or more programs;

[0045] When one or more programs are executed by the one or more processors, the one or more processors implement a structured grating phase shift three-dimensional reconstruction method based on transfer learning.

[0046] Based on the same inventive concept, the present invention also discloses a computer-readable medium on which a computer program is stored. When the program is executed by a processor, a structured grating phase shift three-dimensional reconstruction method based on transfer learning is implemented.

[0047] Through the neural network-based structured grating phase shift three-dimensional reconstruction method of the present invention, by preparing a data set and training a neural network, collecting data through a data acquisition device, and the subsequent data processing and calculation process, high-quality object surface three-dimensional data information can be quickly obtained. This method uses transfer learning to greatly reduce the amount of collected data. While using dual-frequency data, phase space information is also used through the loss function. Its main advantages are as follows:

[0048] 1. An innovative method for 3D reconstruction of structured gratings based on transfer learning is proposed. A unique network structure is designed. A chain multi-head attention interaction mechanism is designed in the refinement module, and a learnable interaction parameter modulator is used to dynamically adjust the attention weight. The network structure is trained for transfer learning by using simulated data pre-training and collected data reinforcement training. By collecting four-step phase-shifted stripe grating patterns of high frequency and unit frequency within a reasonable range, the wrapped phase is obtained and input into the neural network to obtain the absolute phase for 3D reconstruction, which can quickly, robustly, and end-to-end obtain the 3D information of the object surface.

[0049] 2. An innovative neural network structure for phase unwrapping is proposed. It receives the wrapped phase obtained by four-step phase shift as input, establishes a mapping relationship between the wrapped phase and the unwrapped absolute phase, and quickly obtains the unwrapped phase that is robust to noise. The network inputs two frequencies to obtain the absolute phase, which can use less data than more than two frequencies, and can effectively deal with truncated and isolated surface objects compared to a single frequency. The encoder uses a multi-scale residual network combined with a channel attention mechanism, and the decoder efficiently uses a cascaded refinement module. A unique chain multi-head interactive attention mechanism is designed internally, and an innovative loss function combining L1 loss and residual root mean square loss is proposed. The phase error and smoothness are minimized during training to obtain high-quality absolute phase.

[0050] 3. An innovative method for generating large-scale wrapped phase-unwrapped phase simulation data sets is proposed, and data enhancement is performed through random noise and random mask processing.

[0051] 4. An innovative transfer learning strategy was used. The neural network was first pre-trained on a simulated data set, and then the encoder network parameters were fixed. The real data set consisting of a small amount of real data was used for training. This reduced the workload of collecting real data, solved the problem of difficulty in obtaining real data, and reduced the error of the network's absolute phase prediction.

[0052] 5. Use the triangulation method to establish the mathematical model of the projector light and the camera light to obtain the three-dimensional information of the object surface. The proposed method is verified using simulated data and real data, which shows that this method can quickly and accurately reconstruct the three-dimensional model of the surface of objects with truncated surfaces and isolated areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a system flow chart of the neural network-based structured grating phase shift three-dimensional reconstruction method of the present invention.

[0054] Figure 2 4 is a diagram of a neural network structure according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to further understand the content of the present invention and make the objectives, technical solutions and advantages of the present invention clearer, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0056] Embodiment 1

[0057] As attached Figure 1 As shown, this embodiment provides a method for three-dimensional reconstruction of structured grating phase shift based on transfer learning. Through a specific data acquisition device, a checkerboard calibration plate is used to calibrate the entire data acquisition device; in a dark environment, an image is taken, and the correspondence between the projector light and the camera light is obtained based on the taken image. We use the phase shift method to obtain clear wrapped phase stripes modulated by the object; a specific neural network for recovering from the wrapped phase to the unwrapped phase is proposed; a method for generating simulated data for training the network is proposed, and a data enhancement method is used; through the transfer learning method, the real data set obtained by the collection and calculation is used to continue training the initially trained network to obtain unwrapped phase stripes that meet the characteristics of the actual data; finally, a general triangulation method is used to obtain the three-dimensional data of the object surface. The proposed method is verified on real and simulated data sets, and the results show that this method can quickly and robustly reconstruct the three-dimensional model of the object surface. The main steps are as follows:

[0058] Step S1: Construct a neural network for predicting the unwrapped phase, a multi-scale encoding network for feature extraction, and a decoding network for dense prediction. The overall network is implemented using convolution, channel attention, and multi-head interactive attention mechanisms, as shown in the attached figure. Figure 2As shown in the figure, the encoding network is a multi-scale convolutional network, combined with a channel attention mechanism and including convolutional layers and pooling layers; the decoding network is a dense prediction network composed of cascaded refinement modules. A chain multi-head attention interaction mechanism is designed in the refinement module, and a learnable interactive parameter modulator is used to dynamically adjust the attention weight. The multi-head attention interaction mechanism uses multiple branches to utilize the input features, uses global pooling and fully connected layers to obtain the feature weights of each channel, uses the feature vector learned by the parameter modulator to adjust and fuse the channel feature weights of each branch, and dynamically adjusts the feature maps of each stage according to the weights. The absolute phase is predicted by fusing the multi-scale features extracted by the encoding network and the decoding network.

[0059] Furthermore, in step S1, the number of input features of the encoding network is 2, and the number of features is increased to 16 after the initial convolution module. After four downsamplings of the encoding network, the number of features becomes 32, 64, 128 and 256 respectively, and the length and width of the feature map become the corresponding input size. , , and ;

[0060] Furthermore, in step S1, the input feature numbers of the decoding network are the feature numbers of the encoding network at the corresponding level. Except that the deepest layer only accepts the encoding network at the corresponding level as input, the other levels of the decoding network simultaneously receive the output of the encoding network at the corresponding level and the feature maps output by the previous level. The number of feature maps is determined through actual experiments. In this embodiment, the feature numbers of all layers are 256. Finally, the prediction data of the original size with a feature number of 1 is output through the prediction convolution module.

[0061] Step S2: Simulate data set generation. The generation method uses a random matrix expansion and filling method, and then adds random Gaussian noise and a random number of graphic masks. A set of data includes a two-dimensional matrix data and two grayscale images, which are used for the absolute phase after unwrapping, the high-frequency wrapped phase, and the unit wrapped phase, respectively. The unwrapped phase is used as the true value, and the two wrapped phase images are used as input.

[0062] S2-1, first randomly generate small matrices of different sizes (randomly set to ), and then use quadratic interpolation to scale the matrix to size, and then scale its value linearly to The absolute phase after unwrapping is obtained within the interval. This interval can be set as needed. Continuity is ensured by interpolation, and randomness is ensured by random parameter selection of the initial matrix.

[0063] S2-2, construct the high-frequency wrapped phase by the generated absolute phase, according to the following formula:

[0064]

[0065]

[0066]

[0067] in is the absolute phase of the unwrapped and is the corresponding coordinate, and are the real and imaginary parts, is the wrapped phase. Get the real and imaginary parts of the absolute phase, and use the inverse tangent function to get the high-frequency wrapped phase.

[0068] S2-3, construct unit wrapped phase through absolute phase, and linearly map the absolute phase value to The interval gets the unit wrapped phase.

[0069] S2-4, adds a random SNR range to the partially wrapped phases generated by S2-2 and S2-3 db Gaussian noise to simulate the real environment noise; add a random mask to a set of data sets generated by parts S2-1, S2-2 and S2-3 to divide the continuous phase into several isolated random geometric regions to simulate the discontinuity of the surface phase of the object. Finally, the processed wrapped phases are mapped to The “part” here is determined by the characteristics of the actual acquisition object in the specific usage scenario, and is determined through experiments according to the surface complexity and amount of data of the reconstructed object. The proportion used in this embodiment is 20%.

[0070] Step S3: Real data acquisition and data set generation. The equipment required for the acquisition device includes: a computer, a camera, and a projector. In order to avoid the influence of ambient light, the entire acquisition process is carried out in a dark room. The only light source comes from the projector, which projects phase-shifted structured light stripe patterns in sequence. Collect multiple sets of four-step phase-shifted structured light images at four frequencies after object modulation. , , and (unit frequency), phase difference , a total of 16 images. First, the phase principal value of the wrapped phase of the grating fringe pattern with the same frequency is solved by four-step phase shift, and then the corresponding high-frequency absolute phase is obtained by three-frequency heterodyne synthesis and expansion. The high-frequency absolute phase, the corresponding high-frequency wrapped phase and the unit frequency phase are used to form a set of real data sets.

[0071] Step S4: Neural network training, using the transfer learning strategy, first uses the simulated data set generated by S2 for training, minimizes the loss function, and defines the loss function as the sum of the L1 loss function and the residual root mean square loss:

[0072]

[0073]

[0074]

[0075] in is the true value of the unwrapped phase at this coordinate, is the predicted value of the neural network, , is the difference in the gradient direction corresponding to the true value, , is the difference in the gradient direction corresponding to the predicted value. After the training network converges, the network training parameters are fixed and encoded, and the network is continued to be trained using the data set obtained in S3, using the same loss function until the network converges.

[0076] Step S5: Data acquisition. The equipment required for the acquisition device includes: a computer, a camera, a projector, and a calibration plate. In order to avoid the influence of ambient light, the entire acquisition process is carried out in a dark room. The only light source comes from the projector, which projects phase-shifted structured light stripe patterns in sequence. For non-calibrated data, a total of 16 images are collected for each group, of which 8 stripes are projected in the horizontal and vertical directions, including 4 phase-shifted stripes. High frequency fringes and 4 phase difference For the data used for calibration, finally collect an image of a pure color background projected by the projector for camera calibration. The number of calibration data collected is not less than 4 groups.

[0077] Step S6: Obtain the unwrapped phase, and use the four-step phase shift method to obtain the phase principal value of the wrapped phase of the grating fringe pattern at the same frequency in step S5. Then, for each set of data, the two obtained wrapped phase principal value images of different frequencies are input into the neural network trained in S4 to obtain the absolute phase predicted by the network.

[0078] Step S7: System calibration, using the images processed by S5-S6 for calibration. First, perform camera calibration. With the help of a checkerboard calibration board, use Zhang Zhengyou's calibration method to obtain the camera's intrinsic parameters. Then establish a projector-inverse camera model, obtain the correspondence between the projector light and the camera light through the absolute phase, regard the projector imaging process as the inverse process of camera imaging, and use Zhang Zhengyou's calibration method to calibrate the projector to obtain the projector's intrinsic parameters. Finally, use the binocular calibration method to perform binocular stereo calibration on the camera and projector to obtain the system's extrinsic parameters.

[0079] Step S8: Establish triangulated models of the projector, camera and object, and use the unwrapped phase modulated by the object obtained and processed in S6 and the system parameters obtained in S7 to calculate the three-dimensional information of the object surface according to the following formula:

[0080]

[0081]

[0082]

[0083] in, and The projection matrices of the camera and projector are obtained by calibration. For a given camera pixel coordinate And the projector horizontal coordinates , the three-dimensional coordinates of the corresponding object point can be obtained , and .

[0084] After the initial training in step S4 converges, the percentage of the root mean square error to the generated phase amplitude h on the validation set is as low as 0.2%; after transfer learning, the generalization of the network can be enhanced while retaining the network performance to the maximum extent. The accuracy of the three-dimensional coordinates of the object obtained through subsequent calculations is affected by the specific equipment accuracy of the system, and its error is positively correlated with the phase error.

[0085] Embodiment 2

[0086] Based on the same inventive concept, the present invention also discloses a structured grating phase shift three-dimensional reconstruction system based on transfer learning, including a structured light image acquisition device for acquiring real data used for training and data to be reconstructed three-dimensionally;

[0087] A neural network for predicting an unwrapped phase includes an encoder composed of a residual module embedded in a channel attention module, a decoder composed of a cascaded refinement module, a chain multi-head attention interaction mechanism is designed in the refinement module, and a learnable interaction parameter modulator is used to dynamically adjust the attention weight, and a feature fusion module for fusing multi-scale features extracted by the encoder and decoder to predict the absolute phase, the input of the neural network is the wrapped phase; the training of the neural network is as follows:

[0088] First, use the simulated data set to train the neural network, minimize the L1 regularization loss and the residual root mean square loss, obtain the network parameters, and then perform transfer learning, freeze the encoder network parameters, continue training with the real data set, adjust the decoder network parameters, and obtain a fully trained neural network;

[0089] A simulation data set generation module uses random interpolation, random noise addition and masking methods to generate several sets of wrapped phases and unwrapped phases for neural network training as data sets for neural network training;

[0090] A data processing module processes the data from the data acquisition module to obtain the absolute phase after unwrapping;

[0091] The 3D reconstruction module uses the trained neural network to obtain the unwrapped phase; calibrates the projector to obtain the projector parameters; finally, uses the stereo calibration method to obtain the system parameters; establishes a triangulated model of the projector, camera, and object, and uses the unwrapped phase and system parameters to calculate the 3D information of the object surface.

[0092] Since the system introduced in the second embodiment of the present invention is a system adopted for implementing the structured grating phase shift three-dimensional reconstruction method based on transfer learning in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, technical personnel in this field can understand the specific structure and deformation of the electronic device, so it will not be repeated here.

[0093] Embodiment 3

[0094] Based on the same inventive concept, the present invention also provides an electronic device, including one or more processors; a storage device for storing one or more programs; when one or more programs are executed by the one or more processors, the one or more processors implement the method described in Example 1.

[0095] Since the device introduced in the third embodiment of the present invention is an electronic device used to implement the method for phase shifting three-dimensional reconstruction of structured gratings based on transfer learning in the first embodiment of the present invention, a person skilled in the art can understand the specific structure and deformation of the electronic device based on the method introduced in the first embodiment of the present invention, so it is not repeated here. All electronic devices used in a method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0096] Embodiment 4

[0097] Based on the same inventive concept, the present invention further provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method described in the first embodiment is implemented.

[0098] Since the device introduced in the fourth embodiment of the present invention is a computer-readable medium used to implement the method for three-dimensional reconstruction of phase shift of structured grating based on transfer learning in the first embodiment of the present invention, the technical personnel in the field can understand the specific structure and deformation of the electronic device based on the method introduced in the first embodiment of the present invention, so it is not repeated here. All electronic devices used in a method of the embodiment of the present invention belong to the scope of protection of the present invention.

[0099] The above-described embodiment only expresses a specific implementation of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of this application shall be based on the attached claims.

Claims

1. A three-dimensional reconstruction method of structured grating phase shift based on transfer learning, characterized in that: The following steps are involved: Step S1: constructing a neural network for predicting the unwrapped phase, including an encoder composed of a residual module embedded in a channel attention module, a decoder composed of cascaded refinement modules, and a feature fusion module for fusing multi-scale features extracted by the encoder and the decoder to predict the absolute phase, wherein the input of the neural network is the wrapped phase; Among them, the refinement module adopts a chained multi-head attention interaction mechanism and uses a learnable interactive parameter modulator to dynamically adjust the attention weight. The chained multi-head attention interaction mechanism uses multiple branches to utilize the input features, uses global pooling and fully connected layers to obtain the weights of each channel feature, uses the feature vector learned by the parameter modulator to adjust and fuse the channel feature weights of each branch, and dynamically adjusts the feature maps of each stage according to the weights. Step S2: Use the random matrix expansion and filling method to obtain the absolute phase after unwrapping, and construct the high-frequency wrapped phase through the generated absolute phase according to the following formula: in is the absolute phase of the unwrapped and is the corresponding coordinate, and denote the real part and the imaginary part respectively, To obtain the wrapped phase, the real and imaginary parts of the absolute phase are obtained, and the high-frequency wrapped phase is obtained by the inverse tangent function; The unit wrapped phase is constructed by the absolute phase, and the absolute phase value is linearly mapped to The interval gets the unit wrapping phase; Adding random Gaussian noise and random number pattern masks to the generated high-frequency wrapped phase and unit wrapped phase generates a simulated data set; Step S3: collecting structured light image data, and obtaining the high-frequency absolute phase, the corresponding high-frequency package phase and the unit frequency phase of the structured light image data, and taking the above phase data as a real data set; Step S4: Neural network training: first use the simulated data set to train the neural network, minimize the L1 regularization loss and the residual root mean square loss, obtain the network parameters, and then perform transfer learning, freeze the encoder network parameters, continue training with the real data set, adjust the decoder network parameters, and obtain the trained neural network; Step S5: collecting structured light images; Step S6: obtaining a wrapped phase, and using the neural network to obtain an unwrapped phase; Step S7: Use the unwrapped phase obtained in step S6 to calibrate the projector and obtain projector parameters; finally, use a stereo calibration method to obtain system parameters; Step S8: Establish a triangulated model of the projector, camera, and object, and use the unwrapped phase and the system parameters in step S7 to calculate and obtain the three-dimensional information of the object surface.

2. The method for three-dimensional reconstruction of structured grating phase shift based on transfer learning according to claim 1, characterized in that: The decoder input feature numbers are all the feature numbers of the encoding network at the corresponding level. Except for the deepest layer which only accepts the input of the encoding network at the corresponding level, the other layers of the decoding network simultaneously receive the output of the encoding network at the corresponding level and the feature maps output by the previous layer. Finally, the prediction convolution module outputs the prediction data of the original size with a feature number of 1.

3. The method for three-dimensional reconstruction of structured grating phase shift based on transfer learning according to claim 1, characterized in that: The specific process of generating the simulated data set in step S2 is as follows: Add random Gaussian noise to the generated high-frequency wrapped phase and unit wrapped phase; add random masks to a set of data sets generated by partial absolute phase, high-frequency wrapped phase and unit wrapped phase, divide the continuous phase into several isolated random geometric regions, simulate the discontinuity of the surface phase of the object, and finally map the processed wrapped phase to And generate a grayscale image.

4. The method for three-dimensional reconstruction of structured grating phase shift based on transfer learning according to claim 1, characterized in that: The specific process of step S3 is: Prepare computers, shooting devices and projection devices. The computer controls the shooting devices and projection devices and processes the data collected by the shooting devices. In a dark room, a projector is used to sequentially project phase-shifted structured light stripe patterns; Collect multiple sets of four-step phase-shifted structured light images at four frequencies after being modulated by the object, and select the frequency , , and , phase difference ,Firstly, the phase principal value of the wrapped phase of the grating fringe pattern with the same frequency is solved by four-step phase shift, and then the corresponding high-frequency absolute phase is obtained by three-frequency heterodyne synthesis and expansion. The high-frequency absolute phase, the corresponding high-frequency wrapped phase and the unit frequency phase are used to form a real data set.

5. The method for three-dimensional reconstruction of structured grating phase shift based on transfer learning according to claim 4, characterized in that: Step S6 uses a four-step phase shift method to obtain the wrapping phase.

6. The method for three-dimensional reconstruction of structured grating phase shift based on transfer learning according to claim 1, characterized in that: The hardware for data collection in step S5 also includes a calibration board. For the data used for calibration, an image of a projector projecting a pure color background is collected for camera calibration.

7. The method for three-dimensional reconstruction of structured grating phase shift based on transfer learning according to claim 1, characterized in that: The calibration in step S7 specifically includes: First, the camera is calibrated. With the help of a checkerboard calibration plate, the Zhang Zhengyou calibration method is used to obtain the camera's intrinsic parameters. Then, a projector-inverse camera model is established. The correspondence between the projector light and the camera light is obtained through the absolute phase. The projector imaging process is regarded as the inverse process of camera imaging. The projector is calibrated using the Zhang Zhengyou calibration method to obtain the internal parameters of the projector. Finally, the binocular stereo calibration of the camera and projector is performed using the binocular calibration method to obtain the external parameters of the system.

8. A 3D reconstruction system based on the 3D reconstruction method according to any one of claims 1 to 7, characterized in that: It includes a structured light image acquisition device, a predictive unwrapping phase neural network and a three-dimensional reconstruction module, wherein: The structured light image acquisition device is used to acquire real data of structured light images used for training, as well as structured light image data to be reconstructed in three dimensions; A neural network for predicting unwrapped phase, comprising an encoder composed of a residual module embedded in a channel attention module, a decoder composed of cascaded refinement modules, a chained multi-head attention interaction mechanism designed in the refinement module, and a learnable interaction parameter modulator to dynamically adjust the attention weight, and a feature fusion module for fusing multi-scale features extracted by the encoder and decoder to predict the absolute phase, wherein the input of the neural network is the wrapped phase; The absolute phase after unwrapping is obtained by using the random matrix expansion and filling method, and the high-frequency wrapped phase is constructed by the generated absolute phase according to the following formula: in is the absolute phase of the unwrapped and is the corresponding coordinate, and denote the real part and the imaginary part respectively, To obtain the wrapped phase, the real and imaginary parts of the absolute phase are obtained, and the high-frequency wrapped phase is obtained by the inverse tangent function; The unit wrapped phase is constructed by the absolute phase, and the absolute phase value is linearly mapped to The interval gets the unit wrapping phase; Adding random Gaussian noise and random number pattern masks to the generated high-frequency wrapped phase and unit wrapped phase generates a simulated data set; Collect structured light image data, and obtain the high-frequency absolute phase, the corresponding high-frequency package phase, and the unit frequency phase of the structured light image data, and use the above phase data as a real data set; First, use the simulated data set to train the neural network, minimize the L1 regularization loss and the residual root mean square loss, obtain the network parameters, and then perform transfer learning, freeze the encoder network parameters, continue training with the real data set, adjust the decoder network parameters, and obtain the trained neural network; The three-dimensional reconstruction module first collects the structured light image; obtains the wrapped phase, and uses the trained neural network to obtain the unwrapped phase; uses the unwrapped phase to calibrate the projector and obtain the projector parameters; finally, uses the stereo calibration method to obtain the system parameters; establishes a triangulated model of the projector, camera, and object, and uses the unwrapped phase and the system parameters to calculate the three-dimensional information of the object surface.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images