Adaptive deep learning fringe analysis method based on multi-expert cooperation mechanism

Abstract

The invention discloses an adaptive deep learning fringe analysis method based on a multi-expert cooperation mechanism. The objective of obtaining cross-system generalization is achieved by fusing a data field and a network structure level design. According to the method, a self-adaptive expert mixing idea is adopted, and the composition strategy of the network is determined by analyzing input data. And different from an expert hybrid framework which is directly switched among different networks, a framework with high expansibility and flexibility is formed through real-time synthesis in a parameter space. Once training is completed, the method can automatically adapt to different input data and calculate and fuse the optimal weight of different experts, thereby realizing adaptive fringe image analysis under different imaging conditions. In addition, a denoising auto-encoder method is used in the pre-training stage to reduce resource consumption, and a data enhancement and Fourier loss method is introduced in the training stage to further enhance performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical measurement, and in particular relates to an adaptive deep learning fringe analysis method based on a multi-expert collaboration mechanism. Background Art

[0002] In recent years, as the fringe analysis methods based on deep learning have been widely used in related fields and achieved effective results, some have achieved performance far beyond traditional methods. This new generation of fringe analysis method framework is expected to break through the efficiency and accuracy limitations of traditional methods and become the core technology of the next generation of optical measurement systems. Although effective, these current deep learning fringe analysis methods still face huge challenges. They usually perform well only when the imaging conditions are the same, and when the shooting conditions of the test data vary greatly, the performance of the neural network will drop sharply. Secondly, even if it is the same measurement system, after adjusting the system parameters and re-adjusting, although the task is the same, the network model previously trained on the system cannot be directly applied to the adjusted system. The present invention achieves the goal of obtaining cross-system generalization by integrating the data domain and the network structure level design. Different from the traditional framework based on fixed parameter networks, it adopts the concept of adaptive expert mixing and determines the network composition strategy by analyzing the input data. Moreover, unlike the expert mixing framework that directly switches between different networks, it is realized by real-time synthesis in the parameter space to form an architecture with high scalability and flexibility, thereby solving the cross-system fringe analysis problem in deep learning fringe analysis. Summary of the invention

[0003] The present invention aims to provide an adaptive deep learning fringe analysis method based on a multi-expert collaboration mechanism.

[0004] The technical solution to achieve the purpose of the present invention is: an adaptive deep learning fringe analysis method based on a multi-expert collaboration mechanism, comprising:

[0005] Step 1: Construct a fringe projection dataset with different fringe frequencies and contrasts;

[0006] Step 2: Construct several expert neural networks, and select several groups of data that can cover different data domain requirements from the data sets of different fringe frequencies and contrasts constructed in step 1 to pre-train each expert neural network;

[0007] Step 3: Construct a gate network for controlling the synthesis of the expert neural network, use the fringe projection dataset of step 1 to collaboratively train the pre-trained expert neural network and the gate network, use the parameters obtained from the collaborative training to construct an expert collaboration network for cross-system fringe analysis, and use the fringe projection dataset of step 1 to train the expert collaboration network;

[0008] Step 4: Input the real-time fringe pattern into the trained expert collaboration network to obtain the wrapped phase.

[0009] Preferably, the specific method for constructing a fringe projection data set with different fringe frequencies and contrasts is:

[0010] For different test scenes, a projector is used to project several sinusoidal phase-shifted fringe images with different frequencies and contrasts, and a camera is used to shoot the test scene illuminated by the fringe. The projected fringe images are obtained as a data set and data enhancement operations are performed on the data set.

[0011] The labels of the dataset are the numerator and denominator of the wrapped phase calculated by the twelve-step phase shift method.

[0012] Preferably, the projected fringe image is represented as:

[0013]

[0014] Where (x, y) is the pixel coordinate of the projector, index n = 0, 1, 2, ..., 11; A p , B p , f are the fringe amplitude, modulation degree and spatial frequency respectively.

[0015] Preferably, the structure of the expert neural network adopts the Res-Unet structure, and the pre-training process adopts the self-supervised denoising autoencoder method.

[0016] Preferably, the specific steps of pre-training the expert neural network are:

[0017] For each expert neural network, a series of fringe patterns at set frequency and contrast are taken from the data set constructed in step 1, and a certain proportion of Gaussian noise is added to the fringe patterns as the input of the expert neural network. The original fringe patterns without noise are used as labels, and this is used as a data set for pre-training of the specific expert neural network.

[0018] Preferably, the specific formula for adding noise to the fringe image is:

[0019] I σ (x,y)=I c (x,y)+σ

[0020] Among them I c (x,y) and I σ (x, y) are the normalized original raster image and the image after adding noise, respectively, and σ is the image noise.

[0021] Preferably, the gate network adopts a Vgg network architecture.

[0022] Preferably, the fringe projection data set of step 1 is used to collaboratively train the pre-trained expert neural network and the gate network, and the parameters obtained by the collaborative training are used to construct an expert collaborative network for cross-system fringe analysis. The specific method is:

[0023] The fringe image in the data set is input into the gate network, and the gate network extracts features and then outputs the adaptive synthesis weights;

[0024] The weights output by the gate network are fused with the parameters of the pre-trained network as the parameters of the expert collaboration network, and the structure of the expert collaboration network is the same as that of the expert neural network;

[0025] The fringe pattern in the data set is input into the expert collaboration network to obtain the corresponding wrapped phase numerator and denominator, the loss function of the wrapped phase numerator and denominator output by the expert collaboration network and the label is calculated, and the parameters are updated to complete the training process.

[0026] Preferably, the parameters of the expert collaboration network are specifically:

[0027]

[0028] Where P i are the parameters of each expert neural network, P MoE is the expert collaboration network parameter after dynamic synthesis, w i Output weights for the gate network, n is the number of expert networks.

[0029] Preferably, the loss function of the parcel phase numerator, denominator and label output by the expert collaboration network is specifically:

[0030] L loss =λ1L MSE +λ2L Fourier

[0031] Among them, λ1 and λ2 are the weights of different loss functions, L MSE is the spatial consistency loss function L MSE The calculation method is

[0032]

[0033] where y pred is the network prediction value, y true is the label value, k is the total pixel value of the image;

[0034] L Fourier is the Fourier loss function, L Fourier The expression is:

[0035]

[0036] in, is the discrete Fourier transform.

[0037] Compared with the prior art, the present invention has the following significant advantages: the present invention adopts the concept of adaptive expert hybrid, determines the network composition strategy by analyzing the input data, and forms a highly scalable and flexible architecture through real-time synthesis in the parameter space;

[0038] The present invention uses a denoising autoencoder method in the pre-training stage to reduce resource consumption, and introduces data enhancement and Fourier loss methods in the training stage to further enhance performance;

[0039] The present invention is superior to traditional expert neural networks designed based on single-system data domains and large-parameter networks for multi-system data domains in both seen and unseen situations, and has obvious relative advantages when faced with situations that are more difficult to handle. It can maintain excellent stability and robustness in the face of different systems determined by multiple parameters such as camera settings, fringe characteristics, and environmental factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of an adaptive deep learning fringe analysis method based on multi-expert collaboration mechanism.

[0041] Figure 2 Schematic diagram of the proposed expert neural network structure and pre-training method.

[0042] Figure 3 Schematic diagram of the partial structure of the proposed network gate network.

[0043] Figure 4 The three-dimensional reconstruction results of the parcel phase predicted by this method under unseen circumstances are shown, as well as the comparison with several other methods. DETAILED DESCRIPTION

[0044] The present invention proposes an adaptive deep learning stripe analysis method based on a multi-expert collaboration mechanism, which achieves the goal of obtaining cross-system generalization by integrating the data domain and the network structure level design. Different from the traditional framework based on fixed parameter networks, it adopts the concept of adaptive expert hybrid, and determines the network composition strategy by analyzing the input data. Moreover, unlike the expert hybrid framework that directly switches between different networks, it solves the cross-system stripe analysis problem in deep learning stripe analysis by real-time synthesis in the parameter space to form an architecture with high scalability and flexibility. This method includes the following four steps:

[0045] Step 1: Construct a fringe projection dataset with different fringe frequencies and contrasts. The dataset is constructed by using a projector to project several sinusoidal phase-shifted fringe images with different frequencies and contrasts for each scene under test, and using a camera to shoot the scene under test illuminated by the fringe. The original fringe image is used as the input for formal network training. The expression of the projected fringe image can be written as:

[0046]

[0047] Where (x, y) is the pixel coordinate of the projector, index n = 0, 1, 2, ..., 11. Parameter A p , B p , f are the fringe amplitude, modulation and spatial frequency respectively. The frequency of the projected fringe can be changed by changing the value of f, and the contrast can be adjusted by changing B p The label of the data set is the numerator and denominator of the wrapped phase calculated by the twelve-step phase shift method. The calculation formula is as follows:

[0048]

[0049] Among them I n (x, y) is the nth phase-shifted fringe image, M(x, y) is the numerator, and D(x, y) is the denominator. After that, the data set is augmented by vertically flipping and cropping all training data, then rescaling them and adding the processed data to the training data set. The cropping operation is performed on a random area that occupies 75% of the image area, and then rescaled to the original size of the image through bilinear interpolation. Both operations are performed on the input fringe image and the label at the same time.

[0050] Step 2: Construct several expert neural networks for formal training. Based on the data sets of different stripe frequencies and contrasts constructed in step 1, select several groups of data that can cover different data domain requirements to pre-train each expert neural network. The structure of the expert neural network adopts the Res-Unet structure design, and the pre-training process adopts the self-supervised denoising autoencoder method. The specific training data and method are as follows: for each expert neural network, only a series of input stripe images at specific frequencies and contrasts are taken from the data set constructed in step 1, and a certain proportion of Gaussian noise is manually added to them as input. The original stripe image without noise is used as a label, and this is used as a data set to pre-train a specific expert neural network. The process of adding noise to the stripe image can be expressed as:

[0051] I σ (x,y)=I c (x,y)+σ

[0052] Among them I c(x,y) and I σ (x, y) are the normalized original raster image and the image after adding noise, respectively. σ is the image noise, which follows a normal distribution with a mean of 0 and a variance of σ (in this method, σ = 0.4).

[0053] Step 3: Construct a gate network for controlling the synthesis of the expert neural network, and then use the data set pre-constructed in step 1 and the expert neural network pre-trained in step 2 to train the expert neural network and the gate network together to construct an expert collaborative network for cross-system stripe analysis.

[0054] The gate network adopts the Vgg network architecture design, and the initial parameters are set by random initialization. The initial parameters of the expert neural network are the parameters when the pre-training phase is completed. In the collaborative training phase, the gate network first receives the fringe pattern as input, outputs adaptive synthesis weights, and then uses the weights output by the gate network to combine with the parameters of the pre-training network to form an expert collaborative network for prediction. The expert collaborative network has the same structure as the expert neural network, but all parameters are obtained by fusing the parameters of each expert neural network with the output weights of the gate network. The fusion process can be described by the following formula:

[0055]

[0056] Where P i are the parameters of each expert neural network, P MoE is the expert collaboration network parameter after dynamic synthesis, w i is the output weight of the gate network, and n is the number of expert networks. The sum of all expert weights is 1, that is, The expert collaboration network formed through this process receives the fringe image as input again, and outputs the corresponding wrapped phase numerator and denominator, calculates the loss with the corresponding label value, and then updates the parameters to complete the training process.

[0057] During the training phase, the loss function is expressed as follows:

[0058] L loss =λ1L MSE +λ2L Fourier

[0059] Where λ1 and λ2 are weights of different loss functions. MSE L is the spatial consistency loss function, which can make the prediction value of the expert collaboration network consistent with the label value in spatial intensity distribution. MSE The calculation method is

[0060]

[0061] where y predis the network prediction value, y true is the label value, and k is the total pixel value of the image. Fourier is the Fourier loss function, which constrains the network prediction result in the frequency domain to be consistent with the frequency domain value of the label value. Fourie The expression of r is:

[0062]

[0063] in is the discrete Fourier transform. Fourier loss guides the optimization of the network from another perspective by comparing the difference between the predicted value and the label value in the frequency domain. This loss strategy combining time domain and space domain is helpful to assist the network to utilize the physical laws of the problem being processed to improve its generalization and robustness.

[0064] The present invention achieves the goal of obtaining cross-system generalization by integrating the data domain and the network structure level design. Different from the traditional framework based on fixed parameter networks, it adopts the concept of adaptive expert hybrid, and determines the network composition strategy by analyzing the input data. Moreover, unlike the expert hybrid framework that directly switches between different networks, it solves the cross-system stripe analysis problem in deep learning-based stripe analysis by real-time synthesis in the parameter space to form a highly scalable and flexible architecture.

[0065] In order to compare the actual effect of the test method, the 12-step phase shift method as a benchmark, the baseline network built using the Unet structure, the expert collaborative network, and the traditional single-frame fringe analysis method Fourier profilometry (FTP) were used to analyze the fringe of the same scene, and the phase unwrapping and 3D reconstruction were performed using the time phase unwrapping and calibration data to obtain the 3D measurement results for comparison. The test data fringe frequency was selected to be 120, and the baseline network was trained at a fringe frequency of 96 to show the impact of cross-frequency on its performance. Figure 4 The results of 3D reconstruction by various methods are shown. It can be seen that even in completely unknown situations, the results of this method are comparable to those obtained by the benchmark twelve-step phase shift method. Excellent reconstruction results are obtained for both complex details and flat surfaces of the model. Compared with the results obtained by the traditional FTP method and the baseline network, this method has obvious advantages in reconstruction accuracy, demonstrating its excellent capabilities in cross-system 3D reconstruction tasks.

Claims

1. An adaptive deep learning fringe analysis method based on multi-expert collaboration mechanism, characterized in that: include: Step 1: Construct a fringe projection dataset with different fringe frequencies and contrasts; Step 2: Construct several expert neural networks, and select several groups of data that can cover different data domain requirements from the data sets of different fringe frequencies and contrasts constructed in step 1 to pre-train each expert neural network; Step 3: Construct a gate network for controlling the synthesis of the expert neural network, use the fringe projection dataset of step 1 to collaboratively train the pre-trained expert neural network and the gate network, use the parameters obtained from the collaborative training to construct an expert collaboration network for cross-system fringe analysis, and use the fringe projection dataset of step 1 to train the expert collaboration network; Step 4: Input the real-time fringe pattern into the trained expert collaboration network to obtain the wrapped phase.

2. The adaptive deep learning fringe analysis method based on multi-expert collaboration mechanism according to claim 1 is characterized in that: The specific method for constructing a fringe projection dataset with different fringe frequencies and contrasts is: For different test scenes, a projector is used to project several sinusoidal phase-shifted fringe images with different frequencies and contrasts, and a camera is used to shoot the test scene illuminated by the fringe. The projected fringe images are obtained as a data set and data enhancement operations are performed on the data set. The labels of the dataset are the numerator and denominator of the wrapped phase calculated by the twelve-step phase shift method.

3. The adaptive deep learning fringe analysis method based on multi-expert collaboration mechanism according to claim 1 is characterized in that: The projected fringe image is represented as: Where (x, y) is the pixel coordinate of the projector, index n = 0, 1, 2, ..., 11; A p , B p , f are the fringe amplitude, modulation degree and spatial frequency respectively.

4. The adaptive deep learning fringe analysis method based on multi-expert collaboration mechanism according to claim 1 is characterized in that: The structure of the expert neural network adopts the Res-Unet structure, and the pre-training process adopts the self-supervised denoising autoencoder method.

5. The adaptive deep learning fringe analysis method based on multi-expert collaboration mechanism according to claim 1 is characterized in that: The specific steps for pre-training the expert neural network are: For each expert neural network, a series of fringe patterns at set frequency and contrast are taken from the data set constructed in step 1, and a certain proportion of Gaussian noise is added to the fringe patterns as the input of the expert neural network. The original fringe patterns without noise are used as labels, and this is used as a data set for pre-training of the specific expert neural network.

6. The adaptive deep learning fringe analysis method based on multi-expert collaboration mechanism according to claim 5 is characterized in that: The specific formula for adding noise to the fringe image is: I σ (x,y)=I c (x,y)+σ Among them I c (x,y) and I σ (x, y) are the normalized original raster image and the image after adding noise, respectively, and σ is the image noise.

7. The adaptive deep learning fringe analysis method based on multi-expert collaboration mechanism according to claim 1 is characterized in that: The gate network adopts the Vgg network architecture.

8. The adaptive deep learning fringe analysis method based on multi-expert collaboration mechanism according to claim 1 is characterized in that: The fringe projection data set in step 1 is used to collaboratively train the pre-trained expert neural network and the gate network. The specific method for constructing an expert collaborative network for cross-system fringe analysis using the parameters obtained from the collaborative training is as follows: The stripe image in the data set is input into the gate network, and the gate network outputs the adaptive synthesis weight; The weights output by the gate network are fused with the parameters of the pre-trained network as the parameters of the expert collaboration network, and the structure of the expert collaboration network is the same as that of the expert neural network; The fringe pattern in the data set is input into the expert collaboration network to obtain the corresponding wrapped phase numerator and denominator, the loss function of the wrapped phase numerator and denominator output by the expert collaboration network and the label is calculated, and the parameters are updated to complete the training process.

9. The adaptive deep learning fringe analysis method based on multi-expert collaboration mechanism according to claim 8 is characterized in that: The parameters of the expert collaboration network are as follows: Where P i are the parameters of each expert neural network, P MoE is the expert collaboration network parameter after dynamic synthesis, w i Output weights for the gate network, n is the number of expert networks.

10. The adaptive deep learning fringe analysis method based on multi-expert collaboration mechanism according to claim 8, characterized in that: The loss function of the parcel phase numerator, denominator and label output by the expert collaboration network is as follows: L loss =λ1L MSE +λ2L Fourier Among them, λ1 and λ2 are the weights of different loss functions, L MSE is the spatial consistency loss function L MSE The calculation method is where y pred is the network prediction value, y true is the label value, k is the total pixel value of the image; L Fourier is the Fourier loss function, L Fourier The expression is: in, is the discrete Fourier transform.

Images