An adaptive deep learning fringe analysis method based on multi-expert collaboration mechanism

Abstract

The application discloses a kind of self-adapting deep learning stripe analysis methods based on multi-expert cooperation mechanism, and the present application reaches the goal of obtaining cross-system generalization by fusing data domain and network structure level design.The present application adopts the concept of adaptive expert mixing, determines the composition strategy of network by analyzing input data.And different from the expert mixing framework that directly switches between different networks, it is composed of a highly expandable and flexible architecture by real-time synthesis in the parameter space.Once trained, it can automatically adapt to different input data and calculate the optimal weight of different experts, achieving adaptive fringe image analysis under different imaging conditions.In addition, the present application also uses the denoising autoencoder method to reduce resource consumption during the pre-training stage, and introduces data enhancement and Fourier loss method during the training stage to further enhance performance.

Description

Technical Field

[0001] This invention belongs to the field of optical measurement technology, specifically relating to an adaptive deep learning fringe analysis method based on a multi-expert collaboration mechanism. Background Technology

[0002] In recent years, deep learning-based fringe analysis methods have been widely applied and achieved effective results in related fields, with some even surpassing the performance of traditional methods. This new generation of fringe analysis framework is expected to break through the efficiency and accuracy limitations of traditional methods and become a core technology for next-generation optical measurement systems. Despite their effectiveness, current deep learning fringe analysis methods still face significant challenges. They typically perform well only under identical imaging conditions, and the performance of the neural network drops sharply when the shooting conditions of the test data change significantly. Furthermore, even with the same measurement system, after adjusting the system parameters and readjusting, the network model previously trained on the system cannot be directly applied to the adjusted system, despite the same task. This invention aims to achieve cross-system generalization by fusing data domain and network structure design. Unlike traditional frameworks based on fixed-parameter networks, it adopts an adaptive expert hybrid approach, determining the network composition strategy through analysis of the input data. Moreover, unlike expert hybrid frameworks that directly switch between different networks, it constructs a highly scalable and flexible architecture through real-time synthesis in the parameter space, thereby solving the cross-system fringe analysis problem in deep learning-based fringe analysis. Summary of the Invention

[0003] The purpose of this invention is to provide an adaptive deep learning stripe analysis method based on a multi-expert collaboration mechanism.

[0004] The technical solution for achieving the objective of this invention is: an adaptive deep learning stripe analysis method based on a multi-expert collaboration mechanism, comprising:

[0005] Step 1: Construct a stripe projection dataset with different stripe frequencies and contrasts;

[0006] Step 2: Construct several expert neural networks. Select several sets of data from the datasets with different stripe frequencies and contrasts constructed in Step 1 that can cover the needs of different data domains and pre-train each expert neural network.

[0007] Step 3: Construct a gate network for controlling the synthesis of expert neural networks. Use the fringe projection dataset from Step 1 to train the pre-trained expert neural network and gate network together. Use the parameters obtained from the co-training to construct an expert collaboration network for cross-system fringe analysis, and train the expert collaboration network using the fringe projection dataset from Step 1.

[0008] Step 4: Input the real-time stripe pattern into the trained expert collaboration network to obtain the wrap phase.

[0009] Preferably, the specific method for constructing a fringe projection dataset with different fringe frequencies and contrasts is as follows:

[0010] For different test scenarios, several sinusoidal phase-shifted fringe images with different frequencies and contrasts are projected using a projector. The test scenario illuminated by the fringe is captured by a camera. The projected fringe images are used as a dataset, and data augmentation operations are performed on the dataset.

[0011] The dataset is labeled with the numerator and denominator of the wrapped phase calculated by the twelve-step phase shift method.

[0012] Preferably, the projected stripe image is represented as follows:

[0013]

[0014] Where (x, y) are the pixel coordinates of the projector, with indices n = 0, 1, 2, ..., 11; A p B p f and f represent the fringe amplitude, modulation index, and spatial frequency, respectively.

[0015] Preferably, the expert neural network adopts the Res-Unet structure, and the pre-training process uses a self-supervised denoising autoencoder method.

[0016] Preferably, the specific steps for pre-training the expert neural network are as follows:

[0017] For each expert neural network, a series of stripe patterns with a set frequency and contrast are taken from the dataset constructed in step 1. A certain proportion of Gaussian noise is added to the stripe patterns and used as the input of the expert neural network. The original stripe patterns without noise are used as labels. This dataset is used to pre-train the specific expert neural network.

[0018] Preferably, the specific formula for adding noise to a striped image is as follows:

[0019] I σ (x,y)=I c (x,y)+σ

[0020] Among them I c (x,y) and I σ (x,y) represent the normalized original raster image and the image after adding noise, respectively, and σ represents the image noise.

[0021] Preferably, the gate network adopts a Vgg network architecture.

[0022] Preferably, the method for collaboratively training the pre-trained expert neural network and gate network using the fringe projection dataset from step 1, and constructing an expert collaborative network for cross-system fringe analysis using the parameters obtained from the collaborative training, is as follows:

[0023] The stripe pattern in the dataset is input into a gate network, and after feature extraction by the gate network, adaptive synthetic weights are output.

[0024] The weights output by the gate network are fused with the parameters of the pre-trained network and used as the parameters of the expert collaboration network. The structure of the expert collaboration network is the same as that of the expert neural network.

[0025] Input the stripe pattern from the dataset into the expert collaboration network to obtain the corresponding wrapped phase numerator and denominator. Calculate the loss function between the wrapped phase numerator and denominator output by the expert collaboration network and the label, and update the parameters to complete the training process.

[0026] Preferably, the parameters of the expert collaboration network are as follows:

[0027]

[0028] Where P i P represents the parameters of each expert neural network. MoE For the parameters of the dynamically synthesized expert collaboration network, w i Output weights for the gate network. n represents the number of expert networks.

[0029] Preferably, the loss function for the packaged phase numerator and denominator output by the expert collaboration network is as follows:

[0030] L loss =λ1L MSE +λ2L Fourier

[0031] Where λ1 and λ2 are the weights of different loss functions, L MSE The spatial consistency loss function L MSE The calculation method is as follows

[0032]

[0033] Where y pred y is the network prediction value. true Here, k represents the label value, and k represents the total number of pixels in the image.

[0034] L Fourier Let L be the Fourier loss function. Fourier The expression is:

[0035]

[0036] in, This is the Discrete Fourier Transform.

[0037] Compared with the prior art, the significant advantages of this invention are: this invention adopts the concept of adaptive expert hybrid, determines the network construction strategy by analyzing the input data, and constructs an architecture with high scalability and flexibility by real-time synthesis in the parameter space;

[0038] This invention uses a denoising autoencoder method to reduce resource consumption during the pre-training stage, and introduces data augmentation and Fourier loss methods during the training stage to further enhance performance.

[0039] This invention outperforms traditional expert neural networks designed for single-system data domains and large-parameter networks designed for multi-system data domains in both seen and unseen scenarios. It also has a significant relative advantage when facing more difficult situations and can maintain excellent stability and robustness in different systems where parameters such as camera settings, stripe characteristics, and environmental factors are used together. Attached Figure Description

[0040] Figure 1 This is a flowchart of an adaptive deep learning stripe analysis method based on a multi-expert collaboration mechanism.

[0041] Figure 2 This is a schematic diagram of the proposed expert neural network structure and pre-training method.

[0042] Figure 3 This is a schematic diagram of the proposed network gate network structure.

[0043] Figure 4 This paper presents the 3D reconstruction results of the package phase predicted by our method in the case of no prior sighting, and compares them with several other methods. Detailed Implementation

[0044] This invention proposes an adaptive deep learning fringe analysis method based on a multi-expert collaboration mechanism. It achieves cross-system generalization by fusing data domain and network structure design. Unlike traditional frameworks based on fixed-parameter networks, it adopts an adaptive expert hybrid approach, determining the network construction strategy through input data analysis. Furthermore, unlike expert hybrid frameworks that directly switch between different networks, it constructs a highly scalable and flexible architecture through real-time synthesis in the parameter space, thereby solving the cross-system fringe analysis problem in deep learning-based fringe analysis. This method includes the following four steps:

[0045] Step 1: Construct a fringe projection dataset with different fringe frequencies and contrasts. The dataset construction process involves projecting several sinusoidal phase-shifted fringe images with different frequencies and contrasts onto each test scene using a projector. A camera is used to capture images of the test scene illuminated by the stripes, and the resulting original fringe images are used as input for the network's formal training. The expression for the projected fringe image can be written as:

[0046]

[0047] Where (x, y) are the pixel coordinates of the projector, with indices n = 0, 1, 2, ..., 11. Parameter A p B p f and f represent the fringe amplitude, modulation index, and spatial frequency, respectively. Changing the value of f alters the frequency of the projected fringes, while contrast is achieved by changing the value of B. p The values ​​are set accordingly. The labels of the dataset are the numerator and denominator of the wrapped phase calculated by the twelve-step phase-shifting method, as shown in the following formula:

[0048]

[0049] Among them I n (x,y) represents the nth phase-shifted fringe image, where M(x,y) is the numerator and D(x,y) is the denominator. Following this, data augmentation is performed on the dataset. Specifically, all training data are vertically flipped and cropped once, then rescaled, and the processed data is added back to the training dataset. The cropping operation is performed on random regions occupying 75% of the image area, followed by rescaling to the original image size using bilinear interpolation. Both operations are performed simultaneously on the input fringe pattern and the labels.

[0050] Step 2: Construct several expert neural networks for formal training. Based on the datasets with different stripe frequencies and contrasts constructed in Step 1, select several sets of data that can cover different data domain requirements to pre-train each expert neural network. The expert neural network adopts the Res-Unet architecture, and the pre-training process uses a self-supervised denoising autoencoder method. Specifically, for each expert neural network, from the dataset constructed in Step 1, only a series of input stripe images at specific frequencies and contrasts are taken. A certain proportion of Gaussian noise is manually added to these images before they are used as input. The original stripe images without noise are used as labels. This dataset is used to pre-train the specific expert neural network. The process of adding noise to the stripe image can be described as follows:

[0051] I σ (x,y)=I c (x,y)+σ

[0052] Among them I c(x,y) and I σ (x, y) represent the normalized original raster image and the image after adding noise, respectively. σ represents the image noise, which follows a normal distribution with a mean of 0 and a variance of σ (in this method, σ = 0.4).

[0053] Step 3: Construct a gate network to control the synthesis of expert neural networks. Then, using the dataset pre-constructed in Step 1 and the expert neural network pre-trained in Step 2, train the expert neural network and the gate network together to construct an expert collaboration network for cross-system stripe analysis.

[0054] The gate network adopts a Vgg network architecture, with initial parameters set randomly. The expert neural network, on the other hand, uses the parameters obtained during the pre-training phase. In the co-training phase, the gate network first receives the stripe pattern as input and outputs adaptive synthetic weights. Then, the weights output by the gate network are combined with the parameters of the pre-trained network to form an expert collaborative network for prediction. The expert collaborative network has the same structure as the expert neural network, but all parameters are obtained by fusing the parameters of each expert neural network with the output weights of the gate network. The fusion process can be expressed by the following formula:

[0055]

[0056] Where P i P represents the parameters of each expert neural network. MoE For the parameters of the dynamically synthesized expert collaboration network, w i The output weights of the expert network are n, where n is the number of expert network members. The sum of all expert weights is 1. The expert collaborative network formed through this process receives the stripe pattern as input again and outputs the corresponding wrapped phase numerator and denominator. After calculating the loss with the corresponding label value, the parameters are updated, thus completing the training process.

[0057] During the training phase, the loss function is expressed as follows:

[0058] L loss =λ1L MSE +λ2L Fourier

[0059] Where λ1 and λ2 are the weights for different loss functions. L MSE L is the spatial consistency loss function, which ensures that the predicted values ​​and label values ​​of the expert collaborative network maintain consistency in spatial intensity distribution. MSE The calculation method is as follows

[0060]

[0061] Where y predy is the network prediction value. true L is the label value, and k is the total number of pixels in the image. Fourier L is the Fourier loss function, which constrains the network prediction results to be consistent with the frequency domain values ​​of the labels in the frequency domain. Fourie The expression for r is:

[0062]

[0063] in This is the Discrete Fourier Transform. Fourier loss guides network optimization from another perspective by comparing the difference between the predicted value and the label value in the frequency domain. This loss strategy, which combines the time and spatial domains, helps the network utilize the physical laws of the problem it is dealing with to improve its generalization and robustness.

[0064] This invention achieves cross-system generalization by integrating data domain and network structure design. Unlike traditional frameworks based on fixed-parameter networks, it adopts an adaptive expert hybrid approach, determining the network composition strategy through analysis of input data. Furthermore, unlike expert hybrid frameworks that directly switch between different networks, it constructs a highly scalable and flexible architecture through real-time synthesis in the parameter space, thereby solving the cross-system fringe analysis problem in deep learning-based fringe analysis.

[0065] To compare the actual performance of the testing methods, fringe analysis of the same scene was performed using the 12-step phase-shifting method as a benchmark, a baseline network built using the Unet architecture, an expert collaborative network, and the traditional single-frame fringe analysis method Fourier profilometry (FTP). Temporal phase unwrapping and 3D reconstruction were performed using time-phase unfolding and calibration data to obtain 3D measurement results for comparison. The selected test data had a fringe frequency of 120, while the baseline network was trained at a fringe frequency of 96 to demonstrate the impact of cross-frequency variations on its performance. Figure 4 The results of 3D reconstruction using various methods are presented. It can be seen that even in the case of complete unknowns, the results of this method are comparable to those obtained by the benchmark twelve-step phase-shifting method. Excellent reconstruction results are obtained for complex details and smooth surfaces of the model. Compared with the results obtained by the traditional FTP method and the baseline network, this method has a significant advantage in reconstruction accuracy, demonstrating its excellent ability in cross-system 3D reconstruction tasks.

Claims

1. An adaptive deep learning fringe analysis method based on a multi-expert collaboration mechanism, characterized in that, The application relates to a method for constructing a cross-system fringe analysis expert collaboration network. Step 1: constructing a fringe projection dataset with different fringe frequencies and contrasts; Step 2: constructing a plurality of expert neural networks, selecting a plurality of groups of data capable of covering different data domain requirements from the different fringe frequency and contrast datasets constructed in step 1, and pre-training each expert neural network, the specific steps being as follows: For each expert neural network, a series of fringe images at a set frequency and contrast are taken from the dataset constructed in step 1, and a certain proportion of Gaussian noise is added to the fringe images to serve as the input of the expert neural network, and the original fringe images without noise are taken as labels, so as to pre-train the specific expert neural network with the dataset; Step 3: constructing a gate network for controlling the synthesis of the expert neural network, training the pre-trained expert neural network and the gate network cooperatively by using the fringe projection dataset in step 1, using the parameters obtained through the cooperative training to construct an expert collaboration network for cross-system fringe analysis, and training the expert collaboration network by using the fringe projection dataset in step 1, the specific method being as follows: inputting the fringe images in the dataset into the gate network, and outputting adaptive synthesis weights from the gate network; using the weights output from the gate network and the parameters of the pre-trained network to fuse and serve as the parameters of the expert collaboration network, the structure of the expert collaboration network being the same as that of the expert neural network; inputting the fringe images in the dataset into the expert collaboration network to obtain the corresponding wrapped phase numerator and denominator, calculating the loss function of the wrapped phase numerator and denominator output from the expert collaboration network and the label, and updating the parameters to complete the training process; Step 4: inputting the real-time obtained fringe images into the trained expert collaboration network to obtain the wrapped phase.

2. The adaptive deep learning striation analysis method based on multi-expert collaboration mechanism according to claim 1, characterized in that, The specific method for constructing the fringe projection dataset with different fringe frequencies and contrasts is as follows: for different scenes to be measured, a projector is used to project a plurality of sinusoidal phase shift fringe images with different frequencies and contrasts, a camera is used to shoot the scene to be measured illuminated by the fringe, and the projected fringe images are obtained as a dataset and subjected to a data enhancement operation; wherein the label of the dataset is the numerator and denominator of the wrapped phase obtained through a twelve-step phase shift method. 3.The self-adaptive deep learning striation analysis method based on multi-expert collaboration mechanism according to claim 1, characterized in that, The projected fringe image is represented as: where (x, y) are the pixel coordinates of the projector, index n = 0, 1, 2,..., 11; A p , B p , f are the fringe amplitude, modulation, and spatial frequency, respectively.

4. The adaptive deep learning striation analysis method based on multi-expert collaboration mechanism according to claim 1, characterized in that, The structure of the expert neural network adopts a Res-Unet structure, and the pre-training process adopts a self-supervised denoising autoencoder method.

5. The adaptive deep learning striation analysis method based on multi-expert collaboration mechanism according to claim 1, characterized in that, The specific formula for adding noise to the fringe image is as follows: I σ (x,y) = I c (x,y) + σ where I c (x,y) and I σ (x,y) are the normalized original raster image and the image after adding noise, respectively, and σ is the image noise.

6. The adaptive deep learning striation analysis method based on multi-expert collaboration mechanism according to claim 1, characterized in that, The gate network adopts a Vgg network architecture.

7. The adaptive deep learning striation analysis method based on multi-expert collaboration mechanism according to claim 1, characterized in that, The parameters of the expert collaboration network are as follows: where P i is the parameter of each expert neural network, R MoE is the parameter of the dynamically synthesized expert collaboration network, w i is the gate network output weight, n is the number of expert networks.

8. The adaptive deep learning striation analysis method based on multi-expert collaboration mechanism according to claim 1, characterized in that, The loss function of the wrapped phase numerator and denominator output from the expert collaboration network and the label is as follows: L loss = λ1L MSE + λ2L Fourier where λ1and λ2are the weights of different loss functions, L MSE the spatial consistency loss function L MSE is calculated in the following way where y pred is the network predicted value, y true is the label value, and k is the total pixel value of the image. L Fourier The Fourier loss function, L Fourier is given by the expression: wherein is a discrete Fourier transform.

Images