Near-infrared fringe image rapid denoising method based on lightweight network

Abstract

The invention provides a near-infrared fringe image rapid denoising method based on a lightweight network. The method comprises the following steps: constructing a lightweight convolutional neural network; the method comprises the following steps of: acquiring sample images with different surface shapes in a real scene by using a near-infrared structured light system, setting an optimal parameter for each image, and preparing a data set by using the images with the optimal parameters; inputting the data set into a lightweight denoising neural network for training; and inputting an infrared fringe image with noise, which is acquired in real time, into the trained lightweight denoising model to obtain a denoised optimized fringe image.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical measurement, and in particular relates to a fast denoising method for near-infrared stripe images based on a lightweight network. Background Art

[0002] Most of the existing fringe projection 3D measurement systems use visible light sources, but due to the particularity of the objects collected in certain environments, visible light sources are limited and have strong stimulation to the human eye. Near-infrared light sources can minimize the stimulation to the measured person / object and are not easily affected by ambient light. Therefore, infrared fringe projection is an ideal structured light illumination method. Infrared fringe projection 3D surface measurement uses infrared light sources with a wavelength greater than 780nm. Ordinary industrial CCD cameras have a wide range of light wave sensing. Although the sensitivity to infrared light response decreases with the increase of wavelength, they still have considerable sensitivity to the near-infrared band close to 780nm. During the imaging process of near-infrared fringe images, some unprocessed laser beams will cause uneven illumination of the detection area and generate a large amount of speckle noise in the detector image. The appearance of these inherent speckle noises will cause random distribution of image light intensity, resulting in granular distribution noise on the surface of the collected image, blurring the fine features of the image. Speckle noise can be reduced by using a large step phase shift method, but the efficiency will be reduced.

[0003] Generally, the speckle denoising method based on image processing is mainly filtering method, which is divided into spatial domain denoising method and transform domain denoising method. In recent years, there are also some denoising methods based on convolutional neural network. Although the spatial domain denoising method based on various filters can obtain good speckle denoising effect, it will also bring edge and texture blur, which will cause distortion of details. The denoising method based on Fourier transform is called Windowed Fourier transform (WFT), which is a transform domain denoising method. Appropriate threshold technology is applied to the WFT coefficients of the obtained speckle image to eliminate the spectral contribution of speckle noise. However, for different scenes, the threshold of this method needs to be determined empirically. Chao Ni et al. proposed an iterative method based on wavelet transform (literature "A novel method of infrared image denoising and edge enhancement", author Chao Ni et al.), introduced forward and backward diffusion technology in wavelet diffusion, and the effect of enhancing the edge of the image can be improved with the increase of the number of iterations. However, as an iterative method, 3-9 iterations are required to obtain the optimal denoising result, which consumes a lot of time.

[0004] The image denoising by sparse 3D transform-domain collaborative filtering (Block-matching and 3D filtering, BM3D) method is widely used (reference: "Image denoising by sparse 3D transform-domain collaborative filtering", author K Dabov et al.). In order to better preserve the structure and details of the image, BM3D first searches for similar blocks and filters them in the transform domain to obtain block evaluation values, and then weights each point in the image to obtain the final denoising result. However, the BM3D algorithm is easily affected by the sigma parameter, which needs to be adjusted according to the input source to control the denoising degree of different objects and environmental scenes. In addition, due to the complexity of the algorithm, the denoising time cost is high, which greatly affects the final three-dimensional measurement efficiency.

[0005] Therefore, for three-dimensional imaging technology based on infrared fringe projection, there is currently a lack of a fringe denoising method that has good denoising effect, high accuracy, and a speed that can meet the requirements of real-time three-dimensional measurement. Summary of the invention

[0006] The present invention proposes a fast denoising method for near-infrared streak images based on a lightweight network.

[0007] The technical solution to achieve the purpose of the present invention is: a fast denoising method for near-infrared streak images based on a lightweight network, comprising:

[0008] Step 1: Build a lightweight convolutional neural network;

[0009] Step 2: Use the near-infrared structured light system to collect sample images of different surface shapes in real scenes, set optimal parameters for each image, and prepare a data set using the images with the optimal parameters set;

[0010] Step 3: Input the data set into a lightweight denoising neural network for training;

[0011] Step 4: Input the infrared fringe image with noise collected in real time into the trained lightweight denoising model to obtain the denoised optimized fringe image.

[0012] Preferably, the lightweight denoising neural network includes a downsampling main path, four downsampling branches 1 to 4 and an upsampling main path, each downsampling branch includes a residual module and a downsampling module connected in sequence, the downsampling main path includes a convolutional layer, a residual module, a combination of three groups of downsampling modules and residual modules, and a combination of a downsampling module and a multi-head self-attention module connected in sequence, and the input end of the downsampling branches 1 to 4 is connected to the input end of the combination of the three groups of downsampling modules and residual modules, and the combination of a group of downsampling modules and a multi-head self-attention module in accordance with The upsampling main path includes four groups of upsampling modules and residual modules connected in sequence and a convolution layer. The output end of the downsampling branch 4 is connected to the input end of the first group of upsampling modules and residual modules. The output end of the downsampling branch 3 is connected to the input end of the second group of upsampling modules and residual modules. The output end of the downsampling branch 2 is connected to the input end of the third group of upsampling modules and residual modules. The output end of the downsampling branch 1 is connected to the input end of the fourth group of upsampling modules and residual modules.

[0013] Preferably, the specific method of collecting images of samples of different surface shapes in a real scene using a near-infrared structured light system is:

[0014] The grating image projected by the infrared projector is synchronously photographed by the camera, and the collected three-step phase-shifted grating light intensity image is expressed as:

[0015]

[0016]

[0017]

[0018] Among them, I' 1 (x,y),I' 2 (x,y),I' 3 (x, y) is the corresponding three-step phase-shifted grating intensity, (x, y) is the pixel coordinate of the camera plane, A(x, y) is the background intensity, B(x, y) is the modulation degree of the fringe, is the wrapped phase; the grating intensity diagram of different frequencies Different from , δ(x,y) is multiplicative noise.

[0019] Preferably, the BM3D method is used to denoise the collected noise image, and the noise image I' synchronously photographed by the camera is expressed as follows after BM3D processing:

[0020]

[0021] Among them, I g Represents the denoised image.

[0022] Preferably, the specific method of inputting the data set into the lightweight denoising model for training is:

[0023] The noisy image taken synchronously by the camera is used as input data, and the image after noise removal using the BM3D method is used as the standard value to input into the lightweight denoising neural network;

[0024] Calculate the difference between the standard value and the model output value;

[0025] Combined with the back-propagation method, the internal parameters of the lightweight denoising neural network are iteratively optimized until the loss function converges.

[0026] Preferably, L2loss is used as the loss function.

[0027] Compared with the traditional technology, the present invention has the following significant advantages: the present invention realizes fast and high-precision denoising of near-infrared stripe structured light images, and provides more accurate image information for the subsequent fast and real-time three-dimensional face reconstruction based on MEMS near-infrared stripe structured light; the present invention can be applied to the field of face detection that needs to avoid direct light source stimulation to the human eye, and can realize real-time three-dimensional face measurement in some scenes that need fast and high-precision face measurement, and because the cost of MEMS is relatively low, the cost can be greatly reduced, and the popularization of real-time three-dimensional face measurement can be realized. The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The figure is a flowchart of a fast denoising method for near-infrared streak images based on a lightweight network.

[0029] Figure 2 This is a physical picture of the hardware used to collect infrared stripe images in the present invention.

[0030] Figure 3 This is a structural diagram of the lightweight convolutional neural network based on deep learning of the present invention.

[0031] Figure 4 This is the loss value curve of the lightweight convolutional neural network after 200 rounds of training and verification.

[0032] Figure 5 The denoising effect diagram of a set of data in the test set using different denoising methods, (a) is the collected infrared fringe image with speckle noise, (b) is the denoising result using this method, (c) is the denoising result of the BM3D method with sigma set to 10, (d) is the result obtained by the Gaussian filtering method with the template size set to 7*7, (e) is the denoising result obtained by the guided filtering method with the label as the guide map, and (f) is the denoising result of the wavelet soft thresholding method.

[0033] Figure 6 This paper provides an overall flow chart of the fast denoising method for near-infrared streak images based on a lightweight network. A lightweight convolutional neural network is constructed to fully denoise near-infrared streak images of poor quality while ensuring speed.

[0034] Table 1 shows the speed and accuracy comparison results of the BM3D method and the present method. The average time of denoising the 100 images in the validation set by the BM3D method and the present method is compared, as well as the MAE between the denoising results of the present method and the BM3D method. DETAILED DESCRIPTION

[0035] like Figure 1 As shown in the figure, a fast denoising method for near-infrared streak images based on a lightweight network can perform real-time denoising of infrared streak images and obtain high-precision three-dimensional reconstruction results. The method includes the following four steps:

[0036] Step 1: Build a lightweight convolutional neural network for image denoising, such as Figure 1 shown.

[0037] The lightweight denoising neural network includes a downsampling main path, four downsampling branches 1-4 and an upsampling main path, each downsampling branch includes a residual module and a downsampling module connected in sequence, the downsampling main path includes a convolution layer, a residual module, a combination of three groups of downsampling modules and residual modules, and a combination of a downsampling module and a multi-head self-attention module connected in sequence, and the input ends of the downsampling branches 1-4 are connected in sequence to the input ends of the combination of the three groups of downsampling modules and residual modules and the combination of a group of downsampling modules and a multi-head self-attention module. A corresponding connection is made, the upsampling main path includes four groups of upsampling modules and residual modules connected in sequence and a convolution layer, the output end of the downsampling branch 4 is connected to the input end of the first group of upsampling modules and residual modules, the output end of the downsampling branch 3 is connected to the input end of the second group of upsampling modules and residual modules, the output end of the downsampling branch 2 is connected to the input end of the third group of upsampling modules and residual modules, and the output end of the downsampling branch 1 is connected to the input end of the fourth group of upsampling modules and residual modules.

[0038] The input noise image first passes through the convolution layer and the residual module, the image size remains unchanged, and the number of channels is expanded to 32. The input is respectively input into the combination of the downsampling branch 1 and the downsampling main path downsampling module and the residual module. In the downsampling branch 1, a residual module is first passed through, and the output does not change the image size and the number of channels. Then the output of the residual module is downsampled to obtain the branch output with half the image size. In the downsampling main path, downsampling is first performed to obtain the result of halving the image size, and then passes through the residual module. After that, the same steps are performed. After 4 downsamplings, the outputs of 4 branches are obtained, and the sizes are H / 2*W / 2, H / 4*W / 4, H / 8*W / 8, and H / 16*W / 16 respectively. After the fourth downsampling of the downsampling main path, the residual module is not used anymore, but the multi-head self-attention module (Multi-Head Self-Attention, MHSA layer) is introduced. Here, a four-head self-attention module is used. The output of the downsampled main path is added to the output of branch 4 and then upsampled, and input into the residual module to obtain an output of size H / 8*W / 8. It is added to the output of branch 3 and the previous steps are repeated to obtain an output of size H / 4*W / 4. The output of the main path is continuously added to the output of the branch and then upsampled to finally obtain a result of size H*W. Finally, it passes through the convolution layer and adjusts the number of channels to obtain an output of size H*W with a channel number of 1.

[0039] The entire lightweight network is mainly composed of five functional modules. Among them, the convolution layer (Conv Layer) and the pooling layer (Pooling Layer) are very commonly used modules in traditional convolutional neural networks. The convolution layer consists of multiple convolution kernels. The size of the convolution kernel is 3*3. Compared with the size of the convolution kernel of 5*5, the model using 3*3 filters has a smaller memory footprint and no significant difference in accuracy. The number of convolution kernels is the number of channels of the convolution layer. Each convolution kernel independently performs spatial convolution operations on the input data to generate output tensors. The role of the pooling layer is to compress the input tensor. On the one hand, it makes the input tensor smaller, simplifies the complexity of network calculations, and prevents overfitting; on the other hand, it compresses the input tensor and extracts the main features of the input tensor. Commonly used pooling layers are the average pooling layer (AveragePooling Layer) and the maximum pooling layer (MaxPoolingLayer). In this network, the maximum pooling layer is used to downsample the input tensor by 1 / 2. In addition, multiple residual blocks are used in the network to solve the problem of gradient disappearance in deep networks, prevent overfitting, and accelerate the convergence of the network's loss function. Each residual block contains three convolutional layers and three activation functions (ReLU). Since the computational complexity of the self-attention module is strongly related to the size of the tensor, it is necessary to add a four-head attention module after the last downsampling of the main path. There is no hyperparameter difference, and the training and inference overhead is minimized. The multi-head self-attention module can reduce model parameters while ensuring network stability, thereby minimizing latency. Due to the processing of the maximum pooling layer, the tensor size in the path is not consistent. The output of the branch with the same size as the main path after downsampling is added, and then upsampled. The upsampling block consists of a two-dimensional deconvolution and an activation function (ReLU). The convolution kernel size and step size of the deconvolution are set to 2, so that the tensor size becomes twice the original.

[0040] After the network model is built, the network input data is set to the unprocessed infrared stripe image collected by the camera, and the network output is set to the optimized stripe image after denoising by the BM3D method.

[0041] Step 2: Collect sample images of different surface shapes in real scenes based on the near-infrared structured light system, set the optimal parameters for each image, prepare the data set, and ensure the generalization of the network.

[0042] The near-infrared structured light system is mainly composed of a near-infrared projector and a camera. Its basic process is that the near-infrared projector projects an infrared stripe pattern, which is then collected synchronously by the camera. Based on the collected phase-shifted stripe pattern, the phase modulated by the object surface can be calculated. Based on the positional relationship between the projector and the camera, the corresponding three-dimensional coordinates of each point can be calculated from the phase to obtain the three-dimensional morphology of the object surface.

[0043] The grating image projected by the infrared projector is synchronously photographed by the camera, and the collected three-step phase-shifted grating light intensity image is expressed as:

[0044]

[0045]

[0046]

[0047] Among them, I' 1 (x,y),I' 2 (x,y),I' 3 (x, y) is the corresponding three-step phase-shifted grating intensity, (x, y) is the pixel coordinate of the camera plane, A(x, y) is the background intensity, B(x, y) is the modulation degree of the fringe, is the wrapped phase. The intensity diagram of the grating at different frequencies Different. δ(x,y) is multiplicative noise and affects image quality.

[0048] The BM3D method is used to denoise the collected noisy image. The BM3D algorithm can obtain a high signal-to-noise ratio image and can fully preserve the structure and details of the image. It is currently the best denoising algorithm among traditional algorithms. The BM3D algorithm mainly consists of three parts. First, similar block matching (Block-Matching) is performed on each reference block to obtain a 3D combination; then the 3D combination is collaboratively transformed and filtered (3D-Transform); and then the combined filtering results are aggregated (Aggregation), and the denoised results are obtained for each reference block. In order to obtain better image quality, BM3D performs two-step denoising: preliminary estimation and final estimation. The specific block matching criteria, filtering methods, and integration weights in the final estimation step are somewhat different from the preliminary estimation. The speckle image I' captured by the camera is denoised using the BM3D method. After BM3D processing, it can be expressed as:

[0049]

[0050] Among them, I g Represents the denoised image, which is used as the true value for deep learning convolutional neural network training.

[0051] A dataset was created using 1000 sets of infrared stripe images, of which 20% were used as validation sets, 10% were used as test sets, and the rest were used as training sets.

[0052] Step 3: Input the data set into a lightweight denoising neural network for training;

[0053] Taking I' as input data, I g As the standard value (groundtruth) is sent to the lightweight model. Using the mean square error as the loss function, the standard value I is calculated g Output value with model 1 The difference between them. Combined with the back propagation method, the internal parameters of model 1 are repeatedly optimized iteratively until the loss function converges, at which point the lightweight training ends. During the training, the L2loss with better results is used as the loss function after experiments. The Adam algorithm is used to find the minimum value of the loss function during the network training process. The network is trained for 200 rounds in total, and the loss of the final network training set and validation set is less than 0.01.

[0054] The trained and verified lightweight denoising convolutional neural network is used to test the test set data to evaluate the accuracy and speed of the network. The test data is input, and the lightweight denoising model obtained through training directly outputs the denoised optimized stripe image.

[0055] Step 4: Input the infrared fringe image with noise collected in real time into the trained lightweight denoising model to obtain the denoised optimized fringe image.

[0056] The present invention adopts a deep learning method to denoise infrared stripe images. Compared with the traditional filtering method and transform domain denoising method, it can more fully denoise the near-infrared stripe image while well preserving the image edge detail features. The final prediction result of the test set image is 0.00094 with the MAE of the BM3D method, and the speed is improved by about 60 times compared with the BM3D method.

[0057] Example

[0058] In order to verify the effectiveness of the method described in the present invention, a black and white camera (model acA640-750um, Basler), an infrared projector (model P8508, Micro Vision Sensor) and a computer were used to build a real-time 3D measurement device based on deep learning. Figure 2 As shown. When generating training data, the projector projects a three-step phase-shift fringe pattern onto the object and the camera collects it. The BM3D method is then used to make labels. A total of 1,000 sets of data are taken, of which 700 are used for training sets, 200 are used for validation sets, and 100 are used for test sets. The data of the training set, validation set, and test set are not reused and are shuffled during training. The network loss function is set to mean square error (MSE), the optimizer is Adam, and the network training cycle is set to 200 rounds. Figure 4 This is the loss value curve of the residual convolutional neural network after 200 rounds of training and verification. Figure 4The network stopped converging after 120 rounds, and the final loss value of the training set was about 0.008, and the final loss value of the validation set was about 0.006. A group of images in the test set were selected, and this method was used with the BM3D method, Gaussian filtering method, guided filtering method, and wavelet soft threshold method to denoise this group of images, and the results were compared. Figure 5 The denoising results of these methods are shown. Figure 5 (a) is the un-denoised infrared fringe image. Figure 5 (b) is the result of denoising using this method. Figure 5 (c) is the denoising result of the BM3D method with sigma set to 10. Figure 5 (d) is the result obtained by setting the template size of Gaussian filtering to 7*7. Figure 5 (e) is the denoising result obtained by the guided filtering method using the label as the guide map. Figure 5 (f) is the result of denoising using the wavelet soft threshold method. Figure 5 It shows that this method can fully denoise the object under test while retaining the edge and detail information of the image, thereby ensuring that the modulation information generated by the stripes is not affected, and optimizing the quality of the stripe image. Table 1 shows the denoising of 100 images in the test set using the BM3D method and the lightweight network. The denoising process of the two methods is timed, and the average denoising time of each image is calculated. The denoising time of the lightweight network is shortened by nearly 60 times compared with the BM3D method. The MAE of each image is calculated at the same time. The average value of the MAE of the lightweight network and the label BM3D is 0.00094. It can be seen that the lightweight network with the BM3D method as the label can effectively remove the noise of the infrared projector while improving the speed, and while ensuring the effective removal of the noise interference generated by the projector, it does not affect the modulation information in the stripe image.

[0059] Table 1

[0060] BM3D Lightweight Network Average denoising time per image 2.58s 0.043s MAE with label (BM3D) / 0.00094

Claims

1. A fast denoising method for near-infrared streak images based on a lightweight network, characterized in that: include: Step 1: Build a lightweight convolutional neural network; Step 2: Use the near-infrared structured light system to collect sample images of different surface shapes in real scenes, set optimal parameters for each image, and prepare a data set using the images with the optimal parameters set; Step 3: Input the data set into a lightweight denoising neural network for training; Step 4: Input the infrared fringe image with noise collected in real time into the trained lightweight denoising model to obtain the denoised optimized fringe image.

2. The fast denoising method for near-infrared streak images based on a lightweight network according to claim 1 is characterized in that: The lightweight denoising neural network includes a downsampling main path, four downsampling branches 1-4 and an upsampling main path, each downsampling branch includes a residual module and a downsampling module connected in sequence, the downsampling main path includes a convolution layer, a residual module, a combination of three groups of downsampling modules and residual modules, and a combination of a downsampling module and a multi-head self-attention module connected in sequence, and the input ends of the downsampling branches 1-4 are connected in sequence to the input ends of the combination of the three groups of downsampling modules and residual modules and the combination of a group of downsampling modules and a multi-head self-attention module. A corresponding connection is made, the upsampling main path includes four groups of upsampling modules and residual modules connected in sequence and a convolution layer, the output end of the downsampling branch 4 is connected to the input end of the first group of upsampling modules and residual modules, the output end of the downsampling branch 3 is connected to the input end of the second group of upsampling modules and residual modules, the output end of the downsampling branch 2 is connected to the input end of the third group of upsampling modules and residual modules, and the output end of the downsampling branch 1 is connected to the input end of the fourth group of upsampling modules and residual modules.

3. The fast denoising method for near-infrared streak images based on a lightweight network according to claim 2 is characterized in that: The multi-head self-attention module is a four-head self-attention module.

4. The fast denoising method for near-infrared streak images based on a lightweight network according to claim 1, characterized in that: The specific method of using the near-infrared structured light system to collect images of samples of different surface shapes in real scenes is: The grating image projected by the infrared projector is synchronously photographed by the camera, and the collected three-step phase-shifted grating light intensity image is expressed as: Among them, I'1(x,y), I'2(x,y), I'3(x,y) are the corresponding three-step phase-shifted grating light intensities, (x,y) is the pixel coordinate of the camera plane, A(x,y) is the background light intensity, B(x,y) is the modulation degree of the fringe, is the wrapped phase; the grating intensity diagram of different frequencies Different from , δ(x,y) is multiplicative noise.

5. The fast denoising method for near-infrared streak images based on a lightweight network according to claim 4 is characterized in that: The BM3D method is used to denoise the collected noisy image. The noisy image I' taken synchronously by the camera is expressed as follows after BM3D processing: Among them, I g Represents the denoised image.

6. The fast denoising method for near-infrared streak images based on a lightweight network according to claim 5, characterized in that: The specific method of inputting the data set into the lightweight denoising model for training is: The noisy image taken synchronously by the camera is used as input data, and the image after noise removal using the BM3D method is used as the standard value to input into the lightweight denoising neural network; Calculate the difference between the standard value and the model output value; Combined with the back-propagation method, the internal parameters of the lightweight denoising neural network are iteratively optimized until the loss function converges.

7. The fast denoising method for near-infrared streak images based on a lightweight network according to claim 6, characterized in that: L2loss is used as the loss function.

Images