Lightweight network-based near-infrared stripe image fast denoising method

Abstract

The present application provides a kind of near-infrared stripe image fast denoising method based on light weight network, comprising: constructing light weight convolutional neural network;Different surface shape sample images are collected in real scene using near-infrared structured light system, set the optimal parameters for each image, and prepare dataset using the image with optimal parameters;The dataset is input into the light weight denoising neural network for training;The infrared stripe image with noise collected in real time is input into the trained light weight denoising model, and the optimized stripe image after denoising is obtained.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of optical measurement, and particularly relates to a near-infrared fringe image fast denoising method based on a light-weight network. BACKGROUND

[0002] Most of the existing fringe projection three-dimensional measurement systems adopt visible light sources, but the visible light sources are limited and have strong stimulation to the human eyes due to the particularity of the collected objects in some environments. The near-infrared light source can minimize the stimulation to the measured person / object and is not easily affected by the environmental light, so the infrared fringe projection is an ideal structured light illumination mode. The infrared fringe projection three-dimensional surface measurement uses an infrared light source with a wavelength greater than 780 nm. The ordinary industrial CCD camera has a wide light wave sensing range, although the sensitivity of the response to the infrared light decreases with the increase of the wavelength, but it still has considerable sensitivity to the near-infrared band close to 780 nm. In the imaging process of the near-infrared fringe image, some untreated laser beams can cause uneven illumination of the detection area and generate a large amount of speckle noise in the detector image. The appearance of these inherent speckle noises can cause random distribution of the image light intensity, resulting in granular distribution noise on the collected image surface and blurring the fine features of the image. The speckle noise can be reduced by using a large-step phase shift method, but the efficiency is reduced.

[0003] Generally, the speckle denoising methods based on image processing are mainly filtering methods, which are divided into spatial domain denoising methods and transform domain denoising methods, and in recent years, there are also some denoising methods based on convolutional neural networks. Although the spatial domain denoising method based on various filters can obtain good speckle denoising effect, it will also cause edge and texture blurring, which will lead to distortion in details. The denoising method based on Fourier transform, called Windowed Fourier transform (WFT), is a transform domain denoising method. A proper threshold technique is applied to the WFT coefficients of the obtained speckle image in order to eliminate the spectral contribution of the speckle noise. However, for different scenes, the threshold of this method needs to be determined empirically. Chao Ni et al. proposed an iterative method based on wavelet transform (literature “A novel method of infrared image denoising and edge enhancement”, authors Chao Ni et al.), which introduced forward and backward diffusion techniques in wavelet diffusion, and the effect of enhancing the image edge can be improved with the increase of the iteration number. However, as an iterative method, 3-9 iterations are needed to obtain the optimal denoising result, which greatly consumes time.

[0004] The image denoising method of Block-matching and 3D filtering (BM3D) is widely used (in the document "Image denoising by sparse 3D transform-domain collaborative filtering", by K Dabov et al.). In order to better preserve the structure and details of the image, BM3D first searches for similar blocks in the transform domain for filtering, thereby obtaining block evaluation values, and then obtains the final denoising result after weighting each point in the image. However, the BM3D algorithm is susceptible to the sigma parameter, which needs to be adjusted according to the input source to control the denoising degree of different objects and environmental scenes. In addition, due to the complexity of the algorithm, the denoising time cost is high, which greatly affects the final three-dimensional measurement efficiency.

[0005] Therefore, for the three-dimensional imaging technology based on infrared fringe projection, there is currently a lack of a fringe denoising method that has good denoising effect, high precision, and speed that can meet real-time three-dimensional measurement. SUMMARY

[0006] The present application provides a fast denoising method for near-infrared fringe images based on a lightweight network.

[0007] The technical solution for achieving the purpose of the present application is as follows: a fast denoising method for near-infrared fringe images based on a lightweight network, comprising:

[0008] Step 1: Construct a lightweight convolutional neural network;

[0009] Step 2: Use the near-infrared structured light system to collect different surface shape sample images in a real scene, set the optimal parameters for each image, and use the images with the optimal parameters to prepare a data set;

[0010] Step 3: Input the data set into the lightweight denoising neural network for training;

[0011] Step 4: Input the real-time collected infrared fringe image with noise into the trained lightweight denoising model to obtain the denoised optimized fringe image.

[0012] Preferably, the lightweight denoising neural network comprises one down-sampling main path, four down-sampling branches 1-4, and one up-sampling main path, each down-sampling branch comprises a residual module and a down-sampling module connected in sequence, the down-sampling main path comprises a convolutional layer, a residual module, a combination of three groups of down-sampling modules and residual modules, and a combination of one group of down-sampling modules and a multi-head self-attention module connected in sequence, the input ends of the down-sampling branches 1-4 and the input ends of the combination of three groups of down-sampling modules and residual modules and the combination of one group of down-sampling modules and a multi-head self-attention module are connected in sequence one by one, the up-sampling main path comprises a combination of four groups of up-sampling modules and residual modules and a convolutional layer connected in sequence, the output end of the down-sampling branch 4 is connected to the input end of the first group of up-sampling modules and residual modules, the output end of the down-sampling branch 3 is connected to the input end of the second group of up-sampling modules and residual modules, the output end of the down-sampling branch 2 is connected to the input end of the third group of up-sampling modules and residual modules, and the output end of the down-sampling branch 1 is connected to the input end of the fourth group of up-sampling modules and residual modules.

[0013] Preferably, the specific method for collecting different surface shape sample images in a real scene by using a near-infrared structured light system is as follows:

[0014] The three-step phase shift grating light intensity images collected by synchronously photographing the grating image projected by the infrared projector by the camera are represented as:

[0015]

[0016]

[0017]

[0018] wherein I'1(x,y), I'2(x,y), I'3(x,y) are the corresponding three-step phase shift grating light intensities, (x,y) is the pixel coordinate of the camera plane, A(x,y) is the background light intensity, B(x,y) is the fringe modulation, is the wrapped phase; the of grating light intensity images of different frequencies are different, and δ(x,y) is multiplicative noise.

[0019] Preferably, the BM3D method is used to denoise the collected noise image, and the noise image I' synchronously photographed by the camera after BM3D processing is represented as:

[0020]

[0021] wherein I g represents the denoised image.

[0022] Preferably, the specific method of inputting the data set into the lightweight denoising model for training is as follows:

[0023] The noise image synchronously captured by the camera is taken as the input data, and the image after noise is taken as the standard value input into the lightweight denoising neural network by using the BM3D method.

[0024] The difference between the standard value and the output value of the model is calculated.

[0025] The internal parameters of the lightweight denoising neural network are repeatedly iterated and optimized in combination with the back propagation method until the loss function converges.

[0026] Preferably, the L2loss is taken as the loss function.

[0027] Compared with the prior art, the present application has the following advantages: the present application realizes fast and high-precision denoising of the near-infrared stripe structured light image, provides more accurate image information for subsequent fast and real-time three-dimensional face reconstruction based on the near-infrared stripe structured light of MEMS, can be applied to the face detection field which needs to avoid direct observation of the light source to stimulate the human eye, realizes real-time three-dimensional face measurement in some scenes which need to perform fast and high-precision face measurement, and greatly compresses the cost due to the low cost of MEMS, and realizes the popularization of real-time three-dimensional face measurement. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 It is a flowchart of the near-infrared stripe image fast denoising method based on a lightweight network.

[0029] Figure 2 It is a hardware actual object diagram of the present application for collecting infrared stripe images.

[0030] Figure 3 It is a structure diagram of the lightweight convolutional neural network based on deep learning of the present application.

[0031] Figure 4 It is a loss value curve of the lightweight convolutional neural network after 200 rounds of training and verification.

[0032] Figure 5 It is a denoising effect diagram of a group of data in the test set using different denoising methods, (a) is an infrared stripe image with speckle noise collected, (b) is the denoising result using the present method, (c) is the denoising result of the BM3D method with sigma set to 10, (d) is the result obtained by setting the template size of the Gaussian filter method to 7*7, (e) is the denoising result obtained by taking the label as the guide image by the guide filter method, and (f) is the denoising result of the wavelet soft threshold method.

[0033] Figure 6The application discloses a light-weight network-based near-infrared stripe image fast denoising method, and relates to the technical field of image processing.

[0034] Table 1 is a comparison result of speed and accuracy of the BM3D method and the method. The average time of denoising 100 pictures of the verification set by the BM3D method and the application and the MAE between the denoising results of the application and the BM3D method are compared. DETAILED DESCRIPTION

[0035] As shown in Figure 1 , the application discloses a light-weight network-based near-infrared stripe image fast denoising method, which can realize real-time infrared stripe image denoising and obtain high-precision three-dimensional reconstruction results.

[0036] Step one: a light-weight convolutional neural network is built to realize image denoising, as shown in Figure 1 .

[0037] The light-weight denoising neural network comprises one down-sampling main path, four down-sampling branches 1-4 and one up-sampling main path, each down-sampling branch comprises a residual module and a down-sampling module connected in sequence, the down-sampling main path comprises a convolutional layer, a residual module, a combination of three groups of down-sampling modules and residual modules and a combination of one group of down-sampling modules and a multi-head self-attention module connected in sequence, the input ends of the down-sampling branches 1-4 are connected in one-to-one correspondence with the input ends of the combination of three groups of down-sampling modules and residual modules, the combination of one group of down-sampling modules and a multi-head self-attention module, the up-sampling main path comprises a combination of four groups of up-sampling modules and residual modules and a convolutional layer connected in sequence, the output end of the down-sampling branch 4 is connected with the input end of the first group of up-sampling modules and residual modules, the output end of the down-sampling branch 3 is connected with the input end of the second group of up-sampling modules and residual modules, the output end of the down-sampling branch 2 is connected with the input end of the third group of up-sampling modules and residual modules, and the output end of the down-sampling branch 1 is connected with the input end of the fourth group of up-sampling modules and residual modules.

[0038] The input noise map first passes through a convolutional layer and a residual module, the image size remains unchanged, and the number of channels is expanded to 32. The combination of the down-sampling branch 1 and the down-sampling main path down-sampling module and the residual module are input respectively. In the down-sampling branch 1, first pass through a residual module, the output does not change the image size and the number of channels, and then down-sample the output of the residual module to obtain the branch output with the image size halved. In the down-sampling main path, down-sampling is performed first to obtain a result with the image size halved, and then a residual module is passed through. Thereafter, the same steps are performed, and after 4 times of down-sampling, the outputs of 4 branches are obtained, with sizes of H / 2*W / 2, H / 4*W / 4, H / 8*W / 8, and H / 16*W / 16 respectively. After the fourth down-sampling of the down-sampling main path, the residual module is not continued to be used, but a multi-head self-attention module (Multi-Head Self-Attention, MHSA layer) is introduced, and four-head self-attention modules are used here. The output after down-sampling of the down-sampling main path is added to the output of the branch 4 after up-sampling, and is input into a residual module to obtain an output with a size of H / 8*W / 8. The output is added to the output of the branch 3, and the previous steps are repeated to obtain an output with a size of H / 4*W / 4. The output of the main path is continuously added to the output of the branch, and then up-sampling is performed to finally obtain an output with a size of H*W. Finally, a convolutional layer is passed through to adjust the number of channels to obtain an output with a size of H*W and a number of channels of 1.

[0039] The entire lightweight network is mainly composed of five functional modules. Among them, the convolution layer (Conv Layer) and the pooling layer (Pooling Layer) are very common modules in traditional convolutional neural networks. The convolution layer is composed of multiple convolution kernels, and the size of the convolution kernel is 3*3. Compared with the 5*5 convolution kernel size, the model using the 3*3 filter has smaller memory occupation and no significant difference in accuracy. The number of convolution kernels is the number of channels of the convolution layer, and each convolution kernel independently performs spatial convolution operation on the input data to generate an output tensor. The role of the pooling layer is to compress the input tensor, on the one hand to make the input tensor smaller, simplify the network calculation complexity and prevent overfitting; on the other hand to compress the input tensor and extract the main features of the input tensor. Common pooling layers include average pooling layer (AveragePooling Layer) and maximum pooling layer (MaxPooling Layer). In this network, the maximum pooling layer is used to downsample the input tensor by 1 / 2. In addition, multiple residual blocks (Residual Block) are used in the network to solve the problem of gradient disappearance in deep network, prevent overfitting and accelerate the convergence of the network loss function. Each residual block contains three convolution layers and three activation functions (ReLU). Since the computational complexity of the self-attention module is strongly related to the size of the tensor, a four-head attention module is added after the last down-sampling of the main path, with no hyperparameter difference, the training and inference overhead is minimized. The multi-head self-attention module can reduce model parameters while ensuring network stability, thereby minimizing the delay. Due to the processing of the maximum pooling layer, the tensor size in the path is not consistent. The output of the branch after down-sampling is added to the output of the main path with the same size, and then it is up-sampled. The up-sampling block is composed of two-dimensional deconvolution and activation function (ReLU). The convolution kernel size and step of the deconvolution are both set to 2, so that the tensor size becomes twice the original size.

[0040] After the model of the network is built, the input data of the network is set to be the unprocessed infrared fringe image collected by the camera, and the output of the network is set to be the optimized fringe image denoised by the BM3D method.

[0041] Step two: According to the collection of different surface shape sample images by the near-infrared structured light system in the real scene, set the optimal parameters for each image, prepare the data set, and ensure the generalization of the network.

[0042] The near-infrared structured light system is mainly composed of a near-infrared projector and a camera. The basic process is that the near-infrared projector projects an infrared fringe pattern, and the camera synchronously collects it. According to the collected phase shift fringe pattern, the phase modulated by the object surface can be calculated, and according to the positional relationship between the projector and the camera, the three-dimensional coordinates corresponding to each point can be calculated from the phase of each point to obtain the three-dimensional topography of the object surface.

[0043] The three-step phase-shifted grating light intensity graphs collected by synchronously capturing the grating graph projected by the infrared projector are represented as:

[0044]

[0045]

[0046]

[0047] where I'1(x, y), I'2(x, y), I'3(x, y) are the corresponding three-step phase-shifted grating light intensity, (x, y) is the pixel coordinate of the camera plane, A(x, y) is the background light intensity, B(x, y) is the modulation of the fringe, is the wrapped phase. The grating light intensity graphs of different frequencies are different. δ(x, y) is the multiplicative noise, which affects the image quality.

[0048] The BM3D method is used to denoise the collected noise image. The BM3D algorithm can obtain a higher signal-to-noise ratio image and can fully preserve the structure and details of the image. In the traditional algorithm, it is currently the best denoising algorithm. The BM3D algorithm mainly has three parts. First, the similar block matching (Block-Matching) is performed on each reference block to obtain a 3D combination; then the 3D combination is transformed and filtered (3D-Transform); and finally the filtered results of the combination are integrated (Aggregation), and the corresponding reference block is obtained. The result of the denoised result. In order to obtain better image quality, BM3D performs two steps of denoising: preliminary estimation and final estimation. In the final estimation step, the specific block matching standard, filtering method, and integration weight are different from the preliminary estimation. The speckle image I' captured by the camera is denoised using the BM3D method, and after BM3D processing, it can be represented as:

[0049]

[0050] where I g represents the denoised image, which is used as the true value for training the deep learning convolutional neural network.

[0051] 1000 groups of infrared fringe images collected are used to make a data set, of which 20% is used for verification set, 10% is used for test set, and the rest is used as training set.

[0052] Step three: input the data set into the lightweight denoising neural network for training;

[0053] I' is used as input data, and I gThe standard value (ground truth) is input into the lightweight model. The mean square error is used as the loss function to calculate the difference between the standard value I g and the output value of model 1. The internal parameters of model 1 are repeatedly iterated and optimized by combining the back propagation method until the loss function converges, at which time the lightweight training ends. In the training, the L2loss with better effect is used as the loss function through experiments, and the Adam algorithm is used to find the minimum value of the loss function in the network training process. The network is trained for a total of 200 rounds, and the loss of the final network training set and validation set is less than 0.01.

[0054] The trained and verified lightweight denoising convolutional neural network is used to test the test set data to evaluate the accuracy and speed of the network. The test data is input into the lightweight denoising model obtained by training, and the denoised optimized stripe image is directly output.

[0055] Step four: inputting the real-time collected infrared stripe image with noise into the trained lightweight denoising model to obtain the denoised optimized stripe image.

[0056] The method of the present application adopts the deep learning method for infrared stripe image denoising. Compared with the traditional filtering method and the transform domain denoising method, the method can sufficiently denoise the near-infrared stripe image under the premise of well preserving the image edge detail features. The prediction result of the test set image is 0.00094, which is the MAE of the BM3D method, and the speed is improved by about 60 times compared with the BM3D method.

[0057] Embodiment

[0058] In order to verify the effectiveness of the method, a black and white camera (model acA640-750um, Basler), an infrared projector (model P8508, Microview Sensing) and a computer are used to build an infrared stripe projection real-time three-dimensional measurement device based on deep learning, as shown in Figure 2 When generating training data, the projector projects three-step phase shift stripe images to the object, and the camera collects them, and then the BM3D method is used to make labels. A total of 1000 groups of data are collected, of which 700 groups are used for training set, 200 groups are used for validation set, and 100 groups are used for test set. The data of the training set, the validation set and the test set are not repeated, and are shuffled during training. The loss function of the network is set as mean square error (MSE), the optimizer is Adam, and the training period of the network is set as 200 rounds. Figure 4 The loss value curve of the residual convolutional neural network after 200 rounds of training and verification. Figure 4The display network stops converging after 120 rounds, and the loss value of the final training set is about 0.008, and the loss value of the final validation set is about 0.006. Select a group of pictures in the test set, respectively, using the method and BM3D method, Gaussian filtering method, guided filtering method, wavelet soft threshold method to denoise this group of images, and compare the results, Figure 5 The denoising results of these methods are shown. Figure 5 (a) is the undenoised infrared fringe image, Figure 5 (b) is the result of denoising using the method, Figure 5 (c) is the denoising result of the BM3D method with sigma set to 10, Figure 5 (d) is the result obtained by setting the template size of the Gaussian filtering method to 7*7, Figure 5 (e) is the denoising result obtained by using the label as the guide image of the guided filtering method, Figure 5 (f) is the denoising result of the wavelet soft threshold method. BM3D It is shown that the method can sufficiently denoise the measured object while retaining the edge and detail information of the image, thereby ensuring that the modulation information generated by the fringe is not affected, and the quality of the fringe image is better optimized. Table 1 shows that the BM3D method and the lightweight network are used to denoise 100 images in the test set, the denoising time of the two methods is timed, and the average denoising time of each image is calculated. Compared with the BM3D method, the denoising time of the lightweight network is shortened by nearly 60 times. At the same time, the MAE of each image is calculated, and the average value of the MAE of the lightweight network and the label BM3D is 0.00094. It can be seen that the lightweight network with the BM3D method as the label can effectively remove the noise of the infrared projector while improving the speed, and can effectively remove the noise interference generated by the projector while ensuring that the modulation information in the fringe image is not affected.

[0059] Table 1

[0060] Lightweight network Average denoising time per picture MAE with label (BM3D) 2.58s 0.043s ​ / 0.00094

Claims

1. A fast denoising method for near-infrared stripe images based on lightweight networks, characterized in that, include: Step 1: Construct a lightweight denoising neural network; the lightweight denoising neural network includes one downsampling main path, four downsampling branches, and one upsampling main path. Each downsampling branch includes a residual module and a downsampling module connected in sequence. The downsampling main path includes a convolutional layer, a residual module, a combination of three sets of downsampling modules and a residual module, and a combination of a downsampling module and a multi-head self-attention module connected in sequence. The input terminals of the four downsampling branches are respectively connected to the combination of three sets of downsampling modules and a residual module, and the combination of a downsampling module and a multi-head self-attention module. The input terminals of the combination are connected one-to-one in sequence. The main upsampling path includes four sets of combinations of upsampling modules and residual modules connected in sequence, as well as a convolutional layer. The output terminal of the fourth downsampling branch is connected to the input terminal of the combination of the first set of upsampling modules and residual modules. The output terminal of the third downsampling branch is connected to the input terminal of the combination of the second set of upsampling modules and residual modules. The output terminal of the second downsampling branch is connected to the input terminal of the combination of the third set of upsampling modules and residual modules. The output terminal of the first downsampling branch is connected to the input terminal of the combination of the fourth set of upsampling modules and residual modules. Step 2: Use a near-infrared structured light system to acquire images of samples with different surface shapes in a real scene, set the optimal parameters for each image, and use the images with the optimal parameters to prepare a dataset; Step 3: Input the dataset into the lightweight denoising neural network for training; Step 4: Input the noisy infrared stripe pattern acquired in real time into the trained lightweight denoising neural model to obtain the denoised optimized stripe image.

2. The fast denoising method for near-infrared stripe images based on lightweight networks according to claim 1, characterized in that, The multi-head self-attention module is a four-head self-attention module.

3. The fast denoising method for near-infrared stripe images based on lightweight networks according to claim 1, characterized in that, The specific method for acquiring images of samples with different surface shapes in a real scene using a near-infrared structured light system is as follows: The three-step phase-shifting grating intensity map, acquired by simultaneously capturing the grating image projected by the infrared projector using a camera, is represented as follows: Where I′1(x,y), I′2(x,y), and I′3(x,y) are the corresponding three-step phase-shifting grating light intensities, (x,y) are the pixel coordinates on the camera plane, A(x,y) is the background light intensity, and B(x,y) is the modulation degree of the stripes. For the wrapping phase; grating intensity maps of different frequencies Unlike δ(x,y), δ(x,y) is multiplicative noise.

4. The fast denoising method for near-infrared stripe images based on lightweight networks according to claim 3, characterized in that, The BM3D method is used to denoise the acquired noisy images. The noisy image I′ captured synchronously by the camera is represented by BM3D after processing as follows: Among them, I g This represents the image after denoising.

5. The fast denoising method for near-infrared stripe images based on lightweight networks according to claim 4, characterized in that, The specific method for training a lightweight denoising neural model by inputting the dataset is as follows: The noisy image captured simultaneously by the camera is used as input data, and the image after denoising using the BM3D method is used as the standard value to input into the lightweight denoising neural network. Calculate the difference between the standard value and the model output value; By combining backpropagation, the internal parameters of the lightweight denoising neural network are iteratively optimized until the loss function converges.

6. The fast denoising method for near-infrared stripe images based on lightweight networks according to claim 5, characterized in that, L2 loss is used as the loss function.

Images