Underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction

Abstract

The invention relates to the technical field of underwater imaging, in particular to an underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction, which comprises a light modulation module, a multispectral light intensity data acquisition module and a control and image reconstruction module, the control and image reconstruction module is respectively connected with the light modulation module and the multispectral light intensity data acquisition module, the control and image reconstruction module controls the light modulation module to modulate an illumination light beam and then project the illumination light beam to a target object, and the modulated illumination light beam is reflected by the target object and then received by the multispectral light intensity data acquisition module; and the multispectral light intensity data acquisition module sends the received reflected light beam to the control and image reconstruction module, image reconstruction is performed through a pre-trained deep neural network in the control and image reconstruction module, and a super-resolution image is output. The underwater multispectral ghost imaging system is provided in combination with a deep learning image super-resolution reconstruction algorithm, and the imaging quality and resolution are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater imaging, and particularly to an underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction. Background Art

[0002] Underwater optical imaging technology has important application values in fields such as marine resource exploration, ecological monitoring, and military detection. However, traditional imaging methods are vulnerable to the interference of water body scattering, absorption effects, and backscattering noise, resulting in problems such as blurred images, decreased contrast, and loss of spectral information. Multispectral ghost imaging technology can effectively suppress water body scattering noise and simultaneously obtain the spatial-spectral joint information of the target by correlating the spatial intensity fluctuations of the light field with the target reflection characteristics. Its non-local imaging characteristics are particularly suitable for complex underwater environments. Compared with conventional multispectral imaging systems, this system can achieve high-dimensional data acquisition without mechanical scanning, and at the same time, combining the compressed sensing theory can significantly improve the imaging efficiency. However, limited by underwater channel attenuation and the inherent computational reconstruction characteristics of ghost imaging, the system still faces challenges such as slow reconstruction speed and insufficient spatial resolution, restricting the application requirements of high-precision underwater detection.

[0003] Deep learning methods provide a new technical path for the image super-resolution reconstruction of underwater multispectral ghost imaging systems. Traditional reconstruction algorithms based on interpolation or sparse representation are difficult to effectively restore the high-frequency details and spectral features degraded by the water body. Therefore, the present invention proposes an underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction. Summary of the Invention

[0004] The object of the present invention is to provide an underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction for the problems of poor imaging quality, low resolution, and susceptibility to noise interference existing in the application of traditional imaging systems in underwater scenarios. This system has a compact structure, is easy to install, and has a low cost. Combining the deep learning image super-resolution reconstruction algorithm, it improves the imaging quality and resolution.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] An underwater multi-spectral ghost imaging system based on deep learning image super-resolution reconstruction, comprising: a light modulation module, a multi-spectral light intensity data acquisition module, and a control and image reconstruction module. The control and image reconstruction module is respectively connected to the light modulation module and the multi-spectral light intensity data acquisition module. The control and image reconstruction module controls the light modulation module to modulate the illumination beam and project it onto the target object. The modulated illumination beam is reflected by the target object and then received by the multi-spectral light intensity data acquisition module. The multi-spectral light intensity data acquisition module sends the received reflected beam to the control and image reconstruction module, and the pre-trained deep neural network in the control and image reconstruction module performs image reconstruction to output a super-resolution image.

[0007] Optionally, the light modulation module modulates the illumination beam using Hadamard speckles.

[0008] Optionally, the multi-spectral light intensity data acquisition module includes: a planar array CCD detector, a linear variable filter, a multi-aperture light shield, and a microlens array. Among them, the linear variable filter, the multi-aperture light shield, and the microlens array are arranged in sequence in front of the planar array CCD detector, and the microlens array, the multi-aperture light shield, and the linear variable filter are aligned with the horizontal and vertical directions of the planar array CCD detector. The center of each microlens corresponds one-to-one with the center of the corresponding aperture on the multi-aperture light shield. The planar array CCD detector is at the focal length of the microlens array, and the horizontal angle between the linear variable filter and the microlens array is β.

[0009] Optionally, the modulated illumination beam is reflected by the target object and then received by the multi-spectral light intensity data acquisition module to generate a multi-spectral data set matrix S = [S 1 , S 2 ,..., S k ,..., S K , where k = 1, 2,..., K, K = A × B, A × B represents the number of rows and columns of the microlens array, and S k represents the one-dimensional light field intensity sequence collected by the k-th microlens counted by rows on the microlens array.

[0010] Optionally, the deep neural network includes K branches. After each branch inputs the corresponding one-dimensional light field intensity sequence, it is processed by a fully connected network, a 3×3 convolution, an image reconstruction module, a noise and blur kernel, and a feature extractor, and then outputs the corresponding branch feature; the branch features are fused and then input into a bottleneck layer convolution and a convolution offset generator to generate offset features; the offset features and the branch features are input into a deformable convolution and a reconstruction layer convolution to generate reconstruction features; the reconstruction features and the one-dimensional light field intensity sequence are fused and then output a super-resolution reconstruction image.

[0011] Optionally, the deep neural network is trained by a training set, and the training set includes a multi-spectral data set matrix C n , and the multi-spectral data set matrix C n is obtained by calculating after reducing the size of the original image and randomly moving pixels.

[0012] Optionally, during the training process of the deep neural network, after the multi-spectral data set matrix C n is input into the deep neural network and processed into a two-dimensional image, m noise level maps are added to the two-dimensional image. The pixel values of the noise level maps are randomly generated within a preset range, and Gaussian noise, Poisson noise, additive Gaussian white noise, Gaussian blur, defocus blur, and motion blur are added to the noise level maps.

[0013] Optionally, the loss calculation during the training process of the deep neural network is:

[0014] L(P) = L align + L sr ;

[0015]

[0016] wherein, L align represents the network training loss, and L sr represents the SR reconstruction network loss. is the reference frame, is the low-resolution LR auxiliary frame, is the aligned LR frame corresponding to the training prediction, is the final high-resolution (HR) frame estimate, is the HR frame of the t-th frame.

[0017] The beneficial effects of the present invention are as follows:

[0018] The present invention designs an underwater multi-spectral ghost imaging system with a compact structure, convenient installation, and low cost. By adopting a multi-spectral light intensity data acquisition module, a multi-spectral data set matrix of different bands of the target object can be obtained. At the same time, due to the adoption of a multi-aperture light shielding plate, the interference between multi-spectral data is avoided, and the anti-interference ability and practicability are stronger. In addition, by adopting an image super-resolution reconstruction algorithm based on deep learning, a multi-spectral super-resolution image of the target object can be reconstructed. The present invention is very conducive to the application research of underwater multi-spectral ghost imaging and deep learning technologies, and is expected to be widely applied in the fields of deep sea exploration, underwater target recognition, deep sea mineral exploration, etc. Description of the Drawings

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0020] Figure 1 Schematic structural diagram of the underwater multi-spectral ghost imaging system according to an embodiment of the present invention;

[0021] Figure 2 Schematic structural diagram of the multi-spectral light intensity data acquisition module 103 according to an embodiment of the present invention;

[0022] Figure 3 Front view of the multi-aperture light-shielding plate 203 according to an embodiment of the present invention;

[0023] Figure 4 Schematic diagram of the multi-spectral data set matrix according to an embodiment of the present invention;

[0024] Figure 5 Schematic structural diagram of the deep neural network according to an embodiment of the present invention;

[0025] Figure 6 Schematic structural diagram of the image reconstruction module according to an embodiment of the present invention;

[0026] Figure 7 Schematic structural diagram of the feature extractor according to an embodiment of the present invention;

[0027] Figure 8 Schematic structural diagram of the residual block in the feature extractor according to an embodiment of the present invention;

[0028] Among them, 101, control and image reconstruction module; 102, light modulation module; 103, multi-spectral light intensity data acquisition module; 201, area array CCD detector; 202, linear gradient filter; 203, multi-aperture light-shielding plate; 204, microlens array; 301, aperture; 302, opaque light-shielding plate. Detailed implementation manners

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0030] To make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the drawings and specific implementation manners.

[0031] Through an end-to-end learning mechanism, deep neural networks can mine the non-linear mapping relationships and spatial-spectral correlation features among multi-spectral data. By constructing a multi-scale feature fusion network architecture and combining a physical imaging model with a data-driven strategy, image resolution improvement, noise suppression, and spectral fidelity optimization can be achieved simultaneously. Therefore, the development of high-performance underwater multi-spectral ghost imaging systems and high-quality image reconstruction algorithms contributes to the application and development of underwater imaging technologies.

[0032] This embodiment provides an underwater multi-spectral ghost imaging system based on deep learning image super-resolution reconstruction, as Figure 1 shown, including: a light modulation module 102, a multi-spectral light intensity data acquisition module 103, and a control and image reconstruction module 101. The control and image reconstruction module 101 is respectively connected to the light modulation module 102 and the multi-spectral light intensity data acquisition module 103. The control and image reconstruction module 101 controls the light modulation module 102 to modulate the illumination beam and project it onto the target object. The modulated illumination beam is reflected by the target object and received by the multi-spectral light intensity data acquisition module 103. The multi-spectral light intensity data acquisition module 103 sends the received reflected beam to the control and image reconstruction module 101, and the pre-trained deep neural network in the control and image reconstruction module 101 performs image reconstruction to output a super-resolution image.

[0033] Specifically, this embodiment designs an underwater multi-spectral ghost imaging system with a compact structure, convenient installation, and low cost. By using the multi-spectral light intensity data acquisition module 103, a multi-spectral data set matrix of different bands of the target object can be obtained. At the same time, due to the adoption of a multi-aperture light shield 203, the interference between multi-spectral data is avoided, and the anti-interference ability and practicality are stronger. In addition, by using an image super-resolution reconstruction algorithm based on deep learning, a multi-spectral super-resolution image of the target object can be reconstructed. This embodiment is very conducive to the application research of underwater multi-spectral ghost imaging and deep learning technologies, and is expected to be widely applied in fields such as deep sea exploration, underwater target recognition, and deep sea mineral exploration.

[0034] Furthermore, the light modulation module 102 modulates the illumination beam using Hadamard speckles.

[0035] The multi-spectral light intensity data acquisition module 103 includes: an area array CCD detector 201, a linear variable filter 202, a multi-aperture light shield 203, and a microlens array 204. Among them, the linear variable filter 202, the multi-aperture light shield 203, and the microlens array 204 are arranged in sequence in front of the area array CCD detector 201, and the microlens array 204, the multi-aperture light shield 203, and the linear variable filter 202 are aligned with the horizontal and vertical directions of the area array CCD detector 201. The multi-aperture light shield 203 is formed by setting a number of apertures 301 on an opaque light shield 302. The center of each microlens corresponds one-to-one with the center of the corresponding aperture 301 on the multi-aperture light shield 203. The area array CCD detector 201 is at the focal length of the microlens array 204, and the horizontal included angle between the linear variable filter 202 and the microlens array 204 is β.

[0036] Specifically, as Figure 2 , Figure 3 shown, in this embodiment, the focal length of the microlens array 204 is f, the resolution of the area array CCD detector 201 is X×Y, and the distance between the microlens array 204 and the area array CCD detector is A1. The dimension of the microlens in the microlens array 204 is A×B, and the dimension of the aperture 301 in the multi-aperture light shield 203 is A×B. During the installation process, ensure that the center of each microlens corresponds one-to-one with the center of the corresponding aperture 301 on the multi-aperture light shield 203. Adjust the microlens array 204, the multi-aperture light shield 203, and the linear variable filter 202 with the area array CCD detector 201 to make them aligned in the horizontal and vertical directions. Adjust the distance between the microlens array 204 and the area array CCD detector 201 so that the area array CCD detector 201 is at the focal length of the microlens array 204. Then rotate the linear variable filter 202 clockwise so that its horizontal included angle with the microlens array 204 is β.

[0037] Further, the modulated illumination beam is reflected by the target object and then received by the multi-spectral light intensity data acquisition module 103, generating a multi-spectral data set matrix S = [S 1 , S 2 ,..., S k ,..., S K , where k = 1, 2,..., K, K = A×B, A×B represents the number of rows and columns of the microlens array 204, and S k represents the one-dimensional light field intensity sequence collected by the kth microlens counted by row on the microlens array 204.

[0038] Specifically, in this embodiment, the control and image reconstruction module 101 is used to perform Hadamard speckle H h = H h(x, y) is loaded onto the optical modulation module 102, and the illumination beam is modulated by the Hadamard speckle. The modulated beam is projected onto the target object, and the reflected beam is received by the multi-spectral light intensity data acquisition module 103. Since the microlens array 204 has A × B microlenses, the area corresponding to each microlens on the area array CCD detector 201 is divided, and the corresponding imaging image can be obtained for each microlens. Taking one of the microlenses as an example, its light field intensity value I h is expressed as follows:

[0039] I h = ∫∫O(x, y)H h (x, y)dxdy;

[0040] where O(x, y) is the target object function, h = 1, 2,..., Q, Q is the total number of Hadamard speckles, and (x, y) are the pixel coordinates. The pixel size of the Hadamard speckles is s × s, and s is a positive integer.

[0041] After the computer loads Q Hadamard speckles, each microlens forms a speckle sequence H, which is expressed as:

[0042] H = [H 1 , H 2 ,..., H Q ;

[0043] Corresponding to the speckle sequence H, a one-dimensional light field intensity sequence S can be obtained:

[0044] S = [I 1 , I 2 ,..., I Q T ;

[0045] where [·] T means transposing the row vector into a column vector. Since the multi-spectral light intensity data acquisition module 103 has A × B microlenses, a multi-spectral data set matrix S can be finally obtained, as Figure 4 shown, and its expression is as follows:

[0046] S = [S 1 , S 2 ,..., S k ,..., S K ;

[0047] where k = 1, 2,..., K, K = A × B, and S k represents the one-dimensional light field intensity sequence collected by the k-th microlens counted by row on the microlens array 204, as Figure 4 shown.

[0048] Furthermore, as​Figure 5 As shown, the deep neural network includes K branches. After each branch inputs the corresponding one-dimensional light field intensity sequence, it is processed by a fully connected network, a 3×3 convolution, an image reconstruction module, noise and a blur kernel, and a feature extractor, and then outputs the corresponding branch features. After fusing the branch features, they are input into a bottleneck layer convolution and a convolution offset generator to generate offset features. The offset features and the branch features are input into a deformable convolution and a reconstruction layer convolution to generate reconstruction features. After fusing the reconstruction features and the one-dimensional light field intensity sequence, a super-resolution reconstructed image is output.

[0049] Specifically, in this embodiment, the deep learning image super-resolution reconstruction algorithm obtains the multi-spectral target object image through a deep neural network. The multi-spectral data set matrix S is input into the deep neural network, and the output is the super-resolution image MGI. Its mathematical model can be expressed as follows:

[0050] MGI(x,y) = mgisr(S,P);

[0051] Among them, MGI(x,y) represents the multi-spectral super-resolution image of the target object, mgisr() represents the implicit function of the deep neural network, P is the network parameter of the deep neural network, and (x,y) represents the pixel coordinates.

[0052] As Figure 6 , Figure 7 , Figure 8 shown, the image reconstruction module includes an input layer, an encoder, a decoder, and an output layer connected in sequence. The encoder and the decoder are connected by a residual connection; the feature extractor includes a 3×3 convolution and several residual blocks connected in sequence. Each residual block includes a 3×3 convolution, a ReLu activation function, a 3×3 convolution, batch normalization, and a residual link, and outputs the extracted features.

[0053] Further, the deep neural network is trained through a training set, and the training set includes a multi-spectral data set matrix C n , and the multi-spectral data set matrix C n is obtained by calculating after shrinking the original image and randomly moving the pixels.

[0054] During the training process of the deep neural network, after processing the multi-spectral data set matrix C n into a two-dimensional image and inputting it into the deep neural network, m noise level maps are added to the two-dimensional image. The pixel values of the noise level maps are randomly generated within a preset range, and Gaussian noise, Poisson noise, additive white Gaussian noise, Gaussian blur, defocus blur, and motion blur are added to the noise level maps.

[0055] Specifically, in this embodiment, the training data set C in n The process of generating the multi - spectral data set matrix is as follows:

[0056] Take N different color images with pixel size of b×b as the original target images. For the nth original target image GT n Reduce it by v times, and then randomly move the reduced image e and g pixels in the horizontal and vertical directions respectively, where e, g ∈ [E, G]. For the reduced image, calculate the multi - spectral data set matrix C n =[S 1 , S 2 ,..., S k ,..., S K . In the formula, S k represents the one - dimensional light field intensity sequence collected by the kth microlens counted by rows on the microlens array.

[0057] During the neural network training process, the one - dimensional light field intensity sequence is first input into the network. After being processed by 3 network layers, it becomes a two - dimensional image. Add m noise level maps to the obtained two - dimensional image. The pixel values of the bth noise level map are all U b , where b = 1, 2,..., m. U b is randomly generated in the range of (0, 80]. Then add Gaussian noise, Poisson noise, additive white Gaussian noise, Gaussian blur, defocus blur, motion blur, etc. to the noise level maps. Adding the noise level maps simulates the noise interference in the data acquisition process of the underwater ghost imaging system, and is used to enable the neural network model to have the ability to denoise and remove the noise in the image.

[0058] After performing D training times, the optimized network model parameters can be obtained

[0059] For an underwater target, use the multi - spectral light intensity data acquisition system to obtain the multi - spectral data set matrix S act , and then use the trained neural network model to reconstruct a high - resolution and high - quality multi - spectral image of the target magnified by v times That is

[0060] Furthermore, the loss function of the above - mentioned neural network training is:

[0061] L(P)=L align +L sr ;

[0062] Among them, L align The loss function is used for network training and is:

[0063]

[0064] Among them, is the reference frame, is the low-resolution (LR) auxiliary frame, is the aligned LR frame corresponding to the training prediction.

[0065] L sr represents the loss function of the SR reconstruction network:

[0066]

[0067] Among them, is the final high-resolution (HR) frame estimate, is the HR frame of the t-th frame.

[0068] Since the true value of the aligned low-resolution frame cannot be obtained, in order to optimize the training of the deep neural network, the LR reference frame is used as the label to make the aligned low-resolution frame close to the reference frame.

[0069] The embodiments described above are only descriptions of the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. An underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction, characterized in that: include: A light modulation module, a multi-spectral light intensity data acquisition module, and a control and image reconstruction module. The control and image reconstruction module are respectively connected to the light modulation module and the multi-spectral light intensity data acquisition module. The control and image reconstruction module controls the light modulation module to modulate the illumination light beam and then projects it onto the target object. The modulated illumination light beam is reflected by the target object and then received by the multi-spectral light intensity data acquisition module. The multi-spectral light intensity data acquisition module sends the received reflected light beam to the control and image reconstruction module. The control and image reconstruction module performs image reconstruction using a pre-trained deep neural network to output a super-resolution image.

2. The underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction according to claim 1 is characterized in that: The light modulation module modulates the illumination light beam using Hadamard speckle.

3. The underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction according to claim 1 is characterized in that: The multi-spectral light intensity data acquisition module includes: an area array CCD detector, a linear gradient filter, a multi-aperture shading plate, and a microlens array, wherein the linear gradient filter, the multi-aperture shading plate, and the microlens array are arranged in sequence in front of the area array CCD detector, and the microlens array, the multi-aperture shading plate, and the linear gradient filter are aligned with the area array CCD detector in the horizontal and vertical directions, the center of each microlens corresponds one-to-one to the center of the corresponding aperture on the multi-aperture shading plate, the area array CCD detector is at the focal length of the microlens array, and the horizontal angle between the linear gradient filter and the microlens array is β.

4. The underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction according to claim 3 is characterized in that: The modulated illumination beam is reflected by the target object and then received by the multi-spectral light intensity data acquisition module to generate a multi-spectral data set matrix S=[S1, S2, ..., S k ,...,S K ], where k = 1, 2, ..., K, K = A × B, A × B represents the number of rows and columns of the microlens array, S k Represents the one-dimensional light field intensity sequence collected by the kth microlens counted by row on the microlens array.

5. The underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction according to claim 1, characterized in that: The deep neural network includes K branches. After each branch inputs the corresponding one-dimensional light field intensity sequence, it is processed by a fully connected network, a 3×3 convolution, an image reconstruction module, a noise and blur kernel, and a feature extractor to output the corresponding branch feature; the branch features are feature fused and input into a bottleneck layer convolution and a convolution offset generator to generate an offset feature; the offset feature and the branch feature are input into a deformable convolution and a reconstruction layer convolution to generate a reconstruction feature; the reconstruction feature and the one-dimensional light field intensity sequence are feature fused to output a super-resolution reconstructed image.

6. The underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction according to claim 5, characterized in that: The deep neural network is trained by a training set, wherein the training set includes a multispectral data set matrix C n , the multispectral dataset matrix C n It is calculated by shrinking the original image and randomly moving the pixels.

7. The underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction according to claim 6, characterized in that: During the training process of the deep neural network, the multispectral dataset matrix C n After the deep neural network is input and processed into a two-dimensional image, m noise level maps are added to the two-dimensional image. The pixel values ​​of the noise level maps are randomly generated within a preset range. Gaussian noise, Poisson noise, additive Gaussian white noise, Gaussian blur, defocus blur, and motion blur are added to the noise level map.

8. The underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction according to claim 5, characterized in that: The loss of the deep neural network during training is calculated as: L(P)=L align +L sr ; Among them, L align represents the network training loss, L sr represents the SR reconstruction network loss, is the reference frame, is a low-resolution LR auxiliary frame, For training prediction, the corresponding aligned LR frames are For the final high-resolution (HR) frame estimation, I t HR ∈R sH×sW×C is the tth HR frame.

Images