An underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction

Abstract

This invention relates to the field of underwater imaging technology, and in particular to an underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction. The system includes: an optical modulation module, a multispectral light intensity data acquisition module, and a control and image reconstruction module. The control and image reconstruction module is connected to both the optical modulation module and the multispectral light intensity data acquisition module. The control and image reconstruction module controls the optical modulation module to modulate an illumination beam and project it onto a target object. The modulated illumination beam is reflected by the target object and received by the multispectral light intensity data acquisition module. The multispectral light intensity data acquisition module sends the received reflected beam to the control and image reconstruction module. In the control and image reconstruction module, a pre-trained deep neural network performs image reconstruction, outputting a super-resolution image. This invention combines a deep learning image super-resolution reconstruction algorithm to provide an underwater multispectral ghost imaging system, improving imaging quality and resolution.

Description

Technical Field

[0001] This invention relates to the field of underwater imaging technology, and in particular to an underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction. Background Technology

[0002] Underwater optical imaging technology has significant applications in marine resource exploration, ecological monitoring, and military reconnaissance. However, traditional imaging methods are susceptible to interference from water scattering, absorption effects, and backscattering noise, leading to problems such as image blurring, decreased contrast, and loss of spectral information. Multispectral ghost imaging technology, by correlating spatial intensity fluctuations of the light field with target reflection characteristics, can effectively suppress water scattering noise and simultaneously acquire joint spatial-spectral information of the target. Its non-local imaging characteristics are particularly suitable for complex underwater environments. Compared to conventional multispectral imaging systems, this system can achieve high-dimensional data acquisition without mechanical scanning, and the combination with compressed sensing theory can significantly improve imaging efficiency. However, limited by underwater channel attenuation and the inherent computational reconstruction characteristics of ghost imaging, the system still faces challenges such as slow reconstruction speed and insufficient spatial resolution, restricting the application requirements for high-precision underwater detection.

[0003] Deep learning methods offer a novel technical approach for super-resolution image reconstruction in underwater multispectral ghost imaging systems. Traditional reconstruction algorithms based on interpolation or sparse representation struggle to effectively recover high-frequency details and spectral features degraded by water. Therefore, this invention proposes an underwater multispectral ghost imaging system based on deep learning-based image super-resolution reconstruction. Summary of the Invention

[0004] The purpose of this invention is to address the problems of poor imaging quality, low resolution, and susceptibility to noise interference in traditional imaging systems used in underwater scenarios. It provides an underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction. This system is compact, easy to install, and inexpensive. By combining deep learning image super-resolution reconstruction algorithms, it improves both imaging quality and resolution.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] An underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction includes: a light modulation module, a multispectral light intensity data acquisition module, and a control and image reconstruction module. The control and image reconstruction module is connected to the light modulation module and the multispectral light intensity data acquisition module, respectively. The control and image reconstruction module controls the light modulation module to modulate an illumination beam and project it onto a target object. The modulated illumination beam is reflected by the target object and received by the multispectral light intensity data acquisition module. The multispectral light intensity data acquisition module sends the received reflected beam to the control and image reconstruction module. In the control and image reconstruction module, a pre-trained deep neural network is used to reconstruct the image and output a super-resolution image.

[0007] Optionally, the light modulation module modulates the illumination beam using Hadamard speckle.

[0008] Optionally, the multispectral light intensity data acquisition module includes: an area CCD detector, a linear gradient filter, a multi-aperture light-shielding plate, and a microlens array. The linear gradient filter, the multi-aperture light-shielding plate, and the microlens array are arranged sequentially in front of the area CCD detector, and the microlens array, the multi-aperture light-shielding plate, and the linear gradient filter are aligned with the horizontal and vertical directions of the area CCD detector. The center of each microlens corresponds one-to-one with the center of the corresponding aperture on the multi-aperture light-shielding plate. The area CCD detector is located at the focal length of the microlens array, and the horizontal angle between the linear gradient filter and the microlens array is β.

[0009] Optionally, the modulated illumination beam is reflected by the target object and received by the multispectral light intensity data acquisition module to generate a multispectral dataset matrix S = [S1, S2, ..., S...]. k ,...,S K ], where k = 1, 2, ..., K, K = A × B, A × B represents the number of rows and columns of the microlens array, S k This represents the one-dimensional light field intensity sequence acquired by the k-th microlens, which is counted row by row on the microlens array.

[0010] Optionally, the deep neural network includes K branches. Each branch is input with a corresponding one-dimensional light field intensity sequence, and after processing by a fully connected network, a 3×3 convolution, an image reconstruction module, a noise and blur kernel, and a feature extractor, it outputs the corresponding branch features. The branch features are then fused and input into a bottleneck layer convolution and a convolution offset generator to generate offset features. The offset features and branch features are then input into a deformable convolution and a reconstruction layer convolution to generate reconstruction features. The reconstruction features are then fused with the one-dimensional light field intensity sequence to output a super-resolution reconstructed image.

[0011] Optionally, the deep neural network is trained using a training set, which includes a multispectral dataset matrix C.n The multispectral dataset matrix C n The result was obtained by scaling down the original image and randomly shifting the pixels.

[0012] Optionally, during the training process, the deep neural network uses the multispectral dataset matrix C n After the input is processed into a two-dimensional image by the deep neural network, m noise level images are added to the two-dimensional image. The pixel values ​​of the noise level images are randomly generated within a preset range. Gaussian noise, Poisson noise, additive white Gaussian noise, Gaussian blur, defocus blur, and motion blur are added to the noise level images.

[0013] Optionally, the loss of the deep neural network during training is calculated as follows:

[0014] L(P) = L align +L sr ;

[0015]

[0016] Among them, L align L represents the network training loss. sr This indicates the network loss during SR reconstruction. As a reference frame, For low-resolution LR auxiliary frames, To train and predict the corresponding aligned LR frames, For the final high-resolution (HR) frame estimation, t is the HR frame.

[0017] The beneficial effects of this invention are as follows:

[0018] This invention designs a compact, easy-to-install, and low-cost underwater multispectral ghost imaging system. By employing a multispectral light intensity data acquisition module, a multispectral dataset matrix of different bands of the target object can be obtained. Furthermore, the use of a multi-aperture light-shielding plate avoids mutual interference between multispectral data, enhancing its anti-interference capability and practicality. In addition, by employing a deep learning-based image super-resolution reconstruction algorithm, a multispectral super-resolution image of the target object can be reconstructed. This invention is highly beneficial for the application research of underwater multispectral ghost imaging and deep learning technology, and is expected to be widely used in deep-sea exploration, underwater target identification, and deep-sea mineral exploration. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the underwater multispectral ghost imaging system according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the multispectral light intensity data acquisition module 103 according to an embodiment of the present invention;

[0022] Figure 3 This is a front view of the multi-aperture light-shielding plate 203 according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of the multispectral dataset matrix according to an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of a deep neural network structure according to an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of the image reconstruction module structure according to an embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram of the feature extractor structure according to an embodiment of the present invention;

[0027] Figure 8 This is a schematic diagram of the residual block structure in the feature extractor of this invention.

[0028] Among them, 101 is the control and image reconstruction module; 102 is the light modulation module; 103 is the multispectral light intensity data acquisition module; 201 is the area array CCD detector; 202 is the linear gradient filter; 203 is the multi-aperture light shield; 204 is the microlens array; 301 is the aperture; and 302 is the opaque light shield. Detailed Implementation

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Deep neural networks, through end-to-end learning mechanisms, can uncover nonlinear mapping relationships and spatial-spectral correlation features between multispectral data. By constructing a multi-scale feature fusion network architecture and combining it with physical imaging models and data-driven strategies, image resolution enhancement, noise suppression, and spectral fidelity optimization can be achieved simultaneously. Therefore, developing high-performance underwater multispectral ghost imaging systems and high-quality image reconstruction algorithms will contribute to the application and development of underwater imaging technology.

[0032] This embodiment provides an underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction, such as... Figure 1 As shown, the system includes: a light modulation module 102, a multispectral light intensity data acquisition module 103, and a control and image reconstruction module 101. The control and image reconstruction module 101 is connected to the light modulation module 102 and the multispectral light intensity data acquisition module 103, respectively. The control and image reconstruction module 101 controls the light modulation module 102 to modulate the illumination beam and project it onto the target object. The modulated illumination beam is reflected by the target object and received by the multispectral light intensity data acquisition module 103. The multispectral light intensity data acquisition module 103 sends the received reflected beam to the control and image reconstruction module 101. The control and image reconstruction module 101 performs image reconstruction using a pre-trained deep neural network and outputs a super-resolution image.

[0033] Specifically, this embodiment designs a compact, easy-to-install, and low-cost underwater multispectral ghost imaging system. By employing a multispectral light intensity data acquisition module 103, a multispectral dataset matrix of different bands of the target object can be obtained. Simultaneously, the use of a multi-aperture light-shielding plate 203 avoids mutual interference between multispectral data, enhancing its anti-interference capability and practicality. Furthermore, by employing a deep learning-based image super-resolution reconstruction algorithm, a multispectral super-resolution image of the target object can be reconstructed. This embodiment is highly beneficial for the application research of underwater multispectral ghost imaging and deep learning technology, and is expected to be widely used in fields such as deep-sea exploration, underwater target identification, and deep-sea mineral exploration.

[0034] Furthermore, the light modulation module 102 modulates the illumination beam using Hadamard speckle.

[0035] The multispectral light intensity data acquisition module 103 includes: an area CCD detector 201, a linear gradient filter 202, a multi-aperture light-shielding plate 203, and a microlens array 204. The linear gradient filter 202, the multi-aperture light-shielding plate 203, and the microlens array 204 are arranged sequentially in front of the area CCD detector 201, and the microlens array 204, the multi-aperture light-shielding plate 203, and the linear gradient filter 202 are aligned with the area CCD detector 201 in the horizontal and vertical directions. The multi-aperture light-shielding plate 203 is formed by setting a number of apertures 301 on an opaque light-shielding plate 302. The center of each microlens corresponds one-to-one with the center of the corresponding aperture 301 on the multi-aperture light-shielding plate 203. The area CCD detector 201 is located at the focal length of the microlens array 204, and the horizontal angle between the linear gradient filter 202 and the microlens array 204 is β.

[0036] Specifically, such as Figure 2 , Figure 3 As shown, in this embodiment, the focal length of the microlens array 204 is f, the resolution of the area CCD detector 201 is X×Y, and the distance between the microlens array 204 and the area CCD detector is A1. The dimensions of the microlenses in the microlens array 204 are A×B, and the dimensions of the apertures 301 in the multi-aperture light-shielding plate 203 are A×B. During installation, ensure that the center of each microlens corresponds one-to-one with the center of the corresponding aperture 301 on the multi-aperture light-shielding plate 203. Adjust the microlens array 204, the multi-aperture light-shielding plate 203, and the linear gradient filter 202 to align with the area CCD detector 201 in the horizontal and vertical directions. Adjust the distance between the microlens array 204 and the area CCD detector 201 so that the area CCD detector 201 is at the focal length of the microlens array 204. Then rotate the linear gradient filter 202 clockwise so that the horizontal angle between it and the microlens array 204 is β.

[0037] Furthermore, the modulated illumination beam, after being reflected by the target object, is received by the multispectral light intensity data acquisition module 103, generating a multispectral dataset matrix S = [S1, S2, ..., S...]. k ,...,S K ], where k = 1, 2, ..., K, K = A × B, A × B represents the number of rows and columns of the microlens array 204, S k This represents the one-dimensional light field intensity sequence acquired by the k-th microlens, which is counted row by row on the microlens array 204.

[0038] Specifically, in this embodiment, the control and image reconstruction module 101 is used to reconstruct the Hadamard speckle pattern H. h =H hThe (x, y) light beam is loaded onto the light modulation module 102, and the illumination beam is modulated by Hadamard speckle. The modulated beam is projected onto the target object, and the reflected beam is received by the multispectral light intensity data acquisition module 103. Since the microlens array 204 has A×B microlenses, the corresponding area on the area array CCD detector 201 for each microlens can be divided, and the corresponding imaging image can be obtained for each microlens. Taking one microlens as an example, its light field intensity value I... h It is expressed as follows:

[0039] I h =∫∫O(x,y)H h (x,y)dxdy;

[0040] Where O(x,y) is the object function, h = 1, 2, ..., Q, Q is the total number of Hadamard speckles, and (x,y) are the pixel coordinates. The pixel size of the Hadamard speckles is s × s, where s is a positive integer.

[0041] After the computer loads Q Hadamard speckle patterns, each microlens forms a speckle sequence H, represented as:

[0042] H = [H1,H2,...,H] Q ];

[0043] Corresponding to the speckle sequence H, a one-dimensional light field intensity sequence S can be obtained:

[0044] S = [I1, I2, ..., I Q ] T ;

[0045] in,[·] T This represents the transpose of a row vector into a column vector. Since the multispectral intensity data acquisition module 103 has A×B microlenses, a multispectral dataset matrix S can ultimately be obtained, such as... Figure 4 As shown, it is represented as follows:

[0046] S = [S1,S2,...,S] k ,...,S K ];

[0047] Where k = 1, 2, ..., K, K = A × B, S k This represents the one-dimensional light field intensity sequence acquired by the k-th microlens, counted row by row, on the microlens array 204, such as... Figure 4 As shown.

[0048] Furthermore, if Figure 5As shown, the deep neural network includes K branches. Each branch takes a corresponding one-dimensional light field intensity sequence as input, and after processing by a fully connected network, 3×3 convolution, image reconstruction module, noise and blur kernel, and feature extractor, outputs the corresponding branch features. The branch features are then fused and input into a bottleneck layer convolution and a convolution offset generator to generate offset features. The offset features and branch features are then input into a deformable convolution and a reconstruction layer convolution to generate reconstruction features. The reconstruction features are then fused with the one-dimensional light field intensity sequence to output a super-resolution reconstructed image.

[0049] Specifically, in this embodiment, the deep learning image super-resolution reconstruction algorithm acquires multispectral target images through a deep neural network. The multispectral dataset matrix S is input into the deep neural network, and the output is the super-resolution image MGI. Its mathematical model can be expressed as follows:

[0050] MGI(x,y)=mgisr(S,P);

[0051] Where MGI(x,y) represents the multispectral super-resolution image of the target object, mgisr() represents the hidden function of the deep neural network, P is the network parameter of the deep neural network, and (x,y) represents the pixel coordinates.

[0052] like Figure 6 , Figure 7 , Figure 8 As shown, the image reconstruction module includes an input layer, an encoder, a decoder, and an output layer connected in sequence. The encoder and decoder are connected via residuals. The feature extractor includes a 3×3 convolution and several residual blocks connected in sequence. Each residual block includes a 3×3 convolution, a ReLU activation function, a 3×3 convolution, batch normalization, and residual connections connected in sequence, and outputs the extracted features.

[0053] Furthermore, the deep neural network is trained using a training set, which includes a multispectral dataset matrix C. n The multispectral dataset matrix C n The result was obtained by scaling down the original image and randomly shifting the pixels.

[0054] During the training process, the deep neural network uses the multispectral dataset matrix C n After the input is processed into a two-dimensional image by the deep neural network, m noise level images are added to the two-dimensional image. The pixel values ​​of the noise level images are randomly generated within a preset range. Gaussian noise, Poisson noise, additive white Gaussian noise, Gaussian blur, defocus blur, and motion blur are added to the noise level images.

[0055] Specifically, in this embodiment, the training dataset C inn The process of generating the multispectral dataset matrix is ​​as follows:

[0056] Take N different color images with pixel size b×b as the original target object images. For the nth original target object image GT n The image is reduced by a factor of v, and then randomly shifted by e and g pixels horizontally and vertically, respectively, where e, g ∈ [E, G]. For the reduced image, the multispectral dataset matrix C is calculated. n =[S1,S2,...,S k ,...,S K ]. In the formula S k This represents the one-dimensional light field intensity sequence acquired by the k-th microlens, which is counted row by row on the microlens array.

[0057] During neural network training, a one-dimensional light field intensity sequence is first input into the network. After processing through three network layers, it is transformed into a two-dimensional image. Then, m noise level images are added to the resulting two-dimensional image, where the pixel value of the b-th noise level image is U. b b = 1, 2, ..., m. U b It is randomly generated within the range (0, 80). Then, Gaussian noise, Poisson noise, additive white Gaussian noise, Gaussian blur, defocus blur, motion blur, etc. are added to the noise level map. The addition of the noise level map simulates the noise interference in the data acquisition process of the underwater ghost imaging system, which is used to enable the neural network model to have denoising ability and remove noise from the image.

[0058] After performing D training iterations, the optimized network model parameters can be obtained.

[0059] For an underwater target, the multispectral dataset matrix S is obtained using the multispectral light intensity data acquisition system. act Then, using the trained neural network model, a high-resolution, high-quality multispectral image of the target object magnified by v times can be reconstructed. Right now

[0060] Furthermore, the loss function for training the above neural network is:

[0061] L(P) = L align +L sr ;

[0062] Among them, L align The loss function used for network training is:

[0063]

[0064] in, As a reference frame, For low-resolution (LR) auxiliary frames, To train and predict the corresponding aligned LR frames.

[0065] L sr The loss function for SR reconstruction network is:

[0066]

[0067] in, For the final high-resolution (HR) frame estimation, t is the HR frame.

[0068] Since the true values ​​of the aligned low-resolution frames are unavailable, LR reference frames are used to optimize deep neural network training. As a label, it makes the aligned low-resolution frame closer to the reference frame.

[0069] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. An underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction, characterized in that, include: The system includes an optical modulation module, a multispectral light intensity data acquisition module, and a control and image reconstruction module. The control and image reconstruction module is connected to both the optical modulation module and the multispectral light intensity data acquisition module. The control and image reconstruction module controls the optical modulation module to modulate the illumination beam and project it onto the target object. The modulated illumination beam is reflected by the target object and received by the multispectral light intensity data acquisition module. The multispectral light intensity data acquisition module sends the received reflected beam to the control and image reconstruction module. In the control and image reconstruction module, a pre-trained deep neural network is used to reconstruct the image and output a super-resolution image. The multispectral light intensity data acquisition module includes: an area CCD detector, a linear gradient filter, a multi-aperture light shield, and a microlens array. The linear gradient filter, the multi-aperture light shield, and the microlens array are arranged sequentially in front of the area CCD detector, and the microlens array, the multi-aperture light shield, and the linear gradient filter are aligned with the horizontal and vertical directions of the area CCD detector. The center of each microlens corresponds one-to-one with the center of the corresponding aperture on the multi-aperture light shield. The area CCD detector is located at the focal length of the microlens array, and the horizontal angle between the linear gradient filter and the microlens array is β. The modulated illumination beam is reflected by the target object and received by the multispectral light intensity data acquisition module, generating a multispectral dataset matrix S = [S1, S2, ..., S...]. k ,...,S K ], where k = 1, 2, ..., K, K = A × B, A × B represents the number of rows and columns of the microlens array, S k This represents the one-dimensional light field intensity sequence acquired by the k-th microlens, which is counted row by row on the microlens array. The deep neural network includes K branches. Each branch takes a corresponding one-dimensional light field intensity sequence as input, and after processing by a fully connected network, 3×3 convolution, image reconstruction module, noise and blur kernel, and feature extractor, outputs the corresponding branch features. The branch features are then fused and input into a bottleneck layer convolution and a convolution offset generator to generate offset features. The offset features and branch features are then input into a deformable convolution and a reconstruction layer convolution to generate reconstruction features. The reconstruction features are then fused with the one-dimensional light field intensity sequence to output a super-resolution reconstructed image.

2. The underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction according to claim 1, characterized in that, The light modulation module modulates the illumination beam using Hadamard speckle.

3. The underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction according to claim 1, characterized in that, The deep neural network is trained using a training set, which includes a multispectral dataset matrix C. n The multispectral dataset matrix C n The result was obtained by scaling down the original image and randomly shifting the pixels.

4. The underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction according to claim 3, characterized in that, During the training process, the deep neural network uses the multispectral dataset matrix C n After the input is processed into a two-dimensional image by the deep neural network, m noise level images are added to the two-dimensional image. The pixel values ​​of the noise level images are randomly generated within a preset range. Gaussian noise, Poisson noise, additive white Gaussian noise, Gaussian blur, defocus blur, and motion blur are added to the noise level images.

5. The underwater multispectral ghost imaging system based on deep learning image super-resolution reconstruction according to claim 3, characterized in that, The loss calculation for the deep neural network during training is as follows: L(P)=L align +L sr ; Among them, L align L represents the network training loss. sr This indicates the network loss during SR reconstruction. For reference frame, I i LR For low-resolution LR auxiliary frames, To train and predict the corresponding aligned LR frames, For the final high-resolution HR frame estimation, t is the HR frame.

Images