Sonar video sequence denoising method and system, readable medium and underwater vehicle

Abstract

The invention discloses a sonar video sequence denoising method and system, a readable medium and an underwater vehicle, and has the advantages that weak supervised learning under a non-registered dynamic video is realized, the limitation that input and output pixel level alignment is required by a traditional method is overcome, a supervised target is allowed to have non-rigid infinitesimal displacement in a local field, and the accuracy of the noise reduction of the sonar video sequence is improved. Sonar continuous video frames containing tiny non-rigid jitter are directly used for training. And constructing a time sequence-space combined lightweight mapping mechanism: redefining a channel mixing concept in the LUT, migrating from a space domain dimension to a time sequence dimension, and simultaneously realizing denoising and inter-frame smoothing through table look-up operation on the premise of not increasing the calculation amount. In order to solve the problem of LUT quantization loss under weak supervision, an anti-noise LUT conversion method is provided, it is ensured that under the condition of low bit wide storage, the description capability of a weak sonar target can still be kept, and logarithm transformation of an image is introduced so as to adapt to multiplicative noise caused by speckle noise.

Description

Technical Field

[0001] This invention relates to the field of video sequence noise processing and underwater detection technology, specifically to a sonar video sequence denoising method and system, a readable medium, and an underwater vehicle. Background Technology

[0002] With the deepening development of marine resources, unmanned underwater vehicles (UUVs) and remotely operated vehicles (ROVs) play a core role in subsea pipeline inspection, search and rescue, and topographic mapping. In these tasks, sonar systems, as active detection devices based on acoustic imaging, can generate continuous video images in murky waters with zero visibility, serving as the "eyes" of underwater sensing systems. However, due to the physical characteristics of underwater acoustic channels, the imaging quality of sonar images is far lower than that of optical images, posing a significant challenge to subsequent visual processing tasks. Existing sonar image denoising techniques suffer from several technical problems, including heavy reliance on paired ground truth data, inability to adapt to minute dynamic video jitter, and difficulty in deploying complex denoising models in real-time on underwater edge devices, all of which urgently require improvement. Summary of the Invention

[0003] One of the technical problems to be solved by this application is to overcome the defects of the above-mentioned related technologies and provide a sonar video sequence denoising method and system, a readable storage medium, and an underwater vehicle.

[0004] The technical solution adopted by this sonar video sequence denoising method to solve the technical problem is as follows: A sonar video sequence denoising method, specifically including the following steps: S1: Logarithmic domain transformation and temporal pseudo-channel construction; acquire a continuous video frame sequence acquired by sonar, perform a logarithmic transformation on each frame of the continuous video frame sequence to convert the multiplicative speckle noise in the sonar echo into an additive noise distribution; select a specific frame of sonar image after logarithmic transformation for temporal information fusion to construct temporal pseudo-channel input data; S2: Construct a lightweight logarithmic domain denoising network; Construct a convolutional neural network that includes a Siamese hybrid module and a Γ-type convolutional feature extraction module. The convolutional neural network operates entirely in the logarithmic domain. The Siamese hybrid module is used to extract inter-frame temporal correlation features, and the Γ-type convolutional feature extraction module is used to expand the spatial receptive field while maintaining a low number of parameters. S3: Logarithmic domain anti-jitter weak supervision training; train the convolutional neural network, set the input data and supervision target, and introduce a loss function that allows the supervision target to have non-rigid small displacements in the local domain; S4: Quantization perception and LUT transformation; After training convergence, the convolution operation and activation function in the convolutional neural network are discretized and mapped to convert the network weights into a multidimensional lookup table; S5: Mask-constrained inference and exponential restoration; The sonar video stream to be processed is logarithmically transformed and then input into a lookup table. The lookup table results are weighted and corrected using a distance-angle mask to obtain the inference results. An inverse exponential transformation is performed on the inference results to restore the original intensity distribution of the sonar image, and the final linear sonar video is output.

[0005] Preferably, the specific frame sonar image in step S1 is an odd-numbered frame sequence sonar image consisting of the current frame after logarithmic transformation and its adjacent preceding and following frames.

[0006] Preferably, the odd-numbered frame sequence of sonar images consists of 3, 5, or 7 consecutive sonar frames. Preferably, the timing information fusion method in step S1 is channel stacking or frame difference preprocessing; the input data for channel stacking is a frame sequence using the logarithmic domain ( L t-1 , L t ) to form forward timing pairs, or using ( L t , L t+1 The forward or backward time sequence pairs are composed of a forward time sequence pair and a backward time sequence pair. The input data for frame difference preprocessing is the difference image between the current frame and the adjacent frame. The twin hybrid module performs difference and fusion feature extraction on the forward or backward time sequence pairs in the logarithmic domain, and suppresses the multiplicative noise component that changes with time through homomorphic filtering, that is, stitching the time dimension frames in the channel dimension.

[0007] Preferably, the loss function in step S3 is the neighborhood tolerance loss function, the Charbonnier loss function, or the Huber loss function. This loss function enables the network to learn to remove additive speckle noise while ignoring high-frequency positional errors caused by minute inter-frame jitter.

[0008] Preferably, the Γ-shaped convolution feature extraction module in step S2 uses a non-square Γ-shaped convolution kernel structure. When converted to a lookup table, the output of the Γ-shaped convolution kernel is mapped to a 3D LUT lookup table. The large receptive field characteristic of the Γ-shaped convolution kernel can more effectively separate the low-frequency target echo structure from the high-frequency speckle noise component in the logarithmic domain.

[0009] Preferably, step S4 includes a simulated quantization injection process: during the training process in step S3, a pseudo-quantization noise layer is inserted after the activation layer of the convolutional neural network to map continuous logarithmic domain feature values ​​to discrete interval values ​​corresponding to the LUT index, forcing the convolutional neural network to adapt to the discretization error of the lookup table.

[0010] The technical solution adopted by this sonar video sequence denoising system to solve the technical problem is as follows: a sonar video sequence denoising system based on logarithmic transform, comprising: The preprocessing module performs a logarithmic transformation on the input raw sonar video stream, converting multiplicative noise into additive noise and constructing a temporal pseudo-channel. The lookup table storage module is used to store the converted logarithmic field lightweight LUT file; The fast inference module is used to address and read the lookup table based on the logarithmic field input data, and perform mask weighting to obtain the inference result; The post-processing module is used to perform an inverse exponential transform on the inference results and output the final linear sonar video.

[0011] Compared with related technologies, the advantages of this sonar video sequence denoising method and system are as follows: First, it enables weakly supervised learning in unregistered dynamic videos: overcoming the limitation of traditional Noise2Noise methods that require pixel-level alignment of input and output, it introduces a loss function that allows the supervised target to have non-rigid small displacements in the local neighborhood, so that the model can be trained directly using continuous sonar video frames containing small non-rigid jitters without the need for complex motion compensation or optical flow alignment in advance.

[0012] Second, a lightweight mapping mechanism combining temporal and spatial elements is constructed: breaking through the limitation of existing LUT technology that only processes single-frame spatial information, the concept of channel mixing in LUT is redefined and migrated from the spatial dimension to the temporal dimension, so as to achieve denoising and inter-frame smoothing simultaneously through table lookup operations without increasing the amount of computation.

[0013] Third, address the LUT quantization loss problem under weak supervision: To address the issue that the pseudo-true values ​​generated by Noise2Noise contain residual noise, directly converting them to discrete LUTs can lead to error amplification, a noise-resistant LUT conversion method is provided. This ensures that the ability to describe weak sonar targets can still be preserved under low bit-width storage conditions. At the same time, a logarithmic transformation of the image is introduced to adapt to the multiplicative noise caused by speckle noise.

[0014] The technical solution adopted by this readable medium to solve the technical problem is as follows: a readable medium storing a computer program, which is executed by the sonar video sequence denoising system to implement the sonar video sequence denoising method.

[0015] The technical solution adopted by this underwater vehicle to solve the technical problem is as follows: an underwater vehicle, including a sonar system and an onboard computer, wherein the onboard computer is equipped with the sonar video sequence denoising system, which is used to enhance the sonar video collected by the sonar system in real time during underwater operations. Attached Figure Description

[0016] Figure 1 This is a flowchart of Embodiment 1 of the present invention.

[0017] Figure 2 This is a schematic diagram of the anti-jitter weak supervision training principle of the present invention.

[0018] Figure 3 This is a diagram of the mapping mechanism of the present invention.

[0019] Figure 4 This is a model diagram of the twin hybrid module of the present invention. Detailed Implementation

[0020] First, those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the embodiments of this application and are not intended to limit the scope of protection of the embodiments of this application. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0021] The sonar video sequence denoising method based on logarithmic transform includes the following steps: S1: Logarithmic domain transformation and temporal pseudo-channel construction; acquire a continuous video frame sequence acquired by sonar, perform a logarithmic transformation on each frame of the continuous video frame sequence to convert the multiplicative speckle noise in the sonar echo into an additive noise distribution; select a specific frame of sonar image after logarithmic transformation for temporal information fusion to construct temporal pseudo-channel input data; S2: Construct a lightweight logarithmic domain denoising network; Construct a convolutional neural network that includes a Siamese hybrid module and a Γ-type convolutional feature extraction module. The convolutional neural network operates entirely in the logarithmic domain. The Siamese hybrid module is used to extract inter-frame temporal correlation features, and the Γ-type convolutional feature extraction module is used to expand the spatial receptive field while maintaining a low number of parameters to cover sonar sidelobe interference. S3: Logarithmic domain jitter-resistant weakly supervised training; training the convolutional neural network, setting input data and supervision targets, and introducing a loss function that allows the supervision targets to have non-rigid small displacements in the local neighborhood; S4: Quantization perception and LUT transformation; After training convergence, the convolution operation and activation function in the convolutional neural network are discretized and mapped to convert the network weights into a multidimensional lookup table; S5: Mask-constrained inference and exponential restoration; The sonar video stream to be processed is logarithmically transformed and then input into a lookup table. The lookup table results are weighted and corrected using a distance-angle mask to obtain the inference results. An inverse exponential transformation is performed on the inference results to restore the original intensity distribution of the sonar image, and the final linear sonar video is output.

[0022] This sonar video sequence denoising method is applicable to scenarios involving speckle noise removal and requiring stringent end-side computing power, such as image denoising for forward-looking sonar (FLS), synthetic aperture sonar, and side-scan sonar. The invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Example 1 The sonar system in this embodiment employs forward-looking sonar. Forward-looking sonar images not only contain additive noise with a Gaussian distribution, but also are filled with large-scale multiplicative speckle noise and severe side-lobe interference caused by beamforming mechanisms. Side-lobe leakage creates false arc-shaped artifacts in the image, severely obscuring target texture. Traditional digital image processing methods (such as median filtering, nonlocal mean filtering (NLM), BM3D, etc.) often suffer from two shortcomings when dealing with such non-uniformly distributed strong noise: either insufficient denoising, leaving a large number of speckles; or excessive smoothing, causing the target edges formed by acoustic highlights to be blurred, resulting in the loss of crucial detail information.

[0024] A weakly supervised denoising method for forward-looking sonar video sequences based on logarithmic transform, such as Figure 1 As shown, the specific steps include: Logarithmic domain transformation and construction of temporal pseudochannels: To acquire a continuous video frame sequence of forward-looking sonar, a logarithmic transformation is first performed on each frame of the sonar image to convert the multiplicative speckle noise in the sonar echo into an additive noise distribution. Then, the current frame I after the logarithmic transformation is selected. t and its adjacent preceding and following frames I t-1 I t+1 Stacking them along the channel dimension creates time-series pseudo-channel input data, such as... Figure 1 Step S1' is shown; The logarithmic transformation is used to overcome the interference of multiplicative speckle noise in forward-looking sonar imaging on the convolution operation. The specific transformation formula is as follows: ; I(x,y) For the original sonar image in coordinates (x,y) The acoustic intensity value at that location, ε To prevent the small bias constant with a logarithm of negative infinity, c This is the dynamic range scaling factor. L(x,y) These are the logarithmic domain pixel values ​​after logarithmic transformation; The construction of the temporal pseudochannel specifically involves: utilizing the frame sequence in the logarithmic field (L... t-1 , L t ) form forward timing pairs, using (L t , L t+1 Form backward timing pairs, and use either forward timing pairs or backward timing pairs as input data.

[0025] This embodiment uses three frames (t-1, t, t+1) to construct the time-series input data. In practice, the number of input frames is not limited to 3 frames; it can also be extended to more odd-numbered frame sequences such as 5 or 7 frames, or a recursive input method can be used, using the denoising result of the previous time step as the auxiliary input of the current time step.

[0026] In this embodiment, the temporal information fusion method is channel stacking. Frame difference preprocessing can also be used, with the difference image between the current frame and adjacent frames used as input data for a lightweight logarithmic domain denoising network to enhance sensitivity to moving targets.

[0027] Constructing a lightweight logarithmic domain denoising network: A convolutional neural network based on the SRNet backbone is constructed, comprising a Twin Hybrid Module (THM) and a Gamma-type convolutional feature extraction module. The convolutional neural network operates entirely in the logarithmic domain. The THM module is used to extract inter-frame temporal correlation features, and it performs difference and fusion feature extraction on forward or backward temporal pairs in the logarithmic domain, such as... Figure 4 As shown, the suppression of time-varying multiplicative noise components is essentially achieved through homomorphic filtering, which involves stitching time-dimension frames together in the channel dimension. The Γ-shaped convolution feature extraction module is used to expand the spatial receptive field while maintaining a low parameter count, in order to cover sonar sidelobe interference. The Γ-shaped convolution feature extraction module is specifically configured as follows: it adopts a non-square Γ-shaped convolution kernel structure, and the sampling points of the Γ-shaped convolution kernel after rotation of 0°, 90°, 180°, and 270° do not overlap. When converting to a lookup table, the output of the Γ-shaped convolution kernel is mapped to a 3D LUT lookup table. The large receptive field characteristic of the Γ-shaped convolution kernel can more effectively separate the low-frequency target echo structure from the high-frequency speckle noise component in the logarithmic domain.

[0028] However, due to the limitations of computing power on edge devices and the inadequacy of existing lightweight technologies, underwater robots typically use embedded chips for power supply, resulting in extremely limited computing power and power consumption budgets. Existing high-performance denoising networks have extremely high computational complexity, containing millions of parameters and a huge amount of floating-point operations (FLOPs), making it impossible to achieve real-time operation on underwater edge devices.

[0029] Logarithmic domain jitter-resistant weakly supervised training: Adopting such Figure 2 The improved Noise2Noise strategy shown is used to train the convolutional neural network, with logarithmically transformed adjacent video frames as input data and supervision targets. A Neighborhood Tolerance Loss function is introduced to allow non-rigid small displacements of the supervision targets in the local neighborhood, in order to adapt to the jitter characteristics of the underwater sonar platform. Traditional Noise2Noise methods place extremely high demands on data alignment, typically assuming the scene is static or perfectly registered at the pixel level. During forward-looking sonar video acquisition, the impact of water flow and the movement of the carrier itself inevitably introduces minute, non-rigid jitter between adjacent frames. Due to the weak texture of sonar images, traditional registration algorithms are prone to failure. Directly using imperfectly aligned frames for training with traditional Noise2Noise methods results in noticeable ghosting and loss of high-frequency details in the denoised image. Furthermore, when existing CNN models are directly transferred to the sonar domain, they often struggle to adapt to the characteristics of large speckle patterns and sidelobe artifacts, leading to severe domain drift.

[0030] The neighborhood tolerance loss function (also known as the neighboring tolerance loss function) is defined in the logarithmic space, and its specific calculation formula is as follows: L in For the input logarithmic frame, L （target，q） For adjacent logarithmic frames serving as the monitoring signal, Ω( p (in pixels) p Centered k×k Tolerance neighborhood (also known as tolerance region); the tolerance loss function in this neighborhood enables the convolutional neural network to learn to remove additive speckle noise while ignoring high-frequency positional errors caused by small inter-frame jitter.

[0031] The improved Noise2Noise strategy is a Noise2Noise strategy based on adjacent frames, and can also be combined with self-supervised training strategies such as Noise2Void or Noise2Self. For example, in extreme cases where it is impossible to obtain continuous stable video frames, training can be performed by masking the center pixel of the current frame and using surrounding pixels for prediction. Although the convergence speed may be slower, it still falls into the category of training without ground truth.

[0032] Here, the domain tolerance loss function can be replaced with other loss functions that are jitter-resistant or robust. For example, the Charbonnier loss function or Huber loss function can be used to reduce sensitivity to outliers (maximum sidelobes); or a lightweight optical flow estimation or block matching step can be introduced before calculating the loss. L （target，q） Perform a rough alignment.

[0033] Quantization perception and LUT transformation: After training convergence, the convolution operations and activation functions in the convolutional neural network are discretized and mapped, based on, for example... Figure 3 The mapping mechanism shown converts network weights into a multidimensional lookup table (LUT). The simulation quantization injection process is included: During the training process of the logarithmic domain anti-jitter weak supervision training, a pseudo-quantization noise layer is inserted after the activation layer of the convolutional neural network to map the continuous logarithmic domain feature values ​​to the discrete interval values ​​corresponding to the LUT index, forcing the convolutional neural network to adapt to the discretization error of the lookup table.

[0034] Mask-constrained reasoning and exponential restoration: The sonar video stream to be processed is logarithmically transformed and then input into a lookup table. A distance-angle mask is used to weight and correct the lookup results, such as... Figure 1 Step S5' is shown; finally, perform an inverse exponential transform on the output result, as shown. Figure 1 As shown in S5'', the original intensity distribution of the sonar image is restored and a linear sonar video is output; In the weighted correction, the sector mask constraint of the range-angle mask includes: generating a sector mask matrix M based on the physical beam characteristics of the forward-looking sonar. r,θ ); Because sidelobe interference exhibits specific additive texture features after logarithmic transformation, the fan-shaped mask matrix M( r,θ It is designed to suppress background noise of the LUT output with stronger weights in the far-field region, while preserving the original output of the LUT in the near-field target region; The formula for the inverse exponential transform is: .

[0035] Existing LUT techniques face a dilemma when processing complex images. First, there's the separation of spatial and channel processing: to control lookup table size, existing methods, such as SR-LUT and MuLUT, typically separate spatial convolution and channel processing. For example, using only a spatial LUT to process single-channel data ignores multi-channel information, including the correlation between channels in forward-looking sonar multibeam data. When processing high dynamic range sonar data, existing LUT techniques struggle to simultaneously preserve high-echo-intensity targets and low-intensity background textures due to limited quantization intervals, leading to dynamic range compression losses. Research shows that channel correlation is crucial for modeling noise distribution; ignoring this significantly reduces denoising effectiveness. Second, there's the conflict between receptive field and storage capacity: expanding the receptive field to handle large speckle noise results in an exponential increase in LUT storage size. For example, a simple 2×2 convolution kernel, if its channel depth is expanded, will reach terabyte-level LUT size, making deployment impossible. Even with the use of cascaded LUTs, storage efficiency is often low due to repeated indexing, or the sidelobe interference that is widespread in FLS images cannot be removed due to insufficient receptive field.

[0036] Here, the range-angle mask can be replaced by a partition-based multi-LUT strategy. Instead of using a single LUT in conjunction with the mask, three different LUT files are trained and stored separately for the near-field, mid-field, and far-field of the forward-looking sonar. During inference, the corresponding LUT file is directly called based on the pixel coordinates. This approach sacrifices a small amount of storage space but eliminates mask multiplication operations, further improving inference speed.

[0037] A sonar video sequence denoising system implementing a weakly supervised denoising method based on logarithmic transform for forward-looking sonar video sequences specifically includes: The preprocessing module performs a logarithmic transformation on the input raw sonar video stream, converting multiplicative noise into additive noise and constructing a temporal pseudo-channel. The lookup table storage module is used to store the logarithmic field lightweight LUT file obtained by step quantization sensing and LUT transformation; The fast inference module is used to address and read the lookup table based on the logarithmic field input data and perform mask weighting. The post-processing module is used to perform an inverse exponential transform on the inference results and output the final linear sonar video.

[0038] A readable medium storing a computer program that is executed by a sonar video sequence denoising system to implement the aforementioned sonar video sequence denoising method.

[0039] An underwater vehicle includes a sonar system and an onboard computer, wherein the onboard computer is equipped with the sonar video sequence denoising system described above, which is used to enhance the sonar video acquired by the sonar system in real time during underwater operations.

[0040] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for denoising sonar video sequences, characterized in that, Specifically, the steps include the following: S1: Logarithmic domain transformation and temporal pseudo-channel construction; acquire a continuous video frame sequence acquired by sonar, perform a logarithmic transformation on each frame of the continuous video frame sequence to convert the multiplicative speckle noise in the sonar echo into an additive noise distribution; A specific frame of sonar image after logarithmic transformation is selected for temporal information fusion to construct temporal pseudo-channel input data; S2: Construct a lightweight logarithmic domain denoising network; Construct a convolutional neural network that includes a Siamese hybrid module and a Γ-type convolutional feature extraction module. The convolutional neural network operates entirely in the logarithmic domain. The Siamese hybrid module is used to extract inter-frame temporal correlation features, and the Γ-type convolutional feature extraction module is used to expand the spatial receptive field while maintaining a low number of parameters. S3: Logarithmic domain anti-jitter weak supervision training; train the convolutional neural network, set the input data and supervision target, and introduce a loss function that allows the supervision target to have non-rigid small displacements in the local domain; S4: Quantization perception and LUT transformation; After training convergence, the convolution operation and activation function in the convolutional neural network are discretized and mapped to convert the network weights into a multidimensional lookup table; S5: Mask-constrained inference and exponential restoration; The sonar video stream to be processed is logarithmically transformed and then input into a lookup table. The lookup table results are weighted and corrected using a distance-angle mask to obtain the inference results. An inverse exponential transformation is performed on the inference results to restore the original intensity distribution of the sonar image, and the final linear sonar video is output.

2. The sonar video sequence denoising method according to claim 1, characterized in that: The specific frame sonar image mentioned in step S1 is an odd-numbered frame sequence sonar image consisting of the current frame after logarithmic transformation and its adjacent preceding and following frames.

3. The sonar video sequence denoising method according to claim 2, characterized in that: The odd-numbered frame sequence of sonar images consists of 3, 5, or 7 consecutive frames of sonar images.

4. The sonar video sequence denoising method according to claim 1, characterized in that: The timing information fusion method described in step S1 is channel stacking or frame difference preprocessing; the input data for channel stacking is a frame sequence using the logarithmic domain ( L t-1 , L t ) to form forward timing pairs, or using ( L t , L t+1 The following are the backward time pairs; the input data for frame difference preprocessing is the difference image between the current frame and the adjacent frame.

5. The sonar video sequence denoising method according to claim 4, characterized in that: The loss function in step S3 is either the domain tolerance loss function, the Charbonnier loss function, or the Huber loss function.

6. The sonar video sequence denoising method according to claim 1, characterized in that: The Γ-type convolution feature extraction module in step S2 adopts a non-square Γ-type convolution kernel structure. When converted into a lookup table, the output of the Γ-type convolution kernel is mapped to a 3D LUT lookup table.

7. The sonar video sequence denoising method according to claim 6, characterized in that: Step S4 includes a simulated quantization injection process: during the training process in step S3, a pseudo-quantization noise layer is inserted after the activation layer of the convolutional neural network to map continuous logarithmic domain feature values ​​to discrete interval values ​​corresponding to LUT indices.

8. A sonar video sequence denoising system, characterized in that, include: The preprocessing module performs a logarithmic transformation on the input raw sonar video stream, converting multiplicative noise into additive noise and constructing a temporal pseudo-channel. The lookup table storage module is used to store the converted logarithmic field lightweight LUT file; The fast inference module is used to address and read the lookup table based on the logarithmic field input data, and perform mask weighting to obtain the inference result; The post-processing module is used to perform an inverse exponential transform on the inference results and output the final linear sonar video.

9. A readable medium storing a computer program, characterized in that: The computer program is executed by the sonar video sequence denoising system of claim 8 to implement the sonar video sequence denoising method of any one of claims 1-7.

10. An underwater vehicle, comprising a sonar system and an onboard computer, characterized in that: The airborne computer is equipped with the sonar video sequence denoising system as described in claim 8, which is used to enhance the sonar video acquired by the sonar system in real time during underwater operations.

Images