Multi-beam sonar imaging system and method based on adjacent frame complementary sparse sampling

Abstract

The application belongs to the technical field of underwater acoustic detection and sonar imaging, and discloses a multi-beam sonar imaging system and method based on adjacent frame complementary sparse sampling, which comprises a transducer array, 2N / 3 analog receiving channels, an analog gating network, a beam forming imager, a control unit and a multi-frame fusion reconstruction network. The control unit is used to make adjacent frames adopt different and complementary sparse array patterns, and only 2N / 3 array elements are enabled to receive, so that the hardware cost and power consumption are reduced. Meanwhile, the initial images of the current frame and adjacent frames and corresponding sparse array pattern identifiers are jointly input into the U-net-based multi-frame fusion reconstruction network, the current frame is enhanced and reconstructed by using the complementary observation information between adjacent frames, and a high-quality image close to the full-array receiving condition is output. The application effectively restores the image quality degradation caused by sparse sampling under the premise of significantly reducing the number of analog channels.

Description

Technical Field

[0001] This invention belongs to the field of underwater acoustic detection and sonar imaging technology, and particularly relates to a multi-beam sonar imaging system and method based on complementary sparse sampling of adjacent frames. Background Technology

[0002] Multibeam sonar receives echo signals through array transducers and uses beamforming to obtain the spatial echo intensity distribution along different azimuth angles, thereby forming underwater target images or terrain imaging results. To obtain high angular resolution and imaging quality, traditional multibeam sonar typically requires a large number of array elements to operate simultaneously. However, as the number of array elements increases, the number of hardware channels, data acquisition bandwidth, power consumption, size, and cost of the system also increase. Therefore, how to maintain imaging performance while reducing the number of effective working array elements has become an important research direction in the field of multibeam sonar.

[0003] The closest prior art to this invention is the sparse array design method for multi-beam imaging sonar. For example, patent CN105842702A, "Sidelobe Suppression Method and Array Sparsity Method for Sparse Arrays of Multi-beam Imaging Sonar," proposes: given the main lobe and sidelobe performance requirements, a sparse model of the multi-beam imaging sonar array is established. By optimizing the array element positions and weighting coefficients, the number of array elements is reduced while suppressing sidelobes. This patent explicitly states that its purpose is to solve the problem of "difficulty in sparse multi-beam arrays and lack of effective sparse methods in existing technologies." In addition, patent CN109959917A, "An Array Sparsity Method for Broadband Non-Frequency-Variable Multi-beam Imaging Sonar," further addresses broadband, non-frequency-variable multi-beamforming scenarios, utilizing sparse reconstruction and convex optimization methods to reduce the number of effective array elements and lower the peak sidelobe level while ensuring the main lobe width of the multi-beam is basically stable. The patent citation also lists W.Xia, X.Jin, F.Dou, "Thinned Array Design With Minimum Number of Transducers for Multibeam Imaging Sonar," IEEE Journal of Oceanic Engineering, 2017, indicating that the sparse design of "achieving multibeam imaging with the fewest transducers" is an existing research approach in this field.

[0004] The basic principle of the aforementioned sparse array technique lies in the pre-design or optimization of the element position distribution and element weighting coefficients, so that even when some elements are not working, the array pattern still approximates the pattern of a full array as closely as possible, thereby maintaining the main lobe width, controlling the side lobe level, and achieving multi-beam imaging. The advantage of this method is that it can directly reduce the number of elements or the number of effective working channels, which has significant engineering implications.

[0005] However, this type of method still has obvious shortcomings:

[0006] Firstly, existing sparse array methods mainly focus on optimizing the performance of a single imaging or single-frame beammap, with emphasis on main lobe width, side lobe level, and array structure design, while failing to fully utilize the information complementarity between consecutive pings.

[0007] Secondly, even if the angular resolution can be maintained to some extent through array optimization after the number of array elements is reduced, single-frame imaging results are still prone to problems such as side lobe enhancement, raster lobe artifacts, texture discontinuities, more obvious speckle noise, and image detail degradation, resulting in a decline in the final image quality.

[0008] Third, the above methods usually assume that a certain fixed sparse array structure works for a long time, and have not yet provided a solution for the technical route of "using different sparse styles for different pings and using their cross-frame complementarity for joint recovery".

[0009] A second type of prior art, similar to this invention, involves techniques for enhancing sonar or intensity images using neural networks or generative adversarial networks (GANs). For example, patent US11847764B2, "Neural network based enhancement of intensity images," proposes constructing a GAN containing a generator and a discriminator to enhance intensity images formed by measuring the echo intensity of an energy pulse illuminating a target. Its loss function can consist of mean squared error, perceptual loss, and adversarial loss, and its "clean image" can be formed by combining a set of images of the same target. This indicates that using deep learning, especially GANs, to improve the image quality of actively perceived echo intensity images is already a publicly available technical approach.

[0010] The basic principle of this type of method is usually as follows: input the degraded image into a neural network, and through supervised learning or adversarial learning, make the network output an enhanced image with lower noise, higher contrast, and clearer edges. Its advantage lies in that it can directly learn complex degradation and restoration relationships from the image domain, and has a certain improvement effect on problems such as random noise, insufficient contrast, and local blurring.

[0011] However, this type of method also has its shortcomings:

[0012] First, most existing network augmentation methods treat the input as a generally degraded image and do not explicitly incorporate the sparse sampling mechanism of multibeam sonar arrays into the model. Therefore, it is difficult to perform targeted restoration of specific artifacts caused by "array element reduction, sparse aperture sampling, and different array activation modes".

[0013] Secondly, existing methods typically focus on single-frame image enhancement rather than building a recovery model around the naturally occurring cross-frame complementary observations during multi-beam sonar continuous ping imaging.

[0014] Third, if it is difficult to obtain the "high-quality reference images" required for training, the network's generalization ability, experimental adaptability, and engineering feasibility will be limited.

[0015] A third type of prior art, similar to this invention, utilizes multi-frame fusion enhancement methods based on adjacent sonar frames. For example, the 2025 paper "Reference-Free Enhancement of Forward-Looking SonarImages … and Multi-Frame Fusion" proposes a reference-free forward-looking sonar image enhancement framework that explicitly utilizes complementary information between consecutive sonar frames to suppress speckle noise, enhance target brightness, and improve edge details. This paper points out that its multi-frame fusion strategy improves sonar image quality without relying on external high-quality ground truth images by utilizing complementary information from adjacent frames.

[0016] The basic principle of this type of multi-frame fusion technology is that when the carrier platform moves continuously and the target and background are repeatedly observed at adjacent time points, adjacent frames often contain some redundant information and some complementary information. By extracting features, registering or fusing consecutive frames, random noise can be suppressed to a certain extent, target visibility can be improved and image continuity can be enhanced.

[0017] However, this type of technology is still mainly aimed at multi-frame denoising and enhancement under natural continuous sampling conditions, and it also has the following shortcomings:

[0018] Firstly, this is usually based on the premise that each frame comes from the same or nearly the same imaging hardware aperture conditions, rather than addressing the heterogeneous observation problem caused by "using different sparse array patterns for each frame".

[0019] Secondly, these methods focus on improving image quality by utilizing temporal continuity, without taking array-side information such as array activation mode, sparse aperture mask, and array element activation ratio as recovery constraints.

[0020] Third, existing multi-frame fusion methods are more used to reduce speckle noise, improve brightness and edge sharpness, rather than to serve the system goal of "restoring near-full array imaging effect with only some array elements enabled, thereby reducing hardware channels and power consumption".

[0021] In existing multibeam sonar technology, in order to ensure angular resolution and imaging quality, all array elements and corresponding analog receiving channels usually need to work simultaneously. This results in a large number of front-end analog channels, high hardware costs, high power consumption, large system size, and high pressure on data acquisition and processing, which is not conducive to the realization of low-cost, miniaturized and highly integrated systems.

[0022] On the other hand, while existing sparse array methods can reduce the number of array elements or the number of effective working channels, most of them only optimize the array structure or beammap for a single frame, focusing mainly on indicators such as main lobe width and side lobe level. Therefore, when only some array elements are used, although the resolution of a single frame can be maintained to a certain extent, the image usually still suffers from problems such as enhanced side lobe artifacts, loss of texture details, blurred target edges, increased speckle noise, and deterioration of image continuity, resulting in a decrease in overall imaging quality.

[0023] Furthermore, most existing neural network-based image enhancement methods are geared towards single-frame degraded image restoration and typically do not incorporate the sparse sampling mechanism of multi-beam sonar arrays. Because they lack a dedicated design for complementary sparse patterns between adjacent frames, they also fail to incorporate the complementary relationships between adjacent sparse observations and their temporal correlation features into a unified model.

[0024] In summary, while existing technologies have made some progress in multi-beam sonar sparse array design, neural network image enhancement, and multi-frame fusion of adjacent frames, a joint technical solution for the continuous operation of multi-beam sonar is still lacking. Specifically, this solution should fully utilize the complementary observation information provided by different sparse apertures between adjacent frames, under the condition that only a portion of the array elements are used in each frame and adjacent frames employ different complementary sparse array patterns, to achieve near-full-element image quality through intelligent reconstruction. Current technologies cannot simultaneously address the requirements of element number compression, image quality restoration, cross-frame complementary utilization, and low-power engineering implementation. Summary of the Invention

[0025] The purpose of this invention is to provide a multi-beam sonar imaging system and method based on complementary sparse sampling of adjacent frames, so as to solve the above-mentioned technical problems.

[0026] To address the aforementioned technical problems, this invention constructs the complementarity of sampling between adjacent frames by employing different array sparsity patterns, and reconstructs sonar images by using a generative network to learn target motion correlation and the complementarity of adjacent frames. The specific technical solution is as follows:

[0027] A multibeam sonar imaging system based on complementary sparse sampling of adjacent frames includes:

[0028] A transducer array consisting of N array elements;

[0029] 2N / 3 analog receiving channels, where N is an integer multiple of 3;

[0030] An analog gating network is connected between the transducer array and the analog receiving channel;

[0031] A beamforming imager is connected after the analog receiving channel;

[0032] A control unit is used to control the analog gating network so that adjacent frames adopt different sparse array patterns with complementary relationships, and select the corresponding 2N / 3 array elements to 2N / 3 analog receiving channels;

[0033] A multi-frame fusion reconstruction network based on U-net is used to take the initial images of the current frame and its neighboring frames, as well as the sparse array pattern identifiers corresponding to each frame, as inputs and output the reconstructed image of the current frame.

[0034] The complementary relationship between adjacent frames is that the union of the sparse array patterns of any two adjacent frames covers all N array elements.

[0035] Furthermore, the sparse array style includes three templates:

[0036] Template A: Select the first two elements from every three consecutive array elements;

[0037] Template B: Select the first and last two array elements from every three consecutive array elements;

[0038] Template C: Select the last two elements from every three consecutive array elements;

[0039] The control unit causes adjacent frames to cycle through in the order of template A, template B, and template C.

[0040] Furthermore, the U-net-based multi-frame fusion reconstruction network includes:

[0041] The encoding end is used to extract multi-scale features from the initial image of the previous frame, the initial image of the current frame, and the initial image of the next frame, respectively. The encoding branch structures of the three frames are the same and share parameters.

[0042] The bottleneck fusion end is used to concatenate and fuse the deep features extracted from the three frames by the encoder with the sparse array pattern identifiers corresponding to each frame, and output the joint deep features.

[0043] The decoding end is used to perform step-by-step upsampling and feature recovery on the joint deep features and output the reconstructed image of the current frame;

[0044] The decoding end introduces multi-scale features retained in the encoding stage through a bridging structure. The bridging structure fuses the encoded features of three adjacent frames at corresponding scales and then inputs them into the corresponding layer of the decoder.

[0045] Furthermore, the sparse array pattern identifier is not directly input into the encoder, but is concatenated with deep features at the bottleneck layer in the form of one-hot encoding or embedding encoding before being input.

[0046] Furthermore, the multi-frame fusion reconstruction network based on U-net is trained using a supervised learning method. The training samples are formed by extracting 2N / 3 channel data from continuous raw echo data under full-channel reception conditions according to a preset sparse array pattern template to form a sparse sampling initial image, and the corresponding full-channel current frame image is used as the supervision label.

[0047] This invention also discloses a multi-beam sonar imaging method based on complementary sparse sampling of adjacent frames, applied to the aforementioned system, comprising the following steps:

[0048] Step 1: The control unit controls the analog gating network to make adjacent frames adopt different sparse array patterns with complementary relationships, and sequentially selects 2N / 3 array elements from N array elements to access 2N / 3 analog receiving channels;

[0049] Step 2: The analog receiving channel amplifies, filters, and performs analog-to-digital conversion on the received signal to obtain echo data under sparse sampling;

[0050] Step 3: The beamforming imager performs beamforming on the echo data to generate an initial image of the current frame and its adjacent frames;

[0051] Step 4: Input the initial images of the current frame and its adjacent frames, as well as the sparse array pattern identifiers corresponding to each frame, into the pre-trained multi-frame fusion reconstruction network based on U-net.

[0052] Step 5: The multi-frame fusion reconstruction network based on U-net outputs the reconstructed image of the current frame as the final imaging result of the current frame.

[0053] Furthermore, the processing procedure of the U-net-based multi-frame fusion reconstruction network described in step 4 includes:

[0054] Encoding stage: Multi-scale feature extraction is performed on the initial images of the previous frame, the current frame, and the next frame respectively. The three frames use encoding branches with the same structure and shared parameters.

[0055] Bottleneck fusion stage: The deep features extracted from the three frames by the encoder are spliced ​​and fused with the sparse array pattern identifiers corresponding to each frame to form joint deep features;

[0056] Decoding stage: The joint deep features are upsampled step by step, and the multi-scale features retained in the encoding stage are introduced through the bridging structure, and finally the reconstructed image of the current frame is output;

[0057] The bridging structure fuses the encoded features of three adjacent frames at corresponding scales and then inputs them into the corresponding layer of the decoder.

[0058] Furthermore, the U-net-based multi-frame fusion reconstruction network is trained using the following loss function:

[0059] The loss function comprises a weighted sum of reconstruction loss, structural constraint loss, and regularization loss;

[0060] The reconstruction loss is used to constrain the pixel differences between the network output image and the current frame label across all channels.

[0061] The structural constraint loss is used to improve the consistency of the reconstructed image with the current frame label across all channels in terms of edges, texture, and local structure;

[0062] The regularization loss is used to suppress abnormal high-frequency noise and artifacts in the network output.

[0063] Furthermore, the reconstruction loss is either the mean absolute error loss or the mean square error loss;

[0064] The structural constraint loss adopts either gradient consistency loss or structural similarity loss;

[0065] The regularization loss employs either a weight decay term or a smoothing constraint term.

[0066] The multi-beam sonar imaging system and method based on complementary sparse sampling of adjacent frames of the present invention have the following advantages:

[0067] 1. Reduced hardware cost and power consumption: This invention retains N array elements but uses only 2N / 3 analog receiving channels and achieves dynamic switching of different sparse array patterns through an analog gating network, thereby reducing the number of front-end analog links and lowering system hardware cost, power consumption and implementation complexity.

[0068] 2. Maintaining high imaging quality while reducing the number of channels: This invention does not simply use a fixed sparse array, but instead uses a sparse array pattern with complementary relationships between adjacent frames, and combines a multi-frame fusion reconstruction method to enhance the current frame, effectively compensating for the image quality degradation caused by sparse sampling, so that the system can still obtain imaging effects close to those of full-channel reception under limited channel conditions.

[0069] 3. It can make full use of the complementary information between adjacent frames: The present invention inputs adjacent frame images and their corresponding sparse array patterns into the reconstruction network, so that the network can learn the inter-frame complementary relationship under different pattern combinations. Compared with existing methods that only rely on single frame information, it can more effectively recover the details, edges and local structures in the current frame.

[0070] 4. Improve the targeting and effectiveness of image reconstruction: This invention introduces sparse array pattern identifiers as conditional information into the network, enabling the network to not only utilize the image content itself, but also combine array gating modes for reconstruction, thereby enhancing the modeling ability of sparse sampling degradation characteristics and improving the accuracy and stability of image restoration.

[0071] 5. Applicable to low-cost, miniaturized and highly integrated multibeam sonar systems: This invention balances hardware channel compression and image quality restoration, making it suitable for multibeam sonar systems with high requirements for cost, power consumption, size and integration, and has good engineering application value. Attached Figure Description

[0072] Figure 1 This is a system block diagram of the present invention.

[0073] Figure 2 This is a schematic diagram of the channel changes between adjacent frames in this invention; blue circles represent open channels, and hollow circles represent closed channels.

[0074] Figure 3 This is a schematic diagram of the multi-frame fusion reconstruction network architecture based on U-net of the present invention. Detailed Implementation

[0075] To better understand the purpose, structure, and function of this invention, the following detailed description of the multibeam sonar imaging system and method based on complementary sparse sampling of adjacent frames is provided in conjunction with the accompanying drawings.

[0076] The core idea of ​​this invention is to retain N array elements in a multi-beam sonar system, but only set up 2N / 3 analog receiving channels, and use an analog gating network to enable different frames to be received using different complementary sparse array patterns; when the resolution of a single frame can be basically guaranteed but the image quality is reduced, the complementary relationship between adjacent frame images and their sparse patterns is further utilized to restore and enhance the current frame image through an intelligent reconstruction network, thereby reducing the number of analog channels, reducing hardware costs and power consumption, while obtaining imaging quality close to that of full-element reception.

[0077] like Figure 1 As shown, the multi-beam sonar imaging system based on complementary sparse sampling of adjacent frames of the present invention includes a transducer array composed of N array elements, an analog gating network, 2N / 3 analog receiving channels, a beamforming imager, a control unit, and a multi-frame fusion reconstruction network based on U-net.

[0078] This system's transducer array retains a full set of N elements (e.g., N is a multiple of 3, such as 96, 192, or 384) to ensure the physical aperture size and potential angular resolution. However, unlike traditional schemes where each element is connected to an independent analog receiver channel, this system uses only 2N / 3 analog receiver channels. This design reduces the number of front-end analog links while maintaining the same total number of elements, thereby lowering hardware cost, power consumption, and system size.

[0079] The analog gating network is a key component for achieving dynamic sparse sampling. It is located between N array elements and 2N / 3 analog receiving channels and is controlled by the control unit. The analog gating network is used to select 2N / 3 predetermined array elements from the N array elements, based on the sparse array pattern corresponding to the current frame, and connect them to the 2N / 3 analog receiving channels. This achieves dynamic sparse reception with a limited number of channels without reducing the total number of array elements. Simultaneously, the analog gating network is also used to switch different array patterns between adjacent frames, providing a hardware foundation for subsequent image reconstruction using cross-frame complementary information. In existing ultrasound / acoustic front-ends, a common practice is to use analog multiplexers or high-voltage analog switches to dynamically connect more transducer units to a smaller number of front-end channels. This can be achieved by using analog multiplexers or programmable gating circuits, arranged between the array elements and the analog receiving channels. The control unit outputs a gating signal before the start of each frame to control the corresponding switch to close or open, establishing the required array element-channel connection relationship for the current frame.

[0080] Having the analog gating network, we also need to design the gating pattern, which is the array sparse pattern. If the entire array has N elements, and N is divisible by 3, then the three elements of the k-th group are denoted as (3k, 3k+1, 3k+2). Template A selects the first two (110), template B selects the first and last two (101), and template C selects the last two (011). During operation, the control unit causes the sparse pattern of adjacent frames to switch sequentially in a cyclical manner. The sparse pattern changes of adjacent frame channels as follows: Figure 2As shown. In frame n-1, one of every three array element channels is closed. In frame n, the previously closed channel is opened, and another channel is selected from every three channels to be closed. In frame n+1, the channel not selected in the previous two frames is selected from every three channels to be closed. For example, frame n-1 uses template A, frame n uses template B, and frame n+1 uses template C. This design ensures that: (a) the array element utilization rate of each frame is 2 / 3, achieving stable channel number compression; (b) the sparse patterns of any two adjacent frames are complementary (e.g., the union of A and B is [1,1,1]), their observation information can complement each other, achieving near full coverage of the union of any two adjacent frames; (c) the union of any three consecutive frames covers all array elements, each frame is uniformly sampled, ensuring the integrity of information and no information gaps. This invention uses different sparse array patterns; each sparse pattern has a preset complementary relationship, so that the spatial sampling information of the target area in different frames can complement each other. Therefore, this invention breaks through the existing method of relying on single-frame imaging with a single fixed sparse array, and provides a richer source of cross-frame information for subsequent image restoration.

[0081] After passing through an analog gating network, 2N / 3 analog receiving channels amplify, filter, and perform analog-to-digital conversion on the received echo signals to obtain sparsely sampled echo data. A beamforming imager receives this data and processes it using conventional beamforming algorithms (such as delay-summing beamforming). Since the input data comes from only a subset of the array elements, the initial output image, while maintaining basic angular resolution, inevitably suffers from sidelobe enhancement, raster lobe artifacts, texture discontinuities, and increased speckle noise.

[0082] To address the aforementioned problem of initial image quality degradation, this invention designs a U-net autoencoder-decoder network for multi-frame fusion. This network fully utilizes the complementary information arising from different sparse array patterns between adjacent frames to enhance and reconstruct the current frame image.

[0083] The overall design concept of this network is as follows: taking the current frame image as the reconstruction center, while introducing the previous frame image and the next frame image as auxiliary inputs, and encoding the sparse array pattern information corresponding to each frame together. In the process of feature extraction and cross-frame fusion, the network can not only utilize the temporal correlation between adjacent images, but also explicitly utilize the observation complementarity between different sparse array patterns, thereby recovering the missing information and degraded components caused by sparse sampling in the current frame image, and outputting a high-quality reconstructed image close to the full array element receiving conditions.

[0084] like Figure 3As shown, the network adopts a U-net structure, consisting of three parts: an encoder, a bottleneck fusion unit, and a decoder. The encoder is used to extract multi-scale features from multiple frames of input images; the bottleneck fusion unit is used to jointly fuse the features of the current frame, the features of adjacent frames, and the sparse array pattern features; the decoder is used to restore the image resolution step by step, and retain shallow detail information by combining skip connections, and finally outputs the reconstructed image of the current frame.

[0085] (a) Network Input Design

[0086] The network input includes: the initial image I of the current frame. n The initial image I of the previous frame n-1 The initial image I of the next frame n+1， And the sparse array pattern identifiers M1, M2, and M3 corresponding to the above frames. n I n-1 I n+1 As input to the U-net network encoder, multi-scale feature extraction is performed through coding branches with identical structures and shared parameters to obtain corresponding deep feature representations. Unlike traditional single-frame U-net networks, this invention further introduces a sparse array pattern identifier corresponding to each frame in addition to the input image, used to characterize the array element gating mode corresponding to the formation of each frame image. Preferably, the sparse array pattern identifier M... n-1 M n M n+1 Instead of being directly used as an image input encoder, it is introduced at the bottleneck layer; the style identifier is encoded or mapped to form conditional features, and then associated with I. n I n-1 I n+1 The deep features extracted by the encoder are concatenated in the bottleneck layer and then jointly fused through convolution. This allows the network to utilize information from adjacent frames and their corresponding sparse array pattern information when reconstructing the current frame. By introducing sparse array pattern identifiers in the bottleneck layer, the network can distinguish between image differences caused by scene variations and observation differences caused by sparse array pattern differences during multi-frame fusion, thus more effectively learning the complementary relationships between adjacent frames.

[0087] (b) Encoder

[0088] The encoder is used to process the initial image I of the previous frame respectively. n The initial image of the current frame I n and the initial image I of the next frame n+1 Multi-scale feature extraction is performed to obtain feature representations that characterize the local details and high-level structure information of each frame of the image, providing input for subsequent bottleneck layer fusion and decoding reconstruction.

[0089] The encoder employs a two-dimensional convolutional coding structure, which is set in the coding branches corresponding to the three input images. Each coding branch has the same structure and shares network parameters. Each coding branch includes multiple levels of cascaded convolutional extraction units and downsampling units. The convolutional extraction units are used to extract image features at the current scale, while the downsampling units are used to reduce the feature map space size and expand the receptive field, thereby gradually obtaining multi-layer feature representations at different scales.

[0090] Specifically, the input image is processed by a first-level convolutional extraction unit to obtain shallow features, which mainly represent detailed information such as edges, textures, and local echo intensity changes in the image. Subsequently, it is downsampled and enters a deeper convolutional extraction unit to form mid-level and deep features step by step. The mid-level features mainly represent the target contour and local structural information, while the deep features mainly represent high-level structural information and global context information in the image. Through multi-level convolution and downsampling processing, each frame of the image obtains a corresponding multi-scale feature set.

[0091] The encoder functions in two ways: firstly, by extracting image features at different spatial resolutions through multi-scale convolution, the network can simultaneously retain shallow detail information and deep structural information; secondly, by processing three frames of images using encoding branches with shared parameters, different frames of images are encoded in a unified feature representation space, which facilitates the subsequent fusion of features of adjacent frames and the establishment of inter-frame complementary relationships.

[0092] In each coding layer, the convolutional extraction unit may consist of one or more two-dimensional convolutional layers, nonlinear activation layers, and normalization layers; the downsampling unit may be implemented by strided convolutional layers, pooling layers, or other structures that can reduce the resolution of the feature map. More preferably, to improve feature extraction capability and training stability, the convolutional extraction unit may also employ a residual block structure.

[0093] In this invention, the encoder is only responsible for feature extraction of each input frame image and does not directly complete the modeling of inter-frame relationships and image reconstruction; the learning of complementary relationships between adjacent frames and the joint modeling with sparse array pattern information are mainly implemented end-to-end by the entire network in the subsequent bottleneck layer fusion and decoding recovery process.

[0094] (c) Bottleneck layer integration design

[0095] The bottleneck layer is positioned between the encoder and the decoder and is used to process the initial image I of the previous frame. n-1 The initial image of the current frame I n and the initial image I of the next frame n+1 The deep features extracted by the encoder are fused together, and the sparse array pattern identifier M corresponding to each frame is used. n-1 M n Mn+1 A fusion process is introduced to form a joint deep feature representation for image reconstruction of the current frame.

[0096] Specifically, let F be the features output by the encoder at the deepest layer of the three frames. n-1 (L), F n (L) and F n+1 (L), where L represents the deepest level of the encoder. The sparse array pattern identifier is not directly used as the raw symbol input, but is introduced into the bottleneck layer as conditional information. The pattern encoding unit can use one-hot encoding, embedded encoding, or other encoding methods suitable for representing a finite number of templates.

[0097] Preferably, the deep features of the three images are concatenated with the corresponding pattern-encoded conditional features at the bottleneck layer to form joint input features:

[0098] H(L) = Concat(F) n-1 (L), F n (L), F n+1 (L), M n-1 M n M n+1 )

[0099] Here, Concat() represents the concatenation function. Subsequently, the joint input features H(L) are fused and transformed through one or more convolutional units to obtain the bottleneck layer output features:

[0100] B=Φ(H(L))

[0101] in, This represents a deep feature fusion mapping consisting of convolutional layers, nonlinear activation layers, normalization layers, residual blocks, or combinations thereof.

[0102] The bottleneck layer fusion design serves two purposes: firstly, it aggregates the high-level structural information extracted from the deepest layer of the three images, enabling the network to comprehensively utilize the complementary information of adjacent frames at the global semantic level; secondly, it introduces sparse array pattern identifiers as conditional features into the bottleneck layer, allowing the network to not only utilize the image content itself during the fusion process, but also learn the inter-frame complementary relationships under different sparse pattern combinations by combining the array gating modes corresponding to each frame.

[0103] (d) Decoder and bridging design

[0104] The decoder is positioned after the bottleneck layer and is used to perform stepwise upsampling and feature recovery on the joint deep features output by the bottleneck layer to reconstruct the current frame I. nThe enhanced image. The decoder and encoder together form a U-net encoder-decoder structure, characterized by the introduction of multi-scale features retained in the encoding stage through a bridging structure during the decoding process, so as to preserve shallow detail information while recovering high-level structural information.

[0105] Specifically, assuming the bottleneck layer output feature is B, the decoder starts from the deepest layer, upsamples the feature level by level, and fuses it with the corresponding cross-feature at each scale, finally outputting the reconstructed image of the current frame. Suppose the decoder comprises L scale levels, then the... The decoding features at each scale are denoted as ,in .

[0106] Unlike traditional single-frame U-net, the bridging structure in this invention does not simply input the single-frame encoded features of the current frame directly into the decoder. Instead, it fuses the encoded features of three adjacent frames at the corresponding scale to form a joint bridging feature at that scale, which is then input into the corresponding layer of the decoder. Let the three frames be in the... The joint bridging features at each scale can be expressed as:

[0107]

[0108] in, Indicates the first Feature fusion mapping at various scales can be achieved by feature concatenation, convolutional fusion, weighted summation, residual compensation, attention weighting, or a combination thereof.

[0109] (e) Training methods

[0110] The image reconstruction network employs supervised learning for parameter optimization during the training phase, enabling the network to output a high-quality image of the current frame that is close to the full-channel reception condition based on the three adjacent sparsely sampled images and their corresponding sparse array pattern information.

[0111] Specifically, during the training process, the previous initial image I n-1 The initial image of the current frame I n The initial image of the next frame I n+1 As the image input to the network, M is identified by a sparse array pattern corresponding to the three frames mentioned above. n-1 M n M n+1 As a conditional input, the network outputs the reconstruction result of the current frame. Wherein, the initial image I of the current frame n and the initial image I of adjacent frames n-1 I n+1 All images were generated under sparse array gating conditions.

[0112] The supervision label is used in the current frame I. n The current frame image I formed by the full-channel reception method at the same time, in the same scene, and under the same frame conditions. n full That is, the goal of network training is to improve the reconstruction results. To approximate the current frame reference image as closely as possible under full-channel reception conditions, its mapping relationship can be expressed as:

[0113]

[0114] Through the above training method, the network can learn the compensation rules of adjacent frames to the current frame under different combinations of sparse array patterns, thereby achieving approximate recovery from sparsely sampled images to full-channel current frame images.

[0115] In a preferred embodiment, the training samples are constructed as follows: first, multiple consecutive frames of raw echo data are acquired under full-element, full-channel conditions; then, according to a preset sparse array pattern template, corresponding 2N / 3 channel data are extracted from the full-channel data to form the initial images of the previous frame, current frame, and next frame under sparse sampling conditions; simultaneously, the corresponding full-channel current frame image in this set of data is used as the supervision label. This ensures the consistency between the network input and the supervision label in terms of time, scene, and target state, improving the accuracy and stability of training.

[0116] The training loss function can be expressed as:

[0117] +

[0118] Among them, L rec L represents the reconstruction loss. str L represents the structural constraint loss. reg Let λ1, λ2, and λ3 represent the regularization loss, and λ3 be the corresponding weight coefficients. The reconstruction loss is used to constrain the pixel differences between the network's output image and the current frame label across all channels; it can employ either mean absolute error loss or mean squared error loss. rec L rec and L reg Represented as:

[0119]

[0120]

[0121]

[0122] The structural constraint loss is used to improve the consistency of the reconstructed image with the current frame label across all channels in terms of edges, texture, and local structure. Preferably, structural similarity loss, gradient consistency loss, or a combination thereof can be used. The regularization loss is used to suppress abnormal high-frequency noise and artifacts in the network output and enhance the stability of the training process. Preferably, weight decay terms, smoothing constraint terms, or feature regularization terms can be used.

[0123] This invention utilizes autoencoder generator networks or other reconstruction networks from machine learning. Taking adjacent frame images and their corresponding array sparse pattern information as input, it reconstructs or enhances the current frame image. The network learns the complementary relationships between adjacent images under different sparse sampling patterns, thereby recovering the image quality loss caused by array sparse sampling. Compared to existing single-frame enhancement methods, this invention not only utilizes the temporal correlation between images but also leverages the observational complementarity brought about by different sparse array patterns, thus improving the reconstruction effect.

[0124] In summary, this invention does not simply pursue image post-processing enhancement, but rather considers "reducing the number of analog channels" and "restoring image quality" in a unified manner, forming an overall technical solution of "limited channel sparse sampling - adjacent frame complementary observation - intelligent reconstruction compensation", which enables the system to obtain image quality close to that of full-channel operation under the condition of using only 2 / 3N analog channels.

[0125] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A multibeam sonar imaging system based on complementary sparse sampling of adjacent frames, characterized in that, include: A transducer array consisting of N array elements; 2N / 3 analog receiving channels, where N is an integer multiple of 3; An analog gating network is connected between the transducer array and the analog receiving channel; A beamforming imager is connected after the analog receiving channel; A control unit is used to control the analog gating network so that adjacent frames adopt different sparse array patterns with complementary relationships, and select the corresponding 2N / 3 array elements to 2N / 3 analog receiving channels; A multi-frame fusion reconstruction network based on U-net is used to take the initial images of the current frame and its neighboring frames, as well as the sparse array pattern identifiers corresponding to each frame, as inputs and output the reconstructed image of the current frame. The complementary relationship between adjacent frames is that the union of the sparse array patterns of any two adjacent frames covers all N array elements. The sparse array style includes three templates: Template A: Select the first two elements from every three consecutive array elements; Template B: Select the first and last two array elements from every three consecutive array elements; Template C: Select the last two elements from every three consecutive array elements; The control unit causes adjacent frames to switch cyclically in the order of template A, template B, and template C; The U-net-based multi-frame fusion reconstruction network includes: The encoding end is used to extract multi-scale features from the initial image of the previous frame, the initial image of the current frame, and the initial image of the next frame, respectively. The encoding branch structures of the three frames are the same and share parameters. The bottleneck fusion end is used to concatenate and fuse the deep features extracted from the three frames by the encoder with the sparse array pattern identifiers corresponding to each frame, and output the joint deep features. The decoding end is used to perform step-by-step upsampling and feature recovery on the joint deep features and output the reconstructed image of the current frame; The decoding end introduces multi-scale features retained in the encoding stage through a bridging structure. The bridging structure fuses the encoded features of three adjacent frames at corresponding scales and then inputs them into the corresponding layer of the decoder.

2. The multibeam sonar imaging system based on complementary sparse sampling of adjacent frames according to claim 1, characterized in that, The sparse array pattern identifier is not directly input into the encoder, but is concatenated with deep features at the bottleneck layer in the form of one-hot encoding or embedding encoding.

3. The multibeam sonar imaging system based on complementary sparse sampling of adjacent frames according to claim 1, characterized in that, The multi-frame fusion reconstruction network based on U-net is trained using supervised learning. The training samples are formed by extracting 2N / 3 channels of raw echo data from multiple consecutive frames under full-channel reception conditions, according to a preset sparse array pattern template, and using the corresponding full-channel current frame image as the supervision label.

4. A multibeam sonar imaging method based on complementary sparse sampling of adjacent frames, applied to the system according to any one of claims 1-3, characterized in that, Includes the following steps: Step 1: The control unit controls the analog gating network to make adjacent frames adopt different sparse array patterns with complementary relationships, and sequentially selects 2N / 3 array elements from N array elements to access 2N / 3 analog receiving channels; Step 2: The analog receiving channel amplifies, filters, and performs analog-to-digital conversion on the received signal to obtain echo data under sparse sampling; Step 3: The beamforming imager performs beamforming on the echo data to generate an initial image of the current frame and its adjacent frames; Step 4: Input the initial images of the current frame and its adjacent frames, as well as the sparse array pattern identifiers corresponding to each frame, into the pre-trained multi-frame fusion reconstruction network based on U-net. Step 5: The multi-frame fusion reconstruction network based on U-net outputs the reconstructed image of the current frame as the final imaging result of the current frame.

5. The multibeam sonar imaging method based on complementary sparse sampling of adjacent frames according to claim 4, characterized in that, The processing steps of the U-net-based multi-frame fusion reconstruction network described in step 4 include: Encoding stage: Multi-scale feature extraction is performed on the initial images of the previous frame, the current frame, and the next frame respectively. The three frames use encoding branches with the same structure and shared parameters. Bottleneck fusion stage: The deep features extracted from the three frames by the encoder are spliced ​​and fused with the sparse array pattern identifiers corresponding to each frame to form joint deep features; Decoding stage: The joint deep features are upsampled step by step, and the multi-scale features retained in the encoding stage are introduced through the bridging structure, and finally the reconstructed image of the current frame is output; The bridging structure fuses the encoded features of three adjacent frames at corresponding scales and then inputs them into the corresponding layer of the decoder.

6. The multibeam sonar imaging method based on complementary sparse sampling of adjacent frames according to claim 4, characterized in that, The multi-frame fusion reconstruction network based on U-net is trained using the following loss function: The loss function comprises a weighted sum of reconstruction loss, structural constraint loss, and regularization loss; The reconstruction loss is used to constrain the pixel differences between the network output image and the current frame label across all channels. The structural constraint loss is used to improve the consistency of the reconstructed image with the current frame label across all channels in terms of edges, texture, and local structure; The regularization loss is used to suppress abnormal high-frequency noise and artifacts in the network output.

7. The multibeam sonar imaging method based on complementary sparse sampling of adjacent frames according to claim 6, characterized in that, The reconstruction loss is either the mean absolute error loss or the mean square error loss. The structural constraint loss adopts either gradient consistency loss or structural similarity loss; The regularization loss employs either a weight decay term or a smoothing constraint term.

Images