Underwater target detection method based on multi-modal features and domain adaptation

Abstract

The application provides an underwater target detection method based on multi-modal features and domain adaptation, which comprises the following steps: S11, collecting sonar images, optical images and environmental data; S12, extracting sonar features and optical features, encoding the environmental data into environmental channel weights, dynamically adjusting the fusion ratio of the sonar features and the optical features through the environmental channel weights, and obtaining fused features; S13, performing spatial attention calculation on the sonar features to obtain a spatial weight map, enhancing the optical features by using the spatial weight map, and forcibly aligning the sonar features and the optical features at the target edge; S14, decoupling the fused features into synthetic domain features, decoupling the fused features and the environmental data into real domain features, gradually aligning the synthetic domain features and the real domain features through an asymptotic domain alignment, and completing the construction of a target detection model. Through the combination of multi-modal data acquisition, dynamic feature fusion and decoupling, and embedded real-time detection, the precision of underwater target monitoring is significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and in particular to an underwater target detection method based on multi-modal features and domain adaptation. BACKGROUND

[0002] Underwater target detection is extremely important in the fields of marine exploration, environmental monitoring, and underwater resource development. However, the complex underwater environment poses great challenges to target detection. Light attenuation, scattering, and absorption in water result in underwater images with high noise, blurred texture, low contrast, and color distortion. Suspended particles and plankton further scatter light, reducing visibility and blurring target edges and details. Sonar can penetrate turbid water, but is easily disturbed by marine environmental noise, such as water flow noise and biological noise, leading to decreased detection accuracy. Therefore, underwater target detection often requires the combination of multiple modal data such as optical images and sonar images. However, different modal data have different features and properties, and their complementary nature is difficult to effectively utilize. For example, optical images have rich visual information but poor quality in turbid water, while sonar images have strong penetration ability but low resolution. Traditional fixed weight fusion methods cannot fully exploit the potential value of heterogeneous data, resulting in poor fusion results and affecting detection performance. In addition, the difference between synthetic data and real scenes is large, and models trained based on synthetic data perform poorly in actual applications. The lighting conditions, water quality characteristics, and other factors of synthetic data are different from real underwater environments, making it difficult for models to accurately identify and locate targets. The dynamic changes of underwater environments, such as water flow, light intensity, and direction, also pose higher requirements on the generalization ability of models.

[0003] Traditional target detection methods rely on fixed weight fusion or single modal detection. Fixed weight fusion methods lack flexibility and cannot be adaptively adjusted. Single modal detection methods are limited by single data source information and cannot take advantage of multi-modal data. In turbid water or dynamic lighting conditions, these traditional methods perform poorly and cannot meet actual needs. SUMMARY

[0004] The present application aims to provide an underwater target detection method based on multi-modal features and domain adaptation, which significantly improves the accuracy of underwater target monitoring by combining multi-modal data acquisition, dynamic feature fusion and decoupling, and embedded real-time detection.

[0005] To achieve the above-mentioned application purposes, the present application provides an underwater target detection method based on multi-modal features and domain adaptation, which comprises:

[0006] S11, collecting underwater target data, the target data including sonar images, optical images, and environmental data;

[0007] S12, respectively, the features of the sonar image and the optical image are extracted, the sonar features and the optical features are obtained, the environment data is encoded into environment channel weights, the fusion ratio of the sonar features and the optical features is dynamically adjusted through the environment channel weights, and the fusion features are obtained;

[0008] S13, the sonar features are subjected to spatial attention calculation, a spatial weight map is obtained, the optical features are enhanced by using the spatial weight map, and the enhanced optical features and the sonar features are forced to align at the target edge through comparison learning constraints and a Sobel operator;

[0009] S14, the fusion features are decoupled into synthetic domain features, the fusion features and the environment data are decoupled into real domain features, the synthetic domain features and the real domain features are gradually aligned through asymptotic domain alignment, and the construction of the target detection model is completed.

[0010] Further, the features of the sonar image and the optical image are extracted, specifically including:

[0011] The sonar image is subjected to feature extraction through a Wavelet-CNN, the sonar image is decomposed into high-frequency components by using a Haar wavelet, and the expression is:

[0012]

[0013] wherein, is the sonar image, , is a real set, is a single channel, is a height, is a width, is a spatial size, is the sonar feature, , , the high-low, low-high, and high-high frequency components of the Haar wavelet decomposition are respectively;

[0014] The high-frequency components are retained and the features are further extracted through a CNN, and the extracted sonar features are output wherein C1 is the channel number of the sonar features;

[0015] The features of the optical image are extracted through a ResNet-50, and the optical features are output wherein C2 is the channel number of the optical features.

[0016] Further, the environment data is encoded into environment channel weights, specifically including:

[0017] The environment data is converted into an environment parameter vector , the environment channel weights are obtained through​ Will Encode the expression as follows:

[0018]

[0019] in, , Channel weights generated for environmental parameters, , Turbidity Light intensity, The water is deep.

[0020] Furthermore, the fusion ratio of sonar features and optical features is dynamically adjusted through environmental channel weights, specifically including:

[0021] S21. Merge the optical and sonar features along the channel dimension to obtain the stitched features, expressed as:

[0022]

[0023] in, This is a feature obtained by splicing optical and sonar features. ;

[0024] S22, will After performing global average pooling, through Compression yields the weight control vector, expressed as:

[0025]

[0026]

[0027] in, For compression features, , This is the weight control vector. ;

[0028] S23, will and After element-wise multiplication, the channel weight matrix is ​​obtained by compression using the Sigmoid function, and the expression is:

[0029]

[0030]

[0031] in, for and The result obtained by element-wise multiplication. This is the channel weight matrix. is a Sigmoid function, is an element-wise multiplication operation;

[0032] S24, after the channel weight matrix is expanded to the spatial dimension through the broadcast mechanism, the sonar feature and the optical feature are fused to obtain a fused feature, and the expression is:

[0033]

[0034]

[0035] wherein, is the result obtained after W is expanded to the spatial dimension through the broadcast mechanism, , is the fused feature.

[0036] Further, after the spatial attention calculation of the sonar feature, a normalization operation is performed to generate a spatial attention map, and the expression is:

[0037]

[0038]

[0039] wherein, is a spatial weight map, is a normalization operation, is a one-dimensional convolution operation, is an exponential function, is a spatial attention map the element value of the spatial attention map located at the hth row and the wth column, is a spatial attention map or a specific position in the sonar feature map, wherein h is the row index and w is the column index, is a variable representing the spatial attention map or the position in the sonar feature map, used for summation operation.

[0040] Further, the optical feature is enhanced by using the spatial weight map, specifically including:

[0041] The optical feature is enhanced by using the spatial weight map, and the expression is:

[0042]

[0043] wherein, is the enhanced optical feature, is an element-wise multiplication operation, is the spatial weight map is element-wise multiplied with each channel corresponding position of the optical feature , to add the weighted optical feature to the original optical feature.

[0044] Further, the enhanced optical feature and the sonar feature are forced to align at the target edge by a contrastive learning constraint and Sobel operators, specifically including:

[0045] S31, respectively using a horizontal Sobel operator and a vertical Sobel operator convolution operation is performed on to obtain a horizontal gradient map and a vertical gradient map .

[0046] S32, respectively using a horizontal Sobel operator and a vertical Sobel operator convolution operation is performed on to obtain a horizontal gradient map and a vertical gradient map .

[0047] S33, for each position on the feature map, an L2 norm between the enhanced optical feature gradient vector and the sonar feature gradient vector is calculated, and the L2 norm is a Euclidean distance.

[0048] S34, the gradient difference of all positions on the feature map is summed to obtain a contrastive learning constraint loss , and the expression is:

[0049] .

[0050] S35, the is updated by a loss function and back propagation, so that is gradually reduced to realize the alignment of the optical feature and the sonar feature at the target edge.

[0051] Further, the fused feature is decoupled into a synthetic domain feature, specifically including:

[0052] S41, the fused feature is forward propagated and back propagated, and the expression is:

[0053]

[0054]

[0055] wherein, is a feature generated by forward propagation, for the gradient reversal operation, for the gradient operator, for the adversarial loss function, for a hyperparameter, i.e., the gradient reversal coefficient;

[0056] S42, calculating the adversarial loss of the input discriminator network , forcing the feature extractor to generate synthetic domain features that cannot be distinguished by the discriminator .

[0057] Further, decouple the fusion features and the environment data into real domain features, specifically including:

[0058] Embed the fusion features and the environment data into the feature space to obtain the real domain features, and the expression is:

[0059]

[0060] wherein, is the real domain feature, is a scaling factor, is an offset factor;

[0061] The calculation expression of is:

[0062]

[0063] The calculation expression of is:

[0064]

[0065] Input the environment parameter vector , , split the output neurons into and each C dimension, obtain and , and the expression is:

[0066]

[0067] wherein, .

[0068] Further, gradually align the synthetic domain features and the real domain features through asymptotic domain alignment, specifically including:

[0069] S51, calculate the distribution difference between the synthetic domain features and the real domain features using the MMD metric, to obtain ; ​

[0070] S52, detecting the real domain data, and calculating a detection loss according to a detection result and a real label

[0071] S53, calculating a total loss of the progressive domain alignment, expressed as:

[0072]

[0073] wherein, the total loss of the progressive domain alignment, the number of iterations of the training, the weight coefficient of the domain alignment loss, the weight coefficient of the detection loss;

[0074] S54, calculating the total loss using a back propagation algorithm the gradient of the target detection model parameter, updating the parameter of the target detection model according to the gradient, so that the total loss gradually decreases;

[0075] S55, iteratively training steps S51 to S54 until a preset number of training times is reached, and completing the construction of the target detection model.

[0076] Compared with the prior art, the beneficial effects of the present application are:

[0077] The underwater target detection method based on multi-modal features and domain self-adaptation provided by the present application overcomes the limitations of single modal detection methods by fusing the features of sonar images and optical images and dynamically adjusting the fusion ratio of sonar features and optical features in combination with environmental data. The model can adaptively optimize the feature fusion effect according to different underwater environments and target characteristics, thereby more accurately identifying and positioning the target, and significantly improving the precision and reliability of underwater target detection. At the same time, through spatial attention calculation and target edge forced alignment technology, the differences in target edges and details of multi-modal data are solved, further improving the model's ability to capture target edges and details. In addition, the asymptotic domain alignment strategy can gradually reduce the differences between synthetic data and real scenes, reduce the model's dependence on synthetic data, and enhance the model's adaptability and generalization ability in different underwater environments. The present application can still maintain excellent detection performance in complex and variable underwater environments, providing stronger technical support for marine exploration, environmental monitoring, and underwater resource development. BRIEF DESCRIPTION OF DRAWINGS

[0078] ​To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0079] Figure 1 This is a schematic diagram of the underwater target detection method based on multimodal features and domain adaptation provided in an embodiment of the present invention. Detailed Implementation

[0080] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0081] Reference Figure 1 This embodiment provides an underwater target detection method based on multimodal features and domain adaptation, the method comprising:

[0082] S11. Acquire underwater target data, including sonar images, optical images, and environmental data. In this embodiment, underwater target acquisition can be performed simultaneously using an optical camera (RGB+multispectral), side-scan sonar, and environmental sensors (turbidity, illumination, depth). Any acquisition device capable of acquiring sonar images, optical images, and environmental data is also applicable to this invention, and this invention does not limit the scope of the invention.

[0083] S12. Features are extracted from the sonar and optical images respectively to obtain sonar features and optical features. After encoding the environmental data into environmental channel weights, the fusion ratio of sonar features and optical features is dynamically adjusted through the environmental channel weights to obtain fused features. In this embodiment, the fusion ratio of optical and sonar is dynamically adjusted according to the environmental parameter vector to solve the problem of insufficient adaptability of fixed weight fusion, that is, the pixel-level fusion of optical and sonar is driven by the environmental parameter vector.

[0084] Feature extraction was performed on both sonar and optical images, specifically including:

[0085] Feature extraction of sonar images is performed using Wavelet-CNN, and sonar images are processed using Haar wavelets. Decomposed into high-frequency components, the expression is:

[0086]

[0087] in, is a sonar image, , is a real set, is a single channel, is a height, is a width, is a spatial size, is a sonar feature, , , are high-low, low-high, high-high frequency components of the Haar wavelet decomposition, respectively.

[0088] The high frequency components are retained and further extracted by a CNN to output the extracted sonar feature where C1 is the number of channels of the sonar feature.

[0089] In this embodiment, the wavelet decomposition can effectively remove the high frequency noise of the sonar image, retain the high frequency components to highlight the target contour, and suppress the noise.

[0090] The features of the optical image are extracted by ResNet-50 to output the optical feature where C2 is the number of channels of the optical feature.

[0091] In this embodiment, the input optical image is processed by a 50-layer residual network (ResNet-50) to finally obtain a feature map with a specific number of channels, height and width This feature map contains high-level semantic features of the input image and can be used for subsequent multi-modal fusion and target detection tasks. ResNet-50 contains 4 residual blocks, each block contains multiple convolution layers (such as 3x3 convolution), batch normalization (BN) operation and ReLU activation function.

[0092] The parameter meanings of, R represents a real set, and R is used to describe the value range of the feature tensor. C represents the number of channels, and in the output of ResNet-50, the number of channels C depends on the specific structure and configuration of the network, and the number of channels output by the residual block in the last stage is usually the number of channels of the final output feature. For example, if the number of channels output by the residual block in the last stage is 256, then C=256. The number of channels reflects the richness of the features, and each channel can be regarded as a representation of different features of the image. H represents the height of the feature map, and after a series of convolution and pooling operations, the height of the feature map will gradually decrease, and the final height H depends on the size of the input image, the size of the convolution kernel, the step and the padding. W represents the width of the feature map, which is obtained after convolution and pooling operations, and it reflects the size of the feature map in the horizontal direction.

[0093] Encoding the environmental data into environmental channel weights, specifically including:

[0094] Transforming the environmental data into an environmental parameter vector , by encoding , the expression is:

[0095]

[0096] wherein, , is the channel weight generated by the environmental parameter, , is the turbidity, ranging from [0, 1], is the light intensity, in lux, is the water depth, in meters.

[0097] In this embodiment, the environmental parameter vector is mapped into a weight vector of channel dimension, is the channel weight generated by the environmental parameter, to control the contribution ratio of sonar / optical, for example: when the turbidity , [C=0] (low frequency channel) can be encoded as 0.9. The turbidity as input, after a nonlinear transformation by , a higher value can be generated in the low frequency channel (such as C=0). For example: when , may output [C=0]=0.9. The structure is: input layer: 3-dimensional ; hidden layer: 64 neurons, ReLU activation; output layer: C-dimensional (consistent with the number of feature channels).

[0098] wherein, the channel weight generated by the environmental parameter is a set of weight values obtained by processing the environmental parameter vector E, used to adjust the importance of different channels in the subsequent multi-modal data (such as optical and sonar data) fusion process. The generation process: transform the environmental data into an environmental parameter vector, then input it into for encoding, is a multi-layer perceptron, consisting of an input layer, a hidden layer and an output layer. The input layer receives a three-dimensional environmental parameter vector, the hidden layer performs nonlinear transformation, and the output layer outputs a vector with a dimension of C, i.e. the channel weight generated by the environmental parameter .

[0099] Adjusting the fusion ratio of sonar features and optical features dynamically through environmental channel weights, specifically including:

[0100] S21, merging the optical feature and the sonar feature in the channel dimension to obtain a spliced feature, expressed as:

[0101]

[0102] wherein, is the spliced feature of the optical feature and the sonar feature, .

[0103] In this embodiment, the optical feature and the sonar feature are merged in the channel dimension, with the precondition that and the spatial size (height H and width W) must be the same, that is, , ∈ are spliced in the channel dimension, and the spliced feature has a dimension of In the deep learning framework (such as PyTorch), the torch.cat() function can be used to implement it.

[0104] S22, after global average pooling of , the compressed feature is obtained through compression, and the expression is:

[0105]

[0106]

[0107] wherein, is the compressed feature, , is the weight control vector, .

[0108] In this embodiment, after global average pooling of the spliced feature , the compressed feature is obtained, and in the processing of multi-modal data, the optical feature and the sonar feature are spliced after being extracted, and the obtained contains rich information, but the dimension is high. In order to more efficiently process these features, a global average pooling operation is performed on it. The compressed feature will be compressed through a multi-layer perceptron to obtain a weight control vector , which is used to control the weight of each channel during multi-modal data fusion. It is usually composed of an input layer, several hidden layers and an output layer. The number of neurons in the input layer is equal to the number of channels C1+C2 of the spliced feature, and the number of neurons in the output layer is C, that is, dimensions; the number of neurons and the number of layers of the hidden layer can be adjusted according to specific tasks and data sets.

[0109] Forward propagation process, input layer: the features after global average pooling are flattened into a one-dimensional vector as input. Assuming ∈ , the dimension of the flattened vector is . Hidden layer: the input vector passes through each hidden layer in turn, each hidden layer contains a number of neurons, and the neurons are connected by full connection. In each hidden layer, the input vector is multiplied by the weight matrix of the layer, then added to the bias vector, and finally nonlinearly transformed by the activation function (such as ReLU). Output layer: after processing by several hidden layers, the output of the last hidden layer is input to the output layer, which also performs weighted summation and bias operation, but usually does not use the activation function (unless there is a special requirement), to obtain the final output vector .

[0110] S23, after element-wise multiplication of and , the channel weight matrix is obtained by compressing through the Sigmoid function, and the expression is:

[0111]

[0112]

[0113] In this embodiment, is the channel weight generated by the environmental parameter and the weight control vector is obtained by element-wise multiplication, which integrates the influence of environmental factors and multi-modal data characteristics on channel weights, and preliminarily fuses the weight information of two different sources, preparing for subsequent generation of the final channel weight matrix. W is the final channel weight matrix obtained by compressing through the Sigmoid function, and the element value in W is limited between 0 and 1, which can be used as a weight to reasonably fuse each channel of multi-modal data. is the Sigmoid function, and its expression is σ(x)= , which compresses the input value to the interval of 0 to 1.

[0114] Element-wise multiplication of and is performed to combine environmental parameters and feature statistical information, for example: if [C=0]=0.9 (high turbidity) and [C=0]=0.5 (characteristic shows that low frequency is important), then [C=0]=0.45. When , The value of the low frequency channel is high, resulting in a lower W, thereby increasing the sonar weight 1-W. Reflects the importance of the current input features, for example: if the sonar feature responds strongly in the low frequency channel, [C=0] is higher, further increasing the sonar weight. Sonar weight = 1-W, for example: if W[C=0]=0.2, then the sonar weight in this channel is 1-0.2=0.8.

[0115] Low frequency channel characteristics, the low frequency component of the sonar (such as contour information) is less affected by turbidity, when turbidity is high, the low frequency channel of the optical feature may contain more noise, by increasing the sonar weight can suppress these noises. Weight fusion dynamically generates fine-grained fusion weights through the synergy of environmental parameters and feature statistics, when the turbidity exceeds the threshold, the sonar weight of the low frequency channel is significantly increased, effectively suppressing optical noise, realizing adaptive fusion of multi-modal data.

[0116] S24, after the channel weight matrix is expanded to spatial dimensions through the broadcast mechanism, the sonar feature and the optical feature are fused to obtain a fused feature, the expression is:

[0117]

[0118]

[0119] In this embodiment, is the result obtained by expanding the channel weight matrix W to spatial dimensions through the broadcast mechanism, its main role is to weight and fuse the sonar feature and the optical feature in the spatial dimension to generate a fused feature , so as to fully utilize the complementary information of two different modal data and improve the accuracy and robustness of target detection. The broadcast mechanism is a mechanism that allows tensors of different shapes to perform element-level operations without actually copying data. Specifically, it is to copy the weight value of each channel in W in the height and width directions, so as to cover the entire spatial range of the feature map, thereby obtaining .

[0120] The channel weight is extended to the spatial dimension to realize pixel-by-pixel dynamic adjustment, and the gating feature fusion is a spatial refinement step of dynamic weight fusion. By extending the channel weight to the spatial dimension, the modal contribution adjustment is realized pixel by pixel. This fine-grained control mechanism can effectively suppress environmental noise (such as turbidity interference) and enhance the modality complementarity, thereby ultimately improving the robustness of underwater target detection. If C=256, H=256, and W=256, the weight of each channel is copied to all spatial positions. For each channel C, each spatial position The expression is:

[0121]

[0122] The two modalities are balanced by linear interpolation to avoid feature conflicts, and the element-by-element multiplication operation supports backpropagation, which facilitates end-to-end training. When the sonar detects the target outline (such as a metal pipe), but the optical image is blurred, the gating fusion increases the contribution of the sonar channel in the region to strengthen the outline features. For example: in the region with high turbidity (such as ), the sonar weight 1 W is increased to 0.8 in the low-frequency channel to suppress the optical noise in the region.

[0123] S13, spatial attention calculation is performed on the sonar features to obtain a spatial weight map. After the optical features are enhanced using the spatial weight map, the enhanced optical features are forced to align with the sonar features at the target edge through comparison learning constraints and Sobel operators. The outline information of the sonar is used to enhance the target region of the optical image, solving the edge blur problem. The sonar data is sensitive to the target outline (low-frequency components are stable), but the resolution is low; the optical image is rich in details but easy to blur. Through the attention mechanism, the outline knowledge of the sonar is transferred to the optical features, strengthening the target edge.

[0124] After the spatial attention calculation is performed on the sonar features, a normalization operation is performed to generate a spatial attention map, and the expression is:

[0125]

[0126]

[0127] wherein, is the spatial weight map, is the normalization operation, is a one-dimensional convolution operation, is an exponential function, is the spatial attention map is the value of the element located in the hth row and wth column in the spatial attention map, is the spatial attention map Or a specific location in the sonar feature map, where h is the row index and w is the column index. The variable used for summation operations represents the location in the spatial attention map or sonar feature map.

[0128] In this embodiment, the spatial attention map It is a two-dimensional matrix with dimensions (H×W). The value of is in the range of [0, 1], which reflects the importance of the sonar feature map at the position (h, w). The closer the value is to 1, the more important the feature at that position is for target detection; the closer the value is to 0, the less important the feature at that position is. In the denominator of the Softmax normalization operation inside, It will traverse all locations in the spatial attention map, that is From 0 to H - 1, From 0 to W - 1, then for all positions The values ​​are summed. This summation result is used for each position. The values ​​are normalized to ensure The sum of all elements in the array is 1.

[0129] This is a 1×1 convolution operation. A 1×1 convolution kernel is a special type of convolution operation with a kernel size of 1×1, which converts sonar features... Compressing from C channels to 1 channel. For the input feature map Each position in Convolution performs a weighted summation of the C channel values ​​at a given position, resulting in a new value. This process can be viewed as a linear combination of features from different channels, thus compressing multi-channel features into single-channel features. The Softmax function converts each element of the input vector or tensor into a probability value, ensuring that the sum of all elements is 1. The Softmax function processes the single-channel feature map compressed by the 1×1 convolution, generating a spatial weight map. To highlight the target contour region detected by sonar.

[0130] The spatial weight graph is a two-dimensional matrix with dimension 1. In the spatial weight map, each element represents the importance of a sonar feature at its corresponding spatial location. A larger value indicates a more important sonar feature at that location, meaning that the optical features at that location will receive more attention and enhancement during subsequent optical feature enhancement. Due to the softmax normalization process, Each element in the array has a value in the interval [0, 1], and the sum of all elements is 1. For example, in underwater target detection, if The value in the region is large, indicating that the sonar detects a clear target profile in this region, and the information can be used to enhance the corresponding region of the optical image.

[0131] The expression for enhancing the optical feature using the spatial weight map is:

[0132]

[0133] wherein, is the enhanced optical feature, is an element-wise multiplication operation, is the spatial weight map is multiplied element-wise with the optical feature at the corresponding position of each channel, is the weighted optical feature added to the original optical feature.

[0134] In this embodiment, is to strengthen the target region indicated by the sonar, superimpose the original feature to retain background information, and avoid over-enhancement. According to the importance of the sonar feature, the optical feature is weighted, and in the region where the sonar detects the target profile (i.e., the region with a large value in , the response of the optical feature at these positions will be enhanced; while in the region considered unimportant by the sonar , the value is small, the response of the optical feature will be suppressed. In order to retain the background information and other details in the original optical feature, avoid information loss caused by over-enhancement, and highlight the target region feature indicated by the sonar. For example: when the sonar detects the profile of a metal pipe, the value in this region is 0.9, and the response of the corresponding region of the optical feature is improved by 90%.

[0135] The sonar data is sensitive to the target profile, and can relatively stably detect the approximate profile of the target even in complex environments; while the optical image is rich in details, but is easily affected by environmental factors (such as light, turbidity, etc.), resulting in blurred target edges. By using the spatial weight map generated by the sonar feature to weight the optical feature, the profile information of the sonar can be transferred to the optical feature, thereby strengthening the features of the target region in the optical image and solving the problem of blurred edges.

[0136] By comparing the learning constraints and the Sobel operator, the enhanced optical feature and the sonar feature are forced to align at the target edge, which specifically includes:

[0137] S31, respectively using horizontal Sobel operator and vertical Sobel operator on Perform a convolution operation to obtain the gradient map in the horizontal direction. and gradient plot in the vertical direction ;

[0138] S32. Apply the horizontal Sobel operator respectively. and vertical Sobel operator right Perform a convolution operation to obtain the gradient map in the horizontal direction. and gradient plot in the vertical direction ;

[0139] S33. For each location on the feature map Calculate the enhanced optical feature gradient vector and sonar feature gradient vector The L2 norm between them, where the L2 norm is the Euclidean distance;

[0140] S34. For all positions on the feature map The gradient differences are summed to obtain the contrastive learning constrained loss. The expression is:

[0141] ;

[0142] S35, will The parameters are updated using the loss function and backpropagation, so that... Gradually reduce the size to align the optical and sonar features at the target edge.

[0143] In this embodiment, by utilizing contrastive learning constraints and the Sobel operator, the enhanced optical and sonar features can be forced to remain consistent at the target edge, thereby improving the effect of multimodal feature fusion. To compare the learning constraint loss, used to measure the enhanced optical features and primitive sonar features The goal of addressing the difference in edge information is to minimize this loss, aligning the edges of the enhanced optical features as closely as possible with the edges of the sonar features, thereby resolving modal conflicts and ensuring consistency between the features of the two modalities at the target edges. Increasing the sonar weight automatically increases the sonar weight, further suppressing optical noise.

[0144] To enhance the optical features at position The gradient vector at a given location represents the edge intensity and orientation information of the enhanced optical feature at that position. For sonar features at location The gradient vector at this position represents the edge strength and direction information of the sonar feature at this position. The gradient difference of all positions on the feature map is summed up, that is, the gradient difference of each position on the feature map is accumulated to obtain an overall loss value. The gradient difference of all positions on the feature map is summed up, that is, the gradient difference of each position on the feature map is accumulated to obtain an overall loss value. Sobel operator in the horizontal direction, a 3x3 convolution kernel, used to detect the edges of the image or feature map in the horizontal direction. Sobel operator in the vertical direction, which is the transpose of the horizontal direction Sobel operator, used to detect the edges of the image or feature map in the vertical direction.

[0145] Through the sonar guided attention mechanism, the edge of the optical feature is enhanced before fusion, solving the problem of possible blur of the fused feature. The output of the enhanced optical feature not only optimizes the weight distribution of the fused feature, but also directly improves the positioning and classification performance of the detection head, forming a closed-loop optimization of multi-modal features. The edges of the enhanced optical features are sharper, improving the positioning accuracy of the detection frame, and the alignment of the sonar-optical edges reduces the feature ambiguity and improves the classification confidence.

[0146] S14, decoupling the fused feature into a synthetic domain feature, decoupling the fused feature and the environment data into a real domain feature, and gradually aligning the synthetic domain feature and the real domain feature through asymptotic domain alignment to complete the construction of the target detection model. In this embodiment, the distribution difference between the synthetic data and the real water area causes the overall prediction performance to decline. By separating the target intrinsic feature through adversarial training, it is not affected by environmental changes. Separate the synthetic domain feature (target intrinsic attribute or environment invariant feature) from the real domain feature (illumination, turbidity, i.e. environment related feature) to improve cross-domain generalization capability.

[0147] Decoupling the fused feature into a synthetic domain feature, specifically including:

[0148] S41, forward propagation and back propagation of the fused feature, the expression is:

[0149]

[0150]

[0151] wherein, is the feature generated by forward propagation, is a gradient inversion operation, is a gradient operator, is an adversarial loss function, is a hyperparameter, i.e. the gradient inversion coefficient.

[0152] In this embodiment, in the neural network, forward propagation means that data starts from the input layer,

[0153] The process of passing through each hidden layer in turn and finally reaching the output layer. The purpose of forward propagation is to extract and process input features through a series of calculations and transformations in the network, to obtain more meaningful feature representations that can be used for subsequent tasks such as classification, detection, etc. Gradient reversal operation (GRL) In normal neural network training, the gradient is directional information used to update network parameters to minimize the loss function. Gradient reversal operation will reverse the direction of the gradient when backpropagation, when data passes through the GRL layer for forward propagation, it simply outputs the input as is, that is are numerically the same. But when backpropagation, the gradient passing through the GRL layer will be multiplied by a negative coefficient (usually -1), so that the direction of the gradient is reversed.

[0154] is used to calculate the gradient of a certain function with respect to the parameter . The gradient represents the gradient of the adversarial loss with respect to the network parameters , which reflects the rate and direction of the change of the loss function with respect to the parameter . is a positive value that is used to control the strength of gradient reversal. When backpropagation, the gradient will be multiplied by , that is .The value of will affect the balance between domain alignment and target detection of the model. If takes a larger value, the effect of gradient reversal is stronger, and the model will pay more attention to domain alignment, that is, to make the feature distribution of data in different domains more similar; if takes a smaller value, the model will focus more on the accuracy of target detection. Therefore, the choice of needs to be adjusted according to the specific task and data set to achieve the best performance. is used to measure the ability of the discriminator to distinguish different domain data.

[0155] S42, input into the discriminator network to calculate the adversarial loss , forcing the feature extractor to generate synthetic domain features that cannot be distinguished by the discriminator.

[0156] In this embodiment, adversarial training forces the feature extractor to generate synthetic domain features that cannot be distinguished by the discriminator through gradient reversal during backpropagation.These features capture the essential properties of the target (e.g., shape, material), working together with the fused features and the enhanced optical features to significantly improve the model's generalization ability in the synthetic→real domain.

[0157] Adversarial loss The computational expression is:

[0158]

[0159] where, is the expectation of the real domain features, from the real data processing flow, , is the expectation of the synthetic domain features, from the real data processing flow, , is the discriminator network, with input and output as the domain probability (0=synthetic domain, 1=real domain).

[0160] Discriminator is a binary classification network to determine the domain label (synthetic / real) of the features. The synthetic domain refers to the features extracted from synthetic data, such as images generated by simulating underwater environments. The real domain refers to the fused features and environmental data extracted from real-world data, such as optical and sonar images captured in actual water environments. The feature extractor, or generator, is used to generate synthetic domain features.

[0161] The synthetic domain refers to the domain of synthetic data generated by simulating underwater environments and other methods. "Synthetic domain refers to fused features" means that the features extracted from synthetic data and subjected to fusion operations are the fused features. These fused features are generated by fusing sonar features and optical features through a series of operations mentioned earlier, and are used to represent the synthetic data at the feature level, so that the discriminator D can determine whether the features come from the synthetic domain.

[0162] The real domain refers to the domain of data captured in the real world, such as optical and sonar images obtained in actual water environments. "Real domain refers to fused features and environmental data" means that in the real domain, the data used for discriminator D judgment not only includes fused features extracted from real-world optical and sonar images and other data, but also includes related environmental data. Environmental data may include some background information, lighting conditions, water quality, and other auxiliary information of the actual water environment. These information, together with the fused features, form a complete representation of the real domain data, allowing the discriminator D to more accurately determine whether the features come from the real domain and distinguish the differences between the real domain and the synthetic domain.

[0163] Decoupling the fused features and environmental data into real domain features includes:

[0164] The fusion feature and the environmental data are embedded into a feature space to obtain real domain features, and an expression is as follows:

[0165]

[0166] wherein, is the real domain feature, is a scaling factor, is an offset factor;

[0167] A calculation expression of the real domain feature is as follows:

[0168]

[0169] A calculation expression of the real domain feature is as follows:

[0170]

[0171] The environmental parameter vector is inputted, and the output neuron is split into and each C dimension, to obtain and An expression is as follows:

[0172] wherein,

[0173] .

[0174] In the embodiment, the environmental parameters such as illumination and turbidity of different water areas are greatly different, leading to changes in feature distribution. When the turbidity is high, the scaling factor of the low-frequency channel is reduced; when the illumination is low, the offset of the brightness channel is increased, . For example, when the turbidity is high, the noise of the low-frequency channel of the optical feature is increased, and the gain needs to be reduced. The feature value is scaled to a similar range through normalization, facilitating subsequent processing.

[0175] The structure of the neural network includes an input layer: 3 neurons (corresponding to ); a hidden layer: 64 neurons, ReLU activation; and an output layer: 2C neurons (split into and each C dimension), and an element-by-element operation is performed for each channel C and each spatial position , and an expression is as follows:

[0176]

[0177] ​When the environment is in high turbidity (T = 1000 NTU), , Output (the low-frequency channel gain is reduced by 50%) to suppress noise in the sonar feature. When the environment is in low light (l = 10 lux), Output (the brightness channel offset is increased by 0.3) to enhance the visibility of the optical feature.

[0178] The synthetic domain feature is gradually aligned with the real domain feature by asymptotic domain alignment, which specifically includes:

[0179] S51, calculate the distribution difference between the synthetic domain feature and the real domain feature using the MMD metric, to obtain .

[0180] S52, detect the real domain data, and calculate the detection loss according to the detection result and the real label.

[0181] S53, calculate the total loss of the asymptotic domain alignment, which is expressed as:

[0182]

[0183] wherein, is the total loss of the asymptotic domain alignment, is the number of iterations of the training, is the weight coefficient of the domain alignment loss, is the weight coefficient of the detection loss.

[0184] In this embodiment, the synthetic domain data and the real domain data are collected, the synthetic domain data can generate optical and sonar images by simulating the underwater environment, and the real domain data is the optical and sonar images collected from the actual water area. After processing the above data, the fusion feature and the environment data are obtained, the fusion feature is decoupled to obtain the synthetic domain feature, and the fusion feature and the environment data are decoupled to obtain the real domain feature.

[0185] is the weighted sum of the domain alignment loss and the detection loss, which is used to consider the domain alignment and the detection accuracy at the same time during the training process; the value of t increases continuously with the training; With the increase of the number of training iterations t, the value gradually decreases, which means that in the early stage of training, the model pays more attention to domain alignment. Maximum Mean Discrepancy (MMD) loss is used to measure the difference between the synthetic domain feature and the real domain feature The smaller the MMD value, the closer the feature distribution of the two domains. The value of the coefficient gradually increases with the increase of the number of training iterations t, which means that in the later stage of training, the coefficient is larger, and the model pays more attention to the detection accuracy. The value of the coefficient gradually increases with the increase of the number of training iterations t, which means that in the later stage of training, the coefficient is larger, and the model pays more attention to the detection accuracy. The value of the coefficient gradually increases with the increase of the number of training iterations t, which means that in the later stage of training, the coefficient is larger, and the model pays more attention to the detection accuracy.

[0186] S54, calculating the total loss using the back propagation algorithm Regarding the gradient of the target detection model parameters, updating the parameters of the target detection model according to the gradient, so that the total loss gradually decreases.

[0187] S55, iteratively training steps S51 to S54 until a preset number of training times is reached, completing the construction of the target detection model.

[0188] In this embodiment, in the cross-domain target detection task, the environment invariant feature (synthetic domain data) is a mathematical representation of the essential attribute of the target, for example: the invariant feature of a metal target may include a high-frequency reflectivity pattern; the invariant feature of a cylindrical target may include a circular edge response. The environment-dependent feature (real domain data) is a feature affected by the environment, for example: the intensity of light in an optical image, such as the pixel value of an overexposed area; the noise level of a sonar image, which is positively correlated with turbidity; the color shift of a multispectral image, which is affected by water depth. There is a distribution difference between the synthetic domain data and the real domain data. Directly applying the model trained in the synthetic domain to the real domain will often result in a significant decrease in model performance, and the purpose of domain alignment is to reduce the feature distribution difference between the synthetic domain and the real domain, so that the model can perform well on different domains.

[0189] However, simply performing domain alignment may ignore the target of the detection task itself, resulting in a model that is aligned on the domain but has low detection accuracy. Therefore, the progressive domain alignment strategy is adopted, which pays more attention to domain alignment in the early stage of training, so that the model first learns the common features between the synthetic domain and the real domain; in the later stage of training, more attention is paid to detection accuracy to ensure that the model can accurately detect the target. This can balance the relationship between domain alignment and detection accuracy, and improve the generalization ability and detection performance of the model on the real domain.

[0190] By means of adversarial training and conditional normalization, the fusion features are decoupled into target intrinsic attributes (invariant features) and environmental disturbances (relevant features). This separation mechanism not only preserves the core features shared across domains, but also provides an explicit compensation channel for environmental changes, thereby achieving robust underwater target detection. The invariant features learn the reflection patterns of metal targets in synthetic data, which can be directly transferred to real water areas. The relevant features encode factors such as illumination and turbidity into adjustable scaling factors, for example: in high turbidity, the gain of optical features is reduced to suppress noise. Through the above progressive domain alignment method, the domain alignment and detection accuracy are dynamically balanced during training, improving the performance of the model in cross-domain scenarios.

[0191] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for underwater target detection based on multi-modal features and domain adaptation, characterized in that, The method comprises: S11, collecting underwater target data, the target data comprising sonar images, optical images and environment data, the environment data comprising turbidity data, illumination intensity data and depth data collected by an environment sensor; S12, extracting features of the sonar images and the optical images respectively to obtain sonar features and optical features, encoding the environment data as environment channel weights, dynamically adjusting a fusion ratio of the sonar features and the optical features through the environment channel weights, and obtaining fused features; S13, performing spatial attention calculation on the sonar features to obtain a spatial weight map, enhancing the optical features using the spatial weight map, and forcibly aligning the enhanced optical features and the sonar features at a target edge through a comparison learning constraint and a Sobel operator; S14, decoupling the fused features into synthetic domain features, decoupling the fused features and the environment data into real domain features, and gradually aligning the synthetic domain features and the real domain features through an asymptotic domain alignment to complete construction of a target detection model; The gradual alignment of the synthetic domain features and the real domain features comprises: S51, compute distribution difference between synthetic domain features and real domain features using MMD metric and real domain features ;​ S52, detecting the real domain data, calculating a detection loss according to a detection result and a real label ; S53, calculating a total loss of the asymptotic domain alignment, the expression being: wherein, is the total loss for progressive domain alignment, is the number of iterations for training, is the weight coefficient of the domain alignment loss, is the weight coefficient of the detection loss; S54, calculating the total loss using a back propagation algorithm Regarding the gradient of the parameters of the target detection model, the parameters of the target detection model are updated according to the gradient, so that the total loss is gradually reduced; S55, iteratively training steps S51 to S54 until a preset training number is reached to complete construction of the target detection model.

2. The underwater target detection method based on multi-modal features and domain adaptation according to claim 1, characterized in that, The features of the sonar images and the optical images are extracted respectively, and the extraction comprises: The sonar image is decomposed into high-frequency components by using a Haar wavelet to express the sonar image as: wherein, is a sonar image, , is a real set, is a single channel, is a height, is a width, is a spatial dimension, is a sonar feature, , , are high-low, low-high, high-high frequency components of a Haar wavelet decomposition, respectively; Reserve high frequency components and further extract features through CNN, output the extracted sonar features Wherein C1 is the number of channels of sonar features; extracting features of the optical image by a ResNet-50 to output optical features wherein C2 is a channel number of the optical features.

3. The underwater target detection method based on multi-modal features and domain adaptation according to claim 2, characterized in that, The environment data is encoded as environment channel weights, and the encoding comprises: Transforming environmental data into an environmental parameter vector , by encoding , the expression is: wherein, , channel weights generated for environmental parameters, , is turbidity, is light intensity, is water depth.

4. The underwater target detection method based on multi-modal features and domain adaptation according to claim 3, characterized in that, The fusion ratio of the sonar features and the optical features is dynamically adjusted through the environment channel weights, and the adjustment comprises: S21, merging the optical features and the sonar features in a channel dimension to obtain spliced features, the expression being: wherein, is the feature after the optical feature and the sonar feature are spliced, ; S22, will After performing global average pooling, through Compression yields the weight control vector, expressed as: wherein, is a compression feature, , is a weight control vector, ; S23, will and After element-wise multiplication, the channel weight matrix is ​​obtained by compression using the Sigmoid function, and the expression is: wherein, is with the result of an element-wise multiplication, is a channel weight matrix, is a Sigmoid function, is an element-wise multiplication operation; S24, expanding the channel weight matrix to a spatial dimension through a broadcast mechanism, fusing the sonar features and the optical features to obtain fused features, the expression being: wherein, is the result of W extended to the spatial dimension via a broadcasting mechanism, , is the fused feature.

5. The method of claim 2, wherein the method further comprises: After the spatial attention calculation on the sonar features, a normalization operation is performed to generate a spatial attention map, the expression being: wherein, is a spatial weight map, is a normalization operation, is a one-dimensional convolution operation, is an exponential function, is a spatial attention map is the value of an element in the spatial attention map or in the sonar feature map located at the h-th row and w-th column, is a spatial attention map or a specific position in the sonar feature map, where h is the row index and w is the column index, is a variable representing a position in the spatial attention map or in the sonar feature map used for the summation operation.

6. The method of claim 5, wherein the method further comprises: The optical features are enhanced using the spatial weight map, and the enhancement comprises: The optical features are enhanced using the spatial weight map, the expression being: wherein, is the enhanced optical feature, is an element-wise multiplication operation, is the spatial weight map is the weighted optical feature, is an element-wise multiplication operation, is the addition of the weighted optical feature and the original optical feature.

7. The underwater target detection method based on multi-modal features and domain adaptation according to claim 6, characterized in that, The enhanced optical features and the sonar features are forcibly aligned at a target edge through a comparison learning constraint and a Sobel operator, and the alignment comprises: S31, respectively using horizontal Sobel operator and vertical Sobel operator perform convolution operation on to obtain horizontal direction gradient map and vertical direction gradient map ; S32, respectively using a horizontal Sobel operator and a vertical Sobel operator performing convolution operations to obtain a horizontal direction gradient map and a vertical direction gradient map ;​ S33、for each position on the feature map , calculate the L2 norm between the enhanced optical feature gradient vector and the sonar feature gradient vector , which is the Euclidean distance; S34, summing gradient differences for all positions on the feature map to get the contrastive learning constraint loss , the expression is: ； S35, will The parameters are updated using the loss function and backpropagation, so that... Gradually reduce the size to align the optical and sonar features at the target edge.

8. The method of claim 4, wherein the method further comprises: The fused features are decoupled into synthetic domain features, and the decoupling comprises: S41, performing forward propagation and backward propagation on the fused features, the expression being: wherein, are features generated for the forward propagation, are gradient reversal operations, are gradient operators, is an adversarial loss function, is a hyperparameter, i.e., a gradient reversal coefficient; S42, will Input discriminator network calculates adversarial loss This forces the feature extractor to generate synthetic domain features that the discriminator cannot distinguish. .

9. The method of claim 8, wherein the method further comprises: The fused features and the environment data are decoupled into real domain features, and the decoupling comprises: The fused features and the environment data are embedded into a feature space to obtain real domain features, the expression being: wherein, is a real domain feature, is a scaling factor, is an offset factor; The calculation expression is: The calculation expression is: The environmental parameter vector Input , the Output neurons are split into and Each C dimension, get and , the expression is: wherein .

Images