Method and system for target recognition based on sonar image assisted optical image

Abstract

The present application relates to underwater target detection technical field, specifically to a kind of target identification method and system based on sonar image auxiliary optical image;Method includes: obtaining underwater optical image and sonar image, input double-flow multi-scale feature extraction backbone network, generate optical multi-scale feature map and sonar multi-scale feature map;Through optical-sonar adaptive feature fusion module, multi-scale feature map is handled, and enhanced multimodal fusion feature is generated;Multi-modal fusion feature is input detection head, and underwater target detection result is output;The present application effectively solves optical-sonar image modal heterogeneity and spatial mismatch problem, significantly improves the precision and robustness of underwater target detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of underwater target detection, and particularly relates to a target recognition method and system based on sonar image assisted optical image. BACKGROUND

[0002] With the increasing demand for marine resource exploration, biodiversity survey and underwater infrastructure maintenance, underwater target detection technology has become a key core technology for remote-controlled robots (ROV) to realize autonomous underwater operation. At present, the ROV platform mainly relies on optical imaging sensors or sonar imaging sensors for underwater environment perception.

[0003] However, single modality perception has significant limitations: optical images have rich color and texture details, but are extremely susceptible to water turbidity, light attenuation, suspended particle scattering and underwater noise, and their performance drops sharply in low-visibility environments; sonar images can achieve long-distance imaging and are not limited by lighting conditions, but usually have low resolution, lack color and texture details, and are susceptible to reverberation and noise interference, making it difficult to support fine target recognition.

[0004] To solve the above problems, multi-modal information fusion technology has become an effective way to improve the perception and detection performance of underwater autonomous vehicles. By combining the respective advantages of optical and acoustic images, information complementation can be achieved, and a more reliable underwater perception system can be constructed. However, underwater optical-sonar image fusion faces unique technical challenges: optical images present two-dimensional visual information in the X-Y plane in space, while sonar constructs distance-azimuth image information in the X-Z plane based on sound wave reflection characteristics, and there are significant differences in imaging dimensions, resolution, information density, etc. between the two, resulting in serious spatial misalignment and semantic gap between modalities. In addition, the collection of underwater multi-modal data sets requires complex and expensive equipment, and the target correspondence of the two modalities needs to be manually annotated, resulting in a lack of underwater optical-sonar multi-modal target detection data sets, which restricts the development of related technologies.

[0005] In recent years, multi-modal feature fusion technology has made significant progress in remote sensing imaging, autonomous driving and other fields, mainly divided into early fusion, mid-fusion and late fusion. Among them, mid-fusion based on local feature interaction has become the mainstream, but its performance is highly dependent on the quality of fine-grained feature alignment between different modalities. Existing methods rely on artificial calibration or attention mechanisms to achieve cross-modal alignment, but artificial calibration is easily invalidated by physical disturbances in dynamic underwater environments, and general attention mechanisms are difficult to handle the serious spatial mismatch problem between optical and sonar modalities.

[0006] Therefore, there is an urgent need for an underwater target detection method capable of adaptively processing optical-sonar modal heterogeneity and realizing fine-grained feature alignment and deep fusion, so as to fully utilize the complementary advantages of cross-modal information and improve the accuracy and robustness of underwater target detection. SUMMARY

[0007] To solve the above technical problems, the present application provides a target recognition method and system based on sonar image assisted optical image, aiming to realize accurate matching and deep fusion of optical-sonar multi-modal features and significantly improve the performance of underwater target detection.

[0008] In one aspect, the present application provides a target recognition method based on sonar image assisted optical image, comprising the following steps:

[0009] S100, obtaining underwater optical images, sonar images and a trained underwater optical-sonar attention fusion detection network; wherein the underwater optical-sonar attention fusion detection network comprises a double-flow multi-scale feature extraction backbone network, an optical-sonar adaptive feature fusion module and a detection head;

[0010] S200, inputting the underwater optical images and sonar images into the double-flow multi-scale feature extraction backbone network, and performing feature extraction on the optical images and sonar images respectively through the dynamic snake convolution module in the double-flow multi-scale feature extraction backbone network to generate optical preliminary feature maps and sonar preliminary feature maps; and performing multi-scale feature fusion on the optical preliminary feature maps and the sonar preliminary feature maps through the enhanced spatial pyramid pooling module in the double-flow multi-scale feature extraction backbone network to generate optical multi-scale feature maps and sonar multi-scale feature maps;

[0011] S300, processing the multi-scale features through the optical-sonar adaptive feature fusion module to generate enhanced multi-modal fusion features;

[0012] S400, inputting the multi-modal fusion features into the detection head to output underwater target detection results.

[0013] In another aspect, the present application provides a target recognition system based on sonar image assisted optical image, comprising:

[0014] At least one processor;

[0015] At least one memory for storing at least one program;

[0016] When the at least one program is executed by the at least one processor, the at least one processor realizes the method as described in any one of the above.

[0017] In another aspect, an embodiment of the present application provides a computer-readable storage medium having stored therein a program executable by a processor for performing any of the above-described methods when executed by the processor.

[0018] Embodiments of the present application include the following advantages:

[0019] The application discloses a target recognition method and system based on sonar image assisted optical image, and the method comprises the following steps: constructing a double-flow dynamic snake convolution backbone network and an enhanced spatial pyramid pooling mechanism, and effectively extracting texture details of an optical mode and geometric structure features of a sonar mode; designing an optical-sonar adaptive feature fusion module VSAF, realizing fine-grained alignment and deep fusion of cross-modal features in channel and spatial dimensions, and solving the problems of underwater optical-sonar image modal heterogeneity and spatial mismatch; introducing a Slide Loss loss function, dynamically adjusting sample weights, and improving the attention degree to difficult samples; constructing a time-space synchronous UMOD dataset, and providing a benchmark platform for multi-modal underwater target detection. Experimental results show that the application has an mAP@0.5 of 94.1% on the UMOD dataset, is significantly better than existing single-modal and multi-modal detection methods, and verifies the generalization ability on a land multi-modal dataset, thereby providing reliable technical support for underwater autonomous detection of an ROV platform. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described in the following only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0021] Figure 1 is a flowchart of a target recognition method based on sonar image assisted optical image in an embodiment of the present application;

[0022] Figure 2 is a whole architecture diagram of a VSAFDet network in an embodiment of the present application;

[0023] Figure 3 is a structure diagram of a DSC2f module in an embodiment of the present application;

[0024] Figure 4 is a comparison diagram of a DSConv and other convolution methods in an embodiment of the present application;

[0025] Figure 5 is a structure diagram of an ENSPP module in an embodiment of the present application;

[0026] Figure 6is a structural schematic diagram of a VSAF module in an embodiment of the present application;

[0027] Figure 7 is a category number statistical chart of a UMOD data set in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The disclosed concept, specific structure and generated technical effects will be described clearly and completely in the following with reference to the embodiments and the accompanying drawings, so as to fully understand the purposes, schemes and effects of the present application. It should be noted that the embodiments and the features in the embodiments in the present application can be combined with each other without conflict.

[0029] In order to make the purposes, technical schemes and advantages of the present application more clear, the present application will be further described in detail in the following with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application. When the following description relates to the drawings, the same numerals in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application, and they are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0030] It can be understood that the terms "first", "second" and the like used in the present application can be used in the present application to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0031] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0032] Unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as understood by a person skilled in the art to which the present application belongs. The terms used in the present application are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0033] Underwater target detection is a prerequisite for ROV to realize autonomous underwater detection. Traditional underwater target detection methods mainly rely on image enhancement and feature engineering, such as Retinex theory-based de-noising algorithm and multi-scale feature fusion network. However, these methods only use a single optical modality, which is difficult to overcome the limited visibility and light attenuation problems of underwater optical imaging. Although sonar target recognition technology is not limited by optical conditions, the target in the sonar image is easily affected by noise interference and reverberation, resulting in geometric distortion and detection difficulty.

[0034] In recent years, multi-modal feature fusion technology has made progress in remote sensing imaging, autonomous driving and other fields. For example, BEVFusion realizes multi-sensor fusion by mapping camera images to LiDAR point clouds; Cross-modal transformer realizes speech-text fusion through cross-modal attention mechanism. However, these technologies face serious challenges in underwater optical-sonar fusion: severe spatial misalignment between modalities, semantic gap between heterogeneous features, and modality loss in complex environments. Existing underwater multi-modal detection methods such as UAMFDet rely on deformable self-alignment modules, which are prone to feature fusion distortion in extreme misalignment or modality loss situations.

[0035] To solve the above problems, the present application proposes a underwater multi-modal target detection network VSAFDet based on attention mechanism, which effectively solves the problem of optical-sonar modality heterogeneity and spatial misalignment through adaptive cross-modal feature fusion strategy.

[0036] Reference Figure 1 As Figure 1 shown is a target recognition method based on sonar image assisted optical image provided by an embodiment of the present application, the method comprises the following steps:

[0037] S100, obtaining underwater optical image, sonar image and trained underwater optical-sonar attention fusion detection network; wherein the underwater optical-sonar attention fusion detection network comprises a double-flow multi-scale feature extraction backbone network, an optical-sonar adaptive feature fusion module and a detection head;

[0038] S200, inputting the underwater optical image and the sonar image into the double-flow multi-scale feature extraction backbone network, and respectively extracting features of the optical image and the sonar image through the dynamic snake convolution module in the double-flow multi-scale feature extraction backbone network to generate optical preliminary feature map and sonar preliminary feature map; the multi-scale feature fusion of the optical preliminary feature map and the sonar preliminary feature map is carried out through the enhanced spatial pyramid pooling module in the double-flow multi-scale feature extraction backbone network to generate optical multi-scale feature map and sonar multi-scale feature map;

[0039] S300, processing the multi-scale features through the optical-sonar adaptive feature fusion module to generate enhanced multi-modal fusion features;

[0040] S400, inputting the multi-modal fusion features into the detection head to output an underwater target detection result.

[0041] The overall structure design of the underwater optical-sonar attention fusion detection network (VSAFDet) proposed in the application is as shown in the figure. Figure 2 When the backbone network for feature extraction is performed, a double-branch design is adopted to extract feature information of different scales of optical images and sonar images respectively. Subsequently, before the double-modal multi-scale information flow enters the neck of the network, the VSAF module is designed in this embodiment. VSAF includes two core sub-modules, optical-sonar cross-modal attention (VSCA) and optical-sonar pyramid attention (VSPA). The former realizes global context fusion through bidirectional channel weight modulation, establishes a feature dependency relationship between modalities, and thus realizes automatic alignment of multi-modal features. The latter reorganizes the features by using a multi-scale spatial attention map to solve the spatial mismatch problem of cross-modal features. This hierarchical feature interaction mode effectively solves the geometric distortion and resolution difference problems between modalities in the underwater environment. Finally, the multi-scale information flow enters the neck and head of the network to realize underwater multi-modal feature fusion target detection.

[0042] In the YOLOv8 network, the C2f module as a core component can effectively aggregate multi-scale information and compress features, significantly improving the feature expression ability of the model. However, the standard convolution operation adopted by this module limits the receptive field to a fixed geometric shape, which has obvious limitations in processing target detection in complex underwater environments. The targets in underwater optical and sonar images usually exhibit features such as blur, deformation, and low contrast, so conventional convolution cannot adapt to this unstructured feature extraction requirement.

[0043] In some embodiments, in S200, the dynamic snake convolution module is constructed based on dynamic snake convolution DSConv, and the dynamic snake convolution module in the double-flow multi-scale feature extraction backbone network is used to extract features from the optical image and the sonar image respectively to generate optical preliminary feature maps and sonar preliminary feature maps, including:

[0044] S211, determining the center coordinates (x, y) of the 3x3 standard convolution kernel and the fixed sampling coordinate set K={(x-1, y-1), (x-1, y), (x-1, y+1), (x, y-1), (x, y), (x, y+1), (x+1, y-1), (x+1, y), (x+1, y+1)};

[0045] S212, introducing a deformation offset Δ, and based on the previous sampling point The position of the current sampling point is iteratively calculated , and the calculation formula is wherein c represents the horizontal distance of the sampling point to the center of the grid, is the offset learned through the input features;

[0046] S213, the optical image and the sonar image are respectively convolved through the dynamic snake convolution DSConv to generate optical primary feature maps and sonar primary feature maps.

[0047] Based on the above analysis, the DSC2f module is designed based on the dynamic snake convolution module (DSConv), as shown in Figure 3 . DSConv can adaptively adjust the sampling position of the convolution kernel according to the contour feature information and texture information of the underwater target through the deformable convolution mechanism. It adopts a constrained coordinate accumulation method to ensure that the convolution kernel can better fit the edge features of the target, while avoiding the receptive field deviation caused by underwater noise interference. Specifically, taking a 3*3 standard convolution as an example, (x, y) represents the center coordinates of the convolution kernel, and the fixed sampling coordinates are:

[0048] (1);

[0049] wherein, is the 9 fixed sampling point set of the 3*3 convolution kernel.

[0050] In order to make the convolution kernel focus on the complex geometric features of the target, DSConv introduces a deformation offset Δ, and uses an iterative strategy to constrain the learning process of the offset, so that the receptive field of the convolution kernel can closely fit the target structure without excessive divergence, as shown in Figure 4 . For each sampling point in the convolution kernel is iteratively calculated based on the position of the previous sampling point , and the calculation formula is:

[0051] (2);

[0052] wherein c represents the horizontal distance of the sampling point to the center of the grid. The offset is the offset learned in the input features. This iterative accumulation mechanism ensures the spatial continuity between adjacent points, effectively avoids the receptive field from deviating from the target area, and enhances the ability to extract local features of irregular structures. DSConv and other convolution methods of feature extraction are shown in Figure 4 .

[0053] In some embodiments, in S200, the multi-scale feature fusion of the optical primary feature map and the sonar primary feature map through the enhanced spatial pyramid pooling module in the double-flow multi-scale feature extraction backbone network generates an optical multi-scale feature map and a sonar multi-scale feature map, including:

[0054] S221, adaptive global maximum pooling and adaptive global average pooling dual-branch processing are performed on the input primary feature map, wherein the actual size of the pooling window of the adaptive global maximum pooling and the adaptive global average pooling is determined by calculating the actual height and the actual width of the pooling window, the actual height of the pooling window is the result of rounding up the ratio of the height of the input feature map to the height of the output feature map, and the actual width of the pooling window is the result of rounding up the ratio of the width of the input feature map to the width of the output feature map;

[0055] S222, multi-layer local feature extraction is performed on the primary feature map;

[0056] S223, the adaptive global maximum pooling result, the adaptive global average pooling result and the multi-layer local feature extraction result are spliced and fused in the channel dimension to generate a multi-scale feature map.

[0057] In order to improve the ability of the network to capture multi-scale context information in the underwater environment, the application proposes an enhanced spatial pyramid pooling module (Enhance Spatial Pyramid Pooling, ENSPP) fusion mechanism, as shown in Figure 5 This embodiment increases adaptive global maximum pooling and average pooling dual-branch based on multi-level local feature extraction. The adaptive global maximum pooling is conducive to the network focusing on the most prominent target area in the underwater image, and the adaptive average pooling provides the context information of the overall scene, effectively suppressing background interference. ENSPP splices and fuses the multi-scale features in the channel dimension, so that the network has both detail perception ability and global understanding ability, while the computational cost is extremely small.

[0058] The module converts an input feature map of any size into a fixed size output through an adaptive pooling mechanism, and the core is to dynamically calculate the size of the pooling window. Given the feature map size and The actual size of each pooling window is determined by the following formula:

[0059] (3);

[0060] The pooling output calculation result is:

[0061] (4);

[0062] Where i and j represent the row and column indices of the feature map, respectively. , These are the heights of the input feature map and the output feature map, respectively. , Here, represents the width of the input feature map and the width of the output feature map, respectively. Pool(·) represents the pooling operation, and X(m,n) represents the pixel value at position (m,n) in the input feature map. , These are the height and width of the pooling window, respectively. This indicates a round-up operation. This is used to calculate the pixel value at position (i,j) of the pooled output feature map.

[0063] This mechanism adjusts dynamically. and Ensure any output size Both can produce fixed output results Subsequently, the global features output by the two adaptive pooling branches, the preserved original feature details, and the multi-scale local features extracted by the three-level cascaded max pooling branches are concatenated along the channel dimension to form a feature representation that combines local detail awareness with global contextual information. Finally, standardized convolutional operations are used for integration and compression.

[0064] In some embodiments, in S300, the optical-sonar adaptive feature fusion module includes an optical-sonar cross-modal attention submodule and an optical-sonar pyramid attention submodule. The process of processing the multi-scale features through the optical-sonar adaptive feature fusion module to generate enhanced multi-modal fusion features includes:

[0065] S310, The optical multi-scale feature map and the sonar multi-scale feature map are input into the optical-sonar cross-modal attention submodule, and preliminary enhanced optical features and preliminary enhanced sonar features are generated through channel attention mechanism and multi-scale feature fusion.

[0066] S320, the initially enhanced optical features and the initially enhanced sonar features are input into the optical-sonar pyramid attention submodule, a unified attention weight is generated through multi-scale spatial attention fusion, and the optical features and sonar features are bidirectionally enhanced based on the unified attention weight to obtain enhanced multimodal fusion features.

[0067] In view of the significant difference between optical and sonar imaging modalities, the present application proposes a visual-sonar adaptive feature fusion module (VSAF), which is designed between the network backbone and the neck to process the multi-scale feature fusion problem of optical and sonar images, as shown in Figure 6 Firstly, the visual-sonar cross-modal attention sub-module (VSCA) uses cross-modal channel attention to enhance the key channels of optical features with sonar features, while modulating sonar features with optical features. Secondly, the visual-sonar pyramid attention sub-module (VSPA) realizes the synergistic enhancement of important regions through multi-scale spatial attention. This design innovatively introduces dynamic receptive field adjustment and cross-level feature compensation strategies to adaptively adjust the feature extraction range at different levels and retain shallow detail features through residual connection. Finally, a gating weighting strategy is used to dynamically balance the contributions of the two modalities to achieve the enhancement of optical features.

[0068] In some embodiments, in S300, the preliminary enhanced optical features and the preliminary enhanced sonar features are generated through the channel attention mechanism and the multi-scale feature fusion, including:

[0069] S311, the optical multi-scale feature map and the sonar multi-scale feature map are respectively subjected to global average pooling processing to generate optical global features and sonar global features;

[0070] S312, the optical global features and the sonar global features are respectively input into a multi-layer perception to generate optical channel attention weights and sonar channel attention weights;

[0071] S313, the optical channel attention weights are weighted with the optical multi-scale feature map to generate channel-enhanced optical features, and the sonar channel attention weights are weighted with the sonar multi-scale feature map to generate channel-enhanced sonar features;

[0072] S314, the channel-enhanced optical features and the channel-enhanced sonar features are respectively subjected to grouped convolution processing to extract multi-scale detail features;

[0073] S315, the optical multi-scale detail features and the sonar multi-scale detail features are spliced in the channel dimension to generate a cross-modal fusion feature map through 3x3 convolution operation;

[0074] S316, according to the relationship between the minimum dimension of the input feature map and the size of the convolution kernel, 1x1 convolution or global average pooling is dynamically selected to generate global context information;

[0075] S317, the cross-modal fusion feature map is fused with global context information to generate a preliminary enhanced optical feature and a preliminary enhanced sonar feature.

[0076] The VSCA realizes channel-level adaptive fusion of optical and sonar features through a double-link interaction mechanism, as shown in FIG. 3. Figure 6 This module contains four key designs. First, channel attention weights of optical and sonar features are respectively generated through global average pooling and fully connected layers:

[0077] (5)

[0078] (6)

[0079] wherein σ represents a Sigmoid function, MLP represents a multi-layer perceptron composed of two fully connected layers and an activation function ReLU, and is specifically represented as MLP(z) = L2(ReLU(L1(z)), z is an input global average pooling result, L1 and L2 represent linear transformations of the first and second fully connected layers respectively. AvgPool(·) represents a global average pooling operation; 、 are respectively the original input optical and sonar modal features, i.e., the feature maps of the corresponding modal obtained after feature extraction of the optical image and the sonar image by the double-flow multi-scale feature extraction backbone network.

[0080] Subsequently, group convolution is used to extract multi-scale features of the optical and sonar and , and optical and sonar fusion feature maps are generated through splicing and convolution operations :

[0081] (7);

[0082] (8);

[0083] (9);

[0084] wherein and respectively represent the convolution kernel size of the optical and sonar modal, the optical default convolution kernel size is 5, which has a larger receptive field, and the sonar default convolution kernel size is 4, which is suitable for lower resolution. represents a convolution kernel parameter, Concat represents a channel dimension splicing operation, represents a 3x3 standard convolution, represents a depth separable convolution, which independently convolves each input channel, and the number of output channels remains unchanged, and SiLU is an activation function, respectively are optical modality multi-scale feature map and sonar modality multi-scale feature map, is the fusion feature map obtained by integrating optical and sonar features after channel concatenation through 3x3 convolution.

[0085] Further, in order to realize cross-modal shared context, the generation mode of global context is dynamically selected according to the input size:

[0086] (10);

[0087] The last two modalities use each other's attention mechanism to enhance the current modality, obtaining and , to realize complementary feature mining:

[0088] (11);

[0089] (12);

[0090] wherein, represents the spatial sampling range size of the convolution kernel when performing feature extraction, that is, the side length dimension of the sliding window in the convolution operation. For example, in the optical modality default convolution kernel size mentioned in formulas (7) and (8) is 5, and the sonar modality default convolution kernel size is 4, and here corresponds to the convolution kernel spatial size of 5x5 and 4x4 respectively, which is used to control the range of the input feature map region covered by a single convolution operation, and then affect the receptive field size and detail capture ability of feature extraction. represents the smaller value of the height H and the width W of the feature map, when the smaller value is greater than or equal to the preset convolution kernel size (i.e. ), a 1x1 convolution operation ( ) is performed on the fusion feature map to generate global context information ; otherwise, the global context information is obtained through the mean pooling operation (MeanPool). represents the element-wise multiplication operation, represents matrix multiplication, , are enhanced optical modality feature map and enhanced sonar modality feature map respectively.

[0091] In some embodiments, in S320, the unified attention weight is generated by multi-scale spatial attention fusion, and the optical feature and the sonar feature are bidirectionally enhanced based on the unified attention weight to obtain enhanced multi-modal fusion features, including:

[0092] ​S321, the preliminary enhanced optical features are processed using a 3x3 convolution kernel to generate an optical multi-scale spatial attention feature map; the preliminary enhanced sonar features are processed in parallel using a 7x7 convolution kernel to generate a sonar multi-scale spatial attention feature map;

[0093] S322, the optical multi-scale spatial attention feature map and the sonar multi-scale spatial attention feature map are fused by 5x5 convolution to generate unified spatial attention weights;

[0094] S323, the spatial attention weights are used to bidirectionally enhance the optical features and the sonar features, so that the optical features are modulated by the sonar attention map to strengthen the depth information region, and the sonar features are modulated by the optical attention map to optimize the edge information of the texture-rich region;

[0095] S324, the significant region weights of the common attention of the two modalities are introduced to mark the occlusion edges and high-texture regions, and the bidirectionally enhanced optical features and sonar features are synergistically enhanced to obtain enhanced multi-modal fusion features.

[0096] Specifically, the preliminarily enhanced features are input into the VSPA, and the optical and sonar modal features are fused through adaptive spatial weight distribution. This module includes three key designs. First, 3x3 and 7x7 convolution kernels are used to process the two modalities in parallel to generate multi-scale spatial attention feature maps for optical and sonar, respectively 、 :

[0097] (13);

[0098] (14);

[0099] The unified attention weights are generated by 5x5 convolution fusion , 、 are the fused optical modal spatial attention weights and the fused sonar modal spatial attention weights, respectively.

[0100] Subsequently, bidirectional enhancement is performed, and the multi-modal features exchange attention weights. The optical features are modulated by the sonar attention map to strengthen the depth information region provided by the sonar. The sonar features are modulated by the optical attention map to optimize the edge information of the texture-rich region. The specific representation is as follows:

[0101] (15);

[0102] ;

[0103] wherein, represents a global shared space attention weight map, such as an occlusion edge, a high texture area, and the expression is , Significant regions jointly focused by the two modalities are marked; 、 The optical modal fusion feature map and the sonar modal fusion feature map are respectively.

[0104] In some embodiments, in S400, the multi-modal fusion feature is input into the detection head, and an underwater target detection result is output.

[0105] In S410, the detection head performs target class prediction and bounding box position regression on the multi-modal fusion feature, to generate an initial predicted bounding box and a class probability distribution.

[0106] In S420, based on the initial predicted bounding box and the class probability distribution, the bounding box position and the class probability are optimized and adjusted in combination with a modulation coefficient; wherein the modulation coefficient satisfies: when x≤μ-0.1, f(x)=1; when μ-0.1<x<μ, ; when x≥μ, f(x)=0.1. f(x) represents the modulation coefficient, x is the sample quality, the sample quality is the IoU value of the predicted box and the real box, μ is a dynamic IoU threshold, and the dynamic IoU threshold is the average value of the IoU values of all predicted bounding boxes and real boxes in the current batch;

[0107] In S430, after multiple rounds of iterative optimization, the final underwater target class and accurate bounding box position information are output.

[0108] The detection head VSAFDet of the present application is designed based on a single-stage target detector YOLOv8. Although the original CIoU loss function of the detection head is suitable for general detection scenarios, it is difficult to adapt to underwater detection tasks under harsh environmental conditions. The CIoU loss function adopts a unified penalty mechanism to apply the same optimization strategy to all predicted results. However, in the underwater environment, the optical and sonar images have inherent defects such as blurred target edges and partial occlusion, which result in the IoU values of a large number of predicted bounding boxes being generally at a low level. Therefore, the excessive punishment of CIoU on these low-quality samples will lead to unstable gradients affecting model convergence and loss of useful information. In view of the above challenges, the bounding loss of the regression box is improved in the present embodiment, and a sliding loss is introduced to solve the balance problem between simple samples and difficult samples. The formula is expressed as:

[0109] (17)

[0110] where f(x) is the modulation coefficient, x is the prediction box quality, that is, the prediction box and the real box IoU value, and μ represents the dynamic IoU threshold, which is the average value of all prediction boundary boxes and real IoU, and its formula expression is:

[0111] (18);

[0112] where, 、 are the i-th prediction boundary box and the i-th real boundary box, and N is the number of real boundary boxes in the current batch. The sliding loss can dynamically adjust the threshold to adaptively distinguish positive and negative samples, and differentially weight according to the sample difficulty. This design makes the model pay more attention to critical samples with learning value during training, thereby improving the convergence speed of model training and the recall rate of low-quality underwater targets.

[0113] The present application constructs an underwater multi-modal target detection data set UMOD, which provides a benchmark platform for technical verification.

[0114] Data acquisition: In a 7m*6m*5m pool, a ROV platform equipped with a Blue View M900 two-dimensional forward-looking sonar (130° field of view, 6.25cm resolution) and a Zed2 underwater binocular optical camera (1280*800 / 640*400 resolution, 60fps) is used for synchronous acquisition.

[0115] Data processing: The optical data and sonar data are time-stamped at the same time to ensure consistency in the time dimension. The video data is frame-sampled at an interval of 5 frames, and image pairs containing 9 types of targets including cages, frames, hooks, anchors, tires, rov, plastic buckets, fish and oil drums are selected. After redundancy screening and standardization processing, 4000 pairs of strictly calibrated optical-sonar images are finally obtained.

[0116] Data division: The data set is divided into training set, validation set and test set in the ratio of 7:2:1. The number of each category is shown in Figure 7 .

[0117] To verify the effectiveness of VSAFDet, the present application carries out systematic ablation experiments and comparative experiments.

[0118] Experimental setup: The experiment was conducted on a server equipped with three Nvidia A100 GPUs, with a software environment of Python 3.11, PyTorch 2.3.0, and CUDA 12.2. The model was pre-trained for 10 rounds before training, followed by 25 rounds of training, with a batch size of 48, a learning rate of 0.001, and an SGD optimizer. All comparative models used the same training strategy to ensure fairness.

[0119] Evaluation metrics: The average recall R, precision mAP, average precision mAP@0.5 with an IoU threshold of 0.5, parameter quantity Params, and computational complexity GFLOPs were used for evaluation.

[0120] Ablation experiment results: As shown in Table 1, Group 0 (single optical modal YOLOv8) mAP@0.5 was 91.8%. After introducing the double-branch structure, simple Add or Concat fusion (Group 1 and 2) performance decreased instead. Group 3 using VSAF module reached 93.4%, an increase of 1.6% compared to the baseline. Group 4 introducing VSAFDet backbone network (DSC2f and ENSPP) further improved to 94.1%, verifying the effectiveness of each module.

[0121] Table 1: Ablation experiment analysis results;

[0122]

[0123] Comparison experiment results: As shown in Table 2, VSAFDet achieved mAP@0.5 of 94.1% on the UMOD dataset, significantly better than single-modal detectors such as YOLOv8 (91.8%), YOLOv9 (90.7%), YOLOv10 (85.6%), YOLOv11 (90.9%), and YOLOv12 (89.9%), as well as underwater specialized detectors such as UGC-YOLO (92.5%) and LFN-YOLO (92.2%). In terms of model efficiency, VSAFDet achieved a good balance between precision and efficiency with 4.4M parameters and 11.1 GFLOPs.

[0124] Table 2: Performance comparison of different target detection models on the UMOD dataset;

[0125]

[0126] Cross-modal fusion module comparison: As shown in Table 3, compared with CCIFNet (91.0%mAP@0.5) designed for RGB-T task, ICAFusion (88.6%), DEYOLO (89.2%) and YOLOrs (90.1%) designed for remote sensing images, VSAFDet (94.1%) shows significant advantages. This is due to the VSAF module specially designed for underwater optical-sonar modal characteristics, which can effectively handle modal heterogeneity and spatial mismatch problems.

[0127] Table 3: Comparison results of different methods on UMOD dataset.

[0128]

[0129] Generalization verification: As shown in Table 4 and Table 5, VSAFDet also performs well on the land RGB-IR dataset DroneVehicle and FLIR. On the DroneVehicle dataset, the mAP@0.5 of VSAFDet reaches 80.6%, which is better than ICAFusion (79.1%) and YOLOv12 (76.2%). On the FLIR dataset, it reaches 70.2%, verifying the cross-domain generalization ability of the method.

[0130] Table 4: Comparison results of different methods on DroneVehicle.

[0131]

[0132] Table 5: Comparison results of different methods on FLIR dataset.

[0133]

[0134] In summary, the VSAFDet network proposed in the present application can be deployed on underwater robot platforms and widely applied in fields such as ocean resource exploration, underwater infrastructure detection, marine biodiversity survey, etc. The method effectively solves the target detection problem in complex underwater environments, improves the detection accuracy and robustness, and has important application value and market prospect.

[0135] The embodiment of the present application also provides a target recognition system based on sonar image assisted optical image, comprising:

[0136] At least one processor;

[0137] At least one memory for storing at least one program;

[0138] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0139] The contents in the above method embodiments are applicable to this embodiment, the function implemented in this embodiment is the same as that in the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments, which will not be repeated here.

[0140] The embodiment of the application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0141] It can be understood that the contents in the above method embodiments are applicable to the device embodiment, the function implemented in the device embodiment is the same as that in the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0142] The embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0143] It can be understood that the contents in the above method embodiments are applicable to the storage medium embodiment, the function implemented in the storage medium embodiment is the same as that in the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0144] The embodiment of the application further provides a computer program product, which comprises a computer program or computer instructions, the computer program or computer instructions are stored in a memory, a processor of a computer device reads the computer program or computer instructions from the memory, and the processor executes the computer program or computer instructions to enable the computer device to execute the above method.

[0145] It can be understood that the contents in the above method embodiments are applicable to the program product embodiment, the function implemented in the program product embodiment is the same as that in the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0146] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0147] Those of ordinary skill in the art will appreciate that all or some steps, systems of the above-disclosed methods can be implemented as software, firmware, hardware, or suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a micro-processing unit, or implemented as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on computer readable media, which can comprise computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, it should be appreciated by those skilled in the art that computer storage media generally can refer to any medium that can be used to store desired information and that can be accessed by a computer. Moreover, those skilled in the art will appreciate that the term communication media generally can refer to any medium that can be used to carry desired information and which can be accessed by a computer.

[0148] The device embodiments described above are merely illustrative and the units described as separate components can or can not be physically separate, i.e. can be located in one place or distributed over a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments.

Claims

1. A target recognition method based on sonar image-assisted optical image, characterized in that, The method includes the following steps: S100: Acquire underwater optical images, sonar images, and a trained underwater optical-sonar attention fusion detection network; wherein, the underwater optical-sonar attention fusion detection network includes a dual-stream multi-scale feature extraction backbone network, an optical-sonar adaptive feature fusion module, and a detection head; S200: The underwater optical image and sonar image are input into a dual-stream multi-scale feature extraction backbone network. The dynamic serpentine convolution module in the dual-stream multi-scale feature extraction backbone network extracts features from the optical image and sonar image respectively, generating preliminary optical feature maps and preliminary sonar feature maps. The enhanced spatial pyramid pooling module in the dual-stream multi-scale feature extraction backbone network performs multi-scale feature fusion on the preliminary optical feature maps and preliminary sonar feature maps, generating optical multi-scale feature maps and sonar multi-scale feature maps. S300, the multi-scale features are processed by the optical-sonar adaptive feature fusion module to generate enhanced multimodal fusion features; S400, input the multimodal fusion features into the detection head, and output the underwater target detection result; In S200, the step of performing multi-scale feature fusion on the preliminary optical feature map and the preliminary sonar feature map through the enhanced spatial pyramid pooling module in the dual-stream multi-scale feature extraction backbone network to generate optical multi-scale feature maps and sonar multi-scale feature maps includes: S221, perform adaptive global max pooling and adaptive global average pooling dual-branch processing on the input preliminary feature map. The actual size of the pooling window for both adaptive global max pooling and adaptive global average pooling is determined by calculating the actual height and actual width of the pooling window. The actual height of the pooling window is the result of rounding up the ratio of the height of the input feature map to the height of the output feature map. The actual width of the pooling window is the result of rounding up the ratio of the width of the input feature map to the width of the output feature map. S222, Perform multi-layer local feature extraction on the preliminary feature map; S223, the adaptive global max pooling result, the adaptive global average pooling result, and the multi-layer local feature extraction result are spliced ​​and fused in the channel dimension to generate a multi-scale feature map; In S300, the optical-sonar adaptive feature fusion module includes an optical-sonar cross-modal attention submodule and an optical-sonar pyramid attention submodule. The process of processing the multi-scale features through the optical-sonar adaptive feature fusion module to generate enhanced multi-modal fusion features includes: S310, The optical multi-scale feature map and the sonar multi-scale feature map are input into the optical-sonar cross-modal attention submodule, and preliminary enhanced optical features and preliminary enhanced sonar features are generated through channel attention mechanism and multi-scale feature fusion. S320, the initially enhanced optical features and the initially enhanced sonar features are input into the optical-sonar pyramid attention submodule, a unified attention weight is generated through multi-scale spatial attention fusion, and the optical features and sonar features are bidirectionally enhanced based on the unified attention weight to obtain enhanced multimodal fusion features.

2. The method according to claim 1, characterized in that, In S200, the dynamic serpentine convolution module is constructed based on the dynamic serpentine convolution DSConv. The process of extracting features from the optical image and sonar image using the dynamic serpentine convolution module in the dual-stream multi-scale feature extraction backbone network to generate preliminary optical and sonar feature maps includes: S211, determine the center coordinates (x,y) of the 3×3 standard convolution kernel and the fixed sampling coordinate set K={(x-1,y-1),(x-1,y),(x-1,y+1),(x,y-1),(x,y),(x,y+1),(x+1,y-1),(x+1,y),(x+1,y+1)}; S212, introduces a deformation offset Δ, based on the previous sampling point. Position iteration calculation of the current sampling point The calculation formula is: Where c represents the horizontal distance from the sampling point to the center of the grid. It is the offset learned through input features; S213, the optical image and the sonar image are convolved by the dynamic snake convolution DSConv to generate preliminary optical feature maps and preliminary sonar feature maps respectively.

3. The method according to claim 1, characterized in that, In S310, the generation of initially enhanced optical features and initially enhanced sonar features through channel attention mechanism and multi-scale feature fusion includes: S311, regarding the optical multi-scale feature map Harmony sonar multi-scale feature map Global average pooling is performed separately to generate optical global features and sonar global features; S312, The optical global features and sonar global features are respectively input into a multilayer perceptron to generate optical channel attention weights. Sonar channel attention weights ; S313, regarding the optical multi-scale feature map Harmony sonar multi-scale feature map Group convolution processing is performed separately to extract multi-scale detail features. , ; S314 will display optical multi-scale details. Harmony sonar multi-scale detail features The concatenation is performed along the channel dimension, and a cross-modal fusion feature map is generated through a 3×3 convolution operation. ; S315, based on cross-modal fusion feature map The relationship between the minimum dimension and the kernel size is used to dynamically select whether to use 1×1 convolution or global average pooling to generate global context information. ; S316, the two modes enhance the current mode by utilizing each other's attention mechanisms, generating initially enhanced optical features. and preliminary enhancements in sonar signatures : ； ； Where σ represents the Sigmoid function, This indicates an element-wise multiplication operation. This represents matrix multiplication.

4. The method according to claim 1, characterized in that, In S400, the step of inputting the multimodal fusion features into the detection head and outputting the underwater target detection result includes: S410, the detection head performs target category prediction and bounding box position regression on the multimodal fusion features to generate an initial predicted bounding box and category probability distribution; S420. Optimally adjust the bounding box position and class probability based on the initial predicted bounding box and class probability distribution, in combination with a modulation coefficient; wherein, the modulation coefficient satisfies: when x ≤ μ - 0.1, f(x) = 1; when μ - 0.1 < x < μ, ; when x ≥ μ, , f(x) represents the modulation coefficient, x is the sample quality, the sample quality is the IoU value between the predicted box and the ground truth box, μ is the dynamic IoU threshold, and the dynamic IoU threshold is the average of the IoU values between all predicted bounding boxes and the ground truth boxes in the current batch; After multiple rounds of iterative optimization, the S430 outputs the final underwater target category and precise bounding box location information.

5. A target recognition system based on sonar image-assisted optical image, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 4.

Images