An underwater target detection and recognition method based on acoustic-optical fusion

Abstract

This invention discloses an underwater target detection and recognition method based on acoustic-optical fusion, comprising the following steps: acquiring optical and acoustic images of the same underwater target; employing a style transfer-based data augmentation method to obtain a transferred image with the acoustic image style; using an optical image and its corresponding acoustic image transferred image as a set of acoustic-optical fusion data to construct an acoustic-optical fusion dataset; constructing a composite-connected reparameterizable backbone network; training the composite-connected reparameterizable backbone network using the acoustic-optical fusion dataset to obtain a usable underwater target detection and recognition model; and using the usable underwater target detection and recognition model to achieve underwater target detection.

Description

Technical Field

[0001] This invention belongs to the field of underwater target detection technology, specifically an underwater target detection and recognition method based on acoustic-optical fusion. Background Technology

[0002] Vision-based sensors are widely used in autonomous underwater vehicles. In environmental monitoring or geological surveys, the value of optical sensors lies in their ability to contain high-detail and color information, which are crucial for underwater applications. High-resolution data provided by optical cameras is particularly significant in applications such as coral reef monitoring, ship inspection, motion estimation, and archaeological surveys.

[0003] However, optical imaging is dependent on the surrounding environment; light attenuation and water turbidity can severely affect the shooting range and performance of optical sensors. Another problem is that in underwater environments, submarines stir up silt, which also affects the performance of optical sensors. In contrast, sonar can adapt to dim environments and has a much wider detection range. Different sonar systems can provide different information about the seabed and sunken objects. However, compared to optical cameras, sonar has lower resolution and does not include color.

[0004] A multimodal target detection algorithm combining acoustic and optical data has been proposed, but this algorithm has the following drawbacks:

[0005] (1) Current multimodal target detection algorithms ignore the multi-scale problem of target images during feature extraction;

[0006] (2) Current multimodal target detection algorithms only fuse a single feature layer when fusing acoustic and optical data, resulting in low stability of multi-scale detection tasks for small targets;

[0007] (3) The current multimodal target detection algorithm uses a direct serial fusion method, which ignores the differences in feature information of each mode under different underwater lighting conditions, resulting in insufficient detection results;

[0008] (4) Current multimodal target detection algorithms focus on sample magnification and image mechanisms, without considering the real environment of the target, such as the obstruction of mud and sand, the omission of target parts, the various states of the target, the shadows and reflective objects of real sonar data, and the lack of underwater artificial datasets with acoustic and optical support. Summary of the Invention

[0009] Purpose of the invention: To address the problem of low stability in detecting small targets in existing multimodal target detection algorithms, and to solve the problem of insufficient detection results in existing multimodal target detection algorithms, this invention proposes an underwater target detection and recognition method based on acoustic-optical fusion. By utilizing an integrated model of optical and acoustic sensors, the accuracy of underwater environmental data collection is improved.

[0010] Technical solution: An underwater target detection and recognition method based on acoustic-optical fusion, comprising the following steps:

[0011] Optical and acoustic images of the same underwater target are acquired. A style transfer-based data augmentation method is used to obtain a transferred image with the style of the acoustic image. An optical image and the transferred image with the style of the corresponding optical image are used as a set of acoustic-optical fusion data to construct an acoustic-optical fusion dataset.

[0012] Construct a reparameterizable backbone network with composite connections;

[0013] A reparameterizable backbone network with composite connections was trained using an acoustic-optical fusion dataset to obtain a usable underwater target detection and recognition model.

[0014] Underwater target detection is achieved by using available underwater target detection and recognition models;

[0015] The composite-connected reparameterizable backbone network includes a backbone layer, a bottleneck layer, a first detection head, a second detection head, and a third detection head.

[0016] The backbone layer includes a first feature extraction network for extracting feature data from optical images and a second feature extraction network for extracting feature data from transferred images with acoustic image style. The first and second feature extraction networks have the same structure, both consisting of MobileOne blocks and several convolutional layers connected in series. The feature data extracted by the last three convolutional layers in the first and second feature extraction networks are taken as the output of the backbone layer.

[0017] The bottleneck layer includes a first Fusion module, a second Fusion module, a third Fusion module, a first ELAN module, a second ELAN module, a third ELAN module, a fourth ELAN module, and an SPPCSPC module;

[0018] The feature data extracted by the third-to-last convolutional layer of the first feature extraction network and the feature data extracted by the third-to-last convolutional layer of the second feature extraction network are input into the first Fusion module;

[0019] The feature data extracted by the penultimate convolutional layer of the first feature extraction network and the feature data extracted by the penultimate convolutional layer of the second feature extraction network are input into the second Fusion module;

[0020] The feature data extracted by the penultimate convolutional layer of the first feature extraction network and the feature data extracted by the penultimate convolutional layer of the second feature extraction network are input into the third Fusion module;

[0021] The fused feature data output from the first Fusion module, the second Fusion module, and the third Fusion module are used as inputs to the first ELAN module, the second ELAN module, and the SPPCSPC module, respectively.

[0022] The feature data output by the first ELAN module and the feature data output by the second ELAN module are taken as the input of the third ELAN module;

[0023] The feature data output by the third ELAN module and the feature data output by the SPPC module are taken as the input of the fourth ELAN module.

[0024] The feature data output by the second ELAN module is input to the first detection head, and the first detection head outputs the first prediction box.

[0025] The feature data output by the third ELAN module is input to the second detection head, and the second detection head outputs the second prediction box.

[0026] The feature data output by the fourth ELAN module is input to the third detection head, and the third detection head outputs the third prediction box.

[0027] Furthermore, the method of training a composite-connected, reparameterizable backbone network using an acoustic-optical fusion dataset to obtain a usable underwater target detection and recognition model specifically includes:

[0028] Step 1: By minimizing the loss function LOSS as shown in Equation (21), the optimal network parameters of the composite-connected reparameterizable backbone network are obtained, thereby obtaining the trained composite-connected reparameterizable backbone network.

[0029] LOSS=ω box Loss box +ω obj Loss obj +ω cls Loss cls (twenty one)

[0030] In the formula, Loss box Let ω represent the loss function for detecting bounding boxes. box The weights of the loss function for detecting bounding boxes are represented by Loss. obj Represents the object loss function, ω obj The weights of the object loss function, Loss cls Let ω represent the binary cross-entropy loss function. cls The weights represent the weights of the binary cross-entropy loss function;

[0031] The detection box loss function Loss box , represented as:

[0032]

[0033]

[0034] Among them, s 2 This represents an s×s grid, where each grid generates B candidate boxes. The function determines whether there is an object in candidate box j in grid i. If there is, it outputs 1; otherwise, it outputs 0. P∩G represents the intersection of the ground truth bounding box and the predicted bounding box, and P∪G represents the union of the ground truth bounding box and the predicted bounding box. The ground truth bounding box is the one that comes with the acousto-optic fusion data in the acousto-optic fusion dataset.

[0035] The object loss function Loss obj , represented as:

[0036]

[0037] Where Δ represents distance loss and Ω represents shape loss;

[0038] The binary cross-entropy loss function Loss cls , represented as:

[0039]

[0040] In the formula, p(c) represents the probability of predicting that the object belongs to category c. gt (c) represents the probability that the object is actually of category c; classes represents the set of categories;

[0041] Step 2: Reparameterize the MobileOne block in the reparameterizable backbone network with composite connections to obtain a usable underwater target detection and recognition model.

[0042] Furthermore, the operations performed in the first Fusion module, the second Fusion module, and the third Fusion module are the same, as shown below:

[0043]

[0044]

[0045]

[0046]

[0047] Among them, C i and C i' represents the feature data output from layer i of the first feature extraction network and the feature data output from layer i of the second feature extraction network, respectively. ⊙ is the Hadamard product, Conv(·) represents a 3×3 convolutional layer, and σ(·) represents a residual layer. and Denotes activation features, AP(·) represents average pooling operation, MP(·) represents max pooling operation, and B i Represents the residual characteristic, f i This indicates the final fusion feature.

[0048] Furthermore, the method of using style transfer-based data augmentation to obtain a transferred image with an acoustic image style specifically includes:

[0049] Using deep neural networks, underwater targets in optical images are labeled and annotated to obtain optical images with real bounding boxes and labels, as well as corresponding annotation files;

[0050] The optical image with a real bounding box and markers is subjected to automatic color level processing, color inversion processing and grayscale adjustment in sequence to obtain the processed optical image;

[0051] Using the background of the acoustic image as the style image and the processed optical image as the content image, the style image and the content image are input into the style transfer network to obtain a transferred image with the style of the acoustic image.

[0052] The style transfer network is trained using style images and content images by minimizing the loss function shown in equation (2).

[0053] L(a,f,p)=α×L style (p,f)+β×L content (a,f) (2)

[0054] Where L(a,f,p) represents the loss function, L style (p,f) represents a transferred image f with an acoustic image style that is more similar in style to the background p of the acoustic image. content (a,f) represents a transferred image f with an acoustic image style that is more similar to an optical image a in terms of content; f represents a transferred image with an acoustic image style, p represents the background of the acoustic image, and a represents the optical image.

[0055] Furthermore, the optical image used to construct the acousto-optic fusion data is processed according to the following steps:

[0056] The processed optical image is subjected to random Gaussian blur and random gamma adjustment to obtain the optical image used to construct the acousto-optic fusion data.

[0057] Furthermore, the distance loss Δ is expressed as:

[0058]

[0059]

[0060]

[0061] ξ=2-Λ (16)

[0062]

[0063]

[0064]

[0065] Among them, (c w ,c h ) represents the width and height of the minimum bounding matrix between the ground truth bounding box and the predicted bounding box; Represents the center coordinates of the actual bounding box. Indicates the center coordinates of the prediction box; c h σ represents the height distance between the center point of the ground truth bounding box and the predicted bounding box; max(·) represents the maximum output value, and min(·) represents the minimum output value.

[0066] Furthermore, the shape loss Ω is expressed as:

[0067]

[0068]

[0069]

[0070] Where w and h represent the width and height of the prediction box, respectively. θ represents the width and height of the true bounding box, respectively, and θ represents the degree of concern for shape loss.

[0071] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0072] (1) This invention improves the correlation between different modalities by proposing a composite-connected reparameterizable backbone network, and introduces a fusion module to improve the efficiency and accuracy of fusion. It also solves the problem of lack of dataset by using fast style transformation to generate artificial datasets. The composite-connected reparameterizable backbone network is trained with sufficient sample size of acoustic-optical fusion data, thereby improving the detection performance of multimodal underwater algorithms.

[0073] (2) The method of the present invention can obtain the best target detection results under different lighting conditions, which means that it can have a lower false detection rate and false detection rate in actual underwater scenarios, and can better guarantee the quality of underwater target detection, achieving a good balance between accuracy and speed.

[0074] (3) The method of the present invention can effectively detect dense and overlapping targets;

[0075] (4) The method of the present invention reduces the number of parameters by reparameterization, so it can be applied to portable graphics computing cards, solving the problem that it cannot be applied to industrial sites due to the large number of parameters. The method of the present invention has better overall applicability. Attached Figure Description

[0076] Figure 1 This is a flowchart of an underwater target detection and recognition method based on acoustic-optical fusion.

[0077] Figure 2 A flowchart for generating an acoustic-optical fusion dataset;

[0078] Figure 3 A schematic diagram of a reparameterizable backbone network with composite connections;

[0079] Figure 4 This is a structural diagram of the ELAN module, SPPC module, and Fusion module.

[0080] Figure 5 This is a structural diagram of the MobileOne block;

[0081] Figure 6 This is a schematic diagram comparing the results of the method of this invention with the YOLOv7 target detection algorithm;

[0082] Figure 7 This is a schematic diagram illustrating the effect of the method of the present invention on detecting dense and overlapping targets. Detailed Implementation

[0083] The technical solution of the present invention will now be further described in conjunction with the accompanying drawings and embodiments.

[0084] This embodiment proposes an underwater target detection and recognition method based on acoustic-optical fusion, such as... Figure 1 As shown, the main steps include:

[0085] Step 1: Optical images of the underwater target are acquired using a green laser underwater camera, while acoustic images of the underwater target are acquired simultaneously using sonar. To address the issue of scarce samples, a style transfer-based data augmentation method is used to obtain transferred images with the acoustic image style. An acoustic-optical fusion dataset is constructed by grouping an optical image with its corresponding acoustic image style. For example... Figure 2 As shown, the specific operations include:

[0086] S110: The underwater scene is captured by a green laser underwater camera to obtain a video source. Optical images containing underwater targets are extracted from the video source. Using a deep neural network, the underwater targets in the optical images are marked and annotated to obtain optical images with real bounding boxes and markings, as well as corresponding annotation files. For ease of representation, the optical images mentioned in the following steps refer to optical images with real bounding boxes and markings. Each optical image has a corresponding annotation file.

[0087] S120: Enhance each optical image to obtain a processed optical image; the enhancement process mentioned in this step refers to enhancing the shadows and features of underwater targets in the optical image, and the specific operations include:

[0088] The optical image is processed sequentially with automatic color levels, color inversion, and grayscale adjustment, as shown below:

[0089]

[0090] Where Autolevel(·) represents automatic color levels, I(·) represents color inversion, g(·) represents the gamma function, and γ represents the threshold of the gamma function. This represents the processed optical image.

[0091] S130: Using the background of the acoustic image as the style image and the processed optical image as the content image, the style image and the content image are input into the style transfer network to obtain a transferred image with the style of the acoustic image. This transferred image with the style of the acoustic image is more similar to the background of the acoustic image in terms of style and more similar to the processed optical image in terms of content.

[0092] The style transfer network mentioned above is trained using style images and content images, minimizing the loss function shown in equation (2);

[0093] L(a,f,p)=α×L style (p,f)+β×L content (a,f) (2)

[0094] Where L(a,f,p) represents the loss function, L style (p,f) represents a transferred image f with an acoustic image style that is more similar in style to the background p of the acoustic image. content (a,f) indicates that the transferred image f with an acoustic image style is more similar to the optical image a in terms of content; f represents the transferred image with an acoustic image style, p represents the background of the acoustic image, and a represents the optical image;

[0095] S140: In order to simulate optical images acquired in a dim underwater environment and to test the stability of the subsequent reparameterizable backbone network, the processed optical images will be subjected to random Gaussian blurring and random gamma value adjustment to obtain the final optical images used to construct the acousto-optic fusion dataset.

[0096] S150: Normalize the transferred image with acoustic image style to the same numerical range as the optical image, i.e. [0,1]. Then, take an optical image and the corresponding transferred image with acoustic image style as a group to construct the final acoustic-optical fusion dataset.

[0097] Constructing annotated image datasets is typically done manually, which is very time-consuming. Furthermore, in real-world underwater environments, the variable nature of the environment prevents the acquisition of stable optical images, resulting in negative effects such as low light, defocusing, and shakiness. Consequently, the number of initially usable samples is limited. To address this scarcity, this step employs a style transfer-based data augmentation method to obtain stable transferred images with acoustic image styles. Simultaneously, this embodiment also simulates a dimly lit underwater environment by altering the gamma value of random optical images and applying Gaussian blur, ensuring higher accuracy for the subsequently trained, reparameterizable backbone network with composite connections, thereby improving detection and recognition performance.

[0098] Step 2: Construct a composite-connected reparameterizable backbone network. Use the acoustic-optical fusion dataset obtained in Step 1 to train the composite-connected reparameterizable backbone network to obtain a usable underwater target detection and recognition model.

[0099] The reparameterizable backbone network constructed in this embodiment comprises a backbone layer, a bottleneck layer, and a detection head.

[0100] The backbone layer processes the input optical image and the transferred image with an acoustic image style. Two feature extraction networks are used in the backbone layer to independently process the input optical image and the transferred image with an acoustic image style. The backbone layer outputs a 6-channel tensor, where three channels are RGB image feature data and the other three channels are feature data of the transferred image with an acoustic image style. Specifically, the feature extraction network of the backbone layer consists of a MobileOne block and several convolutional layers. Figure 3 As shown, the input to the MobileOne block is an optical image / a transferred image with an acoustic image style. The output of the MobileOne block is connected to the input of the first convolutional layer, the output of the first convolutional layer is connected to the input of the second convolutional layer, the output of the second convolutional layer is connected to the input of the third convolutional layer, and so on, forming a feature extraction network. As the number of layers increases, the feature map becomes smaller and smaller, while the number of features increases.

[0101] The MobileOne block in this embodiment uses a convolutional layer design, which is decomposed into depth and point layers. Furthermore, overparameterized branches are introduced to provide additional benefits. For example... Figure 5 As shown, the basic module is based on the 3×3 depthwise convolutions and 1×1 dot convolutions of MobileNet-V1, introducing reparameterizable skip connections and batch normalization, as well as branches with duplicated structures, and using ReLU as the activation function. In this embodiment, the overparameterization factor is set to 4, meaning there are four repeated 3×3 depthwise convolutional blocks.

[0102] The feature maps extracted by the last three convolutional layers of the two feature extraction networks (denoted as C3, C4, and C5, and C'3, C'4, and C'5) are taken as the output of the backbone layer, i.e., a 6-channel tensor.

[0103] The bottleneck layer includes three Fusion modules, four ELAN modules, and one SPPCSPC module. Specifically, the 6-channel tensor output from the backbone layer is used as the input to the three Fusion modules. Feature maps C3 and C'3 are input to the first Fusion module, feature maps C4 and C'4 are input to the second Fusion module, and feature maps C5 and C'5 are input to the third Fusion module. The fused feature maps output by the three Fusion modules (denoted as F3, F4, and F5) are used as the inputs to the first ELAN module, the second ELAN module, and the SPPCSPC module, respectively. Feature map P3 output by the first ELAN module and feature map P4 output by the second ELAN module are used as the input to the third ELAN module. Feature map N4 output by the third ELAN module and feature map P5 output by the SPPCSPC module are used as the input to the fourth ELAN module.

[0104] The feature map P3 output by the second ELAN module is input to the first detection head, the feature map N4 output by the third ELAN module is input to the second detection head, and the feature map N5 output by the fourth ELAN module is input to the third detection head. The first, second, and third detection heads are used to process detection at three different resolutions. For the feature maps at the three different resolutions, three prediction boxes are created respectively.

[0105] like Figure 4 As shown, the structure of the ELAN and SPPCSPC modules will now be explained.

[0106] The ELAN module is a convolutional aggregation architecture whose main purpose is to address the problem that the convergence of the model gradually deteriorates when scaling. By designing multiple branches, the model has richer gradient combinations, thereby better extracting feature information.

[0107] The SPPC module employs a feature pyramid structure, primarily to avoid image distortion caused by image cropping and scaling operations. It also addresses the issue of repetitive feature extraction related to graphs in convolutional networks, thereby improving the efficiency of generating candidate boxes.

[0108] like Figure 3 As shown, feature maps from different layers of the backbone layer are combined using the Fusion module; for example, C'3 and C3 are fused to generate F3. During training and testing, the sonar and image data here consist of a dataset composed of optical images adjusted with random Gaussian blur and random gamma values, and sonar images generated by inputting the optical images into the style transfer network. In practical applications, real-time acquired optical and sonar data are used.

[0109] In order to effectively combine image features from different modalities, this embodiment proposes the aforementioned Fusion module, and chooses to perform internal fusion after the base layer. The operation performed in the Fusion module is explained below.

[0110] like Figure 4 As shown, two modal features C are obtained through the following operations. i and C' i Feature representation after fusion.

[0111]

[0112]

[0113]

[0114]

[0115] Among them, C i and C' i represent the feature data output from layer i of the first feature extraction network and the feature data output from layer i of the second feature extraction network, respectively. ⊙ is the Hadamard product, Conv(·) represents a 3×3 convolutional layer, and σ(·) represents a residual layer. and Denotes activation features, AP(·) represents average pooling operation, MP(·) represents max pooling operation, and B i Represents the residual characteristic, f i This indicates the final fusion feature.

[0116] This embodiment chooses to perform internal fusion after the backbone layer, which can enhance activation while retaining most of the features of the multimodal data. Since the fused feature map retains more useful information, the network can extract better deep features, which in turn improves the final accuracy.

[0117] The composite-connected reparameterizable backbone network was trained using an acousto-optic fusion dataset. The loss function used for training the composite-connected reparameterizable backbone network was a weighted sum of the detection box loss function, the object loss function, and the class loss function: the detection box loss function measures the error of the predicted box in terms of coordinate localization; the object loss function reflects the confidence error of the predicted box; and the class loss function reflects the error caused by the predicted box's incorrect prediction of the target category.

[0118] In this example, the detection box loss function is expressed as the intersection-over-union (IoU) loss of the average ground truth and predicted boxes:

[0119]

[0120]

[0121] Among them, s 2 This represents an s×s grid, where each grid generates B candidate boxes. The function determines whether there is an object in candidate box j within grid i; it outputs 1 if there is, and 0 otherwise. P∩G represents the intersection of the ground truth bounding box and the predicted bounding box, and P∪G represents the union of the ground truth bounding box and the predicted bounding box. The ground truth bounding boxes are inherent to the audio-visual fusion data in the audio-visual fusion dataset. This example uses the binary cross-entropy loss function to implement the class loss function, expressed as:

[0122]

[0123] In the formula, p(c) represents the probability of predicting that the object belongs to category c. This represents the probability that the object actually belongs to category c.

[0124] This embodiment uses the SIoU loss function as the object loss function. The SIoU loss function introduces the vector angle between the ground reality box and the predicted box to redefine the correlation, and includes four parts: angle loss, distance loss, shape loss, and overlap (IoU) loss.

[0125] The angle loss is defined by the following formula.

[0126]

[0127] Among them, c h σ represents the height distance between the center point of the ground truth bounding box and the predicted bounding box, and σ represents the distance between the center points of the ground truth bounding box and the predicted bounding box.

[0128]

[0129]

[0130] in, Indicates the center coordinates of the actual bounding box. This represents the center coordinates of the prediction box. max(·) represents the maximum output value, and min(·) represents the minimum output value.

[0131] Distance loss is defined by the following formula.

[0132]

[0133]

[0134]

[0135] ξ=2-Λ (16)

[0136] Among them, (c w ,c h ) represents the width and height of the minimum bounding matrix between the true bounding box and the predicted bounding box.

[0137] The shape loss is defined by the following formula.

[0138]

[0139]

[0140]

[0141] Where w and h represent the width and height of the prediction box, respectively. θ represents the width and height of the actual bounding box, respectively, and θ represents the degree of concern regarding shape loss.

[0142] The overlap (IoU) loss is expressed as:

[0143]

[0144] Where A represents the intersection of the ground truth bounding box and the predicted bounding box, and B represents the union of the ground truth bounding box and the predicted bounding box.

[0145] In summary, the object loss function, i.e., the SIoU loss, is composed of the following formula:

[0146]

[0147] In summary, the network's loss function is a weighted sum of three parts:

[0148] LOSS=ω box Loss box +ω obj Loss obj +ω clsLoss cls (twenty one)

[0149] In this embodiment, the weight parameter is selected as: ω box =0.05, ω obj =1, ω cls =0.025.

[0150] like Figure 5 As shown, the MobileOne block in the reparameterizable backbone network with composite connections can be reparameterized, i.e., other redundant branches in the MobileOne model can be removed. During training, the potential of the multi-branch model is fully utilized to learn parameters, while during inference, the high computational speed of the branchless model can be leveraged. Specifically:

[0151] In a trained, reparameterizable backbone network with composite connections, the structure of the MobileOne block is identified and analyzed. Typically, this includes multiple convolutional layers, batch standard layers, and activation functions.

[0152] Folded convolutional layers and batch standard layers. For a convolutional kernel size of K and an input channel size of C. in The output channel size is C out The convolutional layer, with its weight matrix represented as... The bias is represented as The batch standard layer contains the cumulative mean μ, cumulative standard deviation σ, scale γ, and bias β. The convolutional layer and the immediately following batch standard layer are folded into a single convolutional layer. For the weights... and bias The calculations are as follows:

[0153] and

[0154] The batch standard layer in the skip connection is folded into an equivalent 1×1 convolutional kernel. This can be achieved by combining the parameters of the batch standard layer with the parameters of the 1×1 convolutional kernel.

[0155] In a multi-branch structure, the weights and biases of each branch are summed to obtain the weights and biases of the final convolutional layer after merging the branches. For the convolutional layer weights W and biases b during inference, we have:

[0156]

[0157] Where M is the number of branches.

[0158] The redundant branches, folded convolutional layers, and batch standard layers in the original MobileOne block are replaced with a simplified structure after reparameterization.

[0159] Step 3: Conduct ablation and comparative experiments on the composite-connected reparameterizable backbone network to analyze its performance.

[0160] This embodiment uses multi-class average precision (mAP) based on overlap IoU to evaluate the model's performance metrics.

[0161] IoU represents the ratio of the overlap between the predicted bounding box detected and identified by the model and the actual target bounding box. The specific calculation formula is shown below.

[0162]

[0163] The prediction is considered correct when the IoU value is higher than a certain threshold.

[0164] Multi-class average precision (mAP) is related to AP, meaning that the average of APs for multiple classes can be used to obtain mAP.

[0165]

[0166] Where N represents the total number of categories.

[0167] mAP can reflect the recognition accuracy of each type of target and is used to evaluate the effect of multiple target detection and recognition. Obviously, the larger the AP and mAP, the better the performance of the designed model.

[0168] FPS represents the number of frames a model can process per second, measuring the computational speed of the designed model and the processing performance of the computing platform used. The calculation formula is shown below.

[0169]

[0170] Step 4: The reparameterizable backbone network with composite connections is synthesized onto the underwater data collection device to achieve real-time underwater target identification. In some embodiments, the underwater data collection device consists of a green laser underwater camera and a DIDSON dual-frequency identification sonar, which acquire optical and acoustic images of underwater targets in pairs. The device uses an embedded graphics computing card to run a model and detect underwater targets in real time. The identification results display the target bounding box and target type to the user.

[0171] This embodiment improves the accuracy of underwater target identification by leveraging the different advantages of optical information and sonar through deep learning.

[0172] To verify the underwater target recognition accuracy of the underwater target detection and recognition method based on acoustic-optical fusion proposed in this embodiment, the YOLOv7 target detection algorithm is used as a comparative example. The same underwater target is detected using both the method of this embodiment and the comparative method. Figure 6As shown, this embodiment has a higher underwater target recognition accuracy relative to the scale.

[0173] To verify that the underwater target detection and recognition method based on acoustic-optical fusion proposed in this embodiment can effectively detect dense and overlapping targets, underwater images with multiple overlapping underwater targets were selected for detection, such as... Figure 7 As shown, the method in this embodiment can effectively detect dense and overlapping targets.

Claims

1. A method for underwater target detection and recognition based on acoustic-optical fusion, characterized in that: Includes the following steps: Optical and acoustic images of the same underwater target are acquired. A style transfer-based data augmentation method is used to obtain a transferred image with the style of the acoustic image. An optical image and the transferred image with the style of the corresponding optical image are used as a set of acoustic-optical fusion data to construct an acoustic-optical fusion dataset. Construct a reparameterizable backbone network with composite connections; A reparameterizable backbone network with composite connections was trained using an acoustic-optical fusion dataset to obtain a usable underwater target detection and recognition model. Underwater target detection is achieved by using available underwater target detection and recognition models; The composite-connected reparameterizable backbone network includes a backbone layer, a bottleneck layer, a first detection head, a second detection head, and a third detection head. The backbone layer includes a first feature extraction network for extracting feature data from optical images and a second feature extraction network for extracting feature data from transferred images with acoustic image style. The first and second feature extraction networks have the same structure, both consisting of MobileOne blocks and several convolutional layers connected in series. The feature data extracted by the last three convolutional layers in the first and second feature extraction networks are taken as the output of the backbone layer. The bottleneck layer includes a first Fusion module, a second Fusion module, a third Fusion module, a first ELAN module, a second ELAN module, a third ELAN module, a fourth ELAN module, and an SPPCSPC module; The feature data extracted by the third-to-last convolutional layer of the first feature extraction network and the feature data extracted by the third-to-last convolutional layer of the second feature extraction network are input into the first Fusion module; The feature data extracted by the penultimate convolutional layer of the first feature extraction network and the feature data extracted by the penultimate convolutional layer of the second feature extraction network are input into the second Fusion module; The feature data extracted by the penultimate convolutional layer of the first feature extraction network and the feature data extracted by the penultimate convolutional layer of the second feature extraction network are input into the third Fusion module; The fused feature data output from the first Fusion module, the second Fusion module, and the third Fusion module are used as inputs to the first ELAN module, the second ELAN module, and the SPPCSPC module, respectively. The feature data output by the first ELAN module and the feature data output by the second ELAN module are taken as the input of the third ELAN module; The feature data output by the third ELAN module and the feature data output by the SPPC module are taken as the input of the fourth ELAN module. The feature data output by the second ELAN module is input to the first detection head, and the first detection head outputs the first prediction box. The feature data output by the third ELAN module is input to the second detection head, and the second detection head outputs the second prediction box. The feature data output by the fourth ELAN module is input to the third detection head, and the third detection head outputs the third prediction box; The method employing style transfer-based data augmentation to obtain a transferred image with an acoustic image style specifically includes: Using deep neural networks, underwater targets in optical images are labeled and annotated to obtain optical images with real bounding boxes and labels, as well as corresponding annotation files; The optical image with a real bounding box and markers is subjected to automatic color level processing, color inversion processing and grayscale adjustment in sequence to obtain the processed optical image; Using the background of the acoustic image as the style image and the processed optical image as the content image, the style image and the content image are input into the style transfer network to obtain a transferred image with the style of the acoustic image. The style transfer network is trained using style images and content images by minimizing the loss function shown in equation (2). L(a,f,p)=α×L style (p,f)+β×L content (a,f) (2) Where L(a,f,p) represents the loss function, L style (p,f) represents a transferred image f with an acoustic image style that is more similar in style to the background p of the acoustic image. content (a,f) represents a transferred image f with an acoustic image style that is more similar to an optical image a in terms of content; f represents a transferred image with an acoustic image style, p represents the background of the acoustic image, and a represents the optical image.

2. The underwater target detection and recognition method based on acoustic-optical fusion according to claim 1, characterized in that: The method of training a reparameterizable backbone network with composite connections using an acoustic-optical fusion dataset to obtain a usable underwater target detection and recognition model specifically includes: Step 1: By minimizing the loss function LOSS as shown in Equation (21), the optimal network parameters of the composite-connected reparameterizable backbone network are obtained, thereby obtaining the trained composite-connected reparameterizable backbone network. LOSS=ω box Loss box +ω obj Loss obj +ω cls Loss cls (21) In the formula, Loss box Let ω represent the loss function for detecting bounding boxes. box The weights of the loss function for detecting bounding boxes are represented by Loss. obj Represents the object loss function, ω obj The weights of the object loss function, Loss cls Let ω represent the binary cross-entropy loss function. cls The weights represent the weights of the binary cross-entropy loss function; The detection box loss function Loss box , represented as: Among them, s 2 This represents an s×s grid, where each grid generates B candidate boxes. The function determines whether there is an object in candidate box j in grid i. If there is, it outputs 1; otherwise, it outputs 0. P∩G represents the intersection of the ground truth bounding box and the predicted bounding box, and P∪G represents the union of the ground truth bounding box and the predicted bounding box. The ground truth bounding box is the one that comes with the acousto-optic fusion data in the acousto-optic fusion dataset. The object loss function Loss obj , represented as: Where Δ represents distance loss and Ω represents shape loss; The binary cross-entropy loss function Loss cls , represented as: In the formula, p(c) represents the probability of predicting that the object belongs to category c. gt (c) represents the probability that the object is actually of category c; classes represents the set of categories; Step 2: Reparameterize the MobileOne block in the reparameterizable backbone network with composite connections to obtain a usable underwater target detection and recognition model.

3. The underwater target detection and recognition method based on acoustic-optical fusion according to claim 1, characterized in that: The operations performed in the first Fusion module, the second Fusion module, and the third Fusion module are the same, as shown below: Among them, C i and C i ' represents the feature data output from layer i of the first feature extraction network and the feature data output from layer i of the second feature extraction network, respectively. ⊙ is the Hadamard product, Conv(·) represents a 3×3 convolutional layer, and σ(·) represents a residual layer. and Denotes activation features, AP(·) represents average pooling operation, MP(·) represents max pooling operation, and B i Represents the residual characteristic, f i This indicates the final fusion feature.

4. The underwater target detection and recognition method based on acoustic-optical fusion according to claim 1, characterized in that: The optical image used to construct the acoustic-optical fusion data is processed according to the following steps: The processed optical image is subjected to random Gaussian blur and random gamma adjustment to obtain the optical image used to construct the acousto-optic fusion data.

5. The underwater target detection and recognition method based on acoustic-optical fusion according to claim 2, characterized in that: The distance loss Δ is expressed as: Among them, c w The width of the minimum bounding matrix between the ground truth bounding box and the predicted bounding box; Represents the center coordinates of the actual bounding box. Indicates the center coordinates of the prediction box; c h σ represents the height distance between the center point of the ground truth bounding box and the predicted bounding box; max(·) represents the maximum output value, and min(·) represents the minimum output value.

6. The underwater target detection and recognition method based on acoustic-optical fusion according to claim 2, characterized in that: The shape loss Ω is expressed as: Where w and h represent the width and height of the prediction box, respectively. gt ,h gt θ represents the width and height of the true bounding box, respectively, and θ represents the degree of concern for shape loss.

Images