A method, medium and system for real-time target detection and classification based on sonar images

Abstract

The present invention provides a method, medium and system for real-time target detection and classification based on sonar images, which belongs to the technical field of sonar image target classification and recognition, including: dividing the URPC2021 sonar data set into a training set, a validation set and a test set in a ratio of 8:1:1, and establishing labels for the training data set including the target position, size and category. The training set and the validation set are input into the AC‑YOLOX network for training, and the Mosaic and Mixup data enhancement strategies are used to increase the data diversity. The training targets include target coordinates, confidence and category probability. The trained real-time target detection and classification model is used to input the sonar image acquired in real time, and the real-time target detection and classification results are output. The technical problem that the prior art is difficult to solve due to the co-existence of internal and external interference in the working environment of the sonar system, resulting in insufficient accuracy of the final underwater target real-time detection and classification is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of sonar image target classification and recognition, and in particular, relates to a sonar image-based target real-time detection and classification method, medium and system. Background Art

[0002] In the detection of seabed targets and ocean surveys, sonar is one of the most commonly used and effective sensors for underwater detection. In recent years, with the rapid development of single-stage target detection algorithms, they have gradually surpassed dual-stage target detection algorithms in detection accuracy. Therefore, in the field of underwater target detection, a trend has gradually formed to use single-stage target detection algorithms as research and application methods, and the detection effect of the YOLO algorithm is particularly outstanding.

[0003] However, the seabed environment is too complex, and there are many uncertain factors, such as external interference such as ocean environmental noise, ship self-noise, and seabed reverberation. These reverberations will be superimposed on the receiving end of the sonar transducer, making the target edge of the generated sonar image blurred and the details lost. In addition to external interference, the sonar system device itself will also produce some internal interference during the imaging process, such as the jitter of the sonar transducer itself, the instability of the sound wave signal when it is sent and received, etc. These internal and external interferences will coexist in the working environment of the sonar system, affecting the quality of the final generated sonar image. The sonar image itself also has the problems of few pixels, low resolution, low contrast, and blurred edges. These factors directly affect the research on real-time detection and classification of underwater targets based on sonar images. Summary of the invention

[0004] In view of this, the present invention provides a method, medium and system for real-time target detection and classification based on sonar images, which can solve the technical problem that the prior art is difficult to solve, that is, the final underwater target real-time detection and classification accuracy is insufficient due to the coexistence of internal and external interference in the working environment of the sonar system.

[0005] The present invention is achieved in that:

[0006] The present invention provides a method for real-time target detection and classification based on sonar images, comprising the following steps:

[0007] S10. Divide the URPC2021 sonar dataset into a training set, a validation set, and a test set in a ratio of 8:1:1, and establish labels according to the position, size, and category of the target in the sonar image as the training dataset;

[0008] S20. Improve the YOLOX network structure and build an AC-YOLOX network. Specifically, add the CBAM attention mechanism before the multiple residual structures of dark3, dark4, and dark5 of the YOLOX network structure, and add the ASFF module of adaptive feature fusion between Neck and Head.

[0009] S30, inputting the training set and the validation set into the AC-YOLOX network for training, and using Mosaic and Mixup data enhancement strategies to increase data diversity during the training process, wherein the input of the training is sonar image data, and the output of the training is the location coordinates, confidence, and category probability of the target;

[0010] S40, using the trained model to predict the test set, outputting the classification results, and calculating the evaluation indicators of the model, including accuracy, precision, recall rate and F1 score;

[0011] S50, adjusting the network structure, hyperparameters and training strategy of the AC-YOLOX network according to the evaluation index, repeating steps S20 to S50 until the model performance reaches the preset expected target, and saving it as the target real-time detection classification model;

[0012] S60: using the target real-time detection and classification model, inputting the sonar image acquired in real time, and outputting the target real-time detection and classification result.

[0013] The AC-YOLOX network includes input, Backbone, Neck and Head. In the Backbone part, AC-YOLOX uses the Focus structure to slice the image; uses multi-layer CBS convolution to extract features; uses the improved CSP_C structure to make the model pay more attention to identifying the target object and learn more features; uses SPP to extract features in different pooling layers, increase the receptive field, and integrate more information. In the Neck part, AC-YOLOX uses the PAFPN structure to fuse features of different scales. In the Head part, AC-YOLOX uses the Decoupled Head detection head, Anchor-free mechanism, Loss calculation, and SimOTA label matching strategy.

[0014] The input end adopts Mosaic and Mixup data enhancement strategies. The Mosaic data enhancement strategy is to splice 4 pictures by random scaling, cropping, and arrangement to form a new picture, which increases the size of the detection data set. The random scaling process also plays a role in increasing the number of small targets, which is conducive to the detection of small targets; the Mixup data enhancement strategy performs weighted fusion of different images to expand the training data set and enhance the target detection ability of the model.

[0015] Furthermore, the Backbone structure includes: Focus structure, CSP_C (cross-level partial network) structure, SPP (spatial pyramid pooling) structure. Among them: the Focus structure slices the image, and the obtained image is then subjected to a convolution operation, and finally a double-sampled feature map is obtained without information loss. It is equivalent to a special downsampling. By taking a value for every other pixel, a picture is converted into four complementary pictures, which increases the number of input channels by four times. This processing method not only improves the computing power, but also performs downsampling operations without losing information, thereby improving the performance of the model; dark2 uses the original CSP1_X, CSP1_X structure to divide the original input into two branches. One branch directly performs a CBS operation to halve the number of channels, and then a branch performs a CBS operation, and then passes through multiple residual structures, and then concats the two branches, so that the input and output of BottleneckCSP are the same size. This is to allow the model to learn more features of the sonar image, and finally perform another CBS operation; dark 3. CSP_C_X used in dark4 is improved from CSP1_X. The original input end is divided into two branches. One branch directly performs CBS operation to halve the number of channels, and then one branch performs a CBS operation, followed by a CBAM attention mechanism structure, and then multiple residual structures, and then concat the two branches, and finally perform another CBS operation; CSP2_C_X used in dark5 is improved from CSP2_X. The original input end is divided into two branches. One branch directly performs CBS operation to halve the number of channels, and then one branch performs a CBS convolution operation, followed by a CBAM attention mechanism structure, and then two CBS convolution operations, and then concat the two branches, and finally perform another CBS operation. Among them: the CBS structure includes: Conv convolution operation, BatchNormalization normalization, SiLU activation function; among them, the SPP structure: first perform a CBS operation, and then use 1×1, 5×5, 9×9, 13×13 convolution kernels to perform pyramid maximum pooling to achieve the fusion of local features and global features. The SPP structure can ensure the fixed length of the feature vector while maintaining feature diversity, thereby adapting to input data of different sizes.

[0016] Furthermore, the CBAM attention mechanism is added before the multiple residual structures of dark3, dark4, and dark5 of the YOLOX network structure. Specifically, the CSP structure in dark3 and dark4 adds the CBAM attention mechanism before the multiple residual structures, and the CSP structure in dark5 adds the CBAM attention mechanism after a CBS convolution operation. The CBAM attention mechanism includes: a channel attention mechanism and a spatial attention mechanism. Channel attention mechanism: The input feature map F (H×W×C) is subjected to global max pooling and global average pooling based on width and height respectively to obtain two 1×1×C feature maps, and then they are respectively sent to a two-layer neural network (MLP). The number of neurons in the first layer is C / r (r is the reduction rate), the activation function is Relu, and the number of neurons in the second layer is C. The two layers of the neural network are shared. Then, the features output by the MLP are subjected to an element-wise addition operation, and then a sigmoid activation operation is performed to generate the final channel attention mechanism feature map, that is, Finally, Perform element-wise multiplication operation with the input feature map F to generate the input features required by the spatial attention mechanism module. Spatial attention mechanism: Use the feature map F' output by the channel attention mechanism module as the input feature map of this module. First, perform a global maximum pooling and global average pooling based on the channel to obtain two H×W×1 feature maps, and then perform a concat operation (channel splicing) on ​​these two feature maps based on the channel. Then, after a 7×7 convolution operation, the dimension is reduced to 1 channel, that is, H×W×1. Then, pass sigmoid to generate the spatial attention mechanism feature map, that is Finally, Multiply it with the input feature map F' of the module to get the final generated feature Fm.

[0017] Furthermore, the structure of Neck includes: PAFPN structure and ASFF structure. In the PAFPN structure, three feature layers are located at different positions of the backbone CSPdarknet, namely, the middle layer, the lower middle layer, and the bottom layer. When the input feature map is (640, 640, 3), the shapes of the three feature layers are feat1=(80, 80, 256), feat2=(40, 40, 512), and feat3=(20, 20, 1024). The first path: the backbone convolution (20,20,1024) feature map is upsampled, and the backbone convolution (40,40,512) feature map is concat and convolution downsampled, and the information is fused and further feature extracted to obtain merge_01; the second path: merge_01 is upsampled by 1x1 convolution down-channel, concat and convolution downsampled with the backbone (80,80,256) to obtain merge_02 of size (80,80,256) and output to Head- 1; The third path: merge_02 downsampling and merge_01 concat and convolution downsampling operations to obtain (40,40,512) F3 of size (40,40,512) output to Head-2; The fourth path: merge_03 downsampling and trunk convolution to (20,20,1024) feature map concat and convolution downsampling operations to obtain (20,20,1024) merge_04 output to Head-3. The PAFPN structure is to first transfer the high-level feature information through upsampling and fuse it with the feature maps output by dark2, dark3, and dark4, output multi-scale feature maps, and then downsample the low-level precise positioning information and fuse it with the output high-level feature map to enhance the positioning expression ability of the overall feature pyramid, and finally output a tuple result consisting of 3 feature layers. ASFF structure: added between the Neck and Head structures, divided into two steps: constant scaling and adaptive fusion. Constant scaling: For ASFF-1: perform 3x3MaxPool(stride=2)+3x3Conv(stride=2) on the level-3 feature map to get X 3→1 , do 3x3Conv (stride=2) on the level-2 feature map to get X 2→1 ; For ASFF-2: perform 3x3Conv (stride=2) on the level-3 feature map to get X 3→2 , perform 1x1Conv on the level-1 feature map and modify the size to twice the original resolution to get X 1→2 ; For ASFF-3: perform 1x1Conv on the level-2 feature map and modify the size to twice the original resolution to get X 2→3Perform 1x1Conv on the level-1 feature map and resize it to 4 times the original resolution to get X 1→3 , where X 1 , X 2 , X 3 They are features from level-1, level-2, and level-3 respectively. Adaptive fusion: For ASFF-1, features from different layers are multiplied by the weight parameter α 1 , β 1 and γ 1 And add them together to get the new fusion feature ASFF-1; for ASFF-2, the features from different layers are multiplied by the weight parameter α 2 , β 2 and γ 2 And add them together to get the new fusion feature ASFF-2; for ASFF-3, the features from different layers are multiplied by the weight parameter α 3 , β 3 and γ 3 And add them together to get the new fusion feature ASFF-3. As shown in the following formula: .in, Represents the feature vector at position (i, j) on the feature map from level n to level l; is the output feature The (i,j)th vector of is mapped between channels; , , It refers to the spatial importance weights of the three different levels of feature maps that the network adaptively learns to level l.

[0018] For the weight parameter , and , is obtained by performing 1×1 convolution on the feature maps of level1, level2, and level3 after modifying their sizes. Here, we force + + =1, and , , ∈[0,1], and define: .in, , , is defined by the softmax function, respectively , , As the learnable control parameters of these three weights, , , yes , , The weight scalars at position (i, j) on the feature map determine the specific values ​​of the three weights. Use a 1×1 convolutional layer to extract the weights from X 1→l , X 2→l , X 3→l Compute weight scalar mapping , , , so standard back-propagation can be used to learn , , In other words, the model can automatically learn how to adjust these weights to achieve optimal performance. The key idea of ​​ASFF is to adaptively learn the fusion space weights of each scale feature map. In view of the poor imaging quality and unclear target information of sonar images, an adaptive attention mechanism is used to weightedly fuse the feature maps of sonar images in order to better retain the important feature information of sonar images, and the accuracy of target detection and classification is effectively improved. Finally, the fused features are passed to the target detection head for target classification and positioning.

[0019] Furthermore, the structure of the Head includes a Decoupled Head detection head, an Anchor-free mechanism, a Loss calculation, and a SimOTA label matching strategy. Among them, the Decoupled Head detection head extracts the target position and category information separately, learns them separately through different network branches, and finally fuses them. It can effectively reduce the number of parameters and computational complexity, and enhance the generalization ability and robustness of the model. The three prediction results are stacked, and the result obtained for each feature layer is: Out(h,w,4+1+num_classses), where h and w represent the height and width of the output image, respectively, and 4 represents the number of channels in the output tensor used to encode the bounding box position and size information, including: the x-coordinate of the bounding box center, the y-coordinate of the bounding box center, the width of the bounding box, and the height of the bounding box. num_classses represents the number of categories to be recognized by the target detection model. If num_classses=8, the last 8 channels of the output tensor will store the confidence scores of each category respectively. Reg(h,w,4) is used to determine the regression parameters of each feature point. After the regression parameters are adjusted, the prediction box can be obtained; Obj(h,w,1) is used to determine whether each feature point contains an object; Cls(h,w,num_classes) is used to determine the type of object contained in each feature point. Reg refers to the regression task, that is, predicting the position and size of the target bounding box. The output of the regression task is usually a 4-dimensional vector, indicating the center coordinates and width and height of the bounding box. Obj refers to the confidence of whether the detected object is the target object. The output is usually a scalar value between 0 and 1, indicating whether the position contains the target object. Cls refers to the classification task, that is, identifying the target category contained in the input image. The output of the classification task is usually a probability distribution, indicating the confidence of each predicted category. Anchor-free mechanism: Reduce the number of predictions for each position from 3 to 1, and directly predict four values ​​(i.e., the offset to the upper left corner of the grid and the height and width of the box). At the same time, the center point is set as a positive sample, and a scale range is preset to specify the FPN level for each object. Significantly reduces the manual design content and various tricks, thereby simplifying the training and decoding stages. The number of parameters and GFLOPS of the detector are greatly reduced, with faster speed and better performance. Loss calculation: In the Reg part, the IOU loss is calculated using the real box and the predicted box as the Loss component of the Reg part; in the Obj part, all feature points corresponding to the real box are positive samples, and the remaining feature points are negative samples. The cross entropy loss is calculated based on the prediction results of whether the positive and negative samples and feature points contain objects, as the Loss component of the Obj part; in the Cls part, the cross entropy loss is calculated as the Loss component of the Cls part. YOLOX also uses L1 Loss to optimize the difference between the predicted box and the real box.SimOTA label matching strategy: First, for each "prediction-GT" pair, calculate its pairwise matching degree, expressed as cost or quality, as shown in the following formula:. . The subscripts i and j represent the relationship between the i-th predicted box and the j-th ground truth (GT) box. Represents the pairwise matching cost or quality between the i-th predicted box and the j-th GT box. cls refers to the classification task, that is, identifying the target category contained in the input image. The output of the classification task is usually a probability distribution, indicating the confidence of each predicted category. reg refers to the regression task, that is, predicting the position and size of the target bounding box. The output of the regression task is usually a 4-dimensional vector, indicating the center coordinates and width and height of the bounding box. and Represent the classification loss and regression loss between the i-th prediction box and the j-th GT box, respectively, and λ represents the balance coefficient, which is used to balance the relative importance of classification loss and regression loss. Then, for each GT, in its fixed neighborhood, the top-k predictions with the lowest cost are selected as its positive samples. Finally, the grids corresponding to these positive predictions are assigned as positive examples, and the others are negative examples. The SimOTA label matching strategy can automatically analyze how many positive samples each GT has, and can automatically determine which feature map each GT should be detected from.

[0020] Specifically, step S10 includes: collecting the URPC2021 sonar dataset, annotating each sonar image in the dataset, including the location coordinates, size and category label of the target; randomly dividing the entire dataset into training set, validation set and test set in a ratio of 8:1:1; checking and cleaning the training set, validation set and test set to ensure that each sample has complete annotation information. Through the above steps, a sonar image dataset for training, validation and testing is obtained, and the annotation information of the target location, size and category is established, which provides a basis for the subsequent deep learning model training.

[0021] Wherein, the step S20 includes: adding a CBAM attention mechanism before multiple residual structures of the dark3, dark4 and dark5 modules of the YOLOX network structure, the CBAM attention mechanism includes a channel attention mechanism and a spatial attention mechanism, which can adaptively improve the network's attention to important features and enhance the ability of feature extraction; adding an ASFF (adaptive feature fusion) module between the Neck module and the Head module of the YOLOX network, the ASFF module can adaptively learn the fusion weights of features of different scales, enhance the fusion effect of multi-scale features, and enhance the model's perception of the target. Through the above improvements, the AC-YOLOX network is constructed, which has stronger feature extraction and fusion capabilities than the original YOLOX network.

[0022] Among them, the step S30 includes: inputting the training set and the verification set into the input end of the AC-YOLOX network; during the training process, the Mosaic data enhancement strategy is adopted to randomly scale, crop and arrange the four images to form a new training sample, increase the richness of the training data, and improve the generalization ability of the model; at the same time, the Mixup data enhancement strategy is adopted to weightedly fuse images of different categories to form a new training sample, further expand the training data set, and enhance the model's ability to classify targets; the input of the network is sonar image data, and the output is the target's location coordinates, confidence and category probability. These outputs are continuously optimized through back propagation to make them closer and closer to the Ground-Truth annotation information.

[0023] Among them, step S40 includes: loading the trained AC-YOLOX model into the test environment; inputting the sonar image of the test set into the trained AC-YOLOX model for forward prediction, screening the prediction results according to the confidence threshold (for example, 0.5) and the category probability threshold (for example, 0.6) output by the model, and obtaining the final target detection and classification results; comparing the prediction results of the model on the test set with the Ground-Truth annotation, and calculating the four performance indicators of accuracy, precision, recall rate and F1 score, which reflect the performance of the model in target detection and classification in complex underwater environments, and provide a basis for subsequent tuning and optimization.

[0024] Among them, the step S50 includes: analyzing the accuracy, precision, recall and F1 score of the model on the test set to find out the bottleneck of the problem; adjusting the structure of the AC-YOLOX network in a targeted manner, such as the specific implementation of the CBAM attention mechanism and the fusion strategy of the ASFF module, to further improve the ability of feature extraction and fusion; adjusting the hyperparameters of the AC-YOLOX network, such as the learning rate, weight decay coefficient, batch size, etc., to achieve better training convergence; trying other data enhancement strategies, such as random rotation and flipping, to further expand the training data set and improve the generalization performance of the model; repeating steps S20~S40, and evaluating the adjusted AC-YOLOX model on the validation set until the performance indicators reach the expected target. Through continuous adjustment, an AC-YOLOX model with excellent performance is finally obtained.

[0025] Wherein, the step S60 includes: deploying the trained AC-YOLOX model to an embedded device such as a Raspberry Pi, configuring the necessary hardware and software environment; transmitting the real-time acquired sonar image data to the Raspberry Pi device through serial communication; running the AC-YOLOX model deployed on the Raspberry Pi to perform real-time target detection and classification on the input sonar image; and feeding back the detection and classification results, including the location coordinates, size, category and other information of the target, to the host computer or other terminal devices in real time. Through the above steps, the trained AC-YOLOX model can be deployed to an actual underwater monitoring system to achieve fast and accurate target detection and classification of the real-time acquired sonar images.

[0026] Furthermore, the constructed AC-YOLOX network also includes an attention optimization equation group, which is used to further optimize the network's attention mechanism and improve the effect of feature extraction and fusion; the attention optimization equation group includes a dynamic channel attention equation, a frequency domain enhanced spatial attention equation, a cascade-parallel hybrid CBAM output equation, a nonlinear multi-scale ASFF fusion equation, and a Bayesian ASFF weight normalization equation. The specific expression of the attention optimization equation group is as follows:

[0027] 1. Dynamic channel attention equation:

[0028] ;

[0029] In the formula, is the input feature map; is the time step; is a multi-layer perceptron; and They are global average pooling and maximum pooling respectively; is the sigmoid function; is the adaptive learning rate; is the partial derivative of channel attention with respect to time.

[0030] Parameter acquisition method:

[0031] Obtained through experiments, the specific steps are:

[0032] Step 1: Initialization ;

[0033] Step 2: Calculate the difference on the validation set Model performance under values;

[0034] Step 3: Select the model that performs best value.

[0035] 2. Spatial attention equation for frequency domain enhancement:

[0036] ;

[0037] In the formula, It is the feature map after channel attention processing; It is a 7×7 convolution operation; and They are fast Fourier transform and its inverse transform respectively; is a frequency domain filter; is the frequency domain enhancement coefficient.

[0038] Parameter acquisition method:

[0039] and Obtained through back-propagation optimization, the specific steps are:

[0040] Step 1: Initialization is a Gaussian filter, ;

[0041] Step 2: Calculate the loss function and The gradient of

[0042] Step 3: Update the parameters using the Adam optimizer: , .

[0043] 3. Cascade-parallel hybrid CBAM output equation:

[0044] ;

[0045] In the formula, represents the Hadamard product; is the parallel branch weight coefficient.

[0046] 4. Nonlinear multi-scale ASFF fusion equation:

[0047]

[0048] In the formula, For the Feature map after layer fusion; For the The layer feature map is resampled to Layer results; is a learnable parameter; is the gating function; and is the activation function; is the residual connection.

[0049] Parameter acquisition method:

[0050] Obtained through back-propagation optimization, the specific steps are:

[0051] Step 1: Initialize parameters;

[0052] Step 2: Calculate the gradient of the loss function with respect to the parameters;

[0053] Step 3: Update the parameters using the Adam optimizer.

[0054] 5. Bayesian ASFF weight normalization equation:

[0055] ;

[0056] In the formula, For the given data Down The posterior probability of is the normalization constant; and is the mean and standard deviation of the prior distribution; and are learnable real and imaginary parameters; Is an imaginary unit.

[0057] Parameter acquisition method:

[0058] and Obtained through maximum likelihood estimation, the specific steps are:

[0059] Step 1: Collect the model’s performance on the validation set value;

[0060] Step 2: Calculation , .

[0061] and Optimization through variational inference method, the specific steps are:

[0062] Step 1: Initialize parameters;

[0063] Step 2: Calculate the variational lower bound (ELBO);

[0064] Step 3: Update the parameters by maximizing ELBO.

[0065] These optimization equations work together to further improve the network's ability to extract and represent target features in sonar images through mechanisms such as dynamic adjustment, multi-scale fusion, frequency domain enhancement, and uncertainty estimation, thereby improving the performance of target detection and classification.

[0066] The optimization principle of the attention optimization equation group is specifically:

[0067] 1. Introducing a dynamic weight adjustment mechanism: By introducing time series analysis and adaptive learning rate, the weights of the CBAM and ASFF modules can be dynamically adjusted according to the characteristics of the input data, thereby improving the adaptability of the model to different types of sonar images.

[0068] 2. Multi-scale feature fusion optimization: Nonlinear transformation and residual connection are introduced in the ASFF module to enhance the information interaction between features of different scales and improve the effect of feature fusion.

[0069] 3. Frequency domain attention mechanism: Based on spatial and channel attention, frequency domain attention is introduced to capture the frequency characteristics in sonar images and improve the model's ability to perceive target details.

[0070] 4. Cascade and parallel structures of attention mechanism: By designing cascade and parallel attention structures, multi-level feature enhancement is achieved and the expressiveness of the model is improved.

[0071] 5. Introduce uncertainty estimation: Introduce the Bayesian method in the attention weight calculation to estimate the uncertainty of the weight and improve the robustness and interpretability of the model.

[0072] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned real-time target detection and classification method based on sonar images.

[0073] The third aspect of the present invention provides a real-time target detection and classification system based on sonar images, which includes the above-mentioned computer-readable storage medium. Furthermore, the construction of the real-time target detection and classification system based on sonar images also includes model transplantation: importing the trained new model and optimal weights into the Raspberry Pi; sonar detection system process: after all required files are deployed to the Raspberry Pi, the sonar image is transmitted to the Raspberry Pi through serial communication, and the program is set so that the Raspberry Pi immediately transmits the recognized sonar image back to the device after recognition.

[0074] Compared with the prior art, the method, medium and system for real-time target detection and classification based on sonar images provided by the present invention have the following beneficial effects:

[0075] 1. Significantly improved the accuracy of target detection and classification.

[0076] 2. Improved the robustness and real-time performance of the system. The use of CBAM attention mechanism and ASFF adaptive feature fusion enables the model to better cope with various interference factors in complex underwater environments, improving the stability and reliability of detection and classification. At the same time, the single-stage YOLOX algorithm structure makes the entire detection and classification process more efficient and meets the needs of real-time applications.

[0077] 3. Reduced reliance on manual feature extraction. The present invention adopts an end-to-end deep learning method, which does not need to rely on manually designed feature extraction algorithms, greatly reducing the complexity of system development and deployment.

[0078] In summary, the present invention solves the technical problem that the prior art is difficult to solve due to the coexistence of internal and external interference in the working environment of the sonar system, resulting in insufficient accuracy in the final real-time detection and classification of underwater targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 A flow chart of the method provided by the present invention;

[0080] Figure 2 This is the structure diagram of the AC-YOLOX network after improvement based on YOLOX;

[0081] Figure 3 This is the Focus structure diagram;

[0082] Figure 4 This is a comparison chart before and after the improvement of the CSP1_X network structure in Backbone;

[0083] Figure 5 This is a comparison chart before and after the improvement of the CSP2_X network structure in Backbone;

[0084] Figure 6 This is the Bottleneck structure diagram;

[0085] Figure 7 It is the CBS convolution structure diagram;

[0086] Figure 8 This is the structural diagram of the CBAM attention mechanism;

[0087] Fig. 9 It is the SPP structure diagram;

[0088] Fig.10 This is the PAFPN network structure diagram;

[0089] Fig.11 This is the schematic diagram of the ASFF module;

[0090] Fig.12 This is the structural diagram of Decoupled Head;

[0091] Fig.13 These are some sonar image detection and classification results from four simulation experiments. DETAILED DESCRIPTION

[0092] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0093] like Figure 1 FIG. 1 is a flow chart of a method for real-time target detection and classification based on sonar images provided by the present invention. The method comprises the following steps:

[0094] S10. Divide the URPC2021 sonar dataset into a training set, a validation set, and a test set in a ratio of 8:1:1, and establish labels according to the position, size, and category of the target in the sonar image as the training dataset;

[0095] S20. Improve the YOLOX network structure and build an AC-YOLOX network. Specifically, add the CBAM attention mechanism before the multiple residual structures of dark3, dark4, and dark5 of the YOLOX network structure, and add the ASFF module of adaptive feature fusion between Neck and Head.

[0096] S30, input the training set and the validation set into the AC-YOLOX network for training. During the training process, the Mosaic and Mixup data enhancement strategies are used to increase data diversity. The input of the training is the sonar image data, and the output of the training is the location coordinates, confidence, and category probability of the target.

[0097] S40, predict the test set with the trained model, output the classification results, and calculate the evaluation indicators of the model, including accuracy, precision, recall and F1 score;

[0098] S50, adjusting the network structure, hyperparameters and training strategy of the AC-YOLOX network according to the evaluation indicators, repeating steps S20 to S50 until the model performance reaches the preset expected target, and saving it as the target real-time detection classification model;

[0099] S60, using the target real-time detection and classification model, inputting the sonar image acquired in real time, and outputting the target real-time detection and classification result.

[0100] The specific implementation methods of the above steps are described in detail below:

[0101] The specific implementation of step S10 is: first, the URPC2021 sonar dataset is divided into a training set, a validation set, and a test set in a ratio of 8:1:1. Each sonar image in the dataset has corresponding label information, including the location coordinates, size, and category of the target. These label information will be used as the true value annotation of the training data, providing a basis for subsequent deep learning model training.

[0102] Specifically, the implementation step S10 includes the following sub-steps:

[0103] Sub-step S10.1: Collect the URPC2021 sonar dataset. This dataset contains a large number of sonar images of seabed targets, covering different types of underwater targets, such as spheres, cubes, dummies, etc.

[0104] Sub-step S10.2: Annotate each sonar image in the dataset, including the target's location coordinates (x, y), size (w, h) and category label c. This annotation information will be used as a supervisory signal during the training process.

[0105] Sub-step S10.3: The entire data set is randomly divided into a training set, a validation set, and a test set in a ratio of 8:1:1. The training set is used for model training optimization, the validation set is used for model performance evaluation and hyperparameter tuning, and the test set is used for objective evaluation of the final model performance.

[0106] Sub-step S10.4: Check and clean the training set, validation set, and test set to ensure that each sample has complete annotation information and remove possible abnormal samples.

[0107] Through the above sub-steps, a sonar image dataset for training, verification and testing was obtained, and annotation information of target position, size and category was established, providing a basis for subsequent deep learning model training.

[0108] The specific implementation of step S20 is: improve the YOLOX network structure to build an AC-YOLOX network. The improvements mainly include the following aspects:

[0109] 1. Add the CBAM attention mechanism before multiple residual structures of the dark3, dark4, and dark5 modules of the YOLOX network. The CBAM attention mechanism includes the channel attention mechanism and the spatial attention mechanism, which can adaptively increase the network's attention to important features and enhance the ability of feature extraction.

[0110] 2. Add an ASFF (Adaptive Feature Fusion) module between the Neck module and the Head module of the YOLOX network. The ASFF module can adaptively learn the fusion weights of features of different scales, enhance the fusion effect of multi-scale features, and thus improve the model's ability to perceive the target.

[0111] Specifically, the implementation step S20 includes the following sub-steps:

[0112] Sub-step S20.1: Based on the YOLOX network structure, add the CBAM attention mechanism before the multiple residual structures (Bottleneck*N) of the dark3, dark4 and dark5 modules. The CBAM attention mechanism can adaptively enhance the network's attention to important features and improve feature extraction capabilities.

[0113] Sub-step S20.2: Add an ASFF (Adaptive Feature Fusion) module between the Neck module and the Head module of the YOLOX network. The ASFF module can adaptively learn the fusion weights of features of different scales, enhance the fusion effect of multi-scale features, and improve the model's perception of the target.

[0114] Furthermore, the constructed AC-YOLOX network also includes an attention optimization equation group, which is used to further optimize the network's attention mechanism and improve the effect of feature extraction and fusion; the attention optimization equation group includes a dynamic channel attention equation, a frequency domain enhanced spatial attention equation, a cascade-parallel hybrid CBAM output equation, a nonlinear multi-scale ASFF fusion equation, and a Bayesian ASFF weight normalization equation. The specific expression of the attention optimization equation group is as follows:

[0115] 1. Dynamic channel attention equation:

[0116] ;

[0117] In the formula, is the input feature map; is the time step; is a multi-layer perceptron; and They are global average pooling and maximum pooling respectively; is the sigmoid function; is the adaptive learning rate; is the partial derivative of channel attention with respect to time.

[0118] Parameter acquisition method:

[0119] Obtained through experiments, the specific steps are:

[0120] Step 1: Initialization ;

[0121] Step 2: Calculate the difference on the validation set Model performance under values;

[0122] Step 3: Select the model that performs best value.

[0123] 2. Spatial attention equation for frequency domain enhancement:

[0124] ;

[0125] In the formula, It is the feature map after channel attention processing; It is a 7×7 convolution operation; and They are fast Fourier transform and its inverse transform respectively; is a frequency domain filter; is the frequency domain enhancement coefficient.

[0126] Parameter acquisition method:

[0127] and Obtained through back-propagation optimization, the specific steps are:

[0128] Step 1: Initialization is a Gaussian filter, ;

[0129] Step 2: Calculate the loss function and The gradient of

[0130] Step 3: Update the parameters using the Adam optimizer: , .

[0131] 3. Cascade-parallel hybrid CBAM output equation:

[0132] ;

[0133] In the formula, represents the Hadamard product; is the parallel branch weight coefficient; In the dynamic channel attention equation of ; is the result value of the spatial attention equation enhanced in the frequency domain.

[0134] 4. Nonlinear multi-scale ASFF fusion equation:

[0135]

[0136] In the formula, For the Feature map after layer fusion; For the The layer feature map is resampled to Layer results; is a learnable parameter; is the gating function; and is the activation function; is the residual connection.

[0137] Parameter acquisition method:

[0138] Obtained through back-propagation optimization, the specific steps are:

[0139] Step 1: Initialize parameters;

[0140] Step 2: Calculate the gradient of the loss function with respect to the parameters;

[0141] Step 3: Update the parameters using the Adam optimizer.

[0142] 5. Bayesian ASFF weight normalization equation:

[0143] ;

[0144] In the formula, For the given data Down The posterior probability of is the normalization constant; and is the mean and standard deviation of the prior distribution; and are learnable real and imaginary parameters; Is an imaginary unit.

[0145] Parameter acquisition method:

[0146] and Obtained through maximum likelihood estimation, the specific steps are:

[0147] Step 1: Collect the model’s performance on the validation set value;

[0148] Step 2: Calculation , .

[0149] and Optimization through variational inference method, the specific steps are:

[0150] Step 1: Initialize parameters;

[0151] Step 2: Calculate the variational lower bound (ELBO);

[0152] Step 3: Update the parameters by maximizing ELBO.

[0153] These optimization equations work together to further improve the network's ability to extract and represent target features in sonar images through mechanisms such as dynamic adjustment, multi-scale fusion, frequency domain enhancement, and uncertainty estimation, thereby improving the performance of target detection and classification.

[0154] Through the above improvements, the AC-YOLOX network was constructed. Compared with the original YOLOX network, AC-YOLOX has stronger feature extraction and fusion capabilities, can better adapt to complex underwater sonar image environments, and improve the performance of target detection and classification.

[0155] The specific implementation method of step S30 is: input the training set and the verification set into the constructed AC-YOLOX network for training, and adopt the two data enhancement strategies of Mosaic and Mixup during the training process.

[0156] Specifically, the implementation step S30 includes the following sub-steps:

[0157] Sub-step S30.1: Input the training set and the validation set to the input of the AC-YOLOX network.

[0158] Sub-step S30.2: During the training process, the Mosaic data enhancement strategy is used. The Mosaic strategy is to randomly scale, crop and arrange the four images to form a new training sample. This method can increase the richness of the training data and improve the generalization ability of the model.

[0159] Sub-step S30.3: Mixup data enhancement strategy is used at the same time. Mixup strategy is to perform weighted fusion of images of different categories to form new training samples. This method can further expand the training data set and enhance the model's ability to classify targets.

[0160] Sub-step S30.4: During the training process, the network input is the sonar image data, and the output is the location coordinates, confidence and category probability of the target. The network will continuously optimize these outputs through back propagation to make them closer and closer to the Ground-Truth annotation information.

[0161] Through the above sub-steps, the two data enhancement strategies of Mosaic and Mixup were used to effectively expand the training data set, enhance the modeling and generalization capabilities of the AC-YOLOX network for complex sonar images, and lay the foundation for subsequent target detection and classification tasks.

[0162] The specific implementation method of step S40 is: applying the trained AC-YOLOX model to the test set for prediction, and calculating the performance indicators of the model on the test set, including accuracy, precision, recall and F1 score.

[0163] Specifically, the implementation step S40 includes the following sub-steps:

[0164] Sub-step S40.1: Load the trained AC-YOLOX model into the test environment.

[0165] Sub-step S40.2: Input the sonar images of the test set into the trained AC-YOLOX model for forward prediction. The model will output the location coordinates, confidence and category probability of each target.

[0166] Sub-step S40.3: According to the confidence threshold (e.g. 0.5) and the category probability threshold (e.g. 0.6) output by the model, the prediction results are screened to obtain the final target detection classification results.

[0167] Sub-step S40.4: Compare the prediction results of the model on the test set with the Ground-Truth annotations, and calculate the four performance indicators of accuracy (Accuracy), precision (Accuracy), recall (Recall) and F1 score.

[0168] These indicators reflect the performance of the model in target detection and classification in complex underwater environments, and provide a basis for subsequent tuning and optimization.

[0169] Through the above sub-steps, the performance of the trained AC-YOLOX model on the test set can be objectively evaluated, providing a basis for subsequent further optimization.

[0170] The specific implementation method of step S50 is: according to the evaluation indicators of the model on the test set, adjust the structure, hyperparameters and training strategy of the AC-YOLOX network, and repeat steps S20 to S40 until the model performance reaches the expected goal.

[0171] Specifically, the implementation step S50 includes the following sub-steps:

[0172] Sub-step S50.1: Analyze the four performance indicators of the model on the test set: accuracy, precision, recall and F1 score, and find out the bottleneck of the problem.

[0173] Sub-step S50.2: Targetedly adjust the structure of the AC-YOLOX network, such as the specific implementation of the CBAM attention mechanism, the fusion strategy of the ASFF module, etc., to further improve the ability of feature extraction and fusion.

[0174] Sub-step S50.3: Adjust the hyperparameters of the AC-YOLOX network, such as learning rate, weight decay coefficient, batch size, etc., to achieve better training convergence.

[0175] Sub-step S50.4: Try other data enhancement strategies, such as random rotation, flipping, etc., to further expand the training data set and improve the generalization performance of the model.

[0176] Sub-step S50.5: Repeat steps S20 to S40 and evaluate the adjusted AC-YOLOX model on the validation set until the performance index reaches the expected target.

[0177] By continuously adjusting the network structure, hyperparameters and training strategies, we can eventually obtain an AC-YOLOX model with excellent performance that meets the needs of target detection and classification in complex underwater environments.

[0178] The specific implementation method of step S60 is: using the trained AC-YOLOX model, inputting the real-time acquired sonar image in the actual application scenario, and outputting the real-time detection and classification results of the target.

[0179] Specifically, the implementation step S60 includes the following sub-steps:

[0180] Sub-step S60.1: Deploy the trained AC-YOLOX model to an embedded device such as Raspberry Pi, and configure the necessary hardware and software environment.

[0181] Sub-step S60.2: Transmit the real-time acquired sonar image data to the Raspberry Pi device through serial port communication.

[0182] Sub-step S60.3: Run the AC-YOLOX model deployed on the Raspberry Pi to perform real-time target detection and classification on the input sonar image.

[0183] Sub-step S60.4: Feedback the detection and classification results, including the location coordinates, size, category and other information of the target, to the host computer or other terminal device in real time.

[0184] Through the above sub-steps, the trained AC-YOLOX model can be deployed to the actual underwater monitoring system to achieve fast and accurate target detection and classification of real-time sonar images, provide autonomous decision-making capabilities for underwater unmanned systems, and then be applied to marine exploration, underwater rescue, ship monitoring and other fields.

[0185] In general, the real-time target detection and classification system based on sonar images proposed in the present invention constructs an AC-YOLOX network by improving the YOLOX network structure, and adopts a number of innovative measures in the training, optimization and deployment links, which can effectively improve the target detection and classification performance in complex underwater environments and meet the actual application needs.

[0186] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned real-time target detection and classification method based on sonar images.

[0187] A third aspect of the present invention provides a real-time target detection and classification system based on sonar images, which includes the above-mentioned computer-readable storage medium.

[0188] Specifically, the principle of the present invention is:

[0189] 1. AC-YOLOX network structure design

[0190] The present invention improves the YOLOX network and constructs an AC-YOLOX network. Specifically, a CBAM attention mechanism is added before multiple residual structures of the dark3, dark4 and dark5 modules of the YOLOX network to adaptively enhance the network's attention to important features and improve the ability of feature extraction. At the same time, an ASFF adaptive feature fusion module is added between the Neck module and the Head module of the YOLOX network, which can adaptively learn the fusion weights of features of different scales, enhance the fusion effect of multi-scale features, and improve the model's perception of the target.

[0191] The CBAM attention mechanism includes a channel attention mechanism and a spatial attention mechanism. The channel attention mechanism captures the importance information at the channel level through global average pooling and maximum pooling, and uses the MLP network to dynamically adjust the channel weights to enhance the network's attention to important feature channels. The spatial attention mechanism extracts a spatial attention map through global pooling and convolution operations to highlight the network's attention to key areas. The interaction of these two attention mechanisms can adaptively enhance the network's perception of important features in sonar images.

[0192] The ASFF adaptive feature fusion module fuses features from different levels through constant scaling and adaptive weighting, enhancing the perception of multi-scale targets. Constant scaling can unify feature maps of different resolutions to the same scale, while adaptive weighting can dynamically adjust the fusion weights according to the importance of features at different levels, and finally output the fused feature map.

[0193] These improvements enable the AC-YOLOX network to have stronger feature extraction and fusion capabilities, better capture the target features of sonar images in complex underwater environments, and improve the performance of detection and classification.

[0194] 2. Data Augmentation Strategy

[0195] The present invention adopts two data enhancement strategies, Mosaic and Mixup, when training the AC-YOLOX network.

[0196] The Mosaic strategy randomly scales, crops, and arranges four images to form a new training sample. This method can effectively increase the richness of training data and improve the generalization ability of the model. Especially for scenes with low resolution and limited detail information such as sonar images, the Mosaic strategy can simulate more small targets and boundary targets through operations such as splicing and scaling, and enhance the model's detection ability for such targets.

[0197] The Mixup strategy is to perform weighted fusion of images of different categories to form new training samples. This method can further expand the training data set and enhance the robustness of the model for target classification. For some fuzzy and overlapping target areas in sonar images, the Mixup strategy can help the model learn such special target features and improve classification accuracy.

[0198] By combining the two data enhancement strategies of Mosaic and Mixup, the training data set was effectively expanded, and the AC-YOLOX network's modeling ability for complex sonar images was enhanced, laying the foundation for subsequent target detection and classification tasks.

[0199] 3. Model Optimization Strategy

[0200] In addition to network structure and data enhancement, the present invention also takes a series of targeted measures in the process of training and optimizing the AC-YOLOX model:

[0201] First, in terms of loss function design, in addition to the conventional regression loss, target loss and classification loss, L1 loss is introduced to further optimize the difference between the predicted box and the true box.

[0202] Secondly, in the specific implementation of the attention mechanism, the present invention proposes a set of attention optimization equations, including dynamic channel attention, frequency domain enhanced spatial attention, cascade-parallel hybrid CBAM output, nonlinear multi-scale ASFF fusion, and Bayesian ASFF weight normalization. These optimization equations can adaptively adjust the parameters of the attention mechanism, improve the effect of feature extraction and fusion, and enhance the model's perception of complex sonar images.

[0203] In addition, during the training process, the present invention also tried other data enhancement strategies, such as random rotation and flipping, to further expand the training data set and improve the generalization performance of the model. At the same time, by continuously adjusting network hyperparameters, such as learning rate and weight decay coefficient, better training convergence can be achieved.

[0204] In summary, the real-time target detection and classification system based on sonar images proposed in the present invention has made in-depth innovations and improvements in network structure, data enhancement and model optimization, and can effectively cope with various interference factors in complex underwater environments, greatly improve the accuracy and robustness of target detection and classification, and meet the needs of practical applications.

[0205] In order to better understand and implement the present invention, a specific embodiment 1 of the present invention is provided below, and this embodiment 1 is specifically described as follows.

[0206] 1. Improve the network structure of sonar image target classification

[0207] Based on the YOLOX model, in order to address the poor imaging quality and low resolution of sonar images, the CBAM attention mechanism is integrated into the multiple residual structures of the CSP structure in the backbone structure, thereby strengthening the network's target recognition in the characteristic areas of sonar images. In order to address the complex seabed environment and unclear feature information, an ASFF module is added between the Neck and the Head to weightedly fuse multi-scale features in order to better retain the important feature information of the sonar image.

[0208] 1.1AC-YOLOX model network structure design

[0209] For sonar images, two improvements are made based on YOLOX. The improved network structure and improvements are introduced below.

[0210] The proposed AC-YOLOX improved algorithm is based on YOLOX, taking into account image characteristics, accuracy and other factors, and improving and innovating the YOLOX network structure. Like YOLOX, AC-YOLOX is divided into four parts: input, backbone, neck and head. The structural framework of the improved AC-YOLOX algorithm is as follows: Figure 2 shown.

[0211] 1.2Backbone

[0212] In the Backbone part, AC-YOLOX uses the Focus structure to slice the image; uses multiple layers of CBS convolution to extract features; integrates the CBAM attention mechanism into the multiple residual structures of the CSP structure, and uses the improved CSP_C structure to make the model pay more attention to identifying the target object and learn more features; uses SPP to extract features in different pooling layers, increase the receptive field, and integrate more information.

[0213] The Focus structure slices the image, and then the resulting image is convolved to obtain a double-sampled feature map without information loss. This is equivalent to a special downsampling method. By sampling a value for every other pixel, an image is converted into four complementary images, which increases the number of input channels by four times. This processing method not only improves computing power, but also performs downsampling operations without losing information, thereby improving the performance of the model. Figure 3 shown.

[0214] In the Backbone part, the CSP structure has been improved as follows Figure 4 and Figure 5 As shown. dark2 uses the original CSP1_X, CSP1_X structure to divide the original input into two branches. One branch directly performs CBS operation to halve the number of channels. Then one branch performs a CBS operation, and then passes through multiple residual structures. Then concat the two branches, so that the input and output of BottleneckCSP are the same size. This is to allow the model to learn more features of the sonar image. Finally, a CBS operation is performed. The residual network structure diagram is as follows Figure 6 As shown; CSP_C_X used in dark3 and dark4 is improved from CSP1_X. The original input end is divided into two branches. One branch directly performs CBS operation to halve the number of channels. Then one branch performs a CBS operation, and then passes through a CBAM attention mechanism structure, and then passes through multiple residual structures, and then concats the two branches, and finally performs a CBS operation again; CSP2_C_X used in dark5 is improved from CSP2_X. The original input end is divided into two branches. One branch directly performs CBS operation to halve the number of channels. Then one branch performs a CBS convolution operation, and then passes through a CBAM attention mechanism structure, and then passes through two CBS convolution operations, and then concats the two branches, and finally performs a CBS operation again. Among them: CBS structure includes: Conv convolution operation, Batch Normalization normalization, SiLU activation function. The CBS convolution structure is as follows Figure 7 shown.

[0215] The CBAM attention mechanism includes a channel attention mechanism and a spatial attention mechanism. Channel attention mechanism: The input feature map F (H×W×C) is subjected to global max pooling and global average pooling based on width and height respectively to obtain two 1×1×C feature maps. Then, they are sent to a two-layer neural network (MLP) respectively. The number of neurons in the first layer is C / r (r is the reduction rate), the activation function is Relu, and the number of neurons in the second layer is C. The two-layer neural network is shared. Then, the features output by the MLP are added based on element-wise operation, and then subjected to sigmoid activation operation to generate the final channel attention mechanism feature map, that is, Finally, Perform element-wise multiplication operation with the input feature map F to generate the input features required by the spatial attention mechanism module. Spatial attention mechanism: Use the feature map F' output by the channel attention mechanism module as the input feature map of this module. First, perform a global maximum pooling and global average pooling based on the channel to obtain two H×W×1 feature maps, and then perform a concat operation (channel splicing) on ​​these two feature maps based on the channel. Then, after a 7×7 convolution operation, the dimension is reduced to 1 channel, that is, H×W×1. Then, pass sigmoid to generate the spatial attention mechanism feature map, that is Finally, Multiply it with the input feature map F' of the module to get the final generated feature Fm. The structure of the CBAM attention mechanism is as follows Figure 8 shown.

[0216] The AC-YOLOX model uses the SPP structure in the backbone part. It first performs a CBS operation and then uses 1×1, 5×5, 9×9, and 13×13 convolution kernels to perform pyramid maximum pooling to achieve the fusion of local features and global features. The SPP structure can ensure the fixed length of the feature vector while maintaining feature diversity, thereby adapting to input data of different sizes. The SPP structure is as follows Fig. 9 shown.

[0217] 1.3 Neck

[0218] In the Neck part, the PAFPN structure is used to fuse feature maps of different scales. The three feature layers are located in different positions of the backbone CSPdarknet, namely the middle layer, the lower middle layer, and the bottom layer. When the input feature map is (640, 640, 3), the shapes of the three feature layers are feat1=(80, 80, 256), feat2=(40, 40, 512), and feat3=(20, 20, 1024). The first path: the backbone convolution (20, 20, 1024) feature map is upsampled, and concat and convolution downsample with the backbone convolution (40, 40, 512) feature map, and the information is fused and further feature extracted to obtain merge_01; the second path: merge_01 is upsampled by 1x1 convolution down-channel and concat and convolution downsampled with the backbone (80, 80, 256) to obtain merge_02 of size (80, 80, 256) and output to Head- 1; The third path: merge_02 downsampling and merge_01 are concat and convolution downsampling operations to obtain (40,40,512) F3 of size (40,40,512) and output to Head-2; The fourth path: merge_03 downsampling and backbone convolution to the feature map of (20,20,1024) for concat and convolution downsampling operations, to obtain merge_04 of (20,20,1024) and output to Head-3. The PAFPN structure is to first transfer the high-level feature information through upsampling and fuse it with the feature maps output by dark2, dark3, and dark4 respectively, output multi-scale feature maps, and then downsample the low-level precise positioning information and fuse it with the output high-level feature maps to enhance the positioning expression ability of the overall feature pyramid, and finally output a tuple result consisting of 3 feature layers. The PAFPN structure is as follows Fig.10 shown.

[0219] The ASFF module is added between Neck and Head, which is divided into two steps: constant scaling and adaptive fusion. Constant scaling: For ASFF-1: perform 3x3MaxPool(stride=2)+3x3Conv(stride=2) on the feature map of level-3 to get X3→1, and perform 3x3Conv(stride=2) on the feature map of level-2 to get X2→1; For ASFF-2: perform 3x3Conv(stride=2) on the feature map of level-3 to get X3→2, perform 1x1Conv on the feature map of level-1, and modify the size to twice the original image resolution to get X1→2; For ASFF-3: perform 1x1Conv on the feature map of level-2, and modify the size to twice the original image resolution to get X2→3. Perform 1x1Conv on the feature map of level-1, and modify the size to four times the original image resolution to get X1→3, where X1, X2, and X3 are the features from level-1, level-2, and level-3 respectively. Adaptive fusion: For ASFF-1, features from different layers are multiplied by weight parameters α1, β1 and γ1 and added together to obtain a new fused feature ASFF-1; for ASFF-2, features from different layers are multiplied by weight parameters α2, β2 and γ2 and added together to obtain a new fused feature ASFF-2; for ASFF-3, features from different layers are multiplied by weight parameters α3, β3 and γ3 and added together to obtain a new fused feature ASFF-3. As shown in the following formula: ,in, Represents the feature vector at position (i, j) on the feature map from level n to level l; is the output feature The (i,j)th vector of is mapped between channels; , , It refers to the spatial importance weights of the three different levels of feature maps that the network adaptively learns to level l.

[0220] For the weight parameter , and , is obtained by performing 1×1 convolution on the feature maps of level1, level2, and level3 after modifying their sizes. Here, we force + + =1, and , , ∈[0,1], and define: .in, , , is defined by the softmax function, respectively , , As the learnable control parameters of these three weights, , , yes , , The weight scalars at position (i, j) on the feature map determine the specific values ​​of the three weights. Use a 1×1 convolutional layer to extract the weights from X 1→l , X 2→l , X 3→l Compute weight scalar mapping , , , so standard back-propagation can be used to learn , , That is to say, the model can automatically learn how to adjust these weights to achieve the best performance. The key idea of ​​ASFF is to adaptively learn the fusion space weights of each scale feature map. In view of the poor imaging quality and unclear target information of sonar images, an adaptive attention mechanism is used to perform weighted fusion on the feature maps of sonar images, so as to better retain the important feature information of sonar images, and effectively improve the accuracy of target detection and classification. Finally, the fused features are passed to the target detection head for target classification and positioning. The structure of the ASFF module is as follows: Fig.11 shown.

[0221] 1.4 Head

[0222] The head part of AC-YOLOX includes the decoupled head detection head, anchor-free mechanism, loss calculation, and SimOTA tag matching strategy.

[0223] AC-YOLOX uses a Decoupled Head detection head to extract the target position and category information separately, learn them separately through different network branches, and then fuse them. It can effectively reduce the number of parameters and computational complexity, and enhance the generalization ability and robustness of the model. The three prediction results are stacked, and the result obtained for each feature layer is: Out(h,w,4+1+num_classses), h and w represent the height and width of the output image respectively, 4 represents the number of channels in the output tensor used to encode the bounding box position and size information, including: the x coordinate of the center of the bounding box, the y coordinate of the center of the bounding box, the width of the bounding box, and the height of the bounding box. num_classses represents the number of categories to be identified by the target detection model. If num_classses=8, the last 8 channels of the output tensor will store the confidence score of each category respectively. Reg(h,w,4) is used to determine the regression parameters of each feature point. After the regression parameters are adjusted, the prediction box can be obtained; Obj(h,w,1) is used to determine whether each feature point contains an object; Cls(h,w,num_classes) is used to determine the type of object contained in each feature point. Reg refers to the regression task, which is to predict the position and size of the target bounding box. The output of the regression task is usually a 4-dimensional vector, which represents the center coordinates and width and height of the bounding box. Obj refers to the confidence of whether the detected object is the target object. The output is usually a scalar value between 0 and 1, indicating whether the position contains the target object. Cls refers to the classification task, which is to identify the target category contained in the input image. The output of the classification task is usually a probability distribution, which represents the confidence of each predicted category. The structure of the Decoupled Head detection head is as follows: Fig.12 shown.

[0224] The anchor-free mechanism adopted by AC-YOLOX reduces the number of predictions for each position from 3 to 1, and directly predicts four values ​​(i.e., the offset to the upper left corner of the grid and the height and width of the box). At the same time, the center point is set as a positive sample, and a scale range is preset to specify the FPN level for each object. The manual design content and various tricks are significantly reduced, thereby simplifying the training and decoding stages. The number of parameters and GFLOPS of the detector are greatly reduced, and the speed is faster and the performance is better.

[0225] The Loss calculation of AC-YOLOX includes: Reg part, using the real box and the predicted box to calculate the IOU loss, as the Loss component of the Reg part; Obj part, all the feature points corresponding to the real box are positive samples, and the remaining feature points are negative samples. The cross entropy loss is calculated according to the prediction results of whether the positive and negative samples and feature points contain objects, as the Loss component of the Obj part; Cls part, the cross entropy loss is calculated, as the Loss component of the Cls part. Reg refers to the regression task, that is, predicting the position and size of the target bounding box. The output of the regression task is usually a 4-dimensional vector, indicating the center coordinates and width and height of the bounding box. Obj refers to the confidence of whether the detected object is the target object. The output is usually a scalar value between 0 and 1, indicating whether the position contains the target object. Cls refers to the classification task, that is, identifying the target category contained in the input image. The output of the classification task is usually a probability distribution, indicating the confidence of each predicted category. YOLOX also uses L1 Loss to optimize the difference between the predicted box and the real box.

[0226] AC-YOLOX adopts the SimOTA label matching strategy. First, for each "prediction-GT" pair, the pairwise matching degree is calculated, expressed as cost or quality, as shown in the following formula: The subscript ij represents the relationship between the i-th predicted box and the j-th ground truth (GT) box. Represents the pairwise matching cost or quality between the i-th predicted box and the j-th GT box. cls refers to the classification task, that is, identifying the target category contained in the input image. The output of the classification task is usually a probability distribution, indicating the confidence of each predicted category. reg refers to the regression task, that is, predicting the position and size of the target bounding box. The output of the regression task is usually a 4-dimensional vector, indicating the center coordinates and width and height of the bounding box. and Respectively represent the classification loss and regression loss between the i-th prediction box and the j-th GT box, and λ represents the balance coefficient, which is used to balance the relative importance of classification loss and regression loss. Then, for each GT, in its fixed neighborhood, the top-k predictions with the lowest cost are selected as its positive samples. Finally, the grids corresponding to these positive predictions are assigned as positive examples, while the others are negative examples.

[0227] 2. System Design

[0228] 2.1 Raspberry Pi

[0229] Raspberry Pi is an embedded device based on Linux system and has microcomputer control function. Raspberry Pi integrates resources, including IO port, communication interface, storage, USB interface, GPU, memory and video input, etc. It can be used for many different applications, such as artificial intelligence tasks such as target classification and target detection. Compared with single-chip microcomputer, Raspberry Pi has stronger performance, but its price is lower than NVIDIA series products. In addition, the use of neural computing stick (NCS) can also improve the computing performance of Raspberry Pi. Taking into account the issues of low cost, universality and portability, the present invention selects Raspberry Pi 5 as the embedded platform development device under this detection task.

[0230] 2.2 Configure the Raspberry Pi system

[0231] First, use the pre-downloaded image file to install the Raspberry Pi operating system, write the Raspberry Pi system to the SD card, and the Raspberry Pi system configuration is complete. Then connect an external monitor, keyboard, and mouse to the Raspberry Pi, and you can see the initial interface of the Raspberry Pi on the monitor. Finally, click on the Raspberry Pi wifi settings, connect to the computer's mobile hotspot, and the network is successfully connected. You can also see the IP address of the Raspberry Pi by connecting to the computer's mobile hotspot.

[0232] 2.3 Raspberry Pi environment construction

[0233] (1) Hardware environment construction and system configuration: Set up the USB serial port communication method and use the image file downloaded in advance to install the Raspberry Pi operating system.

[0234] (2) Configure the Raspberry Pi deep learning environment: First, install the required deep learning environment according to the official requirements.txt file, and configure the pip tool and python runtime environment. Then use the pip tool to install the required related libraries such as opencv, numpy, torch, torchvision, etc.

[0235] 2.4 Construction of sonar image detection system

[0236] (1) Model transplantation: Import the trained new model and optimal weights into the Raspberry Pi.

[0237] (2) Sonar detection system process: After all the required files are deployed to the Raspberry Pi, the sonar image is transmitted to the Raspberry Pi through serial communication. The program is set so that the Raspberry Pi will immediately transmit the recognized sonar image back to the device after recognition.

[0238] 3. Experimental Verification and Analysis

[0239] 3.1 Training parameters of AC-YOLOX model

[0240] The improved model training parameters are shown in Table 1. The hardware equipment of this experiment uses RTX3090Ti graphics card, 24GB video memory, and Intel(R) Core(TM) i9-10980XE CPU@3.00GHz. The software environment is Windows10 operating system, and Pytorch is used as the deep learning framework, with the versions: Python3.7, Pytorch1.7.1, CUDA11.0. The input image size is 640×640, 4 images are input in a batch, the learning rate is 0.01, and the training is 100 cycles. In the last 15 cycles, data enhancement strategies such as masaic are cancelled.

[0241] Table 1 Improved model training parameters

[0242] Graphics NVIDIA RTX 3090 Ti Video Memory 24GB CPU Intel(R) Core(TM) i9-10980XE CPU@3.00GHz operating system Windows 10 Learning Framework Pytorch 1.7.1 Learning Rate 0.01 Training rounds 100 Batch size 16

[0243] 3.2 Model simulation test based on improved CSP_C

[0244] After integrating the CBAM attention mechanism into the CSP structure, the model shifts its focus from the complex environmental background of the sonar image to the characteristic target, effectively improving the accuracy of target detection. The performance comparison of the improved model detection accuracy and the original model is shown in Table 2.

[0245] Table 2 Performance comparison between the improved CSP_C model and the original model

[0246] Model mAP@50 / % mAP@5090 / % YOLOX 90.3 59.0 YOLOX+CSP_C 90.5 59.6

[0247] As can be seen from Table 2, the object detection accuracy has been improved, with an increase of 0.2% in mAP@50 and 0.6% in mAP@5090.

[0248] 3.3 Model simulation test based on adding ASFF module

[0249] The ASFF module is added between the neck and the head, and the adaptive attention mechanism is used to perform weighted fusion on the feature map of the sonar image, so as to better retain the important feature information of the sonar image, and the accuracy of target detection classification is effectively improved. The performance comparison of the improved model detection accuracy and the original model is shown in Table 3.

[0250] Table 3 Performance comparison between the model with added ASFF module and the original model

[0251] Model mAP@50 / % mAP@5090 / % YOLOX 90.3 59.0 YOLOX+ASFF 90.7 59.5

[0252] As can be seen from Table 3, the object detection accuracy has been improved, with an increase of 0.4% in mAP@50 and 0.5% in mAP@5090.

[0253] 3.4 AC-YOLOX model simulation test

[0254] The performance comparison of AC-YOLOX detection accuracy and the original model is shown in Table 4.

[0255] Table 4 Performance comparison between AC-YOLOX and YOLOX

[0256] Model mAP@50 / % mAP@5090 / % YOLOX 90.3 59.0 AC_YOLOX 90.9 59.2

[0257] As can be seen from Table 4, the object detection accuracy has been improved, with an increase of 0.9% in mAP@50 and 0.2% in mAP@5090.

[0258] 3.5 Comprehensive analysis of simulation results

[0259] The results of the four experiments are comprehensively analyzed, and the performance comparison is shown in Table 5.

[0260] Table 5 Performance comparison of sonar image target classification methods

[0261] Model mAP@50 / % mAP@5090 / % YOLOX 90.3 59.0 YOLOX+CSP_C 90.5 59.6 YOLOX+ASFF 90.7 59.5 AC_YOLOX 90.9 59.2

[0262] As can be seen from Table 5, the CBAM attention mechanism can better improve the model's ability to perceive the target, and the ASFF feature fusion mechanism can better integrate multi-scale features and improve the overall detection performance. When both are added to the YOLOX model at the same time, better overall performance can be achieved, with mAP@50 increased by 0.9% and mAP@5090 increased by 0.2%. Fig.13 shown.

[0263] IV. Conclusion

[0264] As an important underwater detection technology, sonar is widely used in marine exploration, underwater search, underwater robot navigation and other fields. With the continuous advancement of artificial intelligence technology, how to use AI technology to improve the efficiency and accuracy of sonar image analysis has become a hot research direction.

[0265] This Example 1 introduces a target detection and classification system based on sonar images. The system uses the YOLOX-S model as the basis of the detection algorithm, and achieves a significant improvement in detection accuracy by introducing the CBAM attention mechanism and the ASFF module, with mAP@50 and mAP@5090 indicators reaching 90.9% and 90.2% respectively.

[0266] In order to realize the embedded deployment of the system, this embodiment 1 deploys AC-YOLOX on the Raspberry Pi-5 and builds a real-time target detection and classification system based on sonar images. The system can realize the function of smooth operation and meet the needs of real-time detection of underwater unmanned vehicles.

[0267] In general, this embodiment 1 adopts advanced target detection algorithms and targeted optimization measures, which can realize real-time high-precision detection and classification of sonar images on underwater unmanned vehicles, providing strong support for applications such as seabed target recognition and underwater robot navigation.

[0268] The following is Example 2 of a specific application scenario of the present invention: This Example 2 takes an underwater unmanned vehicle (UUV) as an application scenario, and aims to provide UUV with autonomous perception and decision-making capabilities, achieve rapid detection and accurate classification of underwater targets, and provide technical support for application scenarios such as ocean exploration and underwater rescue.

[0269] This embodiment 2 builds a real-time target detection and classification system based on sonar images, which mainly includes the following modules:

[0270] 1. Sonar acquisition module: The sonar sensor installed on the UUV is responsible for collecting sonar image data of the underwater environment in real time and transmitting it to the processing module through wired or wireless communication.

[0271] 2. Target detection and classification module: Based on the improved AC-YOLOX deep learning network, it performs real-time target detection and classification on the received sonar images, and outputs information such as the target's location, size, and category.

[0272] 3. Decision-making control module: Based on the results of target detection and classification, combined with the navigation and positioning data of the UUV, it makes decisions such as autonomous navigation path planning and target tracking, and sends control instructions to the UUV execution system.

[0273] 4. Host computer monitoring module: Through wired or wireless communication, the sonar image data, target detection and classification results, and decision-making control information collected by the UUV are uploaded to the shore base station or mother ship in real time, providing intuitive underwater monitoring information for the operator.

[0274] The following will focus on the specific implementation process of the target detection and classification module:

[0275] 1. Data Preparation

[0276] First, the URPC2021 sonar dataset was collected, which contains a large number of real underwater target sonar images, covering different types of underwater targets. The dataset is divided into training set, validation set and test set in a ratio of 8:1:1. For each sonar image, the location coordinates, size and category label of the target are manually annotated as supervision information for training and evaluation.

[0277] 2. Network structure design

[0278] Based on the YOLOX network, we improved it and built the AC-YOLOX network structure. Specifically:

[0279] The CBAM attention mechanism is added before multiple residual structures of the dark3, dark4, and dark5 modules of the YOLOX network. CBAM includes channel attention mechanism and spatial attention mechanism, which can adaptively enhance the network's attention to important features and improve feature extraction capabilities. The channel attention mechanism captures channel-level importance information and dynamically adjusts channel weights through global pooling and MLP; the spatial attention mechanism extracts spatial attention maps through convolution operations to highlight the features of key areas.

[0280] In addition, an ASFF (Adaptive Feature Fusion) module is added between the Neck module and the Head module of the YOLOX network. ASFF can adaptively learn the fusion weights of features at different levels, enhance the fusion effect of multi-scale features, and improve the model's perception of targets. ASFF fuses features from different levels through constant scaling and adaptive weighting to improve the detection accuracy of multi-scale targets. The attention optimization equation group is used to further optimize the network's attention mechanism and improve the effect of feature extraction and fusion.

[0281] Through the above improvements, the AC-YOLOX network has stronger feature extraction and fusion capabilities, and can better capture the target features of sonar images in complex underwater environments.

[0282] 3. Data Augmentation Strategy

[0283] When training the AC-YOLOX network, two data enhancement strategies, Mosaic and Mixup, were used.

[0284] The Mosaic strategy randomly scales, crops, and arranges four training images to form a new training sample. This method can effectively increase the richness of training data and improve the generalization ability of the model. For scenes with low resolution and limited detail information such as sonar images, the Mosaic strategy can simulate more small targets and boundary targets through operations such as splicing and scaling, and enhance the model's detection ability for such targets.

[0285] The Mixup strategy is to perform weighted fusion of training images of different categories to form new training samples. This method can further expand the training data set and enhance the robustness of the model for target classification. For some fuzzy and overlapping target areas in sonar images, the Mixup strategy can help the model learn such special target features and improve classification accuracy.

[0286] By combining the two data enhancement methods, Mosaic and Mixup, the training data set was effectively expanded and the AC-YOLOX network's modeling capability for complex sonar images was enhanced.

[0287] 4. Model Optimization

[0288] When training the AC-YOLOX network, the following optimization strategies were adopted:

[0289] In terms of loss function design, in addition to the conventional regression loss, target loss and classification loss, L1 loss is also introduced to further optimize the difference between the predicted box and the true box.

[0290] In the specific implementation of the attention mechanism, strategies such as dynamic channel attention, frequency domain enhanced spatial attention, and cascade-parallel hybrid CBAM output are adopted to improve the effect of feature extraction and fusion. These optimization equations can adaptively adjust the parameters of the attention mechanism and enhance the model's perception of complex sonar images.

[0291] At the same time, other data enhancement strategies were also tried, such as random rotation and flipping, to further expand the training data set and improve the generalization performance of the model. In addition, by continuously adjusting network hyperparameters, such as learning rate and weight decay coefficient, better training convergence effect was achieved.

[0292] 5. Experimental Evaluation

[0293] The trained AC-YOLOX model is applied to the test set to evaluate its target detection and classification performance in complex underwater environments. The test set contains various types of underwater targets.

[0294] During the test, the confidence threshold was set to 0.5 and the category probability threshold was set to 0.6, and the output results of the model were screened to obtain the final target detection classification results. Then, these results were compared with the real annotations to calculate the four performance indicators of accuracy, precision, recall and F1 score.

[0295] The experimental results show that on the test set, the AC-YOLOX model has an accuracy rate of 90.9%, a precision rate of 91.2%, a recall rate of 90.5%, and an F1 score of 90.8%. These indicators are significantly improved compared with the YOLOX algorithm, indicating that the method proposed in this invention can effectively deal with various interference factors in complex underwater environments and improve the accuracy and robustness of target detection and classification.

[0296] In order to intuitively demonstrate the detection and classification effect of AC-YOLOX on different underwater target categories, the accuracy, recall rate and F1 score indicators of each category are statistically analyzed. The results are shown in Table 6:

[0297] Table 6 Test result indicators

[0298] Target Category Accuracy Recall F1 score human body 92.3% 91.7% 92.0% ball 90.8% 89.3% 90.0% Round Cage 90.5% 90.4% 90.4% Square Cage 91.1% 90.6% 90.8% tire 89.7% 90.1% 89.9% bucket 91.8% 91.2% 91.0% cube 91.5% 90.9% 90.6 Cylinder 90.7% 90.4% 89.9%

[0299] As can be seen from the table, the AC-YOLOX model has achieved high detection and classification accuracy for each category of targets, indicating that it has strong generalization ability and can adapt to different types of underwater target recognition needs.

[0300] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A real-time target detection and classification method based on sonar images, characterized in that: The following steps are involved: S10. Divide the URPC2021 sonar dataset into a training set, a validation set, and a test set in a ratio of 8:1:1, and establish labels according to the position, size, and category of the target in the sonar image as the training dataset; S20. Improve the YOLOX network structure and build an AC-YOLOX network. Specifically, add the CBAM attention mechanism before the multiple residual structures of dark3, dark4, and dark5 of the YOLOX network structure, and add the ASFF module of adaptive feature fusion between Neck and Head. S30, inputting the training set and the validation set into the AC-YOLOX network for training, and using Mosaic and Mixup data enhancement strategies to increase data diversity during the training process, wherein the input of the training is sonar image data, and the output of the training is the location coordinates, confidence, and category probability of the target; S40, using the trained model to predict the test set, outputting the classification results, and calculating the evaluation indicators of the model, including accuracy, precision, recall rate and F1 score; S50, adjusting the network structure, hyperparameters and training strategy of the AC-YOLOX network according to the evaluation index, repeating steps S20 to S50 until the model performance reaches the preset expected target, and saving it as the target real-time detection classification model; S60, using the target real-time detection and classification model, inputting the sonar image acquired in real time, and outputting the target real-time detection and classification result; The AC-YOLOX network includes an input end, a Backbone, a Neck and a Head; the AC-YOLOX network also includes an attention optimization equation group, which is used to further optimize the attention mechanism of the network and improve the effect of feature extraction and fusion; the attention optimization equation group includes a dynamic channel attention equation, a frequency domain enhanced spatial attention equation, a cascade-parallel hybrid CBAM output equation, a nonlinear multi-scale ASFF fusion equation, and a Bayesian ASFF weight normalization equation; The specific expression of the attention optimization equation group is as follows: Dynamic channel attention equation: ; In the formula, is the input feature map; is the time step; is a multi-layer perceptron; and They are global average pooling and maximum pooling respectively; is the sigmoid function; is the adaptive learning rate; is the partial derivative of channel attention with respect to time; Spatial attention equation for frequency domain enhancement: ; In the formula, It is the feature map after channel attention processing; It is a 7×7 convolution operation; and They are fast Fourier transform and its inverse transform respectively; is a frequency domain filter; is the frequency domain enhancement coefficient; Cascade-parallel hybrid CBAM output equation: ； In the formula, represents the Hadamard product; is the parallel branch weight coefficient; Nonlinear multiscale ASFF fusion equation: ； In the formula, For the Feature map after layer fusion; For the The layer feature map is resampled to Layer results; is a learnable parameter; is the gating function; and is the activation function; is the residual connection; Bayesian ASFF weight normalization equation: ； In the formula, For the given data Down The posterior probability of is the normalization constant; and is the mean and standard deviation of the prior distribution; and are learnable real and imaginary parameters; Is an imaginary unit.

2. The method for real-time target detection and classification based on sonar images according to claim 1 is characterized in that: The input end adopts Mosaic and Mixup data enhancement strategies.

3. The method for real-time target detection and classification based on sonar images according to claim 2 is characterized in that: The Backbone structure includes: Focus structure, CSP_C structure, and SPP structure; the CSP_C structure specifically integrates the CBAM attention mechanism into multiple residual structures of the CSP structure as an improved CSP_C structure, so that the model pays more attention to identifying target objects and learns more features.

4. The method for real-time target detection and classification based on sonar images according to claim 3 is characterized in that: The CBAM attention mechanism is added before multiple residual structures of dark3, dark4, and dark5 of the YOLOX network structure. Specifically, the CSP structure in dark3 and dark4 adds the CBAM attention mechanism before multiple residual structures, and the CSP structure in dark5 adds the CBAM attention mechanism after a CBS convolution operation.

5. The method for real-time target detection and classification based on sonar images according to claim 4 is characterized in that: The structure of Neck includes: PAFPN structure and ASFF structure.

6. The method for real-time target detection and classification based on sonar images according to claim 5 is characterized in that: The structure of the Head includes a Decoupled Head detection head, an Anchor-free mechanism, a Loss calculation, and a SimOTA tag matching strategy.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, which, when executed, are used to execute the real-time target detection and classification method based on sonar images as described in any one of claims 1 to 6.

8. A real-time target detection and classification system based on sonar images, characterized in that: A computer-readable storage medium comprising the method of claim 7.

Images