AI underwater target detection and positioning method based on sonar image

Abstract

The application relates to an AI underwater target detection and positioning method based on a sonar image, and comprises the following steps: collecting sonar image data of an underwater target, pre-processing data and enhancing data, so as to prepare for subsequent training and testing; training and testing underwater target detection; and simultaneously mapping an underwater environment based on a SLAM algorithm and positioning a target device. Based on the successful application of deep learning in underwater target detection and identification, the application can accurately segment the target from a complex marine environment background, and this direction will certainly lead to a technical breakthrough in the field of underwater target identification. Meanwhile, the successful application of deep learning in underwater SLAM will enable an autonomous underwater vehicle to more accurately understand the external environment of an operation area, so as to improve the automation level of underwater operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of underwater target positioning based on SLAM algorithm, and particularly relates to an AI underwater target detection and positioning method based on sonar images. BACKGROUND

[0002] With the increase of underwater operation tasks in the world, the collection of underwater information has become one of the important prerequisites for underwater operation. Since the underwater information collection equipment needs to be maintained and data recovered regularly, and the underwater environment is complex, the recovery of the underwater information collection equipment is a challenging task.

[0003] Traditional underwater target detection methods include image feature matching recognition, general image segmentation, and detection and recognition based on color and shape. Scholars in various countries have studied traditional methods for underwater targets, mostly using underwater artificial targets for underwater test verification. Although these methods are very fast, they are still not ideal for dynamic environments and are not accurate enough. Compared with traditional target detection methods, target detection algorithms based on deep learning are faster and have better robustness in the case of partial occlusion of the target. Therefore, it has gradually become the mainstream method of target detection. At present, deep learning-based algorithms can be divided into single-stage end-to-end algorithms and two-stage region proposal algorithms.

[0004] These deep learning-based detection methods can only get the position of the target in the image, not the map positioning of the target. Target positioning is another important research content for AUV to recover underwater information collection equipment. In these operations, cooperative targets or artificial targets are mostly used to improve positioning efficiency. Common underwater artificial objects usually have regular shapes and specific bright colors, such as underwater special patterns, active laser modules, 3D markers, etc., and the present application uses SLAM algorithm for positioning.

[0005] The underwater environment is much more complex than on land. In the absence of light sources underwater, the underwater video acquisition system must rely on artificial light sources for illumination. When light is transmitted in water, it will be severely attenuated due to absorption, reflection and scattering of water, etc. The collected underwater images will inevitably have limited visible range, blurred, low contrast, non-uniform illumination, color mismatch and noise, etc. resulting in a small target detection range and low accuracy.

[0006] The underwater positioning technology based on the release of markers requires human participation, and the equipment cannot be completed autonomously, which is low in work efficiency and cannot obtain global map information underwater. SUMMARY

[0007] To solve the problems in the prior art, the present application provides an AI underwater target detection and positioning method based on sonar images, comprising the following steps:

[0008] Step S1: sonar image data of underwater targets is collected, preprocessed and enhanced for subsequent training and testing;

[0009] Step S2: training and testing of underwater target detection are performed;

[0010] Step S3: the underwater environment is simultaneously mapped based on the SLAM algorithm, and the target device is positioned.

[0011] The step S2 comprises:

[0012] Step S21: based on the target detection network, the network model is modified, and the sonar image processed in step S1 is subjected to AI learning and training;

[0013] Step S22: the underwater target sonar image is tested by a target detection method based on deep learning.

[0014] The yolov5 network is used to perform AI learning and training on the sonar image processed in step S1.

[0015] The AI learning and training of the sonar image processed in step S1 using the yolov5 network comprises:

[0016] Step S211: a pre-trained yolov5 network is loaded, the feature extraction layer is frozen, the network output layer is modified, and the 80 classifications for visible light images are adjusted to single targets for sonar images;

[0017] Step S212: migration training and learning are performed, the feature extraction layer is unfrozen after the accuracy reaches the expected value, and the network is fine-tuned;

[0018] Step S213: the dimension of the feature extraction layer is reduced, the output result is subjected to loss calculation with the model result trained in step S212, and network distillation is completed.

[0019] In step S3, the algorithm provided by the ORB-SLAM3 architecture is used to simultaneously map the underwater environment and position the target device, and the sonar image feature extraction and matching training are performed based on SuperPoint and SuperGlue.

[0020] The step S3 comprises:

[0021] Step S31: based on the SuperPoint network, the underwater target sonar image set processed in step S2 is subjected to AI learning and training to extract feature points and descriptors.

[0022] Step S32: based on the SuperGlue model, based on the extracted feature points and descriptors, the sonar image is positioned and mapped.

[0023] The step S31 comprises:

[0024] Step S311: for the collected sonar image dataset, the pre-trained BaseDetector network is used to extract the corner points;

[0025] Step S312: the collected sonar image dataset is geometrically transformed to obtain a new picture, and a known pose relationship picture pair is obtained.

[0026] Step S313: load the pre-trained model SuperPoint and parameters, input the picture before the geometrically transformed picture into the network, extract the feature points and descriptors, and perform transfer learning according to the extracted feature points and descriptors and the network extracted corner points obtained in step S311.

[0027] Based on the successful application of deep learning in underwater target detection and recognition, the target can be accurately segmented from the complex marine environment background, and this direction will lead to a breakthrough in the field of underwater target recognition; at the same time, the successful application of deep learning in underwater SLAM will enable the autonomous underwater robot to more accurately understand the external environment of the working area, thereby improving the automation level of underwater operation. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The target network structure diagram of the application.

[0029] Figure 2 The program flow chart of the SLAM algorithm of the application.

[0030] Figure 3 The result demonstration diagram of the SLAM algorithm of the application.

[0031] Figure 4 The execution step diagram of the SuperPoint of the application.

[0032] Figure 5 The model structure diagram of the SuperGlue of the application.

[0033] Figure 6 Another model structure diagram of the SuperGlue of the application.

[0034] Figure 7 The low threshold feature extraction and matching result diagram of the application.

[0035] Figure 8The application adopts high threshold feature extraction and matching result graph.

[0036] Figure 9 The application is based on the method and the matching result comparison graph based on the existing LIFT and SIFT.

[0037] Figure 10 The underwater target result graph identified based on the method provided by the application. DETAILED DESCRIPTION

[0038] In order to have a further understanding of the technical solutions and beneficial effects of the application, the technical solutions of the application and the beneficial effects generated thereby will be described in detail below with reference to the drawings.

[0039] The application is improved based on the current international leading network model, and uses them for target detection, feature point recognition and matching of sonar images, and simultaneously uses a dynamic threshold method, so as to improve the target detection accuracy and performance, effectively shorten the model detection time, and simultaneously improve the accuracy of relative pose calculation between two frames, so as to provide accurate positioning information for subsequent map creation. Meanwhile, the sonar image used in the underwater environment can have a larger detection range and distance than the optical image, and is more suitable for the underwater environment.

[0040] In order to solve the problems existing in the prior art, for accurate recovery of underwater information collection equipment and improvement of underwater operation efficiency, the accuracy and speed of target detection and positioning are key factors. Meanwhile, obtaining target position information and attitude information is beneficial to the development of underwater operation. The first improvement point of the application is to accurately detect underwater targets by using sonar images. The sonar image can solve the defect that the existing underwater visible light detection distance is too short. However, in the aspects of position and orientation determination, the sonar image needs to overcome more technical obstacles than visible light imaging. The application proposes an AI underwater target detection and positioning method based on a sonar image.

[0041] I. The overall technical solution of the application

[0042] 1. Sonar image data collection, data preprocessing and data enhancement are performed on underwater targets, so as to prepare for subsequent training and testing.

[0043] 2. Training and testing are performed on underwater target detection, and specifically include the following sub-steps:

[0044] (1) Based on the target detection network, the network model is modified, and AI learning training is performed on the sonar image processed in step 1.

[0045] (2) The underwater target sonar image is tested by using a target detection method based on deep learning.

[0046] 3. Based on the SLAM algorithm, the underwater environment is mapped synchronously and the target device is positioned.

[0047] II. Underwater target detection

[0048] Figure 1 For the target detection network structure diagram, as shown in Figure 1 The second improvement point of the present application is that in step 2, the sonar image data is trained using the yolov5s network, and finally the target detection of the sonar image is realized. The network performance and accuracy are improved by improving the yolov5s network for transfer learning and distillation. The main improvements involve the following two points:

[0049] Firstly, the pre-trained yolov5s network is loaded, the feature extraction layer is frozen, the network output layer is modified, the network output is compressed, the 80 classifications for visible light images are adjusted to single targets for sonar images, and the transfer learning training is performed. After the accuracy reaches the expected value, the feature extraction layer is unfrozen, and the network is further fine-tuned.

[0050] Secondly, in order to meet the edge computing requirements and reduce the output dimension of the feature extraction layer, the output results are calculated with the loss of the model trained in the previous step, and the network distillation is completed.

[0051] III. SLAM algorithm

[0052] 1, Figure 2 and Figure 3 are the program flowchart and result demonstration diagram of the SLAM algorithm respectively. The third improvement point of the present application is that in the "TRACKING" step shown in Figure 2 SuperPoint and SuperGlue modified network combination scheme is adopted. SuperPoint obtains feature points and descriptors in the image by using deep learning, and SuperGlue matches the feature points and descriptors of two frames of images by using CNN and GNN, and completes the positioning and mapping of underwater devices.

[0053] Through testing and verification, compared with traditional algorithms, SuperPoint and SuperGlue have more correct matches and fewer incorrect matches when processing sonar images.

[0054] 2、The fourth improvement of the application is that in the process of feature extraction and matching training of the sonar image based on SuperPoint and SuperGlue, the SLAM architecture adopted is the latest ORB-SLAM3 architecture, the accuracy and robustness of the latest ORB-SLAM3 architecture are improved, and in addition, the ORB-SLAM3 architecture has a loop detection function. The inventor finds that there are some test data sets that have loops, and if the loop can be correctly detected, the cumulative error can be corrected to a great extent. The inventor performs loop detection through the bag-of-words algorithm, then removes the wrong loop through geometric verification, and then performs global BA optimization. In this way, the accuracy is further improved.

[0055] Figure 4 The execution steps of SuperPoint are as shown in Figure 4 The specific steps of SuperPoint are as follows:

[0056] 1. Use the collected sonar image data set to extract the corner points using the pre-trained BaseDetector network, and this step is called interest point self-labeling.

[0057] 2. Geometrically transform the picture used in the first step to obtain a new picture, so that there are picture pairs with known pose relationships.

[0058] 3. Load the pre-trained model SuperPoint and parameters, input the two pictures in the second step into the network, extract feature points and descriptors, and perform transfer learning.

[0059] The training adopts a self-supervised training method to extract feature points and descriptors, and the self-labeled interest points in the first two steps are used as pseudo-real values, and the feature points obtained by SuperPoint reasoning are used for loss function calculation.

[0060] The loss function includes two parts, one is the feature point detection loss, and the other is the descriptor detection loss. The feature point detection loss adopts a cross-entropy loss function for calculation:

[0061]

[0062] Among them,

[0063]

[0064] At this time, it is similar to a classification task, and the log operation inside is the probability of the elements in the cell being feature points (i.e. the value after softmax), that is, the sample X hwThe probability belonging to the feature. This is a 2D location classifier, each 8x8 range can only have one feature point, i.e. there are at most H*W / 64 feature points in the image.

[0065] The loss function calculation of the descriptor:

[0066]

[0067] Where l d is the hinge loss function

[0068] l d (d, d'; s) = λ d * s * max(0, m p -d T d') + (1-s) * max(0, d T d'-m n )

[0069] The indicator function is

[0070] P in the above formula is the center point coordinate of the cell.

[0071] When the positive example descriptor cosine similarity d T d' is greater than m p , there is no need for punishment at this time; but if the similarity is small, at this time it is punished; the negative sample is to make d T d' small, but the network performance is not good, the value may be large (greater than m n in the above formula), at this time the phenomenon is punished, the network weight is adjusted to make the loss decrease, and the corresponding descriptor similarity decreases.

[0072] Figure 5 and Figure 6 are the model structure diagrams of SuperGlue, as shown in Figure 4 and Figure 5 : SuperGlue is a feature matching network, its input is the feature points and descriptors (both handcrafted features and deep learning features) in two images, and the output is the matching relationship between image features.

[0073] The whole framework consists of two main modules: attention GNN and optimal matching layer. The attention GNN encodes the feature points and descriptors into a vector (which can be understood as a feature matching vector), and then uses self-attention and cross-attention to enhance (repeat L times) the feature matching performance of the vector f. Then enter the optimal matching layer, calculate the inner product of the feature matching vector to get the matching degree score matrix, and then use the Sinkhorn algorithm (iterate T times) to solve the optimal feature assignment matrix.

[0074] The purpose of feature matching is to solve the relative pose between two frames. The feature matching of SuperGlue is solved by solving the differentiable optimization transfer problem, and the loss function is constructed by GNN. SuperGlue proposes a flexible content aggregation mechanism based on attention mechanism, which can simultaneously perceive the potential 3D scene and perform feature matching. As shown in the above figure, compared with traditional and manually designed features, this algorithm can achieve the best result in underwater environment based on sonar image in pose estimation task. The network can achieve real-time on GPU, so it can be integrated into the SLAM algorithm.

[0075] Figure 7 For a lower threshold, the feature extraction and matching results of two sonar images: due to the rich changes in the scene, even in the same sequence, there are very dramatic changes, such as from strong texture to weak texture. The uneven distribution of such scenes will greatly affect the accuracy of the algorithm. Due to the rapid change of the scene, the number of feature points and matches in the same sequence will be extremely uneven, which will cause the BA link to lose accuracy.

[0076] To solve this problem, see Figure 8 The invention selects the probability of the cell element as a feature point (i.e. the value after softmax) inside the log operation in the above, that is, the sample X hw The threshold value of the probability of the feature is increased, and after increasing the threshold value, the number of matching points is reduced. Therefore, the inventor dynamically adjusts the threshold value and other parameters according to the change of the number of feature points extracted in the environment, so that the number of feature points and matches in each image is balanced, the influence of uneven distribution of feature points is reduced, and the accuracy of subsequent positioning and mapping is improved.

[0077] Another key point is the loop detection function (as described above, which is a function unique to the ORB-SLAM3 architecture). The inventor found that there are some test data sets that have loops, and if the loop can be correctly detected, the cumulative error can be corrected to a large extent. The inventor performs loop detection through the bag-of-words algorithm, then removes false loops through geometric verification, and then performs global BA optimization. In this way, the accuracy is further improved.

[0078] In the present application, the association between the features of two images is established by extracting feature points and nearest neighbor matching. One of the difficulties of underwater tasks is the turbid and constantly changing scene. Accurate extraction of feature points and establishment of correct matching between feature points will have a crucial impact on the results. Figure 9 For comparison of the matching results based on the method of the present application and based on the existing LIFT and SIFT, as shown in Figure 9 The feature point extraction of the existing artificial designed traditional algorithm cannot achieve the ideal effect in the complex environment, and the improved AI network successfully extracts the feature points and performs matching.

[0079] Figure 10 The underwater target result image recognized based on the method provided by the present application: Based on the successful application of deep learning in underwater target detection and recognition, the present application can accurately segment the target from the complex marine environment background. This direction will certainly lead to a breakthrough in the technology of underwater target recognition. At the same time, the successful application of deep learning in underwater SLAM will also enable the autonomous underwater robot to more accurately understand the external environment of the working area, thereby improving the automation level of underwater operation.

[0080] Although the present application has been described by the above preferred embodiments, it is not intended to limit the protection scope of the present application, and any person skilled in the art can make various changes and modifications to the above embodiments without departing from the spirit and scope of the present application, which is still within the scope of protection of the present application. Therefore, the protection scope of the present application is defined by the claims.

Claims

1. An AI-based underwater target detection and localization method based on sonar images, characterized in that, Includes the following steps: Step S1: Acquire sonar image data of underwater targets, perform data preprocessing and data augmentation to prepare for subsequent training and testing; Step S2: Train and test the underwater target detection; Step S3: Simultaneously map the underwater environment and locate the target device based on the SLAM algorithm; The sonar images processed in step S1 are used to perform AI learning and training using a YOLOv5 network, including: Step S211: Load the pre-trained YOLOv5 network, freeze the feature extraction layer, modify the network output layer, and adjust the 80 classifications for visible light images to a single target for sonar images. Step S212: Perform transfer learning. Once the accuracy reaches the expected value, unfreeze the feature extraction layer and fine-tune the network. Step S213: Reduce the dimension of the feature extraction layer, calculate the loss between the output result and the model result trained in step S212, and complete the network distillation; In step S3, the algorithm provided by the ORB-SLAM3 architecture is used to simultaneously map the underwater environment and locate the target device. At the same time, sonar image feature extraction and matching training are performed based on SuperPoint and SuperGlue. Step S3 includes: Step S31: Based on the SuperPoint network, perform AI learning training on the underwater target sonar image set processed in step S2 to extract feature points and descriptors; Step S32: Based on the SuperGlue model, localize and map the sonar image based on the extracted feature points and descriptors; Step S31 includes: Step S311: For the acquired sonar image dataset, use the pre-trained BaseDetector network to extract corner points; Step S312: Perform geometric transformation on the acquired sonar image dataset to obtain new images, resulting in image pairs with known pose relationships; Step S313: Load the pre-trained model SuperPoint and parameters, input the image before geometric transformation into the network, extract feature points and descriptors, and perform transfer learning based on the extracted feature points and descriptors, combined with the corner points extracted by the network obtained in step S311.

2. The AI ​​underwater target detection and localization method based on sonar images as described in claim 1, characterized in that: Step S2 includes: Step S21: Based on the target detection network, modify the network model and perform AI learning training on the sonar images processed in step S1; Step S22: Test the underwater target sonar image using a deep learning-based target detection method.

Images