Joint online data augmentation method and system for small sample sonar images

Abstract

The application provides a kind of small sample sonar image joint online data enhancement method and system, comprising: step S1: constructing data set;Step S2: based on the data enhancement method of Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random spot noise enhancement, select the initial reference data enhancement method according to the characteristics of training set;Step S3: based on the initial reference data enhancement method, linkage other data enhancement methods are enhanced to training set;Step S4: the training set after enhancement is used to train deep learning model, the trained deep learning model is verified using verification set, and the most accurate deep learning model is obtained;Step S5: the current most accurate deep learning model is evaluated by test set;Select the joint online data enhancement method that can effectively improve the target detection and recognition accuracy as the final data enhancement method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sonar image target detection, in particular to a joint online data enhancement method and system for small sample sonar images. BACKGROUND

[0002] With the development of ocean exploration technology, sonar imaging technology plays an increasingly important role in underwater target detection, seabed topography mapping and other fields. Sonar images obtain the shape and position information of underwater objects through the reflection and scattering of sound waves in water. However, due to the complexity of underwater environment and the uncertainty of sound wave propagation, sonar images are often disturbed by noise, have low resolution, and cause difficulties in target detection and identification. In practical applications, it is often costly and difficult to obtain high-quality sonar image data, resulting in limited data for training target detection and identification models based on sonar images, which is difficult to meet the model training needs of advanced algorithms such as deep learning. In addition, the complexity of the underwater environment often causes the target outline in the sonar image to be unclear and easily confused with the background. Many existing advanced data enhancement methods are designed for a single angle, such as helping model training angle or enriching data diversity angle. There are few studies on joint data enhancement methods that consider multiple angles, and most data enhancement methods are designed for optical image characteristics without considering the characteristics of small sample sonar images such as small sample, high noise and low resolution. Therefore, it is necessary to develop a joint online data enhancement method suitable for small sample sonar images. SUMMARY

[0003] In view of the defects in the prior art, the purpose of the present application is to provide a joint online data enhancement method and system for small sample sonar images.

[0004] According to the joint online data enhancement method for small sample sonar images provided by the present application, the following steps are included:

[0005] Step S1: Obtain a sonar image to construct a data set, and divide the data set into a training set, a validation set and a test set;

[0006] Step S2: Based on the data enhancement methods of Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random speckle noise enhancement, analyze the training set, and select an initial reference data enhancement method according to the characteristics of the training set;

[0007] Step S3: Based on the selected initial reference data enhancement method, link other data enhancement methods except the reference data enhancement method to enhance the training set, and obtain the enhanced training set;

[0008] Step S4: training the deep learning model by using the enhanced training set, verifying the trained deep learning model by using the verification set, and obtaining the deep learning model with the highest target detection and recognition accuracy;

[0009] Step S5: evaluating the current obtained deep learning model with the highest target detection and recognition accuracy by using the test set; updating the sequence of the data enhancement methods other than the benchmark data enhancement method to enhance the training set, and repeatedly triggering steps S4 to S5; updating the initial benchmark data enhancement method, and repeatedly triggering steps S3 to S5; and selecting the joint online data enhancement method capable of effectively improving the target detection and recognition accuracy as the final data enhancement method.

[0010] Preferably, the Mosaic enhancement comprises: using the Mosaic enhancement to splice images to increase the number of images in the training set, and to make the aspect ratio of the images close to 1.

[0011] The Mixup enhancement comprises: using the Mixup enhancement to randomly synthesize the target and different regional background images at a certain proportion to simulate the case that the target is disturbed by different background noises.

[0012] Preferably, the random affine transformation enhancement comprises: using the random affine transformation enhancement to perform linear transformation on the images, including random flipping, rotation, scaling and translation operations, and enhancing the diversity of the training set.

[0013] The random affine transformation enhancement comprises: transforming each image at a certain probability, and limiting the rotation angle, the scaling scale and the translation distance, so that the input image is transformed within a certain range, and the range is selected to be subject to uniform distribution.

[0014] Preferably, the random speckle noise enhancement comprises: adding multiple different types of noises to the background regions of the original sonar images, including: salt and pepper noise, Gaussian noise and speckle noise; and performing histogram statistics on the pixel values of the images before and after adding the noises, selecting a gamma distribution for fitting, and selecting the distribution closest to the noise type as the data enhancement of the random noise addition.

[0015] Preferably, the evaluation of the current obtained deep learning model with the highest target detection and recognition accuracy by using the test set comprises: evaluating the mAP value of the current obtained deep learning model with the highest target detection and recognition accuracy by using the test set.

[0016] According to the joint online data enhancement system for small sample sonar images provided by the application, the system comprises:

[0017] Module M1: obtaining sonar images to construct a data set, and dividing the data set into a training set, a verification set and a test set.

[0018] Module M2: based on Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random spot noise enhancement, the data enhancement method is used to analyze the training set, and the initial reference data enhancement method is selected according to the characteristics of the training set;

[0019] Module M3: based on the selected initial reference data enhancement method, the other data enhancement methods except the reference data enhancement method are linked to enhance the training set, and the enhanced training set is obtained;

[0020] Module M4: using the enhanced training set to train the deep learning model, using the validation set to verify the trained deep learning model, and obtaining the deep learning model with the highest target detection and recognition accuracy;

[0021] Module M5: the current obtained target detection and recognition accuracy of the highest deep learning model is evaluated through the test set; the enhancement processing sequence of the training set of other data enhancement methods except the reference data enhancement method is updated, and the modules M4 to M5 are repeatedly triggered; the initial reference data enhancement method is updated, and the modules M3 to M5 are repeatedly triggered; the joint online data enhancement method which can effectively improve the target detection and recognition accuracy is selected as the final data enhancement method.

[0022] Preferably, the Mosaic enhancement includes: using Mosaic enhancement to splice images to increase the number of images in the training set, and to make the aspect ratio of the image close to 1.

[0023] The Mixup enhancement includes: using Mixup enhancement to randomly synthesize targets and different regional background images to simulate the case that the target is disturbed by different background noise.

[0024] Preferably, the random affine transformation enhancement includes: using random affine transformation enhancement to perform linear transformation on the image, including random flipping, rotation, scaling and translation operation, to enhance the diversity of the training set.

[0025] The random affine transformation enhancement includes: each image is transformed with a certain probability, and the rotation angle, scaling scale and translation distance are limited, so that the input image is transformed within a certain range, and the range is uniformly distributed.

[0026] Preferably, the random spot noise enhancement includes: taking the background area of the original sonar image to add multiple different types of noise, including: salt and pepper noise, Gaussian noise and spot noise; and histogram statistics are performed on the pixel values of the images before and after adding noise, and gamma distribution is selected for fitting, and the distribution closest to the noise type is selected as the data enhancement of random noise addition.

[0027] Preferably, the current target detection and recognition accuracy of the highest deep learning model obtained by the test set is evaluated, comprising: evaluating the mAP value of the current target detection and recognition accuracy of the highest deep learning model obtained by the test set.

[0028] Compared with the prior art, the present application has the following beneficial effects:

[0029] 1. The present application provides a method for training small sample sonar image online data enhancement, which combines multiple data enhancement methods from the aspects of model training, data diversity and adversarial noise interference, and uses them jointly, which can greatly help the model to train better, enrich the training data set, enhance the ability of adversarial noise interference, and improve the target detection and recognition performance of the deep learning model in the small sample, high noise and low resolution environment.

[0030] 2. The present application combines Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random spot noise enhancement in sequence and applies them to the online training process of the deep learning model, which can improve the target detection and recognition accuracy of the model in the small sample, high noise and low resolution environment. BRIEF DESCRIPTION OF DRAWINGS

[0031] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0032] Figure 1 Flow chart for the method of online data enhancement of small sample sonar image.

[0033] Figure 2 Mosaic enhancement splicing diagram of the forward-looking sonar image according to the present application.

[0034] Figure 3 Pixel value distribution result graph before and after noise addition of the forward-looking sonar image according to the present application.

[0035] Figure 4 Effect graph of the online data enhancement of the forward-looking sonar image according to the present application. DETAILED DESCRIPTION

[0036] The present application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, a number of changes and improvements can be made. These all belong to the protection scope of the present application.

[0037] Example 1

[0038] Aiming at the problem that small sample, high noise and low resolution of sonar images make it difficult to train a high-performance deep learning target detection and recognition model, the application provides a joint online data enhancement method and system for small sample sonar images.

[0039] The joint online data enhancement method for small sample sonar images comprises the following steps: Figure 1

[0040] Step S1: Obtain a sonar image construction dataset, and divide the dataset into a training set, a validation set and a test set;

[0041] Step S2: Based on the data enhancement method of Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random spot noise enhancement, analyze the training set, and select an initial reference data enhancement method according to the characteristics of the training set;

[0042] Step S3: Based on the selected initial reference data enhancement method, link other data enhancement methods except the reference data enhancement method to enhance the training set, and obtain an enhanced training set;

[0043] Step S4: Train a deep learning model using the enhanced training set, verify the trained deep learning model using the validation set, and obtain a deep learning model with the highest target detection and recognition accuracy;

[0044] Step S5: Evaluate the target detection and recognition accuracy of the current obtained deep learning model through the test set; update the enhancement processing sequence of the training set by other data enhancement methods except the reference data enhancement method, and repeat steps S4 to S5; update the initial reference data enhancement method, and repeat steps S3 to S5; select a joint online data enhancement method that can effectively improve the target detection and recognition accuracy as the final data enhancement method.

[0045] More specifically, Mosaic enhancement is used to alleviate the imbalance between positive and negative samples of sonar images in the model training process, and to increase the richness of background information of the sonar image dataset. Mixup enhancement is used to mix sonar images, simulating the case that the target is disturbed by clutter. Random affine transformation enhancement is used to simulate different scales and attitudes of the target, and to expand the diversity of the dataset. Random spot noise enhancement is designed to simulate the spot noise in the sonar image and improve the anti-interference ability of the model.

[0046] Specifically, the step S1 comprises: using the acoustic track forward-looking sonar image dataset of the online perception race of the national underwater robot competition (URPC) in previous years, which contains 10 types of targets, dividing it into a training set, a validation set and a test set, and statistically analyzing the characteristics of the training set targets and images. ​

[0047] Specifically, the step S2 comprises: statistical analysis obtains single image target quantity and image maximum length width determination adopts mosaic enhancement splicing image to increase the target number of single image input to the network, and makes the length-width ratio of the image close to 1, and uses Mixup to randomly synthesize the target and different area background image.

[0048] The affine transformation is used for linear transformation of the image, and specifically includes random flipping, rotation, scaling and translation.

[0049] The background area of the original sonar image is added with a plurality of different types of noise, and the pixel values of the images before and after adding the noise are histogram statistics, the gamma distribution is selected for fitting, and the distribution closest to the noise type is selected as the data enhancement of the random noise.

[0050] A single-stage deep learning target detection network structure based on YOLO is constructed, Mosaic enhancement and Mixup enhancement are determined as the benchmark data enhancement method, then affine transformation enhancement and random spot noise enhancement are combined to train the deep learning model, after the training is completed, the model with the highest target detection and recognition accuracy on the validation set is selected for test set evaluation.

[0051] The application also provides a small sample sonar image joint online data enhancement system, which can be realized by executing the process steps of the small sample sonar image joint online data enhancement method, that is, the small sample sonar image joint online data enhancement method can be understood as the preferred embodiment of the small sample sonar image joint online data enhancement system by those skilled in the art.

[0052] Embodiment 2

[0053] Embodiment 2 is a preferred example of embodiment 1

[0054] According to the small sample sonar image joint online data enhancement method provided by the application, the method comprises the following steps:

[0055] Dataset making and analysis steps: The online perception race acoustic track forward-looking sonar image dataset of the annual national underwater robot competition (URPC) contains 8400 images, including 7600 labeled images and 800 unlabeled images. The dataset contains 10 types of targets, including cube, sphere, cylinder, mannequin, tire, round cage, square cage, iron drum, airplane model, and ROV. First, semi-automatic data cleaning is performed on all data to correct incorrect annotations. Then, the sonar image dataset is divided into training, validation, and test sets as much as possible to ensure consistent distribution. According to the sonar parameters, the sonar image dataset is divided into training, validation, and test sets, with 6800 images in the training set and 800 images in each of the validation and test sets. Further analysis of the dataset includes counting the number of targets, scales, and image aspect ratios of each type of target to guide the selection of subsequent data augmentation methods and the fine-tuning of augmentation method parameters.

[0056] Model training-based data augmentation method design steps:

[0057] Statistical analysis shows that the average number of targets per image is 1.62, and the maximum aspect ratio of the image is 3.8. Mosaic is used to enhance the splicing of images to increase the number of targets in the input to the network and make the aspect ratio of the image close to 1. Figure 2 The Mosaic enhancement splicing diagram of the forward-looking sonar image is shown. The specific method is to first create a square canvas with a length and width of 2S, filled with pixel values of 0, for placing the images to be spliced. Specify the splicing image size as S, and randomly select four images from the dataset. To ensure that the targets in the image do not deform, fix the long side as S and scale the short side proportionally to obtain four images with a long side of S and a short side ≤ S. Second, randomly select a center point center in the yellow box area with a length and width of S as shown in Figure 2 Figure 2 a、 Figure 2 b、 Figure 2 c and Figure 2 d align the center point on the canvas in turn, if the center point is close to the upper left area, then Figure 2 a、 Figure 2 b and Figure 2 c will be cut off, Figure 2 d retains the original image. Similarly, if the center point is close to the lower right area, then Figure 2 b、 Figure 2 c and Figure 2 d will be cut off, Figure 3 a retains the original image. Further, Mixup is used to randomly synthesize targets and different region background images to simulate the situation where targets are disturbed by different background noises.

[0058] Physical rule-based data augmentation method design steps: ​

[0059] Without changing the physical rules of the forward-looking sonar image target, the image is linearly transformed using affine transformation enhancement, specifically including random flipping, rotation, scaling and translation, etc., to increase the diversity of the training data set and improve the generalization performance of the model on the test set. The probabilities of random flipping, rotation, scaling and translation are set. Each image is transformed with a certain probability before inputting the model, and the rotation angle, scaling scale and translation distance are limited according to the physical meaning, so that the input image is transformed within a certain range, and the range is uniformly distributed.

[0060] The steps of designing the noise-based data enhancement method are as follows:

[0061] Part of the background of the original forward-looking sonar image is added with pepper and salt noise, Gaussian noise and spot noise, etc. The original forward-looking sonar image and the forward-looking sonar image with noise are drawn, and the noise type consistent with the noise distribution of the original forward-looking sonar image is preliminarily determined by comparison. Then the pixel values of the original forward-looking sonar image and the image after adding noise are histogram statistics, and the gamma distribution is selected for fitting, as shown in Figure 3 a to Figure 4 d, the pixel value distribution result graph of the forward-looking sonar image before and after adding noise is drawn, the pixel value distribution of the sonar image after adding different types of noise and the pixel value distribution of the original image are analyzed, and the data enhancement method of adding spot noise is selected according to the principle of closest distribution.

[0062] The steps of determining the sequence of joint online data enhancement and test evaluation are as follows:

[0063] A single-stage deep learning target detection network structure based on YOLO is constructed, 1-2 data enhancement methods that have the greatest impact on target detection and recognition accuracy are selected from a variety of data enhancement methods, and are fixed as the benchmark data enhancement method. Then, according to different sequences, other data enhancement methods are combined to train, and the results of the forward-looking sonar image data generated by the combination of multiple data enhancement methods are directly input to the deep learning target detection network for training, without occupying local memory space. The number of training rounds is 300, the SGD optimizer is used, the model is preheated and trained for several rounds, and then the cosine annealing learning rate decay strategy is used for training. After training, the model with the highest target detection and recognition accuracy on the validation set is selected for test set evaluation. Finally, the joint online data enhancement method that can effectively improve the target detection and recognition accuracy of the forward-looking sonar image is determined as the benchmark of Mosaic enhancement and Mixup enhancement, and the random affine transformation enhancement and random spot noise enhancement are added in sequence. The experimental results are shown in Table 1. As shown in ​ , it is the effect diagram of the joint online data enhancement of the forward-looking sonar image according to the present application.

[0064] Table 1 mAP evaluation results of joint online data enhancement method

[0065]

[0066] From Table 1, it can be seen that, taking Mosaic enhancement and Mixup enhancement as the benchmark, the mAP value of the test set reaches 0.530, and after further adding random affine transformation enhancement, the mAP value is increased by 4.9%, and after further adding random speckle noise, the mAP value is increased by 0.7%, therefore, the joint online data enhancement method for small sample sonar image proposed in the application effectively improves the sonar image target detection and recognition accuracy.

[0067] Those skilled in the art know that, in addition to implementing the system provided by the application and each device, module and unit thereof in the form of pure computer readable program code, the system provided by the application and each device, module and unit thereof can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps to achieve the same functions. Therefore, the system provided by the application and each device, module and unit thereof can be considered as a hardware component, and the devices, modules and units included therein for achieving various functions can also be considered as structures within the hardware component; the devices, modules and units for achieving various functions can also be considered as both software modules for implementing methods and structures within hardware components.

[0068] The specific embodiments of the application are described above. It should be understood that the application is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the application. In the case of no conflict, the embodiments of the application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for joint online data augmentation of small sample sonar images, characterized in that, The method comprises the steps of: Step S1: obtaining a sonar image construction dataset and dividing the dataset into a training set, a verification set and a test set; Step S2: analyzing the training set based on a data enhancement method of Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random spot noise enhancement, and selecting an initial reference data enhancement method according to the characteristics of the training set; Step S3: based on the selected initial reference data enhancement method, the other data enhancement methods except the reference data enhancement method are linked to enhance the training set, and an enhanced training set is obtained; Step S4: training a deep learning model using the enhanced training set, verifying the trained deep learning model using the verification set, and obtaining a deep learning model with the highest target detection and recognition accuracy; Step S5: evaluating the current obtained deep learning model with the highest target detection and recognition accuracy through the test set; updating the sequence of enhancing the training set by the other data enhancement methods except the reference data enhancement method, and repeating steps S4 to S5; updating the initial reference data enhancement method, and repeating steps S3 to S5; selecting a joint online data enhancement method capable of effectively improving the target detection and recognition accuracy as the final data enhancement method; The random spot noise enhancement comprises: adding multiple different types of noises to the background area of the original sonar image, including: salt and pepper noise, Gaussian noise and spot noise; and performing histogram statistics on the image pixel values before and after adding the noise, selecting a gamma distribution for fitting, and selecting the corresponding noise type closest to the gamma distribution before and after adding the noise as the data enhancement of random noise addition.

2. The method of claim 1, wherein, The Mosaic enhancement comprises: using Mosaic enhancement to splice images to increase the number of images in the training set, and to make the aspect ratio of the images close to 1; The Mixup enhancement comprises: using Mixup enhancement to randomly synthesize targets and different region background images to simulate the case that the target is disturbed by different background noises.

3. The method of claim 1, wherein, The random affine transformation enhancement comprises: using random affine transformation enhancement to perform linear transformation on the image, including random flipping, rotation, scaling and translation operations, to enhance the diversity of the training set; The random affine transformation enhancement comprises: using random affine transformation enhancement to perform linear transformation on the image, including random flipping, rotation, scaling and translation operations, to enhance the diversity of the training set; 4. The method of claim 1, wherein, The random affine transformation enhancement comprises: using random affine transformation enhancement to perform linear transformation on the image, including random flipping, rotation, scaling and translation operations, to enhance the diversity of the training set; 5. A joint online data augmentation system for small sample sonar images, characterized in that, The evaluation of the current obtained deep learning model with the highest target detection and recognition accuracy through the test set comprises: evaluating the mAP value of the current obtained deep learning model with the highest target detection and recognition accuracy through the test set. The method comprises the steps of: Module M1: obtaining a sonar image construction dataset and dividing the dataset into a training set, a verification set and a test set; Module M2: analyzing the training set based on a data enhancement method of Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random spot noise enhancement, and selecting an initial reference data enhancement method according to the characteristics of the training set; Module M3: based on the selected initial benchmark data augmentation method, the training set is enhanced by linking other data augmentation methods except the benchmark data augmentation method to obtain an enhanced training set; Module M4: the deep learning model is trained using the enhanced training set, and the trained deep learning model is verified using the validation set to obtain a deep learning model with the highest target detection and recognition accuracy; Module M5: the current obtained target detection and recognition accuracy highest deep learning model is evaluated through the test set; update the enhancement processing sequence of the training set by other data augmentation methods except the benchmark data augmentation method to repeat modules M4 to M5; Update the initial benchmark data augmentation method and repeat modules M3 to M5; Select the joint online data augmentation method that can effectively improve the target detection and recognition accuracy as the final data augmentation method; The random speckle noise enhancement includes: adding multiple different types of noise to the background area of the original sonar image, including: salt and pepper noise, Gaussian noise and speckle noise; and histogram statistics are performed on the image pixel values before and after adding noise, and the corresponding noise type with the closest γ distribution before and after adding noise is selected as the data enhancement of random noise addition.

6. The system according to claim 5, wherein, The Mosaic enhancement includes: using Mosaic enhancement to splice images to increase the number of images in the training set, and to make the aspect ratio of the image close to 1; The Mixup enhancement includes: using Mixup enhancement to randomly synthesize targets and different regional background images to simulate the case that the target is disturbed by different background noise.

7. The system according to claim 5, wherein, The random affine transformation enhancement includes: using random affine transformation enhancement to perform linear transformation on the image, including random flipping, rotation, scaling and translation operations, to enhance the diversity of the training set; The random affine transformation enhancement includes: using random affine transformation enhancement to perform linear transformation on the image, including random flipping, rotation, scaling and translation operations, to enhance the diversity of the training set; 8. The system according to claim 5, wherein, The random affine transformation enhancement includes: using random affine transformation enhancement to perform linear transformation on the image, including random flipping, rotation, scaling and translation operations, to enhance the diversity of the training set; The evaluation of the current obtained target detection and recognition accuracy highest deep learning model through the test set includes: evaluating the mAP value of the current obtained target detection and recognition accuracy highest deep learning model through the test set.

Images