Joint online data enhancement method and system for small sample sonar image

Abstract

The invention provides a joint online data enhancement method and system for a small sample sonar image. The method comprises the following steps: S1, constructing a data set; s2, based on data enhancement methods of Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random speckle noise enhancement, selecting an initial reference data enhancement method according to characteristics of the training set; s3, based on the initial reference data enhancement method, other data enhancement methods are linked to carry out enhancement processing on the training set; s4, training the deep learning model by using the enhanced training set, verifying the trained deep learning model by using the verification set, and obtaining the deep learning model with the highest precision; s5, evaluating the current deep learning model with the highest precision through the test set; and selecting a joint online data enhancement method capable of effectively improving the target detection and recognition precision as a final data enhancement method.

Description

Technical Field

[0001] The present invention relates to the technical field of sonar image target detection, and in particular to a joint online data enhancement method and system for small sample sonar images. Background Art

[0002] With the development of ocean exploration technology, sonar imaging technology plays an increasingly important role in underwater target detection, seabed topography mapping and other fields. Sonar images obtain the shape and position information of underwater objects through the reflection and scattering of sound waves in water. However, due to the complexity of the underwater environment and the uncertainty of sound wave propagation, sonar images are often interfered by noise and have low resolution, making it difficult to detect and recognize targets. In practical applications, it is often costly and difficult to obtain high-quality sonar image data, resulting in a limited amount of data for training target detection and recognition models based on sonar images, which is difficult to meet the model training requirements of advanced algorithms such as deep learning. In addition, the complexity of the underwater environment makes sonar images often subject to various interferences, resulting in unclear target contours in sonar images, which are easily confused with the background and difficult to distinguish. Many existing advanced data enhancement methods are proposed for a single angle, such as the angle of helping model training or enriching data diversity. There are few studies on the design of joint data enhancement methods that comprehensively consider multiple angles. In addition, most data enhancement methods are designed for the characteristics of optical images, without comprehensively considering the small sample, high noise and low resolution of sonar images. It is necessary to develop a joint online data enhancement method suitable for small sample sonar images. Summary of the invention

[0003] In view of the defects in the prior art, the object of the present invention is to provide a joint online data enhancement method and system for small sample sonar images.

[0004] A joint online data enhancement method for a small sample sonar image provided by the present invention includes:

[0005] Step S1: Acquire sonar images to construct a data set, and divide the data set into a training set, a validation set, and a test set;

[0006] Step S2: Based on the data enhancement methods of Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random speckle noise enhancement, the training set is analyzed, and the initial benchmark data enhancement method is selected according to the characteristics of the training set;

[0007] Step S3: Based on the selected initial benchmark data enhancement method, other data enhancement methods other than the benchmark data enhancement method are linked to enhance the training set to obtain an enhanced training set;

[0008] Step S4: Use the enhanced training set to train the deep learning model, and use the verification set to verify the trained deep learning model to obtain a deep learning model with the highest target detection and recognition accuracy;

[0009] Step S5: Evaluate the deep learning model with the highest target detection and recognition accuracy currently obtained through the test set; update other data enhancement methods except the baseline data enhancement method to enhance the training set, and repeat the triggering of steps S4 to S5 in sequence; update the initial baseline data enhancement method, and repeat the triggering of steps S3 to S5; select the joint online data enhancement method that can effectively improve the target detection and recognition accuracy as the final data enhancement method.

[0010] Preferably, the Mosaic enhancement comprises: using Mosaic enhancement to stitch images to increase the number of images in the training set and make the aspect ratio of the images close to 1;

[0011] The Mixup enhancement includes: using Mixup enhancement to perform random proportion synthesis on the target and background images of different regions to simulate the situation where the target is disturbed by different background noises.

[0012] Preferably, the random affine transformation enhancement includes: using random affine transformation enhancement to perform linear transformation on the image, including random flipping, rotation, scaling and translation operations, to enhance the diversity of the training set;

[0013] The use of random affine transformation to enhance the linear transformation of the image includes: transforming each image with a certain probability, and limiting the rotation angle, scaling and translation distance, so that the input image is transformed within a certain range, and the range values ​​obey a uniform distribution.

[0014] Preferably, the random speckle noise enhancement comprises: taking the background area of ​​the original sonar image and adding a plurality of different types of noise respectively, including: salt and pepper noise, Gaussian noise and speckle noise; and performing histogram statistics on the image pixel values ​​before and after adding the noise, selecting the γ distribution for fitting, and selecting the distribution closest to the noise type as the data enhancement for randomly added noise.

[0015] Preferably, the evaluating the deep learning model with the highest target detection and recognition accuracy currently obtained through the test set includes: evaluating the mAP value of the deep learning model with the highest target detection and recognition accuracy currently obtained through the test set.

[0016] A joint online data enhancement system for small sample sonar images provided by the present invention includes:

[0017] Module M1: Obtain sonar images to construct a data set, and divide the data set into a training set, a validation set, and a test set;

[0018] Module M2: Analyze the training set based on the data enhancement methods of Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random speckle noise enhancement, and select the initial benchmark data enhancement method according to the characteristics of the training set;

[0019] Module M3: Based on the selected initial benchmark data enhancement method, other data enhancement methods other than the benchmark data enhancement method are linked to enhance the training set to obtain an enhanced training set;

[0020] Module M4: Use the enhanced training set to train the deep learning model, and use the validation set to validate the trained deep learning model to obtain the deep learning model with the highest target detection and recognition accuracy;

[0021] Module M5: Evaluate the deep learning model with the highest target detection and recognition accuracy currently obtained through the test set; update other data enhancement methods except the baseline data enhancement method to enhance the training set and repeatedly trigger modules M4 to M5 in sequence; update the initial baseline data enhancement method and repeatedly trigger modules M3 to M5; select the joint online data enhancement method that can effectively improve the target detection and recognition accuracy as the final data enhancement method.

[0022] Preferably, the Mosaic enhancement comprises: using Mosaic enhancement to stitch images to increase the number of images in the training set and make the aspect ratio of the images close to 1;

[0023] The Mixup enhancement includes: using Mixup enhancement to perform random proportion synthesis on the target and background images of different regions to simulate the situation where the target is disturbed by different background noises.

[0024] Preferably, the random affine transformation enhancement includes: using random affine transformation enhancement to perform linear transformation on the image, including random flipping, rotation, scaling and translation operations, to enhance the diversity of the training set;

[0025] The use of random affine transformation to enhance the linear transformation of the image includes: transforming each image with a certain probability, and limiting the rotation angle, scaling and translation distance, so that the input image is transformed within a certain range, and the range values ​​obey a uniform distribution.

[0026] Preferably, the random speckle noise enhancement comprises: taking the background area of ​​the original sonar image and adding a plurality of different types of noise respectively, including: salt and pepper noise, Gaussian noise and speckle noise; and performing histogram statistics on the image pixel values ​​before and after adding the noise, selecting the γ distribution for fitting, and selecting the distribution closest to the noise type as the data enhancement for randomly added noise.

[0027] Preferably, the evaluating the deep learning model with the highest target detection and recognition accuracy currently obtained through the test set includes: evaluating the mAP value of the deep learning model with the highest target detection and recognition accuracy currently obtained through the test set.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The present invention provides a method for training small sample sonar images with joint online data enhancement. From the perspectives of model training, data diversity and noise interference resistance, multiple data enhancement methods are designed and used in combination, which can greatly help the model to be better trained, enrich the training data set, enhance the ability to resist noise interference, and improve the target detection and recognition performance of the deep learning model in small sample, high noise and low resolution environments;

[0030] 2. The present invention applies Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random speckle noise enhancement in sequence to the online training process of the deep learning model, which can improve the model's target detection and recognition accuracy in small sample, high noise and low resolution environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0032] Figure 1 Flowchart of the joint online data enhancement method for small sample sonar images.

[0033] Figure 2 This is a schematic diagram of the Mosaic enhancement and splicing of the forward-looking sonar image described in the present invention.

[0034] Figure 3 This is a graph showing the pixel value distribution results before and after noise is added to the forward-looking sonar image described in the present invention.

[0035] Figure 4 This is a diagram showing the effect of the forward-looking sonar image combined with online data enhancement described in the present invention. DETAILED DESCRIPTION

[0036] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0037] Example 1

[0038] To address the problem that small samples, high noise and low resolution of sonar images make it difficult to train high-performance deep learning target detection and recognition models, the present invention proposes a joint online data enhancement method and system for small sample sonar images.

[0039] The joint online data enhancement method for small sample sonar images is as follows: Figure 1 As shown, including:

[0040] Step S1: Acquire sonar images to construct a data set, and divide the data set into a training set, a validation set, and a test set;

[0041] Step S2: Based on the data enhancement methods of Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random speckle noise enhancement, the training set is analyzed, and the initial benchmark data enhancement method is selected according to the characteristics of the training set;

[0042] Step S3: Based on the selected initial benchmark data enhancement method, other data enhancement methods other than the benchmark data enhancement method are linked to enhance the training set to obtain an enhanced training set;

[0043] Step S4: Use the enhanced training set to train the deep learning model, and use the verification set to verify the trained deep learning model to obtain a deep learning model with the highest target detection and recognition accuracy;

[0044] Step S5: Evaluate the deep learning model with the highest target detection and recognition accuracy currently obtained through the test set; update other data enhancement methods except the baseline data enhancement method to enhance the training set, and repeat the triggering of steps S4 to S5 in sequence; update the initial baseline data enhancement method, and repeat the triggering of steps S3 to S5; select the joint online data enhancement method that can effectively improve the target detection and recognition accuracy as the final data enhancement method.

[0045] More specifically, Mosaic enhancement is used to alleviate the imbalance of positive and negative samples of sonar image targets during model training, and to increase the richness of background information in the sonar image dataset. Mixup is used to enhance mixed sonar images to simulate the situation where the target is interfered by clutter. Random affine transformation enhancement is used to simulate different scales and postures of targets to expand the diversity of the dataset. Random speckle noise enhancement is designed to simulate speckle noise in sonar images and improve the model's anti-interference ability.

[0046] Specifically, step S1 includes: using the forward-looking sonar image dataset of the acoustic track of the online perception competition of the National Underwater Robot Competition (URPC) in previous years, the dataset contains 10 types of targets, dividing it into a training set, a validation set and a test set, and performing statistical analysis on the training set targets and image characteristics.

[0047] Specifically, step S2 includes: obtaining the number of targets in a single image and the maximum length and width of the image through statistical analysis, determining the number of targets in a single image when the image is input to the network by using Mosaic enhanced splicing, and making the aspect ratio of the image close to 1, and using Mixup to randomly synthesize the target and background images in different regions.

[0048] Use affine transformation to enhance the linear transformation of the image, including random flipping, rotation, scaling and translation, etc. Before each image is input into the model, it is transformed within a certain range according to the target physical rules, and the probability of the transformation is set.

[0049] The background area of ​​the original sonar image is taken and various types of noise are added respectively. The image pixel values ​​before and after adding noise are statistically analyzed by histogram. The γ distribution is selected for fitting, and the one with the distribution closest to the noise type is selected as the data enhancement for randomly added noise.

[0050] A single-stage deep learning target detection network structure based on YOLO was constructed, and Mosaic enhancement and Mixup enhancement were determined as the benchmark data enhancement methods. Then, the deep learning model was trained by combining affine transformation enhancement and random speckle noise enhancement. After the training was completed, the model with the highest target detection and recognition accuracy on the validation set was selected for test set evaluation.

[0051] The present invention also provides a joint online data enhancement system for small sample sonar images. The joint online data enhancement system for small sample sonar images can be implemented by executing the process steps of the joint online data enhancement method for small sample sonar images, that is, those skilled in the art can understand the joint online data enhancement method for small sample sonar images as a preferred implementation of the joint online data enhancement system for small sample sonar images.

[0052] Example 2

[0053] Embodiment 2 is a preferred embodiment of Embodiment 1

[0054] A joint online data enhancement method for a small sample sonar image provided by the present invention includes:

[0055] Dataset preparation and analysis steps: A total of 8,400 images of the forward-looking sonar image dataset of the acoustic track of the online perception competition of the National Underwater Robot Competition (URPC) were used, including 7,600 annotated images and 800 unannotated images. The dataset contains 10 types of targets, namely cubes, spheres, cylinders, human models, tires, round cages, square cages, iron barrels, aircraft models, and ROVs. First, all data were semi-automatically cleaned and incorrect labels were corrected. Then, the distribution of the training, validation, and test sets was ensured to be consistent as much as possible, and the sonar image dataset was divided into training set, validation set, and test set according to the sonar parameters, including 6,800 training sets, 800 validation sets, and 800 test sets. The dataset was further analyzed to count the number, scale, and image aspect ratio of each type of target, to guide the selection of subsequent data enhancement methods and the refinement of enhancement method parameters.

[0056] Design steps for data enhancement method based on model training:

[0057] After statistical analysis, the average number of objects in a single image is 1.62, and the maximum aspect ratio of the image is 3.8. Mosaic enhancement is used to increase the number of objects in a single image when input to the network, and make the aspect ratio of the image close to 1. Figure 2 As shown in the schematic diagram of forward-looking sonar image mosaic enhancement, the specific method is to first create a square canvas with a length and width of 2S, and fill the pixel value with 0 to place the image to be stitched. Specify the stitching image size as S, randomly select four images from the data set, and to ensure that the target in the image is not deformed, fix the long side to S, and transform the short side in proportion to obtain four images to be stitched with a long side of S and a short side ≤ S. Figure 2 Randomly select the splicing center point center in the yellow box area with length and width S, Figure 2 a. Figure 2 b. Figure 2 c and Figure 2 d. Align the center points on the canvas one by one and stitch them together. If the center point is close to the upper left area, then Figure 2 a. Figure 2 b and Figure 2 The part of c that exceeds the canvas will be clipped. Figure 2 d Keep the original image. Similarly, if the center point is close to the lower right area, then Figure 2 b. Figure 2 c and Figure 2 The part of d that exceeds the canvas will be clipped. Figure 2 a The original image is retained. Mixup is further used to randomly synthesize the target and background images in different regions to simulate the situation where the target is disturbed by different background noises.

[0058] Design steps of data enhancement method based on physical rules:

[0059] Without changing the physical rules of the forward-looking sonar image target, affine transformation enhancement is used to perform linear transformations on the image, including random flipping, rotation, scaling and translation, etc., to increase the diversity of the training data set and improve the generalization performance of the model on the test set. The probability of random flipping, rotation, scaling and translation is set, and each image is transformed with a certain probability before being input into the model. The rotation angle, scaling scale and translation distance are restricted according to the physical meaning, so that the input image is transformed within a certain range, and the range values ​​are uniformly distributed.

[0060] Design steps of noise-based data enhancement method:

[0061] Take part of the background of the original forward-looking sonar image and add salt and pepper noise, Gaussian noise, and speckle noise, etc., and draw the original forward-looking sonar image and the forward-looking sonar image with noise. By comparison, preliminarily determine the noise type that is consistent with the noise distribution of the original forward-looking sonar image. Then, the pixel values ​​of the original forward-looking sonar image and the image with noise are histogram-statisticed, and the γ distribution is selected for fitting, such as Figure 3 a to Figure 3 As shown in Figure d, the pixel value distribution result diagram of the forward-looking sonar image before and after noise is added is plotted, the pixel value distribution of the sonar image after adding different types of noise and the pixel value distribution of the original image are analyzed, and the data enhancement method of adding speckle noise is selected according to the principle of closest distribution.

[0062] Joint online data augmentation order determination and test evaluation steps:

[0063] Construct a single-stage deep learning target detection network structure based on YOLO, select 1 to 2 data enhancement methods that have the greatest impact on target detection and recognition accuracy from a variety of data enhancement methods, fix them as the benchmark data enhancement methods, and then train with other data enhancement methods in different orders. The forward-looking sonar image data generated by a variety of data enhancement methods is directly input online into the deep learning target detection network training without occupying local memory space. The number of training rounds is 300, and the SGD optimizer is used. First, warm-up training is performed for several rounds, and then training is performed using the cosine annealing learning rate decay strategy. After training, the model with the highest target detection and recognition accuracy on the validation set is selected for test set evaluation. It was finally determined that the joint online data enhancement method that can effectively improve the accuracy of forward-looking sonar image target detection and recognition is to use Mosaic enhancement and Mixup enhancement as the benchmark, and add random affine transformation enhancement and random speckle noise enhancement in turn. The experimental results are shown in Table 1. Figure 4 As shown, this is a diagram showing the effect of the forward-looking sonar image combined with online data enhancement according to the present invention.

[0064] Table 1 mAP evaluation results of joint online data enhancement method

[0065]

[0066] It can be seen from Table 1 that, with Mosaic enhancement and Mixup enhancement as the benchmark, the mAP value of the test set reached 0.530. After further adding random affine transformation enhancement, the mAP value increased by 4.9%. After further adding random speckle noise, the mAP value increased by 0.7%. Therefore, the joint online data enhancement method for small sample sonar images proposed in the present invention effectively improves the accuracy of sonar image target detection and recognition.

[0067] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0068] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A joint online data enhancement method for small sample sonar images, characterized in that: include: Step S1: Acquire sonar images to construct a data set, and divide the data set into a training set, a validation set, and a test set; Step S2: Based on the data enhancement methods of Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random speckle noise enhancement, the training set is analyzed, and the initial benchmark data enhancement method is selected according to the characteristics of the training set; Step S3: Based on the selected initial benchmark data enhancement method, other data enhancement methods other than the benchmark data enhancement method are linked to enhance the training set to obtain an enhanced training set; Step S4: Use the enhanced training set to train the deep learning model, and use the verification set to verify the trained deep learning model to obtain a deep learning model with the highest target detection and recognition accuracy; Step S5: Evaluate the deep learning model with the highest target detection and recognition accuracy currently obtained through the test set; update the order of enhancing the training set by other data enhancement methods except the benchmark data enhancement method, and repeatedly trigger steps S4 to S5; Update the initial benchmark data enhancement method and repeatedly trigger steps S3 to S5; A joint online data enhancement method that can effectively improve the accuracy of target detection and recognition is selected as the final data enhancement method.

2. The method for joint online data enhancement of small sample sonar images according to claim 1, characterized in that: The Mosaic enhancement includes: using Mosaic enhancement to stitch images to increase the number of images in the training set and make the aspect ratio of the images close to 1; The Mixup enhancement includes: using Mixup enhancement to perform random proportion synthesis on the target and background images of different regions to simulate the situation where the target is disturbed by different background noises.

3. The method for joint online data enhancement of small sample sonar images according to claim 1, characterized in that: The random affine transformation enhancement includes: using random affine transformation enhancement to perform linear transformation on the image, including random flipping, rotation, scaling and translation operations, to enhance the diversity of the training set; The use of random affine transformation to enhance the linear transformation of the image includes: transforming each image with a certain probability, and limiting the rotation angle, scaling and translation distance, so that the input image is transformed within a certain range, and the range values ​​obey a uniform distribution.

4. The method for joint online data enhancement of small sample sonar images according to claim 1, characterized in that: The random speckle noise enhancement includes: taking the background area of ​​the original sonar image and adding a plurality of different types of noise, including salt and pepper noise, Gaussian noise and speckle noise; and performing histogram statistics on the image pixel values ​​before and after adding the noise, selecting the gamma distribution for fitting, and selecting the data with the distribution closest to the noise type as the data enhancement for the random added noise.

5. The method for joint online data enhancement of small sample sonar images according to claim 1, characterized in that: The evaluating the deep learning model with the highest target detection and recognition accuracy currently obtained through the test set includes: evaluating the mAP value of the deep learning model with the highest target detection and recognition accuracy currently obtained through the test set.

6. A joint online data enhancement system for small sample sonar images, characterized in that: include: Module M1: Obtain sonar images to construct a data set, and divide the data set into a training set, a validation set, and a test set; Module M2: Analyze the training set based on the data enhancement methods of Mosaic enhancement, Mixup enhancement, random affine transformation enhancement and random speckle noise enhancement, and select the initial benchmark data enhancement method according to the characteristics of the training set; Module M3: Based on the selected initial benchmark data enhancement method, other data enhancement methods other than the benchmark data enhancement method are linked to enhance the training set to obtain an enhanced training set; Module M4: Use the enhanced training set to train the deep learning model, and use the validation set to validate the trained deep learning model to obtain the deep learning model with the highest target detection and recognition accuracy; Module M5: Evaluate the deep learning model with the highest target detection and recognition accuracy currently obtained through the test set; update other data enhancement methods except the benchmark data enhancement method to enhance the training set and repeatedly trigger modules M4 to M5 in sequence; Update the initial benchmark data enhancement method and repeatedly trigger modules M3 to M5; A joint online data enhancement method that can effectively improve the accuracy of target detection and recognition is selected as the final data enhancement method.

7. The joint online data enhancement system for small sample sonar images according to claim 6, characterized in that: The Mosaic enhancement includes: using Mosaic enhancement to stitch images to increase the number of images in the training set and make the aspect ratio of the images close to 1; The Mixup enhancement includes: using Mixup enhancement to perform random proportion synthesis on the target and background images of different regions to simulate the situation where the target is disturbed by different background noises.

8. The joint online data enhancement system for small sample sonar images according to claim 6, characterized in that: The random affine transformation enhancement includes: using random affine transformation enhancement to perform linear transformation on the image, including random flipping, rotation, scaling and translation operations, to enhance the diversity of the training set; The use of random affine transformation to enhance the linear transformation of the image includes: transforming each image with a certain probability, and limiting the rotation angle, scaling and translation distance, so that the input image is transformed within a certain range, and the range values ​​obey a uniform distribution.

9. The joint online data enhancement system for small sample sonar images according to claim 6, characterized in that: The random speckle noise enhancement includes: taking the background area of ​​the original sonar image and adding a plurality of different types of noise, including salt and pepper noise, Gaussian noise and speckle noise; and performing histogram statistics on the image pixel values ​​before and after adding the noise, selecting the gamma distribution for fitting, and selecting the data with the distribution closest to the noise type as the data enhancement for the random added noise.

10. The joint online data enhancement system for small sample sonar images according to claim 6, characterized in that: The test set is used to evaluate the deep learning model with the highest target detection and recognition accuracy currently obtained, including: The mAP value of the deep learning model with the highest target detection and recognition accuracy currently obtained is evaluated through the test set.

Images